OpenStack Havana over IPv6

© 2013 nephos6 and/or its affiliates. All rights reserved.

OpenStack Havana On IPv6

Shixiong Shang Randy Tuttle Ciprian Popoviciu !Version 1.9.3


§ Introduction

§ IPv6 and Cloud

§ IPv6 Refreshment

§ Proof of Concept

§ Proposed Blueprint

§ Next Steps

Agenda

�2


§ Nephos6 – Service assurance company

– Founded in June, 2011

– Twitter: @Nephos6

– Web: http://www.nephos6.com

!§ Shixiong Shang

– Head of Engineering

– Twitter: @shshang

– Email: [email protected]

Introduction

�3

§ Ciprian Popoviciu – Founder, CEO

– IPv6 expert

– Twitter: @Nephos6


!§ Randy Tuttle

– Network Consulting Engineer

– Twitter: @randyttl


http://www.nephos6.com

mailto:[email protected]




IP Comparison

�4

IPv4 IPv6Address 32-bit,

Network Address Translation128-bit, Multiple Scopes

ICMP ICMP ICMPv6

Autoconfiguration DHCP SLAAC, DHCPv6, DHCP-PD

Routing RIPv2, OSPFv2, ISIS, MP-BGP, EIGRP

RIPng, OSPFv3, ISIS-ST/MT, MP-BGP, EIGRPv6

IP Multicast IGMP/PIM/Multicast BGP MLD/PIM/Multicast BGP, Scope Identifier

“IPv6 Is an Evolution, Not a Revolution of the Internet Protocol”


IPv6 and Cloud

�5

IPv6 Strength Business Value

Sufficient address space

Direct access to resources

Simplified Address Assignment

Native support of multicast and flow labelNew architectural models

“The promise of Cloud cannot be fully met without IPv6”

}}

Great opportunity for innovation

Easier management and lower operational cost


IPv6 Address Auto-Configuration

�6

SLAAC* DHCPv6

Address Assignment (non-link-local)

By exchanging Router Solicitation and Router Advertisement messages with neighboring routers.

From DHCPv6 server

Additional Information None From DHCPv6 server

Default Gateway The only way to announce default route is using Router Advertisement!

Pros Plug and play IPv4-like approach, but better More control

Cons Doesn’t provide Hostname, DNS server, WINS, etc.

Operational overhead (extra DHCP server, HA, etc.)

Our focus today! Working in progress!

* StateLess Address AutoConfiguration


§ RFC 4861 - “Neighbor Discovery for IP Version 6 (IPv6)” and RFC 4862 - “IPv6 Stateless Address Autoconfiguration”

§ Rely on ICMPv6 (IPv6 control plane!)

SLAAC

�7

Router Solicitation (RS)

Router Advertisement (RA) subnet prefix lifetime autoconfig flag

Router Solicitation (RS)

ICMPv6 Type 133

IPv6 Source A Link Local

IPv6 DestinationLink-local scope all-routers address (FF02::2)

Router Advertisement (RA)

ICMPv6 Type 134

IPv6 Source A Link Local

IPv6 DestinationLink-local scope all-nodes address (FF02::1)

§ VM sends Router Solicitation at boot time to solicit Router Advertisement

§ Default route points to router’s link-local address

§ Router sends RA to all-nodes address periodically

§ Router can also unicast RA back to VM upon receiving RS

Host Router


§ IPv6 SLAAC = network portion (i.e. /64 Prefix in RA) + interface id (i.e. EUI64)

SLAAC Address Calculation

�8

FA 16 3E 73 83 D9

FA 16 3E FF FE 73 83 D9

1111 1010

1111 1000

F8 16 3E FF FE 73 83 D9

MAC

2001:7:10:180:F816:3EFF:FE73:83D9

Insert 0xFFFE in the middle

Change 7th bit in OUI part

IPv6 address =

EUI-‐64


OpenStack IPv6 Readiness

�9

OpenStack Havana OpenStack Icehouse

Limited IPv6 support out of box Neutron will support IPv6…

Neutron IPv6 roadmap is still in preliminary stage

Blueprint: IPv6 Feature Parity (working in progress…)

No clear IPv6 roadmap for other OpenStack projects

Neutron-IPv6-Subteam (ongoing)

Very limited documentation

Biggest risk of all: IPv4 way of thinking


Proof Of Concept

�10

Success with

both Grizzly and

Havana!

Motivation Goals

We are believers All OpenStack infrastructure nodes should be able to communicate with each other by IPv6

What it is v.s. What it should be OpenStack should be able to spin up dual-stack VMs in multi-tenant environment

We are doers…but we are not hackers, or developers :)

VMs should be able to gain connectivity to external IPv6 network beyond OpenStack’s control

Mission Statement: To make these two inflection points, IPv6 and Cloud work together seamlessly!


POC Architecture

�11

eth0

dnsmasq

neutron-openvswitch-agent

neutron-dhcp-agent

neutron-l3-agent

Network Node

openvswitch

neutron-openvswitch-agent

Compute Node

Tenant Data Networks(Tenant 1: VLAN 511)(Tenant 2: VLAN 512)

Tenant 1 External Network172.26.184.0/242001:172:26:184::/64

Router

glance

cinder

neutron-server

nova-consoleauth

Controller Node

nova-scheduler

nova-cert

nova-api

nova-conductor

7.10.180.1012001:7:10:180::101

eth0

keystone

horizon

mysql db

rabbitmq

Common Node

openvswitch nova-compute

eth0 eth1 eth2eth3

vlan 511vlan 512

eth0eth3

vlan 511vlan 512

Management and API network 7.10.180.0/242001:7:10:180::/64

7.10.180.1022001:7:10:180::102

7.10.180.1032001:7:10:180::103

7.10.180.1042001:7:10:180::104

Tenant 2 External Network172.26.185.0/242001:172:26:185::/64

nova-novncproxy

neutron-metadata-agent

Management and API network

External Network

Data Network

© 2013 nephos6 and/or its affiliates. All rights reserved. �12

1. All OpenStack infrastructure nodes should be ab le to communicate with each other by IPv6

-‐ IT IS ALL ABOUT CONFIGURATION


Enable IPv6 On Infrastructure

�13

Nodes Components Configuration Files Field Value

Common

Keystone /etc/keystone/keystone.conf bind_host 2001:7:10:180::101

MySQL DB /etc/mysql/my.cnf bind-address ::

Apache /etc/apache2/ports.conf Listen 80

Controller

Nova /etc/nova/nova.conf

my_ip

use_ipv6

osapi_compute_listen

metadata_listen

novncproxy_host

2001:7:10:180::102

true

2001:7:10:180::102

7.10.180.102

2001:7:10:180::102

Glance/etc/glance/glance-api.conf

bind_host

registry_host

2001:7:10:180::102

net-glance.sandbox.com/etc/glance/glance-registry.conf bind_host 2001:7:10:180::102

http://net-glance.sandbox.com


Enable IPv6 On Infrastructure

�14

Nodes Components Configuration Files Field Value

ControllerCinder /etc/cinder/cinder.conf

my_ip

glance_host

osapi_volume_listen

2001:7:10:180::102

2001:7:10:180::102

2001:7:10:180::102

Neutron /etc/neutron/neutron.conf bind_host 2001:7:10:180::102

Network Neutron /etc/neutron/neutron.conf bind_host 2001:7:10:180::103

ComputeNova /etc/nova/nova.conf

my_ip

use_ipv6

osapi_compute_listen

metadata_listen

novncproxy_host

2001:7:10:180::102

true

2001:7:10:180::102

7.10.180.102

2001:7:10:180::102

Neutron /etc/neutron/neutron.conf bind_host 2001:7:10:180::103

© 2013 nephos6 and/or its affiliates. All rights reserved. �15

2. OpenStack should be able to spin up dual-‐stack VMs in multi-‐tenant environment

-‐ IT IS ALL ABOUT IPV6 ADDRESS ASSIGNMENT


Neutron Tenant Network Provisioning

�16

neutron router-create --tenant-id tenant2-id router2

!neutron net-create --tenant-id tenant2-id net2_192_168_2 --provider:network_type vlan --provider:physical_network physnet3 --provider:segmentation_id 512

!neutron subnet-create --tenant-id tenant2-id --ip-version 4 --name sub2_192_168_2 net2_192_168_2 192.168.2.0/24

neutron subnet-create —tenant-id tenant2-id --ip-version 6 --name sub2_2001_192_168_2 net2_192_168_2 2001:192:168:2::/64

!neutron router-interface-add router2 sub2_192_168_2

neutron router-interface-add router2 sub2_2001_192_168_2

Specify IP version 6

IPv6 tenant subnet

Port is associated with tenant subnet


Neutron Tenant Network

�17

ns-‐74f270ff-‐01 (192.168.2.2)

qr-‐2f573f07-‐d9 (192.168.2.1)

qr-‐6dbfb73d-‐89 (2001:192:168:2::1)

br-‐eth2

eth2

br-‐eth3

eth3

br-‐int

br-‐eth3

eth3

tap74f270ff-‐01

br-‐int

To External Network

Netw

ork Node

Compute N

odetap-‐intf

qdhcp namespace

Tenant 2 Network

3. Need dnsmasq to send RA from default gateway interface

VM 192.168.2.3 (ipv6 address)

1. Need ip6tables filter rules to enable ICMPv6 at inbound direction

dnsmasq binding interface (ipv4)

RA

2. OpenStack needs to know this self-‐calculated IPv6 SLAAC address…

qrouter namespace

Default Gateway Interface (ipv4)

Default Gateway Interface (ipv6)


Enable RA Within Router Namespace

�18

§ Method “spawn_process” in neutron.agent.linux.dhcp.py on Network Node

Enable dnsmasq with RA and SLAAC

Derive router’s namespace and gateway interface

Specify IPv6 DHCP range. Taken from CLI

Bind to IPv6 qr-‐ interface

Launch dnsmasq in router’s namespace

Add IP version check

© 2013 nephos6 and/or its affiliates. All rights reserved.© 2013 nephos6 and/or its affiliates. All rights reserved. �19

3. VMs should be able to gain connectivity to external IPv6 network beyond OpenStack’s control

-‐ Support dual-‐stack on a single external interface -‐ Utilize existing VLAN/Segmentation ID !-‐ Eliminate NAT and GARP for IPv6 subnets


§ Option #1: Use next-hop RA and SLAAC to allow external GW interface defined IPv6 address

§ Option #2: Statically assign IPv6 address to external GW interface for the router – neutron router-gateway-set router2 ext-net-185

Dual-Stack options

�20


Neutron External Network

�21

ns-‐74f270ff-‐01 (192.168.2.2)

qr-‐2f573f07-‐d9 (192.168.2.1)

qr-‐6dbfb73d-‐89 (2001:192:168:2::1)

br-‐eth2

eth2

br-‐eth3

eth3

br-‐int

br-‐eth3

eth3

tap74f270ff-‐01

br-‐int

To External Network

tap-‐intf

Namespace: qrouter-‐94662c71-‐bf80-‐4c2f-‐9841-‐09a2112e3f58

Namespace: qdhcp-‐bfc3d877-‐ 44b6-‐4879-‐a83e-‐d37455e77f71

Tenant 2 Network


VM 192.168.2.3

(2001:192:168:2::1)

Need ip6tables filter rules to enable ICMPv6 at inbound direction


qg-‐3dac3be9-‐1b (172.26.185.70)

(SLAAC or statically assigned)

Disable NAT and GARP for IPV6

Netw

ork Node

Compute N

ode

RA


§ For Option #2, there exists a limitation on static IP address assignment for dual-stack implementation.

§ The L3 (server and agent) only allows a single IP address per network (VLAN) within the Linux namespace representing the tenant's router.

§ This limitation precluded the possibility of a dual-stack arrangement utilizing static assignments without code changes.

Dual-stack options

�22


Dual-stack solution

�23

To accomplish a static dual-‐stack arrangement, ip_version, cidr, ip_address and gateway_ip, was essential for L3 agent to build dual-‐stack interface inside router’s namespace.


§ For the tenant router, learn the default route from the upstream router through RA. When adding an external gateway

– net.ipv6.conf.<gateway_interface>.accept_ra=2

– net.ipv6.conf.<gateway_interface>.forwarding=1

– net.ipv6.conf.<gateway_interface>.accept_ra_defrtr=1

§ Prevent learning a default route from RA from internal tenant network

– net.ipv6.conf.<internal_interface>.accept_ra_defrtr=0

§ When the subnet assigned is an IPv6, don’t apply NAT configuration or perform GARP.

Dual-stack configuration

�24


Summary

�25

Findings FixesRA is not sent to IPv6 enabled internal tenant network by default

Enable RA on dnsmasq

DHCP process is bound to interface other than default gateway of tenant network

Launch dnsmasq process inside router namespace

IPv6 address chosen by OpenStack is not based on SLAAC standard

Calculate VM’s IPv6 address based on unique MAC address

Neighbor Discovery packet is dropped by ip6tables filter rules

Add ip6tables rules to allow ND related ICMPv6 packets

NAT and GARP are turned on for IPv6 subnets. Not desirable!

Only perform NAT and GARP for IPv4 subnets

Whitepaper: http://www.nephos6.com/pdf/OpenStack-Havana-on-IPv6.pdf


§ From openstack-dev mailer: – Short term, my goal is to get provider networks up and running, where

instances can get RA's from an upstream router outside of OpenStack and configure themselves.

– Medium term, we want to make dnsmasq configuration more flexible.

– More long term, I'd like to make it so that if there is an upstream router doing RA's - Neutron should send a PD automatically on network creation, and populate a subnet from the response given by the upstream router.

§ Service Provider focused; may not work entirely with L3 Agent without revisions

§ Integrate this PoC work with Blueprint to address broader OpenStack community and address L3 Agent

Proposed Blueprint

�26


Our Next Step

�27

Tactical Strategical

DHCPv6 IPv6 mindset

Migration Strategy IPv6 understanding / education

SLAAC + DHCPv6 Participation in IPv6 + Cloud efforts

Support for dual-stack infrastructure

Icehouse release validation


Technology

OpenStack Havana over IPv6