Software Defined Network in OpenStack ～OpenStack Neutron～

Icehouse Update and Towards Juno

Akihiro Motoki (@ritchey98)

Jun 12th, 2014

Who am I? • Akihiro Motoki / 元木 顕弘

– @ritchey98, amotoki on freenode IRC – NEC Knowledge Discovery Research Laboratories – Long experiences in developing network equipments/applications

• IP routers, wide-area Ethernet • Hardware spam filter appliance, FPGA-based IDS

– Recently working on OpenFlow, OpenStack and related testbeds

• OpenStack Developer

– Neutron and Horizon Core Developer – I18N (Internationalizationn) team sub-lead

• Linux JM (Japanese manuals) Project Maintainer – Translating manpages from Linux documentation project (LDP), GNU

tools and Linux networking tools

2013/12/12 Okinawa OpenDays 2013 2

Agenda

• Concept of Neutron API abstraction

– Layer2/Layer3 abstraction

– Advanced network services and Service Insertion

• Icehouse Update

• Topics in “Juno” (next release)

• Recent topics in the development community

??? Neutron

OpenStack Project と Neutron

Compute

Storage

Network

Nova

Swift (Objects)

Glance (Images)

Cinder (Block)

Identity Keystone

*-as-a-Service Capability OpenStack Service

Neutron Goal #1

• Provide Network-as-a-Service (NaaS)

– To allow tenants to control their own private networks

• Ability to create “multi-tier” networks

• Control IP addressing (IP address overlapping)

– Neutron API for operating logical networks

• Separate logical operations and backend provisioning

• Backend technologies are provisioned/configured by Neutron plugins/drivers.

Neutron Goal #2

• Support multiple emerging network technologies – New requirements never before in cloud data centers

• multi tenancy, remote data center, VM mobility, advanced network services, …

– Nova provides only legacy basic technology (VLAN + iptables)

– Needs new architecture/framework to support emerging network technologies • SDN/OpenFlow-based network

• Overlay tunneling (VXLAN, NVGRE, STT, …)

• L2 Fabric (Fabric Path, QFabric, …)

Neutron provides common abstract network API independent from specific technologies/vendors

Neutron Basic Network Model

TenantA-VM1 10.0.0.2

TenantA-VM3 9.0.0.2

TenantA-VM2 10.0.0.3

External Net

88.0.0.0/18

Tenant-A Net1 10.0.0.0/24

Tenant-A Net2 9.0.0.0/24

L2 virtual network

virtual port

virtual server

virtual interface (VIF)

virtual subnet

SNAT for external traffic Floating IP

• Tenant can create multiple (Layer2) networks • Subnet defines L3 subnet information (CIDR, gateway etc)

– Multiple subnets can be associated to a network • E.g., IPv4 subnet + IPv6 subnet. Multiple IPv4 address pools

– Support IP address overlapping among tenants

• Network have multiple ports (similar to physical L2SW ports) – Virtual NICs of VMs or router interfaces are associated with neutron ports.

Neutron

Network API • Networking API is RESTful API. • Only “logical” information are visible in APIs for regular users. • In admin API, some physical/backend information such VLAN ID can be specified.

GET http://localhost:9696/v2.0/networks/20b49737-c718-4f10-8edc-7fe9f8ee7cfd { "network": { "admin_state_up": true, "id": "20b49737-c718-4f10-8edc-7fe9f8ee7cfd", "name": "net1", "router:external": false, "shared": false, "status": "ACTIVE", "subnets": [ "644f729f-1c86-48b3-91a2-761e29398e81" ], "tenant_id": "be11ce3cabc74ee68d4f22253ffbb5e8" } }

GET http://localhost:9696/v2.0/networks/20b49737-c718-4f10-8edc-7fe9f8ee7cfd { "network": { "admin_state_up": true, "id": "20b49737-c718-4f10-8edc-7fe9f8ee7cfd", "name": "net1", "router:external": false, "shared": false, "status": "ACTIVE", "subnets": [ "644f729f-1c86-48b3-91a2-761e29398e81" ], "tenant_id": "be11ce3cabc74ee68d4f22253ffbb5e8", "provider:network_type": "vlan", "provider:physical_network": "physnet_1", "provider:segmentation_id": 201 } } Regular user Admin user

Advanced Network Service

• Network Service – LBaaS, FWaaS, VPNaaS and so on.

– Provide additional network features rather than simple data transfer (L2/L3)

• Would like to insert such network services on demand. – Use network services in “Cloud” style (through API

and on demand)

– It fits into SDN.

Service deployment pattern – Firewall

Network 1

External Network

Network 2

Network 1

FW on network

FW on Port (security group?)

FW on Router

Service deployment pattern – load balancer

Network 1

Network 1

Network 1

Network 2

Direct Return

2-arm LB

1-arm LB

- terminated - Router

Network Service Insertion

Network 1 Network 2

External Network

VPN Connection

Service Insertion

Understanding Neutron API

Virtual Network

Virtual Router VM VM VM

Compute API L2/L3 API

Horizon, Heat or Orchestration

LB Service

VM

Process Based LB (HAProxy)

HW Appliance

Advanced network service API

OpenStack provides “Abstraction” and “Integration”. NOT Virtualization (e.g., OpenStack deos not develop hypervisors)

Neutron has various layers of API abstraction L2/L3 API or compute API are primitive APIs. Advanced network service APIs are higher layer API. They

consumes primitive APIs internally.

LBaaS FWaaS VPNaaS

Nova

Neutron Internal Architecture • A plugin per feature API • There are multiple plugins (Core plugin + Service plugins * N)

– ML2 plugin for Layer2 API, LBaaS plugin for LBaaS API

• The traditional model (one plugin for all APIs) is supported. • By switching a plugin implementation , various network technologies can be used. • Neutron also can be used as standalone, though it is a part of OpenStack.

– Can/May we avoid vendor lock-in by adopting Neutron API?

Neutron Server Compute Node

VM VM

vSwitch

HW Switch

L2/L3 Plugin FWaaS Plugin

LBaaS Plugin

Agent

L2 API L3 API FWaaS

API LBaaS

API

Network Controller

OpenFlow Controller など

Neutron API

L3 Agent

HAProxy Agent

Compute API

XXX Driver

Neutron

SDN controller with OpenStack - centralized controller model -

Nova

OpenFlow Plugin Nova Compute

Virtual Switch

VM VM

Nova Compute

Virtual Switch

VM VM

HW Switch

CLI / Dashboard (Horizon) / Orchestration Tool

OpenFlow Controller

Network

OFC API

Neutron API

OpenFlow Protocol

Nova API

agent agent

- NEC OpenFlow/Trema - OpenDaylight - NSX ….

Neutron

SDN controllers with OpenStack - Controller on every compute node -

Nova

ML2 Plugin

(OpenFlow Agent) Nova Compute

Virtual Switch

VM VM

Nova Compute

Virtual Switch

VM VM

HW Switch

CLI / Dashboard (Horizon) / Orchestration Tool

In OpenFlow agent mech driver in ML2, OpenFlow controllers

are deployed per compute node and each OF controller controls OVS on each node.

Neutron API

OpenFlow Protocol

Nova API

agent agent

OpenFlow Controller

OpenFlow Controller

Ryu

Neutron ML2 Plugin • ML2 = Modular Layer 2

– New framework for Layer2 network functions to avoid duplicated/copy-and-paste efforts across plugins.

– Default reference implementation in Neutron. OVS / Linux Bridge plugins will be removed in the near future.

– We can use multiple network technologies at the same time. – Two types of drivers: Type Driver and Mechanism Driver

• Type Driver – Manages logical network resources depending on “network type” (e.g., VLAN-ID/VXLAN VNI. – “network type” is specified by “provider:network_type” attribute of the provider network

extension when creating a network. – Supported network types: VLAN, VXLAN, GRE, Local, Flat

• Mechanism Driver – Configures/provisions network devices. – Multiple mechanism drivers can be enabled (ordered list). Each mechanism driver configures

corresponding network devices if the driver supports a specified network type. – Only one mechanism driver can handle plugging VM VIF to a network (bind_port). – Open vSwitch agent driver, Linux Bridge agent driver, OpenFlow Agent driver, … – Cisco Nexus 1000V driver, Arista driver, ….. – L2 population driver

• Install MAC addresses of endpoints into OVS/Linux bridges to reduce broadcast in tunneling.

Features of OVS/Linux Bridge Plugin are now frozen. All new features are added

to ML2 plugin. Let’s use ML2 plugin.

Icehouse Update

• Focused on improvements in performance and stability – Multiple API worker / RPC worker – Improvements in dhcp-agent, l3-agent, OVS agent … – Scenario test / API test coverage in tempest – Third party testing (vendor driver testing)

• nova event callback – Notify nova of port status changes when fixed_ips/floatingips change – It ensure a VM is launched after network connection is available.

• Deprecate OVS Plugin and Linux Bridge plugin – Migration to ML2 plugin is recommended.

• Not so many feature additions

– We added many feature additions in the past releases. We improved the quality of the whole neutron codebase and now are ready for next feature additions :-)

Key New Features During Icehouse cycle the team focused on stability and testing of the Neutron codebase. Many of the existing plugins and drivers were revised to address known performance and stability issues.

Icehouse Release Notes

Multiple API/RPC Worker

Neutron server

Message queue

OVS agent

Dhcp agent

l3-agent

OVS agent

Dhcp agent

l3-agent

OVS agent

Dhcp agent

l3-agent

DB

RPC worker

Message queue

OVS agent

Dhcp agent

l3-agent

OVS agent

Dhcp agent

l3-agent

OVS agent

Dhcp agent

l3-agent

DB RPC

worker RPC

worker

API worker

API worker

API worker

API request

RPC message

https://www.openstack.org/assets/presentation-media/junohpneutronatscalefinal.pdf

Compute host

VM

Nova event callbacks • Mechanism to inform nova of state changes in Neutron ports • Nova waits VM launching until network connection becomes ready

– Previously Nova launched VMs without checking network connection status. As a result, IP address allocations by DHCP sometimes fail with timeout.

nova-compute

VM

OVS

VIF

Neutron Server

OVS agent

(1) Create Port

(2) Create Tap device

Network

(3) Setup NW connection

(4) Report “port is ready”

(5) Notify Nova

(6) VM Launch

Dhcp server

Wait notification from neutron before launching VM

Vendor Plugin/Driver Testing • Vendor plugin/driver scenario/API testing (so-called “Third Party Testing” is now one of

requirements for new vendor/driver addition. It also applies to all existing plugins/drivers. • All third party CI system should run scenario/API tests for every patch proposal to Gerrit. • Test cases of scenario/API tests are developed by OpenStack community (“tempest” projecct). • Previously there is no way to check vendor plugins/drivers are tested.

http://www.rcbops.com/gerrit/reports/neutron-cireport.html

Neutron Icehouse update detail (1/2)

• Main Focuses are “Stabilization” and “Test Coverage” (rather than new features) – Scenario and API test coverage in tempest – Enable parallel testing on tempest

• Nova-Network Feature Parity – Nova-network feature parity towards deprecating nova-network – A lot of progresses but the completion will be Juno (or later)

• Nova Integration : nova-event-callback – Notify nova of port status changes when fixed_ips/floatingips change – It ensure a VM is launched after network connection is available.

• ML2 (Modular Layer2) plugin – Add support for vnic type request to be managed by ML2 mechanism drivers

• Allows nova to request vnic type for a port : normal, direct (pass-through), macvtap

– Implement the binding:profile port attribute in ML2 – Extensible port attribute for plugin to provide details to VIF driver – Migration from OVS / linux bridge plugin to ML2

• Provide a tool to migration DB data from deprecated plugins to ML2.

• L3 : Operational Status for Floating IPs • Advanced Network Services (LBaaS, VPNaaS, FWaaS)

– No major features – Some vendor drivers: Radware/Embrane (LBaaS), Cisco (VPNaaS)

Neutron Icehouse update detail (2/2)

• New ML2 mechanism drivers – OpenDayLight plugin (It works with Hydrogen ODL release) – OpenFlow agent (ryu) driver (OpenFlow controller per compute node) – Big Switch, Mellanox, Brocade

• New plugins : IBM SDN-VE, Nuage, OneConvergence • Existing plugins:

– BigSwtich (async ope, secgroup, resync with backend) – Vmware NSX (renamed from Nicira NVP plugin), Plumgrid (provider network) – NEC plugin : packet filter extension support for PFC

• Items deferred to Juno ….

– IPv6 related attributes to control IPv6 RA and address assignments (ipv6-two-attributes) • API layer is defined but backend implementation is not merged.

– Layer 3 HA • L3-agent HA • Distributed Virtual Router (with OVS)

– Advanced Network Service “flavor” support – Advanced service in VM – and so on

Neutron topics in “Juno” • Migration from “nova-network”

– OpenStack has two network implementation: nova-network and neutron. – The community seems to have a consensus that both are merged into Neutrn. – Feature parity to nova-network – Clarify the migration scenarios from nova-network in the production

environments. • Migration of Network/Port/Security Group/… from nova to neutron. • Exploring a migration way without stopping tenant data plane traffic. One of the current

goals is to avoid breaking TCP connection. • Requires stoping API service during the migration.

– Tracking the progress in Neutron weekly meeting.

• Distributed Virtual Router (DVR) • L3-agent HA • IPv6 full support • Baremetal (Ironic) Integration

– Multi Tenant isolation with baremetal instance

Neutron topics in “Juno” • Distributed Virtual Router (DVR) with Open vSwitch

– In the current OSS implementation, all traffic across logical routers needs to be transferred via L3-agent nodes. It can be a bottleneck in both performance and HA perspectives.

– DVR implements distributed virtual routers using OVS bridge. The routing functionality is distributed over all compute nodes.

– It is expected to be an alternative of nova-network “multi_host” feature (which is a common solution to achieve network HA in nova-network).

Neutron topics in “Juno” Network Services • Flavor Framework

– A mechanism to provide multiple service levels or implementations in advanced network services such as LBaaS.

– Operators can choose backend implementations according to the service level of a LBaaS resource (e.g., physical appliance for “gold” load balancer and virtual appliance for “bronze” load balancer).

• Service Insertion / Service Chaining • Traffic Steering

– Allows to redirect traffic which matches a defined policy to a specific logical port. It can be used to implement service chaining or insert “L2-transparent” appliances.

• Group Policy Framework

Neutron topics in “Juno” Network Services • Service VM

– VM which provides network services; “virtual appliance” – Service VM itself is related to multiple projects such as neutron

and nova, so it will be a separate project on stackforge. – We decided to gather requirements for neutron for service VMs

and implement them into Neutron. Neutron team is also interested in these efforts and continues to support the effort. • Disabling Security Group / Anti-spoofing rule • Trunk VLAN support for VM port • Traffic Steering • ….

– https://etherpad.openstack.org/p/servicevm

Recent topics in developer community

• NFV Sub-team

• Cross-Project design summit session

• Ops-Dev Meet-up

• Blueprint Spec Review

Backup slides

Neutron Functional modules • Neutron Server

– Manage logical networks – Neutron plugin is a part of neutron server (now)

• Agent – Service processes other than neutron-server. They can be located on a

different host from neutrn-server. – Some plugins uses agents and some do not. It depends on plugin

implementations.

• Various agents – Plugin Agent

• Mainly configures and monitos virtual switches. • Runs on compute nodes and network nodes.

– DHCP Agent • Manages DHCP servers per logica network

– L3 Agent • Manages logical routers. • Network namespace and itpables are used to create logical routers in l3-agent.

– Metadata Agent • Proxies metadata access from neutron network to the nova metadata service.

Nova(CC)

Neutron Server

Compute Node

Compute Node

Data Network

eth1 eth1

eth0 eth0 eth0

OVS (br-int)

OVS (br-int)

Nova Compute

Nova Compute

Plugin Agent Plugin Agent

Management Network

Neutron modules and deployment

Controller Node

eth0

eth1

OVS (br-int)

DHCP agent

L3 agent

Plugin Agent

OVS (br-ex)

Network Node

eth2

External network

metadata agt

Neutron

Nova

Neutron Plugin

Nova Compute

Virtual Switch

VM VM

Nova Compute

Virtual Switch

VM VM

HW Switch

CLI / Dashboard (Horizon) / Orchestration Tool

Physical Network

Neutron API Nova API

agent agent

LBaaS • Added in Grizzly

– Define LBaaS API (tenant API) – HAProxy reference implementation

• Havana – Service Type Framework

• Allow to specify a backend driver. It allows operators to enable mutiple load- balancer drivers and choose it when creating a load-balancer insntace.

– New vendor LBaaS drivers and plugins

• Icehouse (planned) – Multiple VIPs per pool, moving VIP across LB pools – Layer7 support – SSL termination – Routed Service Insertion (LVS based) ? – https://etherpad.openstack.org/p/icehouse-neutron-lbaas

[OLD[ Slide at the beginning of Icehouse dev-cycle

FWaaS • Added in Havana

– Horizon, Heat support – FWaaS API (rule, policy, firewall resource) – L3-agent based impl, some vendor plugin – Limitations

• 1 firewall per tenant (with l3-agent) • Firewall applies to all routers

• Icehouse (planned) – Service Insertion/Service Type Framework – Zone – Apply Operation – Counters – https://etherpad.openstack.org/p/icehouse-neutron-fwaas

[OLD[ Slide at the beginning of Icehouse dev-cycle

VPNaaS

• Added in Havana – IPSec VPN (L3 VPN) is supported – Horizon, Heat support

• Icehouse (planned) – Service Type Framework

• Support multiple typs of VPNs at the same time

– SSL-VPN – IPSec Certification Support – MPLS/BGP – (L2-VPN?) – https://etherpad.openstack.org/p/NeutronVPNaaSIce

House

[OLD[ Slide at the beginning of Icehouse dev-cycle