Neutron CI Run on Docker

Copyright©2014 NTT corp. All Rights Reserved.

Neutron CI Run on Docker

Hirofumi IchiharaNTT

2Copyright©2014 NTT corp. All Rights Reserved.

• Neutron has many vendor plugins• 38 vendor plugins and drivers in Neutron(include

proposing)• Impossible to test all vendor plugins

Neutron plugins

http://stackalytics.com/report/driverlog?project_id=openstack%2Fneutron

A10 Networks LBaaS DriverArista Neutron ML2 DriverBig Switch Neutron ML2 DriverBig Switch Controller PluginBrocade Neutron pluginBrocade Neutron ML2 Mechanism driver for VDX/VCS SwitchesBrocade VyattaCSR1kv router service pluginNeutron ML2 Driver For Cisco Nexus DevicesCisco APIC ML2 DriverCisco APIC L3 ServiceCisco VPNaaSNetScaler LBaas driverCloudbase Hyper-V PluginEmbrane Neutron PluginEmbrane LBaaS DriverExtreme Networks PluginFreescale SDN ML2 Mechanism DriverIBM SDN-VE PluginJuniper Networks plug-in for OpenStack NeutronJuniper Opencontrail plugin for Neutron resources

Mellanox Neutron PluginMellanox Neutron ML2 DriverMellanox Neutron ML2 SR-IOV DriverMidoNet PluginNEC OpenFlow PluginNuage Networks Neutron PluginNuage Networks ML2 driverOne Convergence NVSD ControllerOpenDaylight ML2 MechanismDriverMeta PluginRyu OpenFlow Controller PluginOpenFlow AgentPLUMgrid OpenStack Networking Suite Neutron PluginRadware LBaaS Driver for Alteon.Tail-f NCS ML2 Mechanism DriverVMware NSX Network Virtualization Platform PluginvArmour-Firewall

3Copyright©2014 NTT corp. All Rights Reserved.

• Establish third party CI rules for Neutron• https://wiki.openstack.org/wiki/NeutronThirdPartyTesting

Neutron Third Party CI System

https://review.openstack.org/#/c/131905/

4Copyright©2014 NTT corp. All Rights Reserved.

• Metaplugin• provided since Folsom version• allows you to use multiple plugins at the same time

• Metaplugin CI System

Neutron CI System for Metaplugin

1000 tests/monthMax 3 parallels

5Copyright©2014 NTT corp. All Rights Reserved.

• Test must be run in clean environment

Neutron CI System Requirement

Test Node

start

openstack

OVS settingPackage

ENV setting

install openstack

run testsfinish tests

stop openstack

Clean Node

CleanCleanClean

OVS settingPackage

ENV setting

start

openstack

install openstack

fail to install or test

Jenkins: http://jenkins-ci.org/

6Copyright©2014 NTT corp. All Rights Reserved.

• Metaplugin CI System uses VM

Neutron CI System with VM

create VMstart test

VM

install openstackOpenStack

VMOpenStack

VMOpenStack

finish testsdelete VM

certain Public Cloud Service

VM VM VM

Jenkins: http://jenkins-ci.org/

7Copyright©2014 NTT corp. All Rights Reserved.

• VM startup is very slow• VM performance is not good

VM Problems

But a VM is very cheap:)

8Copyright©2014 NTT corp. All Rights Reserved.

• VM startup is very slow• VM performance is not good

VM Problems

But a VM is very cheap:) Really......?

9Copyright©2014 NTT corp. All Rights Reserved.

Not cheap!

10Copyright©2014 NTT corp. All Rights Reserved.

Let’s use container

11Copyright©2014 NTT corp. All Rights Reserved.

Docker

docker: https://www.docker.com

12Copyright©2014 NTT corp. All Rights Reserved.

• Dockenstack• Run OpenStack on Docker• Use nova-docker project to run nova instances• Written by Paul Czarkowski, Eric Windisch• Provide to https://github.com/ewindisch/dockenstack

OpenStack on Docker

ContainerOpenStack

instance instance

13Copyright©2014 NTT corp. All Rights Reserved.

OpenStack on Docker Problems• Well known

• Not support OpenvSwitch• Not support Cinder Volume

• Devstack failed to• Reserve keystone port • install nova-docker• Restart docker

• Nova-docker didn’t support Neutron ML2 Linuxbridge driver

• Neutron failed to ip6tables

14Copyright©2014 NTT corp. All Rights Reserved.

• OpenvSwitch• Select ML2 Linuxbridge as Neutron plugin

• Cinder Volume• Don’t use volume

• Reserve Keystone port ⇒ Bugfix #125740• Install nova-docker ⇒ Bugfix #128746• Restart docker

• Replace command (Workaround)• start-stop-daemon --stop --pidfile "/var/run/docker.pid”

• ML2 Linuxbridge driver ⇒ Bugfix #130520• Don’t use ip6tables

• Disable ipv6 in container(Workaround)• echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6

OpenStack on Docker Run

15Copyright©2014 NTT corp. All Rights Reserved.

• Jenkins runs shell script

Jenkins Run

#!/usr/bin/env bashset -emkdir -p $BUILD_NUMBER/logsdocker run -v $WORKSPACE/$BUILD_NUMBER/logs:/opt/stack/logs --privileged $DOCKER_IMAGE $GERRIT_REFSPEC

• Docker can bind mount a volume from the host

• Storing log became very easy

16Copyright©2014 NTT corp. All Rights Reserved.

• Jenkins detects test success inside container

Jenkins Success

17Copyright©2014 NTT corp. All Rights Reserved.

• Tempest Neutron API test is successful

Neutron Test Result

18Copyright©2014 NTT corp. All Rights Reserved.

• Jenkins detects test failure inside container

Jenkins Failure

19Copyright©2014 NTT corp. All Rights Reserved.

$ git clone -b neutron-ci https://github.com/hichihara/dockenstack.git

$ cd dockenstack/dockenstack

$ docker build .

TRY

20Copyright©2014 NTT corp. All Rights Reserved.

Contact

ichihara.hirofumi@lab.ntt.co.jp