Introduction of OpenStack cascading solution

HUAWEI TECHNOLOGIES CO., LTD. Page 1Huawei Confidential

Introduction of

OpenStack cascading solution

Chaoyi Huang ( [email protected] )

Hongning Wu ( [email protected] )

Last edited Apr. 27 2015Last update Jan.12, 2015


• Caution• All design in this document is the PoC design, the formal open source project of

OpenStack cascading solution is Tricircle (https://github.com/openstack/tricircle, https://wiki.openstack.org/wiki/Tricircle), where stateless design is applied, and the stateless design is different what’s described in here. It’s useful to learn how the design is evolved from the state-ful to stateless.

https://github.com/openstack/tricircle

https://wiki.openstack.org/wiki/Tricircle


• Requirement and driving forces• The answer – OpenStack cascading solution• Technical diving – OpenStack cascading solution


MotivationThe requirement and driving forces for multi-site/multi-dc clouds integration is as following:

1. Multi-site/multi-dc cloud has co-existence requirement of multi-vendor OpenStack distribution, multi-OpenStack instance, multi-OpenStack versionMulti-vendors’ OpenStack distribution: anti-vendor lock in business policy.Multi OpenStack instance: each vendor has his own OpenStack solution distribution, different site/dc built with different OpenStack instance.Multi OpenStack version: step-wise cloud construction, upgrade gradually

2. Ecosystem friendly global open API for tenant resources in multi-site/multi-dc cloudCross DC orchestration: tenant virtual resources will be distributed in multi-site/multi-dc but inter-connected by L2/L3 networking.Ecosystem friendly open API : It takes almost 4 years for OpenStack to grow the eco-system, the OpenStack API must be retained for distributed multi-site/multi-dc cloud.

3. Restful open API /CLI for each site/dceach site/dc installation/upgrade/maintenance decoupled , make the cloud always workable and manageable standalone in each site. Any part upgrade/update will not lead to the whole cloud upgrade/update at the same time.OpenStack API in each site: Open, de facto standard API


Motivation

Issues of multiple independent OpenStack instances in multi-site:

•Missing global view and control of tenant quota•Missing global view and control of tenant total resource and utilization•Missing global view for tenant level metering data•No global view for tenant level IP address space management•Issue of clone security groups across OpenStack instances•Issue of establishing tenant level L2/L3 networking across region for application high availability across OpenStack instances•Lack of ssh keys clone from one region to another•Lack of image replication across region•Lack the function like volume replication across regions for Geo-site Redundancy/Disaster Recovery•…


• Requirement and driving forces for multi-site cloud• The answer – OpenStack cascading solution• Technical diving – OpenStack cascading solution


DC 1

OpenStack

(Vendor1 /

Version 2.0)DC3

DC 2

OpenStack

(Vendor2 /

Version 2.1)

OpenStack

(Vendor3 /

Version 2.1)

OpenStack cascading is “OpenStack orchestrate OpenStacks” solution for multi-site cloud with unified global OpenStack API exposed

OpenStack API

OpenStack

OpenStack cascading solution

OpenStack API

OpenStack API

OpenStack API


DC 1

OpenStack

(Vendor1 /

Version 2.0)DC3

DC 2

OpenStack

(Vendor2 /

Version 2.1)

OpenStack

(Vendor3 /

Version 2.1)

OpenStack cascading is “OpenStack orchestrate OpenStacks” solution for multi-site cloud with unified OpenStack API exposed

OpenStack API

OpenStack

Value to cloud admin

OpenStack API

OpenStack API

OpenStack API

1 One global single OpenStack API, rich APP ecosystem

3co-existence of Multi-Vendor/Multi-Version OpenStack

2 Geo-distributed multi-site/multi-DC cloud

clearly separated

autonomous sub-cloud

inside a large scale

distributed cloud


DC 1

OpenStack

(Vendor1 /

Version 2.0)DC3

OpenStack

(Vendor3 /

Version 2.1)


OpenStack API

OpenStack


OpenStack API

OpenStack API

OpenStack API

5

4Plug & Play fast integration for multi-site cloud through OpenStack API

DC 2

OpenStack

(Vendor2 /

Version 2.1)

Fault isolation. Any fault can only affect part of the cloud

clearly separated



distributed cloud


DC 1

OpenStack

(Vendor1 /

Version 2.0)DC3

DC 2

OpenStack

(Vendor2 /

Version 2.1)

OpenStack

(Vendor3 /

Version 2.1)


OpenStack API

OpenStack


OpenStack API

OpenStack API

OpenStack API

6Clear OAM boundary. Reduce upgrade/troubleshooting/etc challenge

Bug Fix6

Clear OAM boundary. Reduce upgrade/troubleshooting/etc challenge

clearly separated



distributed cloud


Value to end user• Tenant has global view for resources in multi-clouds

The tenant's resources VM, Volume may be distributed in multi-OpenStacks which using shared KeyStone or KeyStone federation, and also these resources are inter-connected through L2/L3 networking with advanced service like FW,LB,VPN. The tenant's distributed resources can be managed through the cascading OpenStack, like the tenant has one virtual OpenStack allocated to him, the tenant has global view for his resources like image, metering data, VM/volume/network, etc. The tenant also has global quota control and resources utilization through the cascading OpenStack.

• Tenant level global IP address management.The cascading OpenStack can work as the global IP address management for the tenant across multiple cascaded OpenStack.

• High availability application across different physical data center.With the aid of overlay virtual L2/L3 networking across data centers and image Replication function, application backup/disaster recovery/load balance is easy to implement in the distributed cloud.

• Virtual machine / volume migration / vApp migration from one data center to another:With the help of OpenStack cascading, VM/Volume migration from one DC to another one is feasible.


Value to scalability

OpenStack …

…

1 2 100

1 2 1000

OpenStack

…1 2 1000

OpenStack

…1 2 1000

OpenStack

Max. 100 data centers

Max. 100k physical server nodes

Max. 1 million VMs

OpenStack API OpenStack API OpenStack API

OpenStack API

Scalability inside one data center, multi-

data centers or multi-sites

How large scale is up to the scalability of the cascading OpenStack

Test report: http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers

http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers


Stop and thinking, why build a very large cascading OpenStack for all tenants?

In fact, each tenant only cares about his own resources.

Why not just dynamically allocate or assign one virtual OpenStack to serve the tenant for his resources which is distributed in many OpenStacks, no matter these OpenStacks in one data center or multi-data centers?

Cascading OpenStack can work as tenant level virtual OpenStack service.

Evolve to unlimited scalability

*Here the OpenStack mainly indicates service to provide IaaS function like Nova,Cinder,Neutron, Glance, Ceilometer,KeyStone


Evolve to unlimited scalability

Cascading

OpenStack

(Tenant 1)

…Cascading

OpenStack

(Tenant 2)

Cascading

OpenStack

(Tenant x)

…

Tenant 1

Virtual Resources

Tenant 2

Virtual Resources

Tenant x

Virtual Resources

Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y

https://tenant1.OpenStack/ https://tenant2.OpenStack/ https://tenantx.OpenStack/

OpenStack API

OpenStack API

OpenStack API

OpenStack API


OpenStack APIOpenStack API

OpenStack API

OpenStack API

If the tenant wants resources in multi-OpenStacks, allocate one cascading OpenStack for him (or share one cascading OpenStack with others to reduce management resource, but make it as small as possible, to ease disaster recovery, backup, …)

If the tenant wants resource only in one OpenStack, no cascading required.

1. Fully distributed, no central point at all, no scalability bottleneck.

2. Unlimited OpenStack instances in one cloud or federated clouds.

3. Provide tenant with seamless one OpenStack experience no matter how many OpenStack instances behind


• Requirement and driving forces for multi-site cloud• The answer – OpenStack cascading solution• Technical diving – OpenStack cascading solution


Neutron Server

OVS

Agent

Cinder API

Rabbit-MQ

Cinder

VolumeCinder

Volume

Nova API

RabbitMQ

Nova

ComputeNova

Compute

Libvirt

Driver

Nova

Driver

LVM

Driver

Cinder

Driver

KVM Nova LVM Cinder

RabbitMQ

OVS

L3

Agent

Linux

Router

SchedulerScheduler

Neutron

Agent

Neutron

Magic happens by just considering OpenStack as its own backend !

Glance Image1:

Loc1: NFSLoc2: Glance1

Loc3:Glance2

Glance1 Glance2

Image1:

Loc1: Ceph

Image1:

Loc1: Ceph

Ceilometer

API

hBase

StoreStore

CeilometerCeilometer

Nova as hypervisor Cinder as block storage Neutron as networking device Glance as Img. Location Ceilometer as store

** Architecture simplified for illustration only


Neutron

ServerCinder API

Rabbit-MQ

Cinder

Proxy

Nova API

RabbitMQ

Nova

Proxy

Nova Cinder

RabbitMQ

SchedulerScheduler

Neutron

Proxy(L2/L3/LB/V

PN/FW)

Neutron

Components introduced for cascading

Glance

Glance1 Glance2

Ceilometer

API

Ceilometer-

Proxy

CeilometerCeilometer

Replic-

Manager

CascadingLayer

CascadedLayer

Components introduced for cascading

*KeyStone is global service shared or federated by cascading and cascaded layer* Heat will use OpenStack API to do orchestration, no cascading required.


Design Pattern – fractal (self similar)

Fractal: Recursive self-similar, growth to scalehttp://ethshar.wikia.com/wiki/File:Tricircle_fractal.gif

Cascading: Also recursive self-similar, growth to scale. Just treat the cascaded OpenStack as a huge compute-node


Nova Cascading – how it works

Nova-API

Nova-Scheduler

RbbitMQ

Nova-Proxy

(host1/AZ1)

Nova

Nova-Api

Nova-Proxy

(host2/AZ2)

Nova

Nova-Api

…

Cascading OpenStack

Cascaded OpenStack

1.Launch VM1 (AZ1)

2.Host1 scheduled for it belongs to AZ1

3.Translate the Launch VM 1 request to

configured Nova

4.Launch VM1 in AZ1

5.Reboot VM1 @ host1

6. Translate the Reboot VM1

request to configured Nova

7. Reboot VM1

Solved Challenges:

• UUID mapping

• VM status replication

• Host aggregate

• Flavor

• Metadata modification

• Password / Key pair update

• Image / Cinder / Neutron orchestration

• Cold migration cross Nova

AZ1 AZ2

AZ: availability zone

*Only AvailabilityZone(AZ) and HostAggregate related filter will be configured in the cascading layer nova-scheduler

8.Launch VM2 (AZ2)

9. Host2 scheduled for it belongs to AZ2

10Translate the Launch VM2

request to configured Nova

11.Launch VM2 in AZ2


Cinder Cascading – how it works

Cinder-API

Cinder-Scheduler

RbbitMQ

Cinder-Proxy

(host1/AZ1)

Cinder

Cinder-Api

Cinder-Proxy

(host2/AZ2)

Cinder

Cinder-Api

…

Cascading OpenStack

Cascaded OpenStack

1.Create Volume1 (AZ1)


3.Translate the Create Volume1

request to configured Cinder

4.Create Volume1 in AZ1

5.Resize Volume1 @ host1

6. Translate the Resize

volume request to configured

Cinder

7. Resize Volume

Solved Challenges:

• UUID maping

• Timestamp based query

• Volume type / QoS

• Volume migration cross Cinder

AZ1 AZ2

AZ: availability zone

*Only AvailabilityZone(AZ) filter will be configured in the cascading layer cinder-scheduler

8.Create Volume2 (AZ2)


10.Translate the Create Volume2

request to configured Cinder

11.Create Volume2 in AZ2


Neutron Cascading – L2 networking (VxLAN Point2Point)

Neuton-API

L2/L3-Proxy

L2/L3-Proxy

Neutron Neutron

VM1 VM2

VxLAN0

VM3 VM4

AZ1 AZ2

VM1 VM2

VxLAN0 DVR

VM3 VM4

VxLAN0DVR

DVR

2.Periodic polling port

status( for example

VM2 port)

3. VM2 Port

status up

4. L2 population

5. fdb_add ( Port for VM2 IP / VM 2 mac / Host IP )

6. Create virtual remote Port for VM2

(with VM2 IP / VM2 mac / VM2 host IP)

VM2

7. Internal L2 population for

virtual remote port for VM2

Virtual remote port

Cascading OpenStack

Cascaded OpenStack

1. Internal L2 population for VM2

If VM2 was added to VxLAN0…

Self similar L2 population happened

in the cascading levelAZ1 AZ2


Neutron Cascading – L2 networking (VxLAN L2GW)

Neuton-API

L2/L3-Proxy

L2/L3-Proxy

Neutron Neutron

VM1 VM2

VxLAN0

VM3 VM4

AZ1 AZ2

VM1 VM2

VxLAN0DVR

VM3 VM4

VxLAN0DVR

DVR

2.Periodic polling port

status( for example

VM2 port)

3. VM2 Port

status up

4. L2 population

5. fdb_add ( Port for VM2 IP / VM 2 mac / Host IP )

6. Create virtual remote Port for VM2

(with VM2 IP / VM2 mac / VM2 host IP)

Cascading OpenStack

Cascaded OpenStack

1. Internal L2

population for VM2

If VM2 was added to VxLAN0…

Self similar L2 population happened

in the cascading level

VM2

1. Virtual remote port attached

to L2GW

2. ARP responder proxy

3. Reduce L2 population here

L2GW


Neutron Cascading – L2 networking (VxLAN L2GW - continue)

Neuton-API

L2/L3-Proxy

L2/L3-Proxy

Neutron Neutron

VM1 VM2

VxLAN0

VM3 VM4

AZ1 AZ2

VM1 VM2

VxLAN0DVR

VM3 VM4

VxLAN0DVR

DVR Cascading OpenStack

Cascaded OpenStack

L2GWL2GW

Solved Challenges:

1. Not all compute nodes need to be

routable, only GW nodes to be

routable, reduce physical networking

complexity

2. Reduce L2 population traffic in the

cascaded layer

3. Easy to build L3 networking routing

rule for cross OpenStack networking,

for example, VLAN 2 VxLAN, VLAN 2

VLAN, VxLAN 2 VxLAN

4. L2GW mentioned and implemented

in the current BPs can be leveraged


Neutron Cascading – L2 networking (VLAN)

Neuton-API

L2/L3-Proxy L2/L3-Proxy

Neutron Neutron

VM1 VM2

VLAN0

VM3 VM4

AZ1AZ2

VM1 VM2

VLAN0Router

VM3 VM4

VLAN1

DVR

Cascading OpenStack

Cascaded OpenStack

VLAN1

Router192.168.1.0/24

192.168.2.0/24

DVR

network_vlan_ranges

=AZ2:300:2000

network_vlan_ranges =AZ1:100:4000

network_vlan_ranges =AZ1:100:4000

tenant_network_types = vlan,vxlan

network_vlan_ranges =AZ1:100:4000, AZ2:300:2000

vni_ranges=4096:100000

network_vlan_ranges

=AZ2:300:2000

1. neutron net-create net01--provider:network_type vlan --provider:physical_network AZ1

Nova-Proxy

2. LaunchVM ( net01 )3. CreatePort (Net01),

get Port: IP/mac

4. Neutron net-create net01--provider:network_type vlan --provider:physical_network AZ1--segmentation_id = 101

5. port01 = CreatePort(net01, IP/mac)6. Launch VM ( net01, port01 )

***the network / subnet / port creation will be moved to L2 proxy after uuid -mapping refectory finished


Neutron Cascading – L3 networking (E-W simplified)

Neuton-API

L2/L3-Proxy L2/L3-Proxy

Neutron Neutron

VM1 VM2

VLAN0

VM3 VM4

AZ1 AZ2

VM1 VM2

VLAN0Router

VM3 VM4

VLAN1

DVR

Cascading OpenStack

Cascaded OpenStack

VLAN1

Router192.168.1.0/24

192.168.2.0/24

1. Router-add-interface(net2)

2. Router-add-update(net2)2. Router-add-update(net2)

3. Update extra-route next hop

“ 100.64.20.20“, destination

“ 192.168.2.0/24 ”

100.64.0.0/10

100.64.10.10 100.64.20.20

3. Update extra-route (next

hop “ 100.64.10.10“,

destination “ 192.168.1.0/24 ”)

GRE tunneling over extra-route is

used for VLAN2VLAN L3 networking

across data centers. Other tunneling

like VPN support later

DVR population


Neuton-API

L2/L3-ProxyL2/L3-Proxy

Neutron Neutron

VM1 VM2

VLAN0

VM3 VM4

AZ1 AZ2

VM1 VM2

VLAN0Router

VM3 VM4

VLAN1

DVR

Cascading OpenStack

Cascaded OpenStack

VLAN1

Router192.168.1.0/24

192.168.2.0/24

1. Router-add-interface(net2)

Neutron Cascading – multi-SDN controller co-exist

OpenDayLight OpenContrail

DCI Controller

METRO

OTN/SDH

OpemFlow

data center interconnection ( DCI )

overlay L2 network

1. One cascaded Neutron one SDN controller, multi-SDN controllers co-exist(each SDN controller almost covers from L2-L7)

2. provide cross data center Network as a Service, one DCI controller (data center interconnection) under cascading Neutron to orchestrate cross DC networking, provide cross DC overlay L2 network.


Neutron Cascading – L3 networking (N-S simplified)

DVR

(Centralized

Node)

N-S

Router

100.64.0.0/10

Internet

VM1 VM2

VLAN0 DVR1. DVR ( external network: 100.64.10.10 )

3. DVR ( next hop “ 100.64.30.30“,

destination “ 0.0.0.0/0 ” )192.168.1.0/24

100.64.10.10

100.64.30.30

AZ1

1. remote subnet addressing

Through extra route to next hop for remote

subnet mapping

2. Piggy network introduced.

Using this space 100.64.0.0/10 for cross

OpenStack physical networking

3. Onlink routes introduced.

N-S routers with onlink route

N-S cross OpenStacks

GRE tunneling over extra-route is

used for VLAN2VLAN L3 networking

across data centers. Other tunneling

like VPN support later

AZ2


Glance Cascading

Glance cascading solution:Just use cascaded Glance as location backend of cascading Glance.

DB

Glance-API

Repli-Manager

Repli-Driver

DB

Glance-API

Storage

Image-

Store

StorageImage-

StoreDB

Glance-API

Storage Image-

Store

Customized image uploading and distribution:

Policy1: Replication when first used

When the image is used in the cascaded OpenStack for the first time, replicate the

image metadata and image data to the cascaded OpenStack. Bad user experience for

the first time, the gain is image replication is much more simpler

Policy2: Replication when upload

---------------------------------------------------------------------------------------

There are 3 scenarios will trigger the replication of Image

1. Upload image data

2. Patch location

3. Create VM snapshot/Volume image

If one the above 3 scenarios happened, the replication manager will check the replication

policy and the image owner, to see if the image should be replication to other

cascaded Glance. If yes, call the Replication-driver to replication the image to

cascaded Glance according to replication-region-list:

1) replication the image metadata to the specified glance

2) replication the image data to the specified region image storage (by copy-from or

import/export or direct copy between storage…)

3) Register the image location to the image in the just replication cascaded Glance

4) Register the new image location( the image URL in the cascaded Glance) to the

cascading image

Better user experience, but complex replication mechanism

1 2 3

Nova-Proxy

Cinder-Proxy

Image

Client

Image

Client

Image registration:

If image has been already distributed in multi-OpenStack instances, just register the image link in

cascaded Glance as one of the image location in cascading Glance


Ceilometer Cascading

Ceilometer

Heat

Cascading

AutoScaling

Alarm request

Ceilometer-Proxy

Ceilometer Ceilometer

Ceilometer

API calling

class Ceilometer-Proxy(base.StorageEngine)

{

UUID mapping injected by

Nova/Cinder/Neutron/Glance

Resource UUID translation

Resource UUID and Ceilometer Location

addressing

Proxy the request to proper Ceilometer

}

The webhook setting ( callback

to HEAT ) for alarm action will

be sent to cascaded Ceilometer

transparently

The webhook

(callback to

HEAT)

Ceilometer cascading solution:Just use cascaded Ceilometers as StogradeEngine of Cascading

OpenStack. All requests from cascading ceilometer will be proxy to

proper CeilometerCascading OpenStack


OpenStack cascading

DB

Message Bus

Nova-API

Nova-Scheduler

Nova-ConductorDB

Message Bus

Cinder-API

Cinder-Scheduler

DB

Message Bus

Neutron-API

Neutron-Plug-in

DB

Message Bus

Nova-API

Nova-Scheduler

Nova-ConductorDB

Message Bus

Cinder-API

Cinder-Scheduler

DB

Message Bus

Neutron-API

Neutron-Plug-in

Cascaded OpenStack 1 Cascaded OpenStack x

…

Controller NodeCompute 1 Compute n

…

Compute 1 Compute n

DB

Message Bus

Nova-API

Nova-Scheduler

Nova-ConductorDB

Message Bus

Cinder-API

Cinder-Scheduler

DB

Message Bus

Neutron-API

Neutron-Plug-in

Cascading OpenStack

Nova-API Cinder-API Neutron-API Nova-API Cinder-API Neutron-API

Controller Node Compute x

Nova-API Cinder-API Neutron-API

Nova

-Pro

xy

Cin

der-P

roxy

L2-P

roxy

L3-P

roxy

LB

-Pro

xy

VP

N-P

roxy

Nova

-Pro

xy

Cin

der-P

roxy

L2-P

roxy

L3-P

roxy

LB

-Pro

xy

VP

N-P

roxy

FW

-Pro

xy

FW

-Pro

xy

Compute 1

VM

VM

VM

VM

Controller Node

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

Cascading OpenStack is normal OpenStack except that the process running in compute-node as proxy which treat the cascaded OpenStack as its own backend


Evolution of OpenStack cascading to hybrid cloud

Cascading

OpenStack

(Tenant 1)

…Cascading

OpenStack

(Tenant 2)

Cascading

OpenStack

(Tenant x)

…

Tenant 1

Virtual Resources

Tenant 2

Virtual Resources

Tenant x

Virtual Resources

Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y

https://tenant1.OpenStack/ https://tenant2.OpenStack/ https://tenantx.OpenStack/

OpenStack API

OpenStack API

OpenStack API

OpenStack API


OpenStack APIOpenStack API

AWS API

Azure API


More information:

Wiki: https://wiki.openstack.org/wiki/OpenStack_cascading_solution

Wiki: https://wiki.openstack.org/wiki/TricircleSource Code: https://github.com/openstack/tricircle

https://wiki.openstack.org/wiki/OpenStack_cascading_solution

https://wiki.openstack.org/wiki/Tricircle

https://github.com/openstack/tricircle

Software

Introduction of OpenStack cascading solution