클라우드데이터센터네트워크아키텍처 Vxlan cisco(20141215)

클라우드 데이터센터 네트워크 아키텍처

VXLAN Deployment Models –Cisco-

KrDAG

[email protected]

2014.12

Cisco BRKDCT-2404

mailto:[email protected]

http://sola99.tistory.com

Overlay Network Evolution

[ Overlay Network 발전 방향 ]

Hybrid Overlays

Network DB

A p p O S

A p p O S

Virtual Physical

Network Overlays

Protocols

Physical Physical

V M O S

V M O S

Virtual Virtual

V M O S

V M O S

Host Overlays

Flooding

[ VXLAN Evolution ]

Overlay Network 발전방향과 VXLAN Evolution

Edge(장비), OTV/VPLS/LISP등 Edge(서버), VXLAN/NVGRE/STT Edge(물리+가상), 통합/연동/오픈/표준

Multicast Independent Head-end replication enables unicast-only mode Control Plane provides dynamic VTEP discovery

Protocol Learning prevents floods Workload MAC addresses learnt by VXLAN NVEs Advertise L2/L3 address-to-VTEP association information in a protocol

External Connectivity VXLAN HW Gateways to other encaps/networks VXLAN HW Gateway redundancy Enable hybrid overlays

IP Services VXLAN Routing Distributed IP Gateways


VXLAN Evolution: Multicast Independent

Using a Control Protocol: VTEP Discovery

VTEP이 BGP를 통해 자신의 VNI 정보를 광고하고 BGP RR을 통하여 전파: VTEP 자동 발견됨 [ VTEP Discovery ]

VTEP IP B

POD2

VTEP IP C

POD3

VTEP IP A

POD1

BGP consolidates and propagates VTEP list for VNI

2

4 VTEP can perform Head-End Replication

Overlay Neighbors POD3 , IP C POD2 , IP B

3 VTEP obtains list of VTEP neighbors for each VNI

1

VTEPs advertise their VNI membership in BGP

BGP Route Reflector

1

1


VXLAN Evolution: Multicast Independent

VXLAN Unicast Mode: Head-end replication

단말이 *BUM 트래픽 발생 시 VTEP에 등록된 네이버 VTEP에게 복제하여 Unicast로 각각 전달 [ Head-end replication ]

Unicast-Only Transport

VTEP IP B

POD2

VTEP

A host sends a L2 BUM* frame

POD1 VXLAN Encap 4

3 VTEP performs Head-End Replication

VTEP

Overlay Neighbors POD3 , IP C POD2 , IP B

2 VTEP retrieves the list of Overlay Neighbors**

BUM Frame 1

IP A => IP B BUM Frame

IP A BUM Frame

*Broadcast, Unknown Unicast or Multicast = BUM 트래픽

5 Frames are unicast to the neighbors

IP C

POD3

IP A => IP C


VXLAN Evolution: Protocol Learning prevents floods

Using a Control Protocol: Protocol Learning

호스트 라우팅 정보(IP+MAC)를 BGP를 통하여 광고하고 BGP RR을 통하여 전파됨 [ Protocol Learning ]

POD2

VTEP IP C

POD3

VTEP IP A

POD1

BGP propagates routes for the host to all other VTEPs

2

VTEP IP B

Overlay Forwarding Table Host1 <MAC,IP> , VTEP IP A Host2 <MAC,IP> , VTEP IP B

3

1

VTEPs advertise host routes (IP+MAC) to local hosts

BGP Route Reflector

Overlay Forwarding Table Host1 <MAC,IP> , VTEP IP A

3 VTEPs obtain host routes for remote hosts and install in RIB/FIB



VXLAN Control Plane: Host and Subnet Route Distribution

Leaf 장비가 MP-BGP를 통하여 호스트 정보 및 외부 네트워크 정보를 광고 및 전파 [ Host and Subnet Route Distribution ]

Route-Reflectors deployed for scaling purposes

Aggregation RR RR

Access

iBGP Adjacnencies

Host Route Distribution decoupled from the Underlay protocol

Use MP-BGP on the leaf nodes to distribute internal host/subnet routes and external

reachability information

iBGP 내에서 Spine(Aggregation)에 BGP RR을 통하여 전파



VXLAN Control Plane: Host Advertisement

Leaf 장비가 MP-BGP를 통하여 호스트 정보 및 외부 네트워크 정보를 광고 및 전파 [ Host Advertisement ]

1) Host Attaches : 단말 생성

2) Attachment NVE advertises host’s MAC (+IP) through BGP RR : NVE(Network Virtualization

Edge)가 host’s MAC과IP 정보를 BGP를 통하여 광고

3) Choice of encapsulation is also advertised : VXLAN 인지, NVGRE인지 선택하여 광고

NLRI: Host MAC1, IP1 NVE IP 1 VNI 5000 Ext.Community: Encapsulation: VXLAN, NVGRE

Sequence 0

MAC IP VNI Next- Hop

Encap Seq

1 1 5000 IP1 VXLAN 0

Access

Aggregation RR RR

VNI 5000

Host 1 VLAN 10



VXLAN Control Plane: Host Move

Leaf 장비가 MP-BGP를 통하여 호스트 정보 및 외부 네트워크 정보를 광고 및 전파 [ Host Move ]

1) Host Moves to NVE3 : 호스트가 NVE3 아래쪽으로 이동

2) NVE3 detects Host1 and advertises H1 with seq#1 : NVE3이 호스트 정보를 광고하고 변경을 알

리는 시퀀스 넘버를 1 증가하여 전파(즉, 최신 정보라는 뜻)

3) NVE1 sees more recent route and withdraws its advertisement : NVE1은 이동을 인지

NLRI: Host MAC1, IP1 NVE IP 3 VNI 5000 Ext.Community: Encapsulation: VXLAN, NVGRE

Sequence 1

MAC IP VNI Next- Hop

Encap Seq

1 1 5000 IP3 VXLAN 1

Access

Aggregation RR RR

VNI 5000

Host 1 VLAN 10


VXLAN Evolution: External Connectivity

VXLAN L2 and L3 Gateways: Connecting VXLAN to the broader network

VXLAN과 외부간 연결 시 L2 or L3 로 연결 가능 [ Connecting VXLAN to the broader network ]

Destination is in another segment. Packet is routed to the new segment

VXLANORANGE VXLANBLUE

Ingress VXLAN packet on Orange segment

VXLAN Router

L2 Gateway: VXLAN to VLAN Bridging VXLANORANGE

Ingress VXLAN packet on Orange segment

Egress interface chosen (bridge may .1Q tag the packet)

VXLAN L2 Gateway

SVI

Egress interface chosen (bridge may .1Q tag the packet)

L3 Gateway: VXLAN to X Routing • VXLAN • VLAN VLAN100 VLAN200

N1Kv N7K w/F3 LC Nexus 3K N5K/6K N9K CSR 1000V ASR1K/ASR9k

L2 Gateway VXGW on 1110 Yes Yes Yes Yes N/A Yes

L3 Gateway CSR 1000V Yes No Roadmap Roadmap Yes Yes


VXLAN Evolution: External Connectivity

The Multi-encapsulation Gateway

다양한 오버레이 프로토콜 지원, L2/L3 Gateway 지원 [ The Multi-encapsulation Gateway ]

Destination is in another segment. Packet is routed to the new segment

VXLANORANGE

Encap Bridge

SVI

Encap Router

VX

LA

NBLU

E

LIS

PG

REEN

NVGREORANGE

VLANORANGE

MPLS

GREEN

Multi-encapsulation Gateway:

• VXLAN, NVGRE, MPLS, LISP, VLAN, OTV

Bridging (L2 Gateway)

Routing (L3 Gateway)


VXLAN Evolution: IP Services

VXLAN Routing: L2 VXLAN Fabric

Leaf Switch와 물리서버에서 VXLAN(VTEP)이 동작하여 내부에 단말 이동성 보장 [ L2 VXLAN Fabric ]

VXLAN L2 Gateway

VXLAN L2 Gateway

VM

OS

VM

OS

VXLAN L2 Gateway

VXLAN L2 Gateway

VXLAN L2 Gateway

VXLAN L2 Gateway



VXLAN Routing: Centralized Routing in a L2 VXLAN Fabric

코어 네트워킹(혹은 외부)경우 상단에 VXLAN L3 장비(이중 배치)를 통하여 통신 [ Centralized Routing in a L2 VXLAN Fabric ]

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L2 Gateway

VXLAN L2 Gateway

VM

OS

VM

OS

VXLAN L2 Gateway

VXLAN L2 Gateway

HSRP

SVI SVI

VXLAN L2 Gateway

Inter-V(X)LAN and Core Routing

VXLAN L2 Gateway

IP Core



VXLAN Routing: VTEP Redundancy in a VXLAN Fabric

vPC는 MAC 정보 동기화, VTEP 이중화를 위하여 VTEP IP를 Anycast로 제공(HSRP peer 사용) [ Centralized Routing in a L2 VXLAN Fabric ]

L3 Fabric

L2 Gateway redundancy based on vPC (anycast VTEP address)

vMAC -> Emulated VTEP VM VM

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L2 Gateway

VXLAN L2 Gateway

L3 GW redundancy based On vPC and HSRP (2 nodes)



Distributed IP Gateways

기존 L2 환경과 L2/L3 Fabric 환경 비교 [ Traditional L2 & L2/L3 Fabric(or overlay) ]

App

OS

App

OS

L3 Boundary

L3 Boundary

App

OS

App

OS

Virtual Physical

L2/L3 Fabric

Virtual Physical

Traditional L2 - centralised L2/L3 boundary

• Always bridge, route only at an aggregation point

• Large amounts of state converge

• Scale problem for large# of L2 segments

• Traditional L2 and L2 overlays

L2/L3 fabric (or overlay)

• Always route (at the leaves), bridge when necessary

• Distribute and disaggregate necessary state

• Optimal scalability

• Enhanced forwarding and L3 overlays



Distributed IP Gateways: Distributed IP Anycast Gateway

호스트는 어느 Leaf 장비 아래에 있어도 동일한 GW IP(MAC)을 발견(이후 외부 통신) [ Distributed IP Anycast Gateway ]

VXLAN L3 Gateway

L3 Fabric

VXLAN L3 Gateway

VM

OS

VM

OS

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

SVI IP Address MAC: 0000.dead.beef IP:

10.1.1.1

SVI IP Address MAC: 0000.dead.beef IP:

10.1.1.1

The same“Anycast” SVI IP/MAC is used at all VTEPs/ToRs

A host will always find its SVI anywhere it moves



Distributed IP Anycast Gateway: IP Forwarding within the Same Subnet aka Bridging (ARP)

동일 서브넷 2개의 단말이 통신 과정 소개 [ Distributed IP Anycast Gateway ]

VXLAN L3 Gateway

VXLAN L3 Gateway

L3 Fabric

V M O S

V M O S

VM1 10.10.10.10 PM1 10.10.10.20

1

CPU 2

3

rib

PM1 ARP Cache 10.10.10.1 -> GW_MAC

MAC IP L2 VNI L3 VNI

PM1_MAC 10.10.10.20 10000 50000

1) PM1 sends an ARP request

for Default Gateway –

10.10.10.1

2) The ARP request is

suppressed at TOR and

punted to the Supervisor,

where MAC and IP is learned

and distributed

3) TOR response with Gateway

MAC to PM1

Standard behavior of End-Host (virtual or physical) to ARP for the Default Gateway



Distributed IP Anycast Gateway: IP Forwarding within the Same Subnet aka Bridging (ARP)

VM1단말이 PM1단말에 arp 요청을 leaf 에서 처리 [ Distributed IP Anycast Gateway ]

VXLAN L3 Gateway

VXLAN L3 Gateway

L3 Fabric

V M O S

V M O S

VM1 10.10.10.10 PM1

10.10.10.20

4

CPU 5

6

rib

VM1 ARP Cache 10.10.10.20 -> PM1_MAC


VM1_MAC 10.10.10.10 10000 50000

If there is Unicast RIB miss on TOR, ARP request will be forwarded to all ports except the

original sending port (ARP snooping). ARP response will be punted to Supervisor of destination

TOR for Unicast RIB population (learn) and subsequently forwarded to source TOR.

4) VM1 sends an ARP request for

PM1 – 10.10.10.20

5) The ARP request is suppressed

at TOR and punted to the

Supervisor, where MAC and IP

is learned and distributed

6) Assuming PM1 is known and a

valid route does exist in the

Unicast RIB, TOR responds to

ARP with PM1 MAC as Source

MAC. VM1 can build its ARP

cache



Distributed IP Anycast Gateway: IP Forwarding within the Same Subnet aka Bridging (Data)

leaf 에서 PM1 MAC을 L2 Lookup 이후 VXLAN 패킷으로 오른쪽 leaf 로 전달 이후 PM1에 도착 [ Distributed IP Anycast Gateway ]

In case of VM1 is not known to PM1, PM1 would ARP for VM1. Destination TOR would Proxy

for VM1. No Silent-Host discovery problem.

VXLAN L3

Gateway

VXLAN L3

Gateway

VM1 10.10.10.10 PM1

10.10.10.20

VLAN 123

DIP:

10.10.10.20 SIP

: 10.10.10.10

DVTEP: DTOR_L0

SVTEP : STOR_L0

VNI 10000

LDM3ACF:

PMa1b_MrAiCc

SMAC: VM1_MAC

DIP: 10.10.10.20

SIP : 10.10.10.10

V M O S

SMAC: VM1_MAC V

M O S

7

DMAC: PM1_MAC

9

VLAN 123 <-> VNI 10000 PM1_MAC -> DTOR_L0, 10000

8

SIP : 10.10.10.10

DIP: 10.10.10.20

VLAN 123

SMAC:

VM1_MAC

DMAC:

PM1_MAC

10

VLAN 123 <-> VNI 10000 PM1_MAC -> eth1/23

7) VM1 generates a data packet with

PM1_MAC as destination MAC

8) TOR receives the packet and performs

Layer-2 lookup for the destination

9) TOR adds VXLAN-Header information

(Destination VTEP, VNI, etc) and

forwards the packet across the Layer-3

fabric, picking one of the equal cost

paths available via the multiple Spines

10) The destination TOR receives the

packet, strips off the VXLAN header

and performs lookup and forwarding

toward PM1



Distributed IP Anycast Gateway: IP Forwarding within the 다른 Subnet aka Routing (ARP)

VM1이 GW ARP 요청을 Leaf응답 [ Distributed IP Anycast Gateway ]

VXLAN L3

Gateway

VXLAN L3

Gateway

L3 Fabric

V M O S

V M O S

SVI IP Address (VRF Blue) MAC: 0000.dead.beef

IP: 20.20.20.1

VM1 10.10.10.10

VM1 ARP Cache 20.20.20.20 -> GW_MAC

PM2 20.20.20.20

1

CPU 2

3

rib

SVI IP Address (VRF Blue) MAC: 0000.dead.beef

IP: 10.10.10.1


VM1_MAC 10.10.10.10 10000 50000

1) VM1 sends ARP request for

Default Gateway –10.10.10.1

2) The ARP request will be

received at TOR and punted

to the Supervisor, where MAC

and IP is learned and

distributed

3) TOR acts as regular Default

Gateway and sends ARP

response with GW_MAC to

VM1



Distributed IP Anycast Gateway: IP Forwarding within the 다른 Subnet aka Routing (Data)

VM1이 데이터를 전송하면, leaf는 L3 Lookup 이후 VXLAN 을 통하여 전달 [ Distributed IP Anycast Gateway ]

4) VM1 generates a data packet

destined to PM2 IP (20.20.20.20)

with GW_MAC as destination

MAC

5) TOR receives the packet and

performs Layer-3 lookup for the

destination (known)

6) TOR adds VXLAN-Header

information (Destination VTEP,

VNI, etc) and forwards the

packet across the Layer-3 fabric,

picking one of the equal cost

paths available via the multiple

Spines

7) The destination TOR receives

the packet, strips off the VXLAN

header and performs lookup

and forwarding toward PM2

VXLAN L3

Gateway

VXLAN L3

Gateway

PM2 20.20.20.20

DVTEP: DTOR_L0

SVTEP : STOR_L0

VNI 50000

LDM3AC:FDT

aORb_MrAicC

SMAC: STOR_MAC

DIP: 20.20.20.20

SIP : 10.10.10.10

V M O S

V M O S

VM1 10.10.10.10

4

DMAC: GW_MAC

SMAC: VM1_MAC

VLAN 123

DIP: 20.20.20.20

SIP : 10.10.10.10

6

20.20.20.20 -> DTOR_L0, 50000

5

SIP : 10.10.10.10

DIP: 20.20.20.20

VLAN 321

SMAC: GW_MAC

DMAC:

PM2_MAC

7

20.20.20.20 -> PM2_MAC PM2_MAC -> eth1/32


SVI/VNI/VLAN Scoping and Provisioning

Orchestration leads to scale optimization : 규모 및 배포

일반적인 L2/L3 overlay 환경과 SDN(Cisco ACI) 환경 비교 [ 2가지 모델 ]

L3 Fabric

L3 GWY L3 GWY L3 GWY L3 GWY L3 GWY L3 GWY

L3 Fabric

L3 GWY L3 GWY L3 GWY L3 GWY L3 GWY L3 GWY

Mgmt

All VNIs/SVIs everywhere

• Umbrella catch-all provisioning

• Full ARP state on all Leaf Nodes

• Can be manually provisioned up-front

• Open to L2 Flooding everywhere

VNIs/SVIs scoped as hosts attach

• Provision on host attach/policy

• ARP state only for local subnets

• Requires orchestration

• L2 Flooding is scoped


Integration with Orchestartors and Host Attachment

호스트 생성 시 통합 관리 시스템과 연동

자동으로 호스트 발견 및 관련 정책 적용 [ Host Attach ]

Virtual Access

Leaf

Hosts

VM

OS

VM

OS

Virtual Physical

Trigger

Tunnel end-point

Overlay

Network

Manager /

Controller

vCloud

Director

API

Spine

Compute

Orchestration

Network

Orchestration

Trigger

TOR TOR

VM

OS

VM

OS

1

2

3

LDAP

4

VLAN, SVI, VRF, BGP

Compute Controller (e.g. vCloud Director) integrated

overlay provisioning

Integrates physical and virtual end-points Topology discovery with triangulation

VM 생성 이후 TOR에 이벤트발생(단말 학습: New MAC with

VLAN, VDP with VNI, VMTracker, LLDP with NIC MAC,

CLI with VNI or VLAN)이후 TOR LDAP 쿼리이후 config 다운


Multi-Data Center Connectivity

LAN Extensions and IP Mobility

독립적인 Fabric 간에 이더넷 확장, IP 트래픽은 최적 라우팅(no hair-pinning) [ LAN Extensions and IP Mobility ]

N7K/ASR

VLAN 20 VLAN 30

L3 Fabric

L3 Domain

N7K/ASR

VLAN 20 VLAN 30

L3 Fabric

Data Center 1 Data Center 2

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L2/L3 Gateway

VXLAN L2/L3

Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L2/L3

Gateway

VXLAN L2/L3 Gateway LAN Extensions (OTV/EVPN)

IP Mobility (LISP)

VNI 5000



LAN Extensions: 데이터센터 도메인 영역 분리 : 운영 및 영향도 분리

2개의 데이터센터간 연동: 데이터센터 내 통신(VXLAN), 센터 간 통신(OVT/EVPN) [ LAN Extensions ]

N7K/ASR

VNI 5000 VLAN 20 VLAN 30

L3 Fabric

N7K/ASR

VLAN 20 VLAN 30

L3 Fabric

DC1 DC2

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

L2 Gateway L2 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

VXLAN L3 Gateway

L2 Gateway L2 Gateway

L3 Domain

VXLAN ✓OTV/EVPN VXLAN

VXLAN

Domain Boundary:

Failure and Event Containment Clear Administrative Delineation

✗



Scoped Control Plane

데이터센터 내(iBGP 통한 광고/전파), 데이터센터 간(eBGP를 통한 광고/전파) [ Scoped Control Plane ]

L3 Domain

VXLAN

L3 Gatewa y

VXLAN

L3 Gatewa y

VXLAN

L3 Gatewa y

VXLAN

L3 Gatewa y

VXLAN

L3 Gatewa y

VXLAN

L3 Gatewa y

DataCenter #1 DataCenter #2

eBGP

eBGP iBGP iBGP

PM2

20.20.20.20

PM1

10.10.10.10

MAC IP VTEP L2 VNI L3 VNI

PM1_MAC 10.10.10.10 STOR_LO 10000 20000

PM2_MAC 20.20.20.20 DTOR_LO 30000 20000



End-to-end Data Plane

데이터센터 간 트래픽 전달 [ End-to-end Data Plane ]

L3 Domain

2

VXLAN 20.20.20.20 -> DTOR_L0, 20000 L3

Gatewa y

VXLAN L3

Gatewa y

VXLAN L3

Gatewa y

VXLAN L3

Gatewa y

20.20.20.20 -> PMV2X_LAMNAC L3

PM2_MAC -> eGtha1te/w3a2

y

VXLAN L3

Gatewa y

DVTEP: DTOR_L0 SVTEP :

STOR_L0 VNI 20000 DMAC:

DTOR_MAC SMAC:

STOR_MAC DIP: 20.20.20.20

SIP : 10.10.10.10

DataCenter #1

DataCenter #2

eBGP

VXLAN VNI 10’000 VXLAN VNI 20’000 VXLAN VNI 30’000

PM2 20.20.20.20

PM1 10.10.10.10

1

DMAC:

GW_MAC

SMAC:

PM1_MAC VLAN

123 DIP:

20.20.20.20 SIP

: 10.10.10.10

3

SIP :

10.10.10.10

DIP:

20.20.20.20

VLAN 321

SMAC:

GW_MAC

DMAC:

PM2_MAC

4

Internet

클라우드데이터센터네트워크아키텍처 Vxlan cisco(20141215)