VMUG - NSX Architettura e Design

VMware NSX

Architettura e Design

Luca Morelli Senior Systems Engineer

Qualche Info sullo Speaker…

© 2016 VMware Inc. All rights reserved. 2

• Nato a Catanzaro, la città delle 3 V, circa 38 anni fà

• Ingegnere Informatico – Università di Rende

• Nell’IT da circa 15 anni – Esperienze in Spagna, Francia, Olanda e altri paesi

• Iniziato con lo sviluppo software quindi prevendita da quasi 9 anni

• Quasi 7 anni con un vendor di rete “fisica”

• “Virtualizzato” dal Gennaio 2015

• Appassionato di subacquea, apnea, running (sto preparando la mia prima maratona), arrampicata e della mia splendida compagna

https://it.linkedin.com/in/lumorelli

1 NSX Refresher and Use Cases

2 NSX Components

3 NSX Design

4 Summary

Agenda

3

“Non” Agenda

4

• Marketing (Well, I will do my best but sometimes I cannot control it J)

• A full NSX overview

• Security design and operations recommendations

• Multi-sites scenarios (e.g. Disaster recovery)

• Deep technical aspects

• Specific jargon and acronyms (see Marketing comments)

5

La virtualizzazione del Networkè divenuta una tendenza.

• Modello operativo collaudato

• Facile dispiegamento

• Strumenti per la gestione operativa

Evoluzione del Software-Defined Networking

2008 2015

Ricerca• OpenFlow• Ambito Universitario• Sperimentale

Prodotti e Architetture• Overlay networks • Control planes centralizzati• Service providers & enterprise

2010 2012 2014

Battaglie Architetturali

2016

NSX - Distributed Services in the Hypervisor

Applications

Virtual Machines

Virtual Networks

Virtual Storage

Data Center Virtualization

Location Independence

Software

Hardware

L2 Switching

L3 Routing

Firewalling/ACLs

Load Balancing

Automated operational model of the SDDC

Network & Security Services Now in the Hypervisor

Pooled compute, network and storage capacity; Vendor independent, best price/perf; Simplified config and mgt.

ComputeCapacity

NetworkCapacity

Storage Capacity

© 2016 VMware Inc. All rights reserved.

Virtual Network Services to the Virtual Switch


NSX vSwitch

With NSXBefore NSX

Default Gateway

UCS Fabric A UCS Fabric B

UCS Blade 1

vswitch

6 wire hops 6 wire hops


UCS Blade 1 UCS Blade 2

vswitch vswitch


0 wire hops


UCS Blade 1 UCS Blade 2

With NSXBefore NSX

East-West Routing / Same host East-West Routing / Host to host

2 wire hops

NSX vSwitch

UCS Blade 1

The Advantage of Distributing ServicesDistributed Logical Routing - More efficient networking, fewer hops

Default Gateway Default Gateway Default Gateway


NSX Edge Services Gateway: Integrated Network Services

….

FirewallLoad Balancer

VPN

Routing/NATDHCP/DNS relayDDI

VM VM VM VM VM

• Integrated L3 – L7 services• Virtual appliance model to provide

rapid deployment and scale-out

Overview

• Real time service instantiation

• Support for dynamic service differentiation per tenant/application

• Uses x86 compute capacity

Benefits

VLAN 20Edge Uplink

External Network

Physical Router

Web1 App1 DB1 Webn Appn DBn

NSX Edge

VXLAN 5020Transit Link

Distributed Routing

Routing Peering

10

How it looks like a Basic NSX Topology

…

High Scale Multi Tenant Topology

External Network

Tenant 1Web Logical Switch App Logical Switch DB Logical Switch

…

Web Logical Switch App Logical Switch DB Logical Switch

Tenant NSX Edge Services Gateway

NSX Edge X-Large (Route Aggregation Layer)

Tenant NSX Edge Services Gateway

VXLAN Uplinks (or VXLAN Trunk)

VXLAN Uplinks (or VXLAN Trunk)

VXLAN 5100 Transit

11


2 NSX Components

3 NSX Design

4 Summary

Agenda

12

NSX Architecture and ComponentsCloud Consumption • Self Service Portal

• vCloud Automation Center, OpenStack, Custom

Data Plane

ESXi Hypervisor Kernel Modules

Distributed Services

• High – Performance Data Plane• Scale-out Distributed Forwarding Model

Management PlaneNSX Manager

• Single configuration portal• REST API entry-point

Control Plane• Manages Logical networks• Control-Plane Protocol• Separation of Control and Data Plane

Logi

cal N

etw

ork

Phys

ical

Net

wor

k

vCenter Server

13

NSX Logical Router Control VM

NSX Controller

NSX Edge

FirewallDistributed

Logical RouterLogicalSwitch

VPN

…

…

NSX is AGNOSTIC to Underlay Network Topology

§ NSX Manager deployed as a virtual appliance• Minimum 16 GB and 4 vCPU• Larger scale use 24 GB and 8 vCPU

§ 1:1 relation with vCenter§ Availability Considerations – same as any appliance model

• Resiliency of NSX Manager provided by vSphere HA • Catastrophic failure of NSX Manager is rare, however periodic

backup is recommended to restore to the last known configurations• During the failure all existing data plane connectivity continue to

work since data and management plane are separated

§ NSX Manager also holds DFW rules logic, flow monitoring and local logging data

NSX ManagerDeployment Considerations

14

Management Cluster

WANInternet

L3

L2

Host 1

Host 3

Host 2

Host 4

§ Provide control plane to distribute network information to ESXi hosts

§ NSX Controllers are clustered for scale out and high availability

§ Network information is distributed across nodes in a Controller Cluster (slicing)

§ Remove the VXLAN dependency on multicast routing/PIM in the physical network

§ Provide suppression of ARP broadcast traffic in VXLAN networks

VXLAN 5000

Logical Router 1VXLAN 5001

Logical Router 2

VXLAN - 5002

Logical Router 3

Controller VXLAN Directory Service

MAC table

ARP table

VTEP table

NSX Controllers Functions

§ Controller nodes are deployed as virtual appliances• 4 vCPU, 4GB of RAM per node• CPU Reservation of 2048 MHz• No memory reservation required

§ 3 Controller nodes is the only supported configuration§ Storage resiliency to ensure not all nodes fails due to storage component failure

§ Controller majority is required for having a functional controller cluster• Data plane activity maintained even under complete controller cluster failure

§ By default the DRS and anti-affinity rules are not enforced for controller VMs• The recommendation is to manually enable DRS and anti-affinity rules• Minimum 3 host is required to enforce the anti-affinity rule

§ Storage resiliency is required in terms of access, paths, disks and LUNs• Making sure not all three controller does not end in the same storage or partition such

a failure of storage component should not constitute a failure of all three control VM• Oversubscription of IO does not result in contention among three controller VMs

NSX Controllers

16

Management Cluster

WANInternet

L3

L2

Host 1

Host 3

Host 2

Host 4

Distributed Logical Routing Components

§ The Distributed Logical Router Control Plane is provided by a per instance DLR Control VM and the NSX Controller

§ Dynamic Routing Protocols supported with DLR –OSPF and BGP

• Control VM forms the adjacencies with Edge node

§ Communicates with NSX Manager and Controller Cluster

• NSX Manager sends LIF information to the Control VM and Controller Cluster

• Control VM sends Routing updates to the Controller Cluster

§ DLR Control VM and NSX Controller are not in the data path

§ High availability supported through Active-Standby configuration

§ Can exist in edge cluster or in compute cluster

Logical Router Control VM

§ Logical Interfaces (LIFs) on a Distributed Logical Router Instance• There are internal LIFs and uplink LIFs• VM Default Gateway traffic is handled by LIFs on the

appropriate network• LIFs are distributed across every hypervisor prepared for

NSX• An ARP table is maintained per LIF

§ vMAC is the MAC address of an internal LIF• vMAC is same across all hypervisors and it is never seen

by the physical network (only by VMs)

• Routing table on each ESXi hosts is programed via controller

DLR Kernel Module

vSphere Host

LIF1 LIF2

Transit VXLAN

Uplink


2 NSX Components

3 NSX Design

4 Summary

Agenda

18

NSX sizing is based on a modular footprint• NSX footprint is sized based on customer requirements

• Once these requirements are defined, then map NSX components to infrastructure resources

• Similarly separate VCs for Management and Compute is not an NSX requirement

§ Network Virtualization with NSX enables greater flexibility regardless of physical network design

§ NSX capabilities are independent of network topology

19

• Flexibility with NSX components– Controller are in management cluster with single VC– Edge Resources are flexible in terms of vCPU and memory– NSX stack is flexible – DFW only vs full stack– Three tire licensing allow flexibility that maps to cost and growth

VMware NSX for vSphere Network Virtualization Design Guide ver 3.0https://communities.vmware.com/docs/DOC-27683

VMkernel Networking – L2 OR L3 Topology Agnostic

20

vSphere Host (ESXi)

Layer 2 or Layer 3 Uplinksbased on topology

VLAN Trunk (802.1Q)

VLAN 100

Mgmt10.100.1.25/26DGW: 10.66.1.1

VLAN 101

vMotion10.101.1.25/26DGW: 10.77.1.1

VLAN 102

Storage10.102.1.25/26DGW: 10.88.1.1

VLAN 103

VXLAN10.103.1.25/26DGW: 10.99.1.1

SVI 100: 10.100.1.1/26SVI 101: 10.101.1.1/26SVI 102: 10.102.1.1/26SVI 103: 10.103.9.1.1/26

Span

of V

LAN

s

Span

of V

LAN

s

VMkernel Interface

Transport Zone, VTEP, Logical Networks and VDS § Transport Zone: collection of VXLAN prepared ESXi

clusters

§ Normally a TZ defines the span of Logical Switches (Layer 2 communication domains)

§ VTEP (VXLAN Tunnel EndPoint) is a logical interface (VMkernel) connects to TZ for encap/decap VXLAN traffic

§ VTEP VMkernel interface belongs to a specific VLAN backed port-group dynamically created during the cluster VXLAN preparation

§ One or more VDS can be part of the same TZ

§ A given Logical Switch can span multiple VDS

§ Multiple VDS allows flexibility, connectivity and operational control

§ Need to consider L2 vs L3 VTEP Design

§ IP addressing

§ Bandwidth and availably considerations VTEP Design21

vSphere Host

VXLAN Transport Network

10.20.10.10

Host 1

10.20.10.11VTEP1 VTEP2

VM

VXLAN 5002 MAC2

vSphere Host10.20.10.12

Host 2

10.20.10.13

VM

MAC4

VM

MAC1

VM

MAC3

VTEP3 VTEP4

vSphere Distributed Switch vSphere Distributed Switch

vSphere Host

VXLAN Transport Network

10.20.10.10

Host 1

10.20.10.11VTEP1 VTEP2

VM

VXLAN 5002 MAC2

vSphere Host10.20.10.12

Host 2

10.20.10.13

VM

MAC4

VM

MAC1

VM

MAC3

VTEP3 VTEP4

vSphere Distributed Switch vSphere Distributed Switch

VTEP Design § Number of VTEPs deployed depends on teaming mode

• Single VTEP for LACP or Explicit Failover• Multiple VTEPs (based on number of host uplinks) for Src-ID teaming

option

§ Single VTEP is sufficient for • Workloads do not drive more than 10G of throughput• Simple operational model • Deterministic traffic mapping to uplink is desired (Explicit Failover only)

§ Multiple VTEPs or LACP is required for § Workloads require > 10G of throughput• Allows flexibility of choosing teaming mode for other traffic types (non-

LACP)

• IP addressing for VTEP • Common VTEP subnet for a L2 fabric• Multiple VTEP subnets (one per rack) for L3 fabrics

§ IP Pools or DHCP can be use for IP address assignment

22

Design Considerations – VDS and Transport ZoneManagement Cluster

Edge Cluster

WebVM

WebVM

VM

VM

WebVM

WebVM

VM

VM

Compute A Compute N

NSX ManagerController Cluster NSX Edges

VXLAN Transport Zone Spanning Three Clusters

Compute VDS Edge VDS

VTEPvSphere Host

vSphere Host

192.168.230.100

192.168.230.101

Compute Cluster 1vSphere Host

192.168.240.100

Compute Cluster N

vSphere Host

192.168.240.101

vSphere Host

vSphere Host

192.168.220.100

192.168.220.101

VTEP VTEP

Active

VPN

VDS Uplink Design• NSX create dvUplink port-groups for VXLAN enabled hosts. This

uplink connectivity carrying VXLAN traffic

• Must be consistent for all hosts belonging to the VDS

• For the VXLAN traffic, the choice in teaming mode depends on• Simplicity• Bandwidth requirement

• Recommended teaming mode is “route Based on Originating Port”.

• LACP teaming mode is discouraged:• LACP teaming mode only allows single VTEP• LACP teaming forces all the other traffic types to use the

same teaming mode• LACP requires additional configuration and operational

consistency • Having multiple VDS for compute and Edge allow flexibility

of teaming more for uplink configuration

24

Teaming and Failover Mode

NSX Support

Multi-VTEPSupport

Uplink Behavior2 x 10G

Route based on Originating Port

✓ ✓ Both Active

Route based on Source MAC hash

✓ ✓ Both Active

LACP ✓ × Flow based –both active

Route based on IP Hash (Static

EtherChannnel)

✓ × Flow based –both active

Explicit Failover Order ✓ × Only one link is active

Route based on Physical NIC Load

(LBT)

× × ×

DC Topologies – NSX is Agnostic

25

VLANs & IP Subnet Defined at each ToR SVI Interface VLAN ID IP Subnet

Management 100 10.100.R_ID.x/24

vMotion 101 10.101.R_ID.x/24

Storage 102 10.102.R_ID.x/24

VXLAN 103 10.103.R_ID.x/24

VLANs & IP Subnet Defined for POD A

SVI Interface VLAN ID IP Subnet

Management 100 10.100.A.x/24

vMotion 101 10.101.A.x/24

Storage 102 10.102.A.x/24

VXLAN 103 10.103.A.x/24

VLANs & IP Subnet Defined for POD B

SVI Interface VLAN ID IP Subnet

Management 200 10.200.B.x/24

vMotion 201 10.201.B.x/24

Storage 202 10.202.B.x/24

VXLAN 103 10.103.B.x/24

VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters)

VLAN ID 100, 101 & 102 Scope VLAN ID 200, 201 and 203 Scope

Compute Cluster A32 Hosts

Compute Cluster B32 Hosts

POD A POD BL3

L2

VLAN ID 100, 101 & 102 Scope VLAN ID 200, 201 and 203 Scope

VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters)

Compute Cluster A32 Hosts

Compute Cluster B32 Hosts

L3

L2

Cluster Design with NSX

26

• Flexible DC Design– Dedicated VC for management cluster– Dedicated VC that reside in management cluster but

manages NSX domain (Edge and Compute)• Avoid circular dependencies – manager should be always

be outside of the infrastructure it manages• Mobility of mgmt. cluster for remote operations• Integration with existing VC • Ability to have more than one NSX-domains with a

dedicated VC for mgmt • Upgrade of VC not affecting – non-NSX domains. • SRM and other explicit state management is possible

– Separate Rack and ToR for• Management Cluster • Edge Cluster – Covered under the Edge design

• Simple DC Design§ Single VC for managing all components§ Separate cluster for compute § Separate Edge & MGMT Cluster§ Two VDS that spans VXLAN – compute

and Edge§ The Edge and MGMT can share the

same rack/ToR

§ Resources and components that may be hostsed Edge clusters

• Edge VMs for N-S Traffic

• Control-VM for DLR routing

• Services VM - Load Balancer and VPN

• Optional - Controllers, Monitoring, Log Insight VMs

§ Type of workload & oversubscriptions desired

• N-S vs. east-west

• Host uplinks, NIC & CPU

§ Size of DC – Small, Medium & Large

§ Single vs. multi-VC and DR with SRM

§ Availability and Scaling

• Limiting VLANs sprawl for peering with ToR

• Rack Availability and BW considerations

• Oversubscription options

27

DC Design ConsiderationEdge Cluster Design

Management&

Edge Clusters

Separate ComputeCommon Edge and Management Cluster

with NSX

WANInternet

L3

L2

ComputeCluster

Host 1

Host 3

Host 2

Host 6

Host 5

Host 4

§ Edge Cluster design and capacity planning depends on many factors

§ Components in Edge clusters• Edge VMs for N-S Traffic

• Control-VM for DLR routing

• Services VM - Load Balancer and VPN

• Optional - Controllers, Monitoring, Log Insight VMs

§ Type of workload & oversubscriptions desired• N-S vs. east-west

• Host uplinks, NIC & CPU

§ Single vs. multi-VC and DR with SRM

§ Availability and Scaling• Limiting VLANs sprawl for peering with ToR

• HA, DRS, BW and Rack Availability

§ Size of DC – Small, Medium & Large

28

Edge Cluster Design

Should I mix clusters?

Cluster Design Choices and Components Sizing

WANInternet

L3

L2

Host 1

Host 3

Host 2

Host z

Host y

Host x

What type of Edge mode is right?

Where would I put control VM?

How do I scale?

Growth & advance functions?

What is the minimum configuration?

Edge Cluster Design

29

• Minimum four hosts Cluster• Two host to hold two ECMP Edge VMs• Other two for DLR Control-VM• Not to mix ECMP and DLR-Control VM

• Avoiding race condition due to dual failures of components during host failure

• Anti-affinity is automatically enabled for DLR Control-VM• Need anti-affinity and DRS protection group for ECMP VMs

• Host Uplink & VDS • Use “SRC_ID with Failover” teaming for VXLAN traffic • Route peering maps to unique link

• Performance & Sizing• Intel, Broadcom or Emulex supporting VXLAN offload

including RSS and TSO offload• Higher CPU Clock is better• Up to 4 ECMP VMs per hosts with 4x10 GB NIC

• Oversubscription Dependent on• Upstream connectivity from the ToR• Application requirements• Density of Edge VM per hosts

L3

L2

VLAN 10 VLAN 20

Host 1 Host 2

ESG-01 ESG-02 ESG-03 ESG-04

Host 3 Host 4

VM DRS Group 1

VM DRS Group 3

VM DRS Group 2

Bridge Instance

L3

L2

Host 1

VLAN 10 VLAN 20

L3

L2

VLAN 10 VLAN 20

No over subscription 1:2 over subscription

Host 2

Host 3 Host 4Host 1 Host 2

30

Active-Standby Edge Design

L3 - ToR

Routing Adjacency

vSphere Host vSphere Host

VXLAN 5020Transit Link

Active-StandbyStateful

FW/NAT/LB

• Active-Standby Edge Services Gateway enables stateful services • Perimeter FW, NAT, LB, SSL-VPN, North-South routing• Deployed in pair with heartbeat and synchronization of

services state• Heartbeat and sync both use the same internal vNIC • L2 connectivity required between active and standby• Form factor – X-Large to Compact (one license)

• Multi-interface routing Support – OSPF & BGP• Must tune protocol timers to 40/120(hello/hold timer)

• Anti-affinity is automatically created• Active and Standby Edges are placed on different hosts• Minimum three hosts are recommended• Automatic resource reservation – from 6.2.4 onward

• Multiple instance to Edge can be deployed• LB Edge can be deployed near application tire• Multiple tenants can have separate Edge services

31

ECMP Based Edge Design -

ECMP EdgesNon-Stateful

VXLAN

VLAN

Transit VXLAN

E1 E2 E7 E8

R1 R2

External Network

VLAN 10

VLAN 20

ECMP Active NSX Edges

Customer Routers

…

• ECMP Edge enables scalable north-south traffic forwarding services • 8 instances of Edge • Stateful services are not supported due to

asymmetric traffic behavior• No heartbeat and sync between Edge nodes • L2 connectivity required for peering to

physical• Form factor – X-Large to Compact (one

license)• Multi-interface routing Support – OSPF &

BGP• Aggressing timers tuning supported 1/3

(hello/hold timer)• Anti-affinity configuration is required

• Minimum three hosts are recommended• Multiple tenants can have separate Edge

services

High Availability with Edge ClusterEdge Active-Standby Guideline• Minimum three hosts• Anti-affinity automatic enforced for Edge

Active-Standby mode • Heart beat timers can be reduced from 15

seconds default to 9 seconds– Internal link for heartbeat for failure detection

• vSphere HA and standby takeover with HB failure routing timer 40/120 Sec.

• Control VM active-standby anti-affinity is automatic– HB timers for control VM typically do not

requires modification

32

ECMP HA Guideline• Minimum 3 hosts, more if higher BW (VM)

is required• DRS and HA policy governed by how

many VMs per host (dependency on BW and uplinks per hosts – 2x10 or 4x10)

• Active Control-VM and Edge VM can not be in same host to avoid dual failure

• Aggressive timers support up to 1/3 sec• Control VM active-standby anti-affinity is

automatic– HB timers for control VM typically do not

requires modification

• In either Edge mode, the vMotion is supported and does not have impact on control plan

ECMP with DLR and Edge§ ECMP support on the DLR and on the NSX Edge

• Both have the capability of installing in their forwarding tables up to 8 equal cost routes toward a given destination

• 8 NSX Edges can be simultaneously deployed for a given DLR+Edge combination - per tenant

§ Increase the available bandwidth for North-South communication • Reduces the traffic outage in an ESG failure scenario

(only 1/N of the flows are affected)

§ The routing protocol timer can be reduced up to 1 second for hello and 3 second for hold for both protocol OSPF and BGP

§ Recommended best practices is not to have ECMP Edge VM and DLR control VM on the same host to avoid the dual failure and triggering much longer outage due to aggressive timers • Anti-affinity is automatic between active-standby

control VM and enable DRS protection between ECMP edge VM and DLR control-VM

• Disabled GR on ECMP Edge to improve convergence

Web DB

DLR

E1

Physical Router

App

DC Fabric

Active ECMPE8

VLAN 20

Transit VXLAN

Active Standby

RoutingAdjacency…

VLAN 10

Edge HA Models Comparison – BW, Services & ConvergenceActive/Standby HA Model

Bandwidth Single Path(~10 Gbps/Tenant)

Stateful Services Supported - NAT, SLB, FW

Availability Low convergence with stateful services enabled

ECMP Yes with Two Uplinks on Edge HA

ECMP Model

Bandwidth Up to 8 Paths(~80 Gbps/Tenant)

Stateful Services Not Supported

AvailabilityHigh

~ 3-4 sec with (1,3 sec) timers tuning

E1

Physical Router

Active StandbyE2

Routing Adjacency

Web DB

DLR

AppActive Standby

DLRControl VM

…E8E3E1

Physical Router

E2

Routing Adjacencies

Web DB

DLR

App

Active Standby

DLRControl VM

Routing Protocol & Topology• Enterprise vs. DC Routing

– Core & Aggregation Routing – Handled by backbone team– DC Spine-Leaf handled by DC Team– Protocol choices and its connectivity differs in DC

• 10G Links, Convergence is TCP centric – not sub-seconds

• DC Routing Protocol Choices– Mostly OSPF, BGP is prevalent in large scale design and

growing acceptance in enterprise DC fabric

• DC Routing Design and NSX Scope– NSX domain is connected as Edge routing

• Edge routing features is sufficient for most needs• Does not act as transport network

– Simplified protocol features and connectivity for NSX• Use consistent one protocol end-to-end between ESG and

physical and DLR to ESG• Adopting the established routing best practices

– Summarization– reducing the routing churn as well as protecting black hole of traffic– Treating NSX as another autonomous system or stub topology

35

OSPF/BGP

L3

L2

WAN/Internet

AS 65501

AS 65002

NSX Routing Domain

DLR

Compute

NSX Component Mapping

36

Separation of compute, management and Edge function with following design advantage

• Management Cluster• Can co-exist with Edge Cluster in same UCS Chassis• Minimum three hosts – more if needed• LACP can be used on rack-mount

• Edge Cluster• Should be independent UCS C series• Edge VM for North-south traffic • Active-standby Control-VM• Can hold NSX Controller is optimization of resources

is desired

• Compute Cluster

• VXLAN is enabled per cluster• Can co-exist with physical bare-metal compute

Function NSX ComponentsRecommended

Cluster Designation

ManagementPlane NSX Manager & VC Management

Cluster

Control Plane

NSX Controller Cluster

Management or Edge Cluster

Logical Router Control VM Edge Cluster

Data Plane East-West

VXLAN forwarding -compute and edge

VDS kernelcomponents &

DLR(Distributed Logical Routers)

Compute and Edge Cluster

Data PlaneNorth-South

ECMP Edge or Edge Services Edge Cluster

Bridging Traffic DLR Control VM Edge Cluster

§ Mgmt Cluster is typically provisioned with limited severs:• Follows minimum CPU & Memory for management components

• VC, Mgmt, vRA, vROPs etc.

• Static allocation, idle capacity can be used for other resources

§ The single rack design still requires redundant uplinks from host to ToR carrying VLANs for management• LACP teaming mode can be helpful

§ Larger design tends to have dedicated management cluster • Exclude management cluster from preparing VXLAN

§ In a smaller design, management and Edge cluster can be combined

• NSX manager and Controllers automatically excluded from DFW functions.

• Put vCenter server in DFW exclusion list

• Use resource reservation to protect the Edge VMs

LeafL2

L3

VMkernel VLANs

VLANs for Management VMs

L2

VMkernel VLANs

DC Fabric

802.1Q Trunk

Management Cluster Connectivity

L2

37

Management Cluster Deployment Considerations

• Rack based vs. multi-rack (horizontal) stripping• Availability vs. localized domain – CPU & mobility constraint &

simplification of connectivity (IP, VTEP, Automation)

• Lifecycle of the workload drives the consideration for• Growth, availability and changes in the application flows• Multi-rack, zoning ( type of customer, tenancy etc.)

• Typically rack connectivity is streamlined and repeated • Same four VLANs typically streamlines the configuration of ToR• Connectivity to the fabric and requirement for additional capacity

remains the same since its abstracted from infrastructure

• Workloads type, compliance and SLA can be met via• Cluster separation• Separate VXLAN network• Per tenant separation routing domains• DRS

Compute Cluster Connectivity

38

DC Design ConsiderationCompute Cluster

Management

WANInternet

L3

L2

ComputeCluster

Host 1

Host 3

Host 2

Host 6

Host 5

Host 4

Host 1

Host 3

Host 2

Host 6

Host 5

Host 4

Compute Clusters

L3

L2

DC Fabric

Edge Clusters

Organizing Compute, Management & Edge Separation of compute, management and Edge function with following design advantage

• Managing life-cycle of resources for compute and Edge functions

• Ability to isolate and develop span of control• Capacity planning – CPU, Memory & NIC• Upgrades & migration flexibility

• High availability based on functional need• Workload specific SLA (DRS & FT)• Network centric connectivity – P/V, ECMP• vMotion boundary

• Automation control over area or function that requires frequent changes

• app-tier, micro-segmentation & load-balancer

Three areas of technology require considerations• Interaction with physical network• Overlay (VXLAN) impact• Integration with vSphere clustering

Management

WANInternet

L3

L2

Compute

Host 1

Host 3

Host 2

Host 6

Host 5

Host 4

Host 1

Host 3

Host 2

Host 6

Host 5

Host 4

Compute

L3

L2

DC Fabric

Edge

NSXEDGE

NSXEDGE

NSXEDGE

NSXEDGE

Large DC Cluster Design

40

• Workload characteristics– Variable– On-demand– Compliance requirements

• For cross-VC and SRM Deployment– Separation of management cluster is inevitable

• Large scale Edge Cluster Design– Dedicated minimum four hosts– Minimum four ECMP Edge (Quad Large) 40 GB total BW

– Separate host with DRS protection between ECMP Edge VM and Active Control-VM

– Capacity for services VMs

• Edge VM CPU Guideline– Ideally >= 2.6 GHz with 10 core to hold min two ECMP VMs

for 20 GB (2x10 NIC) bandwidth– Higher cores can be used to consolidate VMs but may need

4x10 GB network ports– Keep the CPU/Socket consistent for Edge cluster to have

flexibility

Edge Cluster

Separate Management Compute& Edge and Cluster with NSX

Management

WANInternet

L3

L2

Compute

Host 1

Host 3

Host 2

Host 6

Host 5

Host 4

Host 1

Host 3

Host 2

Host 6

Host 5

Host 4

Compute

L3

L2

DC Fabric

Cluster Design with Medium DC

41

• Mixing compute and edge workload requires– Balanced Compute workload can be mixed with Edge VM

resources– However the growth of compute can put additional burden on

managing resource reservation to protect the Edge VM CPU

• Collapsing edge OR compute with management components (VC and NSX manager) – Requires management component to be dependent on VXLAN since

VXLAN enablement is per cluster bases– Expansion or decoupling of management required for growth

• moving management cluster to remote location• Having multiple VCs to manage separation

• Mixing Edge and Management is a better strategy– Consistent static requirements of the resources – mgmt. is relatively

time idle resources compared to compute workload

• For growth consider separation of edge and mgmt. cluster

Management&

Edge Clusters


with NSX

WANInternet

L3

L2

ComputeCluster

Host 1

Host 3

Host 2

Host 32

Host y

Host x

Cluster Design with Medium DC - Continue

42

• Small to medium cluster can utilize the edge service gateway features where – N-S BW is not more then 10 G– Desire to reduce external FW usage with Edge FW

functionality– Using built in Load Balancer– Use VPN or SSL functionality

• Edge Services Sizing– Start with Large (2 vCPU) if the line rate BW is not

required– Can be upgraded to Quad-Large (4 vCPU) for growth

in BW

• Consider LB in single arm mode to be near the application segments

• Decouple the need to Edge service mode choice if only LB service is required

Management&

Edge Clusters


with NSX

WANInternet

L3

L2

ComputeCluster

Host 1

Host 3

Host 2

Host 32

Host y

Host x

Small Design Considerations• Understanding the workload that impact NSX

component selection – Edge workload is CPU centric with consistent memory –

except in L7 load-balancer Edge services– Edge resources requires external connectivity to VLANs

thus restricting it location to avoid VLAN sprawl

• Single cluster for small design, expand to medium with separation of compute cluster– Single cluster can start with DFW only design

• NSX Manger is the only component required• VDS license comes with NSX

– Progress to full stack for other services such as FW, LB, VPN and VxLAN• ESG in active-standby – Large form factor• Quad-large if needed for firewall throughput• Static routing for simplicity and reduced need of

deploying control-VM

43

Single Cluster with NSX

WANInternet

L3

L2

Host 1

Host 3

Host 2

Host 32

Host y

Host x

Small, Medium and Large Virtualized DC – NSX Scales Consistently

44

Sizing VC Workload Edge Type N-S BW GB Cluster Choice Requirement ResourceReservation

Small 1 ConsistentLarge - 2 vCPU

< 20 G Collapsed Harder to

separate Mgmt. later

Need for Edge VMsESG or ECMP

Med 1Consistence

Some on-demand

Large to Quad 2 or 4 vCPU

< 40G Mgmt./EdgeGrowth not likely, No other smaller

DC

Need for Edge VMs

ESG or ECMP

Med (with multiple DC or compute

growth)

>1 On-demandWith DR

Quad – 4 vCPU

<= 40G

Separate Mgmt., Edge and ComputeCluster

Growth or other DC integration

must

If needed formix useECMP for N-S

ESG for local LB

Large >1

Variable, on-demand, DR,Inter-site and

dev-ops

Quad – 4 vCPU> 40G and multi-tenant

Separate Mgmt., Edge and ComputeCluster

Scale & Availability NA

Multi-tier for services


2 NSX Components

3 NSX Design

4 Summary

Agenda

45

SecurityInherently secure infrastructure

Automation IT at the speed of business

Application continuityData center anywhere

NSX customer use cases

Micro-segmentation

DMZ anywhere

Secure end user

IT automating IT

Multi-tenant infrastructure

Developer cloud

Disaster recovery

Cross cloud

Multi data center pooling

47

NSX Reference Designs

NSX Platform

Hardening

NSX Getting StartedGuides

SDDCValidatedSolutions

NSX PartnerWhite

papers

Reference Designs & Technical Papers on VMware Communities: https://communities.vmware.com/docs

Reference Designs and Technical Papers on the NSX Portal: http://www.vmware.com/products/nsx/resources.html

NSX andFabric

Vendors

VMware NSX Collateral Landscape

Resources

Read vmware.com

Design, Admin, and User Guides

Ops Guides (Monitoring,

Troubleshooting)

White PapersTrylabs.hol.vmware.com

Hands-on Labs

Takevmware.com/education

Courses – ICM, Design and Deploy, Troubleshooting and OperationsCertifications – VCA, VCP, VCIX, VCDX

Operations Transformation Services (OTS)

Usevmware.com/consulting

Thank you.

Technology

VMUG - NSX Architettura e Design