Varrow Madness 2014 - VSAN Deep Drive

2014 VMware Inc. All rights reserved.

VMware Virtual SAN Below the Marketing

Jason Nash, VCDX Chief Technology Officer, Varrow [email protected] @TheJasonNash

Storage & Availability Technical Marketing Resources

Team

Group Manager Charu Chaubal [email protected]

Storage Rawlinson Rivera

[email protected] Wade Holmes [email protected]

Availability Ken Werneburg

[email protected] Jeff Hunter [email protected] GS Khalsa [email protected]

Big Data Justin Murray [email protected]

Resources

Blog: http://blogs.vmware.com/vsphere

Tablet & Mobile app: http://vmware.com/go/vmkp

Hands-on Labs: http://hol.vmware.com

Product Walkthrough Demos: http://vmwarewalkthroughs.com

VM Vault: search by topic or author

2

Virtual SAN Technical Walkthrough

1 SDS and Virtual SAN Overview

2 Use Cases

3 Hardware Requirements

4 Technical Characteristics and Architecture

5 Configuration Walkthrough

6 Virtual Machine Provisioning Operations

7 Resiliency and Failed Scenarios

8 Interoperability

9 Design and Sizing

10 Troubleshooting 3

2011-2013

2008-2010

2005-2007

vSphere 4.x Thin Provisioning Storage I/O control Boot from SAN VAAI Linked mode

2014+

VI 3.x VMFS Snapshots Storage vMotion NAS & iSCSI

support

vSphere 5.x Storage DRS Profile-driven Storage VASA Sphere Storage

Appliance vSphere Data Protection vSphere Replication vSphere Flash Read

VMware Storage Innovations

4

Software-defined Storage

Hypervisor-Converged Opportunities

Why the virtualization platform can play a critical role to solve storage problems?

Inherent knowledge of application

Global view of infrastructure

Hardware agnostic

Hypervisor-Converged storage solutions abstract the plumbing to optimize storage for applications

vSphere

SAN & NAS All Flash BLOB DAS

Server Side Flash

5

Virtual SAN

6

vSAN

VSAN

SAN / NAS

VVOL

VMware Software-defined Storage

7

Bringing computes operational model to storage

Abstraction and pooling Infrastructure integration New storage tiers

VM centric data services Third-party services

integration

Common policy-based automation and orchestration

LUN LUN LUN

LUN LUN

LUN

SAN/NAS Pool

Virtual Data Plane

x86 Servers

Hypervisor-converged Storage pool

Object Storage Pool

Cloud Object Storage

Virtual Data Services

Data Protection Mobility Performance

Policy-driven Control Plane

Virtual SAN

VMware Virtual SAN

Software-defined storage software solution.

Aggregates locally attached storage from each ESXi host in a cluster.

Flash optimized storage solution.

VM-Centric data operations and policy driven management principals.

Resilient design based on a Distributed RAID architecture No single points of failures

Fully integrated with vSphere.

8

vSphere + Virtual SAN

Hard disks Hard disks SSD SSD Hard disks

SSD

Virtual SAN Shared Datastore

Hypervisor-Converged storage platform

9

Data Protection

Disaster Recovery

vSphere

Cloud Ops and Automation

Snapshots Linked clones

Site Recovery Manager vCenter Operations Manager vCloud Automation Center

Virtual Desktop VDP Advanced

vSphere Replication

Storage Policy-Based Management

VMware Horizon View vMotion vSphere HA

DRS Storage vMotion

IaaS

Deeply Integrated with VMware Stack

Bringing the benefit of VMwares products to make Storage Easy

Virtual SAN is NOT a Virtual Storage Appliance

10

Virtual SAN is fully integrated with vSphere (ESXi & vCenter) Drivers embedded in ESXi 5.5 contain the Virtual SAN smarts Kernel modules:

Provide the shortest path for I/O Remove unnecessary management overheads when dealing with an

appliance Do not consume resources unnecessarily

Virtual SAN Embedded into vSphere Virtual SAN Not a VSA

VSA

VMware Virtual SAN

Hybrid storage solution Magnetic disks (HDD) Flash based disks (SSD)

Storage scale out architecture built into the hypervisor

Dynamic capacity and performance scalability

Object based storage architecture

Interoperable with vSphere and enterprise features: vMotion, DRS, vSphere HA

11


Hard disks Hard disks SSD SSD Hard disks

SSD

Virtual SAN Shared Datastore

Radically Simple Hypervisor-Converged Storage Software

12

Installs in two clicks Managed from vSphere

Client

Policy-based management Self-tuning and elastic Deep integration with

VMware stack

Radically Simple

Embedded in vSphere kernel

Flash-accelerated Up to 915K IOPs from 16

nodes cluster

Matches the VDI density of all flash array

Best price/performance

High Performance Lower TCO

Eliminates large upfront investments (CAPEX)

Grow-as-you-go (OPEX) Flexible choice of industry

standard hardware Does not require

specialized skills

Virtual SAN Key Benefits

Storage Policy-Based Management

VSAN Shared Datastore

Simplifies and Automates Storage Management

13

Per VM storage service levels from a single self-tuning datastore

Capacity

Performance

Availability

Per VM Storage Policies

Policies set based on application needs

vSphere + VSAN

SLAs

Software automates control of service levels

No more LUNs!

No overprovisioning Less resources, less time Easy to change

Today

5. Consume from pre-allocated bin

4. Select appropriate bin

3. Expose pre-allocated bins

2. Pre-allocate static bins

1. Pre-define storage configurations

1. Define storage policy

2. Apply policy at VM creation

VSAN

VSAN Shared

Datastore

Resource and data service are automatically provisioned and

maintained

Overprovisioning (better safe than sorry!) Wasted resources, wasted time Frequent Data Migrations

Virtual SAN Puts The App In Charge

Simpler and automated storage management through application centric approach

14

Oh yeah! Scalability..

15

vsanDatastore

4.4 Petabytes

2 Million IOPS

32 Hosts

VMware Virtual SAN Use Cases

17

Management Clusters

Use Cases

Backup and DR Target

DMZ / Isolated

Tier 2 / Tier 3 Test / Dev / Staging

Private cloud

Virtual Desktop

ROBO

VDI

Site A Site B

vSphere VSAN

VMware Virtual SAN Hardware Requirements

Hardware Requirements

19

Any Server on the VMware Compatibility Guide

SSD, HDD, and Storage Controllers must be listed on the VMware Compatibility Guide for VSAN http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan

ESXi 5.5 Hosts: Minimum 3, Maximum 32

1Gb/10Gb NIC

SAS/SATA Controllers (RAID Controllers must work in pass-through or RAID0 mode

SAS/SATA/PCIe SSD

SAS/NL-SAS/SATA HDD

At least 1 of each

4GB to 8GB USB, SD Cards

Flash Based Devices

In Virtual SAN ALL read and write operations always go directly to the Flash tier.

Flash based devices serve two purposes in Virtual SAN 1. Non-volatile Write Buffer (30%)

Writes are acknowledged when they enter prepare stage on SSD. Reduces latency for writes

2. Read Cache (70%) Cache hits reduces read latency Cache miss retrieve data from HDD

Choice of hardware is the #1 performance differentiator between Virtual SAN configurations.

20

Flash Based Devices

VMware SSD Performance Classes Class A: 2,500-5,000 writes per second Class B: 5,000-10,000 writes per second Class C: 10,000-20,000 writes per second Class D: 20,000-30,000 writes per second Class E: 30,000+ writes per second

Examples Intel DC S3700 SSD ~36000 writes per

second -> Class E Toshiba SAS SSD MK2001GRZB ~16000

writes per second -> Class C

Workload Definition Queue Depth: 16 or less Transfer Length: 4KB Operations: write Pattern: 100% random Latency: less than 5 ms

Endurance 10 Drive Writes per Day (DWPD), and Random write endurance up to 3.5 PB on

8KB transfer size per NAND module, or 2.5 PB on 4KB transfer size per NAND module

21

Magnetic Disks (HDD)

SAS/NL-SAS/SATA HDDs supported 7200 RPM for capacity 10000 RPM for performance 15000 RPM for additional performance

NL SAS will provide higher HDD controller queue depth at same drive rotational speed and similar price point NL SAS recommended if choosing between SATA and NL SAS

Differentiate performance between clusters with SSD selection, and SSD:HDD ratio. Rule of thumb guideline is 10% of anticipated capacity usage

22

Storage Controllers

SAS/SATA Storage Controllers Pass-through or RAID0 mode supported

Performance using RAID0 mode is controller dependent Check with your vendor for SSD performance behind a RAID-controller

Storage Controller Queue Depth matters Higher storage controller queue depth will increase performance

Validate number of drives supported for each controller

23

Storage Controllers RAID0 Mode

Configure all disks in RAID0 mode Flash based devices (SSD) Magnetic disks (HDD)

Disable the storage controller cache Allows better performance as cache is controlled by Virtual SAN

Disks Device cache support Flash based devices leverage write through caching

ESXi may not be able to differentiate flash based devices from magnetic devices. Use ESXCLI to manually flag the devices as SSD

24

Network

1Gb / 10Gb supported 10Gb shared with NIOC for QoS will support

most environments If 1GB then recommend dedicated links for Virtual SAN

Jumbo Frames will provide nominal performance increase Enable for greenfield deployments

Virtual SAN supports both VSS & VDS NIOC requires VDS Nexus 1000v Should work but hasn't been fully tested

Network bandwidth performance has more impact on host evacuation, rebuild times than on workload performance

25

Firewalls

Virtual SAN Vendor Provider (VSANVP) Inbound and outbound - TCP 8080

Cluster Monitoring, Membership, and Monitoring Services (CMMDS) Inbound and outbound UDP 12345 - 23451

Reliable Datagram Transport (RDT) Inbound and outbound TCP 2233

26

VMware Compatibility Guide

27

Multi-level cell SSD (or better) or PCIe SSD

SAS/NL-SAS HDD Select SATA HDDs

Any Server on vSphere Hardware Compatibility List

* Note: For additional details, please refer to Virtual SAN VMware Compatibility Guide Page

6Gb enterprise-grade HBA/RAID Controller

1 2 Build your own VSAN Ready Node

with 10 different options between multiple 3rd party vendors available at GA

Preconfigured server ready to use VSAN

using the VSAN Compatibility Guide*

Choose individual components

Two Ways to Build a Virtual SAN Node

Radically Simple Hypervisor-Converged Storage

VMware Virtual SAN Technical Characteristics and Architecture

Technical Characteristics Virtual SAN is a cluster level feature similar to:

vSphere DRS vSphere HA Virtual SAN

Deployed, configured and manage from vCenter through the vSphere Web Client (ONLY!).

Radically simple Configure VMkernel interface for Virtual SAN Enable Virtual SAN by clicking Turn On

30

Virtual SAN Implementation Requirements

Virtual SAN requires: Minimum of 3 hosts in a cluster

configuration All 3 host MUST!!! contribute

storage vSphere 5.5 U1 or later

Maximum of 32 hosts

Locally attached disks Magnetic disks (HDD) Flash-based devices (SSD)

Network connectivity 1GB Ethernet 10GB Ethernet (preferred)

31

esxi-01

local storage local storage local storage

vSphere 5.5 U1 Cluster

esxi-02 esxi-03

cluster

HDD HDD HDD

Storage Policy-based Management

SPBM is a storage policy framework built into vSphere that enables virtual machine policy driven provisioning.

Virtual SAN leverages this new framework in conjunction with VASA APIs to expose storage characteristics to vCenter: Storage capabilities

Underlying storage surfaces up to vCenter and what it is capable of offering.

Virtual machine storage requirements Requirements can only be used against available capabilities.

VM Storage Policies Construct that stores virtual machines storage provisioning requirements based on

storage capabilities.

32

Storage Policy Wizard

SPBM

VSAN object

VSAN object manager

virtual disk

VSAN objects may be (1) mirrored across hosts & (2) striped across disks/hosts to meet VM storage profile policies

Datastore Prole

Virtual SAN SPBM Object Provisioning Mechanism

Virtual SAN Constructs and Artifacts

New Virtual SAN constructs, artifacts and terminologies:

Disk Groups. VSAN Datastore. Objects. Components. Virtual SAN Network.

34

Virtual SAN Disk Groups Virtual SAN uses the concept of disk groups to pool together flash

devices and magnetic disks as single management constructs.

Disk groups are composed of at least 1 flash device and 1 magnetic disk. Flash devices are use for performance (Read cache + Write buffer). Magnetic disks are used for storage capacity. Disk groups cannot be created without a flash device.

35

disk group disk group disk group disk group Each host: 5 disk groups max. Each disk group: 1 SSD + 1 to 7 HDDs

disk group

HDD HDD HDD HDD HDD

Virtual SAN Datastore Virtual SAN is an object store solution that is presented to vSphere as

a file system.

The object store mounts the VMFS volumes from all hosts in a cluster and presents them as a single shared datastore. Only members of the cluster can access the Virtual SAN datastore Not all hosts need to contribute storage, but its recommended.

36

disk group disk group disk group disk group

Each host: 5 disk groups max. Each disk group: 1 SSD + 1 to 7 HDDs

disk group

VSAN network VSAN network VSAN network VSAN network VSAN network

vsanDatastore

HDD HDD HDD HDD HDD

Virtual SAN Objects Virtual SAN manages data in the form of flexible data containers called

objects. virtual machine files are referred to as objects.

Virtual machines files are referred to as objects. There are four different types of virtual machine objects:

VM Home VM swap VMDK Snapshots

Virtual machine objects are split into multiple components based on performance and availability requirements defined in VM Storage profile.

37



disk group


vsanDatastore

HDD HDD HDD HDD HDD

Virtual SAN Components Virtual SAN components are chunks of objects distributes across

multiple hosts in a cluster in order to tolerate simultaneous failures and meet performance requirements.

Virtual SAN utilizes a Distributed RAID architecture to distribute data across the cluster.

Components are distributed with the use of two main techniques: Striping (RAID0) Mirroring (RAID1)

Number of component replicas and copies created is based on the object policy definition.

38

disk group disk group disk group disk group disk group


vsanDatastore

replica-1 replica-2 RAID1

HDD HDD HDD HDD HDD

Object and Components Layout

39


Virtual SAN Storage Objects

R1

R0 R0 R0

Availability defined as number of copies

Low level storage objects would reside on different hosts

VMFS VMFS VMFS

rolo2.vmdk

The VM Home directory object is formatted with VMFS to allow a VMs configuration files to be stored on it.

Performance may include a stripe width

VMFS

rolo1.vmdk rolo.vmx, .log, etc

/vmfs/volumes/vsanDatastore/rolo/rolo.vmdk

disk group

HDD

disk group

HDD

disk group

HDD

disk group

HDD

disk group

HDD

Virtual SAN Network New Virtual SAN traffic VMkernel interface.

Dedicated for Virtual SAN intra-cluster communication and data replication.

Supports both Standard and Distributes vSwitches Leverage NIOC for QoS in shared scenarios

NIC teaming used for availability and not for bandwidth aggregation. Layer 2 Multicast must be enabled on physical switches.

Much easier to manage and implement than Layer 3 Multicast

40

Management Virtual Machines vMotion Virtual SAN

Distributed Switch

20 shares 30 shares 50 shares 100 shares

uplink1 uplink2

vmk1 vmk2 vmk0

Virtual SAN Network

NIC teamed and load balancing algorithms: Route based on Port ID

active / passive with explicit failover

Route based on IP Hash active / active with LACP port channel

Route based on Physical NIC load active / active with LACP port channel

Management Virtual Machines vMotion Virtual SAN

Distributed Switch

100 shares 150 shares 250 shares 500 shares

uplink1 uplink2

vmk1 vmk2 vmk0

Multi chassis link aggregation capable switches

Virtual SAN Scalable Architecture

42

Scale up and Scale out architecture granular and linearly storage, performance and compute scaling capabilities Per magnetic disks for capacity Per flash based device for performance Per disk group for performance and capacity Per node for compute capacity

disk group disk group disk group

VSAN network VSAN network VSAN network

vsanDatastore

HDD

disk group

HDD HDD HDD

disk group

VSAN network

HDD scal

e up

scale out

VMware Virtual SAN Configuration Walkthrough Demo

Configuring VMware Virtual SAN

Radically Simple configuration procedure

44

Setup Virtual SAN Network

Enable Virtual SAN on the

Cluster

Select Manual or Automatic

If Manual, create disk

groups

Configure Network

45

Configure the new dedicated Virtual SAN network vSphere Web Client network template configuration feature.

Enable Virtual SAN One click away!!!

Virtual SAN configured in Automatic mode, all empty local disks are claimed by Virtual SAN for the creation of the distributed vsanDatastore.

Virtual SAN configured in Manual mode, the administrator must manually select disks to add the the distributed vsanDatastore by creating Disk Groups.

46

Disk Management

Each host in the cluster creates a single or multiple disk groups which contain a combination of HDDs, and SSDs.

47

Virtual SAN Datastore

A single Virtual SAN Datastore is created and mounted, using storage from all multiple hosts and disk groups in the cluster.

Virtual SAN Datastore is automatically presented to all hosts in the cluster.

Virtual SAN Datastore enforces thin-provisioning storage allocation by default.

48

VM Storage Policies

VM Storage Policies are accessible from vSphere Web Client Home screen.

49

Virtual SAN Capabilities

Virtual SAN currently surfaces five unique storage capabilities to vCenter.

50

Number of Failures to Tolerate

Number of failures to tolerate Defines the number of hosts, disk or network failures a storage object can

tolerate. For n failures tolerated, n+1 copies of the object are created and 2n+1 host contributing storage are required.

51

vsan network

vmdk vmdk witness

esxi-01 esxi-02 esxi-03 esxi-04

~50% of I/O ~50% of I/O

Virtual SAN Policy: Number of failures to tolerate = 1

raid-1

Number of Disk Stripes Per Object

Number of disk stripes per object The number of HDDs across which each replica of a storage object is

distributed. Higher values may result in better performance.

52

vsan network

stripe-2b witness


stripe-1b stripe-1a stripe-2a

raid-0 raid-0

VSAN Policy: Number of failures to tolerate = 1 + Stripe Width =2

raid-1

Virtual SAN Storage Capabilities

Force provisioning if yes, the object will be provisioned even is the policy specified in the

storage policy is not satisfiable with the resources currently available.

Flash read cache reservation (%) Flash capacity reserved as read cache for the storage object. Specified as a

percentage of logical size of the object.

Object space reservation (%) Percentage of the logical size of the storage object that will be reserved

(thick provisioned) upon VM provisioning. The rest of the storage object is thin provisioned.

53

Virtual SAN I/O flow Write Acknowledgement

vsan network

vmdk vmdk


VSAN mirrors write IOs to all active mirrors, these are acknowledged when they hit the flash buffer!

witness

Destaging to HDD is done independently between hosts.

raid-1

Virtual SAN I/O flow 1MB increment striping

vsan network

witness


stripe-1b

stripe-1a 1MB(1)

raid-0 raid-0

VSAN is thin provisioned by default, stripes grow in increments of 1MB

raid-1

1MB(3) 1MB(5) 1MB(2) 1MB(4)

(x) indicates stripe segment.

Components and Objects Visualization

Visualization of mapping and layout of all objects and components vSphere Web Client RVC

56

Storage Capabilities Recommended Practices

57

Storage Capability Use Case Value

Number of failures to tolerate (RAID 1 Mirror) Redundancy

Default 1 Max 3

Number of disk stripes per object (RAID 0 Stripe) Performance

Default 1 Max 12

Object space reservation Thick Provisioning Default 0 Max 100%

Flash read cache reservation Performance Default 0 Max 100%

Force provisioning Override policy Disabled

VM Storage Policies Recommendations

Number of Disk Stripes per object Should be left at 1, unless the IOPS requirements of the VM is not being

met by the flash layer.

Flash Read Cache Reservation Should be left at 0, unless there is a specific performance requirement to be

met by a VM.

Proportional Capacity Should be left at 0, unless thick provisioning of virtual machines is required.

Force Provisioning Should be left disabled, unless the VM needs to be provisioned, even if not

in compliance.

58

VMware Virtual SAN Virtual Machine Provisioning Operations Demo

Virtual Machine Provisioning Operations

All VM provisioning operation include access to VM Storage Policies

60


If the VSAN Datastore understands the capabilities in the VM Storage Policy, it will be displayed as a matching resource.

61


If the VSAN Datastore can satisfy the VM Storage Policy, the VM Summary tab will display the VM as compliant.

If not, due to failures, or the force

provisioning capability, the VM will be shown as non-compliant.

62

Virtual Machine Policy Management

Modify VM performance, capacity, and availability requirements without downtime.

63

VMware Virtual SAN Resiliency & Failure Scenarios

Understanding Failure Events

Virtual SAN recognized two different types of hardware device events in order to define the type of failed scenario: Absent Degraded

Absent events are responsible to trigger the 60 minutes recovery operations. Virtual SAN will wait 60 minutes before starting the object and component

recovery operations 60 minutes is the default setting for all absent events Configurable value via hosts advanced settings

65

Understanding Failure Events

Degraded events are responsible to trigger the immediate recovery operations. Triggers the immediate recovery operation of objects and components Not configurable

Any of the following detected I/O errors are always deemed degraded: Magnetic disk failures Flash based devices failures Storage controller failures

Any of the following detected I/O errors are always deemed absent: Network failures Network Interface Cards (NICs) Host failures

66

Failure handling philosophy

Traditional SANs Physical drive needs to be replaced to get back to full redundancy Hot-spare disks are set aside to take role of failed disks immediately In both cases: 1:1 replacement of disk

Virtual SAN Entire cluster is a hot-spare, we always want to get back to full redundancy When a disk fails, many small components (stripes or mirrors of objects) fail

New copies of these components can be spread around the cluster for balancing Replacement of the physical disk just adds back resources

Managing Failure Scenarios

Through policies, VMs on Virtual SAN can tolerate multiple failures Disk Failure degraded event SSD Failure degraded event Controller Failure degraded event Network Failure absent event Server Failure absent event

VMs continue to run Parallel rebuilds minimize performance pain

SSD Fail immediately HDD Fail immediately Controller Fail immediately Network Fail 60 minutes Host Fail 60 minutes

68

Virtual SAN Access Rules

Components Access Rules At least 1 mirror copy intact All stripes must be intact Greater than 50% of components must be available

Including witnesses

1 Mirror Copy All stripes available

> 50% components

and witnesses

Power on Operation

Logic is implemented per object

Magnetic Disk Failure Instant mirror copy

Degraded - All impacted components on the failed disk will be instantaneously created onto other disk, disk groups, or hosts.

vsan network

vmdk vmdk witness


vmdk

new mirror copy Instant!

Disk failure, instant mirror copy of impacted component

raid-1

Flash Based Device Failure Instant mirror copy

Degraded - All impacted components on the failed disk will be instantaneously created onto other disk, disk groups, or hosts.

Greater impact on the cluster overall storage capacity

vsan network

vmdk vmdk witness


vmdk

new mirror copy Instant!

Disk failure, instant mirror copy of impacted component

raid-1

Host Failure 60 Minute Delay

Absent will wait the default time setting of 60 minutes before starting the copy of objects and components onto other disk, disk groups, or hosts.

Greater impact on the cluster overall compute and storage capacity.

vsan network

vmdk vmdk witness


vmdk

new mirror copy 60 minute wait

Host failure, 60 minutes wait copy of impacted component

raid-1

Network Failure 60 Minute Delay

Absent will wait the default time setting of 60 minutes before starting the copy of objects and components onto other disk, disk groups, or hosts.

NIC failures, physical network failures can lead to network partitions. Multiple hosts could be impacted in the cluster.

vsan network

vmdk vmdk witness


vmdk

new mirror copy 60 minute wait

Network failure, 60 minutes wait copy of impacted component

raid-1

Virtual SAN 1 host isolated HA restart

vsan network

vmdk vmdk witness

esxi-01 esxi-02 esxi-03 esxi-04 isolated!

HA restart raid-1

vSphere HA restarts VM

Virtual SAN 2 hosts isolated HA restart

vsan network

vmdk vmdk witness

esxi-01 esxi-02 esxi-03 esxi-04 isolated! isolated!

HA restart raid-1

vSphere HA restarts VM on ESXi-02 / ESXi-03, they own > 50% of components!

Virtual SAN partition With HA restart

vsan network

vmdk vmdk witness

esxi-01 esxi-02 esxi-03 esxi-04 Partition 1 Partition 2

HA restart

vSphere HA restarts VM in Partition 2, it owns > 50% of components!

raid-1

Maintenance Mode planned downtime

3 Maintenance mode options:

Ensure accessibility Full data migration No data migration

VMware Virtual SAN Interoperability Technologies and Products

Technology Interoperability

Virtual SAN is fully integrated with many of VMwares storage and vSphere availability enterprise features.

79

Supported Not applicable Future

Virtual Machine Snapshots Storage IO Control (SIOC) 62 TB VMDKs

vSphere HA Storage DRS vCOPS

vSphere DRS Distributed Power Management (DPM)

vMotion

Horizon View Virtual SAN and Horizon View:

Handle peak performance such as boot, login, read/write storms

Seamless granular scaling without huge upfront investments

Support high VDI density

Support high end virtual desktop GPU requirements

Virtual SAN is compatible with the following Horizon View versions:

Horizon View 5.3 (SPBM manually implemented)

Policies maintained across operations such as refresh/refresh no need to re-associate


Hard disks

Hard disks SSD SSD Hard disks

SSD

Full Clone Policies FTT = 1 for persistent

FTT = 0 for non-persistent

Provisioning 100% reserved

Linked Clone Policies OS Disk: FTT = 1 for dedicated

pools,

OS Disk: FTT = 0 for floating pool

Replica Disk: FTT = 1

Replica Disk: Read Cache Reservation 10%

Provisioning: Thin

vSphere Replication and Site Recovery Manager

Virtual SAN is compatible with: vSphere Replication 5.5 (vSphere Web Client) SPBM configured as part of replication vCenter Site Recovery Manager 5.5 (vSphere C#) SRM configuration based on VR replication

vSphere Replication & vCenter Site Recovery Manager Asynchronous replication 15 minute RPO VM-Centric based protection Provide automated DR operation & orchestration Automated failover execution of user defined plans Automated failback reverser original recovery plan Planned migration ensure zero data loss Point-in-Time Recovery multiple recovery points Non-disruptive test automate test on isolated network

vCenter Server VR/SRM

vSphere

VMFS

vCenter Server VR/SRM

production site recovery site

replication

Hard disks

SSD


Hard disks

SSD Hard disks

SSD

vSphere Data Protection Virtual SAN and vSphere Data Protection

Radically simple to deploy and manage Integrated User Interface vSphere Web Client Highly available storage solution Increase operation efficiency

vSphere Data Protection Advanced 5.5 Source and target De-duplication capabilities

Bidirectional replication capabilities

Secure, easy, reliable, network-efficient replication

Application-consistent backup and recovery capabilities

Higher RTO and RPO 24 hours RTO, minutes hours RPO

Incorporated technologies vStorage API for Data protection

Change Block Tracking (CBT)

Avamar variable-length segment algorithm

vCenter Server

Hard disks

SSD


Hard disks

SSD Hard disks

SSD

vSphere

VMFS

vCenter Server

vCloud Automation Center

vCloud Automation Center provides Virtual SAN: Centralized provisioning, governance, infrastructure

management capabilities Simple and self-service consumption capabilities Entitlement compliance monitoring, and enforcement Leverage existing business processes and tools Delegation control of resources

Custom use of VM Storage Policies: Virtual SAN default policy Blueprints VM templates Via vCenter Orchestrator with custom workflow Via vCloud Automation Center designer modifying

provisioning workflow


Hard disks

Hard disks SSD SSD Hard disks

SSD

OpenStack

Virtual SAN and OpenStack Framework Cloud Ready App to Hypervisor

Converged solution Leverage the use of Flash Optimized

storage in OpenStack Resiliency for legacy and Cloud Ready

applications vSphere Web Plug-in for OpenStack UI

Virtual SAN interoperates with OpenStack Framework. vSphere Driver vSphere Datastore

Swift

object store

Glance

image store

HorizonDashboard

OpenStack FrameworkKeyStone

identity service

NSX

driver

Neutron

networking

Nova

compute node

vsphere datastore

driver

Cinder

volume service

vsphere

driver


Hard disks Hard disks SSD SSD Hard disks SSD

VMware Virtual SAN Design & Sizing Guidelines Exercise

Virtual SAN Datastore Distributed datastore capacity determined by aggregating the disk

groups found across multiple hosts that are members of a vSphere cluster and the size of the magnetic disks.

Only the usable capacity of the magnetic disks count towards the total capacity of the Virtual SAN datastore.

The capacity of the flash based devices is specifically dedicated to Virtual SAN's caching layer.



disk group


vsanDatastore

HDD HDD HDD HDD HDD

Objects

Individual storage block device that is compatible with SCSI semantics. Each object that resides on the Virtual SAN datastore is comprised of

multiple components.

Objects are assigned storage performance and availability services requirements through VM Storage Profiles.

Object Types Definitions VM Home Location where all virtual machines configuration files reside (.vmx, log files, etc.)

Swap Unique storage object only created when virtual machines are powered on.

VMDK Virtual machine disk file

Snapshots Unique storage object created for virtual machines

Components Objects are comprised of components that are distributed across hosts

in vSphere cluster. Virtual SAN 5.5 currently supports a maximum of 3000 components per

host.

Objects greater than of 255 gigabytes in capacity are automatically divided into multiple components.

Each component consumes 2 megabytes of disk capacity for metadata.

Witness Witness components are part of every storage object. Only contain object metadata. Serve as tiebreakers when availability decisions are made in the Virtual

SAN cluster in order to avoid split-brain behavior.

Each Virtual SAN witness component also consumes 2 megabytes of capacity.

Virtual SAN Datastore Sizing Considerations

It is important to understand the impact of availability and performance storage capabilities on the consumption of storage capacity. Number of Failures to Tolerate Number of Disk Stripes per Object Flash Read Cache Reservation Object Space Reservation

Disk Groups A single flash based device (SAS/SATA/PCIe SSD) and one or more

magnetic disks (SAS/SATA HDD). Disk Groups make up the distributed flash tier and storage capacity of

the Virtual SAN Datastore.

Formatted with a modified on-disk file system (VMFS-L) and are then mounted onto the Object Store File System datastore as a single datastore

VMFS-L on-disk file system formatting consumes a total of 750 megabytes of capacity per disk.

Artifacts Minimums Maximums Disk Groups 1 Per Host 5 per host

Flash Devices (SAS/SAS/PCIe SSD) 1 Per Disk Group 1 Per Disk Group

Magnetic Disk Devices 1 HDD Per Disk Group 7 HDD Per Disk Group

Disk Formatting Overhead 750 MB Per HDD 750 MB Per HDD

Number of Failures to Tolerate

Largest impact on the consumption of storage capacity in Virtual SAN. Based on the availability requirements of a virtual machine, the setting

defined in a VM Storage Policy can lead to the consumption of up to four times the virtual machine or individual disks capacity

2 full copies of data + 1 witness

Number of Disk Stripes Per Object

If the Number of Disk Stripes per Object is increased beyond the default value of 1, then each stripe will count as a separate component.

This has an impact on the of total number of components supported per host.

Disk Group Design

One Flash Device Per Disk Group Multiple flash based devices, multiple disk groups will be created to

leverage the additional flash

Higher the ratio of flash based device capacity to magnetic disks capacity, the greater the size of the cache layer.

Define and reduce the storage failure domains.

Failure domain



disk group


HDD HDD HDD HDD HDD

Flash Capacity Sizing

The general recommendation for sizing Virtual SAN's flash capacity is to have 10% of the anticipated consumed storage capacity before the Number of Failures To Tolerate is considered.

Total flash capacity percentage should be based on use case, capacity and performance requirements.

10% is a general recommendation, could be too much or it may not be enough.

Measurement Requirements Values Projected VM space usage 20GB

Projected number of VMs 1000

Total projected space consumption per VM 20GB x 1000 = 20,000 GB = 20 TB

Target flash capacity percentage 10%

Total flash capacity required 20TB x .10 = 2 TB

Sizing Exercise Formulas

96

Constraints VSAN components and VMFS metadata overhead (VSANmetaDataOverhead):

1GB per disk Variables Number of Hosts Per cluster (Hst) = 8 Number of Disk Groups (DskGrp) = 5 Number of Disks Per Disk Group (DskPerDskGrp) = 7 Size of Disks (SzHDD) = 4000 GB Number of Failures To Tolerate (ftt) = 1 Number of Virtual Machines (VMs) = 800 Number of Disks per Virtual Machine (NumOfVMDK) = 1 Memory Per Virtual Machine (vmSwp) = 10 GB

Cluster RAW Capacity Formula: Hst x NumDskGrpPerHst x NumDskPerDskGrp x SzHDD = y Example: 8 x 5 x 7 x 4000 GB =1,120,000 GB =1,120 TB


97

VMFS Meta Data Formula: VMFSMetadata x NumDskGrpPerHst x NumDskPerDskGrp = y Example: 750 MB x 5 x 7 = 26,250 MB = 26.2 GB VMFS Metadata

Objects Formula: VMs x [VMnamespace + vmSwap + NumOfVMDK] = y Example: 800 x [1 + 1 + 1] = 2400 Objects Note: Snaps, Clones and >1 Disk Stripes would add more objects Components Formula: Object x [ftt x 2 + 1] = y Example: 2400 x (1 x 2 + 1) = 7200 Components = 900 average components per host (max is 3000 per host) Components Metadata Formula: NumComponents x compMetadata = y Example: 7200 Components x 2 MB = 14.4 GB Component Metadata


98

VSAN Meta Data Formula: compMetadata + VMFSMetadata = y Example: 14.4 GB + 26.2 GB = 40.6 GB VSAN Metadata Simplified Formula: NumDskGrpPerHst x NumDskPerDskGrp x NumHosts x 1 GB = y Simplified Example: 5 x 7 x 8 x 1 GB = 280 GB

Accounts for factors such as snapshots, additional stripes, etc. Swap Utilization

Formula: (VMs x vmSwp x 2) Example: Swap Space = (800 x 10GB x 2) = 16000 GB

Available Capacity = Raw Capacity Swap Capacity = 1120000 16000 = 1104000 = 1,104 TB Disk Capacity

Usable Capacity Formula: (DiskCapacity VSAN Meta Data) / (ftt + 1) Example: (1104000 GB - 280 GB) / 2 = 1103720 GB / 2 = 551,860 GB Usable Capacity Best practice is to allocate no more than 80% to virtual disks

Memory and CPU

Memory requirements for Virtual SAN are defined based on the number of disks groups and disk that are managed by hypervisor.

As long as vSphere hosts have greater memory configurations than 32 gigabytes of RAM, they will be able to support the maximum disk group and disks configuration supported in Virtual SAN.

Virtual SAN is designed to introduce no more than 10% of CPU overhead per hosts. Consider this fact in Virtual SAN implementations with high consolidation ratios and CPU intensive applications requirements.

Network Virtual SAN network activities can potentially saturate and overwhelm an

entire 1GbE network, particularly during rebuild and synchronization operations.

Separate the different traffic types (Management, vMotion, Virtual Machine, Virtual SAN) onto different VLANs and use shares as a Quality of Service mechanism to sustain the level of performance expected during possible contentions scenarios.

Virtual SAN requires for IP multicast to be enabled on the layer 2 physical network segment utilized for Virtual SAN communication

VMware Virtual SAN Monitoring & Troubleshooting

Network Status reports

Misconfiguration detected: Verify physical network Enable multicast

Disabling IGMP snooping Configure IGMP snooping for

selective traffic

Validate the virtual switch configuration VLAN VSAN Traffic service enabled NIC team failover policy

Failover Policy

NIC Teaming failover load balancing: policy with route based on port ID Active / Standby

103

Command Line Tools

VMKPING vmkping

Example 10.4.90.27 To validate network accessibility

ESXCLI esxcli vsan network list

104

Disk Claiming Operation

Automatic disk claiming operation fails to claim disks Is local: true disks are automatically claimed Is local: false disks are shared thus not automatically claimed but can be

manually marked local

105

Ruby vSphere Console

RVC VSAN vsan.disks_info Size, disk type, manufacturers, model, local/non-local

106

Disk Groups Creation Fails

Disk Groups Creation Fails VSAN license needs to be added to the cluster

Home > licenses > Cluster tab > Select cluster object > Assign License Key vSphere Web Client refresh time out

Log out and back in

Unable to delete Disk Group VSAN disk claiming operation set to automatic, change to manual

vsan.host_wipe_vsan_disks --force wipe disks used by VSAN

CONFIDENTIAL 107

Observing performance Monitor performance: Ruby vSphere Console & VSAN Observer

In-depth monitoring of VSANs physical disk layer performance, cache hit rates, latencies, etc.

VSAN Observer

Starting the VSAN Observer Performance stats

1

2

VSAN Observer Monitoring Flash Devices

Monitor read cache hit rate Flash based devices evictions to magnetic disks

110

VSAN Observer

Monitor disk groups aggregate and disk layers

Virtual SAN Logs

Virtual SAN related logs. Individually maintained per hosts

112


Disk Capacity used and reserved capacity

113

Monitoring VSAN Component Limits


Virtual SAN what if failure analysis Simulate host failure impact to cluster


VSAN Observer recommendations Deploy a VCVA appliance to use for the Observer Run the observer session on the newly deployed or remote VCVA appliance Increase the data gathering time beyond the default (2 hours) if necessary.

116

!!!

Jason [email protected]!

@TheJason Nash"

THANK YOU

Graphics by Duncan Epping & Rawlinson

Documents

Varrow Madness 2014 - VSAN Deep Drive