© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

김일호, Solutions Architect03-May-2016

Tips about performance on AWSAmazed by AWS

EC2 performance dive deep

InstancesAPIs

Networking

EC2EC2

Purchase Options

Amazon Elastic Compute Cloud is Big

Host ServerHypervisor

Guest 1 Guest 2 Guest n

Amazon EC2 Instances

2006 2008 2010 2012 2014 2016

m1.small

m1.largem1.xlarge

c1.mediumc1.xlarge

m2.xlarge

m2.4xlargem2.2xlarge

cc1.4xlarge

t1.micro

cg1.4xlarge

cc2.8xlarge

m1.medium

hi1.4xlarge

m3.xlargem3.2xlarge

hs1.8xlarge

cr1.8xlarge

c3.largec3.xlarge

c3.2xlargec3.4xlargec3.8xlargeg2.2xlarge

i2.xlargei2.2xlargei2.4xlargei2.4xlarge

m3.mediumm3.large

r3.larger3.xlarger3.2xlarger3.4xlarger3.8xlarge

t2.microt2.smallt2.med

c4.largec4.xlargec4.2xlargec4.4xlargec4.8xlarge

d2.xlarged2.2xlarged2.4xlarged2.8xlargeg2.8xlarge

t2.largem4.large

m4.xlargem4.2xlargem4.4xlarge

m4.10xlarge

Amazon EC2 Instances History

What to Expect from the Session

• Defining system performance and how it is characterized for different workloads

• How Amazon EC2 instances deliver performance while providing flexibility and agility

• How to make the most of your EC2 instance experience through the lens of several instance types

Defining Performance

• Servers are hired to do jobs• Performance is measured differently depending on the job

Hiring a Server

?

• What performance means depend on your perspective:– Response time– Throughput– Consistency

Defining Performance: Perspective Matters

Application

System libraries

System calls

Kernel

Devices

Workload

Simple Performance Model for Single Thread

• Using CPU: executing (in user mode)

• Not using CPU: waiting for turn on CPU, waiting for disk or network I/O, thread locks, memory paging, or for more work.

Performance Factors

Resource Performance factors Key indicatorsCPU Sockets, number of cores, clock

frequency, bursting capabilityCPU utilization, run queue length

Memory Memory capacity Free memory, anonymous paging, thread swapping

Network interface

Max bandwidth, packet rate Receive throughput, transmit throughput over max bandwidth

Disks Input / output operations per second, throughput

Wait queue length, device utilization, device errors

Resource Utilization

• For given performance, how efficiently are resources being used• Something at 100% utilization can’t accept any more work• Low utilization can indicate more resource is being purchased

than needed

Example: Web Application• MediaWiki installed on Apache with 140 pages of content• Load increased in intervals over time

Example: Web Application• Memory stats

Example: Web Application• Disk stats

Example: Web Application• Network stats

Example: Web Application• CPU stats

• Picking an instance is tantamount to resource performance tuning• Give back instances as easily as you can acquire new ones• Find an ideal instance type and workload combination

Instance Selection = Performance Tuning

Delivering Compute Performance with Amazon EC2 Instances

CPU Instructions and Protection Levels

• CPU has at least two protection levels: ring0 and ring1• Privileged instructions can’t be executed in user mode to protect

system. Applications leverage system calls to the kernel.

Kernel

Application

Example: Web application system calls

X86 CPU Virtualization: Prior to Intel VT-x

• Binary translation for privileged instructions• Para-virtualization (PV)

• PV requires going through the VMM, adding latency• Applications that are system call bound are most affected

VMM

Application

Kernel

PV

X86 CPU Virtualization: After Intel VT-x• Hardware assisted virtualization (HVM)• PV-HVM uses PV drivers opportunistically for operations that are

slow emulated: • e.g. network and block I/O

KernelApplication

VMM

PV-HVM

Tip: Use PV-HVM AMIs with EBS

Time Keeping Explained

• Time keeping in an instance is deceptively hard• gettimeofday(), clock_gettime(), QueryPerformanceCounter()• The TSC

• CPU counter, accessible from userspace• Requires calibration, vDSO• Invariant on Sandy Bridge+ processors

• Xen pvclock; does not support vDSO• On current generation instances, use TSC as clocksource

Tip: Use TSC as clocksource

tsc

source=tsc

CPU Performance and Scheduling

• Hypervisor ensures every guest receives CPU time• Fixed allocation

• Uncapped vs. capped• Variable allocation

• Different schedulers can be used depending on the goal• Fairness• Response time / deadline• Shares

Review: C4 Instances

Custom Intel E5-2666 v3 at 2.9 GHzP-state and C-state controls

Model vCPU Memory (GiB) EBS (Mbps)c4.large 2 3.75 500c4.xlarge 4 7.5 750c4.2xlarge 8 15 1,000c4.4xlarge 16 30 2,000c4.8xlarge 36 60 4,000

What’s new in C4: P-state and C-state control

• By entering deeper idle states, non-idle cores can achieve up to 300MHz higher clock frequencies

• But… deeper idle states require more time to exit, may not be appropriate for latency sensitive workloads

Tip: P-state control for AVX2

• If an application makes heavy use of AVX2 on all cores, the processor may attempt to draw more power than it should

• Processor will transparently reduce frequency• Frequent changes of CPU frequency can slow an application

Review: T2 Instances

• Lowest cost EC2 Instance at $0.013 per hour• Burstable performance• Fixed allocation enforced with CPU Credits

Model vCPU CPU Credits / Hour

Memory (GiB)

Storage

t2.micro 1 6 1 EBS Onlyt2.small 1 12 2 EBS Onlyt2.medium 2 24 4 EBS Onlyt2.large 2 36 8 EBS Only

How Credits Work

• A CPU Credit provides the performance of a full CPU core for one minute

• An instance earns CPU credits at a steady rate

• An instance consumes credits when active

• Credits expire (leak) after 24 hours

Baseline Rate

Credit Balance

BurstRate

Tip: Monitor CPU credit balance

Monitoring CPU Performance in Guest

• Indicators that work is being done• User time• System time (kernel mode)• Wait I/O, threads blocked on disk I/O

• Else, Idle• What happens if OS is scheduled off the CPU?

Tip: How to interpret Steal Time

• Fixed CPU allocations of CPU can be offered through CPU caps

• Steal time happens when CPU cap is enforced• Leverage CloudWatch metrics

Delivering I/O Performance with Amazon EC2 Instances

I/O and Devices Virtualization

• Scheduling I/O requests between virtual devices and shared physical hardware

• Split driver model • Intel VT-d

• Direct pass through and IOMMU for dedicated devices• Enhanced Networking

Hardware

Split Driver ModelDriver Domain Guest Domain Guest Domain

VMM

Frontend driver

Frontend driver

Backend driver

DeviceDriver

Physical CPU

Physical Memory

Network Device

Virtual CPU Virtual Memory

CPU Scheduling

Split Driver Model

• Each virtual device has two main components• Communication ring buffer • An event channel signaling activity in the ring buffer

• Data is transferred through shared pages• Shared pages requires inter domain permissions, or granting

Review: I2 Instances

16 vCPU: 3.2 TB SSD; 32 vCPU: 6.4 TB SSD365K random read IOPS for 32 vCPU instance

Model vCPU Memory (GiB)

Storage Read IOPS Write IOPS

i2.xlarge 4 30.5 1 x 800 SSD 35,000 35,000i2.2xlarge 8 61 2 x 800 SSD 75,000 75,000i2.4xlarge 16 122 4 x 800 SSD 175,000 155,000i2.8xlarge 32 244 8 x 800 SSD 365,000 315,000

Granting in pre-3.8.0 Kernels

• Requires “grant mapping” prior to 3.8.0• Grant mappings are expensive operations due to TLB flushes

read(fd, buffer,…)

Granting in 3.8.0+ Kernels, Persistent and Indirect

• Grant mappings are setup in a pool once• Data is copied in and out of the grant pool

read(fd, buffer…)Copy to and from grant pool

Tip: Use 3.8+ kernel

• Amazon Linux 13.09 or later• Ubuntu 14.04 or later• RHEL7 or later• Etc.

Event Handling

• Guest vCPUs are interrupted to process events. • Pre-2.6.36 kernels: notifications went to a single virtual

hardware interrupt • Post-2.6.36 kernels: allow instance to tell hypervisor to deliver

notification to a specific vCPU for balancing• Check "dmesg" for the following text: "Xen HVM callback vector for

event delivery is enabled“• Also, check version of irqbalance is 1.0.7 or higher

Hardware

Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain

VMM

Frontend driver

Frontend driver

Backend driver

DeviceDriver

Physical CPU

Physical Memory

Network Device

Virtual CPU Virtual Memory

CPU Scheduling

Sockets

Application

Hardware

Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain

VMM

Frontend driver

Frontend driver

Backend driver

DeviceDriver

Physical CPU

Physical Memory

Network Device

Virtual CPU Virtual Memory

CPU Scheduling

Sockets

Application

Hardware

Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain

VMM

Frontend driver

Frontend driver

Backend driver

DeviceDriver

Physical CPU

Physical Memory

Network Device

Virtual CPU Virtual Memory

CPU Scheduling

Sockets

Application

Hardware

Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain

VMM

Frontend driver

Frontend driver

Backend driver

DeviceDriver

Physical CPU

Physical Memory

Network Device

Virtual CPU Virtual Memory

CPU Scheduling

Sockets

Application

Hardware

Split Driver Model: NetworkingDriver Domain Guest Domain Guest Domain

VMM

Frontend driver

Frontend driver

Backend driver

DeviceDriver

Physical CPU

Physical Memory

Network Device

Virtual CPU Virtual Memory

CPU Scheduling

Sockets

Application

Device Pass Through: Enhanced Networking

• SR-IOV eliminates need for driver domain• Physical network device exposes virtual function to

instance• Requires a specialized driver, which means:

• Your instance OS needs to know about it • EC2 needs to be told your instance can use it

Hardware

After Enhanced NetworkingDriver Domain Guest Domain Guest Domain

VMM

Frontend driver

NIC Driver

Backend driver

DeviceDriver

Physical CPU

Physical Memory

SR-IOV Network Device

Virtual CPU Virtual Memory

CPU Scheduling

Sockets

Application

Hardware

After Enhanced NetworkingDriver Domain Guest Domain Guest Domain

VMM

Frontend driver

NIC Driver

Backend driver

DeviceDriver

Physical CPU

Physical Memory

SR-IOV Network Device

Virtual CPU Virtual Memory

CPU Scheduling

Sockets

Application

Hardware

After Enhanced NetworkingDriver Domain Guest Domain Guest Domain

VMM

Frontend driver

NIC Driver

Backend driver

DeviceDriver

Physical CPU

Physical Memory

SR-IOV Network Device

Virtual CPU Virtual Memory

CPU Scheduling

Sockets

Application

Tip: Use Enhanced Networking

• Highest packets-per-second• Lowest variance in latency• Instance OS must support it• Look for SR-IOV property of instance or image

Inter-instance latency

How to build enhanced network driver on Linux

Let’s start to create AMI for Enhanced networking enabled

CentOS 6.5 with c4.8xlarge

Let’s start with the AMI officially provided CentOS, should be clean. CentOS is provided in awsmarketplace.

Only missing c4.8xlargeL

Anyway, let’s go to AMI search

Find and Select AMI

CentOS 6 x86_64 (2014_09_29) EBS HVM-74e73035-3435-48d6-88e0-89cc02ad83ee-ami-a8a117c0.2

ami-c2a818aa (IAD)

This is the CentOS AMI at awsmarketplace CentOS. Nice ?!?!

Check the requirements ofEnabling Enhanced Networking on Linux• C3, C4, D2, I2, M4 and R3• HVM AMI with Linux kernel above V.2.6.32 • Launch the instance in VPC

• A network driver to support enhanced networking on Linux.

Check kernel version and network driver

[root@ip-192-168-1-171 ~]# cat /etc/redhat-releaseCentOS release 6.5 (Final)

[root@ip-192-168-1-171 ~]# uname -na

Linux ip-192-168-1-171 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@ip-192-168-1-171 ~]# modinfo ixgbevfmodinfo ixgbevf

filename: /lib/modules/2.6.32-431.29.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.koversion: 2.7.12-k

license: GPL

description: Intel(R) 82599 Virtual Function Driver

author: Intel Corporation, <linux.nics@intel.com>

srcversion: E75203124BB105EC871944Falias: pci:v00008086d00001515sv*sd*bc*sc*i*

alias: pci:v00008086d000010EDsv*sd*bc*sc*i*depends:

vermagic: 2.6.32-431.29.2.el6.x86_64 SMP mod_unload modversionsparm: debug:Debug level (0=none,...,16=all) (int)

Launch c4.large instance and login

Let’s update all including kernel

root@ip-192-168-1-171 ~]# yum update -yLoaded plugins: fastestmirror, prestoLoading mirror speeds from cached hostfile* base: mirrors.mit.edu* extras: linux.cc.lehigh.edu* updates: mirrors.lga7.us.voxel.net

Setting up Update ProcessResolving Dependencies--> Running transaction check---> Package audit.x86_64 0:2.2-4.el6_5 will be updated---> Package audit.x86_64 0:2.3.7-5.el6 will be an update---> Package audit-libs.x86_64 0:2.2-4.el6_5 will be updated---> Package audit-libs.x86_64 0:2.3.7-5.el6 will be an update---> Package authconfig.x86_64 0:6.1.12-13.el6 will be updated---> Package authconfig.x86_64 0:6.1.12-19.el6 will be an update---> Package bash.x86_64 0:4.1.2-15.el6_5.2 will be updated---> Package bash.x86_64 0:4.1.2-29.el6 will be an update---> Package binutils.x86_64 0:2.20.51.0.2-5.36.el6 will be updated---> Package binutils.x86_64 0:2.20.51.0.2-5.42.el6 will be an update---> Package ca-certificates.noarch 0:2014.1.98-65.0.el6_5 will be updated---> Package ca-certificates.noarch 0:2014.1.98-65.1.el6 will be an update---> Package centos-release.x86_64 0:6-5.el6.centos.11.2 will be updated---> Package centos-release.x86_64 0:6-6.el6.centos.12.2 will be an update---> Package coreutils.x86_64 0:8.4-31.el6_5.2 will be updated---> Package coreutils.x86_64 0:8.4-37.el6 will be an update---> Package coreutils-libs.x86_64 0:8.4-31.el6_5.2 will be updated………………………

Reboot and check the updated

[root@ip-192-168-1-171 ~]# cat /etc/redhat-release

CentOS release 6.6 (Final)

[root@ip-192-168-1-171 ~]# uname -na

Linux ip-192-168-1-171 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@ip-192-168-1-171 ~]# modinfo ixgbevf

filename: /lib/modules/2.6.32-504.12.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko

version: 2.12.1-k

license: GPL

description: Intel(R) 82599 Virtual Function Driver

author: Intel Corporation, <linux.nics@intel.com>

srcversion: 8797AC845BB302315230490

alias: pci:v00008086d00001515sv*sd*bc*sc*i*

alias: pci:v00008086d000010EDsv*sd*bc*sc*i*

depends:

vermagic: 2.6.32-504.12.2.el6.x86_64 SMP mod_unload modversions

parm: debug:Debug level (0=none,...,16=all) (int)

Upgraded network driver installedversion: 2.7.12-k -> 2.12.1-k

Enable SR-IOV

Install AWS CLI or EC2 CLI toolsNot supported at AWS Console yet L

1. Stop the instance2. Enable SR-IOV of the instance with CLI3. Check the status4. Start the instance

a82066443ffe:~ ilho$ aws ec2 modify-instance-attribute --instance-id i-681280bf \--sriov-net-support simple

a82066443ffe:~ ilho$ aws ec2 describe-instance-attribute --instance-id i-681280bf \--attribute sriovNetSupport

{

"InstanceId": "i-681280bf",

"SriovNetSupport": {

"Value": "simple"

}

}

A problem with more than 32 vCPUs on Linux

CentOS 6.x does not support more than 32 vCPUs in kernel.It can not boot when you launch c4.8xlarge(36 vCPUs) L

• d2.8xlarge and m4.10xlarge? -> L

A solution is to add an option to kernel boot parameter

Add maxcpus option to kernel boot parameter

$ vi /boot/grub/menu.lstAdd maxcpus=32

[root@ip-10-10-10-242 ~]# cat /boot/grub/menu.lst

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You do not have a /boot partition. This means that

# all kernel and initrd paths are relative to /, eg.

# root (hd0,0)

# kernel /boot/vmlinuz-version ro root=/dev/vda1

# initrd /boot/initrd-[generic-]version.img

#boot=/dev/vda

default=0

timeout=1

serial --unit=0 --speed=115200

terminal --timeout=1 serial console

title CentOS (2.6.32-431.29.2.el6.x86_64)

root (hd0,0)

kernel /boot/vmlinuz-2.6.32-431.29.2.el6.x86_64 maxcpus=32 ro root=UUID=dcb1645e-05a6-4311-8bce-a9c12bec5801 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD console=ttyS0,115200 crashkernel=auto SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM

initrd /boot/initramfs-2.6.32-431.29.2.el6.x86_64.img

Now let’s launch c4.8xlarge instance

I think it’s ready to go.RebootLaunch a c4.8xlarge instance

It can not launch.

Why?

AWS Marketplace AMI does not support to launch an instance type not in the list.

Change the base AMI

Find CentOS 6.5 community version AMI : CentOS-6.5-base-20150305 (ami-0e80db66)

Repeat three steps1. $ sudo yum update –y2. Enable SR-IOV3. Add maxcpus=32

Network driver version should be checked

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.htmlTo enable enhanced networking on your instance, you must ensure that its kernel has the ixgbevf module installed and that you set the sriovNetSupport attribute for the instance. For the best performance, we recommend that the ixgbevf module is version 2.14.2 or higher.

Build and Install the network driver #2

[ec2-user@ip-192-168-1-50 src]$ make;sudo make installmake -C /lib/modules/2.6.32-504.12.2.el6.x86_64/build SUBDIRS=/home/ec2-user/ixgbevf-2.16.1/srcmodulesmake[1]: Entering directory ̀ /usr/src/kernels/2.6.32-504.12.2.el6.x86_64'

CC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf_main.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf_param.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf_ethtool.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/kcompat.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbe_vf.oCC [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbe_mbx.oLD [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.oBuilding modules, stage 2.MODPOST 1 modulesCC /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.mod.oLD [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.ko.unsignedNO SIGN [M] /home/ec2-user/ixgbevf-2.16.1/src/ixgbevf.ko

make[1]: Leaving directory /̀usr/src/kernels/2.6.32-504.12.2.el6.x86_64'make -C /lib/modules/2.6.32-504.12.2.el6.x86_64/build SUBDIRS=/home/ec2-user/ixgbevf-2.16.1/srcmodulesmake[1]: Entering directory ̀ /usr/src/kernels/2.6.32-504.12.2.el6.x86_64'

Building modules, stage 2.MODPOST 1 modules

make[1]: Leaving directory /̀usr/src/kernels/2.6.32-504.12.2.el6.x86_64'gzip -c ../ixgbevf.7 > ixgbevf.7.gz# remove all old versions of the driverfind /lib/modules/2.6.32-504.12.2.el6.x86_64 -name ixgbevf.ko -exec rm -f {} \; || truefind /lib/modules/2.6.32-504.12.2.el6.x86_64 -name ixgbevf.ko.gz -exec rm -f {} \; || trueinstall -D -m 644 ixgbevf.ko /lib/modules/2.6.32-504.12.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko/sbin/depmod -a 2.6.32-504.12.2.el6.x86_64 || trueinstall -D -m 644 ixgbevf.7.gz /usr/share/man/man7/ixgbevf.7.gzman -c -P'cat > /dev/null' ixgbevf || true

http://sourceforge.net/projects/e1000/files/ixgbevf%20stable/

Check the new driver installed

[ec2-user@ip-192-168-1-50 src]$ modinfo ixgbevf

filename: /lib/modules/2.6.32-504.12.2.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko

version: 2.16.1

license: GPL

description: Intel(R) 10 Gigabit Virtual Function Network Driver

author: Intel Corporation, <linux.nics@intel.com>

srcversion: 3B690FE23A02C25EF74012F

alias: pci:v00008086d00001515sv*sd*bc*sc*i*

alias: pci:v00008086d000010EDsv*sd*bc*sc*i*

depends:

vermagic: 2.6.32-504.12.2.el6.x86_64 SMP mod_unload modversions

parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (956-488281, 0=off, 1=dynamic), default 1 (array of int)

reboot

[ec2-user@ip-192-168-1-50 ~]$ ethtool -i eth0

driver: ixgbevf

version: 2.16.1

firmware-version: N/A

bus-info: 0000:00:03.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: yes

supports-priv-flags: no

Share AMI to the users

Sharing the customized AMI with your team ~

Let’s get some sleep… J

Reply back with 1/2 (OS unreachable)

Root cause found

the 70-persistent-net.rules file /etc/udev/rules.d/ having an entry with the MAC address of the original instance the AMI was taken from. When the image is taken, the MAC from the original instance persists, which tells the OS after the new instance is deployed that the now-nonexistent MAC of the original instance should be eth0. Since the OS cannot find the device with the original MAC address, eth0 fails to be identified and isn't brought up.

A solution to avoid this caveat

Before creating AMI, You must remove/etc/udev/rules.d/70-persistent-net.rules file

Create AMIShare AMI Now launch c4 instance successfully with the latest driver.

A small contribution

The note was added in public document “enhanced networking” section.

Lesson learned

Just use CentOS 7 or Amazon Linux. J

EBS designing for performance

Amazon EBS overview

EBS =

What is EBS?

• Network block storage as a service

• EBS volumes attach to any Amazon EC2 instance in the

same Availability Zone

• Designed for five nines of availability

• 2 million volumes created every day

EBS volume types

Magnetic General purpose (SSD) Provisioned IOPS (SSD)

EBS volume types

IOPS: Typically 100, best effortThroughput: 40-90 MB/sLatency: Read 10-40ms, Write 2-10msBest for infrequently accessed data

Magnetic

EBS volume types

IOPS Baseline: 100-10,000 (3 / GiB)IOPS Burst: 30 minutes @ 3,000Throughput: Up to 160 MB/sLatency: Single-digit msPerformance consistency: 99%Most workloadsGeneral purpose (SSD)

EBS volume types

IOPS: 100-20,000 (customer provisioned)Throughput: Up to 320 MB/sLatency: Single-digit msPerformance consistency: 99.9%Mission Critical workloadsProvisioned IOPS (SSD)

Performance

Queuing theory

Little’s law is the foundation for performance tuning theory• Mathematically proven by John Little in 1961

𝑾 =𝑳𝑨

W = Wait time = average wait time per requestL = Queue length = average number of requests waitingA = Arrival rate = the rate of requests arriving

EBS performance is related to this law

Performance optimization is measured by:

IOPS: Read/write I/O rate (IOPS)

Latency: Time between I/O submission and completion (ms)

Throughput: Read/write transfer rate (MB/s); throughput = IOPS X I/O size

Key components to performance

EC2 instance

I/O

EBS

Network link

A day in the life of an I/O

A day in the life: I/O

All I/O must pass through I/O domainRequires “grant mapping” prior to 3.8.0Grant mappings are expensive operations due to TLB flushes

EBS

Grant mappingread(fd, buffer,

BLOCK_SIZE)

I/O domainInstance

A day in the life: I/O (continued)

Responses

Requests

Instance I/O domain

READ8KB @ 1234

Request queue is a single memory pageEach I/O request has 11 grant references (4KiB/reference)Maximum data in queue = 1408 KiB

3.8.0+ Kernels – Persistent grants

Grant mappings are setup in a pool onceData is copied in and out of the grant poolCopying is significantly faster than remapping

EBS

Grant poolread(fd, buffer,

BLOCK_SIZE)

I/O domainInstance

3.8.0+ Kernels – Indirect grants

Responses

Requests

Instance

READ8KB @ 1234

I/O domain

Each I/O request has grant references that contain grant referencesMaximum data in queue = 4096 KiB (default)

Instance I/O: Before 3.8.0

0 1 2 3 4 5 30 31

128KiB

44KiB 44KiB 40KiB

Instance I/O: Linux 3.8.0+

0 1 2 3 4 5 30 31

128KiB

Tip: Use 3.8+ kernel

Amazon Linux 2013.09 or laterUbuntu 14.04 or laterRHEL7 or laterEtc.

Queue depth

An I/O operation

EBS

After it’s gone, it’s gone

EC2

Queue depth is the pending I/O for a volume

Workload/ software

Typical block size

Random/Seq?

Max EBS @ 500 MB/s instances

Max EBS @ 1 GB/s instances

Max EBS @ 10 GB/s instances

Oracle DB Configurable:2 KB–16 KBDefault 8 KB

random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS

Microsoft SQL Server

8 KB w/ 64 KB extents

random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS

MySQL 16 KB random ~4,000 IOPS ~7,800 IOPS ~48,000 IOPS

PostgreSQL 8 KB random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS

MongoDB 4 KB sequential ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS

Apache Cassandra

4 KB random ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS

GlusterFS 128 KB sequential ~500 IOPS ~1,000 IOPS ~6,000 IOPS

Cheat sheet: Sample storage workloads on AWS

Example workload

Transaction (OLTP)Examples: eCommerce website, metadata storageBenchmark: MySQL + sysbench

Tip: Workload

Where possible, use real production workloads for performance testing

Baseline configuration

Availability Zone: US West (Oregon)Instance type: m2.4xlarge

vCPU: 8Memory: 68.4GiBEBS-optimized

Data volume: 500GiB EBS magneticOS: Amazon Linux 2015.03.1

Optimization: Increase parallelism

MySQL threads

Tran

sact

ions

(n)

Baseline

2 n

Tip: Parallelism

Increase parallelism of your system

Key components to performance

EC2 instance

I/O

EBS

Network link

m2.4xlargeCPU: Intel Xeon vCPU: 8Memory: 68.4 GiBPrice: $0.98/hour*

Instance selection

r3.2xlargeCPU: Intel Xeon E5-2670 v2vCPU: 8Memory: 61 GiBEnhanced networkingPrice: $0.70/hour*

* All pricing from US West (Oregon)

EBS optimized instances

• Most instance families support the EBS-optimized flag

• EBS-optimized instances now support up to 4 Gb/s

• Drive 32,000 16K IOPS or 500 MB/s

• Available by default on newer instance types

• EC2 *.8xlarge instances support 10 Gb/s network

• Max IOPS per node supported is ~48,000 IOPS @ 16K I/O

Tip: Use EBS-optimized instances

Use EBS-optimized instances for consistent EBS performance

Updated configuration: Instance type

Availability Zone: US West (Oregon)Instance type: r3.2xlarge

vCPU: 8Memory: 61 GiBEBS-optimized

EBS volume: 500GiB magneticOS: Amazon Linux 2015.03.1

25%

Optimization: Current generation instances

MySQL threads

Tran

sact

ions

(n)

Baseline

r3.2xlarge

2 n

Tip: Instance selection

Use the right instance family for your workload

Use current generation instances

Key components to performance

EC2 instance

I/O

EBS

Network link

Volume selection

EBS magneticLatency:

Read: 10-40msWrite: 2-10ms

SSD backedLatency:

Read/Write: Single-digit ms

File systems

Use a modern, journaled filesystemext4, xfs, etc.

Ensure partitions are aligned on 4KiB boundaries

Pre-warming

Volume initialization

Newly created volumes• Just attach, mount, and go!• Pre-warming is no longer recommended

Volumes restored from snapshots• You can use your volume right away• Accelerate data loading by reading

Updated configuration: EBS volumes

Availability Zone: US West (Oregon)Instance type: r3.2xlarge

vCPU: 8Memory: 61 GiBEBS-optimized

Boot volume: 8 GiB – EBS general purposeData volume: 500 GiB – EBS general purposeOS: Amazon Linux 2015.03.1

Optimization: Volume selectionTr

ansa

ctio

ns (n

) 19% 50%

MySQL threads

Baseline

r3.2xlarge

r3.2xlarge gp2

2 n

Tip: Volume selection

Use SSD backed volumes when performance matters

EBS IOPS vs. Throughput

20,000 IOPS PIOPS volume

20,000 IOPS

320 MB/s throughput

You can achieve 20,000 IOPS when driving smaller I/O operations

You can achieve up to 320 MB/s when driving larger I/O operations

EBS IOPS vs. Throughput

8,000 IOPS PIOPS volume

8,000 IOPS

320 MB/s throughput

8,000 x 8 KB = 64 MB/s8,000 x 16 KB = 128 MB/s8,000 x 32 KB = 256 MB/s

16,000 x 8 KB = 128 MB/s

8,000 x 64 KB=512 MB/s

5,000 x 64 KB = 320 MB/s

Striping

Increases performance, or capacity, or both

Don’t mix volume types

Typically RAID 0 or LVM stripe

Avoid RAID for redundancyEBS

EC2

Striping: Snapshots

Quiesce I/O1. Database: FLUSH and LOCK tables2. Filesystem: sync and fsfreeze3. EBS: snapshot all volumes

When snapshot API returns, it is safe to resume

EBS-optimized instance

Four key components: Balanced

EC2

A “boatload” of I/O

Right-sized EBS

Monitoring

Amazon CloudWatch

Important Amazon CloudWatch metrics: • IOPS and bandwidth

• Latency

• Queue depth

All EBS metrics are prefixed with “Volume”

CloudWatch: Instance bandwidth

m4.2xlargeInstance: 128MB/s

m4.4xlargeInstance: 256MB/s

m4.10xlargeVolume: 320MB/s

S3 Performance tips

Distributing Key Names

Don’t do this

<my_bucket>/2013_11_13-164533125.jpg<my_bucket>/2013_11_13-051033564.jpg

<my_bucket>/2013_11_13-061133789.jpg<my_bucket>/2013_11_13-051033458.jpg<my_bucket>/2013_11_12-063433125.jpg<my_bucket>/2013_11_12-021033564.jpg<my_bucket>/2013_11_12-065533789.jpg

<my_bucket>/2013_11_12-011033458.jpg<my_bucket>/2013_11_11-022333125.jpg<my_bucket>/2013_11_11-153433564.jpg<my_bucket>/2013_11_11-065233789.jpg<my_bucket>/2013_11_11-065633458.jpg

Distributing Key Names

Add randomness to the beginning of the key name

<my_bucket>/521335461-2013_11_13.jpg<my_bucket>/465330151-2013_11_13.jpg

<my_bucket>/987331160-2013_11_13.jpg<my_bucket>/465765461-2013_11_13.jpg<my_bucket>/125631151-2013_11_13.jpg<my_bucket>/934563160-2013_11_13.jpg<my_bucket>/532132341-2013_11_13.jpg

<my_bucket>/565437681-2013_11_13.jpg<my_bucket>/234567460-2013_11_13.jpg<my_bucket>/456767561-2013_11_13.jpg<my_bucket>/345565651-2013_11_13.jpg<my_bucket>/431345660-2013_11_13.jpg

Other Techniques for Distributing Key Names

Store objects as a hash of their name• add the original name as metadata

• “deadmau5_mix.mp3” à 0aa316fb000eae52921aab1b4697424958a53ad9– watch for duplicate names!

• prepend keyname with short hash• 0aa3-deadmau5_mix.mp3

Epoch time (reverse)• 5321354831-deadmau5_mix.mp3

Randomness in a Key Name Can Be an Anti-Pattern

Lifecycle policies LISTs with prefix filtersMaintaining thumbnails of images

• craig.jpg -> stored as orig-09329jed0fc • thumb-09329jed0fc

When you need to recover a file with its original name

Solving for the Anti-Pattern

Add additional prefixes to help sorting

Amazon S3 maintains keys lexicographically in its internal indices

<my_bucket>/images/521335461-2013_11_13.jpg

<my_bucket>/images/465330151-2013_11_13.jpg<my_bucket>/movies/293924440-2013_11_13.jpg<my_bucket>/movies/987331160-2013_11_13.jpg<my_bucket>/thumbs-small/838434842-2013_11_13.jpg<my_bucket>/thumbs-small/342532454-2013_11_13.jpg

<my_bucket>/thumbs-small/345233453-2013_11_13.jpg<my_bucket>/thumbs-small/345453454-2013_11_13.jpg

Amazon CloudFront

CloudFront dynamic content accleratoion

Region

Edge Location

12 Regions32 Availability Zones54 Edge Locations

Need to update

We’re here J

Configure multiple origins

Elastic Load Balancing

Dynamic content

Amazon EC2

Static content

Amazon S3

* (default)

/error/*/assets/*

Amazon CloudFrontexample.com

CloudFront Behaviors

CloudFront

CustomerLocationwww.mysite.com

Path Pattern Matching/*.jpg; /*.php etc.

GET http://mysite.com/images/1.jpg to ORIGIN AGET http://mysite.com/index.php to ORIGIN B

GET http://mysite.com/web/home.css to ORIGIN CGET http://mysite.com/* (DEFAULT) to ORIGIN D

Origin A: S3 bucket

Origin B: www.mysite.com

Origin C: S3 Bucket

Origin D: www.mysite.com

Path Pattern Matching

/*.php

/images/*.jpg

/web/*.css

/*.* (DEFAULT)

Region

Edge Location

12 Regions32 Availability Zones54 Edge Locations

Need to update AWS optimized network

Internet

Demo J

S3 Transfer AccelerationNew~

Amazon S3 Transfer AccelerationEmbedded WAN acceleration

S3 BucketAWS EdgeLocation

Uploader

OptimizedThroughput!

Move over long geographic distances

Up to 300% (6x) faster

No firewall mods, no client software

54 global edge locations

Change your endpoint, not your code

Accelerate Speed Comparison

• Test URL• http://s3-accelerate-speedtest.s3-

accelerate.amazonaws.com/en/accelerate-speed-comparsion.html

• bit.ly/news3ta• Test Result (May-02-2016)

• Tested on May-02-2016, LGU+ Wifi at GSTower in Seoul• http://bit.ly/newss3taresult

Testing S3 Transfer Accelerator by AWSCLI

$ sudo pip install –upgrade awscli

$ aws configure set default.s3.use_accelerate_endpoint true

Testing S3 Transfer Accelerator by AWSCLI

$ aws s3 cp 33MB.pptx s3://ilho-saopaulo-01/$ aws s3 cp 33MB.pptx s3://ilho-saopaulo-01/ --endpoint-url http://ilho-saopaulo-01.s3-accelerate.amazonaws.com

S3 Transfer Acceleration Pricing

Starting at $0.04/GB transferred (+ usual bandwidth charges). Up to $0.08/GB in some regions

Pay only for what you use

Accelerated performance or no charge

Compare to hardware, per-GB or licenses