The Future of Fiber Optics in the Data Center · The Future of Fiber Optics in the Data Center Today, Tomorrow and Next Year Dwayne Crawford – Global PLM Fiber Connectivity

The Future of Fiber Optics in the Data CenterToday, Tomorrow and Next Year

Dwayne Crawford – Global PLM Fiber Connectivity

The World We Live In …The Internet of Everything …

Internet of Information60T web pages (Google Index, October 2014)

Internet of People1.4B Facebook active users (December 2014)

Internet of MobilityMobile devices account for 44% of all IP traffic (2013)

… Connected to 50B Things by 2020

Image: Cisco Connections Counter

2015 Tech TrendsThe Merging of the Real and the Virtual Worlds

1 Computing Everywhere

2 Internet of Things

3 3D Printing

Intelligence Everywhere

4 Advanced Pervasive Invisible Analytics

5 Context-Rich Systems

6 Smart Machines

IT for the Digital Business

7 Cloud/Client Architecture

8 Software-Defined Infrastructure & Applications

9 Web-Scale IT

10 Risk-Based Security & Self-Protection

Source: Gartner, 2014

Data Characteristics

Volume

Source

Location

Flow

Frequency

Diversity

APPLICATIONS

DATANETWORK

Multiple “layer 0” technologies required toaddress future needs

Connectivity

Cabling

Infrastructure

Producers• Applications generate data and/or results • Services that connect collaborative

producers• Hardware and Connectivity are simply

Enablers

Consumers• Utilize Applications and Services utilizing

Hardware and Connectivity

Producers and Consumers in the Enterprise

Data Center (Applications)LAN (Services)

• Standards driven using commoditized technologies

• Installed base responding to current needs• Modest growth tied to

– New construction– IP Convergence– Power-over-cabling– WLAN– POLAN & DAS

Two Sets of Market Needs

• Rapidly transforming (and departing from LAN)• Robust growth fuelled by

− More data− Greater bandwidth− Improved efficiency

• Cloud providers are changing the rules!− Hotbed for new ideas, new technologies,

new topologies

• Mature and conservative • Dynamic and fast-paced

• Enterprise Data Centers Declining• Data Centers Consolidating

Applications Are Moving

• Bigger Data Centers Growing• Economies of Scale• Shift of Power & Influence

Cloud Shift

How Fast Are Applications Moving?

Cloud$1.1B TAM, +31% CAGR

Enterprise Owned$3.4B TAM, -4% CAGR

Sources: Dell Oro, Gartner, 451, Cisco Networking Report

Multi-Tenant$1.2B TAM, +2% CAGR

Key Attributes:• Capital investment model

for new builds• Full spectrum of scale and

technologies reside in enterprise owned DCs

Key Attributes:• IT owned by enterprise

client, while space and power are leased

• Providers expanding into hosting services

Key Attributes:• Operating expense model

for clients• Enabler of accelerating

mobile / cloud usage

Cloud Server Shipments Will Exceed Enterprise Server Shipments by 2018

Let’s Take a Closer Look:Data Center (Applications)

• Throughput and Protocols • Compute & Silicon Photonics• Connectors

Data Center Evolution

Domain 2 – Switch to Switch• Infrastructure – Long Trunks • Cable Plant Cost ($/strand)• Low Pressure on Transceiver Cost

Domain 1 – Switch to Server• Interconnects – Short Patch Cords• Power • Cleaning• High Pressure on Transceiver Cost

Protocols: The Long and Short of IT

For Domain 2: Switch to Switch

Protocols: Options Today

Ethernet Protocol

100G-SR10 (MPO-24) Cable Cost

40G-SR4 Solution Cost

40G-SR4 (4 x 10G) High Density, Scalable

40G-SR-BD (BiDi) Lack of Standard

10G-SR Too Slow?

AOCs (10G/40G/100G) Installation

For Domain 1: Server to Switch

Protocols: Options Today

Ethernet Protocol

100G / 40G CPU Overhead

40G-SR4 (4 x 10G) High Density, Scalable

40G-SR-BD (BiDi) Transceiver Cost

10G-SR Too Slow?

AOC 40G (QSFP+ to 4x10G SFP+) High Density, Cost

AOC 10G (SFP+ to SFP+) Cost

Protocols: Options Tomorrow

Why Does This Happen Today?

LAN

SAN

10G

16G

40G Channel on a Modern Server

21 GbpsEthernet

39 GbpsInfiniband(RDMA)

http://www.nas.nasa.gov/assets/pdf/papers/NAS_Technical_Report_NAS-2014-01.pdf

Fabric of the Future

LAN

SANConverged

Ethernet

25G

RoCE (RDMA over Converged Ethernet) v.2

Sca

labi

lity

/ P

orta

ble

•“D

ocke

r” C

onta

iner

s

Big

Dat

a / C

lust

er•

Apa

che

Had

oop

• 25G @ Server enables Convergence

New Rising Star: 25G Ethernet

Key Promoters & Adopters

25G Timeline• New in January 2014−Market excitement & rapid momentum

• Cloud deployment starts 2015/2016• Eclipses 40G by 2017

Why the Excitement• Less $/Gb• LC Duplex or AOC Architecture• Replaces FibreChannel & Infiniband w/ RoCE• Cost effective building block for 50G/100G/400G

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se

rver - SR (50G Lane)

- SR4 (25G Lane)- AOC

- SR- AOC

- SR (50G Lane)- AOC

Ethernet


Compute Challenge

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se



- SR- AOC


Ethernet


Compute Challenge

High Speed Networking Options AOC (Active Optical Cable)• High Performance • Low Power

Image: cisco.com

Image: sfpex.com

Transceivers• Long Reach Structured Cabling • Cost effective alternative

10G Lanes: SFP+/QSFP+25G Lanes: SFP28/QSFP28

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se



- SR- AOC

- SR- AOC


Ethernet

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se



- SR- AOC


Ethernet


Compute Challenge

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se



- SR- AOC

- SR- AOC


Ethernet

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se



- SR- AOC


Ethernet


Compute Challenge

Birth of Bi-Directional (BiDi)

WDM (Wave Division Multiplexing)• 2 Fibers• 2 x 10G / fiber• OM3 & OM4 Fiber

BiDi Balances: $/Transceiver vs. $/Channel (Cable)

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se

rver

-BiDi (OM3/4, 2f)- AOC

- SR (50G Lane)- SR4 (25G Lane)

- AOC

- SR- AOC

- SR- AOC


Ethernet

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se



- SR- AOC

- SR- AOC


Ethernet


Compute Challenge

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se

rver



- AOC

- SR- AOC

- SR- AOC


Ethernet

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se



- SR- AOC

- SR- AOC


Ethernet


Compute Challenge

WDM Approach @ 400G over MMF

SR16 Was Posed to Fail• 32 x 25G Fibers!• New MPO-32

SR4 Ultra Wide Band (UWB) over MMF• Low-cost WDM• “OM5” Fiber

UWB Upside for 100G over MMF

• MPO-12 (12f)• OM3, OM4

• LC Duplex (2f)• OM4, OM5

• MPO-24 (24f)• OM3, OM4

50G Gray Zone!

• LC Duplex (2f)• OM4?, OM5

• Connector?• OM4?, OM5

• 50G Lane?• OM4?, OM5

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se

rver



- AOC

- SR- AOC

- SR- AOC


- SR (50G Lane)-UWB (OM5, 2f)

- AOC

-UWB (OM5, 2f)-SR4 (25G Lane)

- AOC

- SR8 (50G Lane)-UWB (OM5, 4f)-SR16 (25G Lane)

- AOC

Ethernet


Compute Challenge

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se

rver

-PSM4 (SM, 8f)



- AOC

- SR- AOC

- SR- AOC



- AOC


- AOC


- AOC

Ethernet

-PSM4 (SM, 8f)


Compute Challenge

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se

rver

-PSM4 (SM, 8f)



- AOC

- SR- AOC

- SR- AOC



- AOC


- AOC


- AOC

Ethernet

-PSM4 (SM, 8f)

Protocols: Quick Summary

Compute Challenge

Single Mode Dominance

Faster Duplex UWB + OM5

AOC




Current Compute Designs Nearing LimitsGrowing Parallelism • Moore’s Law

Serial Limitations• Limited Clock Rate• VCore Floor• Maximized Power Dissipation / Area

• Parallel Physical Limitations– Pin Count – Power Dissipation– Parallel Trace Routing & Noise

• Serial Reach Limitations– Frequency vs. Reach

http://dsp-fpga.com/articles/can-pcb-handle-speed/

http://www.amkor.com/go/3D-Stacked-Die-Packaging

Practical Limitations of I/O

Amdahl & Gustafson – Computing Limitations

"AmdahlsLaw" by Daniels220 at English Wikipedia - Own work based on: File:AmdahlsLaw.png. Licensed under CC BY-SA 3.0 via Wikimedia Commons -http://commons.wikimedia.org/wiki/File:AmdahlsLaw.svg#mediaviewer/File:AmdahlsLaw.svg

"Gustafson" by Peahihawaii - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Gustafson.png#/media/File:Gustafson.png

“The Speedup of a program using multiple processors in parallel computing is limited by the time needed for the sequential fraction of the program.” - Amdahl

The challenge is serial

“…the effort expended on achieving high parallel processing rates is wasted unless it is accompanied by achievements in sequential processing rates of very nearly the same magnitude” - Gustafson

• Component Separation Issues• Intel QPI 8.0 GT/s <12”• PCIe Gen3 (~126 Gbps) <20”• SATA 3.2 (~16 Gbps) <40”

Limited Compute Architectures

Intel Xeon Phi MBS4600LH

Data Centers Need: – Faster Lane Rates– Longer Reach

End Game: Compute Disaggregation

Evolving the Compute Architecture

I/O

Local Storage

Memory

Compute

• Chip to Chip• Board to Board• Box to Box

Future is Silicon Photonics! What is This?




Compute Runs on Hex!

• Hexadecimal– Base 16– Programming (8/16/32/64/128 bits)– Buses (16/32/64/128 bits)

• Where is Fiber Today– Senary (Base 6)– Duodecimal (Base 12)

NEW:• Lime• Tan • Olive• Magenta

Base 8/16 ShiftBASE: 6/12 – LC & MPO-12• 12/24/36/48/72/96/144 Fiber

BASE: 8/16 – LC, MPO-16, MXC• 16/32/48/64/96/128/192 Fiber

Today

RSN

Challenges• Loss• Reach• Cable count

Challenges• Power• Density• Cleaning

Unique Connector Challenges Within The Rack

MXC

Evolving Connector Standards

MPO-16 / MPO-32• 1 to 4 Rows• 16 Fibers / Row (64f Ferrule)• Micro-Lensed Optics• Short Reach “ToR Patch Cord”−Switch to Server−Compute Disaggregation

• 1 or 2 Rows• 16 Fibers / Row (32f Ferrule)• Traditional Polished Endface

− Low Loss, Cleaning Sensitive • Longer Reach Structured Cabling−Switch to Switch

Image: intel.com

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se

rver

-PSM4 (SM, 8f)



- AOC

- SR- AOC

- SR- AOC



- AOC


- AOC


- AOC

Ethernet

-PSM4 (SM, 8f)

Protocols: Change of State

Compute Challenge

25G 50G40G 100G 400G

<500m

<100m

<15m

Dom

ain

2:

Switc

h-Sw

itch

Dom

ain

1:

Switc

h-Se

rver

-PSM4 (SM, 8f)



- AOC

- SR- AOC

- SR- AOC



- AOC


- AOC


- AOC

Ethernet

-PSM4 (SM, 8f)

Protocols: Change of State

Compute Challenge

Single ModeSilicon Photonics

Dominance

Parting Words

• Exciting Times • Rapid Evolution • New Technology & Approaches

• Plan Accordingly• Short vs. Long Term

Documents

The Future of Fiber Optics in the Data Center · The Future of Fiber Optics in the Data Center Today, Tomorrow and Next Year Dwayne Crawford – Global PLM Fiber Connectivity