The Future of Fiber Optics in the Data CenterToday, Tomorrow and Next Year
Dwayne Crawford – Global PLM Fiber Connectivity
The World We Live In …The Internet of Everything …
Internet of Information60T web pages (Google Index, October 2014)
Internet of People1.4B Facebook active users (December 2014)
Internet of MobilityMobile devices account for 44% of all IP traffic (2013)
… Connected to 50B Things by 2020
Image: Cisco Connections Counter
2015 Tech TrendsThe Merging of the Real and the Virtual Worlds
1 Computing Everywhere
2 Internet of Things
3 3D Printing
Intelligence Everywhere
4 Advanced Pervasive Invisible Analytics
5 Context-Rich Systems
6 Smart Machines
IT for the Digital Business
7 Cloud/Client Architecture
8 Software-Defined Infrastructure & Applications
9 Web-Scale IT
10 Risk-Based Security & Self-Protection
Source: Gartner, 2014
Data Characteristics
Volume
Source
Location
Flow
Frequency
Diversity
APPLICATIONS
DATANETWORK
Multiple “layer 0” technologies required toaddress future needs
Connectivity
Cabling
Infrastructure
Producers• Applications generate data and/or results • Services that connect collaborative
producers• Hardware and Connectivity are simply
Enablers
Consumers• Utilize Applications and Services utilizing
Hardware and Connectivity
Producers and Consumers in the Enterprise
Data Center (Applications)LAN (Services)
• Standards driven using commoditized technologies
• Installed base responding to current needs• Modest growth tied to
– New construction– IP Convergence– Power-over-cabling– WLAN– POLAN & DAS
Two Sets of Market Needs
• Rapidly transforming (and departing from LAN)• Robust growth fuelled by
− More data− Greater bandwidth− Improved efficiency
• Cloud providers are changing the rules!− Hotbed for new ideas, new technologies,
new topologies
• Mature and conservative • Dynamic and fast-paced
• Enterprise Data Centers Declining• Data Centers Consolidating
Applications Are Moving
• Bigger Data Centers Growing• Economies of Scale• Shift of Power & Influence
Cloud Shift
How Fast Are Applications Moving?
Cloud$1.1B TAM, +31% CAGR
Enterprise Owned$3.4B TAM, -4% CAGR
Sources: Dell Oro, Gartner, 451, Cisco Networking Report
Multi-Tenant$1.2B TAM, +2% CAGR
Key Attributes:• Capital investment model
for new builds• Full spectrum of scale and
technologies reside in enterprise owned DCs
Key Attributes:• IT owned by enterprise
client, while space and power are leased
• Providers expanding into hosting services
Key Attributes:• Operating expense model
for clients• Enabler of accelerating
mobile / cloud usage
Cloud Server Shipments Will Exceed Enterprise Server Shipments by 2018
Let’s Take a Closer Look:Data Center (Applications)
• Throughput and Protocols • Compute & Silicon Photonics• Connectors
Data Center Evolution
Domain 2 – Switch to Switch• Infrastructure – Long Trunks • Cable Plant Cost ($/strand)• Low Pressure on Transceiver Cost
Domain 1 – Switch to Server• Interconnects – Short Patch Cords• Power • Cleaning• High Pressure on Transceiver Cost
Protocols: The Long and Short of IT
For Domain 2: Switch to Switch
Protocols: Options Today
Ethernet Protocol
100G-SR10 (MPO-24) Cable Cost
40G-SR4 Solution Cost
40G-SR4 (4 x 10G) High Density, Scalable
40G-SR-BD (BiDi) Lack of Standard
10G-SR Too Slow?
AOCs (10G/40G/100G) Installation
For Domain 1: Server to Switch
Protocols: Options Today
Ethernet Protocol
100G / 40G CPU Overhead
40G-SR4 (4 x 10G) High Density, Scalable
40G-SR-BD (BiDi) Transceiver Cost
10G-SR Too Slow?
AOC 40G (QSFP+ to 4x10G SFP+) High Density, Cost
AOC 10G (SFP+ to SFP+) Cost
40G Channel on a Modern Server
21 GbpsEthernet
39 GbpsInfiniband(RDMA)
http://www.nas.nasa.gov/assets/pdf/papers/NAS_Technical_Report_NAS-2014-01.pdf
RoCE (RDMA over Converged Ethernet) v.2
Sca
labi
lity
/ P
orta
ble
•“D
ocke
r” C
onta
iner
s
Big
Dat
a / C
lust
er•
Apa
che
Had
oop
• 25G @ Server enables Convergence
New Rising Star: 25G Ethernet
Key Promoters & Adopters
25G Timeline• New in January 2014−Market excitement & rapid momentum
• Cloud deployment starts 2015/2016• Eclipses 40G by 2017
Why the Excitement• Less $/Gb• LC Duplex or AOC Architecture• Replaces FibreChannel & Infiniband w/ RoCE• Cost effective building block for 50G/100G/400G
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver - SR (50G Lane)
- SR4 (25G Lane)- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
Protocols: Options Tomorrow
Compute Challenge
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver - SR (50G Lane)
- SR4 (25G Lane)- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
Protocols: Options Tomorrow
Compute Challenge
High Speed Networking Options AOC (Active Optical Cable)• High Performance • Low Power
Image: cisco.com
Image: sfpex.com
Transceivers• Long Reach Structured Cabling • Cost effective alternative
10G Lanes: SFP+/QSFP+25G Lanes: SFP28/QSFP28
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver - SR (50G Lane)
- SR4 (25G Lane)- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver - SR (50G Lane)
- SR4 (25G Lane)- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
Protocols: Options Tomorrow
Compute Challenge
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver - SR (50G Lane)
- SR4 (25G Lane)- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver - SR (50G Lane)
- SR4 (25G Lane)- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
Protocols: Options Tomorrow
Compute Challenge
Birth of Bi-Directional (BiDi)
WDM (Wave Division Multiplexing)• 2 Fibers• 2 x 10G / fiber• OM3 & OM4 Fiber
BiDi Balances: $/Transceiver vs. $/Channel (Cable)
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver
-BiDi (OM3/4, 2f)- AOC
- SR (50G Lane)- SR4 (25G Lane)
- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver - SR (50G Lane)
- SR4 (25G Lane)- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
Protocols: Options Tomorrow
Compute Challenge
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver
-BiDi (OM3/4, 2f)- AOC
- SR (50G Lane)- SR4 (25G Lane)
- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver - SR (50G Lane)
- SR4 (25G Lane)- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
Ethernet
Protocols: Options Tomorrow
Compute Challenge
WDM Approach @ 400G over MMF
SR16 Was Posed to Fail• 32 x 25G Fibers!• New MPO-32
SR4 Ultra Wide Band (UWB) over MMF• Low-cost WDM• “OM5” Fiber
UWB Upside for 100G over MMF
• MPO-12 (12f)• OM3, OM4
• LC Duplex (2f)• OM4, OM5
• MPO-24 (24f)• OM3, OM4
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver
-BiDi (OM3/4, 2f)- AOC
- SR (50G Lane)- SR4 (25G Lane)
- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
- SR (50G Lane)-UWB (OM5, 2f)
- AOC
-UWB (OM5, 2f)-SR4 (25G Lane)
- AOC
- SR8 (50G Lane)-UWB (OM5, 4f)-SR16 (25G Lane)
- AOC
Ethernet
Protocols: Options Tomorrow
Compute Challenge
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver
-PSM4 (SM, 8f)
-BiDi (OM3/4, 2f)- AOC
- SR (50G Lane)- SR4 (25G Lane)
- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
- SR (50G Lane)-UWB (OM5, 2f)
- AOC
-UWB (OM5, 2f)-SR4 (25G Lane)
- AOC
- SR8 (50G Lane)-UWB (OM5, 4f)-SR16 (25G Lane)
- AOC
Ethernet
-PSM4 (SM, 8f)
Protocols: Options Tomorrow
Compute Challenge
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver
-PSM4 (SM, 8f)
-BiDi (OM3/4, 2f)- AOC
- SR (50G Lane)- SR4 (25G Lane)
- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
- SR (50G Lane)-UWB (OM5, 2f)
- AOC
-UWB (OM5, 2f)-SR4 (25G Lane)
- AOC
- SR8 (50G Lane)-UWB (OM5, 4f)-SR16 (25G Lane)
- AOC
Ethernet
-PSM4 (SM, 8f)
Protocols: Quick Summary
Compute Challenge
Single Mode Dominance
Faster Duplex UWB + OM5
AOC
Data Center Evolution
Let’s Take a Closer Look:Data Center (Applications)
• Throughput and Protocols • Compute & Silicon Photonics• Connectors
Current Compute Designs Nearing LimitsGrowing Parallelism • Moore’s Law
Serial Limitations• Limited Clock Rate• VCore Floor• Maximized Power Dissipation / Area
• Parallel Physical Limitations– Pin Count – Power Dissipation– Parallel Trace Routing & Noise
• Serial Reach Limitations– Frequency vs. Reach
http://dsp-fpga.com/articles/can-pcb-handle-speed/
http://www.amkor.com/go/3D-Stacked-Die-Packaging
Practical Limitations of I/O
Amdahl & Gustafson – Computing Limitations
"AmdahlsLaw" by Daniels220 at English Wikipedia - Own work based on: File:AmdahlsLaw.png. Licensed under CC BY-SA 3.0 via Wikimedia Commons -http://commons.wikimedia.org/wiki/File:AmdahlsLaw.svg#mediaviewer/File:AmdahlsLaw.svg
"Gustafson" by Peahihawaii - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Gustafson.png#/media/File:Gustafson.png
“The Speedup of a program using multiple processors in parallel computing is limited by the time needed for the sequential fraction of the program.” - Amdahl
The challenge is serial
“…the effort expended on achieving high parallel processing rates is wasted unless it is accompanied by achievements in sequential processing rates of very nearly the same magnitude” - Gustafson
• Component Separation Issues• Intel QPI 8.0 GT/s <12”• PCIe Gen3 (~126 Gbps) <20”• SATA 3.2 (~16 Gbps) <40”
Limited Compute Architectures
Intel Xeon Phi MBS4600LH
Data Centers Need: – Faster Lane Rates– Longer Reach
End Game: Compute Disaggregation
Evolving the Compute Architecture
I/O
Local Storage
Memory
Compute
Data Center Evolution
Let’s Take a Closer Look:Data Center (Applications)
• Throughput and Protocols • Compute & Silicon Photonics• Connectors
Compute Runs on Hex!
• Hexadecimal– Base 16– Programming (8/16/32/64/128 bits)– Buses (16/32/64/128 bits)
• Where is Fiber Today– Senary (Base 6)– Duodecimal (Base 12)
NEW:• Lime• Tan • Olive• Magenta
Base 8/16 ShiftBASE: 6/12 – LC & MPO-12• 12/24/36/48/72/96/144 Fiber
BASE: 8/16 – LC, MPO-16, MXC• 16/32/48/64/96/128/192 Fiber
Today
RSN
Challenges• Loss• Reach• Cable count
Challenges• Power• Density• Cleaning
Unique Connector Challenges Within The Rack
MXC
Evolving Connector Standards
MPO-16 / MPO-32• 1 to 4 Rows• 16 Fibers / Row (64f Ferrule)• Micro-Lensed Optics• Short Reach “ToR Patch Cord”−Switch to Server−Compute Disaggregation
• 1 or 2 Rows• 16 Fibers / Row (32f Ferrule)• Traditional Polished Endface
− Low Loss, Cleaning Sensitive • Longer Reach Structured Cabling−Switch to Switch
Image: intel.com
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver
-PSM4 (SM, 8f)
-BiDi (OM3/4, 2f)- AOC
- SR (50G Lane)- SR4 (25G Lane)
- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
- SR (50G Lane)-UWB (OM5, 2f)
- AOC
-UWB (OM5, 2f)-SR4 (25G Lane)
- AOC
- SR8 (50G Lane)-UWB (OM5, 4f)-SR16 (25G Lane)
- AOC
Ethernet
-PSM4 (SM, 8f)
Protocols: Change of State
Compute Challenge
25G 50G40G 100G 400G
<500m
<100m
<15m
Dom
ain
2:
Switc
h-Sw
itch
Dom
ain
1:
Switc
h-Se
rver
-PSM4 (SM, 8f)
-BiDi (OM3/4, 2f)- AOC
- SR (50G Lane)- SR4 (25G Lane)
- AOC
- SR- AOC
- SR- AOC
- SR (50G Lane)- AOC
- SR (50G Lane)-UWB (OM5, 2f)
- AOC
-UWB (OM5, 2f)-SR4 (25G Lane)
- AOC
- SR8 (50G Lane)-UWB (OM5, 4f)-SR16 (25G Lane)
- AOC
Ethernet
-PSM4 (SM, 8f)
Protocols: Change of State
Compute Challenge
Single ModeSilicon Photonics
Dominance