View
216
Download
3
Category
Preview:
Citation preview
1
400G Demonstrator for ISC ‘13400G Demonstrator for ISC ‘13400G Demonstrator for ISC ‘13400G Demonstrator for ISC ‘13
HPCN Workshop, Braunschweig, 7. Mai 2013HPCN Workshop, Braunschweig, 7. Mai 2013HPCN Workshop, Braunschweig, 7. Mai 2013HPCN Workshop, Braunschweig, 7. Mai 2013
Wolfgang Wünsch, Technische Universität Dresden
Eduard Beier, T-Systems International
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13
Agenda
� Partner
� Purpose
� Project Structure
� Topology
� Turbine Development
� Climate Computing
� Service Recipient Relations
� Data Path
� The Big Picture
� Project Lifetime
� Timeline
� DATE
� Test items
just click on the just click on the just click on the just click on the
hyperlinked topics! hyperlinked topics! hyperlinked topics! hyperlinked topics!
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 3
Partner
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 4
Purpose
The purpose of the project is: to demonstrate, that bandwidth beyond 100GBit/s is feasible and useful
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 5
Project Structure
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
Project Project Project Project BoardBoardBoardBoardProf. Dr. A. Bode / Prof. Prof. Dr. A. Bode / Prof. Prof. Dr. A. Bode / Prof. Prof. Dr. A. Bode / Prof. DrDrDrDr .W. Nagel .W. Nagel .W. Nagel .W. Nagel
Dr. A. KlugeDr. A. KlugeDr. A. KlugeDr. A. Kluge F. SchneiderF. SchneiderF. SchneiderF. Schneider Prof. Dr. W. GentzschProf. Dr. W. GentzschProf. Dr. W. GentzschProf. Dr. W. Gentzsch R. R. R. R. WienekeWienekeWienekeWieneke
M. M. M. M. ZappolinoZappolinoZappolinoZappolino Dr. A. GeigerDr. A. GeigerDr. A. GeigerDr. A. Geiger M. M. M. M. RoosenRoosenRoosenRoosen M. FuchsM. FuchsM. FuchsM. Fuchs
A. A. A. A. ClaubergClaubergClaubergClauberg T. T. T. T. WeselowskiWeselowskiWeselowskiWeselowski Jan Jan Jan Jan HeichlerHeichlerHeichlerHeichler
Project ManagementProject ManagementProject ManagementProject ManagementE.BeierE.BeierE.BeierE.Beier W. WünschW. WünschW. WünschW. Wünsch n.nn.nn.nn.n....
WP1WP1WP1WP1PerformancetTestsPerformancetTestsPerformancetTestsPerformancetTests
Andy GeorgiAndy GeorgiAndy GeorgiAndy Georgi
System Performance
Metering
WP2WP2WP2WP2Parallel Parallel Parallel Parallel FilesystemsFilesystemsFilesystemsFilesystems
Klaus GottschalkKlaus GottschalkKlaus GottschalkKlaus Gottschalk
Filesystem
Optimizing
WP3WP3WP3WP3Server & StorageServer & StorageServer & StorageServer & Storage
Beier/WünschBeier/WünschBeier/WünschBeier/Wünsch
Server & Storage
Project
Management
WP4WP4WP4WP4TransportTransportTransportTransport
MaskosMaskosMaskosMaskos / Mayer/ Mayer/ Mayer/ Mayer
PlanningPlanningPlanningPlanning / / / / EngineeringEngineeringEngineeringEngineering
WDM Project
Management
WP5WP5WP5WP5Layer 2/3Layer 2/3Layer 2/3Layer 2/3
Daniel Daniel Daniel Daniel NowaraNowaraNowaraNowara
Router Project
Management
WP7WP7WP7WP7ApplicationsApplicationsApplicationsApplications
Ferdinand Ferdinand Ferdinand Ferdinand JamitzkyJamitzkyJamitzkyJamitzky
Applications
Project
Management
WP6WP6WP6WP6SDN & NFVSDN & NFVSDN & NFVSDN & NFV
Ralf BraunRalf BraunRalf BraunRalf Braun
SDN & NFV &
Security
WP8WP8WP8WP8Public RelationsPublic RelationsPublic RelationsPublic Relations
Udo SchäferUdo SchäferUdo SchäferUdo Schäfer
Project
Marketing
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 6
WP1: Performance Tests
• Performance of subsystems (e.g. storage) and total performance measurements
•Feedback for subsystem optimization
•Conformance to measurement standards
•Input for publications
•
•WP lead: Andy GeorgiWP lead: Andy GeorgiWP lead: Andy GeorgiWP lead: Andy Georgi
Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 7
WP1: Targets
Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure
3x700G 3x700G 3x700G 3x700G FlashFlashFlashFlash
ServerServerServerServer Router Router Router Router 6 6 6 6 GByteGByteGByteGByte/s/s/s/s 5 5 5 5 GByteGByteGByteGByte/s/s/s/s
∼∼∼∼7000 7000 7000 7000 corescorescorescores
IB FDR IB FDR IB FDR IB FDR NetworkNetworkNetworkNetwork
7 7 7 7 G
Byte
GBy
teG
Byte
GBy
te/s/s /s/s
36x700G 36x700G 36x700G 36x700G FlashFlashFlashFlash
12x 12x 12x 12x ServerServerServerServer
RouterRouterRouterRouter72GByte/s72GByte/s72GByte/s72GByte/s 60GByte/s60GByte/s60GByte/s60GByte/s
∼∼∼∼7000 7000 7000 7000 corescorescorescores
IB FDR IB FDR IB FDR IB FDR NetworkNetworkNetworkNetwork
84G
Byte
/s84
GBy
te/s
84G
Byte
/s84
GBy
te/s
55550GByte/s0GByte/s0GByte/s0GByte/s
Per ServerPer ServerPer ServerPer Server
TotalTotalTotalTotal
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 8
WP2: Parallel File System
•Planning, roll out, optimization and operation of the Parallel File system in coordination with other WPs and partners
•Configure and parameterize the Parallel File System (e.g. TCP buffers)
•Coordinate the communication between Clusters, File System and Network (IP Concept)
•Input for publications
•
•WP lead: Klaus GottschalkWP lead: Klaus GottschalkWP lead: Klaus GottschalkWP lead: Klaus Gottschalk
Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 9
WP3: Server & Storage & IB & etc
•Planning, roll out, optimization and operation of the server, storage and Infiniband infrastructure in coordination with other WPs and partners
•Input for publications
•
•WP lead : WP lead : WP lead : WP lead : ProjektProjektProjektProjekt Management Management Management Management
Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 10
WP4: Transport
•Planning, roll out, optimization and operation of the fiber and WDM infrastructure in coordination with other WPs and partners
•Input for publications
•
•WP lead: Stefan WP lead: Stefan WP lead: Stefan WP lead: Stefan MaskosMaskosMaskosMaskos (Planning) / Heinz Mayer (Technology)(Planning) / Heinz Mayer (Technology)(Planning) / Heinz Mayer (Technology)(Planning) / Heinz Mayer (Technology)
Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 11
WP5: Layer 2/3
•Planning, roll out, optimization and operation of the Router infrastructure in coordination with other WPs and partners
•Input for publication
•
•WPWPWPWP----LeiterLeiterLeiterLeiter: Daniel : Daniel : Daniel : Daniel NowaraNowaraNowaraNowara
Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 12
WP6: SDN & NFV & Security
• attend SDN and NFV approaches
• set up a security concept in coordination with the partners
•Implement that concept
• input for publications
•WP lead Ralf Braun (TWP lead Ralf Braun (TWP lead Ralf Braun (TWP lead Ralf Braun (T----Labs)Labs)Labs)Labs)
Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 13
WP7: Applications
•Coordination of the application teams
•Input for publications
•
•WP lead: Ferdinand WP lead: Ferdinand WP lead: Ferdinand WP lead: Ferdinand JamitzkyJamitzkyJamitzkyJamitzky
Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 14
WP8: Public Relations
•Coordinate partners and activities towards optimum project marketing
•Coordinate press release activities
•Produce and attend project PR material (flyer, articles, etc)
•Coordinate ISC booth activities (flyer, logo, sessions, poster, give aways, etc)
•Coordinate the ISC application demonstration (incl. Internet access)
•
•WP lead: Udo WP lead: Udo WP lead: Udo WP lead: Udo SchäferSchäferSchäferSchäfer
Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 15
100G Sachsen operational
100G Sachsen planning
400G Demonstrator p
Chemnitz
Topology
10GbE for Demonstrator
Computing Center
Euro Industriepark
München
DT PoP
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
Turbine Development
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
� Cooperation with DLR
� Workflow Demonstration
� Preprocessing
� Solver 1
� Solver 2
� Postprocessing
� Turbine model calculation with n Eigenmodes and m Phase Angels
Details:Details:Details:Details:
� Data volume: ~ 1TB
� Overall Workflow:
� Multitude of independent simulation runs (HTC).
� Simulations running on HPC resources at different sites.
� Every simulation produces input data for subsequent simulations.
� Subsequent simulations again run at different sites.
Thus to avoid knock-on delays in workflow execution data instantly should be availableat different sites!
GPFS:GPFS:GPFS:GPFS:
� Adopted feature: Active File Management (AFM) and Stretched Cluster
� Cross site data replication allows running simulations without prior copying
� Implicit data consistent backup via AFM data replication
Turbine Development: Benefits of GPFS Usage on 400G
Back to Back to Back to Back to Turbine Development Turbine Development Turbine Development Turbine Development
400G:400G:400G:400G:
� Possible job distributions on HPC resources:
� n * m jobs (n Eigenmodes, m phase angles)
� a * b jobs on cluster (a parallel jobs running on c cores, b serial jobs) with a * b ≥ n * m
� Bandwidth requirements: bandwidth = �����∙���∆�
, where� ���� = Avg. file size written per job
� ����� = No. of running in parallel (Here: corresponds to a)
� ∆� = Avg. time for disk access
Turbine Development: Benefits of GPFS Usage on 400G
No. cores
time
a = 6
n * m = 28 ≥ a * b = 30
∆t
Solver 1
Solver 2
b = 5
240min
Back to Back to Back to Back to Turbine Development Turbine Development Turbine Development Turbine Development
400G: 400G: 400G: 400G: BandwidthBandwidthBandwidthBandwidth requirementsrequirementsrequirementsrequirements forforforfor different different different different jobjobjobjob distributiondistributiondistributiondistribution setupssetupssetupssetups
� Extreme/HTC setup with a = n * m = 300, b = 1:
� Assuming jobs all writing within 15min to disk an avg. file size of 150MB (i.e. write peak):
� Required bandwidth: 400GBit/s400GBit/s400GBit/s400GBit/s
� Required machine size: > 19200 Cores (when single jobs run on 64 cores)
� „Gentle“ setup with a = 50, b = 6:
� Assuming jobs having an avg. runtime of 240min continously writing 150MB ofdata to disk to represent runtime differences over larger values b:
� Required bandwidth: 4GBit/s4GBit/s4GBit/s4GBit/s
� Required machine size: > 3200 Cores (when single jobs run on 64 cores)
Turbine Development: Benefits of GPFS Usage on 400G
Back to Back to Back to Back to Turbine Development Turbine Development Turbine Development Turbine Development
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 20
Chemnitz
Turbine Development Setup
1. Preprocessing @ DSI 1. Preprocessing @ DSI 1. Preprocessing @ DSI 1. Preprocessing @ DSI
2. Flow Model 2. Flow Model 2. Flow Model 2. Flow Model Calculation Calculation Calculation Calculation (Solver 1) (Solver 1) (Solver 1) (Solver 1)
3. Flow Model 3. Flow Model 3. Flow Model 3. Flow Model Calculation Calculation Calculation Calculation (Solver 2)(Solver 2)(Solver 2)(Solver 2)
4. 4. 4. 4. PostprocessingPostprocessingPostprocessingPostprocessing @ DSI @ DSI @ DSI @ DSI
Back to Back to Back to Back to Turbine Development Turbine Development Turbine Development Turbine Development
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 21
Chemnitz
Turbine Development & GPFS
Parallel Distributed Parallel Distributed Parallel Distributed Parallel Distributed
File SystemFile SystemFile SystemFile System
GPFSGPFSGPFSGPFS
Back to Back to Back to Back to Turbine Development Turbine Development Turbine Development Turbine Development
� Order 30 different Order 30 different Order 30 different Order 30 different modelsmodelsmodelsmodels areareareare usedusedusedused worldwideworldwideworldwideworldwide
� Experiments Experiments Experiments Experiments withwithwithwith thesethesethesethese modelsmodelsmodelsmodels produce10s produce10s produce10s produce10s ofofofofPBytesPBytesPBytesPBytes
� 100s 100s 100s 100s ofofofof PbytesPbytesPbytesPbytes ofofofof datadatadatadata needneedneedneed totototo bebebebe comparedcomparedcomparedcomparedbetweenbetweenbetweenbetween multiple multiple multiple multiple sitessitessitessites worldwideworldwideworldwideworldwide
� Movement Movement Movement Movement ofofofof datadatadatadata shouldshouldshouldshould bebebebe withinwithinwithinwithin monthsmonthsmonthsmonths ****
Transfer Rate
Time totransport 1 PB of Data
10 Mbps ~ 27 years
1 Gbps ~ 97 days
100 Gbps ~ 23 hours* Otherwise the questions will be forgotten ;-) Statistics taken from: „BER Network Requirements Workshop”,
LBNL report LBNL-4089E 2010, P 33. Recommended Reading
Climate Computing
ExtremelyExtremelyExtremelyExtremely High High High High BandwidthBandwidthBandwidthBandwidth RequirementsRequirementsRequirementsRequirements
‘‘‘‘VeryVeryVeryVery Big Data‘Big Data‘Big Data‘Big Data‘
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 23
Climate Computing Application Setup
Folder 1Folder 1Folder 1Folder 1CMIPCMIPCMIPCMIP
Folder 2Folder 2Folder 2Folder 2CMIPCMIPCMIPCMIP
FederationFederationFederationFederation
PreallocationPreallocationPreallocationPreallocation
Model postModel postModel postModel post----processing and processing and processing and processing and analysisanalysisanalysisanalysisVisualisiationVisualisiationVisualisiationVisualisiation @ ISC ’13 @ ISC ’13 @ ISC ’13 @ ISC ’13
LeipzigLeipzigLeipzigLeipzig
Folder Folder Folder Folder 3333CMIPCMIPCMIPCMIP
Back to Back to Back to Back to Climate Climate Climate Climate ComputingComputingComputingComputing
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 24
CCA & GPFS & iRODS
GPFS and/or GPFS and/or GPFS and/or GPFS and/or
Global Namespace Global Namespace Global Namespace Global Namespace
iRODSiRODSiRODSiRODS
Back to Back to Back to Back to Climate Climate Climate Climate ComputingComputingComputingComputing
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 25
Service Recipient Relations
Distributed FoldersDistributed FoldersDistributed FoldersDistributed Folders
Distributed Folders Distributed Folders Distributed Folders Distributed Folders
ServiceServiceServiceService RecipientRecipientRecipientRecipient
Federation, Federation, Federation, Federation, PreallocationPreallocationPreallocationPreallocation
Research ClientResearch ClientResearch ClientResearch Client
PREP & POST on PREP & POST on PREP & POST on PREP & POST on Cloud ResourcesCloud ResourcesCloud ResourcesCloud Resources
CalculationCalculationCalculationCalculation
Climate ComputingClimate ComputingClimate ComputingClimate Computing
TRACE on HPC TRACE on HPC TRACE on HPC TRACE on HPC ResourcesResourcesResourcesResources
TRACE on HPC TRACE on HPC TRACE on HPC TRACE on HPC ResourcesResourcesResourcesResources
Turbine DevelopmentTurbine DevelopmentTurbine DevelopmentTurbine Development
Client evaluating Client evaluating Client evaluating Client evaluating results, e.g. TECPLOTresults, e.g. TECPLOTresults, e.g. TECPLOTresults, e.g. TECPLOT
5T5T5T5T 5T5T5T5T
5T5T5T5T
5T5T5T5T
5T5T5T5T
5T5T5T5T
Distributed FoldersDistributed FoldersDistributed FoldersDistributed Folders 5T5T5T5T
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 26
400GBit/s Data Path
Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch400G WDM 400G WDM 400G WDM 400G WDM Super Channel Super Channel Super Channel Super Channel (4x100GbE)(4x100GbE)(4x100GbE)(4x100GbE)
LinkLinkLinkLinkSerSerSerServerververver
StorageStorageStorageStorage
GPFS GPFS GPFS GPFS FilesystemFilesystemFilesystemFilesystem
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
RouterRouterRouterRouterRouterRouterRouterRouter RouterRouterRouterRouterRouterRouterRouterRouter12x 12x 12x 12x ServerServerServerServer
12x4012x4012x4012x4012x4012x4012x4012x40GbEGbEGbEGbE
12x 12x 12x 12x ServerServerServerServer
12x4012x4012x4012x4012x4012x4012x4012x40GbEGbEGbEGbE
GBitGBitGBitGBit/s/s/s/s400 400 400 400
GBitGBitGBitGBit/s/s/s/s
36x700G 36x700G 36x700G 36x700G FlashFlashFlashFlash
36x700G 36x700G 36x700G 36x700G FlashFlashFlashFlash
∼∼∼∼7000 7000 7000 7000 corescorescorescores
IB FDR IB FDR IB FDR IB FDR NetworkNetworkNetworkNetwork
12xI
B 12
xIB
12xI
B 12
xIB
FDR
FDR
FDR
FDR
12xI
B 12
xIB
12xI
B 12
xIB
FDR
FDR
FDR
FDR
TUD Cluster
∼∼∼∼2000 2000 2000 2000 corescorescorescores
IB FDRIB FDRIB FDRIB FDR10 10 10 10 NetworkNetworkNetworkNetwork
12xI
B 12
xIB
12xI
B 12
xIB
FDR
FDR
FDR
FDR
1010 1010
12xI
B 12
xIB
12xI
B 12
xIB
FDR
FDR
FDR
FDR
1010 1010
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 27
The Big Picture
Firewall / EncryptionFirewall / EncryptionFirewall / EncryptionFirewall / Encryption
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch400G WDM 400G WDM 400G WDM 400G WDM Super Channel Super Channel Super Channel Super Channel (4x100GbE)(4x100GbE)(4x100GbE)(4x100GbE)
LinkLinkLinkLink
SerSerSerServerververver
StorageStorageStorageStorage
GPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System
17x 17x 17x 17x ServerServerServerServer
FirewallFirewallFirewallFirewall∼∼∼∼200 200 200 200 corescorescorescores
1x10 1x10 1x10 1x10 1x10 1x10 1x10 1x10 GbEGbEGbEGbE
36x700G 36x700G 36x700G 36x700G FlashFlashFlashFlash
12x 12x 12x 12x ServerServerServerServer
RouterRouterRouterRouter17x 17x 17x 17x ServerServerServerServer
17x1017x1017x1017x1017x1017x1017x1017x10GbEGbEGbEGbE
RouterRouterRouterRouter
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
DATE Cluster 1
DATE Cluster 2
12x4012x4012x4012x4012x4012x4012x4012x40GbEGbEGbEGbE
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
RouterRouterRouterRouter
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
GBitGBitGBitGBit/s/s/s/s400 400 400 400
GBitGBitGBitGBit/s/s/s/s
36x700G 36x700G 36x700G 36x700G FlashFlashFlashFlash
∼∼∼∼2000 2000 2000 2000 corescorescorescores
12x 12x 12x 12x ServerServerServerServer
12x4012x4012x4012x4012x4012x4012x4012x40GbEGbEGbEGbE
∼∼∼∼7000 7000 7000 7000 corescorescorescores
TUD Cluster RZG Cluster
IB FDRIB FDRIB FDRIB FDR10 10 10 10 NetworkNetworkNetworkNetwork
12xI
B 12
xIB
12xI
B 12
xIB
FDR
FDR
FDR
FDR
1010 1010
12xI
B 12
xIB
12xI
B 12
xIB
FDR
FDR
FDR
FDR
1010 1010
IB FDR IB FDR IB FDR IB FDR NetworkNetworkNetworkNetwork
12xI
B 12
xIB
12xI
B 12
xIB
FDR
FDR
FDR
FDR
12xI
B 12
xIB
12xI
B 12
xIB
FDR
FDR
FDR
FDR
SGI SGI SGI SGI ClusterClusterClusterCluster
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 28
ConnectionConnectionConnectionConnection RZG Router RZG Router RZG Router RZG Router –––– RZGRZGRZGRZG----WDM and WDM and WDM and WDM and Connection TUD RouterConnection TUD RouterConnection TUD RouterConnection TUD Router–––– TUD WDMTUD WDMTUD WDMTUD WDM
Type 2 x LC (100GBaseLR4)
length TUD: 10m; RZG: 10m
Volume 4 each (8 total)
7750 7750 7750 7750 SR12ESR12ESR12ESR12E7750 7750 7750 7750
SR12ESR12ESR12ESR12E
7750 7750 7750 7750 SR12SR12SR12SR127750 7750 7750 7750 SR12SR12SR12SR12
1830 1830 1830 1830 PSSPSSPSSPSS
4x10
04x
100
4x10
04x
100
GbE
GbE
GbE
GbE
4x10
04x
100
4x10
04x
100
GbE
GbE
GbE
GbE
Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch
400G WDM 400G WDM 400G WDM 400G WDM Super Channel Super Channel Super Channel Super Channel
LinkLinkLinkLinkWDM TerminalWDM TerminalWDM TerminalWDM Terminal
AmplifierAmplifierAmplifierAmplifier
1830 1830 1830 1830 PSSPSSPSSPSS
4x10
04x
100
4x10
04x
100
GbE
GbE
GbE
GbE
4x10
04x
100
4x10
04x
100
GbE
GbE
GbE
GbE
coherent Super Channel (2 x 16QAM@ 50 coherent Super Channel (2 x 16QAM@ 50 coherent Super Channel (2 x 16QAM@ 50 coherent Super Channel (2 x 16QAM@ 50 GHzGridGHzGridGHzGridGHzGrid / 2 x 200 / 2 x 200 / 2 x 200 / 2 x 200 GBitGBitGBitGBit/s) /s) /s) /s)
OLA
+ O
LA+
OLA
+ O
LA+
DG
ED
GE
DG
ED
GE
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
OLA
70km
70km
70km
70km 70
km70
km70
km70
km
70km70km70km70km 70km70km70km70km 70km70km70km70km 70km70km70km70km70km70km70km70km70km70km70km70km70km70km70km70km
400 G WDM Super Channel
Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 29
ConnectionConnectionConnectionConnection TUD Server TUD Server TUD Server TUD Server –––– TUD IBTUD IBTUD IBTUD IB----Switch Switch Switch Switch andandandandConnectionConnectionConnectionConnection RZG Server RZG Server RZG Server RZG Server –––– RZG IB SwitchRZG IB SwitchRZG IB SwitchRZG IB Switch
Type MPO (Infiniband FDR (56GBit/s))
length TUD: 10m, RZG: 10m
Volume 12 each side (24 total)
Infiniband Connections
MellanoxMellanoxMellanoxMellanox ConnectConnectConnectConnect----IB IB IB IB Volume:12on each side (24total)Volume:12on each side (24total)Volume:12on each side (24total)Volume:12on each side (24total)
Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture
MellanoxMellanoxMellanoxMellanox active Cable (incl. QSFP)active Cable (incl. QSFP)active Cable (incl. QSFP)active Cable (incl. QSFP)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 30
ConnectionConnectionConnectionConnection TUD Server TUD Server TUD Server TUD Server –––– TUD Router TUD Router TUD Router TUD Router and and and and ConnectionConnectionConnectionConnection RZG Server RZG Server RZG Server RZG Server –––– RZG RouterRZG RouterRZG RouterRZG Router
Type MPO (40GBaseSR4)
length TUD: 10m; RZG: 10m
Volume 24x10m MellanoxMellanoxMellanoxMellanox Connect X3 Connect X3 Connect X3 Connect X3 Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)
MellanoxMellanoxMellanoxMellanox active Cable (incl. QSFP)active Cable (incl. QSFP)active Cable (incl. QSFP)active Cable (incl. QSFP)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)
40GbE Connections
AlcatelAlcatelAlcatelAlcatel----Lucent 3Lucent 3Lucent 3Lucent 3----port 40GbE IMM port 40GbE IMM port 40GbE IMM port 40GbE IMM Volume: 4 @ RZG Volume: 4 @ RZG Volume: 4 @ RZG Volume: 4 @ RZG
AlcatelAlcatelAlcatelAlcatel----Lucent 6Lucent 6Lucent 6Lucent 6----port 40GbE IMM port 40GbE IMM port 40GbE IMM port 40GbE IMM (no picture) (no picture) (no picture) (no picture) Volume:2@TUDVolume:2@TUDVolume:2@TUDVolume:2@TUD
Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 31
IBM IBM IBM IBM iDataPlexiDataPlexiDataPlexiDataPlex dx360 M4 dx360 M4 dx360 M4 dx360 M4 (Volume: 12@RZG)(Volume: 12@RZG)(Volume: 12@RZG)(Volume: 12@RZG)
Bull Bull Bull Bull NovaScaleNovaScaleNovaScaleNovaScale R460 F3 R460 F3 R460 F3 R460 F3 (Volume: 12@TUD) (Volume: 12@TUD) (Volume: 12@TUD) (Volume: 12@TUD)
Server
Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 32
ConnectionConnectionConnectionConnection Router Router Router Router –––– WDM WDM WDM WDM
Type LC singelmode(100GBaseLR10)
length 10m
Volume 2
Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture
AlcatelAlcatelAlcatelAlcatel----Lucent 2Lucent 2Lucent 2Lucent 2----port 100GbE IMM port 100GbE IMM port 100GbE IMM port 100GbE IMM Volume: 3 @ TUD Volume: 3 @ TUD Volume: 3 @ TUD Volume: 3 @ TUD
AlcatelAlcatelAlcatelAlcatel----Lucent 1Lucent 1Lucent 1Lucent 1----port 100GbE IMM port 100GbE IMM port 100GbE IMM port 100GbE IMM (no picture) (no picture) (no picture) (no picture) Volume:4@RZGVolume:4@RZGVolume:4@RZGVolume:4@RZG
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 33
ConnectionConnectionConnectionConnection TUD Router TUD Router TUD Router TUD Router –––– TUD 10GbE ClusterTUD 10GbE ClusterTUD 10GbE ClusterTUD 10GbE Cluster
Type LC duplex multimode (10GBaseSR)
length ?
Volume 17
Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 34
Alcatel Lucent 7750 SR12Alcatel Lucent 7750 SR12Alcatel Lucent 7750 SR12Alcatel Lucent 7750 SR12
@RZG@RZG@RZG@RZG
Alcatel Lucent 7750 SR12 EAlcatel Lucent 7750 SR12 EAlcatel Lucent 7750 SR12 EAlcatel Lucent 7750 SR12 E
@TUD@TUD@TUD@TUD
Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 35
Alcatel Lucent 1830 PSS 32Alcatel Lucent 1830 PSS 32Alcatel Lucent 1830 PSS 32Alcatel Lucent 1830 PSS 32
Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 36
Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture
EMCEMCEMCEMC2222 XtremSFXtremSFXtremSFXtremSF 700GB SLC700GB SLC700GB SLC700GB SLC
Volume: 36 Volume: 36 Volume: 36 Volume: 36 PCIePCIePCIePCIe cards on each side (72 total)cards on each side (72 total)cards on each side (72 total)cards on each side (72 total)
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13
closingclosingclosingclosing
WSWSWSWS21.6.21.6.21.6.21.6.----????
37
Project Lifetime
Demonstrator SetupDemonstrator SetupDemonstrator SetupDemonstrator Setup Getting through Getting through Getting through Getting through Test Item List TILTest Item List TILTest Item List TILTest Item List TIL
ISCISCISCISC28.128.128.128.1---- 15.6. 15.6. 15.6. 15.6. KickoffKickoffKickoffKickoff
WSWSWSWS
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13
Press Press Press Press ReleaseReleaseReleaseRelease
Timeline Rev. C
Setup Server, Storage, Setup Server, Storage, Setup Server, Storage, Setup Server, Storage, InfinibandInfinibandInfinibandInfiniband Setup GPFSSetup GPFSSetup GPFSSetup GPFS
7.6.
7.6.
7.6.
7.6.
final final final final PerfomancePerfomancePerfomancePerfomance TestsTestsTestsTests
12.6
.12
.6.
12.6
.12
.6.
16.6
.16
.6.
16.6
.16
.6.
RfSRfSRfSRfS WDM WDM WDM WDM & IP & IP & IP & IP
Performance Performance Performance Performance Tests Tests Tests Tests donedonedonedone RfSRfSRfSRfS GPFSGPFSGPFSGPFS
Setup WDM, RouterSetup WDM, RouterSetup WDM, RouterSetup WDM, Router
DATE DATE DATE DATE
Applications Applications Applications Applications final final final final configconfigconfigconfig
RfSRfSRfSRfS Tunnel, ServerTunnel, ServerTunnel, ServerTunnel, Server
RfSRfSRfSRfSApplicationsApplicationsApplicationsApplications
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
29.5
29.5
29.5
29.5
.
no Application Tests possibleno Application Tests possibleno Application Tests possibleno Application Tests possible
GPFS optimizationGPFS optimizationGPFS optimizationGPFS optimization
CW19CW19CW19CW19 CW20CW20CW20CW20 CW21CW21CW21CW21 CW22CW22CW22CW22 CW23CW23CW23CW23 CW24CW24CW24CW24
We are hereWe are hereWe are hereWe are here
10.6
.10
.6.
10.6
.10
.6.
20.5
20.5
20.5
20.5
.
14.6
.14
.6.
14.6
.14
.6.
Stop Stop Stop Stop DATEDATEDATEDATE
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 39
Demonstrator Application Test Environment
DATE
ObjectiveObjectiveObjectiveObjective
•3 weeks between RfS IP and the begin of the ISC is by far not enough time to get two high sophisticated applications running @400G
•Therefore the support of the applications as early as possible is an integrated part of the project; the application teams get access as soon possible on new building blocks of the ‘big picture’
1. April 1. April 1. April 1. April ---- 17. May 17. May 17. May 17. May
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 40
DATE Phase 1
Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch400G WDM 400G WDM 400G WDM 400G WDM Super Channel Super Channel Super Channel Super Channel (4x100GbE)(4x100GbE)(4x100GbE)(4x100GbE)
LinkLinkLinkLink
17x 17x 17x 17x ServerServerServerServer
SerSerSerServerververver
StorageStorageStorageStorage
Firewall / EncryptionFirewall / EncryptionFirewall / EncryptionFirewall / Encryption GPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System
7750 7750 7750 7750 SR12SR12SR12SR12
17x 17x 17x 17x ServerServerServerServer
17x1017x1017x1017x1017x1017x1017x1017x10GbEGbEGbEGbE
7750 7750 7750 7750 SR12 SR12 SR12 SR12
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
DATE Cluster 1
DATE Cluster 2
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
1. April 1. April 1. April 1. April ---- 29. April 29. April 29. April 29. April
Back to Back to Back to Back to TimelineTimelineTimelineTimeline
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 41
DATE Phase 2
Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch400G WDM 400G WDM 400G WDM 400G WDM Super Channel Super Channel Super Channel Super Channel (4x100GbE)(4x100GbE)(4x100GbE)(4x100GbE)
LinkLinkLinkLink
SerSerSerServerververver
StorageStorageStorageStorage
Firewall / EncryptionFirewall / EncryptionFirewall / EncryptionFirewall / Encryption GPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System
17x 17x 17x 17x ServerServerServerServer
3xGPFS 3xGPFS 3xGPFS 3xGPFS ServerServerServerServer
7750 7750 7750 7750 SR12SR12SR12SR12
17x 17x 17x 17x ServerServerServerServer
17x1017x1017x1017x1017x1017x1017x1017x10GbEGbEGbEGbE
7750 7750 7750 7750 SR12 SR12 SR12 SR12
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
DATE Cluster 1
DATE Cluster 2
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
2xGPFS 2xGPFS 2xGPFS 2xGPFS ServerServerServerServer
2xGPFS 2xGPFS 2xGPFS 2xGPFS ServerServerServerServer
2x402x402x402x402x402x402x402x40GbEGbEGbEGbE
29. April 29. April 29. April 29. April ---- 6. May 6. May 6. May 6. May
2x402x402x402x402x402x402x402x40GbEGbEGbEGbE
Back to Back to Back to Back to TimelineTimelineTimelineTimeline
4x700G 4x700G 4x700G 4x700G FlashFlashFlashFlash
4x700G 4x700G 4x700G 4x700G FlashFlashFlashFlash
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 42
DATE Phase 3
Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch400G WDM 400G WDM 400G WDM 400G WDM Super Channel Super Channel Super Channel Super Channel (4x100GbE)(4x100GbE)(4x100GbE)(4x100GbE)
LinkLinkLinkLink
SerSerSerServerververver
StorageStorageStorageStorage
Firewall / EncryptionFirewall / EncryptionFirewall / EncryptionFirewall / Encryption GPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System
17x 17x 17x 17x ServerServerServerServer
3xGPFS 3xGPFS 3xGPFS 3xGPFS ServerServerServerServer
7750 7750 7750 7750 SR12SR12SR12SR12
17x 17x 17x 17x ServerServerServerServer
17x1017x1017x1017x1017x1017x1017x1017x10GbEGbEGbEGbE
7750 7750 7750 7750 SR12 SR12 SR12 SR12
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
DATE Cluster 1
DATE Cluster 2
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
∼∼∼∼7000 7000 7000 7000 corescorescorescores
TUD Cluster
IB FDR IB FDR IB FDR IB FDR NetworkNetworkNetworkNetwork
2xGPFS 2xGPFS 2xGPFS 2xGPFS ServerServerServerServer
2xGPFS 2xGPFS 2xGPFS 2xGPFS ServerServerServerServer
2x402x402x402x402x402x402x402x40GbEGbEGbEGbE
23xI
B 23
xIB
23xI
B 23
xIB
FDR
FDR
FDR
FDR
23xI
B 23
xIB
23xI
B 23
xIB
FDR
FDR
FDR
FDR
7. May 7. May 7. May 7. May ---- 16. May 16. May 16. May 16. May
2x402x402x402x402x402x402x402x40GbEGbEGbEGbE
Back to Back to Back to Back to TimelineTimelineTimelineTimeline
4x700G 4x700G 4x700G 4x700G FlashFlashFlashFlash
4x700G 4x700G 4x700G 4x700G FlashFlashFlashFlash
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 43
Test Items
ObjectiveObjectiveObjectiveObjective
•The Demonstrator is a great opportunity for getting valuable findings in networking , HPC, virtualization and other fields
•the project schedule just allows some synthetic performance tests and the implementation of the applications
•Therefore the partners have agreed to set up a test item list where each partner is invited to contribute proposals
Back to Back to Back to Back to Project Project Project Project LifetimeLifetimeLifetimeLifetime
21.6.21.6.21.6.21.6.----????21.6.21.6.21.6.21.6.----????
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 44
Test Item List
NFV (TSI)NFV (TSI)NFV (TSI)NFV (TSI)
Back to Back to Back to Back to AgendaAgendaAgendaAgenda
LoadbalancerLoadbalancerLoadbalancerLoadbalancer / Bundling / Performance / / Bundling / Performance / / Bundling / Performance / / Bundling / Performance / CoSCoSCoSCoS / FCAPS (T/ FCAPS (T/ FCAPS (T/ FCAPS (T----Labs) Labs) Labs) Labs)
21.6.21.6.21.6.21.6.----????21.6.21.6.21.6.21.6.----????
GPFS Network (TSIGPFS Network (TSIGPFS Network (TSIGPFS Network (TSI----SfRSfRSfRSfR) ) ) ) SDN (TUD) SDN (TUD) SDN (TUD) SDN (TUD)
RDMA over Ethernet RDMA over Ethernet RDMA over Ethernet RDMA over Ethernet
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 45
Network Functions Virtualisation (NFV)
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 46
Example NFV Use Case
ESX ESX ESX ESX ClusterClusterClusterClusterFirewallFirewallFirewallFirewall FirewallFirewallFirewallFirewall
Customer Customer Customer Customer BBBB
VLAN 2VLAN 2VLAN 2VLAN 2VLAN 1VLAN 1VLAN 1VLAN 1
Each customer configures Each customer configures Each customer configures Each customer configures his own FW entityhis own FW entityhis own FW entityhis own FW entity
FirewallFirewallFirewallFirewall
VLAN 1VLAN 1VLAN 1VLAN 1
CustomerCustomerCustomerCustomer AAAA
VLAN 1VLAN 1VLAN 1VLAN 1VLAN 2VLAN 2VLAN 2VLAN 2
VLAN 2VLAN 2VLAN 2VLAN 2VLAN 2VLAN 2VLAN 2VLAN 2
VLAN 1VLAN 1VLAN 1VLAN 1VLAN 2VLAN 2VLAN 2VLAN 2
InternetInternetInternetInternet
Router/Switch Router/Switch Router/Switch Router/Switch
Firewall VMFirewall VMFirewall VMFirewall VM
Standby Firewall VMStandby Firewall VMStandby Firewall VMStandby Firewall VM
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 47
NFV & ESX Test Setup
ESX ESX ESX ESX Cluster Cluster Cluster Cluster
ClientClientClientClient
40
40
40
40
GbE
GbE
GbE
GbE40
40
40
40
G
bEG
bEG
bEG
bE
Router/Switch Router/Switch Router/Switch Router/Switch
Firewall VMFirewall VMFirewall VMFirewall VM
Standard HW Standard HW Standard HW Standard HW (2xE5(2xE5(2xE5(2xE5----2670 + 2670 + 2670 + 2670 + 128G RAM)128G RAM)128G RAM)128G RAM)
40 40 40 40
GbE
GbE
GbE
GbE
40 40 40 40
GbE
GbE
GbE
GbE
40 40 40 40
GbE
GbE
GbE
GbE
40 40 40 40
GbE
GbE
GbE
GbE
40 40 40 40
GbE
GbE
GbE
GbE
40 40 40 40
GbE
GbE
GbE
GbE
40 40 40 40
GbE
GbE
GbE
GbE
40 40 40 40
GbE
GbE
GbE
GbE
ClientClientClientClient
40
40
40
40
GbE
GbE
GbE
GbE40
40
40
40
G
bEG
bEG
bEG
bE
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 48
Demonstrator NFV Setup
Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch LinkLinkLinkLink
SerSerSerServerververver
StorageStorageStorageStorageFirewall / Encryption / Firewall / Encryption / Firewall / Encryption / Firewall / Encryption /
CompressionCompressionCompressionCompressionGPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System
17x 17x 17x 17x ServerServerServerServer
FirewallFirewallFirewallFirewall∼∼∼∼200 200 200 200 corescorescorescores
1x10 1x10 1x10 1x10 1x10 1x10 1x10 1x10 GbEGbEGbEGbE
3xGPFS 3xGPFS 3xGPFS 3xGPFS ServerServerServerServer
7750 7750 7750 7750 SR12SR12SR12SR12
17x 17x 17x 17x ServerServerServerServer
17x1017x1017x1017x1017x1017x1017x1017x10GbEGbEGbEGbE
7750 7750 7750 7750 SR12E SR12E SR12E SR12E
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
DATE Cluster 1
DATE Cluster 2
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
SGI SGI SGI SGI ClusterClusterClusterCluster
7750 7750 7750 7750 SR12SR12SR12SR12
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
∼∼∼∼2000 2000 2000 2000 corescorescorescores
3xGPFS 3xGPFS 3xGPFS 3xGPFS ServerServerServerServer
∼∼∼∼7000 7000 7000 7000 corescorescorescores
TUD Cluster RZG Cluster
IB FDRIB FDRIB FDRIB FDR10 10 10 10 NetworkNetworkNetworkNetwork
3xIB
3x
IB
3xIB
3x
IB
FDR
FDR
FDR
FDR
1010 1010
3xIB
3x
IB
3xIB
3x
IB
FDR
FDR
FDR
FDR
1010 1010
IB FDR IB FDR IB FDR IB FDR NetworkNetworkNetworkNetwork
3xIB
FD
R3x
IB F
DR
3xIB
FD
R3x
IB F
DR
1x1001x1001x1001x1001x1001x1001x1001x100GbEGbEGbEGbE3xFEC3xFEC3xFEC3xFEC 3x403x403x403x403x403x403x403x40
GbEGbEGbEGbE3x403x403x403x403x403x403x403x40GbEGbEGbEGbE 3xFEC3xFEC3xFEC3xFEC
3x403x403x403x403x403x403x403x40GbEGbEGbEGbE
3x403x403x403x403x403x403x403x40GbEGbEGbEGbE
9x700G 9x700G 9x700G 9x700G FlashFlashFlashFlash
9x700G 9x700G 9x700G 9x700G FlashFlashFlashFlash
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 49
ScheduledScheduledScheduledScheduled for: 21.6.for: 21.6.for: 21.6.for: 21.6.----5.7.?5.7.?5.7.?5.7.?ObjectivesObjectivesObjectivesObjectives
•Is 5GByte/s Firewall / Encryption / Compression with standard hardware (2xE5-2670 & PCIe3.0) feasible?•Even hypervisored?•What is the impact for the application?
NFV Objectives & Comments
CommentsCommentsCommentsComments
•In ‘real life’ network functions like encryption, firewalling and compression can become very important , even in HPC environments•There is no need for 400GBit/s; 100GBit/s is equivalent
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 50
Loadbalancer / Bundling / Performance / CoS / FCAPS Setup
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 51
GPFS Network
17x 17x 17x 17x ServerServerServerServer
FirewallFirewallFirewallFirewall∼∼∼∼200 200 200 200 corescorescorescores
1x10 1x10 1x10 1x10 1x10 1x10 1x10 1x10 GbEGbEGbEGbE
3xGPFS 3xGPFS 3xGPFS 3xGPFS ServerServerServerServer
7750 7750 7750 7750 SR12SR12SR12SR12
17x 17x 17x 17x ServerServerServerServer
17x1017x1017x1017x1017x1017x1017x1017x10GbEGbEGbEGbE
7750 7750 7750 7750 SR12E SR12E SR12E SR12E
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
2x10
0 2x
100
2x10
0 2x
100
GbE
GbE
GbE
GbE
DATE Cluster 1
DATE Cluster 2
3x403x403x403x403x403x403x403x40GbEGbEGbEGbE
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x1
0 17
x10
17x1
0 17
x10
GbE
GbE
GbE
GbE
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk
7750 7750 7750 7750 SR12SR12SR12SR12
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
∼∼∼∼2000 2000 2000 2000 corescorescorescores
3xGPFS 3xGPFS 3xGPFS 3xGPFS ServerServerServerServer
3x403x403x403x403x403x403x403x40GbEGbEGbEGbE
∼∼∼∼7000 7000 7000 7000 corescorescorescores
TUD Cluster RZG Cluster
IB FDRIB FDRIB FDRIB FDR10 10 10 10 NetworkNetworkNetworkNetwork
3xIB
3x
IB
3xIB
3x
IB
FDR
FDR
FDR
FDR
1010 1010
3xIB
3x
IB
3xIB
3x
IB
FDR
FDR
FDR
FDR
1010 1010
IB FDR IB FDR IB FDR IB FDR NetworkNetworkNetworkNetwork
3xIB
FD
R3x
IB F
DR
3xIB
FD
R3x
IB F
DR
1x1001x1001x1001x1001x1001x1001x1001x100GbEGbEGbEGbE
3xGPFS 3xGPFS 3xGPFS 3xGPFS ServerServerServerServer
3x40
3x
40
3x40
3x
40
GbE
GbE
GbE
GbE
3x40
3x
40
3x40
3x
40
GbE
GbE
GbE
GbE
Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch
LinkLinkLinkLink
SerSerSerServerververver
StorageStorageStorageStorageFirewall / Encryption / Firewall / Encryption / Firewall / Encryption / Firewall / Encryption /
CompressionCompressionCompressionCompression
GPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System
12x700G 12x700G 12x700G 12x700G FlashFlashFlashFlash
12x700G 12x700G 12x700G 12x700G FlashFlashFlashFlash
12x700G 12x700G 12x700G 12x700G FlashFlashFlashFlash
SGI SGI SGI SGI ClusterClusterClusterCluster
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
1x10
1x
10
1x10
1x
10
GbE
GbE
GbE
GbE
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 52
ScheduledScheduledScheduledScheduled for: 21.6.for: 21.6.for: 21.6.for: 21.6.----5.7.?5.7.?5.7.?5.7.?ObjectivesObjectivesObjectivesObjectives
•Wie verhält sich GPFS,wenn die Daten nicht zwischen 2, sondern 3 Sites (Option 4 Sites, Hub be TUD) gespiegelt werden? (Server Überlast?)•Was passiert, wenn z.B eine Site mit geringerer Bandbreite angebunden ist? “Kommt das GPFS noch nach”?
Test Item GPFS Network
CommentsCommentsCommentsComments
•There is no need for 400GBit/s; 100GBit/s is equivalent
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 53
Test Item SDN
CommentsCommentsCommentsComments
* Aufsetzen einer virtuellen SDN Umgebung zwischen ZIH & RZG mittels vSwitch* Wünschenswert wäre die Einbindung von aktiven Netzwerkelementen mit OpenFlowUnterstützung (bspw. auch Barracudas SDN Gateway)* Gegenüberstellung verschiedener verfügbarer OpenFlow Controller (Beacon, Floodlight, FlowER, OpenDaylight, ...)* Zeitraum: 1 Monat, kann aber parallel zu anderen Untersuchungen laufen
Da vermutlich nicht besonders viel Zeit zur Verfügung stehen wird, glaube ich das dies bereits sehr ambitioniert ist. Ich muss dann schauen wieviele Controller aufgesetzt und getestet werden können. Aber ich würde die Umgebung so aufbauen das ich auch nach dem 400G Showcase noch damit arbeiten kann.
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 54
Test Item RDMA over Ethernet
CommentsCommentsCommentsComments
- Analyse vorhandener Protokolle für RDMA über Ethernet- Vergleich mit RDMA über InfiniBand- Performanceanalyse und -optimierung innerhalb einer 40GbE Testumgebung- Gegenüberstellung der Ergebnisse mit FDR InfiniBand
Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List
Recommended