7/31/2019 AFDS 2012 Keynote Phillip Pokorny

1/22

HPC COMPUTING,FROM THE INSIDE OUT

An Insiders Perspective Philip Pokorny, CTOPenguin Computing, Inc.

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

2/22

2 | HPC Computing, from the Inside Out | June 14, 2012

PENGUIN COMPUTING HOW DID WE GET HERE?

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

3/22

3 | HPC Computing, from the Inside Out | June 14, 2012

BEGINNING AT THE END Worlds First APU Cluster

Delivered September 2011

Collaboration among AMD, Sandia National Lab,Penguin Computing

104 Altus 2A00 nodesA8-3850 APU, 16GB of memory, SSD scratchdrive, Qlogic Infiniband

Complete solution with:Management nodes (boot, login,

storage gateways)Networking (Ethernet and Infiniband)Six racksPower distribution with per outlet switchingSerial console monitoring

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

4/22

4 | HPC Computing, from the Inside Out | June 14, 2012

Founded in 1998 by Sam Ockman

Focus on systems running Linux

Focus on reliable systems

Customized systems

First Linux 1U rackmount server

Short depth 1U

2U Storage/Virtual Tape server with sixIDE drives

Customer specified filesystem layout

Customer specified RAID configurations

HISTORY OF PENGUIN COMPUTING

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

5/225 | HPC Computing, from the Inside Out | June 14, 2012

MEMORABLE ADVERTISING CAMPAIGNS

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

6/226 | HPC Computing, from the Inside Out | June 14, 2012

AMD USHERS IN THE 64-BIT PC ERA

AMD Opteron Processor AMD64 architecture64-bit registers (and more of them)

Orthogonal instruction setIntegrated memory controllerHyperTransport Interconnect

Linux64-bit ready and tested

Altus 1000eDual socket systemFan control and monitoringTemperature sensor driver(written and contributed to kernel)Penguin Remote Control

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

7/227 | HPC Computing, from the Inside Out | June 14, 2012

FROM SERVERS TO CLUSTERS

Large orders for Altus 1000eBeowulf style clusters

Scyld SoftwareDon BeckerMerged with Penguin Computing in 2003

Complete clusters with all the accessoriesRacks, UPS, keyboard & monitorNetworks, cabling, management

Factory integrated and testedSoftware installed and configuredTested as a complete clusterShipped in a crateReady to plug into network and power on-site

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

8/228 | HPC Computing, from the Inside Out | June 14, 2012

FROM THE INSIDE OUT

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

9/229 | HPC Computing, from the Inside Out | June 14, 2012

Dual Socket AMD Opteron Processor Memory Configurations

CPU CORES AND MEMORY SIZES OVER TIME

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

10/22

10 | HPC Computing, from the Inside Out | June 14, 2012

Dual Socket AMD Opteron Processor Memory Configurations

MEMORY PER CORE OVER TIME

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

11/22

11 | HPC Computing, from the Inside Out | June 14, 2012

Dual Socket AMD Opteron Processor Memory Configurations

WHAT'S THE TREND?

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

12/22

12 | HPC Computing, from the Inside Out | June 14, 2012

Dual Socket AMD Opteron Processor Memory Configurations

LONG-TERM TRENDS IN CPU AND MEMORY

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

13/22

13 | HPC Computing, from the Inside Out | June 14, 2012

MEMORY BANDWIDTH OVER TIME

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

14/22

14 | HPC Computing, from the Inside Out | June 14, 2012

THOUGHTS ON THE FUTURE

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

15/22

15 | HPC Computing, from the Inside Out | June 14, 2012

STORAGE PERFORMANCE

7200 RPM drives from 2002 to 2012

80 GB to 3000 GB

40 MB/sec to 160 MB/sec

Lots more space, not much more performance

RAID

Less than perfect scaling

Reliability level impacts performance

Motherboard SATA ports limit number of drives

Solid State Drives

Limited writes

May need ATA TRIM for better results

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

16/22

16 | HPC Computing, from the Inside Out | June 14, 2012

STORAGE PERFORMANCE

SATA really? Compatible with 1985 MFM drive controller!

Assumes compatibility at register level

Butquirks to every controller One drive per physical portHot swap added onOops, forgot the drive activity signal

SASHigher performance drives (15,000 RPM)Flexible number of drivers per controllerCan host SATA drivesMultiple ports for redundancy and performance

Ask for it

IS THAT A GPU IN YOUR SOCKET?

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

17/22

17 | HPC Computing, from the Inside Out | June 14, 2012

IS THAT A GPU IN YOUR SOCKET?

Mixed number of CPU/GPU

Socket compatible FPGAs Spansion flash controller and NOR flash DIMMs

Just need the hardware

POWER EFFICIENCY AND IDEAS FOR IMPROVEMENT

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

18/22

18 | HPC Computing, from the Inside Out | June 14, 2012

POWER EFFICIENCY AND IDEAS FOR IMPROVEMENT

CPUs well-covered elsewhere

MotherboardsMore efficient voltage converters (VRM)Better VRM tuningGenerate voltages only where needed

Power supplyHigher efficiency

Beware of hidden power distribution boards

Single rail (one voltage, not five or more)

Common infrastructure12.5V distributionShared fansMatch power supplies to server load

WHERE DOES THE POWER GO?

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

19/22

19 | HPC Computing, from the Inside Out | June 14, 2012

WHERE DOES THE POWER GO?

You cant control what you cant measure

Need to compare multiple systems

Need high-speed capture for some analysis

Diagnose problems

Spot trends

CPU versus motherboard versus anything else

Monitor each voltage rail individuallyCPU, motherboard, fans, hard drives

High speed (1000 samples per second or faster)

Correlate samples to code execution

COMING SOON

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

20/22

20 | HPC Computing, from the Inside Out | June 14, 2012

COMING SOON

A glimpse of what we might learn

CPU power draw during make j4

Different phases of compilation

Clearly different power levels

Larger files can load CPU 100% forlong periods of time

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

21/22

21 | HPC Computing, from the Inside Out | June 14, 2012

THANK YOU QUESTIONS?

7/31/2019 AFDS 2012 Keynote Phillip Pokorny

22/22