Lecture 18 - The Future - MIT CSAILgroups.csail.mit.edu/cag/ps3/lectures/6.189-lecture18-future.pdfLecture 18 The Future. ... Visual FoxPro ... Synchronization follows monitor and

Prof. Saman Amarasinghe, MIT. 1 6.189 IAP 2007 MIT

6.189 IAP 2007

Lecture 18

The Future

2 6.189 IAP 2007 MITProf. Saman Amarasinghe, MIT.

Predicting the Future is Always Risky

● "I think there is a world market for maybe five computers.“

– Thomas Watson, chairman of IBM, 1949

● "There is no reason in the world anyone would want a computer in their home. No reason.”

– Ken Olsen, Chairman, DEC, 1977

● "640K of RAM ought to be enough for anybody.”

– Bill Gates, 1981


Future = Evolution + Revolution

● EvolutionRelatively easy to predictExtrapolate the trends

● Revolution A completely new technology or solutionHard to Predict

● Paradigm Shifts can occur in both


Outline

● EvolutionTrendsArchitectureLanguages, Compilers and Tools

● Revolution● Crossing the Abstraction Boundaries


Evolution

● Look at the trendsMoore’s LawPower ConsumptionWire DelayHardware ComplexityParallelizing CompilersProgram Design Methodologies

● Design Drivers are different in Different Generations


1

10

100

1000

10000

100000

1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

Perfo

rman

ce (v

s. V

AX-

11/7

80)

25%/year

52%/year

??%/year

8086

286

386

486

PentiumP2

P3P4

ItaniumItanium 2

The March to Multicore:Moore’s Law

From David Patterson

1,000,000,000

100,000

10,000

1,000,000

10,000,000

100,000,000

From Hennessy and Patterson, Computer Architecture: A Quantitative Approach, 4th edition, 2006

Num

ber of Transistors


The March to Multicore:Uniprocessor Performance (SPECint)

Specint2000

1.00

10.00

100.00

1000.00

10000.00

85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07

i ntel 386

i ntel 486

i ntel pent i um

i ntel pent i um 2

i ntel pent i um 3i ntel pent i um 4

i ntel i tani um

A l pha 21064

A l pha 21164

A l pha 21264

Spar c

Super Spar c

Spar c64

M i ps

HP PAPower PC

AM D K6

AM D K7

AM D x86-64


Power Consumption (watts)

Power

1

10

100

1000

85 87 89 91 93 95 97 99 01 03 05 07

intel 386

intel 486

intel pentium

intel pentium 2

intel pentium 3

intel pentium 4

intel i tanium

Alpha 21064

Alpha 21164

Alpha 21264

Spar c

Super Spar c

Spar c64

Mips

HP PA

Power PC

AMD K6

AMD K7

AMD x86-64


Power Efficiency (watts/spec)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1982 1984 1987 1990 1993 1995 1998 2001 2004 2006

Year

Wat

ts/S

pec

intel 386intel 486intel pentiumintel pentium 2intel pentium 3intel pentium 4intel itaniumAlpha 21064Alpha 21164Alpha 21264SparcSuperSparcSparc64M ipsHP PAPower PCAM D K6AM D K7AM D x86-64


Range of a Wire in One Clock Cycle

00.020.040.060.080.1

0.120.140.160.180.2

0.220.240.260.28

1996 1998 2000 2002 2004 2006 2008 2010 2012 2014Year

Pro

cess

(mic

rons

)

700 MHz

1.25 GHz

2.1 GHz

6 GHz10 GHz

13.5 GHz

• 400 mm2 Die• From the SIA Roadmap


DRAM Access Latency

1

100

10000

1000000

1980

1982

1984

1986

1988

1990

1992

1994

1996

1998

2000

2002

2004

Year

Perf

orm

ance

µProc60%/yr.

(2X/1.5yr)

DRAM9%/yr.

(2X/10 yrs)


Improvement in Automatic Parallelization

1970 1980 1990 2000 2010

Vectorizationtechnology

Automatic Parallelizing

Compilers forFORTRAN

Prevalence of type unsafe languages and

complex data structures (C, C++)

Typesafelanguages (Java, C#)

Demand driven by

Multicores?Compiling for Instruction

Level Parallelism


1985 199019801970 1975 1995 2000 2005

Raw

Power4Opteron

Power6

Niagara

YonahPExtreme

Tanglewood

Cell

IntelTflops

Xbox360

CaviumOcteon

RazaXLR

PA-8800

CiscoCSR-1

PicochipPC102

Boardcom 1480

20??

# ofcores

1

2

4

8

16

32

64

128256

512

Opteron 4PXeon MP

AmbricAM2045

Multicores are here

4004

8008

80868080 286 386 486 Pentium P2 P3P4Itanium

Itanium 2Athlon


Outline




Novel Opportunities in Multicores

● Don’t have to contend with uniprocessorsThe era of Moore’s Law induced performance gains is over!Parallel programming will be required by the masses– not just a few supercomputer super-users


Novel Opportunities in Multicores

● Don’t have to contend with uniprocessorsThe era of Moore’s Law induced performance gains is over!Parallel programming will be required by the masses– not just a few supercomputer super-users

● Not your same old multiprocessor problemHow does going from Multiprocessors to Multicores impact programs?What changed?Where is the Impact?– Communication Bandwidth– Communication Latency


Communication Bandwidth

● How much data can be communicated between two cores?

● What changed?Number of Wires– IO is the true bottleneck– On-chip wire density is very highClock rate– IO is slower than on-chipMultiplexing – No sharing of pins

● Impact on programming model?Massive data exchange is possibleData movement is not the bottleneck

processor affinity not that important

32 Giga bits/sec ~300 Tera bits/sec

10,000X


Communication Latency

● How long does it take for a round trip communication?

● What changed?Length of wire– Very short wires are faster

Pipeline stages– No multiplexing – On-chip is much closer– Bypass and Speculation?

● Impact on programming model?Ultra-fast synchronizationCan run real-time apps on multiple cores

50X

~200 Cycles ~4 cycles


Past, Present and the Future?

PE PE

$$ $$

Memory

PE PE

$$

Memory

$$

PE

$$ X

PE

$$ X

PE

$$ X

PE

$$ X

MemoryMemory

Mem

ory

Mem

ory

Basic MulticoreIBM Power5

Traditional Multiprocessor

Integrated Multicore16 Tile MIT Raw


Outline




The OO Revolution

● Object Oriented revolution did not come out of a vacuum

● Hundreds of small experimental languages

● Rely on lessons learned from lesser-known languagesC++ grew out of C, Simula, and other languagesJava grew out of C++, Eiffel, SmallTalk, Objective C, and Cedar/Mesa1

● Depend on results from research community

J. Gosling, H. McGilton, The Java Language Enviornment1


Object Oriented Languages

● Ada 95● BETA● Boo● C++● C#● ColdFusion● Common Lisp● COOL (Object

Oriented COBOL)● CorbaScript● Clarion● Corn● D● Dylan● Eiffel● F-Script● Fortran 2003● Gambas● Graphtalk● IDLscript● incr Tcl● J● JADE

● Java● Lasso● Lava● Lexico● Lingo● Modula-2● Modula-3● Moto● Nemerle● Nuva● NetRexx● Nuva● Oberon (Oberon-1)● Object REXX● Objective-C● Objective Caml● Object Pascal

(Delphi)● Oz● Perl 5● PHP● Pliant● PRM● PowerBuilder

● ABCL● Python● REALbasic● Revolution● Ruby● Scala● Simula● Smalltalk● Self● Squeak● Squirrel● STOOP (Tcl

extension)● Superx++● TADS● Ubercode● Visual Basic● Visual FoxPro● Visual Prolog● Tcl● ZZT-oop

Source: Wikipedia


Language EvolutionFrom FORTRAN to a few present day languages

Source: Eric Levenez


Origins of C++

Source: B. Stroustrup, The Design and Evolution of C++

1960

1970

1980

1990

Structural influenceFeature influence

FortranAlgol 60

CPL

BCPL

C

ANSI C

Simula 67

C with Classes

C++

C++arm

C++std

ML CluAlgol 68

Ada


Academic Influence on C++

“Exceptions were considered in the original design of C++, butwere postponed because there wasn't time to do a thorough job ofexploring the design and implementation issues.

In retrospect, the greatest influence on the C++ exceptionhandling design was the work on fault-tolerant systems started at the University of Newcastle in England by Brian Randell and his colleagues and continued in many places since.”

-- B. Stroustrup, A History of C++

…


Origins of Java● Java grew out of C++, Eiffel, SmallTalk, Objective C, and Cedar/Mesa● Example lessons learned:

Stumbling blocks of C++ removed (multiple inheritance, preprocessing, operator overloading, automatic coercion, etc.)Pointers removed based on studies of bug injectionGOTO removed based on studies of usage patternsObjects based on Eiffel, SmallTalkJava interfaces based on Objective C protocolsSynchronization follows monitor and condition variable paradigm (introduced by Hoare, implemented in Cedar/Mesa)Bytecode approach validated as early as UCSD P-System (‘70s)

Lesser-known precursors essential to Java’s success

Source: J. Gosling, H. McGilton, The Java Language Enviornment


Why New Programming Models and Languages?● Paradigm shift in architecture

From sequential to multicoreNeed a new “common machine language”

● New application domainsStreamingScriptingEvent-driven (real-time)

● New hardware featuresTransactions IntrospectionScalar Operand Networks or Core-to-core DMA

● New customersMobile devicesThe average programmer!

● Can we achieve parallelism without burdening the programmer?


Domain Specific Languages

● There is no single programming domain!Many programs don’t fit the OO model (ex: scripting and streaming)

● Need to identify new programming models/domainsDevelop domain specific end-to-end systemsDevelop languages, tools, applications ⇒ a body of knowledge

● Stitching multiple domains together is a hard problemA central concept in one domain may not exist in another– Shared memory is critical for transactions, but not available in

streaming Need conceptually simple and formally rigorous interfaces Need integrated toolsBut critical for many applications


programmability

domain specificoptimizations

enable parallelexecution

simple and effective optimizations for domain specific abstractions

boost productivity, enable faster development and rapid prototyping

Compiler-Aware Language Design: StreamIt Experience

● Some programming models are inherently concurrentCoding them using a sequential language is…– Harder than using the right parallel abstraction – All information on inherent parallelism is lost

● There are win-win situations Increasing the programmer productivity while extracting parallelperformance

target tiled architectures, clusters, DSPs, multicores, graphics processors,

…


0123456789

10111213141516171819

1 2 3 4 5 6 7 8 9 10 11 12 13

Benchmarks

Thro

ughp

ut N

orm

aliz

ed to

Sin

gle

Cor

e St

ream

I

StreamIt Performance on Raw

GeoMean


Parallelizing Compilers: SUIF Experience

● Automatic Parallelism is not impossibleCan work well in many domains (example: ILP)

● Automatic Parallelism for multiprocessors “almost” worked in the ‘90sSUIF compiler got the Best SPEC results by automatic parallelization

● But…The compilers were not robustClients were impossible (performance at any cost)Multiprocessor communication was expensive Had to compete with improvements in sequential performanceThe Dogfooding problem

● Today: Programs are even harder to analyzeComplex data structuresComplex control flowComplex build process Aliasing problem (type unsafe languages)


Compilers

● Compilers are critical in reducing the burden on programmers

Identification of data parallel loops can be easily automated, but many current systems (Brook, PeakStream) require the programmer to do it.

● Need to revive the push for automatic parallelizationBest case: totally automated parallelization hidden from the userWorst case: simplify the task of the programmer


Tools

● A lot of progress in tools to improve programmer productivity

● Need tools toIdentify parallelismDebug parallel codeUpdate and maintain parallel codeStitch multiple domains together

● Need an “Eclipse platform for multicores”


Facilitate Evaluation and Feedback for Rapid Evolution

Language/Compiler/ToolsIdea

Implementation

Evaluation

Evaluation

Develop aProgram

FunctionalDebugging

PerformanceDebuggingEvaluate


The Dogfooding ProblemCAD Tools vs. OO Languages

● CAD ToolsUniversally hated by the usersOnly a few can hack itVery painful to use

● OriginsDeveloped by CAD expertsUser community is separate

● Object Oriented LanguagesUser friendly Universal acceptance Use by ordinary programmersHuge improvements in programmer productivity

● OriginsDeveloped by PL expertsThe compiler is always written using the language/toolsRapid feedback

● High Performance LanguagesUser community is separateHard to get feedbackSlow evolution


Rapid Evaluation

● Extremely hard to getReal users have no interest in flaky toolsHard to quantify Superficial users vs. Deep users will give different feedback– Fatal flaws as well as amazing uses may not come out immediately

● Need a huge, sophisticated (and expensive) infrastructure How to get a lot of application experts to use the system? How do you get them to become an expert?How do you get them to use it for a long time?How do you scientifically evaluate?How go you get actionable feedback?

● A “Center for Evaluating Multicore Programming Environments”??


Identify, Collect, Standardize, Adopt

● Good languages/tools cannot be designed by committee

● However, you need a vibrant ecosystem of ideas

● Need a process of natural selection Quantify Productivity and Performance Competition between multiple teamsWinner(s) get to design the final language


Migrate the Dusty Deck

● Impossible to bring them to the new era automatically Badly mangled, hand-optimized, impossible to analyze codeAutomatic compilation, even with a heroic effort, cannot do anything

● Help rewrite the huge stack of dusty deckApplication in useSource code availableProgrammer long gone

● Getting the new program to have the same behavior is hard“Word pagination problem”

● Can take advantage of many recent advancesCreating test casesExtracting invariants Failure oblivious computing


Outline




How about Revolutions?

● What are the far-out technologies?● Wishful Thinking?


Outline




Computer Systems from 10,000 feetfoo(int x){ .. }

class ofcomputation

convenientphysical phenomenon

… we use abstractions to make this easier


The Abstraction Layers Make This Easier

foo(int x) { .. }

ComputationLanguage / APICompiler / OSISAMicro ArchitectureLayoutDesign StyleDesign RulesProcessMaterials SciencePhysics

IBM 360/RISC/TransmetaFortran

Mead & Conway


A Case Against Entrenched Abstractions

foo(int x) { .. }

ComputationLanguage / APICompiler / OSISAMicro ArchitectureLayoutDesign StyleDesign RulesProcessMaterials SciencePhysics

Documents

Lecture 18 - The Future - MIT CSAILgroups.csail.mit.edu/cag/ps3/lectures/6.189-lecture18-future.pdfLecture 18 The Future. ... Visual FoxPro ... Synchronization follows monitor and