1

Homa Karimabadi 1UCSD, La Jolla, CA

Hybrid Code: H. X. Vu2, Y. A. Omelchenko1 , M. Tatineni3, A. Majumdar3, U. Catalyurek4

Fully Kinetic Simulations:W. Daughton5, V. Roytershteyn1

Scientific Visualization, Data Mining and Computer Vision:B. Loring8, T. Sipes2, A. Yilmaz9

MHD Simulations:J. Dorelli6, J. Raeder7

2SciberQuest, Inc., Del Mar, CA 3SDSC, La Jolla, CA 4Ohio State Univ. 5Los Alamos National Lab. 6NASA Goddard Space Flight Center 7Univ. Of New Hampshire 8Lawrence Berkeley National Lab. 9Ohio State University

SIAM Conference on Parallel Processing for Scientific Computing February 15-17, 2012

Computations:NSF’s Kraken – G. Brook / Y. MackDOE’s Jaguar – J. WellsNASA’s Pleiades – J. Chang / M. CaryDOE’s Hopper – F. Verdier

Data Analysis:Black light – J. WellingLonghornNautilus / Kraken – RDAV support has been criticalLENS

What is space plasma physics?plasma = ionized gas examples of plasmas: stars, neon signs, plasma TV

Computational challenge in space sciences

Codes & simulation characteristics

New science enabled through petascale simulations

Remaining bottlenecks

490 million miles or ~ 100 Suns

Video: Animation of Coronal Mass Ejection (CME)A burst of fast material from the sun generates magnetic reconnection events in Earth's magnetic field. This eventuallysends high-speed electrons and protons into Earth's upper atmosphere to form aurorae. Credit: NASA/Goddard SpaceFlight Center Conceptual Image Lab

Earth is embedded in the Sun’s atmosphereEarth’s dipole field shields us for the most part from the effects of the SunBut a process called magnetic reconnection fractures

the Earth’s protective shield, exposing us to the effects of solar activity

Space weather affects the Earth and its technological systems

- Over $4 billion in satellite losses

- Damage sensitive electronics on orbiting spacecraft

- Cause colorful auroras, often seen in the higher latitudes

- Create blackouts on Earth when they cause surges in power grids

Travels at 20km/s to 3200km/s

Can take 1‐5 days to reach the Earth

During solar maxima can have 3 / day

During solar mimina can have 1 every 5 days

Release terawatt of power

8

Sun is about 109 times larger than Earth in diameter and is about 330 times more massive.

Distance of Sun from Earth is ~90 million miles ~ 100 Suns.

Earth weighs about ~ 1.3x1025 pounds.

Solar Wind Inflow Boundary Condition

Simulation Domain

120 x 30 x 30 RE3

20 RE 100 RE

1 RE =130 c/ωpΩ-1=0.5sec

1 RE =6 c/ωpΩ-1=1.5sec

9

Multi-Scale Coupled SystemSpatial scales vary from centimeters to 200 RE (span of 1011 spatial scales!)

Temporal scales vary from less than milliseconds to days (span of 108 temporal scales!)

Kinetic effects have global consequences Requires Particle Simulations

Multi-physicselectron physics: e.g., controls reconnection rate

ion physics: e.g., dominates formation of boundaries and transport

global features and dynamics: e.g., magnetotail/energetic particles

coupling to the ionosphere

Requires yottaflops (1024) and beyond

Even with infinite computational power, numerical round off would swamp the results

MHD (single fluid)used extensively

does not resolve important ion physics

does not correctly capture the physics of reconnection

not suitable for studies of boundaries & discontinuities

Petascale computing has enabled:Hybrid (fluid electrons, kinetic ions)

fluid electrons, kinetic ions

it is the next stage of advance in 3D global simulations

resolves ion spatial scales (ion inertial length) and ion temporal scales (gyroperiod)

Full Particle (kinetic electrons, kinetic ions)it is the most complete description

only 2D is possible

Our largest simulation involves 15 billion cells and 3.2 trillion particles

NonuniformMesh + Multi‐time zones

Discrete Event Technique- robust (stable) and efficient (no idle computation)- works for arbitrary meshes- Predict local Δt for each variable f (“state”) based on its estimated

trajectory, f=fE(t) and a given ΔfE (selected based on local CFL/reactionconditions)

13

Hybrid Code

2D global for capturing the microphysics of reconnection in a global setting

Fully Kinetic Code

14

• Ions: kinetic particles

• Electrons: massless, quasi‐neutral (ene = qini) fluid

• Electromagnetic fields:

‐ Faraday’s law

(∂ /∂ t) B = ‐c ∇ × E

‐ Ampere’s law

∇ × B = 4 π J /c = 4π qini (vi ‐ve) /c

‐ Electric field from electron momentum equation

E = ‐vi × B /c ‐∇pe /(qini)‐B × (∇ × B) /(4π qini)‐ηJ

• Stretched mesh

• Temporal zones15

16

Omelchenko and Karimabadi, JCP, 2011

Large I/O (over 200 TB from a single run)

Memory limited

Expensive checkpointing

“Noisy”, 3D multi‐variate data

Long integration times

17

19

Teraflop: Billion Particles

Petaflop: Trillion Particles

Daughton et al., Nature Physics, 2011

20

Daughton et al., Nature Physics, 2011

Our largest simulation with over 3 trillion particles

24

25

2D: Teraflop 3D: Petaflop

31

Petascale computing is leading to breakthrough science in space sciencesBottlenecks to progress are:

‐ Visualization of multi‐variate, “noisy”, 3D data‐ Sharing of data sets‐ Archiving‐ I/O‐ Disk space‐ Allocation time on supercomputers‐ Algorithmic issues