Upload
gridrebel
View
323
Download
2
Embed Size (px)
DESCRIPTION
Invited talk at workshop "Exascale Computing in Astrophysics" held in Ascona, Switzerland, 8-13 September 2013. http://www.itp.uzh.ch/exastro2013/Home.html
Citation preview
The Einstein Toolkit: A Community Computational Infrastructure for Relativistic Astrophysics
Gabrielle AllenSkolkovo Institute of Science & TechnologyMoscow, Russia
http://www.einsteintoolkit.org
Gravitational Wave Physics
Observations
Models & Simulation
Theory
Scientific Discovery!
G = 8TCompact binaries, supernovae collapse, gamma-ray bursts, oscillating NSs, gravitational waves, …
Basic Formalism: ADM1. Choose initial spacelike surface
and provide initial data (3-metric, extrinsic curvature)
2. Choose coordinates: Construct timelike unit normal to
surface, choose lapse function Choose time axis at each point on
next surface (shift vector)
3. Evolve 3-metric, extrinsic curvature
Usual numerical methods:
Structured meshes (including multi-patch), finite differences (finite volumes for matter), adaptive mesh refinement (since ~2003). High order methods.
Some groups use high accuracy spectral methods for vacuum space times
Black Holes & Vacuum Spacetimes 40+ year effort to model … big
advance in 2005
Now: Accurate waveforms for arbitrary masses, spins, momentum
Numerically generated waveforms now used with gravitational wave data analysis, analytic GR (NINJA, NR-AR)
B.Aylott et al 2009 Class. Quantum Grav. 26 165008
• Opens the door to general relativistic hydrodynamics
New Frontiers: Relativistic Matter General relativity
Nuclear equations of state
Relativistic magnetohydrodynamics (GRMHD)
Radiation Transport (neutrinos/photons) Expensive and complicated! Requires opacities/emissivities
Chemical reactions (thermonuclear, chemical)
Computation: Multiphysics!! GRMHD: petascale problem Radiation transport beyond this
Resolve 102m to 1010m, 500 grid variablesSchnetter et al, PetaScale Computing:
Algorithms and Applications, 2007
Software
Ecosystem
Software: Component Framework
Einstein Toolkit
Cactus Computational Toolkit
Cactus Flesh(APIs and Definitions)
MPI, Threads, New Programming Models
Driver Thorns (Parallelisation)
Group A Thorns
Group B Thorns
CS
CDSE
Computational
Relativists
Domain Scientist
s
Cactus
Framework
Software Platform
Cactus: www.CactusCode.org Open source component framework for HPC
Modular system with high level abstractions Components (“thorns”) defined by parameters, variables,
methods Cactus “flesh” binds together Cactus Computational Toolkit: general thorns
Different application areas Numerical relativity, CFD, coastal science, petroleum,
quantum gravity, cosmology, …
Key Features
Driver thorn provides scheduling, load balancing, parallelization
Application thorns deal only with local part of parallel mesh
Different thorns can be used to provide the same functionality, easily swapped.
Cactus: 1997-today
History: Black Hole Grand Challenge (‘94-’98): multiple codes,
groups trying to collaborate, tech/social challenges, NCSA (USA) group moves to AEI (Germany).
New software needed! Vision … Modular for easy code reuse, community sharing and
development of code Highly portable and flexible to take advantage of new
architectures and technologies (grid computing, networks)
Higher level programming than “MPI”: abstractions Emerging: general to support other applications, better
general code, shared infrastructure
AMR: Carpet Set of Cactus thorns
Developed by Erik Schnetter
Berger-Oliger style adaptive mesh refinement with sub-cycling in time High order differencing
(4,6,8) Domain decomposition Hybrid MPI-OpenMP
2002-03: Design of Cactus means many groups, even competing ones, suddenly had AMR with little code change
AEI (Rezzolla, Kaehler)
E. Schnetter
Carpet Scaling
Schnetter, 2013, arXiv:1308.1343
Numerical Relativity with Cactus 1997: 1st version of Cactus just for relativity
(Funding from MPG/NCSA)
1999: Cactus 4.0: “Cactus Einstein” thorns
1999-2002: EU Network “Sources of Gravitational Waves” Led to Whisky Code for GR Hydro in Cactus
Groups develop codes based on Cactus Einstein
2007: LSU/RIT/PennState/GeorgiaTech: NSF XiRel Improve scaling for multiple codes using Cactus
2009-: LSU/RIT/GeorgiaTech/Caltech/AEI: NSF CIGR Shared cyberinfrastructure including matter Einstein Toolkit from community contributions Sustainable, community supported model
So far …
Over 200 science papers
Over 30 student thesis
Around 50
contributors
14
Coastal Science: CaFUNWAVEPorted from a mature community code:
FUNWAVE (phase-resolving, time-stepping
Boussinesq model for ocean surface
wave propagation in the nearshore)
Now uses total variation diminishing method and
shock capturing scheme
J. Chen (Civil Eng), J. Tao (CCT/LSU), S.Brandt
(CCT/LSU), F. Shi (Delaware)
General applications for CactusCareer implications for staffIncreased funding opportunitiesImproved code base and communityMore support apparatus needed!
Einstein Toolkit
Community
Einstein Toolkit
“The Einstein Toolkit Consortium is developing and supporting open software for relativistic astrophysics. Our aim is to provide the core computational tools that can enable new science, broaden our community, facilitate interdisciplinary research and take advantage of emerging petascale computers and advanced cyberinfrastructure.”
Consortium: 94 members, 49 sites, 14 countries
Sustainable community model: 9 Maintainers from 6 sites: oversee technical developments,
quality control, verification and validation, distributions
and releases Whole consortium engaged in directions, support, development Open development meetings Governance model: still being discussed (looking at CIG, iPlant)
http://www.einsteintoolkit.org
Einstein Toolkit Members
Components
Currently 150 Cactus thorns: Initial data, evolution, analysis, AMR, … Tools, viz, etc Provide extensible standard interface for general relativity
variables (e.g. variables, parameters, data model for output)
Software contributions from other groups More than Cactus
Examples and tutorials Complete production codes for black holes, neutron stars New users: Test account on LONI supercomputer (Louisiana)
Community support
http://www.einsteintoolkit.org
Groups Concentrate on New Science
Einstein Toolkit
Cactus Computational Toolkit
Cactus Flesh(APIs and Definitions)
MPI, Threads, New Programming Models
Driver Thorns (Parallelisation)
Group A Thorns
Group B Thorns
CS
CDSE
Computational
Relativists
Domain Scientist
s
Example: Codes in Analytical Relativity Collaboration (NRAR)
6 out of 9 codes using Einstein Toolkit
Hinder et al (2013), arXiv:1307.5307
Issues
Incentives for software sharing and reuse
Credit for software (and data) as part of e.g. promotion and tenure
How to cite use of Einstein Toolkit in publications and ensure that contributions are cited?
http://www.einsteintoolkit.org 22
Education
Undergraduate Research (LSU REU in Computational Science) Undergraduates can do do
sophisticated AMR simulations on parallel machines with Einstein Toolkit
Graduate Scientific Computing Course at LSU Einstein Toolkit used in
teaching real world simulation related scientific computing concepts
CS, ECE, Civil Eng, Geoscience, Mech Eng
3D Full GR Simulations of Core-Collapse Supernovae
with the Einstein Toolkit
• Multi-Physics, Multi-Scale Simulations ongeneralized cubed-sphere grids with AMR.
• Scaling: time-evolving GR, no elliptic PDEs -> full physics sims scale strongly to O(10000)cores with hybrid OpenMP/MPI.
• Monte-Carlo Boltzmann solver &and efficient M1 radiation-hydro schemeunder development.
• MHD implementation (Moesta et al. 2013)-> study GRB central engines.
Ott et al. 2013, ApJ
Jet-Driven Supernovaewith the Einstein Toolkit
• Full GR -> get special relativity for free.
• GRMHD with full microphysics (EOS, neutrinos) from http://stellarcollapse.org
• Full adaptivity; shock/jet tracking framework.
• Fully reproducible thanks to open source.
Moesta et al. 2013 in prep.
Supermassive Stars and Black Holeswith the Einstein Toolkit
• Making massive seed black holes from supermassive stars:Simulation of fragmentation of differentially rotating polytrope.
• Implementation is full GR:simulate black hole formation self-consistently.
• Get gravitational-waves for “free” from spacetime metric -> probe formation of the first black holes.
Reisswig et al. 2013, PRL
New: GRHydro ET Thorn Base: GRHD public version of Whisky code (EU 5th Framework)
Much development plus new MHD Caltech, LSU, AEI, GATECH, Perimeter, RIT (NSF CIGR Award)
Full 3D and dynamic general relativity
Valencia formalism of GRMHD: Relativistic magnetized fluids in ideal MHD limit
Published text results, convergence arXiv: 1304.5544 (Moesta et al, 2013)
All code, input files etc part of Einstein Toolkit
User support
StrategyExascale
Automatic Code Generation
Einstein equations very complex Coding cumbersome, error prone Deters experimentation
Kranc: Mathematica tool to generate Cactus thorns from PDEs, specify differencing methods
Vision: Generate entire codes from underlying equations/problem specification, optimize codes for target architectures Revolutionize HPC Opportunity to integrate
verification/validation/data description
Governing
Equations
Numerics
KrancEngine
Custom Code
Resource
Chimora Use large scale CPU/GPU systems efficiently for
complex applications
Reduce code rewrite, new programming paradigms
Strategy uses: High level code transformations Loop traversal strategies Dynamically selected data/instruction cache JIT compiler tailored to application
Blazewicz et al (2013), arXiv:1307.6488
1265449
Chimora
Software Partnerships NSF S2I2 Software
Institutes Conceptualization awards
for institute level software development and support
Software Institute for Abstractions and Methodologies for HPC Simulation Codes on Future Architectures (Anshu Dubey)
High-Performance Computational Science with Structured Meshes and Particles (Phil Colella)
Katz, NSF
Some Challenges for Petascale and Beyond
New physics: neutrino transport, photon radiation transport
Massive scalability Local metadata, remove global operations Extend Cactus abstractions for new programming models Robust automatically generated code Multithreading, accelerators
Tools: real time debuggers, profilers, more intelligent application-specific tools
Data, visualization, profiling tools, debugging tools, tools to run codes, archive results, …
Growing complexity of application, programming models, architectures.
Social: how to develop sustainable software for astrophysics? CDSE and supporting career paths? Edcuation?
Conclusions Numerical relativity community generally now comfortable with
sharing software Didn’t happen overnight Some fundamental issues resolved first (BH-BH evolutions) Some trade-offs, flexibility/support
Einstein Toolkit approach Mechanism for injecting new science (e.g. GRHydro) and taking full
benefit of new CS opportunities Need to focus on implications for young researchers, motivation to
contribute, scientific aims Focus on modularity/abstractions reduces dependence on Cactus
Funding Need lightweight governance model to better target funding, help
funding agencies make decisions, enable leveraging international funding
Target limited science funding where it will make a difference, leverage CS funding
Cactus: broader application base has potential to coordinate with other disciplines
Einstein Toolkit Credits
Frank Loeffler (Louisiana State University) Erik Schnetter (Perimeter Institute) Christian Ott (Caltech) Ian Hinder (Albert Einstein Institute) Roland Haas (Caltech) Tanja Bode (Tuebingen) Bruno Mundim (Albert Einstein Institute) Peter Diener (Louisiana State University) Christian Reisswig (Caltech) Joshua Faber (RIT) Philipp Moesta (Caltech) And many others
AdvertisementSunday @ SC13, Denver
http://wssspe.researchcomputing.org.uk