High-Performance Computing, Computational Science, and NeuroInformatics Research Allen D. Malony Department of Computer and Information Science NeuroInformatics

High-Performance Computing, Computational Science, and NeuroInformatics Research

Allen D. Malony

Department of Computer and Information ScienceNeuroInformatics Center (NIC)Computational Science Institute

University of Oregon

April 29, 2004 PNNL UO Visit

Outline

High-performance computing research Interactions and funding Project areas TAU parallel performance system

Computational science at UO Projects Computational Science Institute

Neuroinformatics research NeuroInformatics Center (NIC)

ICONIC Grid


High-Performance Computing Research

Strong associations with DOE national laboratories Los Alamos National Lab Lawrence Livermore National Lab Sandia National Lab (Livermore) Argonne National Lab National Energy Research Supercomputing Center

DOE funding Office of Science, Advance Scientific Computing ASCI/NNSA

NSF funding Academic Research Infrastructure Major Research Instrumentation


Project Areas

Parallel performance evaluation and tools Parallel language systems Tools for parallel system and software interaction Source code analysis Parallel component software Computational services Grid computing Parallel modeling and simulation Scientific problem solving environments

Allen D. Malony Sameer S. Shende

Department of Computer and Information Science

Computational Science Institute

University of Oregon

TAU Parallel Performance System


Parallel Performance Research

Tools for performance problem solving Empirical-based performance optimization process

characterization

PerformanceTuning

PerformanceDiagnosis

PerformanceExperimentation

PerformanceObservation

hypotheses

properties

• Instrumentation• Measurement• Analysis• Visualization

PerformanceTechnology


Complexity Challenges for Performance Tools

Computing system environment complexity Observation integration and optimization Access, accuracy, and granularity constraints Diverse/specialized observation capabilities/technology Restricted modes limit performance problem solving

Sophisticated software development environments Programming paradigms and performance models Performance data mapping to software abstractions Uniformity of performance abstraction across platforms Rich observation capabilities and flexible configuration Common performance problem solving methods


General Problems

How do we create robust and ubiquitous performance technology for the analysis and tuning of parallel and

distributed software and systems in the presence of (evolving) complexity challenges?

How do we apply performance technology effectively for the variety and diversity of performance problems

that arise in the context of complex parallel and distributed computer systems?


TAU Performance System

Tuning and Analysis Utilities Performance system framework for scalable parallel and

distributed high-performance computing Targets a general complex system computation model

nodes / contexts / threads Multi-level: system / software / parallelism Measurement and analysis abstraction

Integrated toolkit for performance instrumentation, measurement, analysis, and visualization Portable performance profiling and tracing facility Open software approach with technology integration

University of Oregon , Forschungszentrum Jülich, LANL


TAU Performance System Architecture

EPILOG

Paraver


TAU Performance System Status

Computing platforms IBM SP / Power4, SGI Origin 2K/3K, ASCI Red, Cray

T3E / SV-1 / X-1, HP (Compaq) SC (Tru64), HP Superdome (HP-UX), Sun, Hitachi SR8000, NEX SX-5/6, Linux clusters (IA-32/64, Alpha, PPC, PA-RISC, Power, Opteron), Apple (G4/5, OS X), Windows

Programming languages C, C++, Fortran 77/90/95, HPF, Java, OpenMP, Python

Communication libraries MPI, PVM, Nexus, shmem, LAMPI, MPIJava

Thread libraries pthreads, SGI sproc, Java,Windows, OpenMP


TAU Performance System Status (continued)

Compilers Intel KAI (KCC, KAP/Pro), PGI, GNU, Fujitsu, Sun,

Microsoft, SGI, Cray, IBM (xlc, xlf), Compaq, Hitachi, NEC, Intel

Application libraries (selected) Blitz++, A++/P++, PETSc, SAMRAI, Overture, PAWS

Application frameworks (selected) POOMA, MC++, ECMF, Uintah, VTF, UPS, GrACE

Performance technology integrated with TAU PAPI, PCL, DyninstAPI, mpiP, MUSE/Magnet

TAU full distribution (Version 2.x, web download) TAU performance system toolkit and user’s guide Automatic software installation and examples


Computational Science

ComputerScience

Biology

Neuroscience

PsychologyPaleontology

Geoscience

Math

Integration of computer sciencein traditional sciencedisciplines

Third model ofscientificresearch

Application ofhigh-performancecomputation, algorithmsand networking Parallel computing Grid computing


Computational Science Projects at UO

Geological science Model coupling for hydrology

Bioinformatics Zebrafish Information Network (ZFIN) Evolution of gene families Oregon Bioinformatics Tool

Neuroinformatics Electronic notebooks Domain-specific problem solving environments

Dinosaur skeleton and motion modeling Computational Science Institute


Computational Science Cognitive Neuroscience

Computational methods applied to scientific research High-performance simulation of complex phenomena Large-scale data analysis and visualization

Understand functional activity of the human cortex Multiple cognitive, clinical, and medical domains Multiple experimental paradigms and methods

Need for coupled/integrated modeling and analysis Multi-modal (electromagnetic, MR, optical) Physical brain models and theoretical cognitive models

Need for robust tools: computational & informatic


Brain Dynamics Analysis Problem

Identify functional components Different cognitive neuroscience research contexts Clinical and medical applications

Interpret with respect to physical and cognitive models Requirements: spatial (structure), temporal (activity) Imaging techniques for analyzing brain dynamics

Blood flow neuroimaging (PET, fMRI) good spatial resolution functional brain mapping temporal limitations to tracking of dynamic activities

Electromagnetic measures (EEG/ERP, MEG) msec temporal resolution to distinguish components spatial resolution sub-optimal (source localization)


Integrated Electromagnetic Brain Analysis

IndividualBrain Analysis

Structural /FunctionalMRI/PET

DenseArray EEG /

MEG

ConstraintAnalysis

Head Analysis

Source Analysis

Signal Analysis

Response Analysis

Experimentsubject

temporaldynamics

neuralconstraints

CorticalActivity Model

ComponentResponse Model

spatial patternrecognition

temporal patternrecognition

Cortical ActivityKnowledge Base

Component ResponseKnowledge Base

good spatialpoor temporal

poor spatialgood temporal neuroimaging

integration


Experimental Methodology and Tool Integration

source localization constrained to cortical surface

processed EEG

BrainVoyager

BESA

CT / MRI

EEG segmentedtissues

16x256bits permillisec(30MB/m)

mesh generation

EMSEInterpolator 3D

NetStation


NeuroInformatics Center (NIC)

Application of computational science methods to cognitive and clinical neuroscience problems Understand functional activity of the brain Help to diagnosis brain-related disorders Utilize high-performance computing and simulation Support large-scale data analysis and visualization

Advance techniques for integrated neuroimaging Coupled modeling (EEG/ERP and MR analysis) Advanced statistical factor analysis FDM/FEM brain models (EEG, CT, MRI) Source localization

Problem-solving environment for brain analysis


NIC Organization

Director, Allen D. Malony Associate Professor, Computer and Information Science

Associate Director, Don M. Tucker Professor, Psychology; CEO, EGI

Computational Scientist, Kevin Glass Ph.D., Computer Science; B.S., Physics

Computational Physicist, Sergei Turovets Ph.D., Computer Science; B.S., Physics

Computer Scientist, Sameer S. Shende Ph.D., Computer Science; parallel computing specialist

Mathematician, Bob Frank M.S., Mathematics


Funding Support

BBMI federal appropriation DoD Telemedicine Advanced Technology Research

Command (TATRC) Initial budget of approximately $750K Oct. 1, 2002 through March 31, 2004

NSF Major Research Instrumentation ICONIC Grid, awarded

New proposal opportunities NIH Human Brain Project Neuroinformatics NSF ITR


NIC Approaches

Optimize spatial resolution MRI structural information Measurement of skull conductivity Convergence / co-recording with MEG and fMRI

Optimize temporal resolution Use EEG/MEG time course for fMRI signal extraction Decomposition of component analysis (ICA, PCA) Single-trial analysis

Computational brain models Boundary and finite element brain models Brain information databases and atlases


EEG/ERP Methodology

Electroencephalogram (EEG) Event-Related Potential (ERP)

Stimulus-locked measures of brain dynamics Generated from subject- and trial-based analysis

Raw EEG datasets processed and analyzed Segmentation to time series waveforms Blink removal and other cleaning ERP analysis

Averaging for increasing signal to noise Characterization with respect to trial conditions Results visualization

Source localization


EGI Geodesics Sensor Net

Electrical Geodesics Inc. Dense-array sensor technology

64/128/256 channels 256-channel geodesics sensor net

AgCl plastic electrodes Carbon fiber leads

Future optical sensors EGI + LANL


EEG/ERP Experiment Management System

Support EEG-based cognitive neuroscience research Based on experiment model

Experiment type Subjects measured for trial types

Management of experiment data Raw and processed datasets and derived statistics Per experiment/subject/trial database Secure protection and storage with selective access

Analysis tools and workflows Generation of results (across experimental variables) Analysis processes with multi-tool workflows


EEG/ERP Experiment Analysis Environment

… …

rawprocessed datasets/ derived resultsanalysis workflow

storageresources

virtualservices

compute resources


Source Localization

Mapping of scalp potentials to cortical generators Single time sample and time series

Requirements Accurate head model and physics

High-resolution 3D structural geometry Precise tissue identification and segmentation Correct tissue conductivity assessment

Computational head model formulation Finite element model (FEM) Finite difference model (FDM) Forward problem calculation

Dipole search strategy


Advanced Image Segmentation

Native MR gives high gray-to-white matter contrast

Edge detection finds region boundaries

Segments formed by edge merger

Color depicts tissue type Investigate more advance

level set methods and hybrid methods


Building Finite Element Brain Models

MRI segmentation of brain tissues Conductivity model

Measure head tissue conductivity Electrical impedance tomography

small currents are injectedbetween electrode pair

resulting potential measuredat remaining electrodes

Finite element forward solution Source inverse modeling

Explicit and implicit methods Bayesian methodology

scalp

CSF

skull

cortex


Conductivity Modeling

Governing Equations ICS/BCS

Discretization

System of Algebraic Equations

Equation (Matrix) Solver

Approximate Solution

Continuous Solutions

Finite-DifferenceFinite-Element

Boundary-ElementFinite-Volume

Spectral

Discrete Nodal Values

TridiagonalADISOR

Gauss-SeidelGaussian elimination

(x,y,z,t)J (x,y,z,t)B (x,y,z,t)


Source Localization Analysis Environment

… …

raw

storageresources

virtualservices

compute resources


NIC Computational Cluster (“Neuronic Cluster”)

Dell computational cluster 16 dual-processor nodes

2.8 MHz Pentium Xeon 4 Gbyte memory 36 Gbyte disk Dual Gigabit ethernet adaptors 2U form factor

Master node (same specs) 2 Gigabit ethernet switches

Brain modeling Component analysis


NIC Relationships

Psychology

CIS

BDL BEL

CSI

OHSU/ OGI

Utah

UCSD

USCAcademic Labs / Centers

LANL Argonne

NCSAInternet2

EGI

Industry

Intel IBM

NIC

UO Departments

UO Centers/Institutes

BBMI CDSI

CNI

Physics

NSI

Sandia


NSF MRI Proposal

Major Research Instrumentation (MRI) “Acquisition of the Oregon ICONIC Grid for

Integrated COgnitive Neuroscience Informatics and Computation”

PIs Computer Science: Malony, Conery Psychology: Tucker, Posner, Nunnally

Senior personnel Computer Science: Douglas, Cuny Psychology: Neville, Awh, White

Approximately $1.2M over three years


SMPServerIBM p655

GraphicsSMP

SGI MARS

SAN Storage System

Gbit Campus Backbone

NIC CIS CIS

Internet 2

SharedMemory

IBM p690

DistributedMemory

IBM JS20

CNI

DistributedMemory

Dell Pentium Xeon

NIC4x8 16 16 2x8 2x16

graphics workstations interactive, immersive viz other campus clusters

ICONIC Grid

5 Terabytes


Cognitive Neuroscience and ICONIC Grid

Common questions to be explored Identifying brain networks Critical periods during normal development Network involvement in psychopathologies Training interventions in network development

Research areas Development of attentional networks Brain plasticity in normal development and deprived Attention and emotion regulation Spatial working memory and selective attention Attention and psychopathology


Computer Science and ICONIC Grid

Scheduling and resource management Assign hardware resources to computation tasks Scheduling of workloads for

PSEs for computational science Provide scientists an entrée to the computational and

data management power of the infrastructure without requiring specialized knowledge of parallel execution

Marine seismic tomograph, molecular evolution Interactive / immersive three-dimensional visualization

Explore multi-sensory visualization Merge 3D graphics with force-feedback haptics

Parallel performance evaluation

Documents

High-Performance Computing, Computational Science, and NeuroInformatics Research Allen D. Malony Department of Computer and Information Science NeuroInformatics