Teamster-G: A Shared-Memory Parallel Programming Environment on Grids 高雄應用科技大學電機系梁廷宇

Teamster-G: A Shared-Memory Parallel Programming Environment on Grids

高雄應用科技大學電機系梁廷宇

HPDSHPDS

原高雄工專，創立於 1963年， 1990 年改大。現擁有「電資學院」、「工學院」、「管理學院」、「人文社會學院」等四個學院、四個博士班、十八個碩士班、十一個產業研發碩士專班、十八個系、以及二個研究中心。

HPDSHPDS

HPDSHPDS

Outline

Background Teamster-G Current State of Teamster-G Next Step of Teamster-G Conclusions

HPDSHPDS

Background

• What is a Computational Grid ?• Ensemble of geographically-distributed resources

• Analogy to Power Grids• deliver computational power and data in a cheap and consistent way

• Grid Features• single sign-on• heterogeneous• dynamic and shared• autonomy

HPDSHPDS

Background

• Why we need computational Grids ?

HPDSHPDS

Background

• LCG : LHC Computing Grids

HPDSHPDS

Background (cont.)

• Providing a familiar programming toolkit is an important factor to make users change their working environments from PCs, workstation, servers, SMPs or clusters to computational grids – RPC– Java RMI– MPI– DSM– OpenMP

HPDSHPDS

Background (cont.)

• DSM (IVY, Li Kai, 1986)– problem: false sharing

P0 P1 Pn…………………

Virtual Shared Memory

mprotect()

M0 M1 Mn

DSMLIB0 DSMLIB1 DSMLIBn

page fault

page request return page

HPDSHPDS

Background (cont.)

• OpenMP: an application programming interface (API) that supports fork-join multithreading programming in C/C++ (1998) and Fortran (1997).– Compiler directives

– Runtime library

– Environment variables

HPDSHPDS

Background (cont.)

• OpenMP program example#include<omp.h>#define NUM_START 1#define NUM_END 100

int main() {

int i;double total=0;

#pragma omp parallel for reduction(+:total) for(i=NUM_START;i<=NUM_END;i++) { total=total+i; }

printf("\n 1+2+3+4+.....+%d = %f\n",NUM_END,total);}

HPDSHPDS

Background (cont.)

• OpenMP Compiler

int _ompc_main(){auto double sumall;auto double total;auto int i;(total)=(0);auto void *__ompc_argv[1];(*(__ompc_argv))=(((void *)(&total)));_ompc_do_parallel(__ompc_func_3,__ompc_argv);

# 20 "sum_omp.c"printf("\012 1+2+3+4+.....+%d = %f\012",100,total);}

HPDSHPDS

Background (cont.)

• OpenMP Compiler__ompc_func_3(void **__ompc_args){auto double *_pp_total;(_pp_total)=(((double *)(*__ompc_args)));{{auto double _p_total;(_p_total)=(0.0);(_p_i_0)=(1);(_p_i_1)=((100)+(1));(_p_i_2)=(1);_ompc_default_sched(&_p_i_0,&_p_i_1,&_p_i_2);

for((_p_i)=(_p_i_0);(_p_i)<(_p_i_1);(_p_i)+=(_p_i_2)){# 16 "sum_omp.c"(_p_total)=((_p_total)+(((double )(_p_i))));}_ompc_reduction(&_p_total,_pp_total);_ompc_barrier();}

HPDSHPDS

Background (cont.)

• Grid-enabled programming toolkit– RPC GridRPC– Java RMI Grid RMI– MPIMPICH-G2– DSM ?– OpenMP ?

• It is necessary to provide the shared memory parallel programming interfaces for grids in order for– Minimizing the programming complexity of grid computing– Allowing users to change their working environment from SMP workst

ations or clusters to computational grids– Enriching grid applications

HPDSHPDS

Background (cont.)

• Teamster-G project : Enabling Shared Memory Programming Interface for Grid Computing– Pthread, OpenMP– Transparent resource discover/allocation

HPDSHPDS

Outline

BackgroundTeamster-GCurrent State of Teamster-GNext Step of Teamster-GConclusions

HPDSHPDS

Teamster-G

• Teamster-G is developed based on Globus toolkit and Teamster. (KUAS, NCKU and Leader, 2004)

• Two work packages– Enabling Teamster for grid computing

– Porting OpenMP API on Teamster

• Two challenges– Minimizing the communication cost of data-consistency m

aintenance

– Keeping source-compatible for OpenMP programs

HPDSHPDS

Teamster-G (cont.)

• Globus toolkit– Resource management service : GRAM (Grid Res

ource Allocation and Management) – Information service : MDS (Meta-computing Direc

tory Services) – Security service : GSI (Grid Security Infrastructur

e) – Data movement and management: GASS (Global

Access to Secondary Storage)

HPDSHPDS

Teamster-G (cont.)

• Teamster (T.B. Chang, NCKU)– a user-level DSM system supporting Linux-based

SMP clusters– global shared address space– eager-released consistency protocol– support POSIX thread

HPDSHPDS

Teamster-G (cont.)

• Components – TGRUN– TGRB (Teamster-G Resource Broker)– TGCM (Teamster-G Cluster Manager)

HPDSHPDS

Teamster-G (cont.)

• Resource allocation

HPDSHPDS

Teamster-G (cont.)

• Job execution

HPDSHPDS

Teamster-G (cont.)

• Data consistency Maintenance : minimizing the number of messages transferred through WANs– Cluster cache

• inter-copyset

• intra-copyset

– Two-level consistency protocol• inter-cluster update propagation

• intra-cluster update propagation

– Hybrid communication protocol• TCP used in WAN connection

• UDP used in LAN connection

HPDSHPDS

Teamster-G (cont.)

Onrequest

Cluster A

Cluster B

B0

A1A2

B1 B2

WAN

A0

X1Y1Z1

X1Y1Z1

Inter-cluster copyset :

A Intra-cluster copyset : A0

B Intra-cluster copyset :

OwnerA0

OwnerA0

OwnerCluster A

OwnerB0

X1Y1Z1

Page owner

Page Fault

Send page requestReturn the page

A2

A

Page Fault

Send page request

B2

Forward page request

Onrequest

Send page request

Page Fault

B1

B

Return the page

X1Y1Z1

Return the page

Return the page

X1Y1Z1

A2

A

OwnerB0

HPDSHPDS

Teamster-G (cont.)

Cluster A

Cluster B

B0

A1A2

B1 B2

WAN

A0

B0 arriveAll nodes in thiscluster are arrived

B2 arriveA0 arriveB1 arriveno diff to flush

X1Y1Z1

X1Y2Z1

Page owner

X2Y1Z1

X1Y1Z1

X1Y1Z2

X1Y1Z1

X1Y1Z1

X1Y1Z1

X1Y1Z1

X1Y1Z1

twinIntra twinInter twin

Intra twinInter twinbarrier

Z2diff

Inter-cluster copyset : A B

A Intra-cluster copyset : A0 A2

B Intra-cluster copyset : B0 B1 B2

ask for intra copyset and flush diff

X2Y1Z2

flush diff

X1Y1Z2

Y2diff

flush diff

X1Y2Z1

diff

flush diff

flush diff

X2Y1Z2

X2Y1Z2

groupdiff

X2X2Z2

ask for inter copyset and flush group diff

X2Y1Z2

X2Y2Z2

forward diff

X2Y2Z2

A2 arriveno diff to flushCluster AAll node are arrived

groupdiff

Y2

flush group diff

X2Y2Z2

forward diff

forward diff

X2Y2Z2

X2Y2Z2

X1Y1Z2

HPDSHPDS

Teamster-G (cont.)

• Performance Evaluation

Application Problem Size CPU Demand Memory Demand

N-Body8192 particles

32 loops661.072 sec/

Pentium IV 3G8 Mbytes

Site CPUs Memory LAN

NCKU.EE 4xPentium IV 3G 1GB100Mbps Ethernet

KUAS.EE 4xPenitum IV 3G 1GB100Mbps Ethernet

Network between Site 1 and Site 2:Average Bandwidth: 3.3MB/secondAverage Round trip time: 2.424 ms, 6 hops

HPDSHPDS

Teamster-G (cont.)

• Performance

Speed up 2 nodes 4 nodes 8 nodes

Nbody in a single cluster 1.93 3.51 6.87

Nbody across two clusters 1.92 3.46 6.36

HPDSHPDS

Teamster-G (cont.)

Ratios of LAN and WAN communication in the N-body application

HPDSHPDS

Teamster-G (cont.)

• Porting OpenMP on Teamster

int var1;int var2 = 0;int main() { int *data=(int*) malloc(sizeof(int)*10); #pragma omp parallel { int id = omp_get_thread_num(); }}

Global variables

Memory allocation function

OpenMP directives

OpenMP library functions

HPDSHPDS

Teamster-G (cont.)

• Four components is implemented– OpenMP Translator (Omni OpenMP compiler, Tsu

kuba, Japan)– OpenMP Runtime Library (OMPLIB)– Distributed Portable Thread Library (DPTHLIB )– Load Balance Mechanism (LBM )

HPDSHPDS

Teamster-G (cont.)

• Compiling OpenMP programs on Teamster-G

OpenMP TranslatorDPTHLIB

OMPLIB

LBM

DSMLIB

Executable files

OpenMP sources

TranslatedSourcecode

Make

Object file

gcc

linking

HPDSHPDS

Teamster-G (cont.)

• Verification

HPDSHPDS

Teamster-G (cont.)

• Performance evaluation

HPDSHPDS

Teamster-G (cont.)

• Performance evaluation

HPDSHPDS

Outline

BackgroundTeamster-GCurrent State of Teamster-GNext Step of Teamster-GConclusion

HPDSHPDS

Current State of Teamster-G

• Teamster-G2 based on GT4 and integrated with well known Grid toolkits such as Ganglia, NWS, Condor, PBS, SGE etc.

– Long-term predictor of resource-availability and performance

– Applications self-adaptive– Data-Racing Avoidance and Deterministic Replay

HPDSHPDS


HPDSHPDS


• Long-term predictor of resource availability– Log resource load– Transfer load history into chaincodes– Discover frequent workload pattern (FWPs)– Predict resource load states by applying– Association rules– Calculate the availabilities of resources.

CPU Worload of day

0

2

4

6

8

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Hour

Lev

el

HPDSHPDS


• Effectiveness of the proposed predictor

HPDSHPDS


• Impact of the proposed predictor

HPDSHPDS


• Progressive Multilayer Reconfiguration– virtual processor layer– node layer– inter-cluster layer

HPDSHPDS


• Data Racing Avoidance and Deterministic Replay

HPDSHPDS


• Verification of DRARS

HPDSHPDS


• Cost of DRARS

HPDSHPDS

Outline

BackgroundTeamster-GCurrent Status of Teamster-GNext Step of Teamster-GConclusion

HPDSHPDS

Next Step of Teamster-G

• IDE : Debugger tool

• Quality of Service

• Checkpoint and Recovery

• Integration of Data/File Sharing (Data Grids)

• Integration of Memory Sharing (RAM Grids)

• Applications : GridERP

HPDSHPDS

Outline

• Background

• Teamster-G

• Current State of Teamster-G

• Next Step of Teamster-G

• Conclusion

HPDSHPDS

Conclusion

• Teamster-G has successfully provided the shared memory parallel programming interfaces for grid computing. As a result, It allows users to easily move their Pthread/OpenMP programs from SMPs or clusters to computational grids for execution.

• However, we must continue to put effort into the development of Teamster-G.

HPDSHPDS

Thank you for your listening& any questions ?

The End

Documents

Teamster-G: A Shared-Memory Parallel Programming Environment on Grids 高雄應用科技大學 電機系 梁廷宇

Teamster-G: A Shared-Memory Parallel Programming Environment on Grids 高雄應用科技大學電機系梁廷宇