1

11Parallel Programming in MPIpart 1Purpose of parallel programming tutorial = Communication = Data transfer among multiple processes.

Each of the processes executes its own program concurrently.

This is parallel processing.

Parallel program is required for parallel processing2Learn how to write parallel programs.

How to Describe Communications in a Program?TCP, UDP ?Good:- Portable: Available on many networks.Bad:- Protocols for connections and data-transfer are complicated.3Possible. But require additional descriptions for protocols.MPI (Message Passing Interface)A set of communication functions designed for parallel processingC, C++, FortranCan be called from C/C++/Fortran programs.

"Message Passing" = Send + ReceiveSend, Receive Actually, more functions other than Send and Receive are available.

Let's see a sample program, first.4555#include #include "mpi.h"

int main(int argc, char *argv[]){ int myid, procs, ierr, i; double myval, val; MPI_Status status; FILE *fp; char s[64];

MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myid); MPI_Comm_size(MPI_COMM_WORLD, &procs); if (myid == 0) { fp = fopen("test.dat", "r"); fscanf(fp, "%lf", &myval); for (i = 1; i < procs; i++){ fscanf(fp, "%lf", &val); MPI_Send(&val, 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD); } fclose(fp); } else MPI_Recv(&myval, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, &status);

printf("PROCS: %d, MYID: %d, MYVAL: %e\n", procs, myid, myval); MPI_Finalize();

return 0;}If my ID is 0input data for this process and keep it in myvaluse MPI_Send to send value in val to process iprocesses with ID other than 0use MPI_Recv to receive data from process 0 and keep it in myvalend of parallel computingSetup MPI environmentGet own ID (= rank) of the processi = procs input data and keep it in valGet total number of processesprint-out its own myval66Flow of the sample program.""Multiple "Processes" execute the program according to their number (= rank).6read data from a fileread datafrom a filesend val to rank 1read data from a filesend val to rank 2print myvalreceive datafrom rank 0

print myvalmyvalprint myvalvalvalreceive datafrom rank 0

myvalmyvalrank 0rank 1rank 2wait for the arrival of the datawait for the arrival of the dataSample of the Result of ExecutionThe order of the output can be different,since each process proceeds execution independently.7PROCS: 4 MYID: 1 MYVAL: 20.0000000000000000PROCS: 4 MYID: 2 MYVAL: 30.0000000000000000PROCS: 4 MYID: 0 MYVAL: 10.0000000000000000PROCS: 4 MYID: 3 MYVAL: 40.0000000000000000rank 1rank 2rank 0rank 388MPICharacteristics of MPI InterfaceMPI CMPI programs are ordinal programs in C-languageNot a new languageEvery process execute the same programEach process executes its own work according to its rank(=process number) A process cannot read or write variables on other process directly

Read file Read fileSendRead fileSendPrint myvalmyvalRank 0Receive

Print myvalmyvalReceive

Print myvalmyvalRank 1Rank 2valval8TCP, UDP vs MPIMPI:Simple interface dedicated for parallel computingSPMD(Single Program Multiple Data-stream) modelAll processes execute the same program

TCP, UDP: Generic interface for various communications,such as internet serversServer/Client model Each process executes its own program.910#include #include "mpi.h"

int main(int argc, char *argv[]){ int myid, procs, ierr, i; double myval, val; MPI_Status status; FILE *fp; char s[64];

MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myid); MPI_Comm_size(MPI_COMM_WORLD, &procs); if (myid == 0) { fp = fopen("test.dat", "r"); fscanf(fp, "%lf", &myval); for (i = 1; i < procs; i++){ fscanf(fp, "%lf", &val); MPI_Send(&val, 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD); } fclose(fp); } else MPI_Recv(&myval, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, &status);

printf("PROCS: %d, MYID: %d, MYVAL: %e\n", procs, myid, myval); MPI_Finalize();

return 0;} sock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP); memset(&echoServAddr, 0, sizeof(echoServAddr)); echoServAddr.sin_family = AF_INET; echoServAddr.sin_addr.s_addr = inet_addr(servIP); echoServAddr.sin_port = htons(echoServPort); connect(sock, (struct sockaddr *) &echoServAddr, sizeof(echoServAddr)); echoStringLen = strlen(echoString); send(sock, echoString, echoStringLen, 0);

totalBytesRcvd = 0; printf("Received: "); while (totalBytesRcvd < echoStringLen){ bytesRcvd = recv(sock, echoBuffer, RCVBUFSIZE - 1, 0); totalBytesRcvd += bytesRcvd; echoBuffer[bytesRcvd] = '\0' ; printf(echoBuffer); } printf("\n"); close(sock); servSock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP); memset(&echoServAddr, 0, sizeof(echoServAddr)); echoServAddr.sin_family = AF_INET; echoServAddr.sin_addr.s_addr = htonl(INADDR_ANY); echoServAddr.sin_port = htons(echoServPort); bind(servSock, (struct sockaddr *) &echoServAddr, sizeof(echoServAddr)); listen(servSock, MAXPENDING); for (;;){ clntLen = sizeof(echoClntAddr); clntSock = accept(servSock,(struct sockaddr *)&echoClntAddr, &clntLen); recvMsgSize = recv(clntSock, echoBuffer, RCVBUFSIZE, 0); while (recvMsgSize > 0){ send(clntSock, echoBuffer, recvMsgSize, 0); recvMsgSize = recv(clntSock, echoBuffer, RCVBUFSIZE, 0); } close(clntSock); }TCP ClientTCP ServerMPIinitializeinitializeinitializeMPILayer of MPIMPIHide the differences of networks11ApplicationsTCPUDPIPEthernet driver, Ethernet cardSocketsXTIMPIHigh-Speed Interconnect(InfiniBand, etc.)MPIHow to compile MPI programsCompile command: mpicc Example) mpicc -O3 test.c -o test

12optimization option O is not 0source file to compileexecutable file to createMPI sandyHow to execute MPI programs (on 'sandy')Prepare a script file

Submit the script file qsub test.shOther commandsqstat (= check status), qdel job_number (= cancel job)

13#!/bin/sh

#PBS -l nodes=2:ppn=4,walltime=00:10:00#PBS -j oe#PBS -q middle

cd $PBS_O_WORKDIR

mpirun -np 8 ./test-mpiNumber of Nodes (ex: 2)Number of Processes per node (ex: 4)Maximum Execution Time (ex: 10min.)Job QueueCommands to be ExecutedSample:cd to the directory from where this job is submittedRun MPI program with specified number (ex: 8) of processesEx 0)MPI Execution of an MPI programsandy.cc.kyushu-u.ac.jp First of all, login to sandy.cc.kyushu-u.ac.jpWindows: Use Putty

MacOS X: Use ssh command from terminal

ID User ID and Password will be announced later.Ex 0)MPI Execution of an MPI programAfter login, try the following commands.

Try changing the number of processes,or modifying the source program.$ cp /tmp/test-mpi.c .$ cp /tmp/test.dat .$ cp /tmp/test.sh .$ cat test-mpi.c$ cat test.dat$ mpicc test-mpi.c o test-mpi$ qsub test.shwait for a while$ ls (check the name of the result file (test.sh.o????))$ less test.sh.o????MPIMPIThe bodies of MPI functions are in "MPI Library".mpicc MPImpicc links the library to the program

MPIMPI Library16main(){ MPI_Init(...); ... MPI_Comm_rank(...); ... MPI_Send(...); ...}MPI_InitMPI_Comm_rank...mpicccompilelinksource programMPI LibraryExecutablefileMPIBasic Structure of MPI Programs17Function for start-upheader file "mpi.h"#include #include "mpi.h"

int main(int argc, char *argv[]){ ...

MPI_Init(&argc, &argv);

...

MPI_Comm_rank(MPI_COMM_WORLD, &myid); MPI_Comm_size(MPI_COMM_WORLD, &procs);

...

MPI_Send(&val, 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD); ... MPI_Recv(&myval, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, &status);

...

MPI_Finalize();

return 0;}Functions for finishYou can call MPI functions in this areaCrucial linesMPIMPI Functions TodayMPI_InitInitializationMPI_FinalizeFinalizationMPI_Comm_sizeGet number of processes MPI_Comm_rankGet rank (= Process number) of this processMPI_Send & MPI_RecvMessage PassingMPI_Bcast & MPI_GatherCollective Communication ( = Group Communication )181919MPI_InitMPIStart parallel execution of in MPIStart processes and establish connectionsamong them.MPIMost be called once before calling otherMPI functions

Parameter:main2Specify pointers of both of the arguments of 'main' function.Each process most share the name of the executable file, and the options given to the mpirun command.

19Usage: int MPI_Init(int *argc, char **argv);#include #include "mpi.h"int main(int argc, char *argv[]){int myid, procs, ierr;double myval, val; MPI_Status status; MPI_Init(&argc, &argv);MPI_Comm_rank(MPI_COMM_WORLD, &myid);MPI_Comm_size(MPI_COMM_WORLD, &procs); ...Example2020MPI_FinalizeFinishes paralles execution MPIMPI functions cannot be calledafter this function.

Every process needs to call this function before exitting the program.20Usage: int MPI_Finalize();main(){ ...

MPI_Finalize();}Example2121MPI_Comm_rankGet the rank(= process number) of the process2Returned in the second argument

= 1st argument = "communicator"An identifier for the group of processesMPI_COMM_WORLD In most cases, just specify MPI_COMM_WORLD, here.MPI_COMM_WORLD: a group that consists all of the processes in this executionProcesses can be devided into multiple groups and attached different jobs.21Usage: int MPI_Comm_rank(MPI_Comm comm, int *rank); ...MPI_Comm_rank(MPI_COMM_WORLD, &myid);...Example2222MPI_Comm_sizeGet the number of processes2

22Usage: int MPI_Comm_size(MPI_Comm comm, int *size); ... MPI_Comm_size(MPI_COMM_WORLD, &procs); ...ExampleMessage PassingCommunication between "sender" and "receiver"""Functions of Sending and Receiving most be called in a correct manner."From" rank and "To" rank are correctSpecified size of the data to be transferred is the same on both side Same "Tag" is specified on both side23Send To: Rank 1 Size: 10 Integer data Tag: 100Receive From: Rank 0 Size: 10 Integer data Tag: 100

Rank 0Rank 1Wait for the message2424MPI_SendInformation of the message to sendstart address of the data ,number of elements ,data type ,rank of the destination ,tag,communicator (= MPI_COMM_WORLD, in most cases)data types:

tag The number attached to each messageUsed in a kind of programs that handles anonymous messages.0 . Usually, you can specify 0.24Usage: int MPI_Send(void *b, int c, MPI_Datatype d, int dest, int t, MPI_Comm comm); ...MPI_Send(&val, 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD);...IntegerMPI_INTReal(Single)MPI_FLOATReal(Double)MPI_DOUBLECharacterMPI_CHARExampleExample of MPI_Send d 1Send the value of an integer variable 'd'

mat 100Send first 100 elements of array 'mat' (with MPI_DOUBLE type)

data 1050Send elements of an integer array 'data' from 10th to 59th element25MPI_Send(&d, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);MPI_Send(mat, 100, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD);MPI_Send(&(data[10]), 50, MPI_INT, 1, 0, MPI_COMM_WORLD);2626MPI_RecvInformation of the message to receivestart address for storing data,number of elements,data type,rank of the source ,tag (= 0, in most cases), communicator (= MPI_COMM_WORLD, in most cases),status

status: An integer array for storing the information of arrived messageConsists the information about the source rank and the tag. ( Not be used in most case )26 ...MPI_Recv(&myval, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD &status); ...Usage: int MPI_Recv(void *b, int c, MPI_Datatype d, int src, int t, MPI_Comm comm, MPI_Status *st);ExampleCollective CommunicationsCommunications among all of the processes in the group Examples)MPI_Bcastcopy a data to otherprocessesMPI_GatherGather data from other processesto an array

MPI_ReduceApply a 'Reduction'operation to the distributed datato produce one array273182Rank 03182Rank 13182Rank 2759Rank 0Rank 1Rank 2759123Rank 0Rank 1Rank 24567891215182828MPI_Bcastcopy a data on a process to all of the processesParameters:start address, number of elements, data type, root rank, communicatorroot rank: rank of the process that has the original dataExample:MPI_Bcast(a, 3, MPI_DOUBLE, 0, MPI_COMM_WORLD);

28Usage: int MPI_Bcast(void *b, int c, MPI_Datatype d, int root, MPI_Comm comm);Rank 0Rank 1Rank 2Rank 3aaaa2929MPI_GatherGather data from other processes to construct an arrayParameters:send data: start address, number of elements, data type, receive data: start address, number of elements, data type, (means only on the root rank)root rank, communicatorroot rank: rank of the process that stores the result arrayExample:MPI_Gather(a, 3, MPI_DOUBLE, b, 3, MPI_DOUBLE, 0, MPI_COMM_WORLD);29Usage: int MPI_Gather(void *sb, int sc MPI_Datatype st, void *rb, int rc, MPI_Datatype rt, int root, MPI_Comm comm);Rank 0Rank 1Rank 2Rank 3aaaab30Usage of Collective CommunicationsEvery process must call the same function MPI_Bcastroot rankFor example, MPI_Bcast must be called not only by the root rank but also all of the other ranks

On functions that require information of both send and receive, the specified ranges of the addresses for sending and receiving cannot be overlapped.MPI_Gather, MPI_Allgather, MPI_Gatherv, MPI_Allgatherv, MPI_Recude, MPI_Allreduce, MPI_Alltoall, MPI_Alltoallv, etc.30SummaryMPIOn MPI, multiple processes run the same programJobs are attached according to the rank(the number) of each processEach process runs on its own memory spaceAccesses to the data on other processes can be made only by explicit communication among processesMPI functionsMPI_Init, MPI_Finalize, MPI_Comm_rankMPI_Send, MPI_RecvMPI_Bcast, MPI_Gather

313232ReferencesMPI Forumhttp://www.mpi-forum.org/specification of "MPI standard"

MPI http://phase.hpcc.jp/phase/mpi-j/ml/

http://accc.riken.jp/HPC/training/mpi/mpi_all_2007-02-07.pdf

32Ex 1) A program that displays random numbersMake a program in which each process displays its own rank with one integer random number

Sample: #include #include #include

int main(int argc, char *argv[]){ int r; struct timeval tv;

gettimeofday(&tv, NULL); srand(tv.tv_usec); r = rand();

printf("%d\n", r);}Ex 1) (cont.)Example of the result of execution

1: 5203910: 9478965003: 17975259402: 5659177804: 16186515065: 2740322936: 12487873507: 828046128Ex 1) Sample of the answer#include #include #include #include "mpi.h"

int main(int argc, char *argv[]){ int r, myid, procs; struct timeval tv;

MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myid); MPI_Comm_size(MPI_COMM_WORLD, &procs);

gettimeofday(&tv, NULL); srand(tv.tv_usec); r = rand();

printf("%d: %d\n", myid, r); MPI_Finalize();}The data is not printed out in the order of ranks.Report: Display in orderEx 1) Modify the program in Ex1), so that: Messages are printed out in the order of the rank of each processExample of the result of the execution

0: 15243946311: 9990945012: 9417636043: 5269563784: 1523746435: 11381541176: 19268147547: 156004811232~3 students will be designated toshow and explain their answer at the next class. Hint 2At least, two methods are possible.

Method 1) (Easy)Gather the data to rank 0, first.Then, let rank 0 to print data in order.

Method 2) (Little hard)Before rank i prints its data, receive a message from rank i-1 (i > 0)After rank i prints its data, send a message to rank i+1 (i