07 ParallelProgramming

Parallelization
programming
Jan Thorbecke
Delft
University of
Technology
Challenge the future
Type to enter text
Contents
Introduction
Programming
OpenMP
MPI
Issues with parallel computing

Running programs
Examples
Amdahls Law
Describes the relation between the parallel portion of your code
and the expected speedup
!
!
!
!
!
!
!
speedup =
(1
1
P) +
P
N
P = parallel portion
N = number of processors used in parallel part
P/N is the ideal parallel speed-up, it will always be less

3
Amdahls Law
1.0
0.99
0.98
60
Speedup
50
40
30
20
10
0
0
10
20
30
40
Number of Processors
50
60
4
Amdahls Law
100
maxspeedup(x)
Maximum Speedup
80
60
40
20
0
1
4 5 7 10
20 30 40 50 70 90
Sequential Portion in %
5
Super Linear speed up

1.0
0.99
0.96
super linear
60
Speedup
50
40
30
20
10
0
0
10
20
30
40
Number of Processors
50
60
6
Concurrency and Parallelism

Concurrency and parallelism are often used synonymously.
Concurrency: The independence of parts of an algorithm

(= independent of each other).
Parallelism(also parallel execution): Two or more parts of a

program are executed at the same moment in time.
Concurrency is a necessary prerequisite for parallel execution

for parallel execution
but
Parallel execution is only one possible consequence of
concurrency.
7
Concurrency vs. Parallelism

Concurrency: two or more threads are in progress at the same time:
!
! Thread 1
!Thread 2
!
!
Parallelism: two or more threads are executing at the same time
!
!Thread 1
!Thread 2
!
Multiple cores needed
Learning
Classical CPUs are sequential
There is an enormous sequential programming knowledge

build into compilers and know by most programmers.
Parallel Programming is requiring new skills and new tools.
!
!
Start to parallelise simple problems and keep on learning along

the way to complex real-world problems.
Recognizing Sequential Processes

Time is inherently sequential
Dynamics and real-time, event driven applications are often difficult to

parallelize effectively
Many games fall into this category
Iterative processes
The results of an iteration depend on the preceding iteration
Audio encoders fall into this category
Parallel Programming Models

Parallel programming models exist as an abstraction above
hardware and memory architectures.
Which model to use is often a combination of what is available

and personal choice. There is no "best" model, although there
certainly are better implementations of some models over others.
Parallel Programming Models

Shared Memory
tasks share a common address space, which they read and write
asynchronously.
Threads (functional)
a single process can have multiple, concurrent execution paths.
Example implementations: POSIX threads & OpenMP
Message Passing
tasks exchange data through communications by sending and
receiving messages. Example: MPI-2 specification.
Data Parallel languages

tasks perform the same operation on their partition of work.
Example: Co-array Fortran (CAF), Unified Parallel C (UPC),
Chapel
Hybrid
Programming Models
New to parallel programming
Experienced programmer
User
Programming Model
Shared Memory
Message passing
Hybrid
Operating system
compilers
System Software
Hardware Layer
Memory
Interconnect
Interconnect
Distributed Memory
Shared Memory
13
Parallel Programming Concepts

Work distribution
Which parallel task is doing what?
Which data is used by which task?
Synchronization
Do the different parallel tasks meet?
Communication
Is communication between parallel parts needed?
Load Balance
Does every task has the same amount of work?
Are all processors of the same speed?
14
Distributing Work and/or Data

Work decomposition
based on loop counter
!
!
Data decomposition
all work for a local portion
of the data is done by the
local processor
do i=1,100!
1: i=1,25!
2: i=26,50!
3: i=51,75!
4: i=76,100
A(1:10,1:25)!
A(1:10,26:50)!
A(11:20,1:25)!
A(11:20,25:50)
!
!
Domain decomposition
decomposition of work and
data
15
Synchronization
!
!
!
!
!
!
!
!
Synchronization
Do i=1,100
a(i) = b(i)+c(i)
Enddo
Do i=1,100
d(i) = 2*a(101-i)
Enddo
i=1..25 | 26..50 | 51..75 | 76..100

execute on the 4 processors
BARRIER synchronization
i=1..25 | 26..50 | 51..75 | 76..100
execute on the 4 processors
Synchronization
is necessary
Synchronization
may cause
idle time on some processors

causes overhead
overhead to execute the synchronization primitive
idle time, when not all tasks are finished at the same time
Hardware Architectures & Parallel Programming Models

[2] Slide 18 / 54 Hchstleistungsrechenzentrum Stuttgart
16
Consider how many calculations you can execute while only

sending 1 Bit from one process to another
(6 s, 1.0 Gflop/s
6000 operations)
Communication
Sending 16 kByte (20x20x5) doubles

53.3 s
(with 300 MB/s bandwidth
53 300 operations)
very often blocks have to wait for their neighbours

but extra work limits parallel efficiency
Communication should only be used if one is quite sure that this is

the best solution
communication is necessary on the boundaries
!
!
!
!
!
!
!
Domain Decomposition
Adamidis/Bnisch
Slide 11
Hchstleistungsrechenzentrum Stuttgart
A(1:25)!
A(26:50)!
A(51:75)!
A(76:100)
do i=2,99!
b(i) = b(i) + h*(a(i-1)-2*a(i)+a(i+1))!
end do!
e.g. b(26) = b(26) + h*(a(25)-2*a(26)+a(27))!

2- Dimensional DD with two Halo Cells
domain decomposition
Mesh Partitioning
Subdomain for each Process
Adamidis/Bnisch
Slide 12
17
tructured Grids
Load Imbalance
Mesh partitioning with special load balancing libraries

Metis
(George Karypis, University of Minnesota)
ParMetis (internally parallel version of Metis)
http://www.cs.umn.edu/~karypis/metis/metis.html
Load imbalance is the time that some processors in the system
http://www.hlrs.de/organization/par/services/tools/loadbalancer/metis.html
Jostle
are
idle due to:
(Chris Walshaw, University of Greenwich)
less parallelism than processors

http://www.gre.ac.uk/jostle
unequal sized tasks together with too little parallelism
http://www.hlrs.de/organization/par/services/tools/loadbalancer/jostle.html
Goals:
unequal processors
Same work load in

each sub-domain
Minimizing the
maximal number of
neighbor-connections
between sub-domains
0
3
12
16
2
4
13
15
17 14
6
9
21
23
22
10
7
11
20
18
19
elization and Iterative Solvers

Rolf Rabenseifner
19 of 51
18
Examples of work distribution

Master Slave
Task Decomposition
19
!
First, decide how data elements should be divided among processors

!
!
!
Second, decide which tasks each processor should be doing
!
Example:
Vector addition, summation, maximum, ..

Large data sets whose elements can be computed independently
- Divide data and associated computation among threads
Find the largest element of an array
13
26 13
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
Core 0
5
13
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
Core 0
5
13
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
68
83
Core 0
5
13
13
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
49
68
83
Core 0
5
13
13
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
49
98
94
Core 0
5
13
13
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
50
98
94
Core 0
5
13
26
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
50
98
94
Core 0
5
13
26
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
50
98
94
Core 0
5
13
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
26
50
26
98
94
Core 0
5
13
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
26
50
50
98
94
Core 0
5
13
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
26
50
98
98
94
Core 0
5
13
Core 1
26 13
Core 2
Core 3
49 34 50 22 12 68 12 98 16 78 31 83 51 94 27 74 64
26
50
98
98
94
Master Slave
Get a heap of work done
Core 0 (master)
Core 1
T
T
Core 2
T
Core 3
33
Task Decomposition
f()
g()
h()
r()
q()
s()
Task Decomposition
Core 0
Core 1
g()
h()
f()
Core 2
r()
q()
s()
Task Decomposition
Core 0
Core 1
g()
h()
f()
Core 2
r()
q()
s()
Task Decomposition
Core 0
Core 1
g()
h()
f()
Core 2
r()
q()
s()
Task Decomposition
Core 0
Core 1
g()
h()
f()
Core 2
r()
q()
s()
Task Decomposition
Core 0
Core 1
g()
h()
f()
Core 2
r()
q()
s()
Evaluating Parallel Models

How do processes share information?
How do processes synchronize?
In shared-memory model, both accomplished through shared

variables
Communication: buffer
Synchronization: semaphore
Shared versus Private Variables
Thread
Private
Variables
!
!
Shared
!
Variables
!
!
!
Thread
Private
Variables
Race Conditions
Parallel threads can race against each other to update resources
Race conditions occur when execution order is assumed but not

guaranteed
Example: un-synchronised access to bank account
Withdraws $100
from account
Deposits $100
into account
Initial balance = $1000

Final balance = ?
Race Conditions
Withdraws $100
from account
Deposits $100
into account
Initial balance = $1000

Final balance = ?
Time
Withdrawal
Load (balance = $1000)
Subtract $100
Load (balance = $1000)
Store (balance = $900)
Add $100
Deposit
Store (balance = $1100)
Code Example in OpenMP

for (i=0; i<NMAX; i++) {!
a[i] = 1;!
b[i] = 2;!
}!
#pragma omp parallel for shared(a,b)!
for (i=0; i<12; i++) {!
a[i+1] = a[i]+b[i];!
}!
!
1:
4:
4:
4:
a= 1.0,
3.0,
5.0,
7.0,
9.0, 11.0, 13.0, 15.0, 17.0, 19.0, 21.0, 23.0!
a= 1.0,
3.0,
5.0,
7.0,
9.0, 11.0, 13.0,
a= 1.0,
3.0,
5.0,
7.0,
9.0, 11.0, 13.0, 15.0, 17.0, 19.0, 21.0, 23.0!
a= 1.0,
3.0,
5.0,
7.0,
9.0, 11.0, 13.0, 15.0, 17.0, 19.0, 21.0, 23.0
3.0,
5.0,
7.0,
9.0, 11.0!
44
Code Example in OpenMP

thread
0
0
0
1
1
1
2
2
2
3
3
computation
a[1] = a[0] + b[0]!
a[2] = a[1] + b[1]!
a[3] = a[2] + b[2] <--|
a[4] = a[3] + b[3] <--|
a[5] = a[4] + b[4]!
a[6] = a[5] + b[5] <--|
a[7] = a[6] + b[6] <--|
a[8] = a[7] + b[7]!
a[9] = a[8] + b[8] <--|
a[10] = a[9] + b[9] <--|
a[11] = a[10] + b[10]!
Problem !
Problem!
Problem !
Problem!
Problem !
Problem!
45
How to Avoid Data Races

Scope variables to be local to threads
Variables declared within threaded functions

Allocate on threads stack
TLS (Thread Local Storage)
Control shared access with critical regions
Mutual exclusion and synchronization

Lock, semaphore, event, critical section, mutex
Examples variables
47
Sequential Code:
!
int a[1000], i;
for (i = 0; i < 1000; i++) a[i] = foo(i);
Sequential Code:
!
int a[1000], i;
for (i = 0; i < 1000; i++) a[i] = foo(i);
Thread 0:
for (i = 0; i < 500; i++) a[i] = foo(i);
Thread 1:
for (i = 500; i < 1000; i++) a[i] = foo(i);
Sequential Code:
!
int a[1000], i;
for (i = 0; i < 1000; i++) a[i] = foo(i);
Thread 0:
for (i = 0; i < 500; i++) a[i] = foo(i);
Thread 1:
for (i = 500; i < 1000; i++) a[i] = foo(i);
Private
Shared
Task Decomposition
int e;
main () {
int x[10], j, k, m;
}
j = f(k);
m = g(k); ...
int f(int k)
{
int a;
a = e * x[k] * x[k];
}
return a;
int g(int k)
{
int a;
k = k-1;
}
a = e / x[k];
return a;
Task Decomposition
int e;
main () {
int x[10], j, k, m;
}
j = f(k);
int f(int k)
{
int a;
a = e * x[k] * x[k];
}
int g(int k)
{
int a;
k = k-1;
}
m = g(k);
Thread 0
return a;
Thread 1
a = e / x[k];
return a;
Task Decomposition
int e;
Static variable: Shared
main () {
int x[10], j, k, m;
}
j = f(k);
int f(int k)
{
int a;
a = e * x[k] * x[k];
}
int g(int k)
{
int a;
k = k-1;
}
m = g(k);
Thread 0
return a;
Thread 1
a = e / x[k];
return a;
Task Decomposition
int e;
Heap variable: Shared
main () {
int x[10], j, k, m;
}
j = f(x, k);
int f(int *x, int k)

{
int a;
a = e * x[k] * x[k];
}
int g(int *x, int k)

{
int a;
k = k-1;
}
m = g(x, k);
Thread 0
return a;
Thread 1
a = e / x[k];
return a;
Task Decomposition
int e;
main () {
int x[10], j, k, m;
j = f(k);
m = g(k);
}
Functions local variables: Private
int f(int k)
{
int a;
a = e * x[k] * x[k];
}
int g(int k)
{
int a;
k = k-1;
}
Thread 0
return a;
Thread 1
a = e / x[k];
return a;
Shared and Private Variables

Shared variables
Static variables
Heap variables
Contents of run-time stack at time of call
Private variables
Loop index variables
Run-time stack of functions invoked by thread
57
What Is OpenMP?
Compiler directives for multithreaded programming
Easy to create threaded Fortran and C/C++ codes
Supports data parallelism model

!
Portable and Standard
Incremental parallelism
Combines serial and parallel code in single source
OpenMP is not ...

!
Not Automatic parallelization !
User explicitly specifies parallel execution !
Compiler does not ignore user directives even if wrong !
!
!
Not just loop level parallelism !

Functionality to enable general parallel parallelism !
!
!
Not a new language !

Structured as extensions to the base !
Minimal functionality with opportunities for extension !
59
Directive based
Directives are special comments in the language !
Fortran fixed form: !$OMP, C$OMP, *$OMP !
Fortran free form: !$OMP !
!
Special comments are interpreted by OpenMP !
compilers !
w = 1.0/n !
sum = 0.0 !
!$OMP PARALLEL DO PRIVATE(x) REDUCTION(+:sum) !
do I=1,n !
x = w*(I-0.5) !
Comment in
sum = sum + f(x) !
Fortran
end do !
pi = w*sum !
but interpreted by
print *,pi !
OpenMP compilers
end !
60
C example
#pragma omp directives in C
!
Ignored by non-OpenMP compilers!
!
w = 1.0/n; !
sum = 0.0; !
#pragma omp parallel for private(x) reduction(+:sum) !
for(i=0, i<n, i++) { !
x = w*((double)i+0.5); !
sum += f(x); !
}!
pi = w*sum; !
printf(pi=%g\n, pi); !
}
61
Architecture of OpenMP
Directives,
Pragmas
Runtime library
routines
Control structures
Control & query routines
number of threads
Work sharing
throughput mode
Synchronization
nested parallism
Data scope attributes
Lock API
private
Environment
variables
Control runtime
schedule type
max threads
nested parallelism
throughput mode
shared
reduction
Orphaning
62
Programming Model
Fork-join parallelism:
Master thread spawns a team of threads as needed
Parallelism is added incrementally: the sequential program
evolves into a parallel program
Master
Thread
Parallel Regions
63
Work-sharing Construct
T
#pragma omp parallel
#pragma omp for
for(i = 0; i < 12; i++)
c[i] = a[i] + b[i]
Threads are assigned an independent
set of iterations
Threads must wait at the end of worksharing construct
#pragma omp for

i=0
i=4
i=8
i=1
i=5
i=9
i=2
i=6
i = 10
i=3
i=7
i = 11
Implicit barrier
64
Combining pragmas
These two code segments are equivalent
{
#pragma omp for
for (i=0; i< MAX; i++) {
res[i] = huge();
}
}
#pragma omp parallel for
for (i=0; i< MAX; i++) {
res[i] = huge();
}
65
IWOMP 2009
TU Dresden
June 3-5, 2009
18
Example
Matrix
times
vector
Matrix-vector example
! "
#!$$ % $$$$

j

=
*

i

TID = 0
TID = 1
for (i=0,1,2,3,4)
i = 0
sum = b[i=0][j]*c[j]
a[0] = sum
i = 1
for (i=5,6,7,8,9)
i = 5
a[5] = sum
i = 6
a[1] = sum
a[6] = sum
... etc ...

RvdP/V1
An Overview of OpenMP 3.0
66
Tutorial IWOMP 2009 TU Dresden, June 3, 2009
MP 2009
19
OpenMP
Performance
Example
Performance is matrix size dependent

Matrix too
small *
scales
Performance (Mflop/s)
Dresden
3-5, 2009

Memory Footprint (KByte)

*) With the IF-clause in OpenMP this performance degradation can 67
be avoided
OpenMP parallelization
OpenMP Team := Master + Workers
A Parallel Region is a block of code executed by all
threads simultaneously
The master thread always has thread ID 0
Thread adjustment (if enabled) is only done before entering a parallel
region
Parallel regions can be nested, but support for this is implementation
dependent
An "if" clause can be used to guard the parallel region; in case the
condition evaluates to "false", the code is executed serially
A work-sharing construct divides the execution of the enclosed

code region among the members of the team; in other words:
they split the work
68
Data Environment
OpenMP uses a shared-memory programming model
Most variables are shared by default.

Global variables are shared among threads
C/C++: File scope variables, static
Not everything is shared, there is often a need for local data as

well
69
Data Environment
... not everything is shared...
Stack variables in functions called from parallel regions are PRIVATE

Automatic variables within a statement block are PRIVATE
Loop index variables are private (with exceptions)
C/C+: The first loop index variable in nested loops following a

#pragma omp for
70
About Variables in SMP

!
Shared variables
Can be accessed by every thread thread. Independent read/write
operations can take place.
Private variables
Every thread has its own copy of the variables that are created/
destroyed upon entering/leaving the procedure. They are not
visible to other threads.
serial code
global
auto local
static
dynamic
parallel code
shared
local
use with care
use with care
71
Data Scope clauses

attribute clauses
!
!
default(shared)
!
!
shared(varname,)
!
!
!
private(varname,)
72
The Private Clause

Reproduces the variable for each thread
Variables are un-initialised; C++ object is default constructed

Any value external to the parallel region is undefined
void* work(float* c, int N)

{
float x, y; int i;
#pragma omp parallel for private(x,y)
for(i=0; i<N; i++) {
x = a[i]; y = b[i];
c[i] = x + y;
}
}
73
Synchronization
Barriers
!
!
Critical sections
!
!
#pragma omp barrier

#pragma omp critical()
Lock library routines
omp_set_lock(omp_lock_t *lock)!
!
omp_unset_lock(omp_lock_t *lock)!
!
....
74
OpenMP Critical Construct

#pragma omp critical [(lock_name)]
Defines a critical region on a structured block

All threads execute the
code, but only one at a
time. Only one calls
consum() thereby
protecting R1 and R2
from race conditions.
Naming the critical
constructs is optional,
but may increase
performance.
float R1, R2;

{ float A, B;
#pragma omp for
for(int i=0; i<niters; i++){
B = big_job(i);
#pragma omp critical(R1_lock)
consum (B, &R1);!

!
A = bigger_job(i);
#pragma omp critical(R2_lock)
consum (A, &R2);

}
}
75
read at a time.
t wait until the current thread exits the critical
OpenMP
an manipulate
values Critical
in the critical region.
fork
x)
- critical region
join
All threads execute the code, but only one at a time.

Day 3: OpenMP
29
Other threads in the group must
2010 wait
Courseuntil
MT1 the current
thread exits the critical region. Thus only one thread can
manipulate values in the critical region.
76
Simple Example: critical

Critical Example
cnt=0
f=7
cnt = 0;
f = 7;
{
#pragma omp for
for (i=0;i<20;i++){
if(b[i] == 0){
#pragma omp critical
cnt ++;
} /* end if */
a[i]=b[i]+f*(i+1);
} /* end for */
} /* omp end parallel */
i =0,4
i=5,9
i= 10,14
i= 20,24
if
i= 20,24
if
if
cnt++
cnt++
a[i]=b[
i]+
a[i]=b[
i]+
cnt++
cnt++
a[i]=b[i]
+
a[i]=b[i]
+
Day 3:77
OpenMP
2010 Course MT1
30
OpenMP Single Construct

Only one thread in the team executes the enclosed code
The Format is:
!
!
#pragma
omp single [nowait][clause, ..]{!
!
block!
!
}
!
The supported clauses on the single directive are:
private (list)!
firstprivate (list)
NOWAIT:
the other threads
will not wait at the
end single directive
78
OpenMP Master directive

!
#pragma
omp master {!
!
code!
!
}
!
All threads but the master, skip the enclosed section of code and
continue
There is no implicit barrier on entry or exit !
!
!
#pragma
omp barrier!
!
Each thread waits until all others in the team have reached this
point.
79
Work Sharing: Single Master

#pragma omp
{!
....!
#pragma omp
{!
....!
}!
#pragma omp
{!
....!
}!
....!
#pragma omp
}
parallel!
single [nowait]!
master!
barrier!
80
Work Sharing: Orphaning

Worksharing constructs may be outside lexical scope of the
parallel region
#pragma omp parallel!
{!
....!
dowork( )!
....!
}!
....
void dowork( )!
{!
#pragma omp for!
for (i=0; i<n; i++) {!
....!
}!
}!
81
Scheduling the work
OMP 2009
U Dresden
e 3-5, 2009
66
schedule ( static | dynamic | guided | auto [, chunk] ) schedule

(runtime)
static
[, chunk]! clause/2
The
schedule
Distribute iterations in blocks of size "chunk" over the threads in a

round-robin fashion
Example
static
schedule
In absence
of "chunk",
each thread executes approx. N/P chunks for
Loopa of
length
16,N 4and
threads:
loop
of length
P threads
Thread

0
*-K
1
L-M
2
N-*7
3
*F-*O

!
*-7
N-*
F-K
**-*7
L-O
*F-*K
P-M
*L-*O
82
dynamic [, chunk]!
Fixed portions of work; size is controlled by the value of chunk
When a thread finishes, it starts on the next portion of work
!
guided [, chunk]!
Same dynamic behavior as "dynamic", but size of the portion of work
decreases exponentially
!
runtime!
Iteration scheduling scheme is set at runtime through environment
variable OMP_SCHEDULE
83
IWOMP 2009
TU Dresden
June 3-5, 2009
68
The experiment
Example
scheduling
500 iterations on 4 threads
3
2
guided, 5
Thread ID
1
0
3
2
1
0
dynamic, 5
3
2
1
0
static
0
100
200
300
400
500
600
Iteration Number
RvdP/V1
An Overview of OpenMP 3.0
84
Tutorial IWOMP 2009 TU Dresden, June 3, 2009
Exercise: OpenMP scheduling

How OpenMP can help parallelize a loop, but as a user you can
help in making it better ;-)
Download code from:

http://www.xs4all.nl/~janth/HPCourse/OMP_schedule.tar
Unpack tar file and check the README for instructions.
This exercise requires already some knowledge about OpenMP.

The OpenMP F-Summary.pdf from the website can be helpful.
85
OpenMP Reduction Clause

reduction (op : list)
The variables in list must be shared in the enclosing parallel

region
Inside parallel or work-sharing construct:
13
A PRIVATE copy of each list variable is created and initialized depending on

the op
These copies are updated locally by threads
At end of construct, local copies are combined through op into a single value
and combined with the value in the original SHARED variable
ReductionOpenMP:
Example
Reduction
performs
reductionomp
on shared
variables in for
list based on the operator provided.
#pragma
parallel
reduction(+:sum)
for C/C++
operator can be any one of :
i<N; i++) {
+, *, -, for(i=0;
^, |, ||, & or &&
At the end sum
of a reduction,
the shared
contains the result obtained upon
+= a[i]
* variable
b[i];
combination of the list of variables processed using the operator specified.
}
sum=0
sum = 0.0
Localomp
copy
of sum for
for each
thread
#pragma
parallel
reduction(+:sum)
All(i=0;
local copies
of sum
for
i < 20;
i++) added together
and
stored in global variable
sum = sum + (a[i] * b[i]);
i=0,4
i=5,9
i=10,14
i=15,19
sum=..
sum=..
sum=..
sum=..
sum
sum=0
87
C/C++ Reduction Operations

A range of associative and commutative operators can be used with
reduction
Initial values are the ones that make sense
Operator
Initial Value
Operator
Initial Value
&
~0
&&
||
FORTRAN:

intrinsic is one of MAX, MIN, IAND, IOR, IEOR

operator is one of +, *, -, .AND., .OR., .EQV., .NEQV.

88
Numerical Integration Example

1
4.0
4
dx =
2
1+x
static
long num_steps=100000;
double step, pi;
void main()
{ int i;
double x, sum = 0.0;
2.0
0.0
TT
1.0
step = 1.0/(double) num_steps;

for (i=0; i<num_steps; i++){
x = (i+0.5)*step;
sum = sum + 4.0/(1.0 + x*x);
}
pi = step * sum;
printf(Pi = %f\n,pi);
89
Numerical Integration to Compute Pi

static long num_steps=100000;
double step, pi;
void main()
{ int i;

x = (i+0.5)*step;
sum = sum + 4.0/(1.0 + x*x);
}
pi = step * sum;
Parallelize the numerical

integration code using
OpenMP
What variables can be shared?
step, num_steps
x, i
What variables need to be

private?
What variables should be set

up for reductions?
sum
90
Solution to Computing Pi
static long num_steps=100000;
double step, pi;
void main()
{ int i;
#pragma omp parallel for private(x) reduction(+:sum)
x = (i+0.5)*step;
sum = sum + 4.0/(1.0 + x*x);
}
pi = step * sum;
}
91
Lets try it out

Go to example MPI_pi.tar and work with openmp_pi2.c
92
Exercise: PI with MPI and OpenMP

cores
OpenMP
9.617728
4.874539
2.455036
1.627149
1.214713
12
0.820746
16
0.616482
93

Pi Scaling
16
Amdahl 1.0
OpenMP Barcelona 2.2 GHz
14
speed up
12
10
8
6
4
2
0
1
12
16
number of CPUs
94
Race condition code example

MPI_pi.tar example and replace
!
#pragma omp parallel for private(x) reduction(+:sum) !
with
!
#pragma omp parallel for private(x) shared(sum)!
solve with!
!
#pragma omp critical before sum
95
Multi-PF Solutions
Cuda Computing PI
CUDA: computing PI
96
2007 IBM Corporation
Environment Variables
The names of the OpenMP environment variables must be
UPPERCASE
The values assigned to them are case insensitive
OMP_NUM_THREADS!
!
OMP_SCHEDULE schedule [chunk]!
!
OMP_NESTED { TRUE | FALSE }!
!
97
Exercise: Shared Cache Trashing

Lets do the exercise.
98
About local and shared data

Consider the following example:
!
!
!
!
!
!
for (i=0; i<10; i++){!

a[i] = b[i] + c[i];!
}
Lets assume we run this on 2 processors:
!
processor 1 for i=0,2,4,6,8
99

Processor 1
Processor 2
for (i1=0,2,4,6,8){!
a[i1] = b[i1] + c[i1];!
}
for (i2=1,3,5,7,9){!
a[i2] = b[i2] + c[i2];!
}
private area
shared area
A
i1
P1
private area
i2
P2
100

This is not an efficient way to do this!
!
!
!
!
Why?
101
Doing it the bad way

Because of cache line usage
!
!
!
!
!
!
!
b[] and c[]:
a[]:
for (i=0; i<10; i++){!

a[i] = b[i] + c[i];!
}
we use half of the data

false sharing
102
False sharing and scalability

The Cause:
Updates on independent data elements that happen to be
part of the same cache line.
The Impact:
Non-scalable parallel applications
The Remedy:
False sharing is often quite simple to solve
103
Poor cache line utilization

Processor 2
B(0)
B(0)
B(1)
B(1)
B(2)
B(2)
B(3)
B(3)
B(4)
B(4)
B(5)
B(5)
B(6)
B(6)
B(7)
B(7)
B(8)
B(8)
B(9)
B(9)
used data
not used data
cache line
Processor 1
Both processors read

the same cache lines
The same holds for array C
104
False Sharing
Processor 1
Processor 2
a[0] = b[0] + c[0];
a[1] = b[1] + c[0];
Write into the line containing a[0]

This marks the cache line
containing a[0] as dirty
Detects the line with a[0] is dirty

Get a fresh copy (from processor 1)
a[2] = b[2] + c[2];

Get a fresh copy (from processor 2)
a[3] = b[3] + c[3];
time
105
Summary
First tune single-processor performance
Tuning parallel programs

Has the program been properly parallelized?
Is enough of the program parallelized (Amdahls law)?
Is the load well-balanced?
location of memory
Cache friendly programs: no special placement needed
Non-cache friendly programs
False sharing?
Use of OpenMP
try to avoid synchronization (barrier, critical, single, ordered)
106
Plenty of Other OpenMP Stuff

Scheduling clauses
Atomic
Barrier
Master & Single
Sections
Tasks (OpenMP 3.0)
API routines
19
Compiling and running OpenMP

Compile with -openmp flag (intel compiler) or -fopenmp (GNU)
Run program with variable:
!
export OMP_NUM_THREADS=4!
108
OpenACC
New set of directives to support accelerators
GPUs
Intels MIC
AMD Fusions processors
109
OpenACC example
void convolution_SM_N(typeToUse A[M][N], typeToUse B[M][N]) {
int i, j, k;
int m=M, n=N;
// OpenACC kernel region
// Define a region of the program to be compiled into a sequence of kernels
// for execution on the accelerator device
#pragma acc kernels pcopyin(A[0:m]) pcopy(B[0:m])
{
typeToUse c11, c12, c13, c21, c22, c23, c31, c32, c33;

c11 = +2.0f; c21 = +5.0f; c31 = -8.0f;
c12 = -3.0f; c22 = +6.0f; c32 = -9.0f;
c13 = +4.0f; c23 = +7.0f; c33 = +10.0f;

// The OpenACC loop gang clause tells the compiler that the iterations of the loops
// are to be executed in parallel across the gangs.
// The argument specifies how many gangs to use to execute the iterations of this loop.
#pragma acc loop gang(64)
for (int i = 1; i < M - 1; ++i) {

// The OpenACC loop worker clause specifies that the iteration of the associated loop are
to be
// executed in parallel across the workers within the gangs created.
// The argument specifies how many workers to use to execute the iterations of this loop.
#pragma acc loop worker(128)
for (int j = 1; j < N - 1; ++j) {
B[i][j] = c11 * A[i - 1][j - 1] + c12 * A[i + 0][j - 1] + c13 * A[i + 1][j - 1]
+ c21 * A[i - 1][j + 0] + c22 * A[i + 0][j + 0] + c23 * A[i + 1][j + 0]
+ c31 * A[i - 1][j + 1] + c32 * A[i + 0][j + 1] + c33 * A[i + 1][j +
1];
}
}
110
}//kernels region
}
MPI
111
Message Passing
Point-to-Point
Requires explicit commands in program

Send, Receive
Must be synchronized among different processors

Sends and Receives must match
Avoid Deadlock -- all processors waiting, none able to communicate
Multi-processor communications
e.g. broadcast, reduce
MPI advantages
Mature and well understood
Backed by widely-supported formal standard (1992)
Porting is easy
Efficiently matches the hardware

Vendor and public implementations available
User interface:
Efficient and simple
Buffer handling
Allow high-level abstractions
Performance
113
MPI disadvantages
MPI 2.0 includes many features beyond message passing
!
!
!
!
!
!
!
!
!
ve
r
u
c
g
n
rni
a
e
L
Execution control environment depends on implementation

114
Programming Model
Explicit parallelism:
All processes starts at the same time at the same point in the code
Full parallelism: there is no sequential part in the program
T
processes
Parallel Region
5
Work Distribution
All processors run the same executable.
Parallel work distribution must be explicitly done by the

programmer:
domain decomposition
master slave
116
A Minimal MPI Program (C)

#include "mpi.h"!
#include <stdio.h>!
!
int main( int argc, char *argv[] )!
{!
MPI_Init( &argc, &argv );!
printf( "Hello, world!\n" );!
MPI_Finalize();!
return 0;!
}
117
A Minimal MPI Program (Fortran 90)

program main!
use MPI!
integer ierr!
!
call MPI_INIT( ierr )!
print *, 'Hello, world!'!
call MPI_FINALIZE( ierr )!
end
118
Starting the MPI Environment

MPI_INIT ( )
Initializes MPI environment. This function must be called and

must be the first MPI function called in a program (exception:
MPI_INITIALIZED)
Syntax!
int MPI_Init (
int *argc, char ***argv )!
!
MPI_INIT ( IERROR )!
INTEGER IERROR
NOTE: Both C and Fortran return error codes for all calls.
119
Exiting the MPI Environment

MPI_FINALIZE ( )
Cleans up all MPI state. Once this routine has been called, no
MPI routine ( even MPI_INIT ) may be called
!
Syntax!
int MPI_Finalize ( );!
!
MPI_FINALIZE
( IERROR )!
INTEGER IERROR!
!
MUST call MPI_FINALIZE when you exit from an MPI program
120
C and Fortran Language

Considerations
Bindings
C
All MPI names have an MPI_ prefix
Defined constants are in all capital letters
Defined types and functions have one capital letter after the
prefix; the remaining letters are lowercase
Fortran
All MPI names have an MPI_ prefix
No capitalization rules apply
last argument is an returned error value
121
Finding Out About the Environment

Two important questions that arise early in a parallel program are:
How many processes are participating in this computation?
Which one am I?
MPI provides functions to answer these questions:

MPI_Comm_size reports the number of processes.
MPI_Comm_rank reports the rank, a number between 0 and size-1,
identifying the calling process
122
Better Hello (C)

#include "mpi.h"!
#include <stdio.h>!
int main( int argc, char *argv[] )!

{!
int rank, size;!
MPI_Init( &argc, &argv );!
MPI_Comm_rank( MPI_COMM_WORLD, &rank );!
MPI_Comm_size( MPI_COMM_WORLD, &size );!
printf( "I am %d of %d\n", rank, size );!
MPI_Finalize();!
return 0;!
}
123
Better Hello (Fortran)

program main!
use MPI!
integer ierr, rank, size!
!
call MPI_COMM_RANK( MPI_COMM_WORLD, rank, ierr )!
call MPI_COMM_SIZE( MPI_COMM_WORLD, size, ierr )!
print *, 'I am ', rank, ' of ', size!
end
124
Some Basic Concepts

Processes can be collected into groups.
Each message is sent in a context, and must be received in the
same context.
communicator.
A group and context together form
MPIaCommunicators
by its rank in the group associated with a
A process is identified
Communicator is an internal object
MPI Programs are made up of communicating
communicator.
processes
There is a default communicator
group
contains
Each process has itswhose
own address
space containing
its all initial
own attributes such as rank, size (and argc, argv, etc.)
processes, called MPI_COMM_WORLD.
MPI provides functions to interact with it
Default communicator is MPI_COMM_WORLD
MPI_COMM_WORLD
All processes are its members

0
It has a size (the number of processes)
Each process has a rank within it
2
One can think of it as an ordered list of processes
Additional communicator(s) can co-exist

A process can belong to more than one communicator
Within a communicator, each process has a unique
rank
5
3
1
6
7
125
Day 2: MPI
2010 Course MT1
14
Communicator
Communication in MPI takes place with respect to
communicators
MPI_COMM_WORLD is one such predefined communicator
(something of type MPI_COMM) and contains group and
context information
MPI_COMM_RANK and MPI_COMM_SIZE return
information based on the communicator passed in as the
first argument
Processes may belong to many different communicators
Rank-->
MPI_COMM_WORLD
126
MPI Basic Send/Receive
Basic message passing process. Send data from one process

to another
Process 0
Process 1
A:
Send
Receive
B:
Questions
To whom is data sent?

Where is the data?
What type of data is sent?
How much of data is sent?
How does the receiver identify it?
127
Data transfer plus synchronization

Process 0
Data
May I Send?
Process 1
Yes
Data
Data
Data
Data
Data
Data
Data
Data
Time
Requires co-operation of sender and receiver

Co-operation not always apparent in code
Communication and synchronization are combined
128
Message Organization in MPI

Envelope
Message is Message
divided into data
and envelope
!
Communication
across
data
Source : process0
processes
is
performed
using
Destination : process1
buffer
Tag : 1234
messages.
count
Communicator : MPI_COMM_WORLD
Each message
datatype consists of a
fixed number
of fields that is
!
used
to distinguish them, called
envelope
MPI_COMM_WORLD
the Message
Envelope
:
process identifier (source/destination)
Envelope comprises source,
messagetag,
tag communicator
destination,
communicator
Message
comprises Envelope +
data
Communicator refers to the

namespace associated with the
group of related processes
3
2
129
6
7

Thus the basic (blocking) send has become:
MPI_Send ( start, count, datatype, dest, tag,
comm )!
Blocking means the function does not return until it is safe to reuse
the data in buffer. The message may not have been received by the
target process.
!
And the receive has become:
MPI_Recv( start, count, datatype, source, tag,
comm, status )!
!
- The source, tag, and the count of the message actually
received can be retrieved from status
130
MPI C Datatypes
131
MPI Fortran Datatypes
132
MPI Tags
Messages are sent with an accompanying user-defined integer
tag, to assist the receiving process in identifying the message.
Messages can be screened at the receiving end by specifying a

specific tag, or not screened by specifying MPI_ANY_TAG as the
tag in a receive.
133
Process Naming and Message Tags
Naming a process
destination is specified by ( rank, group )!
Processes are named according to their rank in the group
Groups are defined by their distinct communicator
MPI_ANY_SOURCE wildcard rank permitted in a receive Tags are
integer variables or constants used to uniquely identify individual
messages
Tags allow programmers to deal with the arrival of messages in

an orderly manner
MPI tags are guaranteed to range from 0 to 32767 by MPI-1

Vendors are free to increase the range in their implementations
MPI_ANY_TAG can be used as a wildcard value

134
Retrieving Further Information

Status is a data structure allocated in the users program.
In C:
int recvd_tag, recvd_from, recvd_count;
MPI_Status status;
MPI_Recv(..., MPI_ANY_SOURCE, MPI_ANY_TAG, ..., &status )
recvd_tag
= status.MPI_TAG;
recvd_from = status.MPI_SOURCE;
MPI_Get_count( &status, datatype, &recvd_count );
In Fortran:
integer recvd_tag, recvd_from, recvd_count
integer status(MPI_STATUS_SIZE)
call MPI_RECV(..., MPI_ANY_SOURCE, MPI_ANY_TAG, .. status, ierr)
tag_recvd
= status(MPI_TAG)
recvd_from = status(MPI_SOURCE)
call MPI_GET_COUNT(status, datatype, recvd_count, ierr)
135
Getting Information About a Message

(Fortran)
The following functions can be used to get information about a
message
INTEGER status(MPI_STATUS_SIZE)
call MPI_Recv( . . . , status, ierr )
tag_of_received_message = status(MPI_TAG)
src_of_received_message = status(MPI_SOURCE)
call MPI_Get_count(status, datatype, count, ierr)
MPI_TAG and MPI_SOURCE are primarily of use when
MPI_ANY_TAG and/or MPI_ANY_SOURCE is used in the receive
The function MPI_GET_COUNT may be used to determine how much
data of a particular type was received
136
Simple Fortran Example - 1/3

program main!
use MPI!
!
integer rank, size, to, from, tag, count, i, ierr!
integer src, dest!
integer st_source, st_tag, st_count!
integer status(MPI_STATUS_SIZE)!
double precision data(10) !
!
call MPI_COMM_RANK( MPI_COMM_WORLD, rank, ierr )!
call MPI_COMM_SIZE( MPI_COMM_WORLD, size, ierr )!
print *, 'Process ', rank, ' of ', size, ' is alive'!
dest = size - 1!
src = 0
137

if (rank .eq. 0) then!
do i=1,10!
data(i) = i!
done!
call MPI_SEND( data, 10, MPI_DOUBLE_PRECISION,!
+
dest, 2001, MPI_COMM_WORLD, ierr)!
!
else if (rank .eq. dest) then!
tag = MPI_ANY_TAG!
source = MPI_ANY_SOURCE!
call MPI_RECV( data, 10, MPI_DOUBLE_PRECISION,!
+
source, tag, MPI_COMM_WORLD,!
+
status, ierr)
138

call MPI_GET_COUNT( status, MPI_DOUBLE_PRECISION,!
st_count, ierr )!
st_source = status( MPI_SOURCE )!
st_tag
= status( MPI_TAG )!
print *, 'status info: source = ', st_source,!
+
' tag = ', st_tag, 'count = ', st_count!
endif!
!
end
139
Is MPI Large or Small?
Is MPI large (128 functions) or small (6 functions)?

MPIs extensive functionality requires many functions
Number of functions not necessarily a measure of complexity
Many programs can be written with just 6 basic functions
MPI_INIT! !
MPI_FINALIZE!
MPI_COMM_SIZE!
MPI_COMM_RANK!
MPI_SEND!
MPI_RECV!
MPI is just right

A small number of concepts
Large number of functions provides flexibility, robustness,
efficiency, modularity, and convenience
One need not master all parts of MPI to use it
140
Example: PI in Fortran 90 and C

#include "mpi.h"
#include <math.h>
int main(int argc, char *argv[])
{

int done = 0, n, myid, numprocs, i, rc;

double PI25DT = 3.141592653589793238462643;

double mypi, pi, h, sum, x, a;

MPI_Init(&argc,&argv);

MPI_Comm_size(MPI_COMM_WORLD,&numprocs);

MPI_Comm_rank(MPI_COMM_WORLD,&myid);

while (!done) {

if (myid == 0) {

printf("Enter the number of intervals: (0 quits) ");

scanf("%d",&n);

}

MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);

if (n == 0) break;

h
= 1.0 / (double) n;

sum = 0.0;

for (i = myid + 1; i <= n; i += numprocs) {

x = h * ((double)i - 0.5);

sum += 4.0 / (1.0 + x*x);

}

mypi = h * sum;

MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0,MPI_COMM_WORLD);

if (myid == 0)

printf("pi is approximately %.16f, Error is %.16f\n",pi, fabs(pi - PI25DT));

}

MPI_Finalize();

return 0;
}
work distribution
141
10
98
program main
use MPI
double precision PI25DT
parameter (PI25DT = 3.141592653589793238462643d0)
double precision mypi, pi, h, sum, x, f, a
integer n, myid, numprocs, i, ierr
function to integrate
f(a) = 4.d0 / (1.d0 + a*a)
call MPI_INIT( ierr )
call MPI_COMM_RANK( MPI_COMM_WORLD, myid, ierr )
call MPI_COMM_SIZE( MPI_COMM_WORLD, numprocs, ierr )
if ( myid .eq. 0 ) then
write(6,98)
format('Enter the number of intervals: (0 quits)')
read(5,(i10)) n
endif
call MPI_BCAST( n, 1, MPI_INTEGER, 0,MPI_COMM_WORLD, ierr)
if ( n .le. 0 ) goto 30
h = 1.0d0/n
sum = 0.0d0
do 20 i = myid+1, n, numprocs
x
= h * (dble(i) - 0.5d0)
sum = sum + f(x)
continue
mypi = h * sum
call MPI_REDUCE( mypi, pi, 1, MPI_DOUBLE_PRECISION,
+
MPI_SUM, 0, MPI_COMM_WORLD,ierr)
if (myid .eq. 0) then
write(6, 97) pi, abs(pi - PI25DT)
format(' pi is approximately: ', F18.16,
+
' Error is: ', F18.16)
endif
goto 10
call MPI_FINALIZE(ierr)
end
work distribution
20
97
30
!
!
!
142

Compile and run for computing PI parallel
Download code from:

http://www.xs4all.nl/~janth/HPCourse/MPI_pi.tar
It is assumed that you use mpich1 and has PBS installed as a job
scheduler. If you have mpich2 let me know.
Use qsub to submit the mpi job (an example script is provided) to
a queue.
143

cores
OpenMP
marken
linux
9.617728
14.10798
22.15252
4.874539
7.071287
9.661745
2.455036
3.532871
5.730912
1.627149
2.356928
3.547961
1.214713
1.832055
2.804715
12
0.820746
1.184123
16
0.616482
0.955704
144

Pi Scaling
16
Amdahl 1.0
MPI marken Xeon 3.1 GHz
MPI linux Xeon 2.4 GHz
14
speed up
12
10
8
6
4
2
0
1
6
number of CPUs
12
16
145

Pi Pi
Scaling
Scaling
OpenMP
MPI
16
30
14
Amdahl
Amdahl1.0
1.0
OpenMP
MPI Core
Xeon
i7-950
E5504
3.06
2.0GHz
GHz
speed up
25
12
20
10
8
15
6
10
4
5
2
4
1
26
8 4
12 6
16 8
number of cores
12
32
16
146
147
Blocking Communication
So far we have discussed blocking communication
MPI_SEND does not complete until buffer is empty (available for reuse)
MPI_RECV does not complete until buffer is full (available for use)
A process sending data will be blocked until data in the send buffer is
emptied
A process receiving data will be blocked until the receive buffer is filled
Completion of communication generally depends on the message size
and the system buffer size
Blocking communication is simple to use but can be prone to
deadlocks
If (my_proc.eq.0) Then
Call mpi_send(..)
Call mpi_recv()
Usually deadlocks-->
Else
Call mpi_send()
<--- UNLESS you reverse send/recv
Call mpi_recv(.)
Endif
148
Sources of Deadlocks
Send a large message from process 0 to process 1
If there is insufficient storage at the destination, the send must
wait for the user to provide the memory space (through a
receive)
What happens with
Process 0
Send(1)
Recv(1)
Process 1
Send(0)
Recv(0)
This is called unsafe because it depends on the availability

of system buffers.
149
Some Solutions to the unsafe

Problem
Order the operations more carefully:
Process 0
Process 1
Send(1)
Recv(1)
Recv(0)
Send(0)
Use non-blocking operations:
Process 0
Process 1
Isend(1)
Irecv(1)
Waitall
Isend(0)
Irecv(0)
Waitall
150
Blocking Send-Receive Diagram

(Receive before Send)
It is important to receive before sending,
for highest performance.
T0: MPI_Recv
Once receive
is called @ T0,
buffer unavailable
to user
T1:MPI_Send
sender
T2
returns
@ T2,
buffer can
be reused
T3: Transfer Complete

T4
send side
Internal completion is soon

followed by return of MPI_Recv
Receive
returns @ T4,
buffer filled
receive side
151
Non-Blocking Send-Receive Diagram

High Performance Implementations
Offer Low Overhead for Non-blocking Calls
Once receive is called @ T0,
buffer unavailable to user
T0: MPI_Irecv
T2: MPI_Isend
T1: Returns
sender
T3
returns @ T3
buffer unavailable
sender
completes @ T5
buffer available
after MPI_Wait
T4: MPI_Wait called
T5
T6
T7: transfer
!
T8
finishes
T6: MPI_Wait
T9: Wait returns
Internal completion is soon
followed by return of MPI_Wait
send side
MPI_Wait, returns @ T8
here, receive buffer filled
receive side152
Non-Blocking Communication
Non-blocking (asynchronous) operations return (immediately)

request handles that can be waited on and queried
MPI_ISEND( start, count, datatype, dest, tag, comm,
request )!
MPI_IRECV( start, count, datatype, src, tag, comm,
request )!
MPI_WAIT( request, status )!
Non-blocking operations allow overlapping computation and

communication.
One can also test without waiting using MPI_TEST!
MPI_TEST( request, flag, status )!
Anywhere you use MPI_Send or MPI_Recv, you can use the

pair of MPI_Isend/MPI_Wait or MPI_Irecv/MPI_Wait!
Combinations of blocking and non-blocking sends/receives can be
used to synchronize execution instead of barriers
153
Multiple Completions
It is often desirable to wait on multiple requests
An example is a worker/manager program, where the manager waits

for one or more workers to send it a message
MPI_WAITALL( count, array_of_requests, array_of_statuses
)!
MPI_WAITANY( count, array_of_requests, index, status )!
MPI_WAITSOME( incount, array_of_requests, outcount,
array_of_indices, array_of_statuses )!
There are corresponding versions of test for each of these
154
Probing the Network for Messages

MPI_PROBE and MPI_IPROBE allow the user to check for
incoming messages without actually receiving them
MPI_IPROBE returns flag == TRUE if there is a matching

message available. MPI_PROBE will not return until there is a
matching receive available
MPI_IPROBE (source, tag, communicator, flag,
status)
MPI_PROBE ( source, tag, communicator, status )
155
Message Completion and Buffering

A send has completed when the user supplied buffer can be
reused
*buf = 3;
MPI_Send ( buf, 1, MPI_INT, ... );
*buf = 4; /* OK, receiver will always receive 3 */
*buf = 3;
MPI_Isend(buf, 1, MPI_INT, ...);
*buf = 4; /* Undefined whether the receiver will get 3 or 4 */
MPI_Wait ( ... );
The send mode used (standard, ready, synchronous, buffered)

may provide additional information
Just because the send completes does not mean that the
receive has completed
Message may be buffered by the system
Message may still be in transit
156
Collective Communications in MPI
Communication is co-ordinated among a group of processes,

as specified by communicator, not on all processes
All collective operations are blocking and no message tags are

used (in MPI-1)
All processes in the communicator group must call the

collective operation
Collective and point-to-point messaging are separated by

different contexts
Three classes of collective operations

Data movement
Collective computation
Synchronization
157
Collective calls
MPI has several collective communication calls, the most frequently
used are:
Synchronization
Barrier
Communication
Broadcast
Gather Scatter
All Gather
Reduction
Reduce
All Reduce
158
MPI Collective Calls: Barrier

MPI_Barrier()
P0
P1
comm )
P2
ronization in a
comm. Each process,
PI_Barrier call, blocks
in the group reach the
l.
P3
MPI_Barrier()
PI_Barrier()
P0
P1
P2
P3
Creates barrier synchronization in a communicator group comm.

Each process, when reaching the MPI_Barrier call, blocks until all
the processes in the group reach the same MPI_Barrier call.
mcs.anl.gov/mpi/www/www3/MPI_Barrier.html
Day 2: MPI
2010 Course MT1
40
159
MPI Basic Collective Operations

Two simple collective operations
MPI_BCAST( start, count, datatype, root, comm )!
MPI_REDUCE( start, result, count, datatype, operation,
root, comm )!
The routine MPI_BCAST sends data from one process to all

others
The routine MPI_REDUCE combines data from all processes,
using a specified operation, and returns the result to a single
process
In many numerical algorithms, SEND/RECEIVE can be replaced
by BCAST/REDUCE, improving both simplicity and efficiency.
160
Broadcast and Reduce

Process Send
Ranks buffer
1
2
Process Send
Ranks buffer
A
A
1
2
Bcast (root=0)
Process Send
Ranks buffer
1
2
B
C
Process Receive
Ranks buffer
Reduce (root=0, op)

X=A op B op C op D
1
2
161
Scatter and Gather

Process Send
Ranks buffer
0 ABCD
Process Receive
Ranks buffer
Scatter (root=0)
????
????
????
Process Send
Ranks buffer
2
3
Process Receive
Ranks buffer
Gather (root=0)
0 ABCD
1
????
????
????
162
MPI Collective Routines

Several routines:
MPI_ALLGATHER!
MPI_ALLTOALL!
! MPI_GATHER!
! MPI_REDUCE_SCATTER!
! MPI_SCATTERV!
MPI_ALLGATHERV! MPI_BCAST
MPI_ALLTOALLV!
MPI_REDUCE!
MPI_GATHERV!
MPI_SCATTER!
MPI_SCAN!
!
MPI_ALLREDUCE!
All versions deliver results to all participating processes

V versions allow the chunks to have different sizes
MPI_ALLREDUCE, MPI_REDUCE, MPI_REDUCE_SCATTER, and

MPI_SCAN take both built-in and user-defined combination
functions
163
Built-In Collective Computation

Operations
164
Timing MPI Programs

MPI_WTIME returns a floating-point number of seconds,
representing elapsed wall-clock time since some time in the past
double MPI_Wtime( void )

DOUBLE PRECISION MPI_WTIME( )
MPI_WTICK returns the resolution of MPI_WTIME in seconds.
It returns, as a double precision value, the number of seconds
between successive clock ticks.
double MPI_Wtick( void )

DOUBLE PRECISION MPI_WTICK( )
165
Extending the Message-Passing

Interface
Dynamic Process Management
Dynamic process startup

Dynamic establishment of connections
One-sided communication
Put/get
Other operations
Parallel I/O
Other MPI-2 features
Generalized requests
Bindings for C++/ Fortran-90; inter-language issues
166
When to use MPI

Portability and Performance
Irregular Data Structures
Building Tools for Others

Libraries
Need to Manage memory on a per processor basis
167
When not to use MPI

Number of cores is limited and OpenMP is doing well on that
number of cores
typically 16-32 cores in SMP
168
for example: wave equation in a 2D domain.
Break up the domain into blocks.

Assign blocks to MPI-processes one-to-one.
Provide a "map" of neighbors to each process.
Write or modify your code so it only updates a single block.
Insert communication subroutine calls where needed.
Adjust the boundary conditions code.
Use "guard cells".
169
-1Break up domain in blocks

Block data decomposition in which dimensions?
Splitting in
one dimension:
communication
= n2*2*w *1
w=width of halo
n3 =size of matrix
p = number of processors
cyclic boundary
> two neighbors
in each direction
two dimensions:
communication
= n2*2*w *2 / p1/2
three dimensions:
optimal for p>11
communication
= n2*2*w *3 / p2/3
A Heat-Transfer Example with MPI

Rolf Rabenseifner
Slide 10
170
[0, 0]
[1, 0]
[0, 1]
[1, 1]
nz
nx
nz_block
-2- Assign blocks to MPI processes
[0, 0]
MPI 0
[1, 0]
MPI 1
[0, 1]
MPI 2
[1, 1]
MPI 3
nx_block
nx_block=nx/2!
nz_block=nz/2
171
-3- Make a map of neighbors

MPI 24
MPI 10
lef=10
rig=40
bot=26
top=24
MPI 40
MPI 11
MPI 26
MPI 41
172
-4- Write code for single MPI block
for (ix=0; ix<nx_block; ix++) {!

for (iz=0; iz<nz_block; iz++) {!
vx[ix][iz] -= rox[ix][iz]*(!
c1*(p[ix][iz]
- p[ix-1][iz]) +!
c2*(p[ix+1][iz] - p[ix-2][iz]));!
}!
}!
173
-5- Communication calls

Examine the algorithm
Does it need data from its neighbors?
Yes? => Insert communication calls to get this data
Need p[-1][iz] and p[-2][iz] to update vx[0][iz]
p[-1][iz] corresponds to p[nx_block-1][iz] from left neighbor

p[-2][iz] corresponds to p[nx_block-2][iz] from left neighbor
Get this data from the neighbor with MPI.
174
communication
for (ix=0; ix<nx_block; ix++) {!
for (iz=0; iz<nz_block; iz++) {!
c1*(p[ix][iz]
- p[ix-1][iz]) +!
c2*(p[ix+1][iz] - p[ix-2][iz]));!
}!
}!
MPI 10
p[iz][nx_block-1]
!
i=nx_block-1
MPI 12
!
vx[0][iz] -= rox[0][iz]*(!
!
c1*(p[0][iz]
- p[i-1][iz])
!
c2*(p[1][iz]
i=0
- p[i-2][iz])
i=1
175
+!
pseudocode
if (my_rank==12) {!
call MPI_Receive(p[-2:-1][iz], .., 10, ..);!
!
c1*(p[ix][iz]
- p[ix-1][iz]) +!
c2*(p[ix+1][iz] - p[ix-2][iz]));!
}!
!
if (my_rank==10) {!
call MPI_Send(p[-2:-1][iz], .., 12, ..);!
}!
176
Update of all blocks [i][j]
[i-1][j]
send(nx-1)
send(nx-2)
receive(-1)
receive(-2)
[i][j]
receive(-1)
receive(-2)
send(nx-1)
send(nx-2)
[i+1][j]
send(nx-1)
send(nx-2)
receive(-1)
receive(-2)
[i+2][j]
receive(-1)
receive(-2)
send(nx-1)
send(nx-2)
mpi_sendrecv(!
p(nx-1:nx-2), 2 ,.., my_neighbor_right,...
p(-2:-1),
2 ,.., my_neighbor_left,...)
177
-6- Domain boundaries

What if my block is at a physical boundary?
Periodic boundary conditions
Non-periodic boundary conditions
178
Periodic Boundary Conditions
MPI 0
left=47
right=10
MPI 10
MPI 25
MPI 38
MPI 47
left=38
right=0
179
Non-Periodic Boundary Conditions

1. Avoid sending or receiving data from a non-existent block.
MPI_PROC_NULL
! if (my_rank == 12) &!

! .! my_neighbor_right = MPI_PROC_NULL!
! call mpi_send(......my_neighbor_right...)!
!no data is sent by
!
!
MPI !0
MPI 10
MPI 25
MPI 38
left=NUL
!
right=10
!
!
proc. 12*
MPI 47
left=38
right=NU
2. Apply boundary conditions.
if (my_neighbor_right == MPI_PROC_NULL) &!

[ APPLY APPROPRIATE BOUNDARY CONDITIONS ]
180
-7- guard cells

Allocate local block including overlapping areas to store values
received from neighbor communication.
Heat: How to parallelize with MP
This allows tidy coding of your loops
80
b1=1
P[i][iz]
guard
cells
-2
-1
inner area
!
!
!
0
outer area
values computed in each iteration

values read in each iteration
i
is
ie
181
halo
Exercise: MPI_Cart
Do I have to make a map of MPI processes myself?
You can use MPI_Cart to create a domain decomposition for you.
Exercise sets up 2D domain with one

periodic and
non-periodic boundary condition
Download code from:

http://www.xs4all.nl/~janth/HPCourse/code/MPI_Cart.tar
Unpack tar file and check the README for instructions
182
! answer
16 MPI tasks
PE-
6: x=1, y=2: Sum = 32 neighb0rs: left=2 right=10 top=5 bottom=7!
PE-
4: x=1, y=0: Sum = 24 neighbors: left=0 right=8 top=-1 bottom=5!
PE-
2: x=0, y=2: Sum = 32 neighbors: left=14 right=6 top=1 bottom=3!
PE-
7: x=1, y=3: Sum = 36 neighbors: left=3 right=11 top=6 bottom=-1!
PE-
PE-
PE-
PE-
3: x=0, y=3: Sum = 36 neighbors: left=15 right=7 top=2 bottom=-1!
PE- 14: x=3, y=2: Sum = 32 neighbors: left=10 right=2 top=13 bottom=15!
PE-
PE-
PE- 11: x=2, y=3: Sum = 36 neighbors: left=7 right=15 top=10 bottom=-1!
PE- 15: x=3, y=3: Sum = 36 neighbors: left=11 right=3 top=14 bottom=-1!
PE- 12: x=3, y=0: Sum = 24 neighbors: left=8 right=0 top=-1 bottom=13!
183
12
13
10
14
11
15
184
Example: 3D-Finite Difference

3D domain decomposition to distribute the work
MPI_Cart_create to set up cartesian domain
Hide communication of halo-areas with computational work
MPI-IO for writing snapshots to disk

use local and global derived types to exclude halo areas write in
global snapshot file
Tutorial about domain decomposition:

http://www.nccs.nasa.gov/tutorials/mpi_tutorial2/
185
MPI_Comm_size(MPI_COMM_WORLD, &size);!
/* divide 3D cube among available processors */!

dims[0]=0; dims[1]=0; dims[2]=0;!
MPI_Dims_create(size, 3, dims);!
!
!
!
npz = 2*halo+nz/dims[0];!
npx = 2*halo+nx/dims[1];!
npy = 2*halo+ny/dims[2];!
/* create cartesian domain decomposition */!
MPI_Cart_create(MPI_COMM_WORLD, 3, dims, period, reorder, &COMM_CART);!
/* find out neighbors of the

the model */!
MPI_Cart_shift(COMM_CART, 1,
rank, MPI_PROC_NULL is a hard boundary of

1, &left, &right);!
1, &top, &bottom);!
1, &front, &back);!
fprintf(stderr, "Rank %d has LR neighbours %d %d FB %d %d TB %d %d\n",

my_rank, left, right, front, back, top, bottom);!
186
subarrays for IO
local
global
nx
px
lnx
ny
py
lny
187
!
!
!
!
!
/* create subarrays(excluding halo areas) to write to file with MPI-IO */!
/* data in the local array */!
sizes[0]=npz; sizes[1]=npx; sizes[2]=npy;!
subsizes[0]=sizes[0]-2*halo;!
starts[0]=halo; starts[1]=halo; starts[2]=halo;!
MPI_Type_create_subarray(3, sizes, subsizes, starts, MPI_ORDER_C,!
MPI_FLOAT, &local_array);!
MPI_Type_commit(&local_array);!
/* data in the global array */!

gsizes[0]=nz; gsizes[1]=nx; gsizes[2]=ny;!
gstarts[0]=subsizes[0]*coord[0];!
MPI_Type_create_subarray(3, gsizes, subsizes, gstarts, MPI_ORDER_C,!
MPI_FLOAT, &global_array);!
MPI_Type_commit(&global_array);!
188
for (it=0; it<nt; it++) {!
/* copy of halo areas */!

copyHalo(vx, vz, p, npx, npy, npz, halo, leftSend, rightSend,
frontSend, backSend, topSend, bottomSend);!
/* start a-synchronous communication of halo areas to neighbours */!

exchangeHalo(leftRecv, leftSend, rightRecv, rightSend, 3*halosizex,
left, right, &reqRecv[0], &reqSend[0], 0);!
exchangeHalo(frontRecv, frontSend, backRecv, backSend, 3*halosizey,
front, back, &reqRecv[2], &reqSend[2], 4);!
exchangeHalo(topRecv, topSend, bottomRecv, bottomSend, 3*halosizez,
top, bottom, &reqRecv[4], &reqSend[4], 8);!
/* update on grid values excluding halo areas */!

updateVelocities(vx, vy, vz, p, ro, ixs, ixe, iys, iye, izs, ize,
npx, npy, npz);!
updatePressure(vx, vy, vz, p, l2m, ixs, ixe, iys, iye, izs, ize,
npx, npy, npz);!
!
!
189
!
!
/* wait for halo communications to be finished */!

waitForHalo(&reqRecv[0], &reqSend[0]);!
t1 = wallclock_time();!
tcomm += t1-t2;!
newHalo(vx, vz, p, npx, npy, npz, halo, leftRecv, rightRecv,

frontRecv, backRecv, topRecv, bottomRecv);!
t2 = wallclock_time();!
tcopy += t2-t1;!
/* compute field on the grid of the halo areas */!

updateVelocities(vx, vy, vz, p, ro, ixs, ixe, iys, iye, izs, ize,
npx, npy, npz);!
updatePressure(vx, vy, vz, p, l2m, ixs, ixe, iys, iye, izs, ize,
npx, npy, npz);!
190
!
!
/* write snapshots to file */!

if (time == snapshottime) {!
MPI_Info_create(&fileinfo);!
MPI_Info_set(fileinfo, "striping_factor", STRIPE_COUNT);!
MPI_Info_set(fileinfo, "striping_unit", STRIPE_SIZE);!
MPI_File_delete(filename, MPI_INFO_NULL);!
rc = MPI_File_open(MPI_COMM_WORLD, filename, MPI_MODE_RDWR|

MPI_MODE_CREATE, fileinfo, &fh);!
disp = 0;!
rc = MPI_File_set_view(fh, disp, MPI_FLOAT, global_array,
"native", fileinfo);!
rc = MPI_File_write_all(fh, p, 1, local_array, status);!
MPI_File_close(&fh);!
}!
} /* end of time loop */!
191
Summary
192
MPI Summary
!
MPI Standard widely accepted by vendors and
programmers
MPI implementations available on most modern platforms
Several MPI applications deployed
Several tools exist to trace and tune MPI applications
!
Simple applications use point-to-point and collective
communication operations
!
Advanced applications use point-to-point, collective,
communicators, datatypes, one-sided, and topology
operations
193
Hybrid systems programming hierarchy

Hybrid System
Pure MPI
Hybrid MPI/OpenMP
OpenMP in SMP nodes

MPI across the int. net.
No overlapping
computation/communication
MPI only out of OpenMP code.
Only master thread with
MPI communication.
Pure OpenMP
Supported by software
distributed shared memory
Overlapping
computation/communication
MPI inside OpenMP code
Hybrid OpenMP/MPI
Natural paradigm for clusters of SMPs
May offer considerable advantages when application mapping and
load balancing is tough
Benefits with slower interconnection networks (overlapping
computation/communication)
Requires work and code analysis to change pure MPI codes

Start with auto parallelization?
Link shared memory librariescheck various thread/MPI
processes combinations
Study carefully the underlying architecture
What is the future of this model? Could it be consolidated in new

languages?
Connection with many-core?
195
Overlapping computation/communication:
Example
P0
P2
TyT T T
T T T T
T T T T
peT T T
T T T TtoT T T
T Type
T T T
T T
to Tenter
en
T T T T T T T
T T T ter
T T T T
T T T TteT T T
T T T T T T T
C. BEKAS
P1 Suppose we wish to solve the PDE
T
T
T
T
TP
T
T
T
Using the Jacobi method: the value of

u at each discretization point is given
by a certain average among its
neighbors, until convergence.
Distributing the mesh to SMP
clusters by Domain Decomposition, it
is clear that the GREEN nodes can
proceed without any comm., while
the Blue nodes have to communicate
first and calculate later.
196
MPI/OpenMPI: Overlapping computation/

communication
Not only the master but other threads communicate. Call MPI
primitives in OpenMP code regions.
!
if (my_thread_id < # ){!
!
MPI_ (communicate needed data)!
} else!
!
/* Perform computations that to not need
communication */!
!
.!
!
.!
}!
/* All threads execute code that requires
communication */!
!
.!!
!
.
197
!
!
!
!
for
!
!
!
(k=0; k < MAXITER; k++){!

/* Start parallel region here */!
#pragma omp parallel private(){!
!
my_id = omp_get_thread_num();!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
if (my_id is given halo points)!

!
MPI_SendRecv(From neighboring MPI process);!
else{!
!
for (i=0; i < # allocated points; i++)!
!
!
newval[i] = avg(oldval[i]);!
}!
!
!
!
if (there are still points I need to do) /* Thi!

!
for (i=0; i< # remaining points; i++)!
!
!
newval[i] = avg(oldval[i]);!
!
!
!
}!
for (i=0; i<(all_my_points); i++)!
!
!
oldval[i] = newval[i];!
! !
! !
! }
}!
MPI_Barrier(); /* Synchronize all MPI processes here */!
198
PGAS
What is PGAS?
How to make use of PGAS as a programmer?
199
Partitioned
Global
Address Space
Partitioned Global
Address
Space
! Explicitly parallel, shared-memory like programming model
! Global addressable space
"
Allows programmers to declare and directly access data distributed across the
machine
! Partitioned address space

"
"
Memory is logically partitioned between local and remote (a two-level hierarchy)

Forces the programmer to pay attention to data locality, by exposing the inherent
NUMA-ness of current architectures
! Single Processor Multiple Data (SPMD) execution model

"
"
"
All threads of control execute the same program

Number of threads fixed at startup
Newer languages such as X10 escape this model, allowing fine-grain threading
! Different language implementations:

"
UPC (C-based), CoArray Fortran (Fortran-based), Titanium and X10 (Java-based)
Montse Farreras mfarrera@ac.upc.edu
200School 2009
PRACE Winter
Partitioned Global Address Space

Process/Thread
Address Space
Message passing
Shared Memory
PGAS
MPI
OpenMP
UPC, CAF, X10
! Computation is performed in multiple

places.
! A place contains data that can be
operated on remotely.
! Data lives in the place it was created, for
its lifetime.
! A datum in one place may point to a datum

in another place.
! Data-structures (e.g. arrays) may be
distributed across many places.
A place expresses locality.

Montse Farreras mfarrera@ac.upc.edu
PRACE Winter School 2009

201
Shared Memory (OpenMP)

Multiple threads share global memory
Most common variant: OpenMP
Program loop iterations distributed to threads, more recent task

features
Each thread has a means to refer to private objects within a parallel
context
Terminology
Thread, thread team
Implementation
Threads map to user threads running on one SMP node
Extensions to multiple servers not so successful
202
OpenMP
memory
threads
203
OpenMP: work distribution

memory
1-8 9-16 17-24 25-32
!$OMP PARALLEL
do i=1,32
a(i)=a(i)*2
end do
threads
204
OpenMP: implementation
memory
process
threads
cpus
205
Message Passing (MPI)

Participating processes communicate using a message-passing
API
Remote data can only be communicated (sent or received) via the

API.
MPI (the Message Passing Interface) is the standard
Implementation:
MPI processes map to processes within one SMP node or across
multiple networked nodes
API provides process numbering, point-to-point and collective
messaging operations
206
MPI
processes
memory
memory
memory
cpu
cpu
cpu
207
MPI
process 0
process 1
memory
memory
cpu
cpu
MPI_Send(a,...,1,) MPI_Recv(a,...,0,)
208

Shortened to PGAS
Participating processes/threads have access to local memory via

standard program mechanisms
Access to remote memory is directly supported by the PGAS

language
209
PGAS
process
process
process
memory
memory
memory
cpu
cpu
cpu
210
PGAS languages
Partitioned Global Address Space (PGAS):
Global address space any process can address memory
on any processor
!
Partitioned GAS retain information about locality
!
Core idea hardest part of writing parallel code is
managing data distribution and communication; make that
simple and explicit
!
PGAS Languages try to simplify parallel programming
(increase programmer productivity).
HPCC PTRANS and PGAS Languages
Data Parallel Languages

Unified Parallel C (UPC) is an extension of the C programming
language designed for high performance computing on large-scale
parallel machines.
http://upc.lbl.gov/
Co-array Fortran (CAF) is part of Fortran 2008 standard. It is a

simple, explicit notation for data decomposition, such as that
often used in message-passing models, expressed in a natural
Fortran-like syntax.
http://www.co-array.org
both need a global address space (which is not equal to SMP)
212
UPC
Unified Parallel C:
An extension of C99
An evolution of AC, PCP, and Split-C
!
Features
SPMD parallelism via replication of threads
Shared and private address spaces
Multiple memory consistency models
!
Benefits
Global view of data
Co-Array Fortran
Co-Array Fortran:
An extension of Fortran 95 and part of Fortran 2008
The language formerly known as F--
!
Features
SPMD parallelism via replication of images
Co-arrays for distributed shared data
!
Benefits
Syntactically transparent communication
Multi-dimensional arrays
Array operations
Basic execution model co-array F- Program executes as if replicated to multiple copies with each
copy executing asynchronously (SPMD)
Each copy (called an image) executes as a normal Fortran

application
New object indexing with [] can be used to access objects on

other images.
New features to inquire about image index, number of images

and to synchronize
215
CAF (F--)
!
!
REAL, DIMENSION(N)[*] :: X,Y!

X(:) = Y(:)[Q]!
Array indices in parentheses follow the normal Fortran rules

within one memory image.
Array indices in square brackets provide an equally convenient
notation for accessing objects across images and follow similar
rules.
216
Coarray execution model

Image 1
coarrays
Image 2
Image 3
memory
memory
memory
cpu
cpu
cpu
Remote access with square bracket indexing: a(:)[2]

217
Basic coarray declaration and usage

integer :: b
integer :: a(4)[*] !coarray
image 1
image 2
image 3
a 1 8 1 5
b 1
a 1 7 9 9
b 3
a 1 7 9 4
b 6
b=a(2)
218

integer :: b
image 1
image 2
image 3
a 1 8 1 5
b 8
a 1 7 9 9
b 7
a 1 7 9 4
b 7
b=a(2)
219

integer :: b
Text
image 1
image 2
image 3
a 1 8 1 5
b 1
a 1 7 9 9
b 3
a 1 7 9 4
b 6
b=a(4)[3]
220

integer :: b
Text
image 1
image 2
image 3
a 1 8 1 5
b 4
a 1 7 9 9
b 4
a 1 7 9 4
b 4
b=a(4)[3]
221
222
Performance Tuning
Performance tuning
S
ideal
second attempt
first attempt
p
Performance tuning of parallel applications is an iterative process
Feb-5-08
223
56
Compiling MPI Programs
224
Compiling and Starting MPI Jobs

Compiling:
Need to link with appropriate MPI and communication subsystem libraries
and set path to MPI Include files
Most vendors provide scripts or wrappers for this (mpxlf, mpif77, mpicc, etc)
Starting jobs:
Most implementations use a special loader named mpirun
mpirun -np <no_of_processors> <prog_name>
!
In MPI-2 it is recommended to use
mpiexec -n <no_of_processors> <prog_name>
225
MPICH: a Portable MPI Environment

MPICH is a high-performance portable implementation of MPI (both 1
and 2).
It runs on MPP's, clusters, and heterogeneous networks of workstations.
The CH comes from Chameleon, the portability layer used in the original
MPICH to provide portability to the existing message-passing systems.
In a wide variety of environments, one can do:

mpicc -mpitrace myprog.c!
mpirun -np 10 myprog!
upshot myprog.log!
to build, compile, run, and analyze performance.

226
MPICH2
MPICH2 is an all-new implementation of the MPI Standard,
designed to implement all of the MPI-2 additions to MPI.
separation between process management and communications

use daemons (mpd) on nodes
dynamic process management,
one-sided operations,
parallel I/O, and others
!
!
http://www.mcs.anl.gov/research/projects/mpich2/
227
Compiling MPI programs

From a command line:
mpicc -o prog prog.c !
Use profiling options (specific to mpich)
-mpilog Generate log files of MPI calls

-mpitrace Trace execution of MPI calls
-mpianim Real-time animation of MPI (not available on all
systems)
--help Find list of available options
The environment variables MPICH_CC, MPICH_CXX, MPICH_F77,

and MPICH_F90 may be used to specify alternate C, C++,
Fortran 77, and Fortran 90 compilers, respectively.
228
example hello.c
#include "mpi.h"!
#include <stdio.h>!
int main(int argc ,char *argv[])!
{!
! int myrank;!
! MPI_Init(&argc, &argv);!
! MPI_Comm_rank(MPI_COMM_WORLD, &myrank);!
! fprintf(stdout, "Hello World, I am process
\n", myrank);!
! MPI_Finalize();!
! return 0;!
}
%d
229
Example: using MPICH1

-bash-3.1$ mpicc -o hello hello.c!
-bash-3.1$ mpirun -np 4 hello!
Hello World, I am process 0!
230
Example: details
If using frontend and compute nodes in machines file use
mpirun -np 2 -machinefile machines hello !
!
If using only compute nodes in machine file use
!
mpirun -nolocal -np 2 -machinefile machines hello
-nolocal - dont start job on frontend

-np 2
- start job on 2 nodes
-machinefile machines - nodes are specified in machines file
hello
- start program hello
231
Notes on clusters
Make sure you have access to the compute node (ssh keys are
generated ssh-keygen) and ask your system administrator.
!
Which mpicc are you using
$ which mpicc
!
command line arguments are not always passed to mpirun/mpiexec
!
(depending on your version). In that case make a script which

calls your program with all its arguments
232
MPICH2 daemons
mpdtrace:
output a list of nodes on which you can run

MPI programs (runs mpd daemons).
The -l option lists full hostnames and the port where the mpd is
listening.
mpd
starts an mpd daemon.
mpdboot
starts a set of mpds on a list of machines.
mpdlistjobs
lists the jobs that the mpds are running.
mpdkilljob kills
mpdlistjobs
!
mpdsigjob
a job specified by the name returned by
delivers a signal to the named job.

233
MPI - Message Passing Interface

MPI or MPI-1 is a library specification for message-passing.
MPI-2: Adds in Parallel I/O, Dynamic Process management,

Remote Memory Operation, C++ & F90 extension
MPI Standard:
http://www-unix.mcs.anl.gov/mpi/standard.html
MPI Standard 1.1 Index:
http://www.mpi-forum.org/docs/mpi-11-html/node182.html
MPI-2 Standard Index:
http://www-unix.mcs.anl.gov/mpi/mpi-standard/mpi-report-2.0/
node306.htm
MPI Forum Home Page:
http://www.mpi-forum.org/index.html
MPI tutorials
http://www.nccs.nasa.gov/tutorials/mpi_tutorial2/
mpi_II_tutorial.html
https://fs.hlrs.de/projects/par/par_prog_ws/
235
MPI Sources
The Standard (3.0) itself:
at http://www.mpi-forum.org
All MPI official releases, in both PDF and HTML
Books:
Using MPI: Portable Parallel Programming with the MessagePassing Interface, by Gropp, Lusk, and Skjellum, MIT Press, 1994.
Parallel Programming with MPI, by Peter Pacheco, MorganKaufmann, 1997.
Other information on Web:
at http://www.mcs.anl.gov/mpi
pointers to lots of stuff, including other talks and tutorials, a FAQ,
other MPI pages
http://mpi.deino.net/mpi_functions/index.htm
236
Job Management and queuing
237

On a large system many users are running simultaneously.
What to do when:
The system is full and you want to run your 512 CPU job?
You want to run 16 jobs, should others wait on that?
Have bigger jobs priority over smaller jobs?
Have longer jobs lower/higher priority?
The job manager and queue system takes care of it.
238
MPICH1 in PBS
#!/bin/bash!
#PBS -N Flank!
#PBS -V!
#PBS -l nodes=11:ppn=2!
#PBS -j eo !
!
cd $PBS_O_WORKDIR!
export nb=`wc -w < $PBS_NODEFILE`!
echo $nb!
!
mpirun -np $nb -machinefile $PBS_NODEFILE ~/bin/
migr_mpi \!
file_vel=$PBS_O_WORKDIR/grad_salt_rot.su \!
file_shot=$PBS_O_WORKDIR/cfp2_sx.su
239

PBS (maui):
http://www.clusterresources.com/pages/products/maui-cluster-scheduler.php
Torque resource manager http://www.clusterresources.com/pages/products/
torque-resource-manager.php
Sun Grid Engine

http://gridengine.sunsource.net/
SLURM
!
!
!
http://slurm.schedmd.com
Luckily their interface is very similar (qsub, qstat, ...)

240
queuing commands
qsub
qstat
qdel
xpbsmon
qsub
#PBS
#PBS
#PBS
#PBS
!
!
-l
-l
-l
-j
nodes=10:ppn=1!
mem=20mb!
walltime=1:00:00!
eo!
submit:
!
qsub
!
-q normal job.scr!
output:
jobname.ejobid!
jobname.ojobid!
qstat
available queue and resources
!
qstat -q!
queued and running jobs
!
qstat (-a)!
243
qdel
deletes job from queue and stops all running executables
!
qdel jobid
244
Submitting many jobs in one script

#!/bin/bash -f!
#!
export xsrc1=93!
export xsrc2=1599!
export dxsrc=3!
xsrc=$xsrc1!
while (( xsrc <= xsrc2 ))!

do!
echo ' modeling shot at x=' $xsrc !
cat << EOF > jobs/pbs_${xsrc}.job!

#!/bin/bash!
#!
#PBS -N fahud_${xsrc}!
#PBS -q verylong!
#PBS -l nodes=1:ppn=1!
#PBS -V!
Be careful !
!
program with arguments!
!
EOF!
!
qsub jobs/pbs_${xsrc}.job!
(( xsrc = $xsrc + $dxsrc))!
done
245
246

Download code from:

http://www.xs4all.nl/~janth/HPCourse/code/MPI_pi.tar
It is assumed that you use mpich1 and has PBS installed as a job
scheduler. If you have mpich2 let me know.
Use qsub to submit the mpi job (an example script is provided) to
a queue.
247

cores
OpenMP
marken (MPI)
linux (MPI)
9.617728
14.10798
22.15252
4.874539
7.071287
9.661745
2.455036
3.532871
5.730912
1.627149
2.356928
3.547961
1.214713
1.832055
2.804715
12
0.820746
1.184123
16
0.616482
0.955704
248

Pi Scaling
16
Amdahl 1.0
MPI marken Xeon 3.1 GHz
MPI linux Xeon 2.4 GHz
14
speed up
12
10
8
6
4
2
0
1
6
number of CPUs
12
16
249
Exercise: OpenMP Max

Download code from:

http://www.xs4all.nl/~janth/HPCourse/code/MPI_pi.tar
The exercise focus on using the available number of cores in an

efficient way.
Also inspect the code and see how the reductions are done, is
there another way of doing the reductions?
250
Exercise: OpenMP details

More details is using OpenMp and shared memory parallelisation
Download code from:

http://www.xs4all.nl/~janth/HPCourse/code/OpenMP.tar
These are old exercises from SGI and give insight in problems
you can encounter using OpenMP.
It requires already some knowledge about OpenMP. The OpenMP

F-Summary.pdf from the website can be helpful.
251
Extend the Exercises

Modify the pi program to use send/receive instead of bcast/
reduce.
Write a program that sends a message around a ring. That is,

process 0 reads a line from the terminal and sends it to process 1,
who sends it to process 2, etc. The last process sends it back to
process 0, who prints it.
Time programs with MPI_WTIME().
252
END
253
254

07 ParallelProgramming

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

07 ParallelProgramming

Transféré par

Droits d'auteur :

Formats disponibles

Parallelization

Challenge the future

Type to enter text

Issues with parallel computing

P/N is the ideal parallel speed-up, it will always be less

Super Linear speed up

Concurrency and Parallelism

Concurrency: The independence of parts of an algorithm

Parallelism(also parallel execution): Two or more parts of a

Concurrency is a necessary prerequisite for parallel execution

Concurrency vs. Parallelism

There is an enormous sequential programming knowledge

Parallel Programming is requiring new skills and new tools.

Start to parallelise simple problems and keep on learning along

Recognizing Sequential Processes

Dynamics and real-time, event driven applications are often difficult to

Many games fall into this category

The results of an iteration depend on the preceding iteration

Audio encoders fall into this category

Parallel Programming Models

Which model to use is often a combination of what is available

Parallel Programming Models

Data Parallel languages

Parallel Programming Concepts

Distributing Work and/or Data

i=1..25 | 26..50 | 51..75 | 76..100

idle time on some processors

Hardware Architectures & Parallel Programming Models

Consider how many calculations you can execute while only

Sending 16 kByte (20x20x5) doubles

very often blocks have to wait for their neighbours

Communication should only be used if one is quite sure that this is

communication is necessary on the boundaries

e.g. b(26) = b(26) + h*(a(25)-2*a(26)+a(27))!

Mesh partitioning with special load balancing libraries

less parallelism than processors

Same work load in

elization and Iterative Solvers

Examples of work distribution

First, decide how data elements should be divided among processors

Vector addition, summation, maximum, ..

Evaluating Parallel Models

How do processes synchronize?

In shared-memory model, both accomplished through shared

Shared versus Private Variables

Race conditions occur when execution order is assumed but not

Example: un-synchronised access to bank account

Initial balance = $1000

Initial balance = $1000

Load (balance = $1000)

Load (balance = $1000)

Store (balance = $900)

Store (balance = $1100)

Code Example in OpenMP

9.0, 11.0, 13.0, 15.0, 17.0, 19.0, 21.0, 23.0!

9.0, 11.0, 13.0,

9.0, 11.0, 13.0, 15.0, 17.0, 19.0, 21.0, 23.0!

9.0, 11.0, 13.0, 15.0, 17.0, 19.0, 21.0, 23.0

Code Example in OpenMP

How to Avoid Data Races

Variables declared within threaded functions

Mutual exclusion and synchronization

Static variable: Shared

e.g. b(26) = b(26) + h(a(25)-2a(26)+a(27))!