Cluster Computing Tutorial

Cluster Computing at PIK
a tutorial
Ciaron Linstead
10th May 2016

Introduction Environment Modules SLURM Python Documentation Questions
Outline
1 Introduction
2 Environment Modules
3 SLURM
4 Python
5 Documentation
6 Questions
Ciaron Linstead IT Services 2

Introduction
Cluster configuration
Environment modules
SLURM - the workload scheduler
Create, submit and monitor jobs
Anaconda Python environment
Documentation

Introduction
Logged into cluster?

Introduction

Run a Python, R, shell script?

Introduction

Compiled and run C, C++, Fortran?

Introduction

Submitted jobs via SLURM?

Introduction

comfortable on the command line (cd, ls, mkdir)?

Introduction

comfortable on the command line (cd, ls, mkdir)?
download/compiled/install 3rd party software in Linux?

Servers
Compute: 312x (2x Intel Xeon E5-2667v3)

Servers
RAM+: 4x (Compute w/ 256GB RAM, 16GB per core)

Servers
GPU: 2x (RAM+ w/ NVidia Tesla K40c)

Servers
Login: 2x (2x E5-2690v3 (12-core), 256GB RAM)

Servers
Visualisation: 2x (2x E5-2680v3, (12-core), 256GB RAM)

Servers
Interconnect
Mellanox FDR Connect-IB Infiniband (non-blocking)

Servers
Interconnect
3.5x throughput compared to iPlex Infiniband

Servers
Interconnect
0.25x latency

Servers
Interconnect
0.25x latency
Filesystem
2 petabyte GPFS (parallel) filesystem storage

Servers
Interconnect
0.25x latency
Filesystem
2 petabyte GPFS (parallel) filesystem storage
/p/projects, /p/tmp, /p/system

Compute node configuration
2x Intel Xeon E5-2667v3 "Haswell" CPUs

8 cores per CPU, 16 cores total
64GB DRAM: TruDDR4 2133MHz
(4GB per core)
No local disk

Compute node - RAM+
Regular compute nodes with 256GB memory

16GB per core
4 available
priority to large-memory jobs (but can run regular jobs)
--partition=ram_gpu

Compute node - GPU
Regular + 256GB RAM + Nvidia Tesla K40c GPU

2 available
1.66 TFlops per card (plus the 16 CPUs)
CUDA/OpenCL interface for programming
--partition=ram_gpu --gres=gpu:1

Outline
1 Introduction
3 SLURM
4 Python
5 Documentation
6 Questions

Environment Modules - motivation
Have a look at the search path: echo $PATH


when you run "command", the shell goes through this list


shell runs the first one it finds


what if I want multiple versions?


what if I want multiple versions?
same goes for library paths, e.g. NetCDF, MPI

Environment Modules
list software: "module avail"

Environment Modules

-------------------- /p/system/modulefiles/compiler -----------------------------------
compiler/gnu/4.9.2 compiler/gnu/5.2.0 compiler/intel/15.0.3 compiler/intel/16.0.0

Environment Modules

"module load compiler/intel/16.0.0" - load one

Environment Modules


"module load compiler/intel"

Environment Modules


15.0.3 or 16.0.0? or 17.0.0? careful!

Environment Modules


15.0.3 or 16.0.0? or 17.0.0? careful!
"module list" - what’s loaded?

Environment Modules


15.0.3 or 16.0.0? or 17.0.0? careful!
"module unload compiler/intel/16.0.0" - unload one

Environment Modules


15.0.3 or 16.0.0? or 17.0.0? careful!
"module purge" - unload all

Environment Modules


15.0.3 or 16.0.0? or 17.0.0? careful!
"module purge" - unload all
"module show compiler/intel/16.0.0" - what’s being
done?

modules - an example modulefile
module show nco/4.5.0

--------------------------------------------------------------
/p/system/modulefiles/tools/nco/4.5.0:
module-whatis Enable usage for nco version 4.5.0

setenv NCOROOT /p/system/packages/nco/4.5.0
prepend-path PATH /p/system/packages/nco/4.5.0/bin
prepend-path INCLUDE /p/system/packages/nco/4.5.0/include
prepend-path LD_LIBRARY_PATH /p/system/packages/nco/4.5.0/lib
prepend-path MANPATH /p/system/packages/nco/4.5.0/share/man
--------------------------------------------------------------

Environment Modules
infrastructure loaded automatically

modules in /p/system/modulefiles, organised by category

Environment Modules
infrastructure loaded automatically

modules in /p/system/modulefiles, organised by category
add your own via $MODULEPATH
"export MODULEPATH=$MODULEPATH:$HOME/modulefiles"
add this to your .bashrc file

Exercise - Let’s add a custom module!
Download and build a library


Install in our $HOME directory


Write a modulefile


Write a modulefile
recommendation: install in
/some/path/<compiler>/package_name/version/


Write a modulefile
e.g. /home/linstead/software/intel/gmp/6.1.0


Write a modulefile
e.g. /home/linstead/software/intel/gmp/6.1.0
Prep:
cp -r /home/linstead/phd16/ $HOME

Hints - build
cd
mkdir -p software/gmp/6.1.0 && cd
software/gmp/6.1.0
tar xvf ../phd16/gmp-6.1.0.tar.xz
cd gmp-6.1.0
module load compiler/gnu/5.2.0
./configure –prefix=$HOME/software/gmp/6.1.0
make && make install

Hints - module
mkdir -p $HOME/modulefiles/gmp
cp /home/linstead/modulefiles/gmp/6.1.0
$HOME/modulefiles/gmp
edit $HOME/modulefiles/gmp/6.1.0 to match your
installation

Outline
1 Introduction
3 SLURM
4 Python
5 Documentation
6 Questions

Simple Linux Utility for Resource Management (SLURM)

SLURM manages and allocates cluster resources
programs (typically) submitted as jobs to a queue
"sbatch myscript.sh"
myscript.sh is a regular script, plus SLURM info

SLURM - a simple submit script
1 #!/bin/bash
2
3 #SBATCH --partition=standard
4 #SBATCH --qos=short
5 #SBATCH --job-name=sumprimes
6 #SBATCH --output=sumprimes-%j.out
7 #SBATCH --error=sumprimes-%j.err
8 #SBATCH --account=its
9 #SBATCH --ntasks=1
10
11 $HOME/mycode/sumprimes 1 10000000001
submit with sbatch <filename>

--partition - physical nodes ("standard" or "ram_gpu")

"sinfo"


"sinfo"
--qos - job type, determines limits and thus priority
"sacctmgr show qos" (or see my alias ssq)
interesting fields: Name, Priority, MaxWall, GrpTRES,
MaxTRES


"sinfo"
--qos - job type, determines limits and thus priority
"sacctmgr show qos" (or see my alias ssq)
interesting fields: Name, Priority, MaxWall, GrpTRES,
MaxTRES
--job-name
distinguish between your jobs in the queue
%j (Job ID) gives jobs unique filenames

--output/--error
location of standard output and error (e.g. "print" statements)
omit "error" and STDERR will go to "output"

--output/--error
--account
the project this job relates to
see "groups" command for the projects you belong to

--output/--error
--account
the project this job relates to
see "groups" command for the projects you belong to
--ntasks
number of copies of this program to run
--ntasks=1 for serial (non-MPI, non parallel) jobs

SLURM - parallel (MPI) submit script
as previous example, with these differences:

1 #SBATCH --ntasks=128
2
3 module purge
4 module load mpi/intel/5.1.3
5 # run parallel code with mpirun
6 mpirun -bootstrap slurm -n $SLURM_NTASKS $HOME/mycode/
sumprimes 0 10000000000

--ntasks=128
give me 128 processor cores, I don’t care where
SLURM will attempt to pack sockets and nodes

--ntasks=128
for performance, I may require packing/blocking
--nodes=8
--tasks-per-node=16

--ntasks=128
for performance, I may require packing/blocking
--nodes=8
--tasks-per-node=16
or
--nodes=16
--tasks-per-node=8
(but see next slide!)

--nodes=16 and --tasks-per-node=8

uses half the cores on each node
the other half are available for other users
implications for memory/disk bandwidth

--nodes=16 and --tasks-per-node=8

uses half the cores on each node
the other half are available for other users
implications for memory/disk bandwidth
--exclusive gets you the whole node
8GB RAM per task with 16/8 above
up to 64GB per tasks on standard nodes
use sparingly!
see also --cpus-per-task

SLURM examples - threaded (OpenMP) submission

script
OpenMP.sh
1 #!/bin/bash
2 # (options omitted for brevity)
3 #SBATCH --nodes=1
4 #SBATCH --tasks-per-node=1
5 #SBATCH --cpus-per-task=16
6
7 export OMP_NUM_THREADS=16
8 # OR export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
9
10 $HOME/mydir/myprog.exe
11 # OR srun $HOME/mydir/myprog.exe
submit with sbatch OpenMP.sh

SLURM - monitoring jobs
show queue: "squeue -u <username>"

alias sq=’squeue -u <username>’

show queue: "squeue -u <username>"

alias sq=’squeue -u <username>’
SQUEUE_FORMAT="%.18i %.9P %.8j %.8u %.2t %.10M
%.10L %.10l %.6D %.6C %.8q %R"
scontrol show job <job_id>


scontrol show job
1 JobId=321608 JobName=sumprimes
2 UserId=linstead(405) GroupId=users(100)
3 Priority=6858 Nice=0 Account=its QOS=short
4 JobState=RUNNING Reason=None Dependency=(null)
5 Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
6 RunTime=00:00:05 TimeLimit=1-00:00:00 TimeMin=N/A
7 SubmitTime=2016-05-04T14:47:13 EligibleTime=2016-05-04T14:47:13
8 StartTime=2016-05-04T14:47:13 EndTime=2016-05-05T14:47:13
9 Partition=standard AllocNode:Sid=login01:15215
10 NodeList=cs-e14c01b[02-05]
11 BatchHost=cs-e14c01b02
12 NumNodes=4 NumCPUs=64 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
13 TRES=cpu=64,mem=229376,node=4
14 Socks/Node=* NtasksPerN:B:S:C=16:0:*:* CoreSpec=*
15 MinCPUsNode=16 MinMemoryCPU=3.50G MinTmpDiskNode=0
16 Command=/home/linstead/cluster-examples/sumprimes/mpi/slurm.sh
17 WorkDir=/home/linstead/cluster-examples/sumprimes/mpi
18 StdErr=/home/linstead/cluster-examples/sumprimes/mpi/sumprimes-321608.err
19 StdOut=/home/linstead/cluster-examples/sumprimes/mpi/sumprimes-321608.out


sview - a graphical monitoring tool


sview - right-click on a job

Did my job(s) finish yet?

check squeue -u <username>
check sacct

sacct
1 [15:47:13] linstead@login01:~$ sacct
2 JobID JobName Partition Account AllocCPUS State ExitCode
3 ------------ ---------- ---------- ---------- ---------- ---------- --------
4 321467 sumprimes standard its 32 CANCELLED+ 0:0
5 321467.batch batch its 8 CANCELLED 0:15
6 321467.0 pmi_proxy its 4 FAILED 7:0
7 321472 sumprimes standard its 32 COMPLETED 0:0
8 321472.batch batch its 8 COMPLETED 0:0
9 321472.0 pmi_proxy its 4 COMPLETED 0:0
10 325608 sumprimes standard its 64 FAILED 9:0
11 325608.batch batch its 16 FAILED 9:0
12 325608.0 pmi_proxy its 4 COMPLETED 0:0

ExitCode n:m
n : code returned by the job script
m : signal which caused the process to terminate (if signalled)

By default sacct goes back to midnight


all my jobs in April:
sacct -S04.01 -E05.01


sacct -S04.01 -E05.01
Add/view extra fields
sacct --format=...
e.g. just ID and end time sacct -ojobid,end


sacct -S04.01 -E05.01
Add/view extra fields
sacct --format=...
e.g. just ID and end time sacct -ojobid,end
see man sacct
the format of most SLURM commands is configurable, either
via --format/-o or $S???_FORMAT variables.

Exercise: what time did jobs 325608 and 325611 start and
end?
Did any fail? If so, what were the exit codes?

Exercise: what time did jobs 325608 and 325611 start and
end?
Did any fail? If so, what were the exit codes?
sacct --jobs 325608,325611 -a --format=start,end,state,exitc

SLURM - job arrays
Submit and manage collections of similar jobs

--array=0-31
each job in the array takes the same settings from submit
script
each job has a unique index
$SLURM_ARRAY_TASK_ID

SLURM - job arrays
Task ID is available to my script:

#!/bin/bash
#SBATCH --qos=short
#SBATCH --partition=standard
#SBATCH --array=0-15
#SBATCH --output=jobarray-%A_%a.out
echo ${SLURM_ARRAY_TASK_ID}
(e.g. "./myprog inputfile_${SLURM_ARRAY_TASK_ID}")

SLURM - job arrays
Read $SLURM_ARRAY_TASK_ID in Python:

import os, sys
try:
task = os.environ[’SLURM_ARRAY_TASK_ID’]
except KeyError:
print "Not running with SLURM job arrays"
sys.exit(1)

SLURM - job arrays
Read $SLURM_ARRAY_TASK_ID in C:
#include <stdio.h>
#include <stdlib.h>
int main ()
{
printf("task ID: %s\n", getenv("SLURM_ARRAY_TASK_ID"));
return(0);
}

Outline
1 Introduction
3 SLURM
4 Python
5 Documentation
6 Questions

Python for scientific computing
Anaconda 2.3.0 (module load anaconda/2.3.0)

/p/system/packages/anaconda/2.3.0/bin/python
conda list


Provides the conda package manager
I can install and maintain my own Python environment...


conda create -n testenv ipython


source activate testenv


conda install <packagename>


source deactivate


source deactivate
...and I can share it with colleagues


source deactivate


source deactivate
conda env export > environment.yml


source deactivate
conda env create -f environment.yml


source deactivate
conda env create -f environment.yml
(can also manage R packages, we can set this up if there’s
interest)

linstead@login01:~$ source activate testenv
discarding /p/system/packages/anaconda/2.3.0/bin from PATH

prepending /home/linstead/.conda/envs/testenv/bin to PATH
(testenv)linstead@login01:~$

Exercise
create a new environment

install packages ipython and matplotlib

Outline
1 Introduction
3 SLURM
4 Python
5 Documentation
6 Questions

Documentation / Help
Cluster User Guides:

https://www.pik-potsdam.de/services/it/hpc/user-guides
Environment Modules: http://modules.sourceforge.net/
conda package manager:
http://conda.pydata.org/docs/using/
SLURM: http://slurm.schedmd.com/
man pages (module, sbatch, srun, sinfo, squeue etc.)

Documentation / Help
Questions/Problems/Requests:
http://www.pik-potsdam.de/services/it/hpc
mailto:cluster-support@pik-potsdam.de

Outline
1 Introduction
3 SLURM
4 Python
5 Documentation
6 Questions

Questions
Any questions?

Cluster Computing Tutorial

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Cluster Computing Tutorial

Transféré par

Droits d'auteur :

Formats disponibles

Cluster Computing at PIK

10th May 2016

Ciaron Linstead IT Services 2

Ciaron Linstead IT Services 3

Logged into cluster?

Ciaron Linstead IT Services 4

Logged into cluster?

Ciaron Linstead IT Services 4

Logged into cluster?

Ciaron Linstead IT Services 4

Logged into cluster?

Ciaron Linstead IT Services 4

Logged into cluster?

Ciaron Linstead IT Services 4

Logged into cluster?

Ciaron Linstead IT Services 4

Ciaron Linstead IT Services 5

Ciaron Linstead IT Services 5

Ciaron Linstead IT Services 5

Ciaron Linstead IT Services 5

Ciaron Linstead IT Services 5

Ciaron Linstead IT Services 5

Ciaron Linstead IT Services 5

Ciaron Linstead IT Services 5

Ciaron Linstead IT Services 5

Ciaron Linstead IT Services 5

Compute node configuration

2x Intel Xeon E5-2667v3 "Haswell" CPUs

Ciaron Linstead IT Services 6

Compute node - RAM+

Regular compute nodes with 256GB memory

Ciaron Linstead IT Services 7

Compute node - GPU

Regular + 256GB RAM + Nvidia Tesla K40c GPU

Ciaron Linstead IT Services 8

Ciaron Linstead IT Services 9

Environment Modules - motivation

Have a look at the search path: echo $PATH

Ciaron Linstead IT Services 10

Environment Modules - motivation

Have a look at the search path: echo $PATH

Ciaron Linstead IT Services 10

Environment Modules - motivation

Have a look at the search path: echo $PATH

Ciaron Linstead IT Services 10

Environment Modules - motivation

Have a look at the search path: echo $PATH

Ciaron Linstead IT Services 10

Environment Modules - motivation

Have a look at the search path: echo $PATH

Ciaron Linstead IT Services 10

list software: "module avail"

Ciaron Linstead IT Services 11

list software: "module avail"

Ciaron Linstead IT Services 11

list software: "module avail"

"module load compiler/intel/16.0.0" - load one

Ciaron Linstead IT Services 11

list software: "module avail"

"module load compiler/intel/16.0.0" - load one

Ciaron Linstead IT Services 11

list software: "module avail"

"module load compiler/intel/16.0.0" - load one

Ciaron Linstead IT Services 11