Hardware

There are currently three GPU nodes available on the Emmy system in Göttingen. These are similar to Skylake compute nodes, but equipped with four GPUs each:

  • 2 x Intel Skylake Gold 6148 CPUs (40 cores per node)
  • 768 GB memory
  • ggpu01: 480 GB local storage on a solid state disk
  • ggpu[02-03]: 2 TB local storage on a solid state disk
  • 1 x Intel Omni-Path host fabric adapter
  • 4 x NVIDIA Tesla V100 32GB

Getting Access

The node can be accessed using the gpu partition. Additionally you can specify, how many GPUs you need (by default you will get access to all) with the -G x options, where x is the number of GPUs you want to access. If you do not use MPI, please use the -c #cores parameter to select the needed CPUs.

Example

the following command gives you access to 20 cores and two GPUs:

Run an interactive shell with access to 20 cores and 2 GPUs 
srun -p gpu --pty -n 1 -c 20 -G 2 --interactive bash


If you want to run multiple concurrent programs, each using one GPU, here is an example:

Run a single GPU program four times concurrently 
#!/bin/bash
#SBATCH -p gpu
#SBATCH -t 12:00:00
#SBATCH -N 1
 
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
wait


More explanation for the above example can be found here.

Software and Libraries

The CUDA and cuDNN modules are only available on the node. Please note that our current cuDNN version is only compatible with the CUDA 10.1 module, not with the older 9.2.

Additionally, the anaconda3/2019.03 python module has a TensorFlow environment tf-gpu that is precompiled for GPU usage. You can use with


[nimboden@glogin1 ~]$ srun -p gpu -c 5 --pty -G 1 --interactive bash
[nimboden@ggpu01 ~]$ module load anaconda3
Module for Anaconda3 2019.03 loaded.
[nimboden@ggpu01 ~]$ source $CONDASH
[nimboden@ggpu01 ~]$ conda activate tf-gpu
(tf-gpu) [nimboden@ggpu01 ~]$ python tf_cnn_benchmarks.py [...]

  • No labels