Horovod

Description

Horovod is a distributed training framework for TensorFlow.

More information

- Homepage: https://github.com/uber/horovod

Available Versions of Horovod

Version	Module	Available on

Version	Module	Available on
0.28.1-foss-2022a-CUDA-11.7.0-TensorFlow-2.11.0	tools/Horovod/0.28.1-foss-2022a-CUDA-11.7.0-TensorFlow-2.11.0	Noctua 2

This table is generated automatically. If you need other versions please click pc2-support@uni-paderborn.de.

Usage Hints for Horovod

If you need support in using this software or example job scripts please click pc2-support@uni-paderborn.de.

Multi-GPU Training with Horovod and DeepMD

Installation as a Python virtual environment (one time)

We recommend installing Python virtual environments in the group directory on permanent storage of your project (/pc2/groups/[project name]) instead of the home directory or the parallel file system (/scratch/[project name]). Please replace [project name] with the name of your compute time project, i.e. hpc-prf-…

module reset
module load toolchain/foss/2022a
module load lang/Python/3.10.4-GCCcore-11.3.0
module load system/CUDA/12.6.0
rm -rf venv
python3 -m venv venv
source /pc2/groups/[project name]/venv_deepmd_horovod/bin/activate

pip install cmake
pip install mpi4py
pip install tensorflow
pip install deepmd-kit[gpu,cu12]
export HOROVOD_WITH_TENSORFLOW=1
export HOROVOD_WITH_MPI=1
pip install horovod[tensorflow]

Usage in a compute job

#!/bin/bash
#SBATCH -N 1
#SBATCH -t 2:00:00
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=32
#SBTACH --gpus-per-task=1
#SBATCH --gres=gpu:a100:2
#SBATCH -p gpu

#alternatively for testing in small/short jobs, see https://upb-pc2.atlassian.net/wiki/spaces/PC2DOK/pages/1902952/Running+Compute+Jobs#Using-GPUs-for-Development-and-Testing-Purposes
##SBATCH -p dgx
##SBATCH -q devel

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_PLACES=cores
export OMP_PROC_BIND=true

export TF_INTER_OP_PARALLELISM_THREADS=2
export TF_INTRA_OP_PARALLELISM_THREADS=`echo "$OMP_NUM_THREADS-$TF_INTER_OP_PARALLELISM_THREADS" | bc`

module reset
module load toolchain/foss/2022a
module load lang/Python/3.10.4-GCCcore-11.3.0
module load system/CUDA/12.6.0

source /pc2/groups/[project name]/venv_deepmd_horovod/bin/activate
srun dp train input.json