Versions Compared

Old Version 27

changes.mady.by.user mrz

Saved on

compared with

New Version Current

changes.mady.by.user Marcel-Brian Wilkowsky

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The JupyterHub service is available for Noctua 1 and Noctua 2.

...

Access

The JupyterHub can be reached at the following address:

Noctua 12: https://jh.noctua1.pc2.uni-paderborn.de

Noctua 21: https://jh.noctua1.pc2.uni-paderborn.de

The JupyterHub can be accessed via VPN or on-site at the University of Paderborn.

...

image-20240422-114054.pngImage Added

Quick Start

Spawn host/resources

Start

Jupyter Session on Noctua 2

Quick Start

Jupyter Session on Noctua 1

Quick Start

Jupyter Notebook on Noctua 2

(Inside Slurm job, 1h runtime, normal partition)

Quick Start

Jupyter Notebook on Noctua 2

(Inside Slurm job, 1h runtime, gpu partition)

Quick Start

Jupyter Notebook on Noctua 1

(Inside Slurm job, 1h runtime, normal partition)

Quick Start

Jupyter Notebook on Noctua 1

(Inside Slurm job, 1h runtime, gpu partition - 1x

RTX 2080Ti

A40)

Quick Start

Server Options

Presets

Simple

...

Pre-set You have the possibility of creating a preset with predefined start options for yourself or your project group.

Note: The preset functionality is currently only available on Noctua 2.

...

Click here to list your presets: https://jh.pc2.uni-paderborn.de/services/presets/

Simple

...

Preset enviroments with predefined values how to start the Jupyter Notebook.

Default and self-created singularity apptainer containers can be used.

Advanced (Slurm)

...

An advanced view with setting options how a Slurm job should be startet started on a HPC cluster.

Expert (Slurm)

...

An expert view with a free text field where you can load additional Slurm flags or load custom environments.

...

Loading additional Jupyter kernels

You can load additional Jupyter kernel using Lmod (module). Following kernel are currently available:

...

If you need new kernel versions or even other programming languages then you are welcome to contact pc2-support!

Apptainer (Singularity) Container

In JupyterHub it is possible to launch Jupyter Notebook instances inside a Singularity container. This has the advantage of being able to use your own built environment. When starting a container, any directories can be mounted inside the container environment.

We provide a set of default Singularity containers:

Container name

Kernels available

Installed software

jupyter_scientific_python

Python

jupyter_datascience

Julia, Python, R

  • All from “jupyter_scientifc_python”

  • rpy2 package

  • The Julia compiler and base environment

  • IJulia to support Julia code in Jupyter notebooks

  • HDF5, Gadfly, RDatasets packages

To learn more about Singularity, see here: Singularity-Introduction

If you want to build your own Singularity container for JupyterHub, see here: Create my own Singularity : https://upb-pc2.atlassian.net/wiki/spaces/PC2DOK/pages/1903131/JupyterHub#Create-my-own-Singularity-container

Remote Desktop (Graphical Environment via

...

Xpra)

To create a remote desktop environment, you can click on "Desktop Environment" in the JupyterLab interface:

...

When you click on the tile ‘Xpra Desktop’, a remote desktop environment is set up in the background. Graphical applications (e.g. loaded via modules) can be started from the started graphical terminal.

How-To

Loading software modules using JupyterLab

To load software modules inside JupyterLab, click on the Lmod extension tab. Then you have the possibility to search, load and unload modules.

If you are using the Classic Notebook View, click on tab "Softwares" to load software modules.

Create my own Singularity container

Installing Jupyter tools

You do not need to install the Jupyter client tools inside your Singularity container.

If the file /opt/conda/bin/jupyterhub-singleuser does not exists inside your container, the JupyterHub binds its own tools inside your container at run time.

If you want to manage your own Jupyter tools/extensions please make sure /opt/conda/bin/jupyterhub-singleuser exists inside your Singularity container.

Using Docker stacks

It is possible to build singularity containers from the official jupyter docker stacks:

https://jupyter-docker-stacks.readthedocs.io/en/latest/

Here are more information on how to build a singularity container from DockerHub:

https://sylabs.io/guides/3.7/user-guide/build_a_container.html

Container Location

...

Creating presets

Note: The preset functionality is currently only available on Noctua 2.

To save time when configuring your Jupyter environenment you have the possibilty to create preset environments for yourself or your compute time group(s).

Created presets can be selected when starting a new Jupyter instance:

...

Create presets here: https://jh.pc2.uni-paderborn.de/services/presets/ (or JupyterHub home → services → presets)

...

Spawner

  • Local spawner (on JupyterHub)

    • Spawning the Jupyter notebook environment on the JupyterHub host. Slurm job flags not needed.

      • Slurm tools, Modules, Remote desktop environment are available.

  • Noctua 2 (via Slurm)

    • Spawning the Jupyter environment inside a Slurm job (on a compute/gpu/fpga node) on Noctua 2. Note: You need to specifiy Slurm job flags.

Preset scopes

Select who can use your preset. You or one of your compute time projects.

Default URL

The URL to which JupyterHub redirects when the server is started.

Example:

/lab -> Spawning JupyterLab environment

/xprahtml5 -> Spawning Remote desktop environment

Notebook directory

The working directory. Used for JupyterLab, the remote desktop environment and the classic Jupyter view.

Apptainer container

Your self-built Apptainer/Singularity container. Have a look here for creating your own container: https://upb-pc2.atlassian.net/wiki/spaces/PC2DOK/pages/1903131/JupyterHub#Create-my-own-Apptainer%2FSingularity-container

Environment variables

Extra environment variables.

Format:

Code Block
MY_ENV_VAR=”Hello World”
FOO=BAR

Modules

Extra Lmod modules to load on start time. All system modules and Jupyter specific kernels are available.

Slurm job flags

Slurm job flags in Slurm batch format. Example:

Code Block
#SBATCH --partition=normal
#SBATCH --time=01:00:00

Create custom IPython kernel inside custom conda environment

  1. Create a conda environment as described here:

    1. Python & Python Package Management

  2. conda activate <your_conda_env>

  3. conda install ipykernel

    1. Or: python3 -m pip install ipykernel

  4. python3 -m ipykernel install --user --name <KERNELNAME> --display-name "<DISPLAY NAME>"

    1. Make sure, that python3 is called from the environment

Create my own Apptainer/Singularity container

Container package requirements

  • python >= 3.10

    • jupyterhub

    • optional, but useful: jupyterlab

Example Apptainer/Singularity recipe

Build containers: Apptainer 

Base recipe
Code Block
Bootstrap: docker
From: debian

%post
apt -y update
export DEBIAN_FRONTEND=noninteractive
apt -y install zsh locales
localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8

python3 -m pip install jupyterhub
Install custom Python kernel inside the container (python 3.12)
Code Block
mkdir /opt/python3.12
cd /opt/python3.12

apt -y install build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python3-openssl git

wget https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tgz
tar -xf Python-3.12.0.tgz
rm Python-3.12.0.tgz
cd Python-3.12.0/
./configure --enable-optimizations
make -j 8
make altinstall

python3.12 --version
python3.12 -m pip install --upgrade pip
python3.12 -m pip install ipykernel

# finally installing ipython kernel
python3.12 -m ipykernel install --sys-prefix  --name <UNIQUE_KERNEL_NAME> --display-name "<KERNEL DISPLAY NAME>"
Install Lmod with the JupyterLab-Lmod extension
Code Block
apt -y install lua5.3 lua-posix

mkdir -p /usr/lib64/lua/5.3
cp /usr/lib/x86_64-linux-gnu/liblua5.3-posix.so.1 /lib64/lua/5.3/
mv /lib64/lua/5.3/liblua5.3-posix.so.1 /lib64/lua/5.3/posix.so
Make Slurm Tools inside my container available
Code Block
groupadd --gid 351 munge
groupadd --gid 567 slurm
useradd -d /var/run/munge -M --gid 351 --uid 994 --shell /sbin/nologin munge
useradd -d /opt/software/slurm -M --gid 567 --uid 567 --shell /bin/false slurm

Container Location

Info

All containers with type .sif will be automatically detected in $HOME/.jupyter/pc2-jupyterhub/

Your new built container can only be placed in your $HOME directory: $HOME/.jupyter/pc2-jupyterhub/

Alternatively you can create a link from your $PC2PFS to your $HOME directory:

Code Block
$ lsln -ls /scratch/pc2hpc-prf-mitarbeiterproject/mawi/jupyter_container.sif   -rw-r--r--. 1 mawi pc2-mitarbeiter 0 Dec 17 07:53 /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif
$ ln -s /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif $HOME/.jupyter/pc2-jupyterhub/
Info

All containers with type .sif will be automatically detected in $HOME/.jupyter/pc2-jupyterhub/

...

$HOME/.jupyter/pc2-jupyterhub/

Access remote JupyterHub server with the local Visual Studio Code instance

You need following extensions for Visual Studio Code:

  1. Create an acccess token in the JupyterHub web interface:

...

  1. Follow following instructions described here: https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter-hub

    1. You need to start a Jupyter session using our JupyterHub web interface. After successful start, you can copy the URL starting with https://jh.pc2.uni-paderborn.de/user/.../...

View Slurm job logs

If the path of the Slurm Job output has not been changed explicity, it can be found here by default:

Noctua 1: $HOME/.jupyter/last_jh_noctua1.log

Noctua 2: $HOME/.jupyter/last_jh_noctua2.log

"HubAuth._api_request" was never awaited

This is a current version conflict due to a feature change within JupyterHub.

For more information see here: https://github.com/jupyterhub/batchspawner/pull/247

We are waiting for a pull request.

“Terminals unavailable”

If you have terminado installed in your $HOME directory (pip3 install --user), please make sure that the version of terminado is at least 0.8.3.

PC² Support

...

PC² Support

If you have any other problems that won’t be solved, please contact the pc2-support@uni-paderborn.de

Troubleshooting

JupyterLab

“A Jupyter server is running.” message

  • This message appears because user settings managed by JupyterLab do not match the new JupyterLab version.

    • Try deleting ~/.jupyter/ in your $HOME directory as follows:

      • rm -r ~/.jupyter/

    • If you want to keep your custom user settings, write an email to pc2-support@uni-paderborn.de