It is possible to run Jupyter notebooks interactively with a Slurm batch job, i.e., not using an interactive job.

If you want to access the Jupyter notebook web interface from your local workstation, you must make sure that you have a tunnel available to the port on which the web interface serves the Jupyter notebook output. This can be achieved with a few simple settings in the batch job and the ssh config on your local workstation.

1 Setting up the batch job
2 Accessing the Jupyter notebook on your local workstation

Setting up the batch job

First, we need to set up the batch job. Login to ALICE or SHARK and create a slurm batch file like this one depending on the cluster that you work on:

ALICE

In this example, we will make use of an existing module on ALICE. If you need a different version or additional packages, you can always create your own virtual environment and install everything in there.

The sbatch settings used here were chosen for demonstration purposes only.

#!/bin/bash

#SBATCH --job-name=jupyter_notebook
#SBATCH --mem=8G
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=cpu-short
#SBATCH --time=01:00:00
#SBATCH --output=%x_%j.out

unset XDG_RUNTIME_DIR
module load IPython/7.7.0-foss-2019a-Python-3.7.2

echo "Running the notebook on $(hostname)"
IPADDR=$(hostname -i)

echo "#### Starting notebook"
srun jupyter notebook --ip=$IPADDR --no-browser --port=8989
echo "#### Terminated notebook. Done"

SHARK

On SHARK, Jupyter notebooks are also available through the Open Ondemand portal which is most likely easier to make use of than this way

In this example, we will assume that you create a virtual environment and installed the jupyterlab package in it. Of course, you can also use conda.

The sbatch settings used here were chosen for demonstration purposes only.

#!/bin/bash

#SBATCH --job-name=jupyter_notebook
#SBATCH --mem=8G
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=short
#SBATCH --time=00:15:00
#SBATCH --output=%x_%j.out

unset XDG_RUNTIME_DIR
module load system/python/3.10.2

# Replace the path to your virtual environment as needed
echo "#### Sourcing virtual env"
source /path/to/your/virtualenv
echo "#### ... Done"

echo "#### Running the notebook on $(hostname)"
IPADDR=$(hostname -i)

echo "#### Starting notebook"
srun jupyter notebook --ip=$IPADDR --no-browser --port=8989
echo "#### Terminated notebook. Done"

Save the batch file for example as jupyter_notebook.slurm and submit it with sbatch jupyter_notebook.slurm.

Open the output file which in our example will be named something like jupyter_notebook_<jobid>.out and take note of the notebook token, the port that is used and the name of the node that the job is running on. Do not specify the node in the batch script. but let slurm take care of which node should be used.

Accessing the Jupyter notebook on your local workstation

On your local computer, set up the connection to the port of the web interface. You can for instance add into your ~/.ssh/config file the following lines:

Port 8989 that is used here is just an example. If a notebook is already running on this node on this port, then Jupyter will launch the notebook on another port. You can see this in the slurm output file. In this case adjust the port in your ssh config setting.

ALICE

Host alice-notebook
   HostName login1.alice.universiteitleiden.nl
   LocalForward 8989 <node_name>:8989
   ProxyJump <username>@ssh-gw.alice.universiteitleiden.nl:22
   User <username>
   ServerAliveInterval 60

where “<username> should be replaced by your ALICE user name and “<node_name> by the name of the node that your job is running on.

SHARK

where “<username> should be replaced by your SHARK user name and “<node_name> by the name of the node that your job is running on.

If you work from within the LUMC network, you do not need the line with the ProxyJump setting.

You can always look up the node that your job is running on with squeue or scontrol show job <job_id> or by looking into the output file where the token is in (assuming that you included the line about printing out the hostname).

Then open a local terminal and run

Afterwards, open your browser and point it to

and voila' you can use jupyter notebooks on the cluster from your local workstation.

HPC wiki

Running Jupyter notebooks in an interactive session

Setting up the batch job

Accessing the Jupyter notebook on your local workstation