How to run a Jupyter notebook on SeaWulf
Jupyter is a popular web-based environment for writing code in Python, as well as other languages. This article will provide a tutorial on how to launch a Jupyter notebook remotely on SeaWulf and connect to to it in your local browser. Parts of this tutorial are adapted from the following page: Jupyter on the Cluster.
The easiest way to run Jupyter Notebooks on SeaWulf is to use Open OnDemand. Please see this FAQ article for detailed instructions on accessing Open OnDemand and running interactive applications, including Jupyter Notebooks.
Alternatively, you may use the following steps to access a Jupyter Notebook via ssh port forwarding.
Jupyter is installed on SeaWulf and can be accessed via the following commands:
module load jupyter/7.4.4
To start a new Jupyter notebook on one of the SeaWulf compute nodes, please do the following:
1. Submit a new Slurm job. For example, using the following template script:
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH -p short-28core
#SBATCH --time 00:30:00
#SBATCH --job-name jupyter-notebook
#SBATCH --output jupyter-notebook-%J.log
# get tunneling info
XDG_RUNTIME_DIR=""
node=$(hostname -s)
user=$(whoami)
port=9001
# print tunneling instructions to the jupyter-log
echo -e "
Command to create ssh tunnel:
ssh -N -f -L ${port}:${node}:${port} ${user}@login.seawulf.stonybrook.edu
Use a Browser on your local machine to go to:
localhost:${port}"
# load modules or conda environments here
module load jupyter/7.4.4
# Run Jupyter
jupyter notebook --no-browser --port=${port} --ip=${node}
The above script will launch a 30 minute job in the short-28core queue, determine the hostname of the compute node (as well as other useful information), and then launch Jupyter Notebook on the specified port. Note that the choice of ports is arbitrary. In this case, we have chosen port 9001 since it's unlikely that this port would be used for any system processes.
2. Once the job has started running, you will see your log file ("output jupyter-notebook-<job-id>.log") which contains instructions for setting up an ssh tunnel from your computer to the SeaWulf compute node running Jupyter.
In this case, let's imagine that the job was running on node sn011. You would then open a new terminal window on your local machine (because you want the tunnel to be opened from the same location as your internet browser) and type the following:
ssh -N -f -L 9001:sn011:9001 <your-netid>@login.seawulf.stonybrook.edu
The above indicates that the ssh command will run in the background and that the port on your local machine (9001) will be forwarded to the port (again 9001 in this case) on SeaWulf compute node cn011. Please note that if you already have a connection to port 9001 open on your computer, you will need to specify an alternative port number.
3. Open a browser of your choice, and navigate to http://localhost:9001/
4. If prompted for a password or token, look again in your slurm log file. There should be a token that you can copy and paste into the browser. From here, you are ready to start a new notebook or resume an old one!
Please note: the instructions above will also work for running R in a Jupyter notebook, except that instead of loading the "jupyter" module, you should do the following instead:
module load r-jupyter/4.0.2