HPC cheat sheet

The commands you'll type most on SeaWulf, on one page. Everything here runs on a login node after you connect. Replace anything in <angle brackets> with your own values.

Jump to: Top 10 · Run work · Job script · Queues · Manage jobs · Files & transfer · Modules · Storage · Efficiency · Troubleshooting

module load slurm

Top 10

Three ways to run work

# Batch: queue a script to run on its own (best for real work)
sbatch job.sh

# Interactive: get a shell on a compute node, then work as usual
srun -p short-40core --time=01:00:00 --pty bash

# Allocate first, then run steps inside the allocation with srun
salloc -p short-40core --time=02:00:00 --ntasks=4

Type exit when you're done with an interactive session so the node is freed. More in Interactive jobs.

Job script template

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=results_%j.txt      # %j becomes the job ID
#SBATCH --error=errors_%j.txt
#SBATCH --partition=short-40core
#SBATCH --time=02:00:00              # hh:mm:ss, keep it realistic
#SBATCH --nodes=1
#SBATCH --ntasks=40
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=<you>@stonybrook.edu

module load intel-stack
mpirun ./my_program               # MPI example; for a serial program, just run it

Prefer not to write it by hand? The job script builder makes one for you. Details in Writing job scripts.

Common queues

These run from milan.seawulf.stonybrook.edu. The queue selection guide lists every queue, including GPU, high-memory, and long queues.

Watch and manage jobs

# Your jobs, with state and the reason a job is waiting
squeue -u $USER --format="%.10i %.20j %.8T %.10M %.6D %R"

# Cancel one job, or all of your pending jobs
scancel <jobid>
scancel -u $USER --state=pending

# Hold a pending job, then let it run again
scontrol hold <jobid>
scontrol release <jobid>

# Your job history since a date
sacct --starttime=<YYYY-MM-DD> --format=JobID,JobName,State,Elapsed,ExitCode

More in Managing jobs.

Files and transfer

Run these on your own computer, not on the cluster.

# Upload a file or a folder to your home directory
scp results.csv <netid>@milan.seawulf.stonybrook.edu:~/
scp -r my_folder/ <netid>@milan.seawulf.stonybrook.edu:~/

# Download a file
scp <netid>@milan.seawulf.stonybrook.edu:~/output.txt ./

# Sync a folder (only copies what changed); try --dry-run first
rsync -avz --progress my_folder/ <netid>@milan.seawulf.stonybrook.edu:~/my_folder/

rsync flags: -a keeps permissions and times, -v lists files, -z compresses, --dry-run shows what would happen without copying. For large data, use Globus. To avoid a Duo prompt for every copy, see fewer Duo prompts.

Software modules

module avail                 # everything available
module avail python          # search by name
module load gcc/12.3.0       # load a specific version
module list                  # what's loaded now
module show <name>           # what a module sets up
module purge                 # unload everything and start clean

Browse what's installed in the software catalog. More in Using modules.

Storage

myquota                               # space and file counts for home and scratch
cd /gpfs/scratch/$USER                # your scratch space (files older than 45 days are deleted)
du -sh * | sort -h                    # what's using the space in this folder
tar -czf archive.tar.gz my_folder/    # bundle a folder into one file
tar -xzf archive.tar.gz               # unpack it

Home is 20 GB and backed up; scratch is 20 TB, not backed up, and purged after 45 days. See Storage layout & policies and Checking storage quotas.

Check efficiency

# While a job runs
sstat -j <jobid> --format=JobID,AveCPU,AveRSS,MaxRSS

# After it finishes
seff <jobid>
sacct -j <jobid> --format=JobID,Elapsed,MaxRSS,AveCPU,State

If seff shows you used far less memory or CPU than you asked for, request less next time. Your jobs will start sooner and fair share works in your favor.

Quick troubleshooting

Still stuck? See When to ask for help or open a ticket. New to all this? Start with the Quick Start Guide and the glossary of HPC terms.