Writing job scripts

A job script tells Slurm what your job needs (cores, memory, time, queue) and what to run. Write it once, submit it with sbatch, and Slurm runs it on a compute node when resources are free. This page gives you a template to copy, explains each part, and lists the options you'll use most.

On this page: The four parts · Template · Asking for resources · Directive reference · Useful variables · Dependencies · Make scripts safer · Troubleshooting

The four parts of a job script

1. Shebang line: #!/bin/bash, so the script runs in bash.
2. #SBATCH lines: what the job needs. Slurm reads these; bash treats them as comments.
3. Environment setup: load modules and set variables.
4. The work: the commands that actually run.

Template

#!/bin/bash

# --- Job identification ---
#SBATCH --job-name=my_job
#SBATCH --output=%x_%j.out            # %x = job name, %j = job ID
#SBATCH --error=%x_%j.err

# --- Resources ---
#SBATCH --partition=short-40core
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=40
#SBATCH --time=02:00:00               # hh:mm:ss

# --- Optional: email when it ends or fails ---
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=<you>@stonybrook.edu

# --- Environment setup ---
module purge
module load intel/oneAPI/2022.2
module load compiler/latest
module load mpi/latest
export OMP_NUM_THREADS=1

echo "Job $SLURM_JOB_ID on $SLURM_JOB_NODELIST, $SLURM_NTASKS tasks, started $(date)"

# --- The work ---
cd $SLURM_SUBMIT_DIR
mpirun ./my_app input.dat > computation.log

echo "Finished $(date)"

Submit it and check on it:

sbatch my_job.slurm
squeue -u $USER

Ready-made scripts for serial, multi-threaded, MPI, GPU, and array jobs are in Example jobs.

Asking for the right resources

• Pick the queue first. The queue decides the node type and the maximum run time. See the queue selection guide.
• Whole node or part of one? Queues without -shared give you whole nodes, with all their memory. For small jobs, use a shared queue and set --mem.
• Tasks vs. CPUs per task. Use --ntasks for MPI processes and --cpus-per-task for threads (OpenMP, Python multiprocessing). Match OMP_NUM_THREADS to $SLURM_CPUS_PER_TASK.
• Time. Ask for a realistic limit with some margin. Shorter requests often start sooner; if the job runs past its limit, Slurm stops it.
• Check afterward. seff <jobid> shows how much CPU and memory you really used, so you can ask for less next time.

Directive reference

Useful variables inside a job

Run jobs in order (dependencies)

For work in steps, such as preprocessing, then analysis, then plotting, you can submit every step at once and have each one wait for the one before it. Slurm holds the later jobs until their condition is met.

# Submit the first step and keep its job ID
jid=$(sbatch --parsable step1.slurm)

# Start step 2 only if step 1 finishes successfully
sbatch --dependency=afterok:$jid step2.slurm

You can wait on several jobs at once, as in afterok:1234:1235. If a condition can never be met (step 1 failed and step 2 needed afterok), the waiting job stays in the queue with the reason DependencyNeverSatisfied. Cancel it with scancel <jobid>.

Make scripts safer and easier to debug

# Stop at the first error instead of carrying on
set -e

# Check the input exists before doing any work
if [ ! -f "input.dat" ]; then
    echo "ERROR: input.dat not found"
    exit 1
fi

# Time the main step
start=$(date +%s)
mpirun ./my_app input.dat
echo "Computation took $(( $(date +%s) - start )) seconds"

• Start with module purge so modules left over from your login session don't interfere.
• Load the same modules you compiled with.
• Test small first: a short run in debug-40core, or an interactive session, before a big submission.
• Keep a template for each kind of job you run often.

Troubleshooting

sed -i 's/\r$//' my_job.slurm

More commands for watching and changing jobs are in Managing jobs and the cheat sheet.