Managing jobs

Once a job is submitted, you'll want to see where it is, change or cancel it if needed, and check afterward how well it used what it asked for. These are the commands for each of those steps.

On this page: Watch your jobs · Hold, release, cancel · After a job finishes · Live view on the node · Use resources well · Why is my job waiting?

Watch your jobs

# A tidier view: ID, name, state, run time, nodes, and the reason it's waiting
squeue -u $USER --format="%.10i %.20j %.8T %.10M %.6D %R"

Hold, release, and cancel

scontrol hold <jobid>                    # keep a waiting job from starting
scontrol release <jobid>                 # let it start again

scancel <jobid>                          # cancel one job
scancel --name=my_job                    # cancel all jobs with this name
scancel --user=$USER --state=PENDING     # cancel all your waiting jobs
scancel --user=$USER                     # cancel ALL your jobs (careful)

After a job finishes

seff <jobid>                                                  # CPU and memory efficiency
sacct -j <jobid> --format=JobID,JobName,State,Elapsed,MaxRSS,CPUTime
/gpfs/software/hpc_tools/get_resource_usage.py                 # SeaWulf summary of CPU and memory use

Reading seff: low CPU efficiency means cores sat idle (ask for fewer, or check your code actually runs in parallel). Memory efficiency far below 100% means you can ask for less memory next time.

Live view on the compute node

1. Find your node. Look in the NODELIST column:

   squeue -u $USER

   For example dn045.
2. Connect to it (only works while your job is running there):

   ssh dn045

3. Watch it with one of these:

   module load glances && glances     # easiest to read
   module load htop && htop           # classic detailed view
   top                                # always available

CPU load is the average number of processes wanting a CPU, shown for the last 1, 5, and 15 minutes. On a busy 40-core node you'd expect about 40. Much lower means cores are idle; much higher means processes are fighting for cores, which slows everything down.

Use resources well

• Match CPU load to the number of cores you asked for.
• Compare peak memory with what you requested, and trim the request.
• Move small jobs to shared queues.
• Update your scripts based on what seff and the live view show. Better-sized jobs start sooner and help your fair-share priority.

Why is my job waiting?

The last column of squeue gives the reason:

Screenshots