Slurm basics

Common user commands

The following is a list of common user commands that will be discussed in more detail in this section and the tutorials.

If you want to get a full overview, have a look at the Slurm documentation or enter man <command> or check the documentation on the Slurm website.

Accounting info

The following list a number of commands to get accounting and job statistics.

Scheduling info

The following commands can provide useful information to understand the scheduling of your job

Environment variables

Any environment variables that you have set with the sbatch command will be passed to your job. For this reason, if your program needs certain environment variables set to function properly, it is best to put them in your job script. This also makes it easier to reproduce your job results later, if necessary.

In addition to setting environment variables yourself, Slurm provides some environment variables of its own that you can use in your job scripts. Information on some of the common Slurm environment variables is listed in the chart below. For additional information, see the man page for sbatch.

Job settings and configuration information

With the slurm command scontrol show you can get a more detailed overview of your running job, node hardware and partitions:

Slurm partitions

Slurm makes nodes on a cluster available as partitions, i.e., one partition gives access to a certain number of nodes and specific resource limits. Jobs are always submitted to either a default partition or a user-specified partition.

sinfo

sinfo provides information about the current state of partitions on a cluster. The following commands are useful:

sinfo -N --Format=Nodelist:20,Available:10,Time:15,Partition:15,StateLong:15,Memory:15,FreeMem:15,CPUsLoad:15,CPUsState:15,GresUsed:15,Features:20,Reason:10

sinfo -p <parition_name> -N --Format=Nodelist:20,Available:10,Time:15,StateLong:15,Memory:15,FreeMem:15,CPUsLoad:15,CPUsState:15,GresUsed:15,Features:20,Reason:10

sinfo --Format=Nodelist:20,Available:10,Time:15,StateLong:15,Memory:15,FreeMem:15,CPUsLoad:15,CPUsState:15,GresUsed:15,Features:20,Reason:10,Partition

Here, we show the output for some the above commands for both ALICE and SHARK. Note that the output will most likely look different when you run it because of the current load of the system and possible changes made to the partition system. We encourage you to try them yourself.

The node states have the following meaning:

The abbreviations (A/I/O/T) in NODES(A/I/O/T) or CPUS(A/I/O/T) mean (Allocated/Idle/Other/Total)

squeue

With squeue, you can get information about your running jobs and jobs from other users:

Jobs typically pass through several states in the course of their execution.
The typical states are PENDING, RUNNING, SUSPENDED, COMPLETING, and COMPLETED. An explanation of some state follows:

Job information and cluster configuration with scontrol

With the Slurm command scontrol you can get a more detailed overview of your running job, node hardware and partitions, e.g.,

[user@res-hpc-lo02 ~]$ scontrol show job 260
JobId=260 JobName=IMB
   UserId=user(225812) GroupId=Domain Users(513) MCS_label=N/A
   Priority=35603 Nice=0 Account=dnst-ict QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:13 TimeLimit=00:30:00 TimeMin=N/A
   SubmitTime=2020-01-23T10:27:45 EligibleTime=2020-01-23T10:27:45
   AccrueTime=2020-01-23T10:27:45
   StartTime=2020-01-23T10:27:45 EndTime=2020-01-23T10:57:45 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-01-23T10:27:45
   Partition=all AllocNode:Sid=res-hpc-ma01:46428
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=res-hpc-exe[013-014]
   BatchHost=res-hpc-exe013
   NumNodes=2 NumCPUs=32 NumTasks=32 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=32,mem=64G,node=2,billing=32
   Socks/Node=* NtasksPerN:B:S:C=16:0:*:* CoreSpec=*
   MinCPUsNode=16 MinMemoryCPU=2G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/home/user/Software/imb/mpi-benchmarks/imb.slurm
   WorkDir=/home/user/Software/imb/mpi-benchmarks
   StdErr=/home/user/Software/imb/mpi-benchmarks/job.%J.err
   StdIn=/dev/null
   StdOut=/home/user/Software/imb/mpi-benchmarks/job.%J.out
   Power=
   MailUser=user@gmail.com MailType=BEGIN,END,FAIL

[user@res-hpc-lo02 ~]$ scontrol show node res-hpc-exe014
NodeName=res-hpc-exe014 Arch=x86_64 CoresPerSocket=12 
   CPUAlloc=16 CPUTot=24 CPULoad=0.00
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=res-hpc-exe014 NodeHostName=res-hpc-exe014 Version=20.02.0-0pre1
   OS=Linux 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019 
   RealMemory=386800 AllocMem=32768 FreeMem=380208 Sockets=2 Boards=1
   State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=all 
   BootTime=2019-12-11T11:51:40 SlurmdStartTime=2020-01-14T15:36:20
   CfgTRES=cpu=24,mem=386800M,billing=24
   AllocTRES=cpu=16,mem=32G
   CapWatts=n/a
   CurrentWatts=0 AveWatts=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

[user@res-hpc-lo02 ~]$ scontrol show partition all
PartitionName=all
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=YES QoS=N/A
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED
   Nodes=res-hpc-exe[013-014]
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=40 TotalNodes=2 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerCPU=2048 MaxMemPerNode=UNLIMITED

Job Resources

Determining what resources to request

Requesting the right amount of resources for jobs is one the most essential aspects of using Slurm (or running any jobs on an HPC).

Before you submit a job for batch processing, it is important to know what the requirements of your program are so that it can run properly. Each program and workflow has unique requirements so we advise that you determine what resources you need before you submit your job.

Keep in mind that increasing the amount of compute resources may also increase the amount of time that your job spends waiting in the queue. Within some limits, you may request whatever resources you need but bear in mind that other researchers need to be able to use those resources as well.

It is vital that you specify the resources you need as detailed as possible. This will help Slurm to better schedule your job and to allocate free resources to other users.

Finding the right settings often requires a bit of trial and error. It usually helps to run a few small test jobs and use their performance to estimate the resources for the production job. After each job, you can check how much of the requested resources was actually use. It is also worthwhile to check this after production jobs to see if you can or should make adjustments.

Below are some ways to specify the resources to ask for in your job script. These are options defined for the sbatch and salloc commands. There are additional options that you can find by checking the man pages for each command or the Slurm website.

Specifying resources and other settings for jobs

Slurm has its own syntax to request compute resources. In addition, Slurm has a number of settings for jobs that make it easier to organize jobs. Below is a summary table of some commonly requested resources and the Slurm syntax to get it. These options can be passed to sbatch or provided in a batch job script using the #SBATCH --option=<value> syntax. For a complete listing of request syntax, run the command man sbatch or check the Slurm website.

Adjusting job settings after resubmission

While scontrol show a powerful command is to show info about your job, with scontrol update, you can change certain settings as long as your job is on hold or pending. First put your job on hold, update the settings and then release your job. Here is brief and generic example:

scontrol hold <jobid>
scontrol update job <jobid> NumNodes=2-2 NumTasks=2 Features=intel16
scontrol release <jobid>

See the man page for the scontrol command or the Slurm website for more information.

Adjusting job priority after submission

You can adjust the priority of your job manually yourself to some extent. You cannot actually change the Priority value because it is not a fixed value, but you can manipulate the “Nice” factor to change the Priority value. The Nice factor will be subtracted from the Priority value that Slurm calculated.

For this, you do not have to hold your pending job, but you can adjust the value directly:

scontrol update job <jobid> Nice=<some_value>

Batch Jobs

Batch processing is the recommended and common way to use a Slurm-controlled HPC cluster. It is a non-interactive workflow for running jobs which is reusable and reproducible if set up correctly. It makes use of a self-contained shell script, a so-called batch script, that is submitted to slurm using the sbatch command. The batch script contains all necessary information for Slurm to setup and run your job without the need for you to take action.

General workflow for batch processing

The workflow for batch processing can be summarized like this:

1. Prepare your job e.g, develop/write/compile program/script/software, download data to the cluster

2. Write your batch script

3. Submit your batch script with sbatch batch_file_name

4. Wait for Slurm to start your job

5. Let Slurm run your job

   1. Optionally, you can monitor your job while it runs

6. Come back after Slurm has finished running your job and retrieve results (and assess job performance)

7. Go back to step 1 or 2 and run more jobs.

You can of course cancel your pending (step 4) or running job (stage 5) at any point in time and go back to stage 2 or even 1, in particular if you encountered any issues.

The batch script

The batch script contains everything Slurm needs to make sure your job runs properly and successfully assuming that your program can run without errors. It is written as

Below, you can find an example for a basic layout of a batch script. Specific examples can be found in the section Tutorials

#!/bin/bash  # Always the first line

###########################
# Settings for slurm
# Not all of them are always necessary
# Some examples,
###########################
#SBATCH --job_-name=<job_name>
#SBATCH --out=%x_%j.out
#SBATCH --mail=<your_e-mail_address>
#SBATCH --mail-type=<set_mail_type>
#SBATCH --partition=<partition>
#SBATCH --time=<d-hh:mm:ss>
#SBATCH --ntasks=<number_of_tasks>
...

###########################
# Set up your software environment
# e.g.,
###########################
# For example, load modules, set environment variables
...

###########################
# Copy your data to local scratch
###########################
# Stage your data on the local scratch
# if you want to avoid using network storage 
# for reading and writing files while your job runs

###########################
# Execute tasks
###########################
# All the commands that you need
# for executing your program
# including the call to your program itself

###########################
# Move data products back to network storage
###########################
# If you staged or wrote data to
# the local scratch storage,
# you probably want to keep some data products
# after your job has finished and move those
# to your scratch directory in network storage

Job arrays

Job arrays allow you to submit multiple jobs with a single batch script. The jobs have to have the same #SBATCH settings to begin with, but subsequent commands in your batch script can make use of SLURM environment variables specific for array jobs that allow you to adapt commands to jobs in the job array. More information is available on the Slurm website: Slurm Workload Manager - Job Array Support (schedmd.com)

The id of array jobs consists of the general job id and the id of the job in the array separate by an underscore, i.e., <jobid>_<array_id_counter>

Batch commands for job arrays

In order to tell Slurm that your batch file should be run as a job array, add the sbatch setting --array your batch script, for example

# For a job array with an array index from 0 till 20
#SBATCH --array=1-20

# For a job array with a specific index
#SBATCH --array=1,3,5,7,9

# For a job array with index from 0 till 20 limited to 4 jobs running at the same time
#SBATCH --array=1-20%4

When submitting an array job, squeue will show the job id as <primary_id>_<array_index>. However, internally Slurm will also log the job id of a job in a job array as <primary_id> + <array_index>

As such you have two options to easily specify your output file with a unique name.

# Option 1 using %j
# Here, %j will be %j=<primary_id> + <array_index> 
#SBATCH --output=%x_%j.out

# Option 2 using %A (primary id) and %a (array index)
#SBATCH --output=%x_%A_%a.out

For seff, you can also specify the job id in both ways.

Limitations for job arrays

In order to limit users from submitting too many jobs that would occupy the cluster for too long, a number of limitations apply:

Environment variables for job arrays

These are some of the Slurm environment variables specific to job arrays:

Interactive Jobs

It is also possible to run interactive jobs on a Slurm-based HPC cluster like ALICE or SHARK. They can be useful for quick interactive tests and data visualization, but they are not recommended for production jobs.

Interactive jobs also require you to specify resources for your job similar to batch jobs and they also have to go through the queue. This means that it depends on the load of the cluster or partition when your interactive job will run and you might have to wait.

The Slurm command salloc allows you to request an interactive job. Here is an example:

If you need X11 forwarding, you can enable it for your interactive session by adding the option --x11, for example:

salloc --ntasks=1 --mem=1G --time=00:05:00 --x11

Job Monitoring

Slurm provides a number of ways for you to monitor the state of your job.

One option is to use squeue to get the overall state.

If you use entire nodes, you can query sinfo to get basic load information.

If your program writes out sufficient information to the slurm output file (or a separate log file), you can check its content on a regular basis.

Another option is to log in to the node on which your job is running. After a job has started to run, Slurm grants you permission to login to the compute node via ssh until the job terminates. You can use this to look at the utilization. Note that anything you do on the compute node, will count against the resources that you requested. So, you should make sure not to run any resource-intensive tasks.

Job Performance

A quick and easy way to get information about the performance of your job is to use the command seff followed by the id of your job, i.e.

seff <jobid>

If you told Slurm to notify you after your job as finished, the e-mail send from slurm will most likely also contain information about used resources.

More detailed performance statistics about your job or the program that you ran cannot be provided through Slurm-based tools. In this case, you need to deploy your own method to gather metrics that allow you to assess the performance.

Cancelling jobs

With the command scancel, you can cancel any of your jobs (running or pending) or even all of your jobs. Here are some example commands:

See the man page for the scontrol command or the Slurm website for more information