SLURM Advanced Submit Techniques

Advanced Submit Techniques

This document provides information on job submission techniques that do not fall into the standard serial/parallel model and need additional considerations. If you know of any other techniques that can be helpful, please let us know so we can include them here.

SLURM Array Tasks

SLURM provides a mechanism to allow users to submit a single job which will be spawned as a range of tasks. Lets say you have a problem such as the following:

• You are using a parallel application that runs best on 8 processors
• You have 48 different data sets you need to run with this application
• You should not iterate the execution of the parallel job within your submit script, like

#SBATCH --name=my_app
#SBATCH --ntasks=8
#SBATCH --time=10:00:00


for i in {1..48}; do
      mpirun -np $SLURM_NTASKS ./my_app inputfile.$i
done

because for longer MPI jobs, its not fair to other users.

With SLURM, you can tell the scheduler that your job is an array task. To do this, you’ll need to add the —array= option to your submit script. Here’s an example:

#SBATCH --name=my_app_array_run
#SBATCH --ntasks=8
#SBATCH --time=00:45:00
#SBATCH --array=1-48

mpirun -np $SLURM_NTASKS ./my_app inputfile.$SLURM_ARRAY_TASK_ID"

You can now arrange your input files, inputfile, as inputfile.[1-48], and submit the tasks to SLURM via the below sbatch command:

[user@host ~]$ sbatch my_app_array_run.sh

With the —array option, SLURM will execute your submit script 48 times. The only difference between these jobs will be the value of $SLURM_ARRAY_TASK_ID which you can use to direct your application to the correct data sets.

A slightly more complex example uses $SLURM_ARRAY_TASK_ID as an index variable to define, for example, a temperature range. Say you want to calculate some property of a system at various temperatures. It is unnecessary to generate all of the input files before hand if you are incrementing through a temperature range. This can be done on the fly with a little shell scripting and regular expressions:

#SBATCH --name=my_app_array_run
#SBATCH --ntasks=8
#SBATCH --time=00:45:00
#SBATCH --array=1-48

# Set the new temperature
BASE_TEMP=100
TEMP=`bc -q < inputfile.$SLURM_ARRAY_TASK_ID

mpirun -np $SLURM_NTASKS ./my_app inputfile.$SLURM_ARRAY_TASK_ID

This will allow you to automatically generate your input file for every run by substituting a line in the template input file with a new line declaring the temperature you wish to work with.

Iterative Parallel Jobs

The Iterative Parallel job is similar to the job described above, but each iteration must be run in sequence because each subsequent run of your application depends upon some data element generated by the previous run. As before, since the scheduler cannot do something like this:

#SBATCH --name=my_app
#SBATCH --ntasks=8
#SBATCH --time=10:00:00

for i in {1..48}; do
      mpirun -np $NSLOTS ./my_app inputfile.$i
done

we are stuck with a problem: How do we run an iterative parallel job without having to modify inputs and resubmit a new job after each iteration?

Job Dependencies

With creative use of SLURM’s array task facility, we can ensure a sort of “blocking” behavior between each run of your job. As you can see in the example below, we have created a submit script which extracts necessary data from previous runs in order to create the parameters for our subsequent runs. This script is for a hypothetical example and will need to be modified to suit your needs:

#SBATCH --name=my_app_array_run
#SBATCH --ntasks=8
#SBATCH --time=00:45:00
#SBATCH --array=1-10%1

# Lets take a specific result from the previous run and the first run
# to form the basis of the next run
if [ $ITR -gt 1 ]; then
     NEW_PARAM_1=`awk '/^RESULT.*\=.*[0-9]+\.[0-9]+$/ { print $NF}' output.$SLURM_JOB_NAME.$((SLURM_ARRAY_TASK_ID - 1))
     NEW_PARAM_2=`awk '/^RESULT.*\=.*[0-9]+\.[0-9]+$/ { print $NF}' output.$SLURM_JOB_NAME.1 | \

     # Generate our new input file
     sed -r "s/PARAMETER1(\s)+\=(\s)+([0-9])+\.([0-9])+/PARAMETER1 = $NEW_PARAM_1/" \
                inputfile.template > inputfile.$SLURM_JOB_NAME.$SLURM_ARRAY_TASK_ID
     # Do the replacement 'inline' to the existing inputfile
     sed -ri "s/PARAMETER2(\s)+\=(\s)+([0-9])+\.([0-9])+/PARAMETER2 = $NEW_PARAM_2/" \
                inputfile.$SLURM_JOB_NAME.$SLURM_ARRAY_TASK_ID
fi

mpirun -np $NSLOTS ./my_app inputfile.$SLURM_JOB_NAME.$SLURM_ARRAY_TASK_ID

We can submit the following like so, to run 10 steps:

[user@host ~]$ sbatch my_app_array_run.sh

In this example, we parse out a particular figure from the previous run’s output (defined as output.$SLURM_JOB_NAME.$((SLURM_ARRAY_TASK_ID - 1)) and we use that value as a parameter in our next run.