Job Prototyping

This document will provide information on prototyping jobs for maximum efficiency.

/usr/bin/time -v [command/script/program]

This runs the specified command and outputs system information. Wall clock (elapsed time) and resident set size (memory utilization) are the most important items. The results of this command can be utilized as submission parameters to the cluster. You can add /usr/bin/time -v to a single command in your sbatch script, or you can use srun to test your whole script.

Here is an example sbatch script named dd.sh. It creates a 20GB file and then makes a copy of it. I'm unsure how many resources it will take to complete the script. A best guest would be that it will not take more than 4GB of ram and 5 minutes of runtime. So I will start a srun session with these parameters. I can then run /usr/bin/time -v with my shell dd.sh.

[user@login0 ~]$ srun --mem=4096 --time=00:05:00 --pty /bin/bash
srun: job 6543210 queued and waiting for resources
srun: job 6543210 has been allocated resources
[user@svc-3024-9-28 ~]$ /usr/bin/time -v /work/u/user/dd.sh
20+0 records in
20+0 records out
20971520000 bytes (21 GB) copied, 30.8695 s, 679 MB/s
        Command being timed: "/work/u/user/dd.sh"
        User time (seconds): 1.38
        System time (seconds): 15.00
        Percent of CPU this job got: 30%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:53.58
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1024788
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 1226133
        Voluntary context switches: 6061
        Involuntary context switches: 28
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
...

The wall clock was 53.58 seconds and the maximum resident set time was 1024.8 megabytes. So the parameters that I would set in dd.sh before submitting to SLURM are :

#!/bin/bash
#SBATCH --job-name=dd
#SBATCH --time=00:01:00
#SBATCH --mem=1200

dd if=/dev/zero of=/work/u/user/testfile bs=1000M count=20
cp /work/u/user/testfile /work/u/user/copyoftestfile

Notice that I over estimated my parameters by a small amount. This will ensure my job will run, while minimizing my resources and impact on the system and other users.

If I wanted to add another command to dd.sh, I could use SRUN and /user/bin/time -v to test just that command.

[user@login0 ~]$ srun --mem=2048 --time=00:01:00 --pty /bin/bash
[user@svc-3024-1-1 user]$ /usr/bin/time -v mv /work/u/user/testfile /home/u/user/Documents/testfile
        Command being timed: "mv /work/u/user/testfile /home/u/user/Documents/testfile"
        User time (seconds): 0.00
        System time (seconds): 10.63
        Percent of CPU this job got: 29%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:36.13
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 4996
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 4451
        Voluntary context switches: 5102
        Involuntary context switches: 6
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Since it took 36.13 seconds and almost 5MB of RAM to move the tesfile. I will add that to my dd.sh script.

#!/bin/bash
#SBATCH --job-name=dd
#SBATCH --time=00:02:00
#SBATCH --mem=1200

dd if=/dev/zero of=/work/u/user/testfile bs=1000M count=20
cp /work/u/user/testfile /work/u/user/copyoftestfile
mv /work/u/user/testfile /home/u/user/Documents/testfile

Selecting Which Partition To Use

These are tools and guides to help you select the appropriate partition to run your job.

Development Partitions

hchg is meant specifically for testing and development. Make use of this partition if available.

See all partitions and their guidelines here: SLURM_Partitions

scontrol show partition

Use this command to check partition State, QOS, CPU, and Memory to find which one to submit your job to. This will allow you to find a partition that fits your needs and if it is immediately available. You can specify a particular partition such as :

[user@login0 ~]$ scontrol show partition hchg
PartitionName=hchg
 AllowGroups=ALL AllowAccounts=ALL AllowQos=hchg,interactive,trouble-shooting
   AllocNodes=ALL Default=NO QoS=N/A
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED
   Nodes=hchg-c[0-1]
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=CANCEL
   State=UP TotalCPUs=48 TotalNodes=2 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerCPU=512 MaxMemPerNode=UNLIMITED

The command sinfo is also useful to see partition status. See SLURM_Status for more information.