Information Technology and Computing Support

If you want to request resources for an interactive session or single command, you can use the Slurm command "srun":  

srun {srun options} {command or shell}

See examples of srun below:

To start an interactive (pseudo-terminal bash shell) session on the cluster:

srun --pty bash

The tcsh and other shells can be used instead of bash for srun (and sbatch) if that is preferred.  

To request an interactive tcsh session with X11 forwarding (e.g. for running a GUI) from the default (share) queue for more than the default 12 hours, e.g. 3 days:

srun --time=3-00:00:00 --x11 --pty tcsh

To request an interactive bash session with 8 OpenMP threads/cores and 50GB RAM from the preempt queue:

srun -p preempt -c 8 --mem=50G --pty bash

To request an interactive bash session to run an application using 12 OpenMP threads and 4GB per thread/cpu from the share queue:

srun --cpus-per-task=12 --mem-per-cpu=4G --pty bash

To request an interactive zsh session with a GPU from the dgxh queue:

srun -p dgxh --gres=gpu:1 --pty zsh

To request an interactive session with a GPU from either the dgx2, gpu or eecs queues using the eecs account:

srun -A eecs -p dgx2,gpu,eecs --gres=gpu:1 --pty bash

To request an interactive session with two GPUs on a specific node in the dgx2 queue using the eecs account:

srun -p dgx2 --nodelist=dgx2-1 --gres=gpu:2 --pty bash

To request an interactive bash session from the class queue using the cs499 account:

srun -A cs499 -p class --pty bash

To request an interactive session to run an MPI application with 8 processes over an arbitrary number of nodes from the preempt queue:

srun --partition=preempt -n 8 --pty bash

To request an interactive session to run an MPI application on 2 nodes with 8 processes per node from the share queue:

srun -N 2 --ntasks-per-node=8 --pty bash

To request an interactive tcsh session on a compute node running the "el8" operating system, from the preempt queue:

srun -p preempt --constraint=el8 --pty tcsh

To request an interactive session with one GPU on a compute node with an A40 GPU, from the preempt queue:

srun -p preempt --gres=gpu:1 --constraint=a40 --pty bash

For more information or options on srun, check out the manual page by typing "man srun".

Submitting batch jobs are useful when you need to run several jobs, or long jobs.  Batch job submissions will not be interrupted by lost network or VPN connections. The Slurm command "sbatch" is used to submit batch jobs.

To submit a batch job to the cluster:

sbatch myBatchFile.sh

where myBatchFile.sh is a script containing the commands you wish to run as well as a list of SBATCH directives specifying the resources or parameters that you need for your job.  See the sample myBatchFile.sh batch script below.  

#!/bin/bash
#SBATCH -J helloWorld						  # name of job
#SBATCH -A mySponsoredAccount	  # name of my sponsored account, e.g. class or research group, NOT ONID!
#SBATCH -p share								  # name of partition or queue
#SBATCH -o helloWorld.out				  # name of output file for this submission script
#SBATCH -e helloWorld.err				  # name of error file for this submission script

# load any software environment module required for app (e.g. matlab, gcc, cuda)
module load software/version

# run my job (e.g. matlab, python)
mySoftwareExecutable

There are additional sample batch scripts located in /apps/samples, so feel free to copy them to your directory and modify them according to your needs.

Here are some other useful SBATCH directives that can be included in a batch script to specify additional resources or different parameters from the defaults:

#SBATCH -p dgx2                                   # partition (default share)
#SBATCH --partition=dgxh                   # partition (default share)
#SBATCH -A mySpecialAccount          # account name to access special or private partition
#SBATCH --account=mySpecialAcct  # account name to access special or private partition
#SBATCH -t 24:30:00                            # time limit on job: 24 hours, 30 minutes (default 12 hours)
#SBATCH --time=2-12:30:00                # time limit on job: 2 days, 12 hours, 30 minutes (default 12 hours)
#SBATCH --gres=gpu:1                         # number of GPUs to request per node (default 0)
#SBATCH --gpus-per-node=1              # number of GPUs per node (default 0)
#SBATCH -G 1                                        # number of GPUs per job (default 0)
#SBATCH --gpus-per-job=1                  # number of GPUs per job (default 0)
#SBATCH --mem=10G                          # request 10 gigabytes memory per node (default depends on node)
#SBATCH --mem-per-cpu=10G           # request 10 gigabytes per core/cpu
#SBATCH -c 4                                         # number of cores/threads per task (default 1)
#SBATCH --cpus-per-task=2                # number of cores/threads per task (default 1)
#SBATCH -n 3                                         # number of MPI tasks per job (default 1)
#SBATCH --ntasks=3                             # number of MPI tasks per job (default 1)
#SBATCH --ntasks-per-node=3           # number of MPI tasks per node (default 1), use for multiple nodes (-N>1)
#SBATCH -N 2                                        # number of nodes per job (default 1)
#SBATCH --nodes=2                              # number of nodes per job (default 1)
#SBATCH --nodelist=node1,node2...  # required node, or comma separated list of required nodes
#SBATCH --exclude=node1,node2...  # exclude node, or comma separated list of excluded nodes
#SBATCH --constraint=el9                   # request node running el9 operating system
#SBATCH --constraint=avx512            # request node with AVX512 instruction set
#SBATCH --constraint=a40                  # request node with A40 GPU
#SBATCH --mail-type=FAIL                  # send email notification if the job fails
#SBATCH --mail-type=BEGIN,END     # send email notification when the job starts and completes
#SBATCH --mail-type=ALL                   # send email notification when job starts, ends, fails, or requeues
#SBATCH --mail-type=TIMELIMIT_90 # send email notification when job reaches 90% of time limit

For more information or options on sbatch, check out the manual page by typing "man sbatch".

Here are the currently available node features that can be requested using the Slurm "--constraint" option:

#SBATCH --constraint=el8                   # request node running RHEL8 based operating system
#SBATCH --constraint=el9                  # request node running RHEL9 based operating system

#SBATCH --constraint=avx                  # request node with AVX instruction set
#SBATCH --constraint=avx2                # request node with AVX2 instruction set
#SBATCH --constraint=avx512            # request node with AVX512 instruction set

#SBATCH --constraint=haswell          # request node with Intel Haswell CPU
#SBATCH --constraint=broadwell      # request node with Intel Broadwell CPU
#SBATCH --constraint=skylake           # request node with Intel Skylake CPU
#SBATCH --constraint=cascadelake    # request node with Intel Cascadelake CPU
#SBATCH --constraint=epyc              # request node with AMD Epic CPU

#SBATCH --constraint=m60                 # request node with M60 GPU
#SBATCH --constraint=t4                     # request node with T4 GPU
#SBATCH --constraint=rtx6000           # request node with RTX6000 GPU
#SBATCH --constraint=rtx8000           # request node with RTX8000 GPU
#SBATCH --constraint=v100                # request node with V100 GPU
#SBATCH --constraint=a40                  # request node with A40 GPU
#SBATCH --constraint=h100               # request node with h100 GPU
#SBATCH --constraint=h200               # request node with h200 GPU
#SBATCH --constraint=l40s               # request node with L40S GPU

Note that when requesting a node with a specific GPU using the constraint option, you must still use the "--gres" option to request the GPU itself.

To cancel a job on the cluster:

scancel {jobid}

To cancel all of your jobs on the cluster:

scancel -u {ONID}

 To cancel all of your jobs on one partition:

scancel -p {partition}

For more information or options on scancel, type "man scancel".

To view a list of queues and their status:

sinfo

For more information or options on sinfo, type "man sinfo".

To view status and available resources (CPU, GPU, RAM) of all nodes in a partition:

nodestat {partition}

To view the list and status of all jobs in all queues:

squeue
-or-
sq

To view the list and status of all jobs in one partition or queue:

squeue -p {partition}

To view the list and status of your jobs:

squeue -u {ONID}
-or-
squ

For more information or options on squeue, type "man squeue".

To view status of jobs in the queue but with an alternate, longer listing format:

sqe

To see more detailed information on a specific pending or running job:

showjob {jobid}

Here is a list of partitions, their descriptions, and a list of accounts that have access to them:

Here is a list of some partitions and their time and resource limits:

The CPU/GPU/GB times are the amount of CPU/GPU/GB time you can have running at one time.

The total number of jobs that one user can submit to all partitions at once is 1000. 

The total number of jobs that one user can have running in all partitions at once is 400.