Submitting Jobs

The qsub command

Jobs are submitted for execution on the cluster by using the qsub command on the submission host (hpc1.cs.bham.ac.uk). All jobs submitted to Torque must be in the form of Shell scripts (Torque will reject attempts submit binaries). It may seem rather cumbersome to be forced to write shell script wrappers for your programs but it is actually very useful. Not only can you include all the setup and configuration steps necessary to run your code but you can also embed commands that are interpreted by Torque to control the way your job is run on the cluster. More information about scripting can be found later in this page.

The simplest way to submit a job would be to execute the command below:

qsub my-script.sh

In this example, my-script.sh is a shell script containing all the commands necessary for running the job. The job would be submitted to the cluster and a confirmation message would be displayed, giving the unique identifier assigned to the job. It is important to note the job ID number since it is required if you wish to modify, cancel or monitor the status of the the job.

Some useful command line options for qsub are listed below. The man page for qsub provide far more detail about the command and its options

Command line options

-l walltime=HH:MM:SS Set the maximum runtime for your job. The default is 2 days, so unless this option is set, any jobs taking longer than 2 days will be killed.
-M email Set email address for notification
-m be Notify user by email when job starts and completes
-o file Redirect STDOUT to file
-e file Redirect STDERR to file
-j y Merge STDOUT and STDERR
-S shell Execute the script using the specified shell rather than the default csh/tcsh
-t index-range Define an array job (see below)
-l complex=value Make a resource request for the complex to have value

Notes

  • Unless otherwise specified, the job is run from the user's home directory irrespective of where the job is submitted (home directories are the default working directories for jobs).
  • By default, the output of the script is redirected to files in the user's home directory with a prefix of the job ID and the letter o (STDOUT) or e (STDERR).
  • Please specify your e-mail address if using e-mail notification when your job starts/ends. If not specified, Torque may guess your e-mail address incorrectly and the messages will bounce!

Array jobs

Array Jobs allow users to repeatedly execute the same job on the cluster. Array jobs are defined using task index notation which has the syntax lower-upper:step. The individual jobs of an array job are allocated unique identifiers consisting of the job id for the array job followed by a task index. The task index is an integer in the range lower to upper with a step size of step.

Example:

qsub -t 1-5:1 my-script.sh

Runs the script my-script.sh five times on the cluster. Each instance will be given a task index in the range 1, 2, ... ,5

Shell scripts

Rather than controlling jobs by specifying qsub options on the command line, these options can be embedded in the jobs's shell script. The syntax for embedding qsub options is to include them after the character sequence #PBS i.e., they appear as comments if the script were to be executed from the command line. A number of "environment variables" relating to the current job can also be referenced within the shell script such as JOB_ID (the job's ID number), HOSTNAME (name of the host where the job is executed), etc.

-

detailed descriptions of all the qsub options and the environment variables that can be referenced within scripts are given in the man page for qsub.