Batch Queue Configuration

Batch jobs on the Beowulf compute cluster are managed by TORQUE, a PBS job scheduler. For a short description of basic TORQUE commands see the article Using TORQUE to Submit and Monitor Jobs.

There are three queues on Beowulf, the primary queue batch and two execution queues: aqm and hpc. Even though you are able to submit jobs to the execution queues, we recommend you always submit to the primary queue: batch.

This is easy. Since batch is the default queue, if you don't ask for a particular queue your job will be routed to the appropriate execution queue automatically.

At this time, execution queues on Beowulf place no restrictions on walltime.

About scratch space

Depending on your job, you may want to use scratch space for temporary storage rather than your shared home or project directory. Read and write access to local scratch space will be faster than to network shares.

The job scheduler automatically creates a scratch directory on each node allocated to your job. The scratch directory is created when your job starts and lasts only for the duration of the run. When your job finishes, the scratch directory is deleted and all files removed. If you want to save any files in the scratch directory, your job should copy them to permanent storage in your home or project directory.

The compute nodes in the Beowulf cluster each allocate about 20GB for use as scratch space. This 20GB is shared by all jobs running on the compute node. If your job is the only one running, you may use all 20GB. Otherwise, you share - first come, first serve.

Access the temporary scratch directory through the shell environment variable $TMPDIR. This variable will contain the name of your scratch directory. Use the environment variable in a job script like this:

# This script copies a file named data1.dat from my home directory
# to my batch scratch space. The program myprog reads this copy
# and writes a new file named data2.dat also to scratch space. If myprog
# completes sucessfully, data2.dat is copied to my home directory 

$ cp $HOME/data1.dat $TMPDIR/data1.dat
if [ $? -eq 0 ]
then
         ./myprog <$TMPDIR/data1.dat >$TMPDIR/data2.dat
           if [ $? -eq 0 ]
           then
                       cp $TMPDIR/data2.dat $HOME/data2.dat
         fi
fi

Important note: If you define $TMPDIR on your own, in your job script or your .profile, the job scheduler will not change it. So, you will not be able to use it to access your job's temporary scratch directory.