pbs queues on gowonda

Owned by Indy Siva
Dec 10, 2015
6 min read

Introduction

We have defined different queues for different purposes. Some queues have restriction placed on them. To get a listing of all the available queues, you may type qstat -q

Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State
---------------- ------ -------- -------- ---- ----- ----- ----  -----
workq              --      --       --     --     12     8   --   E R
testq              --      --       --     --      0     0   --   E R
urgent             --      --       --     --      0     0   --   E R
xl                 --      --       --     --      0     0   --   E R
gpu                --      --       --       4     2     0   --   E R
default            --      --       --     --      0     0   --   E R
backfill            1gb    --    02:00:00    1     0     0   --   E R
holding            --      --       --     --      0     0   --   E R
medium             --      --       --     --      0     0   --   D S
fast               --      --    168:00:0    4     0     0   --   D R
mpi                --      --    999:00:0    4     0     0   --   D R
sp1                --      --    168:00:0    4     0     0   --   E R
gccm               --      --    999:00:0    6     1     0   --   E R
adminq             --      --       --     --      0     0   --   E R
qdong              --      --       --       2     0     0   --   E R
qweek_s            --      --    140:00:0    6     0     0   --   E R
                                               ----- -----
                                                  15     8

Different Queues

workq

This is for all and trhere is no restriction placed on this.

gpu

Restricted to gpu nodes n020, n021, n022 and n023

Currently members of the following unix groups have access to this queue:
nimrodusers, vasp.

Please note the usage:
From command line: qsub -I -q gpu -W group_list=gaussian -l walltime=01:00:00

OR in a pbs script as:

>>>>>>

#!/bin/sh
#PBS -m abe
#PBS -M nnnnn@griffith.edu.au
#PBS -N gpuQueue_test
#PBS -q gpu
#PBS -W group_list=gaussian
#PBS -l walltime=01:00:00
#PBS -l select=1:ncpus=1:ngpus=1
source $HOME/.bashrc
module load cuda/4.0
module list
echo "Hello"
echo "Starting job"
sleep 28
echo "test Done"
echo "Done with job"

>>>>>>>>>>>>

Qmgr: p q gpu
#
# Create queues and set their attributes.
#
#
# Create and define queue gpu
#
create queue gpu
set queue gpu queue_type = Execution
set queue gpu Priority = 250
set queue gpu resources_max.ncpus = 12
set queue gpu resources_max.nodect = 4
set queue gpu resources_min.class = C
set queue gpu resources_default.class = C
set queue gpu acl_group_enable = True
set queue gpu acl_groups = gaussian
set queue gpu acl_groups += nimrodusers
set queue gpu acl_groups += vasp
set queue gpu enabled = True
set queue gpu started = True

mpi

Available to everybody

gccm

Restricted to nodes n040,n041 and n042
set queue gccm acl_user_enable = True

adminq

This is restricted to administrators of the cluster.
set queue adminq acl_user_enable = True

qdong

This is restricted to a group of researchers in School of Engineering, only on node n047.
set queue qdong acl_user_enable = True

qweek_s

This queue is for jobs upto one week walltime or 140 hours. The queue is also desirable for data intensive jobs with the use of fast ssd as local scratch. qweek_s is attached to 3 nodes each with 800GB ssd drive used as local scratch filesystem. It is available to everybody.

If you want to use the queue, be sure make the following changes:

(1)  Change the queue attribute in your pbs script to
#PBS -q qweek_s

(2)  Change walltime  to no more than 140
#PBS -l walltime=120:00:00

If you do not plan to use the fast ssd scratch filesystem, that's all you have to do.
The following steps are for those want to use the local ssd scratch.

(3)  Request the amount of ssd space using the ssd_scratch attribute, in integer format only can be in KB, MB or GB .   
For example, if you want require 1.5GB :
#PBS -l select=1:ncpus=2:ssd_scratch=1.5GB  (WRONG)
#PBS -l select=1:ncpus=2:ssd_scratch=1536MB (CORRECT)

The maxim amount you can request is 734GB.

(4)   Make use of the ssd scratch
To use the ssd scratch, you have to create a directory on the filesystem and named it as the PBS_JOBID. 
Then copy your data to the newly created directory and run your job from there.  At the end of job copy 
your data/results back to your home directory and then delete the directory PBS_JOBID.

You need to modify your pbs script according to the following example:
##########################################################################
#Create a local directory to run and copy your files

mkdir /data/${PBS_JOBID}
cd /data/${PBS_JOBID}

cp path_to_your_files .

run_your_program

# Copy your result back to your home directory or scratch
cp /data/${PBS_JOBID}/* path_to_your_directory

#Clean up your files
/bin/rm -rf /data/${PBS_JOBID}

##########################################################################

Please make sure you always use the name $PBS_JOBID as your directory, otherwise your directory may get deleted by a cron job.  
And also make sure to copy your data back and delete afterwards.

Usage

Reference