Table of Contents

*Qs 1: How do I cite or mention the cluster in papers? or what is the preferred method?

...

No Format

You could try a trace.
cmake --trace . 2>&1 | tee /tmp/cmakeOut.txt

You may have to explicitly mention the path, For example:
cmake . -DLAPACK_LIBRARIES=/sw/library/lapack/lapack-3.6.0/3.6.0/lib64/liblapack.so -DBLAS_LIBRARIES=/sw/library/blas/CBLAS/lib/cblas_LINUX.so
 
OR:
 cmake . -DLAPACK_LIBRARIES=/sw/library/lapack/lapack-3.6.0/3.6.0/lib64/liblapack.so -DBLAS_LIBRARIES=/sw/library/blas/CBLAS/lib/cblas_LINUX.so -DCMAKE_INSTALL_PREFIX=/sw/simbody/353

Qs28: How do I check number of CPUs my job is using

No Format

You can find out on which compute node your job is running:
qstat -1an|grep snumber
>>>>>>>>>>>>>>>>>>>
e.g:
qstat -1an|grep s2761086
4598354.pbsserv s2761086 workq    DT_k-e_04   21795   1   1   10gb 99999 R 523:2 n010/0
Here you see it is running on n010
Then you can do this:
ssh nodename -t "htop"  or ssh nodename -t "htop -u username" 
e.g.ssh n010 -t "htop -u s2761086"
Press <F2> key, go to "Columns", and add PROCESSOR under "Available Columns".
The currently used CPU ID of each process will appear under "CPU" column.

Qs29: How to customize an environmental variable using modules

...

Qs 31: Multi Cores are requested and allocated by PBs but job runs only on 1 core. Why is that?

This contribution is from Nicholas Dhal and is acknowledged. Nick is an active Grifith HPC user.

>>>>>>>>>

When a job is executed, the processes running the job are divided up amongst the requested number of cores on the HPC compute node. The processes have an associated setting called 'affinity' which is a logical (true or false) test that specifies which CPU core the processes are allowed to run on. In serial cases, this usually makes no difference because everything is on a single CPU only. However, in parallel runs with multiple cores, sometimes the affinity is set incorrectly by the scheduler and this limits performance.

...

The command 'htop' uses some CPU resources on the node, so limit your time logged into the node (no more than 10 minutes). To exit 'htop' use the key 'F10'. To exit the node use the command 'exit'.

goo.gl/koYII3

>>>>>>>>>>

Qs 32: How to check the remaining licenses on the license server

No Format

/usr/local/bin/lmstat -a -c 27006@gc-prd-erslic.corp.griffith.edu.au
/sw/sysadminmisc/intelflexlm/lmstat -a -c 27006@gc-prd-erslic.corp.griffith.edu.au
/usr/local/bin/lmstat -a -c 27004@gc-prd-erslic.corp.griffith.edu.au

...

No Format

#!/bin/bash 
#PBS -m abe
#PBS -M YOUREMAIL@griffith.edu.au
#PBS -N Inversion
#PBS -l select=1:ncpus=16:mpiprocs=16:mem=1gb,walltime=300:00:00
##     processor cores
slots=16
cd $PBS_O_WORKDIR
module load mpi/openmpi/4.0.2
module load cmake/3.15.5        
echo "Starting job"
echo Running on host `hostname`
echo Time is `date`
echo Directory is `pwd`
make clean
make SolveInversion_NoRot
mpiexec $PBS_O_WORKDIR/SolveInversion_NoRot > $PBS_O_WORKDIR/Inversion_export.dat
#NP=`wc -l < $PBS_NODEFILE`
#mpirun  -hostfile $PBS_NODEFILE -np $NP mdrun_mpi_d  -deffnm md01

intelmpi (MPI_DIR = /sw/intel/ps/2019up5/compilers_and_libraries_2019.5.281/linux/mpi/intel64)

...

e.g scp -r /export/home/s5284664/folder n061:/lscratch/s5284664/

Qs.56: NCMAS process and application

NCMAS facilities overview and who should apply
https://youtu.be/7ZZVk4HtdDY

NCMAS process and application 2021
https://youtu.be/hmV_j5GFgI0

Qs 57: What kind of storage and compute is available on Griffith HPC

...

The mount point will not be seen by compute nodes and hence, it needs to be copied to /scratch (or home directory). Once it is copied to /scratch, it can be seen by all compute nodes and used as working data from then onwards. The problem is that depending on the size of the data, this can take a while. The output needs to be copied back to the research drive. All of this cannot be done as part of the pbs script and needs to be performed manually. Also, there is the risk of leaving the data behind on /scratch space after use and then forgetting about it. Over time, the /scratch space will fill up and disrupt your jobs and other user’s HPC operations.
The copy in and copy out process will be much slower as it will not be using the fast infiniband interconnect network within the HPC cluster. It would use Griffith network to transfer files. This copy in and copy out will be additional overhead for the researcher/research group.
The shared folder in /scratch will have space limitation (200GB typically). Hence, keeping it well managed is critical.

Data ingress and egress

There is no internet access from the HPC to outside world to bring in lots of data from outside. A limited facility using proxy for application installs is allowed. Within Griffith network, a two hop transfer to local desktop and HPC is suggested.

Please contact cluster admin to discuss this and other cluster issues so that we can enable your HPC work.

...

Additionally and in parallel, you can also apply directly when the application opens
https://my.nci.org.au/mancini/ncmas/2022/
Please note that projects will be given a fixed allocation which is given per quarter on a use it or loose it basis. Allocations cannot be carried forward or backward into other quarters. Standard disk space per project is 75GB in /scratch and if a project needs more you will need to contact help@nci.org.au.

Students cannot be a lead CI on an NCI project however, for the QCIF share postdocs can be. For NCMAS the lead CI is required to have an ARC or NHMRC grant or equivalent which is why larger groups apply for NCMAS. A grant is not required for a project under QCIF. However, the QCIF allocations are small, around 20-50 thousand per quarter. Larger allocations are only available through NCMAS.

Some applications like Mathematica and Matlab are licensed software. Mathematica is only available to ANU researchers on NCI. For Matlab, Griffith will need to get in contact with NCI to set up their institutional license. At the moment this is not available so one cannot use it. Unless you have your own license. But also in that case you would need to get in touch with NCI first to see if you can use Matlab on Gadi or not.

In general, allocations are given in service units SUs. 1 core hour is charged at 2 SUs. So if you have a calculation running using 4 cores and taking 48 hours then you will be charged 4*48*2=384 SUs for that calculation.

If a larger disk space (e.g 300GB) is needed, you would need to talk to NCI to increase the space in /scratch to accommodate this. If a larger RAM (e.g 400GB ), then you would need to make sure you run in a queue that supports that ram request. They could be charged more than 2 SUs per core hour though, so you would need to factor that in.

But talk to NCI, help@nci.org.au, first to see if you could use the application (e.g Matlab) onNCI before you even consider applying for an allocation on NCI.

Qs69: How to install bioinformatics software in your home directory

...

Reference: https://kb.hlrs.de/platforms/index.php/Batch_System_PBSPro_(vulcan)#DISPLAY:_X11_applications_on_interactive_batch_jobs

The login or head node of each cluster is a resource that is shared by many users. Running a GUI job on the login node is prohibited and may adversely affect other users. X11 Forwarding is only possible for interactive jobs.

Please note that there is a performance penalty when running a GUI job on the compute nodes using the method outlined below.

Set up X11 forwarding

To use X11 port forwarding, Install Xming X Server on Windows laptop/desktop first. Install the xming fonts package as well.
See instructions here: https://griffith.atlassian.net/wiki/spaces/GHCD/pages/4035477/xming

...

No Format

https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Exporting the environment.yml file
Note

If you already have an environment.yml file in your current directory, it will be overwritten during this task.

Activate the environment to export: 

Note

Replace myenv with the name of the environment. 

module load anaconda3/20212023.1109

source activate myenv

Export your active environment to a new file:

conda env export > environment.yml
Note

This file handles both the environment's pip packages and conda packages.

Email or copy the exported environment.yml file to the other person.



To setup this environment:
module load anaconda3/20212023.1109
source /usr/local/bin/s3proxy.sh
conda env create -n ENVNAME --file ENV.yml
e.g
conda env create -n labmcdougall --file /tmp/environment.yml
conda env create -n labmcdougallRoot --file /tmp/environment.yml
#To remove an environment, conda remove -n env --all

...

http://mobaxterm.mobatek.net/download-home-edition.html
putty
Filezillia
Windows WSL system lets you run the linux versions of ssh under windows.
wsl --install
This should get you command line: ssh, scp, and sftp;

...

To transfer from remote to locally: scp -r remotehostname:/export/home/snumber/FolderName ~/

e.g scp -r s5323827@10.250.250.3:/export/home/s5323827/FolderName ~/

To transfer from local to remote:

scp -r Foldername s5323827@10.250.250.3:/export/home/s5323827/

You will get a pingID MFA request for this.

Qs80: Tunnel setup

Tunneling setup:

...

No Format

n061 nvidia driver is 530.30.02 and cuda toolkit 12.1. As this is the latest (as of March 2023), pytorch was not compatible. The nvidia drivers have been downgraded to 520.61.05 and cuda toolkit 11.8. Even after the downgrade, torch still could not detect a cuda device.The workaround was to manually compile it 

Here is the installation notes:
>>>>>>
Edit ~/.condarc (vi ~/.condarc)
>>>>>>
channels:
  - defaults

envs_dirs:
  - /lscratch/s12345/.conda/envs
  - /export/home/s12345/.conda/envs

>>>>>>>
source /usr/local/bin/s3proxy.sh
module load anaconda3/2022.10
module load gcc/11.2.0
module load cmake/3.26.4
module load cuda/11.4
#If you have an existing environment, you can use it. If not create it with: conda create -n myTorch
source activate modelT5bmyTorch
conda install astunparse numpy ninja pyyaml mkl mkl-include setuptools  cffi typing_extensions future six requests dataclasses #cmake
conda install -c pytorch magma-cuda118

mkdir  /tmp/bela  #Any name is fine. Here I named it bela. It is temp dir
cd /tmp/bela
git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
git checkout v1.13.1 # Or any version you want
git submodule sync
git submodule update --init --recursive
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py install 2>&1 | tee pythonSetupLogs.txt
>>>>>>>

Now you can test your installation:
source activate myenv
python
Python 3.10.11 (main, Apr 20 2023, 19:02:41) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>>

Another test:
python isCuda.py 

CUDA AVILABLE

>>>>cat isCuda.py<<<<<
import torch
if torch.cuda.is_available():
    print('CUDA AVILABLE')
else:
    print('NO CUDA')
>>>>>>>>>>>>>>>>>>
 
To install torchvision from source:
conda install -c conda-forge libjpeg-turbo
git clone https://github.com/uploadcare/pillow-simd
cd pillow-simd
python setup.py install 2>&1 | tee pythonInstallPillowSimd.txt
git clone https://github.com/pytorch/vision.git
cd vision
git checkout v0.14.1
python setup.py install 2>&1  | tee pythonInstalltorchvision.txt
 For tensorflow:
conda install tensorflow=2.12.0=gpu_py311h65739b5_0  -c pytorch -c nvidia
>>>>>>>>>
Another way:
source /usr/local/bin/s3proxy.sh
module load anaconda3/2023.09
conda create -n mytorchA100 -c pytorch -c nvidia
source activate mytorchA100
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
conda install tensorflow=2.12.0=gpu_py311h65739b5_0  -c pytorch -c nvidia
>>>>>>>>
To ensure that PyTorch was installed correctly, we can verify the installation by running sample PyTorch code. Here we will construct a randomly initialized tensor.

import torch
x = torch.rand(5, 3)
print(x)

The output should be something similar to:

tensor([[0.3380, 0.3845, 0.3217],
        [0.8337, 0.9050, 0.2650],
        [0.2979, 0.7141, 0.9069],
        [0.1449, 0.1132, 0.1375],
        [0.4675, 0.3947, 0.1426]])



Source: https://pytorch.org/get-started/locally/

...

No Format

Requests of this type should be directed to IT Service Centre to raise though the Digital Solutions service management system (Cherwell / GSM). 
The easiest way to do this is to email your request to ithelp@griffith.edu.au. The request will then be redirected to the correct team. 
Given the email list is an LML, this will be Identity Services. 
(e.g Reference GSM case Incident#1081850)
The solution provided for this was this:
>>>>>>
Regarding your Service Request 1081850 emails sent to LML email list are not received by me, logged on 7/11/2023 2:49 PM, 
we have the following question or update:

We have noticed a discrepancy in the Group Membership for LML - GHPC and your Staff Azure GroupMembership was missing from this Azure Group used for email.  A re-synchronization of this group has increased the associated Azure Group Membership from 178 to 216 members of which you were one of the people missing. Thank you for notifying us of this anomaly.

You will receive emails for those sent to this group from now on.

None of the other LML - Groups (you have membership for) have Staff Azure/Staff Email as a Target System  and would never generate email.

This Service Request will now be marked resolved on the basis of the action taken and information provided.

>>>>>>

Qs88: Install perl modules without root access.

No Format

Install perl locally

>>>>>>>>installation notes<<<<<<<<<<<<
Download the perl source file and unzip/untar
wget https://www.cpan.org/src/5.0/perl-5.38.2.tar.gz --no-check-certificate  
 ./Configure -des -Dprefix=~/sw/perl/5.38.2 2>&1 | tee configureLogs.txt
   make 2>&1 |tee makeLogs.txt
   make test 2>&1 | tee  maketestLogs.txt
   make install 2>&1 | tee makeInstallLogs.txt
>>>>>>>>End of perl installation<<<<<

echo "check_certificate = off" >> ~/.wgetrc

Edit the .bash_profile file and add the local MODULEPATH
>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
vi ~/.bash_profile
export MODULEPATH=$MODULEPATH:~/sw/Modules
>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
#copy the main perl module file locally
 cp -r /sw/Modules/lang/perl/ ~/sw/Modules
module load perl/5.38.2

You now have a a copy of the perl installation in your ~/sw/perl directory and made a module file from ~/Modules/perl/5.38.2

source /usr/local/bin/s3proxy.sh

I guess you want to install cpan -i File:FindLib
This will install this module now. Simple press return for username and password for proxy.

Qs89: n061 gpu node : pytorch usage

No Format
module load anaconda3/2022.10 source activate TorchA100

Qs90: How to run the pytorch (container method and conda env method) on the ICT cluster

Container method:

Ref1: Singularity#Convertingadockerimageintosingularityimage

No Format

Minimal docker images are kept here:
/sw/Containers/docker

mkdir -p /export/home/snumber/sw/Containers
cd /export/home/snumber/sw/Containers
Convert the tarball to a Singularity image. 
module load singularity/4.1.3 
singularity build --sandbox pytorch docker-archive:///sw/Containers/docker/pytorchnew.tar
(it takes quite sometimes to create the sandbox. Please be patient. You only have to do this once!)

for testing:
singularity shell -e -B /scratch/snumber:/scratch/snumber -B /export/home/snumber:/export/home/snumber /export/home/snumber/sw/Containers/pytorch

e.g:
Singularity> python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True

So we see it works!

So now you need to integrate this into a slurm or pbs script. 
PBS examples are here



Slurm would be like this:

create a run script:
mkdir /export/home/snumber/slurm/data
create a file /export/home/snumber/slurm/data/myrun1.sh

cat  /export/home/snumber/slurm/data/myrun1.sh
>>>>>>>>>
cd /export/home/snumber/slurm/data
python /export/home/snumber/slurm/data/isCuda.py
>>>>>>>>>>

make it executable:
chmod +x  /export/home/snumber/slurm/data/myrun1.sh

Here is the content of the slurm.01 file

>>>>cat slurm.02 >>>>>>>>>>>>> 
#!/bin/bash 

####SBATCH --account=def-yoursnumber 

#SBATCH --job-name=helloWord 

#SBATCH --cpus-per-task=1 

#SBATCH --mem-per-cpu=1500MB 

#SBATCH --gres=gpu:a100:1 

###SBATCH --qos=work 

#SBATCH --qos=work 

###SBATCH --mem=4000M               # memory per node 

#SBATCH --time=0-03:00

module load singularity/4.1.3

##./program                         # you can use 'nvidia-smi' for a test

singularity exec -e -B /scratch:/scratch -B /export/home/snumber:/export/home/snumber /export/home/snumber/sw/Containers/pytorch "data/myrun1.sh"

>>>>>>>>>>>

To submit the slurm job:
sbatch  slurm.02


Alternatively, to run interactively:
srun --export=PATH,TERM,HOME,LANG  --job-name=hello_word --cpus-per-task=1 --mem-per-cpu=1500MB --time=1:00:00 --qos=work --gres=gpu:a100:1 --pty /bin/bash -l


singularity shell -e -B /scratch/snumber:/scratch/snumber -B /export/home/snumber:/export/home/snumber /export/home/snumber/sw/Containers/pytorch
cd data
sh myrun1.sh 






>>>Some errors you can ignore while building the sandbox>>>
INFO:    Starting build...
INFO:    Fetching OCI image...
INFO:    Extracting OCI image...
2024/05/20 12:37:31  warn rootless{usr/local/nvm/versions/node/v16.20.2/bin/corepack} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2024/05/20 12:37:32  warn rootless{usr/local/nvm/versions/node/v16.20.2/bin/npm} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2024/05/20 12:37:32  warn rootless{usr/local/nvm/versions/node/v16.20.2/bin/npx} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"

2024/05/20 12:38:17  warn rootless{usr/lib/libarrow.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2024/05/20 12:38:17  warn rootless{usr/lib/libarrow.so.1400} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2024/05/20 12:38:17  warn rootless{usr/lib/libarrow_acero.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2024/05/20 12:38:17  warn rootless{usr/lib/libarrow_acero.so.1400} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2024/05/20 12:38:17  warn rootless{usr/lib/libarrow_dataset.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2024/05/20 12:38:17  warn rootless{usr/lib/libarrow_dataset.so.1400} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2024/05/20 12:38:34  warn rootless{usr/lib/libparquet.so} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2024/05/20 12:38:34  warn rootless{usr/lib/libparquet.so.1400} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
INFO:    Inserting Singularity configuration...
INFO:    Creating sandbox directory...
INFO:    Build complete: pytorch
>>>>

The other method is to create a conda environment

No Format

#Go into an interactive run first
srun --export=PATH,TERM,HOME,LANG  --job-name=hello_word --cpus-per-task=1 --mem-per-cpu=60GB --time=19:00:00 --qos=work --gres=gpu:a100_1g.10gb:1  --pty /bin/bash -l
source /usr/local/bin/s3proxy.sh
module load  cuda/12.1
module  load anaconda3/2024.02
conda create --name pytorchCuda121 pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
source activate pytorchCuda121

Qs91: Do you recommend installing miniconda : pytorch usage example

No Format

We do not recommend installing miniconda as we provide conda as a module.
module load anaconda3/2024.02 

If you have installed it, comment out the miniconda section from ~./bashrc which can mess with other non-miniconda env. 
After commenting it out re-log back in.

Here is how you can test pytorch on your home dir for cuda availability:

To test this, please do this;
cd ~/slurm 
There are a couple of sample scripts to run slurm jobs

We will run an interactive job to troubleshoot this problem:
For example:
srun --export=PATH,TERM,HOME,LANG  --job-name=hello_word --cpus-per-task=1 --mem-per-cpu=50GB --time=1:00:00 --qos=work --gres=gpu:a100:1 --pty /bin/bash -l

it would put you inside a job.

squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            <snip>
              1718    LocalQ    myrun s5284664  R    4:17:59      1 dgxlogin


Once inside a job, check if it detects gpu in pytorch app. (if already installed)

module load anaconda3/2024.02 
conda info --envs
# conda environments:
#
base                  *  /export/home/s5305964/miniconda3
PHDenv                   /export/home/s5305964/miniconda3/envs/PHDenv

Unfortunately, you have installed it in miniconda. So leaving aside that, I would build a new env

To install new env and packages, do this:

source /usr/local/bin/s3proxy.sh  (need to get access the internet within a slurm job)
conda create --name pytorchCuda121 pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

(See Qs 90 : https://griffith.atlassian.net/wiki/spaces/GHCD/pages/4030751/FAQ+-+Griffith+HPC+Cluster#FAQ-GriffithHPCCluster-Qs90%3AHowtorunthepytorch(containermethodandcondaenvmethod)ontheICTcluster)

source activate pytorchCuda121
 python ~/isCuda.py 
CUDA AVILABLE
>>>>>>>>>isCuda.py>>>>>>>>>>>>>
import torch
if torch.cuda.is_available():
    print('CUDA AVILABLE')
else:
    print('NO CUDA')

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

PS: Comment out the miniconda install from ~./bashrc which can mess with this env.

If you ever need to use the miniconda env, you can do this:

source deactivate
module purge

__conda_setup="$('~/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
eval "$__conda_setup"
source activate PHDenv
python isCuda.py 
CUDA AVILABLE

Qs92: dgxA100 gpu node : group reservation - dedicated gpus in slurm

No Format

A group named "alan" has 2 A100 gpus reserved.

scontrol show res
ReservationName=alan100 StartTime=2024-07-01T10:33:01 EndTime=2024-08-02T10:33:01 Duration=32-00:00:00
   Nodes=dgxlogin NodeCnt=1 CoreCnt=32 Features=(null) PartitionName=LocalQ Flags=FLEX,MAGNETIC
     NodeName=dgxlogin CoreIDs=0-10,13,40-43,64-79
   TRES=cpu=64,gres/gpu:a100=2,gres/gpu=2
   Users=(null) Groups=alan Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a
   MaxStartDelay=(null)


This is reservation is automated (using cron) so that it is deleted every 30 days and recreated. Which means , these 2 gpus are permanently reserved for alan group. 

Usage is as follows:
For batch runs
==============
Add this inside the slurm script:
#SBATCH --reservation=alan100 
OR
sbatch  --reservation=alan100 <job script>
e.g:   sbatch  --reservation=alan1g10gb slurm.res
Make sure the job script has resource request covered by the reservation (This command will list that: scontrol show res)

For interactive runs
=================
srun --reservation=alan100 --export=PATH,TERM,HOME,LANG  --job-name=myTestRun --cpus-per-task=1 --mem-per-cpu=15GB --time=00:02:00 --qos=work --gres=gpu:a100:1 --pty /bin/bash -l

Qs93: Apptainer/Singularity Container Usage Example on the gowonda HPC cluster

No Format

This will be illustrated using an actual question that came to our support team:

Support Request: 
================
I’ve been trying for some time to get gene regulatory analysis to run on a large dataset on Gowonda. 
Originally I tried doing this using Genie3, which begins running but errors out after about 5 hours 
due to exceeding memory, even if I run it on bigmem requesting 900GB.

There’s a new package – arboreto – that has a better version of Genie3 but also a much more 
computationally efficient methodology called grnboost2. I’ve made a new conda environment 
and have installed arboreto, however can’t get grnboost2 to run. The files load fine, but it 
stalls at the grnboost2 step with the error 
‘TypeError: descriptor '__call__' for 'type' objects doesn't apply to a 'property' object’. 
Help forums indicate that that this may be due to an issue with the Python and Dask versions.
 I’ve tried making different Conda environments using different version combinations, but still 
can’t get it to work.

Our Answer:
===========
We looked at this page: https://github.com/aertslab/pySCENIC/issues/163
and we think the 1st solution using singularity containers is good. 
Containers are better suited when issues with version mismatch occurs 
(e.g specific versions of python, dask etc).
https://pyscenic.readthedocs.io/en/latest/installation.html#docker-and-singularity-images
We went to this link as directed from above link.

This is the copy and paste of the section using Apptainer (used to be called Singularity)
>>>>>> Clip from the link >>>>>>>>>>>>
Singularity/Apptainer
Singularity/Apptainer images can be build from the Docker Hub image as source:
# pySCENIC CLI version.
singularity build aertslab-pyscenic-0.12.1.sif docker://aertslab/pyscenic:0.12.1
apptainer build aertslab-pyscenic-0.12.1.sif docker://aertslab/pyscenic:0.12.1

# pySCENIC CLI version + ipython kernel + scanpy.
singularity build aertslab-pyscenic-scanpy-0.12.1-1.9.1.sif docker://aertslab/pyscenic_scanpy:0.12.1_1.9.1
apptainer build aertslab-pyscenic-0.12.1-1.9.1.sif docker://aertslab/pyscenic_scanpy:0.12.1_1.9.1
>>>>>>>>>End of the clip from the link >>>>>>>>>>

To download the container from any docker link at Griffith, we have to alter the commands slightly 
(also documented on the singularity page of our wiki) 
https://griffith.atlassian.net/wiki/spaces/GHCD/pages/4030761/Singularity

(The change is to replace docker:// with docker://public.docker.itc.griffith.edu.au/)
So the altered command becomes:
singularity build aertslab-pyscenic-0.12.1.sif docker://public.docker.itc.griffith.edu.au/aertslab/pyscenic:0.12.1
singularity build aertslab-pyscenic-scanpy-0.12.1-1.9.1.sif docker://public.docker.itc.griffith.edu.au/aertslab/pyscenic_scanpy:0.12.1_1.9.1

After sourcing the proxy file (source /usr/local/bin/s3proxy.sh), you can run the above two commands to download those containers.
We have done this for you and placed the containers in this directory: ~/Containers 
If this directory does not exist, create it with mkdir ~/Containers

To check the functionality of the container, please do this:
singularity shell -B /scratch/snumber:/scratch --pwd /scratch/snumber:/scratch --pwd /scratch  /export/home/snumber/Containers/aertslab-pyscenic-0.12.1.sif 
>>>>>>>>>
From inside the container shell, you can run a few commands to check for functionality
e.g:
Apptainer>cd scripts
Apptainer> python grnboost3.py 
INFO:root:Initializing Dask client...
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
INFO:root:Dask client initialized: <Client: 'tcp://127.0.0.1:41411' processes=9 threads=72, memory=251.23 GiB>
INFO:root:Loading and transposing expression data...
ERROR:root:An error occurred: [Errno 2] No such file or directory: 'FinalCounts.csv'
<snip>
exit

>>>>>>>>>>>>>
This looks ok (you need to supply the missing files FinalCounts.csv)

If you are happy with this, go to the next step:
(Please note: We have put all the sample pbs and run scripts in your ~scratch/scripts folder)


Next part is to create a pbs script and a run script 

mkdir /scratch/snumber/scripts
mkdir /scratch/snumber/scripts/data
cd  /scratch/snumber/scripts
You can copy all the scripts into this folder as well as create a run script
e.g: cp grnboost3.py ~/scratch/scripts/grnboost3.py

Here is a sample pbs script to use 6 cpus
>>>>>>>>>>>>>>>
#!/bin/bash
#PBS -m e
#PBS -M YourEmail@griffith.edu.au
#PBS -N GRN_pearl
#PBS -q workq
#PBS -l select=1:ncpus=6:mem=66gb,walltime=30:00:00
cd  $PBS_O_WORKDIR
singularity exec -B /scratch/:/scratch --pwd /scratch/snumber:/scratch --pwd /scratch  /export/home/snumber/Containers/aertslab-pyscenic-0.12.1.sif "/scratch/scripts/run1.sh"
#singularity exec -B /scratch/snumber:/scratch --pwd /scratch/snumber:/scratch --pwd /scratch  /export/home/snumber/Containers/aertslab-pyscenic-scanpy-0.12.1-1.9.1.sif "/scratch/scripts/run2.sh"
exit
sleep 2

>>>>>>>>>>>>>>>>


Next create a run script inside : /scratch/snumber/scripts

cat run1.sh
>>>>>>>>>>>>
cd /scratch/scripts
python /scratch/scripts/grnboost3.py

>>>>>>>>>>>>
make it executable:
chmod +x d_run1.sh

Now simply submit the pbs job:
cd  /scratch/snumber/scripts
qsub pbs.01

After the run has completed, check through the output and error files and fix any errors. Voilà!

Versions Compared

Old Version 253

New Version Current

Key

*Qs 1: How do I cite or mention the cluster in papers? or what is the preferred method?

Qs28: How do I check number of CPUs my job is using

Qs29: How to customize an environmental variable using modules

Qs 31: Multi Cores are requested and allocated by PBs but job runs only on 1 core. Why is that?

Qs 32: How to check the remaining licenses on the license server

Qs.56: NCMAS process and application

Qs 57: What kind of storage and compute is available on Griffith HPC

Qs69: How to install bioinformatics software in your home directory

Set up X11 forwarding

Qs80: Tunnel setup

Qs88: Install perl modules without root access.

Qs89: n061 gpu node : pytorch usage

Qs90: How to run the pytorch (container method and conda env method) on the ICT cluster

Qs91: Do you recommend installing miniconda : pytorch usage example

Qs92: dgxA100 gpu node : group reservation - dedicated gpus in slurm

Qs93: Apptainer/Singularity Container Usage Example on the gowonda HPC cluster

Page Comparison

Versions Compared

Old Version 253

New Version Current

Key

*Qs 1: How do I cite or mention the cluster in papers? or what is the preferred method?

Qs28: How do I check number of CPUs my job is using

Qs29: How to customize an environmental variable using modules

Qs 31: Multi Cores are requested and allocated by PBs but job runs only on 1 core. Why is that?

Qs 32: How to check the remaining licenses on the license server

Qs.56: NCMAS process and application

Qs 57: What kind of storage and compute is available on Griffith HPC

Qs69: How to install bioinformatics software in your home directory

Set up X11 forwarding

Qs80: Tunnel setup

Qs88: Install perl modules without root access.

Qs89: n061 gpu node : pytorch usage

Qs90: How to run the pytorch (container method and conda env method) on the ICT cluster

Qs91: Do you recommend installing miniconda : pytorch usage example

Qs92: dgxA100 gpu node : group reservation - dedicated gpus in slurm

Qs93: Apptainer/Singularity Container Usage Example on the gowonda HPC cluster