Initial Prep:
=============

Copy the images into the local container folder:

mkdir ~/Container
cp /sw/Containers/singularity/images/delft3d4_latest.sif  ~/Container
cp /sw/Containers/singularity/images/delft3dfm_latest.sif  ~/Container


Shell into the container and create launcher script:
====================================================

Initial setup

1. shell into the container
=============================
For example:

singularity  shell -B /scratch/s12345:/scratch --pwd /scratch/s12345:/scratch --pwd /scratch  /export/home/s12345/Container/delft3d4_latest.sif

For example:
singularity  shell -B /scratch/s2961948:/scratch --pwd /scratch/s2961948:/scratch --pwd /scratch  /export/home/s2961948/Container/delft3d4_latest.sif

2. Create a variable file you can source

Once in the shell, create a variable file with contents like this:

vi /scratch/delft3denv.sh
export LD_LIBRARY_PATH=/opt/delft3d_latest/lnx64/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/delft3d_latest/lnx64/lib:$LIBRARY_PATH
export PATH=/opt/delft3d_latest/lnx64/bin:$PATH
module load mpich-x86_64

3. Manual Test Run
===================
source /scratch/delft3denv.sh
d_hydro

For example:

Singularity delft3d4_latest.sif:/scratch> d_hydro
d_hydro ABORT: Improper usage.  Execute "d_hydro -?" for command-line syntax


4. Create a Launcher Script;
===========================
Once you are happy that all step 1 ,2 and 3 works, you can create a launcherScript
While still inside the shell:

vi /scratch/d_hydroLauncher.sh

An example of the content is this:
>>>>>>>>>>>>
export LD_LIBRARY_PATH=/opt/delft3d_latest/lnx64/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/delft3d_latest/lnx64/lib:$LIBRARY_PATH
export PATH=/opt/delft3d_latest/lnx64/bin:$PATH
module load mpich-x86_64
cd /scratch
d_hydro 
>>>>>>>>>>>>>
make it executable:
chmod +x d_hydroLauncher.sh
Check if it gives the expected results:

./scratch/d_hydroLauncher.sh


5. Running it as a batch job:
==============================
Exit the shell and create this pbs script anywhere in your home directory:

A: Final d_hydroLauncher.sh
===========================
more ~/scratch/d_hydroLauncher.sh
/opt/delft3d_latest/lnx64/bin/d_hydro

B: Final PBS script
====================
more pbs.01
>>>>>>>>>
#!/bin/bash 
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -N DelftJob  
#PBS -q gworkq  
#PBS -l select=1:ncpus=1:mem=12gb,walltime=00:00:30
cd  $PBS_O_WORKDIR
DELFT_PATH="/opt/delft3d_latest/lnx64/"
##export SINGULARITY_BINDPATH="$DELFT_PATH/bin" # Append this location to the container PATH
##export SINGULARITY_BINDPATH=/scratch,/fast/tmp,/gpfs01
export SINGULARITYENV_LD_LIBRARY_PATH=$DELFT_PATH/lib:$LD_LIBRARY_PATH
export SINGULARITYENV_LIBRARY_PATH=$DELFT_PATH/lib:$LIBRARY_PATH
export SINGULARITYENV_PATH=$DELFT_PATH/bin:$PATH
singularity exec -B /scratch/s12345:/scratch  /export/home/s12345/Container/delft3d4_latest.sif "/scratch/d_hydroLauncher.sh"
##d_hydro
exit
sleep 2
>>>>>>

Submit the job and check the results.

qsub pbs.01

Second method without using the launcher script:
=================================================
#!/bin/bash 
#PBS -m abe
#PBS -V
#PBS -M YourEmail@griffith.edu.au
#PBS -N DelftJob  
#PBS -q gworkq  
#PBS -l select=1:ncpus=1:mem=12gb,walltime=00:00:30
DELFT_PATH="/opt/delft3d_latest/lnx64/"
export SINGULARITYENV_LD_LIBRARY_PATH=$DELFT_PATH/lib:
export SINGULARITYENV_LIBRARY_PATH=$DELFT_PATH/lib
export SINGULARITYENV_PATH=$DELFT_PATH/bin
cd $HOME/scratch
singularity exec /export/home/s12345/Container/delft3d4_latest.sif d_hydro swaninit
exit
sleep 2

1. Find the Docker image ID.
docker images
REPOSITORY        TAG       IMAGE ID        CREATED        SIZE
hello-world       latest    bf756fb1ae65    5 months ago   13.3kB
godlovedc/lolcow  latest    577c1fe8e6d8    2 years ago    241MB

2. Create a tarball of the Docker image. 
For the Docker image you want to port to Griffith HPC, 
for example, godlovedc/lolcow with an image ID of 577c1fe8e6d8, 
create a tarball using the docker save command:
docker save 577c1fe8e6d8 -o lolcow.tar 

3. Copy the tarball to Griffith HPC
Use scp or winscp or cyberduck etc 
scp lolcow.tar 10.250.250.3:/tmp

4. Convert the tarball to a Singularity image.
module load singularity
singularity build --sandbox lolcow docker-archive://lolcow.tar

If the tarball is not in the current working directory, specify the path, for example, /tmp:
singularity build --sandbox lolcow docker-archive:///tmp/lolcow.tar
singularity build --sandbox pytorch docker-archive:////sw/Containers/docker/pytorchnew.tar

In this example, lolcow is the directory name of the Singularity image.

5. Run the Singularity sandbox as usual for testing. Once testing is over, use it in a pbs script (see example above). 
For example: 
singularity shell lolcow
singularity exec lolcow cowsay hello
singularity run lolcow