| Table of Contents |
|---|
Introduction
...
Singularity usage on Griffith HPC
To create containers, we use public.docker.itc.griffith.edu.au
Simple "singularity pull" with s3proxy stopped working for docker hubs around Oct 2020. This is due to the changes (limits) docker introduced. To avoid hitting the limits, Griffith team implemented steps to stop direct access through the s3proxy http proxy to the docker hub registry (see https://www.docker.com/blog/understanding-inner-loop-development-and-pull-rates/ for notes)
| No Format |
|---|
To build an image singularity pull docker://public.docker.itc.griffith.edu.au/biocontainers/blast:2.2.31 To run a container singularity run docker://public.docker.itc.griffith.edu.au/godlovedc/lolcow # Download the container into a permanent image file http_proxy=http://s3proxy.itc.griffith.edu.au:3128 singularity pull my_blast.sif docker://biocontainers/blast:2.2.31 singularity pull my_blast.sif docker://public.docker.itc.griffith.edu.au/biocontainers/blast:2.2.31 Another example: singularity pull lolcow_latest.sif docker://public.docker.itc.griffith.edu.au/godlovedc/lolcow # Now run it from the image file singularity exec my_blast.sif blastp -version blastp: 2.2.31+ Package: blast 2.2.31, build Apr 23 2016 15:49:47 Package: blast 2.2.31, build Apr 23 2016 15:49:47 Using GPU(s) with a Container This is useful on the n060 node with gpu cards A docker or singularity container built to use an NVIDIA GPU should be run with the--nv option singularity pull docker://public.docker.itc.griffith.edu.au/tensorflow/tensorflow:latest-gpu-jupyter singularity run --nv tensorflow_latest-gpu-jupyter.sif (-B /run is needed for some images (using e.g. jupyter, that give an error otherwise. For that, use this syntax: singularity run --nv –B /run tensorflow_latest-gpu-jupyter.sif) Most docker containers run successfully under Singularity Some need to be run with special options Few cannot be used Docker is optimized for running services in isolated containers Need to add back other cluster filesystems with -B option... singularity exec --contain -B /project -B /archive singularity exec -B /scratch/s1234:/scratch --pwd /scratch/scratch/s1234:/scratch --pwd /scratch /export/home/s1234/Container/openfoam6-paraview54_latest.sif-centos env|more singularity exec -B /scratch/s1234:/scratch --pwd /scratch/scratch/s1234:/scratch --pwd /scratch /export/home/s1234/Container/openfoam6-paraview54_latest.sif-centos cat /scratch/singularity_launch.03 look at the runscript, and envrionment ================================= singularity inspect -e /export/home/s1234/Container/openfoam6-paraview54_latest.sif-centos singularity inspect -r /export/home/s1234/Container/openfoam6-paraview54_latest.sif-centos singularity exec -B /scratch/s1234:/scratch --pwd /scratch/scratch/s1234:/scratch --pwd /scratch /export/home/1234/Container/openfoam6-paraview54_latest.sif-centos /.singularity.d/runscript |
...