| Table of Contents |
|---|
...
To create containers, we use public.docker.itc.griffith.edu.au
Simple "singularity pull" with s3proxy stopped working for docker hubs around Oct 2020. This is due to the changes (limits) docker introduced. To avoid hitting the limits, Griffith team implemented steps to stop direct access through the s3proxy http proxy to the docker hub registry (see https://www.docker.com/blog/understanding-inner-loop-development-and-pull-rates/ for notes)
...
| Code Block |
|---|
cat pbs.01 >>>> #!/bin/bash #PBS -m e #PBS -M email@griffith.edu.au #PBS -N CEINMS256Calib #PBS -q medium #PBS -l select=1:ncpus=4:mem=16gb,walltime=30:00:00 cd $PBS_O_WORKDIR singularity exec -B /scratch/s123456:/scratch --pwd /scratch/s123456:/scratch --pwd /scratch /export/home/s123456/Container/ceinms_latest.sif "/scratch/Scripts/ceinmsLauncher.sh" exit sleep 2 >>>>>> cat /scratch/s2984644/Scripts/ceinmsLauncher.sh >>>>>> export LD_LIBRARY_PATH=/usr/local/lib:/opensim/opensim_install/lib export PATH=/CEINMS/install/bin:$PATH CEINMScalibrate -S '/scratch/data/Dev02/Barefoot/ceinms/calibratedSubjects/setupCalibration_t00000000.xml' >>>>>> Before you submit the job, make sure it is executable. You do that with this: chmod 700 /scratch/s2984644/Scripts/ceinmsLauncher.sh >>>>>> qsub pbs.01 >>>>> Results: +-+-+-+-+-+-+ |C|E|I|N|M|S| +-+-+-+-+-+-+-+-+-+-+ |C|a|l|i|b|r|a|t|e|d| +-+-+-+-+-+-+-+-+-+-+-+-+ |E|M|G|-|I|n|f|o|r|m|e|d| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |N|e|u|r|o|m|u|s|c|u|l|o|s|k|e|l|e|t|a|l| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |T|o|o|l|b|o|x| +-+-+-+-+-+-+-+ CEINMScalibrate version 0.30.1 Copyright (C) Nov 3 2022 Claudio Pizzolato, Monica Reggiani, Massimo Sartori, David Lloyd Software developers: Claudio Pizzolato, Monica Reggiani readNMSmodelCfg filesystem error: cannot make canonical path: No such file or directory [/scratch/data/Dev02/Barefoot/ceinms/calibratedSubjects/subjectCalibrated_t00000000.xml]Reading subject file: /scratch/data/Dev02/Barefoot/ceinms/calibratedSubjects/uncalibrated.xml . Contact model found Calibration configuration - Algorithm -- Simulated Annealing --- noEpsilon 4 --- NS 15 --- NT 5 --- maxNoEval 200000 --- rt 0.3 --- T 20 --- epsilon 1e-05 - NMSmodel - CalibrationSteps -- Step Objective Function: TorqueErrorNormalised - Targets Type: Torque - Targets: hip_flexion_r knee_angle_r ankle_angle_r - Weight: 1 - Exponent: 1 - Trials -- /scratch/data/Dev02/Barefoot/ceinms/calibratedSubjects/running10.xml -- /scratch/data/Dev02/Barefoot/ceinms/calibratedSubjects/running11.xml -- /scratch/data/Dev02/Barefoot/ceinms/calibratedSubjects/running7.xml Reading subject file: /scratch/data/Dev02/Barefoot/ceinms/calibratedSubjects/uncalibrated.xml . activeForceLength passiveForceLength forceVelocity tendonForceStrain Assuming addbrev_r pennation angle in radians: 0.114781 Assuming addlong_r pennation angle in radians: 0.13777 Assuming addmagDist_r pennation angle in radians: 0.194705 Assuming addmagIsch_r pennation angle in radians: 0.168044 Assuming addmagMid_r pennation angle in radians: 0.207308 Assuming addmagProx_r pennation angle in radians: 0.311483 <snip> EMG: Reading emg file.../scratch/data/Dev02/Barefoot/ceinms/calibratedSubjects/../../dynamicElaborations/running10/emg.mot Muscle excitations to muscle mapping: addbrev_r -> addbrev_r addlong_r -> addlong_r addmagDist_r -> addmagDist_r addmagIsch_r -> addmagIsch_r addmagMid_r -> addmagMid_r addmagProx_r -> addmagProx_r bflh_r -> bflh_r bfsh_r -> bfsh_r edl_r -> edl_r ehl_r -> ehl_r fdl_r -> fdl_r <snip> >>>> |
Converting a docker image into singularity image
Method 1: Using an Existing Docker Image on Your Local Machine
| No Format |
|---|
1. Find the Docker image ID.
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest bf756fb1ae65 5 months ago 13.3kB
godlovedc/lolcow latest 577c1fe8e6d8 2 years ago 241MB
2. Create a tarball of the Docker image.
For the Docker image you want to port to Griffith HPC,
for example, godlovedc/lolcow with an image ID of 577c1fe8e6d8,
create a tarball using the docker save command:
docker save 577c1fe8e6d8 -o lolcow.tar
3. Copy the tarball to Griffith HPC
Use scp or winscp or cyberduck etc
scp lolcow.tar 10.250.250.3:/tmp
4. Convert the tarball to a Singularity image.
module load singularity
singularity build --sandbox lolcow docker-archive://lolcow.tar
If the tarball is not in the current working directory, specify the path, for example, /tmp:
singularity build --sandbox lolcow docker-archive:///tmp/lolcow.tar
In this example, lolcow is the directory name of the Singularity image.
5. Run the Singularity sandbox as usual for testing. Once testing is over, use it in a pbs script (see example above).
For example:
singularity shell lolcow
singularity exec lolcow cowsay hello
singularity run lolcow |
Method 2: Using an Existing Docker Image on Docker Hub
See section earlier on how to do this.
Method 3: Using a Working Dockerfile Without a Docker Image
| No Format |
|---|
Reference
- http://singularity.lbl.gov/
- https://www.singularity-hub.org/
- https://singularityhub.github.io/containers/registry/singularity-hub-registry/
- https://singularity.lbl.gov/user-guide
- https://hpc.research.uts.edu.au/software_general/singularity/
- https://cran.r-project.org/web/views/HighPerformanceComputing.html
- https://support.pawsey.org.au/documentation/display/US/Running+RStudio+on+Zeus+with+Singularity
- https://www.katacoda.com/courses/docker
- https://pawseysc.github.io/containers-bioinformatics-workshop/5.build/index.html
- https://quay.io/
- https://pawseysc.github.io/containers-bioinformatics-workshop/3.pipeline/index.html
- https://ist.mit.edu/xwin32
- https://pawseysc.github.io/containers-bioinformatics-workshop/1.prep1-ssh/index.html
...