| Table of Contents |
|---|
Introduction
OpenFOAM is a framework for developing application executables that use packaged functionality contained within a collection of approximately 100 C+ libraries. OpenFOAM is shipped with approximately 250 pre-built applications that fall into two categories: solvers, that are each designed to solve a specific problem in fluid (or continuum) mechanics; and utilities, that are designed to perform tasks that involve data manipulation.
Source: https://cfd.direct/openfoam/user-guide/
...
| No Format |
|---|
Singularity is installed on the nodes but if needed latest version of singularity can be loaded with
#module load singularity
The container with openfoam6-paraview54 can be located in: /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
We use singularity shell to start a container, and run a shell in the container.
The -B option is used to “bind” the /scratch/$USER directory to a directory named /scratch in the container.
We also the --pwd option to specify the working directory in the running container (in this case /scratch). This is always recommended.
singularity shell -B /scratch/s123456:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
Using it with PBS scheduler, we need to set up two bash scripts: the launch script that gets executed inside singularity and the PBS scheduler script that instructs the scheduler to start singularity.
==============
Prep the Image
===============
mkdir ~/Container
cp -r /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif-centos ~/Container
======================================
Launcher Script: singularity_launch.01
==========================
source /opt/openfoam6/etc/bashrc
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
#Make sure that it is executable with chmod +x singularity_launch.01
===================
pbs script: pbs.01
==================
#!/bin/bash -l
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -V
#PBS -N TestOPENFOAM
#PBS -q workq
#PBS -l select=1:ncpus=1:ngpus=1:mem=32gb,walltime=30:00:00
cd $PBS_O_WORKDIR
##run singularity
#singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
singularity shell -B /scratch/s12345:/scratch --pwd /scratch/scratch/s12345:/scratch --pwd /scratch --nv /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif /scratch/s12345/singularity_launch.01
#singularity shell -B /share/data:/mnt --nv ubuntu.simg /mnt/[PATH TO ExoGAN]/singularity_launch_gan.sh
cat /etc/os-release
exit
sleep 2
|
Another example of usage on the new cluster
| No Format |
|---|
Initial Prep:
=============
Copy the images into the local container folder:
mkdir ~/Container
cp /sw/Containers/singularity/images/openfoam7-paraview56_latest.sif ~/Container/
Shell into the container and create launcher script:
====================================================
Initial setup
1. shell into the container
=============================
For example:
singularity shell -B /scratch/s12345:/scratch --pwd /scratch/s12345:/scratch --pwd /scratch /export/home/s12345/Container/openfoam7-paraview56_latest.sif
e.g:
singularity shell -B /scratch/s5126720:/scratch --pwd /scratch/s5126720:/scratch --pwd /scratch /export/home/s5126720/Container/openfoam7-paraview56_latest.sif
2. Create a variable file you can source
Once in the shell, create a variable file with contents like this:
vi /scratch/openFOAMenv.sh
export LD_LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LIBRARY_PATH
export PATH=/opt/openfoam7/bin:/opt/paraviewopenfoam56/bin:$PATH
export LDFLAGS=-L/opt/paraviewopenfoam56/lib:$LDFLAGS
export PKG_CONFIG_PATH=/sw/library/loki/0.1.7/lib/pkgconfig:$PKG_CONFIG_PATH
export INCLUDE=/opt/paraviewopenfoam56/include:$INCLUDE
export CPLUS_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$CPLUS_INCLUDE_PATH
export CPATH=/opt/paraviewopenfoam56/include:$CPATH
export C_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$C_INCLUDE_PATH
export FPATH=/opt/paraviewopenfoam56/include:$FPATH
export CFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export FCFLAGS=-I/opt/paraviewopenfoam56/include:$FCFLAGS
export FFLAGS=-I/opt/paraviewopenfoam56/include:$FFLAGS
3. Manual Test Run
===================
source /scratch/openFOAMenv.sh
foamInfo -help
For example:
Singularity openfoam7-paraview56_latest.sif:/opt/openfoam7> foamInfo -help
Usage: foamInfo [OPTIONS] <name>
options:
-all | -a list all tutorials that use <name> (otherwise maximum 10)
4. Create a Launcher Script;
===========================
Once you are happy that all step 1 ,2 and 3 works, you can create a launcherScript
While still inside the shell:
vi /scratch/openfoamLauncher.sh
An example of the content is this:
export LD_LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LIBRARY_PATH
export PATH=/opt/openfoam7/bin:/opt/paraviewopenfoam56/bin:$PATH
export LDFLAGS=-L/opt/paraviewopenfoam56/lib:$LDFLAGS
export PKG_CONFIG_PATH=/sw/library/loki/0.1.7/lib/pkgconfig:$PKG_CONFIG_PATH
export INCLUDE=/opt/paraviewopenfoam56/include:$INCLUDE
export CPLUS_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$CPLUS_INCLUDE_PATH
export CPATH=/opt/paraviewopenfoam56/include:$CPATH
export C_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$C_INCLUDE_PATH
export FPATH=/opt/paraviewopenfoam56/include:$FPATH
export CFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export FCFLAGS=-I/opt/paraviewopenfoam56/include:$FCFLAGS
export FFLAGS=-I/opt/paraviewopenfoam56/include:$FFLAGS
cd /scratch
#Put your run instructions/command below.
foamInfo -help
>>>>>>>>>>>>>
make it executable:
chmod +x /scratch/openfoamLauncher.sh
Run it manually and check if it gives the expected reults:
Check if it gives the expected results:
/scratch/openfoamLauncher.sh
(OR: try cd /scratch;./openfoamLauncher.sh)
5. Running it as a batch job:
==============================
Exit the shell and create this pbs script anywhere in space:
vi pbs.01
>>>>>>>>>
#!/bin/bash
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -N TestOpenFoam
#PBS -q workq
#PBS -l select=1:ncpus=1:mem=2gb,walltime=3:00:00
cd $PBS_O_WORKDIR
#An example is as below. Modify accordingly to suit your needs
singularity exec -B /scratch/s5126720:/scratch --pwd /scratch/s5126720:/scratch --pwd /scratch /export/home/s5126720/Container/openfoam7-paraview56_latest.sif "/scratch/openfoamLauncher.sh"
exit
sleep 2
>>>>>>
Submit the job and check the results.
qsub pbs.01
|