Introduction
OpenFOAM is a framework for developing application executables that use packaged functionality contained within a collection of approximately 100 C+ libraries. OpenFOAM is shipped with approximately 250 pre-built applications that fall into two categories: solvers, that are each designed to solve a specific problem in fluid (or continuum) mechanics; and utilities, that are designed to perform tasks that involve data manipulation.
Source: https://cfd.direct/openfoam/user-guide/
Usage using singularity Container : OpenFOAM 4.1
Creating the singularity image
| No Format |
|---|
module load OpenFoam/4.0.0
echo "alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'" >> $HOME/.bashrc
>>>>>>>>>>>
Add this into your $home/.bashrc
alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'
>>>>>>>>>>>
End User Update: Nov 5th 2018:
=================================
Add this into your $home/.bashrc
alias of40='module load OpenFoam/4.0.0; source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc /sw/OpenFoam/OpenFOAM-4.0/etc/pref.sh WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes';
>>>>>>>>>>>>>>>>>>>>>>>>>
source /sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes
Reminder: Whenever you start a new terminal window or tab, you should run the alias command associated to the OpenFOAM 4.0 shell environment. In other words, run the following command whenever you start a new terminal:
of40 |
Installation
##source: https://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-4.0/CentOS_SL_RHEL
| No Format |
|---|
module load library/libxml2/2.5
module load misc/extralibs/extralibs
module load library/bzip2/1.0.6fpic
module load NetCDF/4.4.1.1-gnu
module load misc/openssl/1.0.2
module load jdk/1.8.0_66
module load library/xz/5.3.3
module load library/gdal/2.3.0
module load misc/curl/7.54.0
module load library/gmp/6.0.0
module load pcre/8.39
module load mpi/openMPI/1.8.5-gcc4.9.0
module load library/mpfr/3.1.2
module load gcc/6.3.0
module load library/geos/3.6.1
module load library/hdf5/1.8.18-parallel
module load library/bzip2/1.0.6
module load library/hdf5/1.8.18
module load library/tiff/4.0.7
module load openmpi-x86_64
module load ATLAS/3.9.39
module load misc/rtmpdump/rtmp
module load library/libgeotiff/1.4.1
module load library/expat/2.2.0
module load library/mpc/1.0.2
module load jags/4.2.0
module load R/3.4.0
module load library/jasper/1.900.1
module load library/zlib/1.2.11
module load library/zlib/1.2.5
cd source /usr/local/bin/s3proxy.sh
mkdir -p $HOME/sw/Containers/
cd $HOME/sw/Containers/
singularity pull docker://public.docker.itc.griffith.edu.au/openfoam/openfoam4-paraview50
#You may skip above steps by copying the images that have been stored in /sw/Containers/singularity/images/openfoam4-paraview50_latest.sif
#mkdir -p $HOME/sw/Containers/;cd $HOME/sw/Containers/;cp /sw/Containers/singularity/images/openfoam4-paraview50_latest.sif . |
Initial setup and test
| No Format |
|---|
mkdir -p $HOME/OpenFOAM/${USER}-4.1
cd $HOME/OpenFOAM/${USER}-4.1
singularity shell -B /scratch/${USER}:/scratch --pwd /scratch/${USER}:/scratch --pwd $HOME/OpenFOAM/${USER}-4.1 $HOME/sw/Containers/openfoam4-paraview50_latest.sif
Once inside the shell:
source /opt/openfoam4/etc/bashrc
mkdir -p $FOAM_RUN
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
paraFoam
|
Launcher script to run a test job
| No Format |
|---|
cat $HOME/OpenFOAM/${USER}-4.1/openfoamLauncher4.sh
>>>>>>>>>>>>>>>>
#!/bin/bash
source /opt/openfoam4/etc/bashrc
mkdir -p $FOAM_RUN
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
paraFoam |
cd $HOME/OpenFOAM/${USER}-4.1/chmod +x openfoamLauncher4.sh
pbs script - A sample to run the test job in the above example using pbs.
cat $HOME/OpenFOAM/${USER}-4.1/pbs.01
| No Format |
|---|
#!/bin/bash
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -N TestOpenFoam4Job1
#PBS -q workq
#PBS -l select=1:ncpus=1:mem=2gb,walltime=3:00:00
cd $PBS_O_WORKDIR
#An example is as below. Modify accordingly to suit your needs
singularity exec -B /scratch/${USER}:/scratch --pwd /scratch/${USER}:/scratch --pwd $HOME/OpenFOAM/${USER}-4.1 $HOME/sw/Containers/openfoam4-paraview50_latest.sif "./openfoamLauncher4.sh"
exit
sleep 2
|
cd $HOME/OpenFOAM/${USER}-4.1/
qsub pbs.01
Interactive pbs job
| No Format |
|---|
qsub -I -l select=1:ncpus=1:mem=2gb,walltime=3:00:00
#Once inside the job:
cd $PBS_O_WORKDIR
singularity shell -B /scratch/${USER}:/scratch --pwd /scratch/${USER}:/scratch --pwd $HOME/OpenFOAM/s3000725-4.1 $HOME/sw/Containers/openfoam4-paraview50_latest.sif
#Once inside the container:
source /opt/openfoam4/etc/bashrc
mkdir -p $FOAM_RUN
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
paraFoam |
openFOAM version 11
Openfoam on the cluster
Ref: https://hub.docker.com/r/openfoam/openfoam4-paraview50
https://openfoam.org/download/4-1-linux/
Old Information: For Legacy reasons only
Usage
| No Format |
|---|
module load OpenFoam/4.0.0
echo "alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'" >> $HOME/.bashrc
>>>>>>>>>>>
Add this into your $home/.bashrc
alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'
>>>>>>>>>>>
End User Update: Nov 5th 2018:
=================================
Add this into your $home/.bashrc
alias of40='module load OpenFoam/4.0.0; source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc /sw/OpenFoam/OpenFOAM-4.0/etc/pref.sh WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes';
>>>>>>>>>>>>>>>>>>>>>>>>>
source /sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes
Reminder: Whenever you start a new terminal window or tab, you should run the alias command associated to the OpenFOAM 4.0 shell environment. In other words, run the following command whenever you start a new terminal:
of40 |
Installation
##source: https://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-4.0/CentOS_SL_RHEL
| No Format |
|---|
module load library/libxml2/2.5
module load misc/extralibs/extralibs
module load library/bzip2/1.0.6fpic
module load NetCDF/4.4.1.1-gnu
module load misc/openssl/1.0.2
module load jdk/1.8.0_66
module load library/xz/5.3.3
module load library/gdal/2.3.0
module load misc/curl/7.54.0
module load library/gmp/6.0.0
module load pcre/8.39
module load mpi/openMPI/1.8.5-gcc4.9.0
module load library/mpfr/3.1.2
module load gcc/6.3.0
module load library/geos/3.6.1
module load library/hdf5/1.8.18-parallel
module load library/bzip2/1.0.6
module load library/hdf5/1.8.18
module load library/tiff/4.0.7
module load openmpi-x86_64
module load ATLAS/3.9.39
module load misc/rtmpdump/rtmp
module load library/libgeotiff/1.4.1
module load library/expat/2.2.0
module load library/mpc/1.0.2
module load jags/4.2.0
module load R/3.4.0
module load library/jasper/1.900.1
module load library/zlib/1.2.11
module load library/zlib/1.2.5
cd /sw/OpenFoam/OpenFOAM-4.0
wget "http://download.openfoam.org/source/4-0" -O OpenFOAM-4.0.tgz
wget "http://download.openfoam.org/third-party/4-0" -O ThirdParty-4.0.tgz
tar -xzf OpenFOAM-4.0.tgz
tar -xzf ThirdParty-4.0.tgz
mv OpenFOAM-4.x-version-4.0 OpenFOAM-4.0
mv ThirdParty-4.x-version-4.0 ThirdParty-4.0
cd ThirdParty-4.0
mkdir download
wget -P download http://www.cmake.org/files/v3.2/cmake-3.2.1.tar.gz
wget -P download https://github.com/CGAL/cgal/releases/download/releases%2FCGAL-4.8/CGAL-4.8.tar.xz
wget -P download http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.bz2
tar -xzf download/cmake-3.2.1.tar.gz
tar -xJf download/CGAL-4.8.tar.xz
tar -xjf download/boost_1_55_0.tar.bz2
cd /sw/OpenFoam/OpenFOAM-4.0
sed -i -e 's/\(boost_version=\)boost-system/\1boost_1_55_0/' OpenFOAM-4.0/etc/config.sh/CGAL
sed -i -e 's/\(cgal_version=\)cgal-system/\1CGAL-4.8/' OpenFOAM-4.0/etc/config.sh/CGAL
ln -s /sw/gcc/4.8.5 /sw/OpenFoam/ThirdParty-4.0/platforms/linux64/gcc-4.8.5
module load openmpi-x86_64
source /sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes
alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0
wget "http://download.openfoam.org/source/4-0" -O OpenFOAM-4.0.tgz
wget "http://download.openfoam.org/third-party/4-0" -O ThirdParty-4.0.tgz
tar -xzf OpenFOAM-4.0.tgz
tar -xzf ThirdParty-4.0.tgz
mv OpenFOAM-4.x-version-4.0 OpenFOAM-4.0
mv ThirdParty-4.x-version-4.0 ThirdParty-4.0
cd ThirdParty-4.0
mkdir download
wget -P download http://www.cmake.org/files/v3.2/cmake-3.2.1.tar.gz
wget -P download https://github.com/CGAL/cgal/releases/download/releases%2FCGAL-4.8/CGAL-4.8.tar.xz
wget -P download http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.bz2
tar -xzf download/cmake-3.2.1.tar.gz
tar -xJf download/CGAL-4.8.tar.xz
tar -xjf download/boost_1_55_0.tar.bz2
cd /sw/OpenFoam/OpenFOAM-4.0
sed -i -e 's/\(boost_version=\)boost-system/\1boost_1_55_0/' OpenFOAM-4.0/etc/config.sh/CGAL
sed -i -e 's/\(cgal_version=\)cgal-system/\1CGAL-4.8/' OpenFOAM-4.0/etc/config.sh/CGAL
ln -s /sw/gcc/4.8.5 /sw/OpenFoam/ThirdParty-4.0/platforms/linux64/gcc-4.8.5
module load openmpi-x86_64
source /sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes
alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'
of40
>>>>>>>>>>>
Add this into your $home/.bashrc
echo "alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'" >> $HOME/.bashrc
>>>>>>>>>>>>/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'
of40
>>>>>>>>>>>
Add this into your $home/.bashrc
echo "alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'" >> $HOME/.bashrc
>>>>>>>>>>>>
cd $WM_THIRD_PARTY_DIR
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getGcc"
wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/makeBinutils"
wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/getBinutils"
chmod +x get* make*
cd $WM_THIRD_PARTY_DIR
./makeCmake > log.makeCmake 2>&1
wmRefresh
./getGcc gcc-4.8.5 gmp-5.1.2 mpfr-3.1.2 mpc-1.0.1
./makeGcc -no-multilib > log.makeGcc 2>&1
wmRefresh
./getBinutils
./makeBinutils gcc-4.8.5 > log.makeBinutils 2>&1
# This next command will take a little while...
./makeCGAL gmp-5.1.2 mpfr-3.1.2 > log.makeCGAL 2>&1
#update the shell environment
wmRefresh
cd $WM_THIRD_PARTY_DIR
#Get the scripts we need
wget https://github.com/wyldckat/scripts4OpenFOAM3rdParty/raw/master/getQt
wget https://github.com/OpenFOAM/ThirdParty-2.4.x/raw/master/makeQt
wget -P etc/tools/ https://github.com/OpenFOAM/ThirdParty-2.4.x/raw/master/etc/tools/QtFunctions
#make them executable
chmod +x getQt makeQt
#define correct download version and download it
sed -i -e 's=4\.6=4.8=' -e 's=4\.8\.4=4.8.6=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt
./getQt
./makeQt qt-4.8.6 > log.makeQt 2>&1
cd $WM_THIRD_PARTY_DIR
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getGcc"
wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/makeBinutils"
wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/getBinutils"
chmod +x get* make*
cd $WM_THIRD_PARTY_DIR
./makeCmake > log.makeCmake 2>&1
wmRefresh
./getGcc gcc-4.8.5 gmp-5.1.2 mpfr-3.1.2 mpc-1.0.1
./makeGcc -no-multilib > log.makeGcc 2>&1
wmRefresh
./getBinutils
./makeBinutils gcc-4.8.5 > log.makeBinutils 2>&1
# This next command will take a little while...
./makeCGAL gmp-5.1.2 mpfr-3.1.2 > log.makeCGAL 2>&1
#update the shell environment
wmRefresh
cd $WM_THIRD_PARTY_DIR
#Get the scripts we need
wget https://github.com/wyldckat/scripts4OpenFOAM3rdParty/raw/master/getQt
wget https://github.com/OpenFOAM/ThirdParty-2.4.x/raw/master/makeQt
wget -P etc/tools/ https://github.com/OpenFOAM/ThirdParty-2.4.x/raw/master/etc/tools/QtFunctions
#make them executable
chmod +x getQt makeQt
#define correct download version and download it
sed -i -e 's=4\.6=4.8=' -e 's=4\.8\.4=4.8.6=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt
./getQt
./makeQt qt-4.8.6 > log.makeQt 2>&1
cd $WM_THIRD_PARTY_DIR
#this will take a while... somewhere between 30 minutes to 2 hours or more
./makeParaView -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.6/bin/qmake -mpi -python > log.makePV 2>&1
Once the makeParaView script is finished running, make sure to check the contents of the file log.makePV and check if there are any errors.
wmRefresh
Now let's build OpenFOAM:Go into OpenFOAM's main source folder:cd $WM_PROJECT_DIR
./Allwmake -j 4 > log.make 2>&1
The "4" refers to the number of cores to be used for building in parallel. In addition, the amount of RAM needed for building scales with the number of cores used, something like 1GB of RAM per core; a minimum of 1.5GB is needed for linking the libraries, which is not done in parallel.
Run it a second time for getting a summary of the installation:./Allwmake -j 4 > log.make 2>&1
To check if everything went well:Check if icoFoam is working, by running this command:
icoFoam -help
Usage: icoFoam [OPTIONS]
options:
-case <dir> specify alternate case directory, default is the cwd
-noFunctionObjects
do not execute functionObjects
-parallel run in parallel
-roots <(dir1 .. dirN)>
slave root directories for distributed running
-srcDoc display source code in browser
-doc display application documentation in browser
-help print the usage
Using: OpenFOAM-4.0 (see www.OpenFOAM.org)
Build: 4.0
Reminder: Whenever you start a new terminal window or tab, you should run the alias command associated to the OpenFOAM 4.0 shell environment. In other words, run the following command whenever you start a new terminal:of40
|
Sample pbs script
| No Format |
|---|
#PBS -m abe
#PBS -M Myemail@griffith.edu.au
#PBS -N MyOpenFOAM
#PBS -l select=1:ncpus=1:mem=1g,walltime=1:00:00
# specifies the input file and output file
##jobname=MyOpenFOAMrun
##jobInput=~/pbs/olafoam/myrun.inp
## sets up the OpenFOAM environment
module load OpenFoam/4.0.0
module list
source /sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes
cd $PBS_O_WORKDIR
# starts simulation
icoFoam -help |
OpenFOAM on GriffithHPC
Ref: https://www.palmetto.clemson.edu/palmetto/software_singularity.html
| No Format |
|---|
Singularity is installed on the nodes but if needed latest version of singularity can be loaded with
#module load singularity
The container with openfoam6-paraview54 can be located in: /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
We use singularity shell to start a container, and run a shell in the container.
The -B option is used to “bind” the /scratch/$USER directory to a directory named /scratch in the container.
We also the --pwd option to specify the working directory in the running container (in this case /scratch). This is always recommended.
#this will take a while... somewhere between 30 minutes to 2 hours or more
./makeParaView -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.6/bin/qmake -mpi -python > log.makePV 2>&1
Once the makeParaView script is finished running, make sure to check the contents of the file log.makePV and check if there are any errors.
wmRefresh
Now let's build OpenFOAM:Go into OpenFOAM's main source folder:cd $WM_PROJECT_DIR
./Allwmake -j 4 > log.make 2>&1
The "4" refers to the number of cores to be used for building in parallel. In addition, the amount of RAM needed for building scales with the number of cores used, something like 1GB of RAM per core; a minimum of 1.5GB is needed for linking the libraries, which is not done in parallel.
Run it a second time for getting a summary of the installation:./Allwmake -j 4 > log.make 2>&1
To check if everything went well:Check if icoFoam is working, by running this command:
icoFoam -help
Usage: icoFoam [OPTIONS]
options:
-case <dir> specify alternate case directory, default is the cwd
-noFunctionObjects
do not execute functionObjects
-parallel run in parallel
-roots <(dir1 .. dirN)>
slave root directories for distributed running
-srcDoc display source code in browser
-doc display application documentation in browser
-help print the usage
Using: OpenFOAM-4.0 (see www.OpenFOAM.org)
Build: 4.0
Reminder: Whenever you start a new terminal window or tab, you should run the alias command associated to the OpenFOAM 4.0 shell environment. In other words, run the following command whenever you start a new terminal:of40
|
Sample pbs script
| No Format |
|---|
#PBS -m abe
#PBS -M Myemail@griffith.edu.au
#PBS -N MyOpenFOAM
#PBS -l select=1:ncpus=1:mem=1g,walltime=1:00:00
# specifies the input file and output file
##jobname=MyOpenFOAMrun
##jobInput=~/pbs/olafoam/myrun.inp
## sets up the OpenFOAM environment
module load OpenFoam/4.0.0
module list
source /sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes
cd $PBS_O_WORKDIR
# starts simulation
icoFoam -help |
OpenFOAM on GriffithHPC
Ref: https://www.palmetto.clemson.edu/palmetto/software_singularity.html
| No Format |
|---|
Singularity is installed on the nodes but if needed latest version of singularity can be loaded with
#module load singularity
The container with openfoam6-paraview54 can be located in: /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
We use singularity shell to start a container, and run a shell in the container.
The -B option is used to “bind” the /scratch/$USER directory to a directory named /scratch in the container.
We also the --pwd option to specify the working directory in the running container (in this case /scratch). This is always recommended.
singularity shell -B /scratch/s123456:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
Using it with PBS scheduler, we need to set up two bash scripts: the launch script that gets executed inside singularity and the PBS scheduler script that instructs the scheduler to start singularity.
==============
Prep the Image
===============
mkdir ~/Container
cp -r /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif-centos ~/Container
======================================
Launcher Script: singularity_launch.01
==========================
source /opt/openfoam6/etc/bashrc
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
#Make sure that it is executable with chmod +x singularity_launch.01
===================
pbs script: pbs.01
==================
#!/bin/bash -l
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -V
#PBS -N TestOPENFOAM
#PBS -q workq
#PBS -l select=1:ncpus=1:ngpus=1:mem=32gb,walltime=30:00:00
cd $PBS_O_WORKDIR
##run singularity
#singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
singularity shell -B /scratch/s123456s12345:/scratch --pwd /scratch/scratch/s12345:/scratch --pwd /scratch --nv /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
Using it with PBS scheduler, we need to set up two bash scripts: the launch script that gets executed inside singularity and the PBS scheduler script that instructs the scheduler to start singularity.
/scratch/s12345/singularity_launch.01
#singularity shell -B /share/data:/mnt --nv ubuntu.simg /mnt/[PATH TO ExoGAN]/singularity_launch_gan.sh
cat /etc/os-release
exit
sleep 2
|
Another example of usage on the new cluster
| No Format |
|---|
Initial Prep:
==============
Prep
Copy the images into the local Image
===============container folder:
mkdir ~/Container
cp -r /sw/OpenFoamContainers/singularity/Containersimages/openfoam6openfoam7-paraview54paraview56_latest.sif-centos ~/Container/
Shell into the container and create launcher script:
======================================
Launcher Script: singularity_launch.01
==============
Initial setup
1. shell into the container
===============================
source /opt/openfoam6/etc/bashrc
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
#Make sure that it is executable with chmod +x singularity_launch.01
===================
pbs script: pbs.01
==================
#!/bin/bash -l
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -V
#PBS -N TestOPENFOAM
#PBS -q workq
#PBS -l select=1:ncpus=1:ngpus=1:mem=32gb,walltime=30:00:00
cd $PBS_O_WORKDIR
##run singularity
#singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
singularity shell -B /scratch/s12345:/scratch --pwd /scratch/scratch/s12345:/scratch --pwd /scratch --nv /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif /scratch/s12345/singularity_launch.01
#singularity shell -B /share/data:/mnt --nv ubuntu.simg /mnt/[PATH TO ExoGAN]/singularity_launch_gan.sh
cat /etc/os-release
exit
sleep 2
|
Another example of usage on the new cluster
| No Format |
|---|
Initial Prep:
=============
Copy the images into the local container folder:
mkdir ~/Container
cp /sw/Containers/singularity/images/openfoam7-paraview56_latest.sif ~/Container/
Shell into the container and create launcher script:
====================================================
Initial setup
1. shell into the container
For example:
singularity shell -B /scratch/s12345:/scratch --pwd /scratch/s12345:/scratch --pwd /scratch /export/home/s12345/Container/openfoam7-paraview56_latest.sif
e.g:
singularity shell -B /scratch/s5126720:/scratch --pwd /scratch/s5126720:/scratch --pwd /scratch /export/home/s5126720/Container/openfoam7-paraview56_latest.sif
2. Create a variable file you can source (While shelled into the singularity container,, do this)
2a. Copy the bashrc file to scratch
cp /opt/openfoam7/etc/bashrc /scratch
Modify the /scratch/bashrc file to suit your environment.
2b. (While shelled in, do this), create a variable file (openFOAMenv7.sh) with contents like this:
vi /scratch/openFOAMenv7.sh
#!/bin/bash
export PATH=/opt/openfoam7:/opt/openfoam7/bin:/opt/paraviewopenfoam56/bin:$PATH
export LD_LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LIBRARY_PATH
export LDFLAGS=-L/opt/paraviewopenfoam56/lib:$LDFLAGS
#export PKG_CONFIG_PATH=/sw/library/loki/0.1.7/lib/pkgconfig:$PKG_CONFIG_PATH
export INCLUDE=/opt/paraviewopenfoam56/include:$INCLUDE
export CPLUS_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$CPLUS_INCLUDE_PATH
export CPATH=/opt/paraviewopenfoam56/include:$CPATH
export C_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$C_INCLUDE_PATH
export FPATH=/opt/paraviewopenfoam56/include:$FPATH
export CFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export FCFLAGS=-I/opt/paraviewopenfoam56/include:$FCFLAGS
export FFLAGS=-I/opt/paraviewopenfoam56/include:$FFLAGS
source bashrc7
3. Manual Test Run
=============================
For example:
singularity shell -B /scratch/s12345:/scratch --pwd /scratch/s12345:/scratch --pwd /scratch /export/home/s12345/Container/
(While shelled into the singularity container, do this)
source /scratch/openFOAMenv7.sh
foamInfo -help
For example:
Singularity openfoam7-paraview56_latest.sif:/opt/openfoam7> foamInfo
e.g-help
Usage: foamInfo singularity[OPTIONS] shell -B /scratch/s5126720:/scratch --pwd /scratch/s5126720:/scratch --pwd /scratch /export/home/s5126720/Container/openfoam7-paraview56_latest.sif
2. Create a variable file you can source (While shelled into the singularity container,, do this)
2a. Copy the bashrc file to scratch
cp /opt/openfoam7/etc/bashrc /scratch
Modify the /scratch/bashrc file to suit your environment.
2b. (While shelled in, do this), create a variable file (openFOAMenv7.sh) with contents like this:<name>
options:
-all | -a list all tutorials that use <name> (otherwise maximum 10)
If the output looks correct, go to the next step
4. Create a Launcher Script;
===========================
Once you are happy that all step 1 ,2 and 3 works, you can create a launcherScript
While still inside the shell:
vi /scratch/openFOAMenv7openfoamLauncher7.sh
An example of the content is this:
#!/bin/bash
export LD_LIBRARY_PATH=/opt/openfoam7:/opt/openfoam7/bin:/opt/paraviewopenfoam56/bin:$PATHlib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LD_LIBRARY$LIBRARY_PATH
export LIBRARY_PATH= PATH=/opt/openfoam7:/opt/openfoam7/bin:/opt/paraviewopenfoam56/libbin:$LIBRARY_PATH$PATH
export LDFLAGS=-L/opt/paraviewopenfoam56/lib:$LDFLAGS
#export PKG_CONFIG_PATH=/sw/library/loki/0.1.7/lib/pkgconfig:$PKG_CONFIG_PATH
export INCLUDE=/opt/paraviewopenfoam56/include:$INCLUDE
export CPLUS_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$CPLUS_INCLUDE_PATH
export CPATH=/opt/paraviewopenfoam56/include:$CPATH
export C_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$C_INCLUDE_PATH
export FPATH=/opt/paraviewopenfoam56/include:$FPATH
export FPATH=CFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export FCFLAGS=-I/opt/paraviewopenfoam56/include:$FPATH$FCFLAGS
export CFLAGSFFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export FCFLAGS=-I/opt/paraviewopenfoam56/include:$FCFLAGS
export FFLAGS=-I/opt/paraviewopenfoam56/include:$FFLAGS
source bashrc7
3. Manual Test Run $FFLAGS
source bashrc7
#Put your run instructions/command below.
foamInfo -help
cd s2940457/currentWaveFlume
chmod +x runCase
./runCase >>>>>>>>>>>>>
make it executable:
chmod +x /scratch/openfoamLauncher7.sh
Run it manually and check if it gives the expected results:
Check if it gives the expected results:
/scratch/openfoamLauncher7.sh
(OR: try cd /scratch;./openfoamLauncher.sh)
5. Running it as a batch job:
====================
(While shelled into==========
Exit the singularityshell container,and docreate this) pbs source /scratch/openFOAMenv7.sh
foamInfo -help
For examplescript anywhere in space:
Singularity openfoam7-paraview56_latest.sif:/opt/openfoam7> foamInfo -help
Usage: foamInfo [OPTIONS] <name>
options:
-all | -a list all tutorials that use <name> (otherwise maximum 10)
If the output looks correct, go to the next step
4. Create a Launcher Script;
===========================
Once you are happy that all step 1 ,2 and 3 works, you can create a launcherScript
While still inside the shell:
vi /scratch/openfoamLauncher7.sh
An example of the content is this:
#!/bin/bash
export LD_LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LIBRARY_PATH
export PATH=/opt/openfoam7:/opt/openfoam7/bin:/opt/paraviewopenfoam56/bin:$PATH
export LDFLAGS=-L/opt/paraviewopenfoam56/lib:$LDFLAGS
#export PKG_CONFIG_PATH=/sw/library/loki/0.1.7/lib/pkgconfig:$PKG_CONFIG_PATH
export INCLUDE=/opt/paraviewopenfoam56/include:$INCLUDE
export CPLUS_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$CPLUS_INCLUDE_PATH
export CPATH=/opt/paraviewopenfoam56/include:$CPATH
export C_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$C_INCLUDE_PATH
export FPATH=/opt/paraviewopenfoam56/include:$FPATH
export CFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export FCFLAGS=-I/opt/paraviewopenfoam56/include:$FCFLAGS
export FFLAGS=-I/opt/paraviewopenfoam56/include:$FFLAGS
source bashrc7
#Put your run instructions/command below.
foamInfo -help
>>>>>>>>>>>>>
make it executable:
chmod +x /scratch/openfoamLauncher7.sh
Run it manually and check if it gives the expected results:
Check if it gives the expected results:
/scratch/openfoamLauncher7.sh
(OR: try cd /scratch;./openfoamLauncher.sh)
5. Runningmkdir /scratch/s2940457/pbs
cd /scratch/s2940457/pbs
vi openfoam7.pbs01
>>>>>>>>>
!/bin/bash
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -N TestOpenFoam7
#PBS -q workq
#PBS -l select=1:ncpus=8:mem=2gb,walltime=3:00:00
cd $PBS_O_WORKDIR
#An example is as below. Modify accordingly to suit your needs
singularity exec -B /scratch/s2940457:/scratch --pwd /scratch/s2940457:/scratch --pwd /scratch /export/home/s2940457/Container/openfoam7-paraview56_latest.sif "./openfoamLauncher7.sh"
exit
sleep 2
>>>>>>
Submit the job and check the results (output and error files).
qsub openfoam7.pbs01
If all goes well, then the only file that needs to be edited in the future is the openfoamLauncher7.sh
To run in parallel, you may look at online documentation.
>>>>>>>>>>>>>>
e.g
How to run in parallel:
https://www.youtube.com/watch?v=YdYHDMygNPU
>>>>>>>>>> |
Another example with openfoam8
| No Format |
|---|
source /usr/local/bin/s3proxy.sh
mkdir -p $HOME/sw/Containers/openfoam8/
cd $HOME/sw/Containers/openfoam8/
singularity pull docker://public.docker.itc.griffith.edu.au/openfoam/openfoam8-paraview56
mkdir -p $HOME/OpenFOAM/${USER}-8
cd $HOME/OpenFOAM/${USER}-8
singularity shell -B /export/home/${USER}:/home --pwd /export/home/${USER}:/home --pwd $HOME/OpenFOAM/${USER}-8 $HOME/sw/Containers/openfoam8/openfoam8-paraview56_latest.sif
Once inside the shell:
source /opt/openfoam8/etc/bashrc
mkdir -p $FOAM_RUN
#echo $FOAM_RUN
#/export/home/s5072653/run
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
paraFoam
To run it as a batch job:
==============================
Exit the shell and create this pbs script anywhere in space:
mkdir /scratch/s2940457/pbs
cd /scratch/s2940457/pbs
vi openfoam7.pbs01
>>>>>>>>>
, create a launcher script:
1. Launcher Script
>>>>>>>
vi ~/pbs/openfoam8Launcher.sh
#!/bin/bash #PBS
-m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -N TestOpenFoam7
#PBS -q workq
#PBS -l select=1:ncpus=8:mem=2gb,walltime=3:00:00
cd $PBS_O_WORKDIR
#An example is as below. Modify accordingly to suit your needs
singularity exec -B /scratch/s2940457:/scratch --pwd /scratch/s2940457:/scratch --pwd /scratch /export/home/s2940457/Container/openfoam7-paraview56_latest.sif "./openfoamLauncher7.sh"
exit
sleep 2
>>>>>>
Submit the job and check the results (output and error files).
qsub openfoam7.pbs01
If all goes well, then the only file that needs to be edited in the future is the openfoamLauncher7.sh
To run in parallel, you may look at online documentation.
>>>>>>>>>>>>>>
e.g
How to run in parallel:
https://www.youtube.com/watch?v=YdYHDMygNPU
>>>>>>>>>>source /opt/openfoam8/etc/bashrc
mkdir -p $FOAM_RUN
#cd ~/run
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
>>>>>>>>
#make it executable
chmod +7 ~/pbs/openfoam8Launcher.sh
2. pbs script
vi ~/pbs/openfoam8.pbs01
>>>>>
#!/bin/bash
#PBS -m abe
#PBS -M YourEmail@griffithuni.edu.au
#PBS -N TestOpenFoam8
#PBS -q workq
#PBS -l select=1:ncpus=8:mem=12gb,walltime=3:00:00
cd $PBS_O_WORKDIR
#An example is as below. Modify accordingly to suit your needs
singularity exec -B ~:/home --pwd ~:/home --pwd /home $HOME/sw/Containers/openfoam8/openfoam8-paraview56_latest.sif "pbs/openfoam8Launcher.sh"
exit
sleep 2
>>>>>>>
Please submit the test pbs script to check functionality
cd ~/pbs
qsub openfoam8.pbs01
|
Reference
- https://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-4.0/CentOS_SL_RHEL
- https://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-4.0
- https://openfoamwiki.net/index.php/Main_Page
- https://cfd.direct/openfoam/user-guide/