Table of Contents

Introduction

OpenFOAM is a framework for developing application executables that use packaged functionality contained within a collection of approximately 100 C+ libraries. OpenFOAM is shipped with approximately 250 pre-built applications that fall into two categories: solvers, that are each designed to solve a specific problem in fluid (or continuum) mechanics; and utilities, that are designed to perform tasks that involve data manipulation.

Source: https://cfd.direct/openfoam/user-guide/

...

No Format

mkdir -p $HOME/OpenFOAM/${USER}-4.1
cd $HOME/OpenFOAM/${USER}-4.1

singularity  shell -B /scratch/${USER}:/scratch --pwd /scratch/${USER}:/scratch --pwd $HOME/OpenFOAM/${USER}-4.1 $HOME/sw/Containers/openfoam4-paraview50_latest.sif

Once inside the shell:
mkdir
-p
$FOAM_RUN
source /opt/openfoam4/etc/bashrc
mkdir -p $FOAM_RUN
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
paraFoam

...

cd $HOME/OpenFOAM/${USER}-4.1/

qsub pbs.01

Ref: https://hub.docker.com/r/openfoam/openfoam4-paraview50
https://openfoam.org/download/4-1-linux/

Old Information: For Legacy reasons only

Usage

...

Interactive pbs job

No Format

qsub -I -l select=1:ncpus=1:mem=2gb,walltime=3:00:00
#Once inside the job:
cd $PBS_O_WORKDIR
singularity  shell -B /scratch/${USER}:/scratch --pwd /scratch/${USER}:/scratch --pwd $HOME/OpenFOAM/s3000725-4.1 $HOME/sw/Containers/openfoam4-paraview50_latest.sif

#Once inside the container:
source /opt/openfoam4/etc/bashrc
mkdir -p $FOAM_RUN
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
paraFoam

openFOAM version 11

Openfoam on the cluster

Ref: https://hub.docker.com/r/openfoam/openfoam4-paraview50
https://openfoam.org/download/4-1-linux/

Old Information: For Legacy reasons only

Usage

No Format

module load OpenFoam/4.0.0
 
echo "alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'"  >> $HOME/.bashrc

>>>>>>>>>>>
EndAdd Userthis Update:into Nov 5th 2018:
====your $home/.bashrc
alias of40=' source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes'  
>>>>>>>>>>>
End User Update: Nov 5th 2018:
=================================
Add this into your $home/.bashrc
alias of40='module load OpenFoam/4.0.0; source \/sw/OpenFoam/OpenFOAM-4.0/etc/bashrc /sw/OpenFoam/OpenFOAM-4.0/etc/pref.sh WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes';
>>>>>>>>>>>>>>>>>>>>>>>>>

source /sw/OpenFoam/OpenFOAM-4.0/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=ThirdParty FOAMY_HEX_MESH=yes

Reminder: Whenever you start a new terminal window or tab, you should run the alias command associated to the OpenFOAM 4.0 shell environment. In other words, run the following command whenever you start a new terminal:

of40

...

Ref: https://www.palmetto.clemson.edu/palmetto/software_singularity.html

...

.edu/palmetto/software_singularity.html

No Format

Singularity is installed on the nodes but if needed latest version of singularity can be loaded with
#module load singularity

The container with openfoam6-paraview54 can be located in: /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif

We use singularity shell to start a container, and run a shell in the container.

The -B option is used to “bind” the /scratch/$USER directory to a directory named /scratch in the container.

We also the --pwd option to specify the working directory in the running container (in this case /scratch). This is always recommended.


singularity shell -B /scratch/s123456:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif

Using it with PBS scheduler, we need to set up two bash scripts: the launch script that gets executed inside singularity and the PBS scheduler script that instructs the scheduler to start singularity.

==============
Prep the Image
===============
mkdir ~/Container
cp -r /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif-centos ~/Container


======================================
Launcher Script: singularity_launch.01
==========================
source /opt/openfoam6/etc/bashrc 
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam

#Make sure that it is executable with chmod +x singularity_launch.01


===================
pbs script: pbs.01
==================

#!/bin/bash -l
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -V
#PBS -N TestOPENFOAM
#PBS -q workq
#PBS -l select=1:ncpus=1:ngpus=1:mem=32gb,walltime=30:00:00
cd  $PBS_O_WORKDIR
##run singularity
#singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif

We use singularity shell to start a container, and run a shell in the container.

The -B option is used to “bind” the /scratch/$USER directory to a directory named /scratch in the container.

We also the --pwd option to specify the working directory in the running container (in this case /scratch). This is always recommended.


singularitylatest.sif
singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
singularity shell -B /scratch/s12345:/scratch --pwd /scratch/scratch/s12345:/scratch --pwd /scratch --nv /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif /scratch/s12345/singularity_launch.01
#singularity shell -B /scratchshare/s123456data:/scratchmnt --pwdnv ubuntu.simg /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif

Using it with PBS scheduler, we need to set up two bash scripts: the launch script that gets executed inside singularity and the PBS scheduler script that instructs the scheduler to start singularity.
mnt/[PATH TO ExoGAN]/singularity_launch_gan.sh
cat /etc/os-release
exit
sleep 2

Another example of usage on the new cluster

No Format

Initial Prep:
=============
 
Copy the images into the local container folder:
 
mkdir ~/Container
cp /sw/Containers/singularity/images/openfoam7-paraview56_latest.sif ~/Container/

Shell into the container and create launcher script:
==============
Prep the Image
===========================
mkdir ~/Container
cp -r /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif-centos ~/Container


===========
 
Initial setup
 
1. shell into the container
===========================
Launcher Script: singularity_launch.01
==========================
source /opt/openfoam6/etc/bashrc 
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam

#Make sure that it is executable with chmod +x singularity_launch.01


===================
pbs script: pbs.01
==================

#!/bin/bash -l
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -V
#PBS -N TestOPENFOAM
#PBS -q workq
#PBS -l select=1:ncpus=1:ngpus=1:mem=32gb,walltime=30:00:00
cd  $PBS_O_WORKDIR
##run singularity
#singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
singularity shell -B /scratch/s12345:/scratch --pwd /scratch /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif
singularity shell -B /scratch/s12345:/scratch --pwd /scratch/scratch/s12345:/scratch --pwd /scratch --nv /sw/OpenFoam/Containers/openfoam6-paraview54_latest.sif /scratch/s12345/singularity_launch.01
#singularity shell -B /share/data:/mnt --nv ubuntu.simg /mnt/[PATH TO ExoGAN]/singularity_launch_gan.sh
cat /etc/os-release
exit
sleep 2

Another example of usage on the new cluster

No Format

Initial Prep:
=============
 
Copy the images into the local container folder:
 
mkdir ~/Container
cp /sw/Containers/singularity/images/openfoam7-paraview56_latest.sif ~/Container/

Shell into the container and create launcher script:
====================================================
 
Initial setup
 
1. shell into the container==
For example:

singularity  shell -B /scratch/s12345:/scratch --pwd /scratch/s12345:/scratch --pwd /scratch  /export/home/s12345/Container/openfoam7-paraview56_latest.sif
 
e.g:

singularity  shell -B /scratch/s5126720:/scratch --pwd /scratch/s5126720:/scratch --pwd /scratch  /export/home/s5126720/Container/openfoam7-paraview56_latest.sif
 
2. Create a variable file you can source (While shelled into the singularity container,, do this)

2a. Copy the bashrc file to scratch
cp   /opt/openfoam7/etc/bashrc /scratch
Modify the /scratch/bashrc file to suit your environment.

2b. (While shelled in, do this), create a variable file (openFOAMenv7.sh) with contents like this:

vi /scratch/openFOAMenv7.sh
#!/bin/bash
export PATH=/opt/openfoam7:/opt/openfoam7/bin:/opt/paraviewopenfoam56/bin:$PATH
export LD_LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LIBRARY_PATH
export   LDFLAGS=-L/opt/paraviewopenfoam56/lib:$LDFLAGS
#export   PKG_CONFIG_PATH=/sw/library/loki/0.1.7/lib/pkgconfig:$PKG_CONFIG_PATH
export   INCLUDE=/opt/paraviewopenfoam56/include:$INCLUDE
export   CPLUS_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$CPLUS_INCLUDE_PATH
export   CPATH=/opt/paraviewopenfoam56/include:$CPATH
export   C_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$C_INCLUDE_PATH
export   FPATH=/opt/paraviewopenfoam56/include:$FPATH
export   CFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export   CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export   CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export   FCFLAGS=-I/opt/paraviewopenfoam56/include:$FCFLAGS
export   FFLAGS=-I/opt/paraviewopenfoam56/include:$FFLAGS 
source bashrc7  

3. Manual Test Run 
=============================
For example:

 
(While shelled into the singularity  shell -Bcontainer, do this)

source /scratch/s12345:/scratch --pwd /scratch/s12345:/scratch --pwd /scratch  /export/home/s12345/Container/openFOAMenv7.sh 
foamInfo -help

For example:
 
Singularity openfoam7-paraview56_latest.sif:/opt/openfoam7> foamInfo -help

Usage: foamInfo [OPTIONS] <name>
e.goptions:
 singularity -all shell -B /scratch/s5126720:/scratch --pwd /scratch/s5126720:/scratch | --pwda /scratch  /export/home/s5126720/Container/openfoam7-paraview56_latest.sif   2. Create alist variableall filetutorials youthat canuse source<name> (Whileotherwise shelledmaximum into10)
the
singularityIf container,,the dooutput this)looks correct, 2a.go Copyto the bashrcnext filestep
to
scratch4. cpCreate a  /opt/openfoam7/etc/bashrc /scratch
Modify the /scratch/bashrc file to suit your environment.

2b. (While shelled in, do this),Launcher Script;
===========================
Once you are happy that all step 1 ,2 and 3 works, you can create a variablelauncherScript
file (openFOAMenv7.sh) with contents like this:While still inside the shell:
 
vi /scratch/openFOAMenv7openfoamLauncher7.sh
 
An example of the content is this: 
#!/bin/bash 
export LD_LIBRARY_PATH=/opt/openfoam7:/opt/openfoam7/bin:/opt/paraviewopenfoam56/bin:$PATHlib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LD$LIBRARY_LIBRARY_PATH
export LIBRARY_PATH=/opt/openfoam7:/opt/openfoam7/bin:/opt/paraviewopenfoam56/libbin:$LIBRARY_PATH$PATH
export   LDFLAGS=-L/opt/paraviewopenfoam56/lib:$LDFLAGS
#export   PKG_CONFIG_PATH=/sw/library/loki/0.1.7/lib/pkgconfig:$PKG_CONFIG_PATH
export   INCLUDE=/opt/paraviewopenfoam56/include:$INCLUDE
export   CPLUS_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$CPLUS_INCLUDE_PATH
export   CPATH=/opt/paraviewopenfoam56/include:$CPATH
export   C_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$C_INCLUDE_PATH
export   FPATH=FPATH=/opt/paraviewopenfoam56/include:$FPATH
export   CFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export   CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export   CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export   FCFLAGS=-I/opt/paraviewopenfoam56/include:$FPATH$FCFLAGS
export   CFLAGSFFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export   CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export   CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export   FCFLAGS=-I/opt/paraviewopenfoam56/include:$FCFLAGS
export   FFLAGS=-I/opt/paraviewopenfoam56/include:$FFLAGS 
source bashrc7  

3. Manual Test Run 
$FFLAGS 
source bashrc7
#Put your run instructions/command below.
foamInfo -help
cd s2940457/currentWaveFlume
chmod +x runCase
./runCase  >>>>>>>>>>>>> 

make it executable:
chmod +x /scratch/openfoamLauncher7.sh

Run it manually and check if it gives the expected results:

Check if it gives the expected results:
/scratch/openfoamLauncher7.sh
(OR: try cd /scratch;./openfoamLauncher.sh)

5. Running it as a batch job:
==============================


(While shelled into Exit the singularityshell container,and docreate this) pbs source /scratch/openFOAMenv7.sh 
foamInfo -help

For examplescript anywhere in space:
 
Singularity openfoam7-paraview56_latest.sif:/opt/openfoam7> foamInfo -help

Usage: foamInfo [OPTIONS] <name>
options:
  -all     | -a        list all tutorials that use <name> (otherwise maximum 10)

If the output looks correct, go to the next step

4. Create a Launcher Script;
===========================
Once you are happy that all step 1 ,2 and 3 works, you can create a launcherScript
While still inside the shell:
 
vi /scratch/openfoamLauncher7.sh
 
An example of the content is this: 
#!/bin/bash 
export LD_LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/paraviewopenfoam56/lib:$LIBRARY_PATH
export PATH=/opt/openfoam7:/opt/openfoam7/bin:/opt/paraviewopenfoam56/bin:$PATH
export   LDFLAGS=-L/opt/paraviewopenfoam56/lib:$LDFLAGS
#export   PKG_CONFIG_PATH=/sw/library/loki/0.1.7/lib/pkgconfig:$PKG_CONFIG_PATH
export   INCLUDE=/opt/paraviewopenfoam56/include:$INCLUDE
export   CPLUS_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$CPLUS_INCLUDE_PATH
export   CPATH=/opt/paraviewopenfoam56/include:$CPATH
export   C_INCLUDE_PATH=/opt/paraviewopenfoam56/include:$C_INCLUDE_PATH
export   FPATH=/opt/paraviewopenfoam56/include:$FPATH
export   CFLAGS=-I/opt/paraviewopenfoam56/include:$CFLAGS
export   CPPFLAGS=-I/opt/paraviewopenfoam56/include:$CPPFLAGS
export   CXXFLAGS=-I/opt/paraviewopenfoam56/include:$CXXFLAGS
export   FCFLAGS=-I/opt/paraviewopenfoam56/include:$FCFLAGS
export   FFLAGS=-I/opt/paraviewopenfoam56/include:$FFLAGS 
source bashrc7
#Put your run instructions/command below.
foamInfo -help
cd s2940457/currentWaveFlume
chmod +x runCase
./runCase  >>>>>>>>>>>>> 

make it executable:
chmod +x /scratch/openfoamLauncher7.sh

Run it manually and check if it gives the expected results:

Check if it gives the expected results:
/scratch/openfoamLauncher7.sh
(OR: try cd /scratch;./openfoamLauncher.sh)

5. Running it as a batch job:
==============================

Exit the shell and create this pbs script anywhere in space:
 
mkdir /scratch/s2940457/pbs
cd /scratch/s2940457/pbs
vi openfoam7.pbs01

>>>>>>>>>  

!/bin/bash
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -N TestOpenFoam7
#PBS -q workq
#PBS -l select=1:ncpus=8:mem=2gb,walltime=3:00:00
cd  $PBS_O_WORKDIR
#An example is as below. Modify accordingly to suit your needs
singularity  exec -B /scratch/s2940457:/scratch --pwd /scratch/s2940457:/scratch --pwd /scratch  /export/home/s2940457/Container/openfoam7-paraview56_latest.sif "./openfoamLauncher7.sh"
exit
sleep 2 
>>>>>> 
 
Submit the job and check the results (output and error files).
 
qsub openfoam7.pbs01  


If all goes well, then the only file that needs to be edited in the future is the openfoamLauncher7.sh

To run in parallel, you may look at online documentation.
>>>>>>>>>>>>>>
e.g
How to run in parallel:
https://www.youtube.com/watch?v=YdYHDMygNPU
>>>>>>>>>>mkdir /scratch/s2940457/pbs
cd /scratch/s2940457/pbs
vi openfoam7.pbs01

>>>>>>>>>  

!/bin/bash
#PBS -m abe
#PBS -M YourEmail@griffith.edu.au
#PBS -N TestOpenFoam7
#PBS -q workq
#PBS -l select=1:ncpus=8:mem=2gb,walltime=3:00:00
cd  $PBS_O_WORKDIR
#An example is as below. Modify accordingly to suit your needs
singularity  exec -B /scratch/s2940457:/scratch --pwd /scratch/s2940457:/scratch --pwd /scratch  /export/home/s2940457/Container/openfoam7-paraview56_latest.sif "./openfoamLauncher7.sh"
exit
sleep 2 
>>>>>> 
 
Submit the job and check the results (output and error files).
 
qsub openfoam7.pbs01  


If all goes well, then the only file that needs to be edited in the future is the openfoamLauncher7.sh

To run in parallel, you may look at online documentation.
>>>>>>>>>>>>>>
e.g
How to run in parallel:
https://www.youtube.com/watch?v=YdYHDMygNPU
>>>>>>>>>>

Another example with openfoam8

No Format

source /usr/local/bin/s3proxy.sh
mkdir -p  $HOME/sw/Containers/openfoam8/
cd  $HOME/sw/Containers/openfoam8/
singularity pull  docker://public.docker.itc.griffith.edu.au/openfoam/openfoam8-paraview56


mkdir -p $HOME/OpenFOAM/${USER}-8
cd $HOME/OpenFOAM/${USER}-8
singularity  shell -B /export/home/${USER}:/home --pwd /export/home/${USER}:/home --pwd $HOME/OpenFOAM/${USER}-8 $HOME/sw/Containers/openfoam8/openfoam8-paraview56_latest.sif

Once inside the shell:

source /opt/openfoam8/etc/bashrc
mkdir -p $FOAM_RUN
#echo $FOAM_RUN
#/export/home/s5072653/run

cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
paraFoam

To run it as a batch job, create a launcher script:
1. Launcher Script
>>>>>>>
vi ~/pbs/openfoam8Launcher.sh
#!/bin/bash 
source /opt/openfoam8/etc/bashrc
mkdir -p $FOAM_RUN
#cd ~/run
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
>>>>>>>>
#make it executable
chmod +7 ~/pbs/openfoam8Launcher.sh

2. pbs script

vi ~/pbs/openfoam8.pbs01
>>>>>
#!/bin/bash
#PBS -m abe
#PBS -M YourEmail@griffithuni.edu.au
#PBS -N TestOpenFoam8
#PBS -q workq
#PBS -l select=1:ncpus=8:mem=12gb,walltime=3:00:00
cd  $PBS_O_WORKDIR
#An example is as below. Modify accordingly to suit your needs
singularity  exec -B ~:/home --pwd ~:/home --pwd /home  $HOME/sw/Containers/openfoam8/openfoam8-paraview56_latest.sif "pbs/openfoam8Launcher.sh"
exit
sleep 2
>>>>>>>

Please submit the test pbs script to check functionality
cd ~/pbs
qsub openfoam8.pbs01

Versions Compared

Old Version 37

New Version Current

Key

Introduction

Old Information: For Legacy reasons only

Usage

Interactive pbs job

openFOAM version 11

Old Information: For Legacy reasons only

Usage

Another example of usage on the new cluster

Another example of usage on the new cluster

Another example with openfoam8

Reference

Page Comparison

Versions Compared

Old Version 37

New Version Current

Key

Introduction

Old Information: For Legacy reasons only

Usage

Interactive pbs job

openFOAM version 11

Old Information: For Legacy reasons only

Usage

Another example of usage on the new cluster

Another example of usage on the new cluster

Another example with openfoam8

Reference