Regional Ocean Modeling System (ROMS)

Owned by Indy Siva
Dec 10, 2015
5 min read

ROMS is a free-surface, terrain-following, primitive equations ocean model widely used by the scientific community for a diverse range of applications

https://www.myroms.org/

https://www.myroms.org/wiki/index.php/Documentation_Portal

Special Note

ROMS has not been made available to all cluster users currently. If there is sufficient interest, it can be made available to all users. However, please see below notes on how to build this package on gowonda.

Usage

Serial Version

 module load roms/3.5serial-gnu

MPI version

 module load roms/3.5mpi-gnu

Shared Memory (Open MP version)

module load roms/3.5omp-gnu

Build Instructions from the installation manual

https://www.myroms.org/wiki/index.php/Getting_Started

You can also type make at the top of the directory structure where the makefile is located but we do not recommend this process because it requires changing the ROMS files which can cause conflicts when you update you ROMS code.

To speed compilation, you may want to add the -j <n> flag to the build command (i.e. build.sh -j 4) where <n> is the number of processors you wish to compile with. Even single processor machines can benefit from the -j flag with <n> = 2.

o make sure your application can compile successfully, you might want to set USE_DEBUG to on in the build script since it will compile faster. Once your application can compile you can unset USE_DEBUG in order to create an optimized executable.

https://www.myroms.org/wiki/index.php/build_Script

As mentioned in makefile, you need to provide settings for some user-defined choices before you can compile ROMS. If you have more than one application (or more than one compiler), you may get tired of editing the makefile. One option is to have a makefile for each configuration. The recommended solution, however, is to used the ROMS build script.

There are two of these build scripts in the ROMS/Bin directory: build.sh (which is surprisingly a csh script) and build.bash. The build scripts use environment variables to provide values for the user-defined make variables, overwriting those found in the ROMS makefile. Just as in the multiple makefile option, you will need as many copies of the build script as you have applications. However, the scope of these variables is local to the build script, allowing you to compile different applications at the same time from the same sources as long as each $(SCRATCH_DIR) is unique.

    Usage:

    ./build.bash [options]

    Options:

    -j [N] Compile in parallel using N CPUs. Omit argument to compile
    on all available CPUs.
    -noclean Do not clean already compiled objects.

    Example:

    ./build.sh -j 2

Test Build

module purge
module load mpi/openMPI/1.4.3-gnu
module load NetCDF/4.1.3-gnu

 module list
Currently Loaded Modulefiles:
  1) mpi/openMPI/1.4.3-gnu   2) library/zlib/1.2.5      3) library/hdf5/1.8.7      4) NetCDF/4.1.3-gnu



cd ROMS/Bin/
cp build.bash build-gnu.bash

vi build-gnu.bash

Item#

User Definable Compilation Options

Modified One

1.

export   ROMS_APPLICATION=UPWELLING

Same

2.

MY_ROOT_DIR=${HOME}/ocean/repository
MY_ROOT_DIR=${HOME}/pbs/dj/modelling/ocean/repository

3.

 MY_PROJECT_DIR=${PWD}

Same

4.

MY_ROMS_SRC=${MY_ROOT_DIR}/trunk
MY_ROMS_SRC=${HOME}/pbs/dj/modelling/roms/src

5.

export         COMPILERS=${MY_ROMS_SRC}/Compilers

same

6.

MY_CPP_FLAGS

same

7.

export           USE_MPI=on            # distributed-memory parallelism
export        USE_MPIF90=on            # compile with mpif90 script

same

8.

export         which_MPI=openmpi       # compile with OpenMPI library

same

9.

#export        USE_OpenMP=on            # shared-memory parallelism
you cannot set USE_OpenMP and USE_MPI at the same time.

same

10.

export              FORT=gfortran

same

11.

export         USE_DEBUG=on            # use Fortran debugging flags

same

12.

 export         USE_LARGE=on            # activate 64-bit compilation

same

13.

export       USE_NETCDF4=on            # compile with NetCDF-4 library

same

14.

#export   USE_PARALLEL_IO=on            # Parallel I/O with Netcdf-4/HDF5

same

15.

#export       USE_MY_LIBS=on            # use my library paths below

Use the custom library paths specified below for coupling libraries, NetCDF libraries, etc.

same

16.

    gfortran )
      if [ "${which_MPI}" = "mpich2" ]; then
        export PATH=/opt/gfortransoft/mpich2/bin:$PATH
      elif [ "${which_MPI}" = "openmpi" ]; then
        export PATH=/sw/openMPI/1.4.3-gnu/bin:$PATH
      fi
      ;;

-

17.

 export     MY_HEADER_DIR=${MY_PROJECT_DIR}  
 export MY_HEADER_DIR=${HOME}/pbs/dj/modelling/roms/ROMS/Include

18.

 export MY_ANALYTICAL_DIR=${MY_PROJECT_DIR}

same

19.

  export  BINDIR=${MY_PROJECT_DIR}

same

20.

   export  SCRATCH_DIR=${MY_PROJECT_DIR}/Build

same

Edit this file as well: src/Compilers/Linux-gfortran.mk
and add

 FC := /sw/openMPI/1.4.3-gnu/bin/mpif90 
  ifdef USE_MPIF90
#               FC := /usr/lib64/mpich2/bin/mpif90
#               FC := /usr/lib64/lam/bin/mpif77
               FC := /sw/openMPI/1.4.3-gnu/bin/mpif90

sh build-gnu.bash 2>&1 |tee roms_build.txt

testing

 ./oceanG

 Model Input Parameters:  ROMS/TOMS version 3.5
                          Wednesday - August 31, 2011 - 10:47:25 AM
 -----------------------------------------------------------------------------

 INP_PAR - Unable to open ROMS/TOMS input script file.
           In distributed-memory applications, the input
           script file is processed in parallel. The Unix
           routine GETARG is used to get script file name.
           For example, in MPI applications make sure that
           command line is something like:

           mpirun -np 4 ocean ocean.in

           and not

           mpirun -np 4 ocean < ocean.in


 Elapsed CPU time (seconds):


 Analytical header files used:

Sample PBS script

cat romsrun-forecast-openMPI.pbs

#PBS -m abe
#PBS -M <YourEmail>@griffith.edu.au
#PBS -N ROMS_OpenMPI
#PBS -l select=5:ncpus=2:mem=4g:mpiprocs=2
source $HOME/.bashrc
module load mpi/openMPI/1.4.3-gnu
module load NetCDF/4.1.3-gnu
## The number of nodes is given by the select =<NUM > above
NODES=5

###$PBS_NODEFILE is a node-list file created with select and mpiprocs options by PBS
###### The number of MPI processes available is mpiprocs * nodes (=NPROCS)
NPROCS=10

ROMS_DIR=/export/home/<snumber>/pbs/dj/modelling/roms/forecast/work
HOME_DIR=/export/home/<snumber>
BIN_DIR=/export/home/snumber/pbs/dj/modelling/roms/src/ROMS/Bin/

echo "Starting job"

/sw/openMPI/1.4.3-gnu/bin/mpirun -machinefile $PBS_NODEFILE -np $NPROCS  env PATH=$PATH  env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ${BIN_DIR}/oceanG  ${ROMS_DIR}/ocean_eau_forecast.in
#
echo "Done with job"

openMP

Compiling roms for openMP (shared memory) run

module purge
module load NetCDF/4.1.3-gnu
sh  build-gnu-openMP.bash  2>&1 |tee roms_openMP_build.txt

Sample pbs file

cat romsrun-forecast-openMP.pbs

#PBS -m abe
#PBS -M <YourEmail>@griffith.edu.au
#PBS -N ROMS_OpenMP
#PBS -l select=1:ncpus=4:mem=12g
source $HOME/.bashrc
module load NetCDF/4.1.3-gnu
#To run ROMS in parallel with shared-memory mode (OpenMP) on four processors
export OMP_NUM_THREADS=4

ROMS_DIR=/export/home/<snumber>/pbs/dj/modelling/roms/forecast/work
HOME_DIR=/export/home/<snumber>
BIN_DIR=/export/home/snumber/pbs/dj/modelling/roms/src/ROMS/Bin/

echo "Starting job"
${BIN_DIR}/oceanG-openMP < ${ROMS_DIR}/ocean_eau_forecast-openMP.in
# 
echo "Done with job"

roms-serial

Compiling roms for serial run

cd /export/home/snumber/pbs/dj/modelling/roms/src/ROMS/Bin
module purge
module load NetCDF/4.1.3-gnu
sh  build-gnu-serial.bash  2>&1 |tee roms_serial_build.txt

Sample pbs file

cat romsrun-forecast-openMP.pbs

#PBS -m abe
#PBS -M <YourEmail>@griffith.edu.au
#PBS -N ROMS_OpenMP
#PBS -l select=1:ncpus=4:mem=12g
source $HOME/.bashrc
module load NetCDF/4.1.3-gnu

ROMS_DIR=/export/home/<snumber>/pbs/dj/modelling/roms/forecast/work
HOME_DIR=/export/home/<snumber>
BIN_DIR=/export/home/snumber/pbs/dj/modelling/roms/src/ROMS/Bin/

echo "Starting job"
${BIN_DIR}/oceanG-serial < ${ROMS_DIR}/ocean_eau_forecast-serial.in
# 
echo "Done with job"