Navigation

Spack

AMD Toolchain with SPACK

Micro Benchmarks/Synthetic

SPACK HPC Applications

Introduction

OpenFOAM® is a C++ toolbox for the development of customized numerical solvers and pre-/post-processing utilities for the solution of continuum mechanics problems, including computational fluid dynamics. OpenFOAM® is one of the leading CFD applications and is increasingly popular due to its features and open source availability.

Official Website for OpenFOAM®: https://www.openfoam.com/

Build OpenFOAM® Using Spack

Reference to add external packages to Spack: Build Customization (Adding external packages to Spack)

# Format For Building OpenFOAM
$ spack -d install -v openfoam@<Version> %aocc@<Version> target=<zen2/zen3> ^amdfftw@<Version> ^openmpi@<Version>
# Example: For building OpenFOAM 2012 with AOCC 3.2.0, AMDFFTW 3.1 and Open MPI
$ spack -d install -v -j 16 openfoam@2012 %aocc@3.2.0 target=zen3 ^amdfftw@3.1 ^openmpi@4.1.1
# Example: For building OpenFOAM 2012 with AOCC 3.1.0, AMDFFTW 3.0, and Open MPI
$ spack -d install -v -j 16 openfoam@2012 %aocc@3.1.0 target=zen3 ^amdfftw@3.0 ^openmpi@4.0.5
# Example: For building OpenFOAM 2012 with AOCC 3.0, AMDFFTW 3.0, and Open MPI
$ spack -d install -v -j 16 openfoam@2012 %aocc@3.0.0 target=zen3 ^amdfftw@3.0 ^adios2~fortran ^cgal@4.13 ^openmpi@4.0.3
# Example: For building OpenFOAM 2012 with AOCC 2.3, AMDFFTW 2.2, and Open MPI
$ spack -d install -v -j 16 openfoam@2012 %aocc@2.3.0 target=zen2 ^amdfftw@2.2 ^adios2~fortran ^cgal@4.13 ^openmpi@4.0.3
# Example: For building OpenFOAM 2012 with AOCC 2.2, AMDFFTW 2.2, and Open MPI
$ spack -d install -v -j 16 openfoam@2012 %aocc@2.2.0 target=zen2 ^amdfftw@2.2 ^adios2~fortran ^cgal@4.13 ^openmpi@4.0.3

Use any combination of the following components/applications and corresponding versions:

Component/Application Versions Applicable
OpenFOAM® 2012
AOCC 3.2.0, 3.1.0, 3.0.0, 2.3.0, 2.2.0
AOCL 3.1, 3.0, 2.2

Specifications and Dependencies

Symbol Meaning
-d To enable debug output
-v To enable verbose
@ To specify version number
-j 16 To enable parallel build
% To specify compiler
^amdfftw Use amdfftw version 2.2 for build
^adios2~fortran Use adios2 without Fortran for build
^cgal@4.13 Use cgal version 4.13 for build
^openmpi@4.0.3 Build with Open MPI

Running OpenFOAM®

While OpenFOAM® can be used for a variety of workloads, a very common benchmark uses the Motorbike data set. The following steps download and run OpenFOAM® using the MotorBike data set on the AMD EPYCTM 7742 processor model that has 128 cores (SMT OFF) per Node.  This example is applicable to both AMD 2nd and 3rd Gen EPYCTM processors.

The run script can be created from the following code block (runme.sh):

Setting Environment
# Load OpenFOAM module into environment
$ spack load openfoam@<Version> %aocc@<Version>
# Example loading OpenFOAM 2012 build with AOCC 3.2
$ spack load openfoam@2012 %aocc@3.2.0

 

Obtaining Benchmarks
# Obtaining Benchmark: motorBike 2M cells
$ wget http://openfoamwiki.net/images/6/62/Motorbike_bench_template.tar.gz

 

runme.sh
#!/bin/bash
tar -xzvf Motorbike_bench_template.tar.gz
cd $PWD/bench_template
sed -i '/#include "streamLines"/c\ ' basecase/system/controlDict
sed -i '/#include "wallBoundedStreamLines"/c\ ' basecase/system/controlDict
unset FOAM_SIGFPE
export FOAM_SIGFPE=false
# Prepare cases
which snappyHexMesh
for in 64 128 256; do
  d=run_$i
  echo "Prepare case ${d}..."
  cp -r basecase $d
   cd $d
  pwd
  if [ $i -eq 1 ]
  then
  mv Allmesh_serial Allmesh
  elif [ $i -gt 128 ]
  then
  sed -i "s|runParallel snappyHexMesh -overwrite|mpirun -np ${i} -mca btl    vader,self  --map-by hwthread -use-hwthread-cpus  snappyHexMesh -parallel    -overwrite > log.snappyHexMesh|" Allmesh
  fi
  sed -i "s/method.*/method scotch;/" system/decomposeParDict
  sed -i "s/numberOfSubdomains.*/numberOfSubdomains           ${i};/" system/decomposeParDict
 ./Allmesh
 cd ..
done
# Run cases
for in 64 128 256; do
 echo "Run for ${i}..."
 cd run_$i
 if [ $i -eq 1 ]
 then
 simpleFoam > log.simpleFoam 2>&1
 elif [ $i -gt 128 ]
 then
 mpirun -np ${i} --map-by hwthread  -use-hwthread-cpus  -mca btl vader,self simpleFoam -parallel > log.simpleFoam 2>&1
sed -i "s|mpirun -np ${i} --map-by hwthread -use-hwthread-cpus  snappyHexMesh -parallel -overwrite > log.snappyHexMesh|runParallel snappyHexMesh -overwrite|" Allmesh
 else
 mpirun -np ${i} --map-by core  simpleFoam -parallel > log.simpleFoam 2>&1
 fi
 cd ..
done
echo "# cores   Wall time (s):"
echo "------------------------"
for in 64 128 256; do
 echo $i `grep Execution run_${i}/log.simpleFoam | tail -n 1 | cut -d " " -f 3`
done