Navigation

Spack

AMD Toolchain with SPACK

Micro Benchmarks/Synthetic

SPACK HPC Applications

Introduction

OpenFOAM® is a C++ toolbox for the development of customized numerical solvers and pre-/post-processing utilities for the solution of continuum mechanics problems, including computational fluid dynamics. OpenFOAM® is one of the leading CFD applications and is increasingly popular due to its features and open source availability.

OpenFOAM® official website: https://www.openfoam.com/

Build OpenFOAM® using Spack

Reference to add external packages to Spack: Build Customization (Adding external packages to Spack)

# Format For Building OpenFOAM
$ spack -d install -v openfoam@<Version Number> %aocc@<Version Number> target=<zen2/zen3> ^amdfftw@<Version Number> ^adios2~fortran ^cgal@<Version Number> ^openmpi@<Version Number> fabrics=auto
# Example: For building OpenFOAM 2012 with AOCC 3.0, AMDFFTW 3.0 and OpenMPI
$ spack -d install -v openfoam@2012 %aocc@3.0.0 target=zen3 ^amdfftw@3.0 ^adios2~fortran ^cgal@4.13 ^openmpi@4.0.3 fabrics=auto
# Example: For building OpenFOAM 2012 with AOCC 2.3, AMDFFTW 2.2 and OpenMPI
$ spack -d install -v openfoam@2012 %aocc@2.3.0 target=zen2 ^amdfftw@2.2 ^adios2~fortran ^cgal@4.13 ^openmpi@4.0.3 fabrics=auto
# Example: For building OpenFOAM 2012 with AOCC 2.2, AMDFFTW 2.2 and OpenMPI
$ spack -d install -v openfoam@2012 %aocc@2.2.0 target=zen2 ^amdfftw@2.2 ^adios2~fortran ^cgal@4.13 ^openmpi@4.0.3 fabrics=auto

Please use any combination of below components/Applications and its versions.

Component/Application Versions Applicable
OpenFOAM® 2012
AOCC 3.0.0, 2.3.0, 2.2.0
AOCL 3.0, 2.2

Specifications and Dependencies

Symbol Meaning
-d To enable debug output
-v To enable verbose
@ To specify version number
% To specify compiler
^amdfftw Use amdfftw version 2.2 for build
^adios2~fortran Use adios2 without fortran for build
^cgal@4.13 Use cgal version 4.13 for build
^openmpi@4.0.3 fabrics=auto Build with openmpi with variant fabrics=auto

Running OpenFOAM®

While OpenFOAM® can be used for a variety of workloads, a very common benchmark uses the Motorbike data set. The following steps download and run OpenFOAM® using the MotorBike data set on the AMD EPYC 7742 processor model that has 128 cores (SMT OFF) per Node.  This example is applicable to both AMD 2nd and 3rd Gen EPYC processors.

Note: The run script can be created from the code block (runme.sh) below.

Setting Environment
# Load OpenFOAM module into environment
$ spack load openfoam@<Version Number> %aocc@<Version Number>
# Example loading OpenFOAM 2012 build with AOCC 3.0
$ spack load openfoam@2012 %aocc@3.0.0

 

Obtaining Benchmarks
# Obtaining Benchmark: motorBike
$ wget http://openfoamwiki.net/images/6/62/Motorbike_bench_template.tar.gz

 

runme.sh
#!/bin/bash
tar -xzvf Motorbike_bench_template.tar.gz
cd $PWD/bench_template
sed -i '/#include "streamLines"/c\ ' basecase/system/controlDict
sed -i '/#include "wallBoundedStreamLines"/c\ ' basecase/system/controlDict
unset FOAM_SIGFPE
export FOAM_SIGFPE=false
# Prepare cases
which snappyHexMesh
for in 64 128 256; do
d=run_$i
echo "Prepare case ${d}..."
cp -r basecase $d
cd $d
pwd
if [ $i -eq 1 ]
then
mv Allmesh_serial Allmesh
elif [ $i -gt 128 ]
then
sed -i "s|runParallel snappyHexMesh -overwrite|mpirun -np ${i} -mca btl vader,self  --map-by hwthread -use-hwthread-cpus  snappyHexMesh -parallel -overwrite > log.snappyHexMesh|" Allmesh
fi
sed -i "s/method.*/method scotch;/" system/decomposeParDict
sed -i "s/numberOfSubdomains.*/numberOfSubdomains ${i};/" system/decomposeParDict
./Allmesh
cd ..
done
# Run cases
for in 64 128 256; do
echo "Run for ${i}..."
cd run_$i
if [ $i -eq 1 ]
then
simpleFoam > log.simpleFoam 2>&1
elif [ $i -gt 128 ]
then
mpirun -np ${i} --map-by hwthread  -use-hwthread-cpus  -mca btl vader,self simpleFoam -parallel > log.simpleFoam 2>&1
sed -i "s|mpirun -np ${i} --map-by hwthread -use-hwthread-cpus  snappyHexMesh -parallel -overwrite > log.snappyHexMesh|runParallel snappyHexMesh -overwrite|" Allmesh
else
mpirun -np ${i} --map-by core  simpleFoam -parallel > log.simpleFoam 2>&1
fi
cd ..
done
echo "# cores   Wall time (s):"
echo "------------------------"
for in 64 128 256; do
echo $i `grep Execution run_${i}/log.simpleFoam | tail -n 1 | cut -d " " -f 3`
done