Navigation

Spack

AMD Toolchain with SPACK

Micro Benchmarks/Synthetic

SPACK HPC Applications

Introduction

The High Performance Conjugate Gradient (HPCG) benchmark is designed to be the successor of the HPL benchmark. The motivation to create a new benchmark was to produce a memory performance metric of HPC systems that better reflects the performance of common application workloads on a modern architecture.  HPC workloads are no longer limited by CPU performance alone but may also be sensitive to memory performance.  Examples of memory sensitivity are computational fluid dynamics applications, such as the open source OPENFOAM® application. Whereas HPL solves a system of linear equations, the HPCG benchmark concentrates on data access patterns.

A reference implementation of the HPCG benchmark can be found at https://www.hpcg-benchmark.org/

The following guide recommends parameters to create the HPCG binary for the system of interest.

Build HPCG using Spack

Reference to add external packages to Spack: Build Customization (Adding external packages to Spack)

# Format for Building HPCG
$ spack -d install -v hpcg@<Version Number> %aocc@<Version Number> +openmp cflags="CFLAGS" target=<zen2/zen3> ^openmpi@<Version Number>
# Example: For Building HPCG-3.1 with AOCC-3.0.0
$ spack -d install -v hpcg@3.1 %aocc@3.0.0 +openmp cflags="-O3" target=zen3 ^openmpi@4.0.3
# Example: For Building HPCG-3.1 with AOCC-2.3.0
$ spack -d install -v hpcg@3.1 %aocc@2.3.0 +openmp cflags="-O3" target=zen2 ^openmpi@4.0.3
# Example: For Building HPCG-3.1 with AOCC-2.2.0
$ spack -d install -v hpcg@3.1 %aocc@2.2.0 +openmp cflags="-O3" target=zen2 ^openmpi@4.0.3

Please use any combination of below components/Applications and its versions.

Component/Application Versions Applicable
HPCG 3.1
AOCC 3.0.0, 2.3.0, 2.2.0

Specifications and Dependencies

Symbol Meaning
-d To enable debug output
-v To enable verbose
@ To specify version number
% To specify compiler
+openmp To build with OPENMP enabled
cflags To add cflags to the Spack environment using command line

Running HPCG

HPCG must be configured to run from the hpcg.dat file.

An example hpcg.dat looks like this:

hpcg.dat
HPCG benchmark input file
[name of lab or company here]
192 192 192
1800

For valid/official HPCG runs, there are two parameters that need to be set in the hpcg.dat file.  Using the example above:

  1. Problem size – (Line 3) A valid run must execute a problem size that is large enough that data arrays accessed in the CG iteration loop do not fit in the cache of the device in a way that would be unrealistic in a real application setting.  The problem size should be large enough to occupy at least 25% of main memory.
  2. Run time – (Line 4) HPCG can be run in just a few minutes from start to finish.  Our experience suggests 30 seconds is enough to produce results that don’t vary significantly with longer runs.  However, official runs must be at least 1800 seconds (30 minutes) as reported in the output file.

HPCG must be be loaded before running as follows with format and loading examples:

Setting Environment
# Format for loading HPCG build with AOCC
$ spack load hpcg@<Version Number> %aocc@<Version Number>
# Example : Load HPCG build with AOCC-3.0 module into environment
$ spack load hpcg %aocc@3.0.0

HPCG can now be run as follows:

Running HPCG
export OMP_PROC_BIND=close
export OMP_PLACES=cores
# Running HPCG
$ mpirun -np 32  –map-by ppr:2:l3cache:pe=4 -x OMP_NUM_THREADS=4 xhpcg