AMD Toolchain with SPACK

Micro Benchmarks/Synthetic

SPACK HPC Applications


The High Performance Conjugate Gradient (HPCG) benchmark is designed to be the successor of the HPL benchmark. The motivation to create a new benchmark was to produce a memory performance metric of HPC systems that better reflects the performance of common application workloads on a modern architecture.  HPC workloads are no longer limited by CPU performance alone but may also be sensitive to memory performance.  Examples of memory sensitivity are computational fluid dynamics applications, such as the open source OPENFOAM® application. Whereas HPL solves a system of linear equations, the HPCG benchmark concentrates on data access patterns.

A reference implementation of the HPCG benchmark can be found at

The following guide recommends parameters to create the HPCG binary for the system of interest.

Build HPCG using Spack

Reference to add external packages to Spack: Build Customization (Adding external packages to Spack)

# Format for Installing HPCG
$ spack -d install -v hpcg@<Version> %aocc@<Version> +openmp target=<zen2/zen3> ^openmpi@<Version>
# Example: For Building HPCG 3.1 with AOCC 3.2.0
$ spack -d install -v hpcg@3.1 %aocc@3.2.0 +openmp target=zen3 ^openmpi@4.1.1
# Example: For Building HPCG 3.1 with AOCC 3.1.0
$ spack -d install -v hpcg@3.1 %aocc@3.1.0 +openmp target=zen3 ^openmpi@4.0.5
# Example: For Building HPCG 3.1 with AOCC 3.0.0
$ spack -d install -v hpcg@3.1 %aocc@3.0.0 +openmp target=zen3 ^openmpi@4.0.3
# Example: For Building HPCG 3.1 with AOCC 2.3.0
$ spack -d install -v hpcg@3.1 %aocc@2.3.0 +openmp target=zen2 ^openmpi@4.0.3
# Example: For Building HPCG 3.1 with AOCC 2.2.0
$ spack -d install -v hpcg@3.1 %aocc@2.2.0 +openmp target=zen2 ^openmpi@4.0.3

Use any combination of below components/applications and its versions:

Component/Application Versions Applicable
HPCG 3.1
AOCC 3.2.0, 3.1.0, 3.0.0, 2.3.0, 2.2.0

Specifications and Dependencies

Symbol Meaning
-d To enable debug output
-v To enable verbose
@ To specify version number
% To specify compiler
+openmp To build with OPENMP enabled
^openmpi To build with Open MPI support

Running HPCG

HPCG must be configured to run from the hpcg.dat file.

An example hpcg.dat looks like this:

HPCG benchmark input file
[name of lab or company here]
192 192 192

For valid/official HPCG runs, there are two parameters that need to be set in the hpcg.dat file.  Using the example above:

  1. Problem size – (Line 3) A valid run must execute a problem size that is large enough that data arrays accessed in the CG iteration loop do not fit in the cache of the device in a way that would be unrealistic in a real application setting.  The problem size should be large enough to occupy at least 25% of main memory.
  2. Run time – (Line 4) HPCG can be run in just a few minutes from start to finish.  Our experience suggests 30 seconds is enough to produce results that don’t vary significantly with longer runs.  However, official runs must be at least 1800 seconds (30 minutes) as reported in the output file.

HPCG must be be loaded before running as follows with format and loading examples:

Setting Environment
# Format for loading HPCG build with AOCC
$ spack load hpcg@<Version> %aocc@<Version>
# Example : Load HPCG build with AOCC 3.2 module into environment
$ spack load hpcg %aocc@3.2.0
Run Command
export OMP_PROC_BIND=close
export OMP_PLACES=cores
# Running HPCG
$ mpirun -np 32 map-by ppr:2:l3cache:pe=4 -x OMP_NUM_THREADS=4 xhpcg