HPL (High-Performance Linpack)

What HPL computes, why it's the standard whole-system stress and acceptance test, HPL.dat parameters (N, NB, P×Q), how to run it on DGX H100 with NVIDIA's container, how to read the output, and how to troubleshoot it. Includes a sizing calculator.

1. What HPL is

HPL (High-Performance Linpack) solves one huge dense system of linear equations, A·x = b, in double precision (FP64), spread across every process in the cluster. It uses LU factorization with partial pivoting, which is mostly giant matrix multiplications (DGEMM).

The matrix A is N × N and filled with random numbers. N is chosen so the matrix fills most of the available memory (GPU HBM for GPU HPL).
The matrix is cut into NB × NB blocks and spread over a P × Q grid of MPI processes (on DGX: usually one process per GPU).
At the end, HPL checks the answer (residual test). If the math came out wrong, the run is FAILED, no matter how fast it was.
The headline result is Gflops / Tflops / Pflops (FP64 floating-point operations per second). This number ranks systems on the TOP500 list.

HPL is a marathon at full sprint for the whole cluster: every GPU does heavy FP64 matrix math at maximum power for minutes to hours, while constantly swapping data over NVLink and InfiniBand. At the finish line, a referee checks that every calculation was correct (the residual check).

2. Why we run it in bring-up

Purpose	What HPL proves
Whole-system stress	GPUs at maximum compute and power and heat at the same time, plus HBM, NVLink, InfiniBand, and CPUs/MPI all busy together
Stability / burn-in	Repeated runs over hours or days reveal marginal hardware: GPUs or HBM failing under heat, power supply problems, flaky links
Correctness	The residual check catches silent data corruption: hardware that's fast but computes wrong answers
Performance acceptance	Measured Rmax vs expected (vendor reference / contract) shows the cluster delivers what was bought
Finding weak nodes	Single-node HPL on every node, compared side by side, exposes throttled or slow nodes
Facility validation	Near-maximum power draw tests PDUs, cooling, and airflow (a DGX H100 is about 10.2 kW max)

HPL is the final, top-of-stack test. Run it after DCGM diagnostics (GPU health), single-node NCCL (NVLink), perftest/ClusterKit (IB links), and multi-node nccl-tests. If HPL fails or is slow, those lower-level tests help pinpoint where.

HPL tells you that something is wrong (slow, unstable, wrong answer), but not exactly what. It's a system-level test. To isolate, go back down to DCGM, nvidia-smi, nccl-tests, and perftest.

3. The math you need

Term	Meaning	Formula / note
FLOPs in an HPL run	Work done	≈ `2/3 · N³ + 2 · N²` (dominated by the N³ term)
Rmax	Achieved performance (the HPL result)	FLOPs ÷ run time
Rpeak	Theoretical peak FP64	#GPUs × FP64 peak per GPU
Efficiency	How close you got to peak	`Rmax / Rpeak`
Matrix memory	Memory needed for A	`N² × 8 bytes` (FP64 = 8 bytes)

H100 numbers to know

	H100 SXM (per GPU)	DGX H100 (8 GPUs)
FP64 (vector)	~34 TFLOPS	~268 TFLOPS
FP64 Tensor Core (what HPL's DGEMM uses)	~67 TFLOPS	~536 TFLOPS
HBM3	80 GB	640 GB

Real-world GPU HPL efficiency against FP64 Tensor Core peak is often around 60–80%. Large multi-node systems tend toward the lower end because of communication, and well-tuned single nodes toward the higher end. Use NVIDIA's reference results for your exact system and container version as the acceptance target, not a generic percentage.

Rpeak = paper. Rmax = proof. Efficiency = proof ÷ paper.

4. HPL.dat: the parameters that matter

HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any)
6            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
266240       Ns                       <- problem size N
1            # of NBs
1024         NBs                      <- block size NB
0            PMAP process mapping (0=Row-,1=Column-major)
1            # of process grids (P x Q)
2            Ps                       <- P
4            Qs                       <- Q   (P x Q = 8 ranks = 8 GPUs)
16.0         threshold                <- residual pass threshold
1            # of panel fact
2            PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
4            NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
1            RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
3            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
1            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
64           swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
8            memory alignment in double (> 0)

Parameter	What it is	Rule of thumb
N	Matrix dimension	As large as memory allows (about 80–90% of total memory), and a multiple of NB. Bigger N = higher efficiency, but a longer run.
NB	Block size for distribution and computation	GPUs like large NB (hundreds to ~1024). CPUs use smaller (~100–400). Start from the vendor's sample files.
P × Q	Process grid	P × Q = number of MPI ranks (usually = number of GPUs). As square as possible with P ≤ Q.
threshold	Residual pass limit	16.0 (standard; don't change it to "pass")
PFACT / RFACT / NBMIN / NDIV	Panel factorization algorithm variants	Netlib (CPU) HPL tuning. Ignored by NVIDIA's GPU HPL.
BCAST	Broadcast algorithm for panels	Netlib tuning. Ignored by NVIDIA HPL, which uses NCCL/NVSHMEM (see `HPL_P2P_AS_BCAST`)
DEPTH	Lookahead (overlap of communication and compute)	Netlib tuning. Ignored by NVIDIA HPL.

Every HPL.dat line, and what NVIDIA HPL does with it

NVIDIA HPL reads the standard Netlib HPL.dat format, but only some lines still matter. NVIDIA's docs list what it ignores: broadcast, panel factorization, look-ahead, L1 layout, U layout, equilibration, and memory alignment.

Line	Netlib meaning	NVIDIA GPU HPL
Lines 1–2	Free-text header	Ignored (keep them; the file is read by line position)
output file name	Name used if device out is a file	Used
device out	`6` = stdout, `7` = stderr, anything else = the named file	Used
# of problem sizes / Ns	How many N values, then the list	Used (N)
# of NBs / NBs	How many block sizes, then the list	Used (NB; NVIDIA sample files use large values such as 576 or 1024)
PMAP	Rank-to-grid mapping: 0 = row-major, 1 = column-major	Used
# of grids / Ps / Qs	Process grids to try	Used (P × Q = ranks = GPUs)
threshold	Residual pass limit (16.0); a negative value skips the check	Used. Never loosen it to make a run pass.
PFACTs, NBMINs, NDIVs, RFACTs	Panel factorization variants and recursion	Ignored
BCASTs	Panel broadcast topology (0 = 1rg … 5 = LnM)	Ignored
DEPTHs	Lookahead depth	Ignored
SWAP, swapping threshold	Row-swap algorithm (0 = binary exchange, 1 = long, 2 = mix) and its cut-over	Not used for tuning: row swaps go through NCCL or NVSHMEM (`HPL_NVSHMEM_SWAP`)
L1 in, U in	Transposed or not	Ignored
Equilibration	Equilibrate during the swap	Ignored
memory alignment	Alignment in doubles	Ignored

Reading the T/V column (Netlib format, for example WR03L2R8): W wall time, R/C = PMAP, then lookahead DEPTH (0), BCAST (3), RFACT (L/C/R), NDIV (2), PFACT (R), NBMIN (8). With NVIDIA HPL these characters only echo what's in the file, because those lines are ignored.

Multiple values on one line (several Ns, NBs, or grids) make HPL run every combination. That's good for a tuning sweep, bad for a quick test. For burn-in, use one well-chosen configuration and repeat it.

5. Choosing N, NB, and P×Q

N from memory

The matrix takes N² × 8 bytes. Solve for N using a fill fraction f (for example 0.85) of total memory M:

N ≈ sqrt( f × M / 8 )        then round DOWN to a multiple of NB

Example: one DGX H100 (GPU HPL uses HBM): M = 8 × 80 GB = 640 GB
N ≈ sqrt(0.85 × 640e9 / 8) ≈ sqrt(6.8e10) ≈ 260,768  → round to a multiple of NB=1024 → 260,096

For multiple nodes, total memory grows linearly, so N grows with the square root of the node count (4× nodes → 2× N).

P × Q

Ranks = nodes × GPUs per node (for example 4 nodes × 8 = 32 ranks).
Pick the factor pair that's closest to square, with P ≤ Q: 8 → 2×4. 16 → 4×4. 32 → 4×8. 64 → 8×8.
Very flat grids (1×32) increase communication imbalance and usually perform worse.

N = fill memory. NB = big blocks for GPUs. P×Q = ranks, square-ish, P ≤ Q.

6. Sizing calculator

Nodes

GPUs per node

Memory per GPU (GB)

Memory fill fraction

FP64 peak per GPU (TFLOPS)

Assumed efficiency (%)

7. Running HPL on DGX H100

NVIDIA ships GPU-optimized HPL in the NVIDIA HPC-Benchmarks container on NGC (nvcr.io/nvidia/hpc-benchmarks:<tag>). It includes HPL, HPL-MxP, HPCG, and STREAM, plus sample HPL.dat files for common GPU counts.

# Slurm + Pyxis/Enroot (NVIDIA's documented form): 8 DGX H100 nodes x 8 GPUs = 64 ranks
CONT='nvcr.io#nvidia/hpc-benchmarks:<tag>'
srun -N 8 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix \
     --container-image="${CONT}" \
     ./hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-64GPUs.dat

# Single node (8 GPUs): the per-node health/performance check, with a custom HPL.dat
srun -N 1 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix \
     --container-image="${CONT}" --container-mounts="$PWD:/my-dat" \
     ./hpl.sh --dat /my-dat/HPL-8GPUs.dat

# Copy the sample HPL.dat and Slurm files out of the container to edit them
srun -N 1 --cpu-bind=none --mpi=pmix --container-image="${CONT}" \
     --container-mounts="$PWD:/home_pwd" --pty bash          # then cp /workspace/hpl-linux-x86_64/sample-dat/* /home_pwd/

# Without Slurm: Docker on one node
docker run --gpus all --shm-size=1g nvcr.io/nvidia/hpc-benchmarks:<tag> \
     mpirun --bind-to none -np 8 ./hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-8GPUs.dat

--cpu-bind=none leaves CPU pinning to hpl.sh (its --cpu-affinity); Slurm's own binding would fight it. --mpi=pmix lets Slurm launch the MPI ranks itself.
Don't nest mpirun inside srun --mpi=pmix: either Slurm launches the ranks or mpirun does, not both.
The package (and the container's /workspace) holds hpl.sh, hpl-mxp.sh, hpcg.sh, stream-gpu-test.sh, and per-benchmark folders such as hpl-linux-x86_64/ with xhpl, sample-dat/, sample-slurm/, and README, RUNNING, and TUNING guides. Arm systems: hpl-linux-aarch64-gpu/ (Grace Hopper / Grace Blackwell) and hpl-aarch64.sh for Grace CPU only. Newer packages split into cuda12/ and cuda13/ trees.

Script names, sample file paths, and affinity options (CPU/GPU/memory/NIC binding) vary between container releases. Check the container's README for your tag. For the exam: use NVIDIA's HPC-Benchmarks container, one MPI rank per GPU, P×Q = total GPUs, and correct CPU/NIC affinity.

Newer Pyxis releases accept the Docker-style nvcr.io/nvidia/... path shown above. If yours rejects it, use Enroot's nvcr.io#nvidia/hpc-benchmarks:<tag> form. For NGC login, Enroot credentials, and pre-staging images, see NGC mastery.

Prerequisites

Healthy GPUs (DCGM diag), Fabric Manager running, NVLink OK (single-node nccl-tests).
Working IB with GPUDirect RDMA for multi-node runs (perftest, multi-node nccl-tests).
Slurm + containers working, with IB devices available inside the container.
Monitoring running during the test (DCGM / nvidia-smi dmon for clocks, power, temperatures, throttling, Xids).

hpl.sh options

hpl.sh wraps xhpl: it sets up the libraries, binds each rank to its GPU, CPU cores, NUMA memory, and NIC, then starts xhpl. NVIDIA HPL expects one GPU per MPI rank, so the number of ranks must equal the number of GPUs.

Option	What it does	When it matters
`--dat <file>`	Path to HPL.dat. Required.	Always. Use a sample file for your GPU count, or your own sized file.
`--gpu-affinity <list>`	Colon-separated GPU index per local rank (`0:1:2:3:4:5:6:7`)	Non-default GPU order, or only some GPUs. Pair with the CPU and memory lists.
`--cpu-affinity <list>`	Colon-separated CPU core ranges per local rank	Pin each rank to cores on its GPU's socket. Wrong pinning = cross-socket traffic and lower Gflops.
`--mem-affinity <list>`	Colon-separated NUMA node per local rank	Keep host buffers on the GPU's socket. Strongly recommended with out-of-core mode.
`--ucx-affinity <list>`	Colon-separated UCX (network) device per local rank, for example `mlx5_0:mlx5_1:…`	Multi-node: give each GPU its nearest NIC (rail).
`--ucx-tls <tls>`	UCX transports to use (sets `UCX_TLS`)	Forcing or excluding a transport while debugging MPI traffic.
`--no-multinode`	Flags for no-network execution	Single-node runs on a node with no working fabric, or to take the network out of the picture.
`--cuda-compat`	Manually enables CUDA forward compatibility	The container's CUDA is newer than the host driver supports (datacenter forward-compat package installed).
`--exec-name <file>`	Use a different HPL executable	Testing another `xhpl` build.

# NVIDIA's documented affinity for DGX H100 and DGX B200 (8 GPUs, 2 sockets)
./hpl.sh --dat HPL-8GPUs.dat \
   --mem-affinity 0:0:0:0:1:1:1:1 \
   --cpu-affinity 0-13:14-27:28-41:42-55:56-69:70-83:84-97:98-111
# DGX A100 (GPUs aren't in socket order):
#  --mem-affinity 2:3:0:1:6:7:4:5 --cpu-affinity 32-47:48-63:0-15:16-31:96-111:112-127:64-79:80-95
# Grace Hopper x4, one node:
#  --cpu-affinity 0-71:72-143:144-215:216-287 --mem-affinity 0:1:2:3

Each affinity list has one entry per local rank, separated by colons. Entry i is for local rank i. GPUs 0–3 on socket 0 and 4–7 on socket 1 gives 0:0:0:0:1:1:1:1.

7b. NVIDIA HPL environment variables

Set them before hpl.sh runs (exported in the batch script, or with srun --export=ALL,VAR=value). The defaults are good on DGX systems: change one at a time and compare Gflops. Defaults below are from NVIDIA's HPC-Benchmarks documentation; check the RUNNING and TUNING guides in your container tag.

Variable	Default	Values	What it does / when it matters
`HPL_P2P_AS_BCAST`	1	0 NCCL bcast, 1 NCCL send/recv, 2 CUDA-aware MPI, 3 host MPI, 4 NVSHMEM	Communication library for the final solve step. Try 3 (host MPI) to rule out NCCL; 2 needs CUDA-aware MPI.
`HPL_USE_NVSHMEM`	1	1 on, 0 off	NVSHMEM support. Set 0 if NVSHMEM fails to initialize on your fabric or MPI, to see whether NVSHMEM is the problem.
`HPL_NVSHMEM_INIT`	0	0 via MPI, 1 via unique ID (UID)	How NVSHMEM bootstraps. Set 1 where MPI bootstrap is problematic.
`HPL_FCT_COMM_POLICY`	0	0 NVSHMEM, 1 host MPI	Communication library for panel factorization.
`HPL_NVSHMEM_SWAP`	0	1 on, 0 off	Row swaps with NVSHMEM instead of NCCL (default).
`HPL_CHUNK_SIZE_NBS`	16	>0	How many NB-sized blocks are grouped per computation.
`HPL_DIST_TRSM_FLAG`	1	1 on, 0 off	Runs the solve (TRSM) in parallel across ranks instead of only on the owners.
`HPL_CTA_PER_FCT`	16	>0	CUDA thread blocks (CTAs) used for factorization.
`HPL_FCT_CHUNK_SIZE`	32	32, 64, 128	Chunk size for factorization.
`HPL_ALLOC_HUGEPAGES`	0	1 on, 0 off	2 MB hugepages for host allocations via `madvise`. Only works if `/sys/kernel/mm/transparent_hugepage/enabled` is `madvise`.
`WARMUP_END_PROG`	5	−1 to 100	Runs a warm-up pass of the main loop first and stops it at that % progress.
`TEST_LOOPS`	1	>0	Runs the main loop that many times. Handy for burn-in: repeated solves in one job.
`HPL_CUSOLVER_MP_TESTS`	1	1 on, 0 off	Runs component tests first (GEMMs, communication). Useful diagnostics; set 0 to start the solve sooner.
`HPL_CUSOLVER_MP_TESTS_GEMM_ITERS`	128	>0	Repeat GEMM calls in those tests.
`HPL_WARMUP_MPI`	0	1 on, 0 off	Makes every MPI call once before the main loop so MPI sets up its internal structures. Can smooth first-iteration slowness at scale.

# Example: three back-to-back solves in one burn-in job, component tests skipped
export TEST_LOOPS=3 HPL_CUSOLVER_MP_TESTS=0
srun -N 8 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix --container-image="${CONT}" \
     ./hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-64GPUs.dat

Out-of-core mode (HPL_OOC_*)

An opt-in mode for matrices larger than GPU memory: the part that doesn't fit stays in host (CPU) memory and streams to the GPUs. The default is in-core. Enable it and raise N; enabling it alone changes nothing. Speed depends on the host-to-GPU link.

Variable	Default	What it does
`HPL_OOC_MODE`	0	1 enables out-of-core mode
`HPL_OOC_MAX_GPU_MEM`	−1 (no limit)	Cap on GPU memory used by out-of-core mode, in GiB
`HPL_OOC_TILE_M`	4096	Row blocking factor
`HPL_OOC_TILE_N`	4096	Column blocking factor
`HPL_OOC_NUM_STREAMS`	3	CUDA streams for out-of-core transfers
`HPL_OOC_SAFE_SIZE`	3.0 GiB (2.0 in older releases)	GPU memory left for the driver. Raise it if out-of-core runs hit GPU out-of-memory.

How much host memory: about 6–16 GiB per GPU on PCIe systems such as x86 DGX; 50–90 GiB on Grace Hopper and Grace Blackwell, whose NVLink-C2C CPU-GPU link is much faster.
Sizing N: take the largest per-GPU memory your in-core runs used, add the host target, and work N out from that total (N² × 8 bytes across all ranks).
Pass the CPU, GPU, and memory affinity options to hpl.sh: out-of-core traffic must stay on the right socket.

HPL_OOC_MODE=1 HPL_OOC_SAFE_SIZE=4.0 ./hpl.sh --dat HPL-ooc.dat \
   --mem-affinity 0:0:0:0:1:1:1:1 --cpu-affinity 0-13:14-27:28-41:42-55:56-69:70-83:84-97:98-111

FP64 emulation on Blackwell (Ozaki schemes)

Opt-in on Blackwell GPUs: DGEMM is emulated with INT8 tensor-core math using the Ozaki schemes, instead of native FP64. The default stays native FP64.

Variable	Default	What it does
`HPL_EMULATE_DOUBLE_PRECISION`	0	0 native FP64, 1 DGEMM with Ozaki-I, 2 DGEMM with Ozaki-II
`HPL_DOUBLE_PRECISION_EMULATION_MANTISSA_BIT_COUNT`	53	Mantissa bits for Ozaki-I (includes FP64's implicit bit); only read in mode 1. Slices = ceil((bits + 1) / 8).
`HPL_DOUBLE_PRECISION_EMULATION_NUM_MODULI`	15	Number of moduli for Ozaki-II (1–20); only read in mode 2. Marked experimental.

With Ozaki-II, NVIDIA recommends a grid with P ≥ Q, the opposite of the usual P ≤ Q rule. An emulated result isn't a native-FP64 result: report which mode you used, and keep the residual check PASSED.

Package, library, and MPI environment variables

Each run script sources hpc-benchmarks-gpu-env.sh (or -cpu-env.sh on Grace CPU), which puts the bundled libraries on LD_LIBRARY_PATH. The scripts use relative paths, so don't move them out of the package.

Variable	What it does / when it matters
`NCCL_PATH`, `NVSHMEM_PATH`	Use your own NCCL or NVSHMEM instead of the bundled one, for example `export NCCL_PATH=/opt/nccl-<ver>/lib`.
`NVPL_BLAS_PATH`, `NVPL_LAPACK_PATH`, `NVPL_SPARSE_PATH`	Same for the NVPL math libraries (Arm only).
`NCCL_`, `UCX_`	HPL's GPU communication runs on NCCL and its MPI on UCX, so the usual `NCCL_DEBUG=INFO`, `NCCL_IB_HCA`, `UCX_NET_DEVICES` apply. See the NCCL environment variables.
`NVSHMEM_REMOTE_TRANSPORT=libfabric`, `NVSHMEM_LIBFABRIC_PROVIDER=cxi`, `NVSHMEM_DISABLE_CUDA_VMM=1`, `MPICH_NO_BUFFER_ALIAS_CHECK=1`	HPE/Cray (Slingshot, MPICH-compatible MPI) only; set by default in the env script.
`MPICH_GPU_SUPPORT_ENABLED=1` plus `LD_PRELOAD="<path>/libmpi_gtl_cuda.so $LD_PRELOAD"`	CUDA-aware Cray MPICH, needed for `HPL_P2P_AS_BCAST=2`. `LD_PRELOAD` is space-separated, not colon-separated.

The applications link to libmpi.so. If your MPI only ships libmpi.so.12 (or similar), create a libmpi.so symlink. On systems without NVIDIA InfiniBand or Spectrum-X, NCCL needs the AWS OFI NCCL plugin, which the package doesn't include.

HPL-MxP options (hpl-mxp.sh)

HPL-MxP takes its whole configuration as command-line options, not an HPL.dat file. It runs xhpl_mxp.

./hpl-mxp.sh --n 280000 --nb 2048 --nprow 2 --npcol 4 --nporder row \
   --gpu-affinity 0:1:2:3:4:5:6:7      # one DGX H100 (8 ranks)

Option	Default	What it does
`--n`, `--nb`	required	Matrix size N and block size NB
`--nprow`, `--npcol`, `--nporder row\|column`	required	Process grid (P, Q) and its layout
`--gpu-affinity`	required	GPU per local rank
`--cpu-affinity`, `--mem-affinity`, `--ucx-affinity`, `--ucx-tls`, `--exec-name`	optional	As in `hpl.sh`
`--sloppy-type`	FP16	Low precision used for the LU: `FP4`, `FP8` (or `1`), `FP16` (or `2`)
`--tolerance`	1e-12	Harness tolerance for the refined result
`--test-loop`	1	Number of test loops
`--use-mpi-panel-broadcast`	1	% of steps that broadcast panels over MPI (30 = first 30%); 0 = NCCL only. Values >0 use CUDA-aware MPI.
`--mpi-use-mpi`, `--use-host-mpi`	0 / off	Fall back to `MPI_Bcast`; don't use CUDA-aware MPI
`--u-panel-chunk-nbs`	8	U panel chunk size in NBs. Known limit: `((N / NB) / npcol) / u-panel-chunk-nbs < 20`.
`--Anq-device`, `--fill-device`, `--fill-device-buffer-size`	0, 0, 3048 MB	How many FP64 matrix columns live on the GPU; fill the GPU (overrides Anq) and leave that buffer free
`--cuda-host-register-step`	2048	FP64 columns pinned (`cudaHostRegister`) at a time
`--preset-gemm-kernel`	0	0 none, 80 = SM80 preset GEMM kernel
`--prioritize-trsm`, `--prioritize-factorization`, `--use-separate-stream-for-gemm`	0, 0, 1	Scheduling of GEMMs relative to TRSMs and factorizations
`--call-dgemv-with-multiple-threads`	0	Rows per host thread for DGEMV (0 = one thread)
`--monitor-gpu`, `--monitor-gpu-interval`	0, 1 s	Monitor GPUs during the run
`--monitor-gpu-clock-warning`, `-power-warning`, `-temp-warning`, `-pcie-width-warning`, `-pcie-gen-warning`	0 (off)	Warn when clocks drop below, power or temperature rise above, or the PCIe link trains below a value. Useful in burn-in to catch a throttling or degraded GPU.
`--skip-tests`, `--gemm-iterations`	0, 100	Skip the internal tests; GEMM iterations in them
`-h`, `--help`		Print all options for your version

Sources: NVIDIA HPC Benchmarks documentation (NVIDIA HPL Benchmark, NVIDIA HPL-MxP Benchmark, Overview, Release Notes) and the NGC hpc-benchmarks container page. Options and defaults change between releases: ./hpl-mxp.sh --help and the container's README, RUNNING, and TUNING files are authoritative for your tag. HPCG and STREAM have their own scripts (hpcg.sh, stream-gpu-test.sh).

8. Reading the output

================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR03L2R8      260096  1024     2     4             27.10             4.335e+05
HPL_pdgesv() start time ...
--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=   0.0021375 ...... PASSED
================================================================================

Field	Meaning
T/V	Encodes the algorithm variant used (factorization, broadcast, etc.)
N, NB, P, Q	The configuration this line ran
Time	Seconds for the solve
Gflops	Rmax for this run. `4.335e+05` Gflops = 433.5 TFLOPS (≈81% of 536 TFLOPS peak in this illustration).
Residual line	Must be below the threshold (16.0) → PASSED. Otherwise FAILED.

A run only counts if the residual check PASSED. A FAILED residual means the computation produced wrong numbers: suspect hardware (GPU/HBM errors, overheating, unstable clocks, memory corruption), not tuning parameters. Drain the node and run DCGM diag level 3/4, check ECC/Xids.

Read two things: "how fast?" (Gflops vs expected) and "how correct?" (PASSED). Fast and FAILED = broken.

9. Validation strategy

Stage	How	Looking for
1. Single-node sweep	Run the same single-node HPL (8 GPUs) on every node	All PASSED. Gflops within a tight band (for example a few %). Outliers = throttling, bad GPU, wrong config.
2. Small multi-node	2, 4, 8 nodes	Near-linear scaling of Rmax. Drops point to network/affinity problems.
3. Full-system run	All nodes, N sized for total memory	Rmax and efficiency vs target. PASSED.
4. Burn-in	Repeat HPL (often mixed with other stress such as DCGM diag -r 4, NCCL, real training) for hours to days	No failures, no Xids/ECC growth, no hangs, stable Gflops (no thermal drift)

One node slower than the rest drags down multi-node HPL, because everyone waits for the slowest rank. That's why the single-node sweep comes first: find and fix outliers before running at scale.

10. HPL vs related benchmarks and tests

Test	Measures	Bound by	Typical % of peak	Used for
HPL	Dense FP64 linear solve	Compute (FP64 Tensor Cores)	High (~60–80%)	TOP500, stress, acceptance
HPL-MxP (formerly HPL-AI)	Same problem with mixed precision (low-precision LU + iterative refinement to FP64 accuracy)	Low-precision Tensor Core compute	Much higher "effective" FLOPS	AI-relevant compute capability. HPL-MxP list.
HPCG	Sparse conjugate gradient	Memory bandwidth and network latency	Very low (a few %)	Realistic HPC application behavior
STREAM	Memory bandwidth	Memory	vs memory BW peak	Memory subsystem check
nccl-tests	Collective communication bandwidth	NVLink / IB	busbw vs link speed	Interconnect validation
DCGM diag	GPU hardware health	n/a (pass/fail)	n/a	Per-node hardware screening
Real training (e.g. NeMo/Megatron)	Actual AI workload throughput	Everything	MFU	Final user-facing validation

HPL is FP64. AI training mostly uses BF16/FP8. A great HPL score proves the system is healthy and stable under full load, but it doesn't directly measure AI training throughput. HPL-MxP and real training workloads are closer to that.

11. What affects HPL performance

Factor	Effect	Check
N too small	Low efficiency (communication and overhead dominate)	Use ~80–90% memory
N too large	Out of memory / crash	Reduce N
Bad P×Q shape	Communication imbalance	Square-ish, P ≤ Q
Power / thermal throttling	One node or GPU slower, or performance falls over time	`nvidia-smi -q -d PERFORMANCE` (throttle reasons), `nvidia-smi dmon`, DCGM, inlet temperatures
Power limit set low	Lower clocks	`nvidia-smi -q -d POWER`
CPU/NIC/GPU affinity wrong	Cross-socket traffic, slow communication	`hpl.sh --cpu-affinity / --mem-affinity / --ucx-affinity` with `srun --cpu-bind=none` (see hpl.sh options)
Tuned the wrong HPL.dat lines	No change at all	NVIDIA HPL ignores BCAST, PFACT/RFACT, DEPTH, and friends. Tune N, NB, P×Q, affinity, and the HPL_* variables.
GPUDirect RDMA not working	Poor multi-node scaling	`nvidia-peermem`, NCCL/UCX logs
Degraded IB link or slow node	The whole job slows to the slowest link/node	ClusterKit, perftest, single-node sweep
Degraded NVLink	Single-node HPL low	`topo -m` (NV18), FM, Xid 74
Wrong container / old software	Below reference	Use the current NVIDIA HPC-Benchmarks container

12. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

srun -N 8 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix --container-image="${CONT}" ./hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-64GPUs.dat

64-GPU HPL across 8 nodes

================================================================================
HPL-NVIDIA 24.09.0  -- NVIDIA accelerated HPL benchmark -- NVIDIA
================================================================================
...
The following parameter values will be used:

N      :  736256
NB     :    1024
PMAP   : Row-major process mapping
P      :       8
Q      :       8
...

- Computational tests pass if scaled residuals are less than 16.0
...
 Prog= 1.21%  N_left= 733184  Time= 0.98  Time_left= 79.70  iGF= 3248451.20  GF= 3285552.80  iGF_per= 50757.05  GF_per= 51336.76
 Prog= 50.07%  N_left= 583680  Time= 40.40  Time_left= 40.28  iGF= 3337495.04  GF= 3307813.76  iGF_per= 52148.36  GF_per= 51684.59
 Prog= 99.91%  N_left= 70656  Time= 80.61  Time_left= 0.07  iGF= 3172599.04  GF= 3266589.76  iGF_per= 49571.86  GF_per= 51040.46
...
================================================================================
T/V                N    NB     P     Q               Time                 Gflops (   per GPU)
--------------------------------------------------------------------------------
WR03L2R8      736256  1024     8     8              80.68              3.298e+06 ( 5.153e+04)
HPL_pdgesv() start time Tue Nov 10 12:10:04 2026

HPL_pdgesv() end time   Tue Nov 10 12:11:25 2026

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0018243 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
...

What to look at: Check that P x Q (8 x 8) equals the 64 ranks and the residual says PASSED, then read Gflops as Rmax: about 3.3 PFLOPS, or about 51.5 TFLOPS per GPU here. If per-GPU Gflops is well below the single-node figure, suspect the network or affinity before the GPUs.

srun -N 1 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix --container-image="${CONT}" --container-mounts="$PWD:/my-dat" ./hpl.sh --dat /my-dat/HPL-8GPUs.dat

Single-node HPL health check

================================================================================
HPL-NVIDIA 24.09.0  -- NVIDIA accelerated HPL benchmark -- NVIDIA
================================================================================
The following parameter values will be used:

N      :  260096
NB     :    1024
PMAP   : Row-major process mapping
P      :       2
Q      :       4
PFACT  :    Right
NBMIN  :       4
NDIV   :       2
RFACT  :    Crout
BCAST  :   2ringM
DEPTH  :       1
SWAP   : Mix (threshold = 64)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

HPL-NVIDIA ignores the following parameters from input file:
  * Broadcast parameters
  * Panel factorization parameters
  ...
...
 Prog= 2.31%  N_left= 258048  Time= 0.63  Time_left= 26.47  iGF= 426997.50  GF= 431874.38  iGF_per= 53374.69  GF_per= 53984.30
 Prog= 99.89%  N_left= 26624  Time= 27.07  Time_left= 0.03  iGF= 438702.00  GF= 434800.50  iGF_per= 54837.75  GF_per= 54350.06
...
================================================================================
T/V                N    NB     P     Q               Time                 Gflops (   per GPU)
--------------------------------------------------------------------------------
WR03L2R8      260096  1024     2     4              27.10              4.335e+05 ( 5.419e+04)
HPL_pdgesv() start time Tue Nov 10 11:02:11 2026

HPL_pdgesv() end time   Tue Nov 10 11:02:38 2026

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0021375 ...... PASSED

What to look at: The parameter echo should match your HPL.dat (N 260096, NB 1024, P 2, Q 4), and NVIDIA HPL tells you which lines it ignores. Rmax is about 433 TFLOPS (about 54 TFLOPS per GPU); a node more than a few percent below its peers, or any FAILED residual, needs investigation.

srun -N 1 --cpu-bind=none --mpi=pmix --container-image="${CONT}" --container-mounts="$PWD:/home_pwd" --pty bashShell in container to copy HPL.dat

pyxis: importing docker image: nvcr.io#nvidia/hpc-benchmarks:<tag>
pyxis: imported docker image: nvcr.io#nvidia/hpc-benchmarks:<tag>
root@dgx01:/workspace# ls /workspace/hpl-linux-x86_64/sample-dat/
HPL-1GPU.dat    HPL-16GPUs.dat  HPL-2GPUs.dat   HPL-32GPUs.dat  HPL-4GPUs.dat
HPL-64GPUs.dat  HPL-8GPUs.dat   ...
root@dgx01:/workspace# cp /workspace/hpl-linux-x86_64/sample-dat/* /home_pwd/
root@dgx01:/workspace# exit

What to look at: The pyxis lines confirm the image was imported, and the sample-dat directory holds one starting HPL.dat per GPU count. Copy them to the mounted directory, edit N, NB and P x Q there, and pass the edited file with --dat.

docker run --gpus all --shm-size=1g nvcr.io/nvidia/hpc-benchmarks:<tag> mpirun --bind-to none -np 8 ./hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-8GPUs.dat

Single-node HPL with Docker


=========================================================
================= NVIDIA HPC Benchmarks =================
=========================================================

NVIDIA Release 24.09
Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
...
WARNING: No InfiniBand devices detected.
         Multi-node communication performance may be reduced.
         Ensure /dev/infiniband is mounted to this container.

================================================================================
HPL-NVIDIA 24.09.0  -- NVIDIA accelerated HPL benchmark -- NVIDIA
================================================================================
...
================================================================================
T/V                N    NB     P     Q               Time                 Gflops (   per GPU)
--------------------------------------------------------------------------------
WR03L2R8      260096  1024     2     4              27.14              4.329e+05 ( 5.411e+04)
...
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0020911 ...... PASSED
...

What to look at: The InfiniBand warning is expected for a single-node Docker run without /dev/infiniband and does not matter here; for multi-node runs it would. The result should match the srun single-node run within a percent or two, with PASSED.

./hpl.sh --dat HPL-8GPUs.dat --mem-affinity 0:0:0:0:1:1:1:1 --cpu-affinity 0-13:14-27:28-41:42-55:56-69:70-83:84-97:98-111

Pin each rank's CPU cores and NUMA

...
HPL-NVIDIA settings from environment variables:
 monitor_gpu from environment variable 0
 warmup_end_prog from environment variable 5.0
 test_loops from environment variable 1

Device info:
 Peak clock frequency 1980 MHz
 SM 90
 Number of SMs 132
 Total memory available 79.11 GB
...
Per-Process Host Memory Estimate: 0.00 GB (MAX) 0.00 GB (MIN)
Per-Process Device Memory Estimate: 67.94 GB (MAX) 67.94 GB (MIN)
...
 **** GEMM - cublas ****
avg time = 46.12 ms, avg = 49172.36. min = 48803.27 [rank 5, host dgx01, gpuID 0000:C3:00.0], max = 49521.08 GFLOPS
...
T/V                N    NB     P     Q               Time                 Gflops (   per GPU)
--------------------------------------------------------------------------------
WR03L2R8      260096  1024     2     4              27.08              4.338e+05 ( 5.423e+04)
...
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0019876 ...... PASSED

What to look at: Device info should show SM 90 and about 79 GB per H100, and the device memory estimate should leave a few GB free. The GEMM component test min/max names the slowest rank and GPU bus ID; with wrong affinity (ranks on the far socket) Gflops drop and vary between runs.

export TEST_LOOPS=3 HPL_CUSOLVER_MP_TESTS=0; srun -N 8 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix --container-image="${CONT}" ./hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-64GPUs.dat

Three back-to-back solves for burn-in

...
HPL-NVIDIA settings from environment variables:
 monitor_gpu from environment variable 0
 warmup_end_prog from environment variable 5.0
 test_loops from environment variable 3
 hpl_cusolver_mp_tests from environment variable 0
...
================================================================================
T/V                N    NB     P     Q               Time                 Gflops (   per GPU)
--------------------------------------------------------------------------------
WR03L2R8      736256  1024     8     8              80.68              3.298e+06 ( 5.153e+04)
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0018243 ...... PASSED
...
WR03L2R8      736256  1024     8     8              80.91              3.288e+06 ( 5.138e+04)
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0017962 ...... PASSED
...
WR03L2R8      736256  1024     8     8              81.07              3.282e+06 ( 5.128e+04)
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0018511 ...... PASSED
================================================================================

Finished      3 tests with the following results:
              3 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

End of Tests.
================================================================================

What to look at: test_loops 3 confirms the setting reached every rank, and the 'Testing HPL components' block is skipped. All three solves must PASS, and Gflops should stay within about 1%; a steady fall from loop to loop points to thermal throttling.

HPL_OOC_MODE=1 HPL_OOC_SAFE_SIZE=4.0 ./hpl.sh --dat HPL-ooc.dat --mem-affinity 0:0:0:0:1:1:1:1 --cpu-affinity 0-13:14-27:28-41:42-55:56-69:70-83:84-97:98-111

Out-of-core run larger than HBM

...
HPL-NVIDIA settings from environment variables:
 test_loops from environment variable 1
 hpl_ooc_mode from environment variable 1
 hpl_ooc_safe_size from environment variable 4.0
...
Per-Process Host Memory Estimate: 61.12 GB (MAX) 61.12 GB (MIN)
Per-Process Device Memory Estimate: 74.81 GB (MAX) 74.81 GB (MIN)
...
================================================================================
T/V                N    NB     P     Q               Time                 Gflops (   per GPU)
--------------------------------------------------------------------------------
WR03L2R8      368640  1024     2     4              90.51              3.690e+05 ( 4.612e+04)
...
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0016530 ...... PASSED

What to look at: N is larger than HBM alone can hold, so part of each rank's matrix lives in host memory (the non-zero host estimate). Expect PASSED but lower per-GPU Gflops than the in-core run; CUDA out-of-memory errors mean HPL_OOC_SAFE_SIZE is too small.

./hpl-mxp.sh --n 280000 --nb 2048 --nprow 2 --npcol 4 --nporder row --gpu-affinity 0:1:2:3:4:5:6:7Mixed-precision HPL-MxP on one node

...
=============================================================================
matrix size (N): 280000
block size (B): 2048
total MPIs: 8
MPI map: P*Q, P: 2, Q: 4
pmap: ROWDIST
sloppy type: FP16
...
LU factorization time: 7.91 s
LU GFLOPS: 1.8502e+06   LU GFLOPS (per GPU): 231270.91
IR time: 3.06 s
overall time: 10.97 s
overall performance: 1.3341e+06 GFlops
Avg performance per MPI: 166759.61 GFlops
Residual check: PASSED
=============================================================================

What to look at: HPL-MxP factors in low precision (FP16 here) and refines to FP64 accuracy, so its overall GFlops are about 3x the FP64 HPL result and are not an Rmax for the TOP500 list. Check the residual passed and that the per-GPU LU rate is similar across nodes.

nvidia-smi -q -d PERFORMANCEWhy clocks are being reduced


==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 11:02:20 2026
Driver Version                            : 550.90.07
CUDA Version                              : 12.4

Attached GPUs                             : 8
GPU 00000000:1B:00.0
    Performance State                     : P0
    Clocks Event Reasons
        Idle                              : Not Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active

...
GPU 00000000:C3:00.0
    Performance State                     : P0
    Clocks Event Reasons
        Idle                              : Not Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Active
        Display Clock Setting             : Not Active

...

What to look at: During an HPL run every reason should be Not Active; here GPU 00000000:C3:00.0 (GPU 5) shows SW Thermal Slowdown Active, so it is throttling on temperature and will drag the whole node's Gflops down (HW Slowdown or HW Power Brake Active would point to power or cooling hardware).

nvidia-smi dmonLive per-GPU power, temperature, clocks

# gpu    pwr  gtemp  mtemp     sm    mem    enc    dec    jpg    ofa   mclk   pclk
# Idx      W      C      C      %      %      %      %      %      %    MHz    MHz
    0    688     61     70    100     54      0      0      0      0   2619   1755
    1    691     63     72    100     54      0      0      0      0   2619   1755
    2    686     60     69    100     54      0      0      0      0   2619   1755
    3    690     62     71    100     54      0      0      0      0   2619   1755
    4    684     64     73    100     54      0      0      0      0   2619   1740
    5    612     86     92    100     54      0      0      0      0   2619   1410
    6    689     62     71    100     54      0      0      0      0   2619   1755
    7    687     61     70    100     54      0      0      0      0   2619   1755
...

What to look at: Under HPL every GPU should run at 100% SM near the 700 W limit with similar temperatures and pclk. GPU 5 here is about 25 C hotter with pclk at 1410 MHz instead of about 1755 MHz, which is the throttling GPU to chase (airflow, fans, inlet temperature).

nvidia-smi -q -d POWERPower draw and power limits


==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 11:03:05 2026
Driver Version                            : 550.90.07
CUDA Version                              : 12.4

Attached GPUs                             : 8
GPU 00000000:1B:00.0
    GPU Power Readings
        Power Draw                        : 689.42 W
        Current Power Limit               : 700.00 W
        Requested Power Limit             : 700.00 W
        Default Power Limit               : 700.00 W
        Min Power Limit                   : 200.00 W
        Max Power Limit                   : 700.00 W
    Power Samples
        Duration                          : 2.38 sec
        Number of Samples                 : 119
        Max                               : 699.86 W
        Min                               : 671.23 W
        Avg                               : 688.17 W
    GPU Memory Power Readings
        Power Draw                        : 96.12 W
...
GPU 00000000:43:00.0
...

What to look at: Current Power Limit should equal the Default Power Limit (700 W on H100 SXM). A lower current or requested limit on one node, for example 500 W, is a power cap that explains a node scoring below its peers; reset it with nvidia-smi -pl or fix the BMC policy.

13. Troubleshooting scenarios

14. Searchable reference

Item	Meaning

15. Flashcards (click to flip)

16. Exam traps

HPL = FP64 dense linear solve (LU), compute-bound, used for TOP500. Not a memory-bandwidth or network-only test.
Residual FAILED → wrong answers → hardware suspicion (GPU/HBM/thermal), not "tune NB."
N fills memory (N² × 8 bytes ≈ 80–90%). P × Q = number of ranks/GPUs, square-ish with P ≤ Q.
Rmax = measured, Rpeak = theoretical, efficiency = Rmax / Rpeak.
HPL is run last in bring-up (after DCGM, NCCL, perftest). It's a system-level test, not a fault isolator.
One slow node in multi-node HPL → run single-node HPL on each node to find the outlier (often thermal/power throttling).
Burn-in = repeated HPL for hours or days to catch intermittent failures.
HPL-MxP = mixed precision. HPCG = memory-bound sparse solver with a low % of peak. Don't confuse them with HPL.
On DGX, use NVIDIA's HPC-Benchmarks NGC container with one rank per GPU.
HPL draws near-maximum power, so it's also a facility (power/cooling) test.
Don't raise the threshold (16.0) to make a failing run "pass."

17. Practice quiz (32 questions)

Score: 0 / 0