gpu-burn & dcgmproftester

Two GPU load tools that are easy to confuse with each other and with dcgmi diag: what each one does, how to run and read it, and when to use it instead of DCGM diagnostics, NVSM stress-test, nvbandwidth, nccl-tests, or HPL.

1. The big picture: three different jobs

Tool	Its job in one sentence	Verdict it gives
`dcgmi diag`	NVIDIA's official GPU health diagnostic: runs a structured battery of tests and says whether each GPU is healthy	PASS / FAIL / WARN per test, suitable as RMA evidence
`gpu-burn`	A community burn-in tool: runs every GPU at maximum matrix-math load and checks the results for computation errors	OK / FAULTY per GPU, plus an error count
`dcgmproftester`	A DCGM utility that generates a controlled load for one specific profiling metric (for example tensor-core activity or memory bandwidth)	No health verdict. It's a load generator, used mainly to validate monitoring.

dcgmi diag is the official vehicle inspection with a certificate. gpu-burn is driving the car flat-out on a track for an hour and checking whether anything breaks. dcgmproftester is a test rig that holds the engine at exactly 3,000 RPM so you can check that the dashboard gauges read correctly.

diag = diagnose (official verdict). burn = burn-in (max load + error check). proftester = profiling tester (drives a chosen metric so you can test monitoring).

2. gpu-burn: what it is

An open-source multi-GPU stress test (commonly from github.com/wilicc/gpu-burn). It's not an official NVIDIA product, but it's widely used for burn-in.
It runs large matrix multiplications (cuBLAS GEMM) continuously on every GPU at once, pushing compute, power, and temperature to the maximum.
It periodically compares results against a reference. Mismatches are counted as errors, which catches GPUs that are fast but compute wrong numbers (silent data corruption, often heat- or power-related).
Works on any CUDA GPU (DGX or not). Needs the NVIDIA driver and a CUDA build (or a container image).

gpu-burn's value is the error check under maximum sustained load. A GPU that passes short tests but starts producing errors after 20 minutes at full power and temperature is exactly what it's designed to catch.

3. Running gpu-burn and reading its output

# Build (needs the CUDA toolkit)
git clone https://github.com/wilicc/gpu-burn.git && cd gpu-burn && make

./gpu_burn 60            # all GPUs, 60 seconds, single precision (FP32)
./gpu_burn -d 3600       # double precision (FP64), 1 hour
./gpu_burn -tc 600       # use Tensor Cores, 10 minutes
./gpu_burn -m 90% 600    # use 90% of GPU memory
./gpu_burn -i 3 300      # only GPU 3
./gpu_burn -l            # list GPUs
./gpu_burn -h            # all options for your version

Flags differ slightly between forks and versions. Always check ./gpu_burn -h. The last argument is usually the duration in seconds.

Output (illustrative)

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-...)
...
Using compare file: compare.ptx
Burning for 600 seconds.
 10.0%  proc'd: 1240 (48210 Gflop/s) - 1238 (48190 Gflop/s) - ...   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 61 C - 63 C - ...
 50.0%  proc'd: 6200 (48180 Gflop/s) - ...                         errors: 0 - 0 - 0 - 17 - 0 - 0 - 0 - 0   temps: 74 C - 75 C - 76 C - 89 C - ...
100.0%  ...
Killing processes.. done

Tested 8 GPUs:
  GPU 0: OK
  GPU 1: OK
  GPU 2: OK
  GPU 3: FAULTY
  GPU 4: OK
  ...

What to read	Meaning
errors per GPU	Must stay 0. Any errors = wrong computation results under load.
Gflop/s per GPU	Should be similar across identical GPUs. An outlier (much lower) suggests throttling or a weak GPU.
temps	One GPU much hotter than its neighbours points to a cooling/airflow problem for that slot.
OK / FAULTY	Final verdict. FAULTY = drain the node and investigate (DCGM diag, ECC, Xids), and RMA if confirmed.

gpu-burn FAULTY is strong evidence, but NVIDIA support normally expects official diagnostics (dcgmi diag -r 3/4, and nvsm dump health on DGX) before an RMA. Use gpu-burn to find the problem and DCGM to document it.

4. dcgmproftester: what it is

A utility that ships with DCGM. The binary name includes the CUDA major version it's built for: dcgmproftester12 (CUDA 12), dcgmproftester11 (CUDA 11).
It runs CUDA workloads designed to drive one chosen profiling metric (a DCGM "DCP" field) to high levels: tensor-core activity, FP64/FP32/FP16 pipe activity, SM activity/occupancy, DRAM bandwidth, PCIe or NVLink traffic.
Originally built to validate DCGM's profiling metrics: "if I generate known tensor-core load, does DCGM report it?"
In practice, admins use it to generate realistic load for testing the monitoring pipeline (DCGM → DCGM Exporter → Prometheus → Grafana/alerts), and as a targeted load generator (for example to heat GPUs or drive tensor cores while watching power and thermals).

dcgmproftester answers "does our monitoring show what the GPU is really doing?" and "give me controlled load of type X." It is not a health diagnostic, and it doesn't produce an official PASS/FAIL for the GPU.

5. Profiling field IDs you need

Field ID	DCGM field name	Measures
1001	`DCGM_FI_PROF_GR_ENGINE_ACTIVE`	Graphics/compute engine active (the "real" GPU busy %)
1002	`DCGM_FI_PROF_SM_ACTIVE`	Fraction of time SMs have work
1003	`DCGM_FI_PROF_SM_OCCUPANCY`	Warps resident vs maximum
1004	`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	Tensor Core activity
1005	`DCGM_FI_PROF_DRAM_ACTIVE`	Memory (HBM) bandwidth utilization
1006	`DCGM_FI_PROF_PIPE_FP64_ACTIVE`	FP64 pipe activity
1007	`DCGM_FI_PROF_PIPE_FP32_ACTIVE`	FP32 pipe activity
1008	`DCGM_FI_PROF_PIPE_FP16_ACTIVE`	FP16 pipe activity
1009 / 1010	`DCGM_FI_PROF_PCIE_TX_BYTES` / `RX_BYTES`	PCIe traffic
1011 / 1012	`DCGM_FI_PROF_NVLINK_TX_BYTES` / `RX_BYTES`	NVLink traffic

"1004 = Tensor" (the most common demo). 1005 = DRAM. 1001/1002 = busy/SM active. 1009–1012 = PCIe/NVLink bytes.

nvidia-smi's "GPU-Util" only means "a kernel was running during the sample period." It can show 100% while the GPU is barely working. SM active (1002) and tensor active (1004) show how busy the GPU really is. dcgmproftester is how you prove those metrics are wired up correctly.

6. Running dcgmproftester

# Tensor-core load on all GPUs for 60 s, just generating load (no self-check through DCGM)
dcgmproftester12 --no-dcgm-validation -t 1004 -d 60

# Drive HBM bandwidth on GPU 0 only for 120 s
dcgmproftester12 --no-dcgm-validation -t 1005 -d 120 -i 0

# FP64 pipe load
dcgmproftester12 --no-dcgm-validation -t 1006 -d 60

# Options for your installed version
dcgmproftester12 --help

Option	Meaning
`-t <fieldId>`	Which profiling metric to drive (1001–1012…)
`-d <seconds>`	Duration
`-i <gpuIds>`	Which GPUs (default: all)
`--no-dcgm-validation`	Just generate load. Don't have the tool itself read the metric back through DCGM and check it. This is the usual mode for monitoring tests, or when another DCGM client (like the exporter) is watching.

Profiling metrics come from hardware counters, and only a limited set can be collected at the same time. If you get profiling-related errors, another tool (for example Nsight Compute, or a second DCGM profiling session) may be holding the counters. Pause it or use --no-dcgm-validation. Option names can vary by DCGM version, so check --help.

In Kubernetes (GPU Operator), the standard way to check that DCGM Exporter metrics reach Prometheus is to run a pod with dcgmproftester12 --no-dcgm-validation -t 1004 -d 30, then confirm DCGM_FI_PROF_PIPE_TENSOR_ACTIVE rises in Grafana.

7. Command builder

Tool

Duration (seconds)

GPU (blank = all)

gpu-burn precision

dcgmproftester metric

Purpose

8. Watching the load

# DCGM: watch the profiling metrics live (in a second terminal)
dcgmi dmon -e 1001,1002,1004,1005 -d 1000        # every 1000 ms
dcgmi dmon -e 155,150,203,1004                   # power (155), GPU temp (150), util (203), tensor (1004)

# nvidia-smi
nvidia-smi dmon -s pucvt                          # power, util, clocks, violations (throttling), temperature
nvidia-smi -q -d PERFORMANCE                      # clock throttle reasons

# Prometheus/Grafana (DCGM Exporter, port 9400)
curl -s localhost:9400/metrics | grep -E "PIPE_TENSOR_ACTIVE|DRAM_ACTIVE|SM_ACTIVE"

Always monitor while stressing. A stress test's value is not just "did it crash?" but also "did clocks throttle, did one GPU run hotter, did power hit the cap, did any Xid or ECC error appear?" Check dmesg for Xids after the run too.

9. Comparison

	gpu-burn	dcgmproftester	dcgmi diag	nvsm stress-test	nvbandwidth	nccl-tests	HPL
From	Open source (community)	DCGM package	DCGM package	DGX OS	NVIDIA (open source)	NVIDIA (open source)	NVIDIA HPC-Benchmarks
Scope	All GPUs in a node	Chosen GPUs, chosen metric	GPUs in a node	Whole DGX	Copy paths in a node	GPU communication (node/cluster)	Whole cluster
Load type	Max GEMM (FP32/FP64/TC)	Targeted: tensor, DRAM, FP64, PCIe, NVLink…	Structured test battery (memory, PCIe/NVLink, targeted stress/power…)	Many components	memcpy bandwidth	Collectives	FP64 LU
Verdict	OK/FAULTY + error count	None (load generator)	Official PASS/FAIL	Pass/fail per component	GB/s matrices	busbw, #wrong	Gflops + residual PASSED
Detects wrong results?	Yes	No	Yes (in relevant tests)	Partly	No	Yes (#wrong)	Yes (residual)
Best for	Burn-in, catching marginal GPUs under heat	Validating monitoring, controlled load	Health screening, RMA evidence	DGX after repair	Weak link/PCIe path	Comm performance	Acceptance, system stability
Works on non-DGX?	Yes	Yes (with DCGM)	Yes	No	Yes	Yes	Yes

diag = "certify it" · gpu-burn = "cook it and check the math" · proftester = "dial a metric, check the gauges" · nvsm = "whole DGX" · nvbandwidth = "road speeds" · nccl-tests = "traffic between GPUs" · HPL = "everything at once."

10. When to use which

Situation	Use	Why not the others
New node: are the GPUs healthy?	`dcgmi diag -r 3`	Official, structured, comparable across the fleet
Burn-in: catch GPUs that fail only after long, hot, full load	gpu-burn for hours (with `dcgmi diag -r 4` and HPL in the burn-in plan)	dcgmproftester has no error check
Intermittent wrong results / NaNs in one node's training	gpu-burn (error counts per GPU) → confirm with dcgmi diag -r 3/4	nccl-tests/nvbandwidth don't stress compute correctness the same way
Grafana shows tensor activity at 0 even during jobs. Is monitoring broken?	dcgmproftester12 -t 1004 while watching the dashboard	gpu-burn's load doesn't target a specific metric
Test power/thermal alerting thresholds with predictable load	dcgmproftester (targeted) or gpu-burn (maximum)	diag is too structured and short
A DGX just had a part replaced	`nvsm show health` → `nvsm stress-test` → `dcgmi diag -r 3` (optionally gpu-burn)	dcgmproftester isn't a validation tool
Slow GPU-to-GPU copies inside a node	nvbandwidth, then single-node nccl-tests	Stress tools don't measure link bandwidth
Slow distributed training	Multi-node nccl-tests	GPU stress tools don't test the network
Whole-cluster acceptance	HPL (+ burn-in)	Single-node tools don't stress the fabric
Evidence for an NVIDIA RMA	`dcgmi diag` results (+ `nvsm dump health` on DGX, `nvidia-bug-report.sh`)	gpu-burn output alone usually isn't enough

The usual order for a node: dcgmi diag -r 3 (is it healthy?) → gpu-burn or dcgmi diag -r 4 (does it stay healthy under long load?) → nvbandwidth / single-node nccl-tests (communication) → cluster tests. dcgmproftester sits on the side, for monitoring validation and controlled load.

11. "Which tool?" chooser

Pick a question above.

12. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

git clone https://github.com/wilicc/gpu-burn.git && cd gpu-burn && makeDownload and build gpu-burn

Cloning into 'gpu-burn'...
remote: Enumerating objects: 412, done.
remote: Counting objects: 100% (180/180), done.
remote: Compressing objects: 100% (61/61), done.
remote: Total 412 (delta 140), reused 119 (delta 119), pack-reused 232
Receiving objects: 100% (412/412), 118.37 KiB | 2.31 MiB/s, done.
Resolving deltas: 100% (241/241), done.
g++ -O3 -Wno-unused-result -I/usr/local/cuda/include -std=c++11 -DIS_JETSON=false -c gpu_burn-drv.cpp
PATH="${PATH}:.:/usr/local/cuda/bin" /usr/local/cuda/bin/nvcc -I/usr/local/cuda/include -std=c++11 -arch=compute_75 -ptx compare.cu -o compare.ptx
g++ -o gpu_burn gpu_burn-drv.o -O3 -lcuda -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64/stubs -lcublas -lcudart

What to look at: A good build ends with the g++ link line and leaves both gpu_burn and compare.ptx in the directory. 'nvcc: not found' or 'cublas_v2.h: No such file' means the CUDA toolkit isn't installed or isn't on the path; the driver alone isn't enough.

./gpu_burn 60Burn all GPUs in FP32 for 60 s

Using compare file: compare.ptx
Burning for 60 seconds.
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000000)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000001)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000002)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000003)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000004)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000005)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000006)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000007)
Initialized device 0 with 81559 MB of memory (80972 MB available, using 72874 MB of it), using FLOATS
Results are 268435456 bytes each, thus performing 282 iterations
...  (same two lines for devices 1-7)
 10.0%  proc'd: 3948 (48210 Gflop/s) - 3948 (48190 Gflop/s) - 3948 (48230 Gflop/s) - 3948 (48175 Gflop/s) - 3948 (48205 Gflop/s) - 3948 (48220 Gflop/s) - 3948 (48180 Gflop/s) - 3948 (48195 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 52 C - 54 C - 53 C - 55 C - 52 C - 53 C - 54 C - 52 C
 50.0%  proc'd: 19740 (48180 Gflop/s) - 19740 (48160 Gflop/s) - 19740 (48200 Gflop/s) - 19740 (48150 Gflop/s) - 19740 (48190 Gflop/s) - 19740 (48170 Gflop/s) - 19740 (48165 Gflop/s) - 19740 (48185 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 63 C - 65 C - 64 C - 66 C - 63 C - 64 C - 65 C - 63 C
100.0%  proc'd: 39480 (48170 Gflop/s) - 39480 (48150 Gflop/s) - 39480 (48190 Gflop/s) - 39480 (48145 Gflop/s) - 39480 (48180 Gflop/s) - 39480 (48160 Gflop/s) - 39480 (48155 Gflop/s) - 39480 (48175 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 66 C - 67 C - 66 C - 68 C - 65 C - 66 C - 67 C - 66 C
Killing processes with SIGTERM (soft kill)
Freed memory for dev 0
Uninitted cublas
...
done

Tested 8 GPUs:
  GPU 0: OK
  GPU 1: OK
  GPU 2: OK
  GPU 3: OK
  GPU 4: OK
  GPU 5: OK
  GPU 6: OK
  GPU 7: OK

What to look at: Every GPU must end OK with errors 0 on every progress line. Gflop/s should be within a few percent across identical GPUs and temperatures should be even; one GPU much slower or hotter than its neighbours points to throttling or a cooling problem in that slot.

./gpu_burn -d 3600Burn all GPUs in FP64 for one hour

Using compare file: compare.ptx
Burning for 3600 seconds.
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000000)
...
Initialized device 0 with 81559 MB of memory (80972 MB available, using 72874 MB of it), using DOUBLES
Results are 536870912 bytes each, thus performing 140 iterations
...
  1.0%  proc'd: 5320 (51840 Gflop/s) - 5320 (51810 Gflop/s) - 5320 (51870 Gflop/s) - 5320 (51790 Gflop/s) - 5320 (51850 Gflop/s) - 5320 (51830 Gflop/s) - 5320 (51800 Gflop/s) - 5320 (51820 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 55 C - 57 C - 56 C - 58 C - 55 C - 56 C - 57 C - 55 C
 50.0%  proc'd: 95760 (51420 Gflop/s) - 95760 (51400 Gflop/s) - 95760 (51450 Gflop/s) - 95760 (51380 Gflop/s) - 95760 (51430 Gflop/s) - 95760 (51410 Gflop/s) - 95760 (51390 Gflop/s) - 95760 (51400 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 69 C - 70 C - 69 C - 71 C - 68 C - 69 C - 70 C - 69 C
100.0%  proc'd: 191520 (51400 Gflop/s) - 191520 (51390 Gflop/s) - 191520 (51430 Gflop/s) - 191520 (51370 Gflop/s) - 191520 (51410 Gflop/s) - 191520 (51400 Gflop/s) - 191520 (51380 Gflop/s) - 191520 (51390 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 70 C - 71 C - 70 C - 72 C - 69 C - 70 C - 71 C - 70 C
Killing processes with SIGTERM (soft kill)
...
done

Tested 8 GPUs:
  GPU 0: OK
  GPU 1: OK
  GPU 2: OK
  GPU 3: OK
  GPU 4: OK
  GPU 5: OK
  GPU 6: OK
  GPU 7: OK

What to look at: Check that the header says 'using DOUBLES' and that errors stay 0 for the whole hour. A small Gflop/s drop as GPUs warm up is normal; a GPU whose rate keeps falling or whose errors start climbing late in the run is the marginal part burn-in is meant to catch.

./gpu_burn -tc 600Burn with Tensor Cores for 10 minutes

Using compare file: compare.ptx
Burning for 600 seconds.
...
Initialized device 0 with 81559 MB of memory (80972 MB available, using 72874 MB of it), using FLOATS, using Tensor Cores
Results are 268435456 bytes each, thus performing 282 iterations
...
 10.0%  proc'd: 24816 (362400 Gflop/s) - 24816 (361900 Gflop/s) - 24816 (362800 Gflop/s) - 24816 (361500 Gflop/s) - 24816 (362100 Gflop/s) - 24816 (362600 Gflop/s) - 24816 (361700 Gflop/s) - 24816 (362300 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 58 C - 60 C - 59 C - 61 C - 58 C - 59 C - 60 C - 58 C
 50.0%  proc'd: 124080 (358900 Gflop/s) - 124080 (358400 Gflop/s) - 124080 (359300 Gflop/s) - 124080 (358100 Gflop/s) - 124080 (358700 Gflop/s) - 124080 (359100 Gflop/s) - 124080 (358300 Gflop/s) - 124080 (358800 Gflop/s)   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 71 C - 72 C - 71 C - 73 C - 70 C - 71 C - 72 C - 71 C
...
done

Tested 8 GPUs:
  GPU 0: OK
  GPU 1: OK
  GPU 2: OK
  GPU 3: OK
  GPU 4: OK
  GPU 5: OK
  GPU 6: OK
  GPU 7: OK

What to look at: Look for 'using Tensor Cores' in the header and a much higher Gflop/s than plain FP32; if the rate looks like the FP32 run, Tensor Cores aren't being used. Tensor runs draw the most power, so watch for temperatures or rates that diverge between GPUs.

./gpu_burn -m 90% 600Burn using 90% of GPU memory

Using compare file: compare.ptx
Burning for 600 seconds.
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000000)
...
Initialized device 0 with 81559 MB of memory (80972 MB available, using 72874 MB of it), using FLOATS
Results are 268435456 bytes each, thus performing 282 iterations
...
 50.0%  proc'd: 19740 (48180 Gflop/s) - 19740 (48160 Gflop/s) - ...   errors: 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0   temps: 64 C - 65 C - ...
...
Tested 8 GPUs:
  GPU 0: OK
  ...
  GPU 7: OK

What to look at: The 'using N MB of it' figure should be about 90% of the available memory. A 'Couldn't allocate' error means another process already holds GPU memory, so check nvidia-smi and make sure the node is drained.

./gpu_burn -i 3 300Burn only GPU 3 for 5 minutes

Using compare file: compare.ptx
Burning for 300 seconds.
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-0a1b2c3d-0000-4000-8000-000000000003)
Initialized device 3 with 81559 MB of memory (80972 MB available, using 72874 MB of it), using FLOATS
Results are 268435456 bytes each, thus performing 282 iterations
  5.0%  proc'd: 987 (48160 Gflop/s)   errors: 0   temps: 61 C
 25.0%  proc'd: 4935 (47920 Gflop/s)   errors: 0   temps: 79 C
 40.0%  proc'd: 7896 (45310 Gflop/s)   errors: 17  (WARNING!)  temps: 87 C
 60.0%  proc'd: 11562 (43870 Gflop/s)   errors: 212  (WARNING!)  temps: 89 C
100.0%  proc'd: 18612 (43790 Gflop/s)   errors: 655  (WARNING!)  temps: 89 C
Killing processes with SIGTERM (soft kill)
Freed memory for dev 3
Uninitted cublas
done

Tested 1 GPUs:
  GPU 3: FAULTY

What to look at: This GPU computes wrong results once it passes about 85 C, and its rate drops as it throttles, so it ends FAULTY. Drain the node, check cooling, ECC, and Xids, then confirm with dcgmi diag -r 3 before an RMA; gpu-burn output alone usually isn't enough evidence.

./gpu_burn -lList the GPUs gpu-burn can use

ID 0: NVIDIA H100 80GB HBM3, 81559MB
ID 1: NVIDIA H100 80GB HBM3, 81559MB
ID 2: NVIDIA H100 80GB HBM3, 81559MB
ID 3: NVIDIA H100 80GB HBM3, 81559MB
ID 4: NVIDIA H100 80GB HBM3, 81559MB
ID 5: NVIDIA H100 80GB HBM3, 81559MB
ID 6: NVIDIA H100 80GB HBM3, 81559MB
ID 7: NVIDIA H100 80GB HBM3, 81559MB

What to look at: The count should match the installed GPUs (8 on a DGX H100), and the IDs are the ones -i takes. A missing ID means CUDA can't see that GPU; CUDA_VISIBLE_DEVICES or a dead GPU are the usual causes.

./gpu_burn -hShow gpu-burn options

GPU Burn
Usage: gpu_burn [OPTIONS] [TIME]

-m X   Use X MB of memory.
-m N%  Use N% of the available GPU memory.  Default is 90%
-d     Use doubles
-tc    Try to use Tensor cores
-l     Lists all GPUs in the system
-i N   Execute only on GPU N
-c FILE        Use FILE as compare kernel.  Default is compare.ptx
-stts T        Set timeout threshold to T seconds for using SIGTERM to abort child processes before using SIGKILL.  Default is 30
-h     Show this help message

Examples:
  gpu-burn -d 3600 # burns all GPUs with doubles for an hour
  gpu-burn -m 50% # burns using 50% of the available GPU memory
  gpu-burn -l # list GPUs
  gpu-burn -i 2 # burns only GPU of index 2

What to look at: Check the help on the copy you built, because flags differ between forks. TIME is the last argument in seconds, and -d means doubles (FP64) here, not duration.

dcgmproftester12 --no-dcgm-validation -t 1004 -d 60Generate Tensor Core load on all GPUs

Skipping CreateDcgmGroups() since DCGM validation is disabled
Skipping CreateDcgmGroups() since DCGM validation is disabled
Skipping WatchFields() since DCGM validation is disabled
Worker 0:0[1004]: Message: Bus ID 00000000:1B:00.0 mapped to cuda device ID 0
Worker 1:0[1004]: Message: Bus ID 00000000:43:00.0 mapped to cuda device ID 1
...
DCGM CudaContext Init completed successfully.

CU_DEVICE_ATTRIBUTE_MAX_THREADS_PER_MULTIPROCESSOR: 2048
CU_DEVICE_ATTRIBUTE_MULTIPROCESSOR_COUNT: 132
CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR: 9
CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR: 0
CU_DEVICE_ATTRIBUTE_GLOBAL_MEMORY_BUS_WIDTH: 5120
Max Memory bandwidth: 3352320000000 bytes (3352.32 GiB)
CU_DEVICE_ATTRIBUTE_ECC_SUPPORT: true

Worker 0:0[1004]: TensorEngineActive: generated ???, dcgm 0.000 (702114.6 gflops)
Worker 1:0[1004]: TensorEngineActive: generated ???, dcgm 0.000 (698532.1 gflops)
Worker 2:0[1004]: TensorEngineActive: generated ???, dcgm 0.000 (705240.9 gflops)
Worker 3:0[1004]: TensorEngineActive: generated ???, dcgm 0.000 (699871.3 gflops)
Worker 4:0[1004]: TensorEngineActive: generated ???, dcgm 0.000 (703318.4 gflops)
Worker 5:0[1004]: TensorEngineActive: generated ???, dcgm 0.000 (700962.7 gflops)
Worker 6:0[1004]: TensorEngineActive: generated ???, dcgm 0.000 (701455.0 gflops)
Worker 7:0[1004]: TensorEngineActive: generated ???, dcgm 0.000 (699204.8 gflops)
...  (one line per GPU per second for 60 s)
Skipping UnwatchFields() since DCGM validation is disabled

What to look at: With --no-dcgm-validation the 'dcgm 0.000' value is expected; the tool isn't reading the metric back, so check DCGM_FI_PROF_PIPE_TENSOR_ACTIVE in dmon or Grafana instead. The gflops figure only shows the load is running and is not a health verdict.

dcgmproftester12 --no-dcgm-validation -t 1005 -d 120 -i 0Drive HBM bandwidth on GPU 0

Skipping CreateDcgmGroups() since DCGM validation is disabled
Skipping WatchFields() since DCGM validation is disabled
Worker 0:0[1005]: Message: Bus ID 00000000:1B:00.0 mapped to cuda device ID 0
DCGM CudaContext Init completed successfully.
...
Worker 0:0[1005]: DramUtil: generated ???, dcgm 0.000 (2841.7 GiB/sec)
Worker 0:0[1005]: DramUtil: generated ???, dcgm 0.000 (2856.3 GiB/sec)
Worker 0:0[1005]: DramUtil: generated ???, dcgm 0.000 (2849.9 GiB/sec)
Worker 0:0[1005]: DramUtil: generated ???, dcgm 0.000 (2853.2 GiB/sec)
...  (one line per second for 120 s)
Skipping UnwatchFields() since DCGM validation is disabled

What to look at: Only Worker 0 appears because -i 0 limits the load to one GPU. While it runs, DRAM_ACTIVE (1005) for GPU 0 should rise close to 1.0 in dmon or Grafana while the other GPUs stay near 0; a flat dashboard means the monitoring pipeline is broken.

dcgmproftester12 --no-dcgm-validation -t 1006 -d 60Generate FP64 pipe load

Skipping CreateDcgmGroups() since DCGM validation is disabled
Skipping WatchFields() since DCGM validation is disabled
...
DCGM CudaContext Init completed successfully.
...
Worker 0:0[1006]: Fp64EngineActive: generated ???, dcgm 0.000 (31842.5 gflops)
Worker 1:0[1006]: Fp64EngineActive: generated ???, dcgm 0.000 (31790.2 gflops)
Worker 2:0[1006]: Fp64EngineActive: generated ???, dcgm 0.000 (31866.0 gflops)
Worker 3:0[1006]: Fp64EngineActive: generated ???, dcgm 0.000 (31805.7 gflops)
...
Skipping UnwatchFields() since DCGM validation is disabled

What to look at: The worker label names the metric being driven (Fp64EngineActive for 1006). Expect DCGM_FI_PROF_PIPE_FP64_ACTIVE to rise while tensor activity stays low; if the wrong metric moves, the dashboard query is pointing at the wrong field.

dcgmproftester12 --helpShow dcgmproftester options

USAGE:

   dcgmproftester12  [-t <fieldIds>] [-d <seconds>] [-i <gpuIds>]
                     [--no-dcgm-validation] [-r <seconds>] [--target-max-value]
                     [--mode <mode>] [--] [--version] [-h]

Where:

   -t <fieldIds>,  --fieldId <fieldIds>
     Profiling fieldIds to test, comma-separated (for example 1002,1004)

   -d <seconds>,  --duration <seconds>
     Duration in seconds to run each test. Default: 30

   -i <gpuIds>,  --gpuIds <gpuIds>
     Comma-separated list of GPU IDs to run on. Default: all GPUs

   --no-dcgm-validation
     Only generate the load. Don't watch the fields through DCGM
     and compare them with the expected values.

   -r <seconds>,  --reportinterval <seconds>
     How often to print a progress line

   -h,  --help
     Displays usage information and exits.
(output trimmed)

What to look at: Confirm the option names on your installed version before scripting, because they change between DCGM releases. The trailing 12 in the binary name is the CUDA major version it was built for, not a test level.

dcgmi dmon -e 1001,1002,1004,1005 -d 1000Watch profiling metrics live

#Entity   GRACT  SMACT  TENSO  DRAMA
ID
GPU 0     0.998  0.991  0.934  0.061
GPU 1     0.997  0.990  0.931  0.060
GPU 2     0.998  0.992  0.936  0.062
GPU 3     0.997  0.989  0.929  0.059
GPU 4     0.998  0.991  0.933  0.061
GPU 5     0.998  0.990  0.932  0.060
GPU 6     0.997  0.991  0.934  0.061
GPU 7     0.998  0.990  0.930  0.060
...
(one block per second until Ctrl+C)

What to look at: During a -t 1004 run, TENSO should jump to about 0.9 on the GPUs under load. If it stays at 0.000 or shows N/A, the profiling counters are held by another tool (for example Nsight Compute) or profiling is paused.

dcgmi dmon -e 155,150,203,1004Watch power, temperature, utilization, tensor

#Entity   POWER    TMPTR  GPUTL  TENSO
ID        W        C      %
GPU 0     688.4    66     100    0.934
GPU 1     691.2    67     100    0.931
GPU 2     686.9    66     100    0.936
GPU 3     690.7    68     100    0.929
GPU 4     689.3    65     100    0.933
GPU 5     692.0    84     100    0.712
GPU 6     687.8    67     100    0.934
GPU 7     690.1    66     100    0.930
...

What to look at: Under the same load, GPUs should read about the same power and temperature. GPU 5 here runs 17 C hotter with lower tensor activity, which is typical of thermal throttling from poor airflow in that slot.

nvidia-smi dmon -s pucvtStream power, util, clocks, violations, PCIe

# gpu    pwr  gtemp  mtemp     sm    mem    enc    dec    jpg    ofa   mclk   pclk  pviol  tviol  rxpci  txpci
# Idx      W      C      C      %      %      %      %      %      %    MHz    MHz      %   bool   MB/s   MB/s
    0    688     66     58    100     12      0      0      0      0   2619   1980      0      0     41     38
    1    691     67     59    100     12      0      0      0      0   2619   1980      0      0     40     37
    2    687     66     58    100     12      0      0      0      0   2619   1980      0      0     42     39
    3    690     68     60    100     12      0      0      0      0   2619   1980      0      0     41     38
    4    689     65     57    100     12      0      0      0      0   2619   1980      0      0     40     38
    5    692     84     77    100     12      0      0      0      0   2619   1590     34      1     41     39
    6    688     67     59    100     12      0      0      0      0   2619   1980      0      0     42     38
    7    690     66     58    100     12      0      0      0      0   2619   1980      0      0     41     37

What to look at: pviol is the percentage of time the GPU was power-capped and tviol flags a thermal violation. GPU 5's lower pclk (1590 MHz vs 1980) with tviol 1 confirms it is thermally throttling.

nvidia-smi -q -d PERFORMANCEShow clock throttle (event) reasons

==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 14:22:31 2026
Driver Version                            : 550.54.15
CUDA Version                              : 12.4

Attached GPUs                             : 8
GPU 00000000:1B:00.0
    Performance State                     : P0
    Clocks Event Reasons
        Idle                              : Not Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active
    Sparse Operation Mode                 : N/A
...
GPU 00000000:C3:00.0
    Performance State                     : P0
    Clocks Event Reasons
        Idle                              : Not Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Active
            HW Thermal Slowdown           : Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Active
        Display Clock Setting             : Not Active
...

What to look at: During a stress run every reason should be Not Active, except SW Power Cap, which can be Active briefly at full power. HW Slowdown or HW Thermal Slowdown Active on one GPU (here C3, GPU 5) means a cooling or hardware problem; older drivers call this section 'Clocks Throttle Reasons'.

curl -s localhost:9400/metrics | grep -E "PIPE_TENSOR_ACTIVE|DRAM_ACTIVE|SM_ACTIVE"Read DCGM Exporter profiling metrics

# HELP DCGM_FI_PROF_SM_ACTIVE The ratio of cycles an SM has at least 1 warp assigned (in %).
# TYPE DCGM_FI_PROF_SM_ACTIVE gauge
DCGM_FI_PROF_SM_ACTIVE{gpu="0",UUID="GPU-0a1b2c3d-0000-4000-8000-000000000000",pci_bus_id="00000000:1B:00.0",device="nvidia0",modelName="NVIDIA H100 80GB HBM3",Hostname="dgx01",DCGM_FI_DRIVER_VERSION="550.54.15"} 0.991
DCGM_FI_PROF_SM_ACTIVE{gpu="1",UUID="GPU-0a1b2c3d-0000-4000-8000-000000000001",pci_bus_id="00000000:43:00.0",device="nvidia1",modelName="NVIDIA H100 80GB HBM3",Hostname="dgx01",DCGM_FI_DRIVER_VERSION="550.54.15"} 0.990
...
# HELP DCGM_FI_PROF_PIPE_TENSOR_ACTIVE Ratio of cycles the tensor (HMMA) pipe is active (in %).
# TYPE DCGM_FI_PROF_PIPE_TENSOR_ACTIVE gauge
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{gpu="0",UUID="GPU-0a1b2c3d-0000-4000-8000-000000000000",pci_bus_id="00000000:1B:00.0",device="nvidia0",modelName="NVIDIA H100 80GB HBM3",Hostname="dgx01",DCGM_FI_DRIVER_VERSION="550.54.15"} 0.934
...
# HELP DCGM_FI_PROF_DRAM_ACTIVE Ratio of cycles the device memory interface is active sending or receiving data (in %).
# TYPE DCGM_FI_PROF_DRAM_ACTIVE gauge
DCGM_FI_PROF_DRAM_ACTIVE{gpu="0",UUID="GPU-0a1b2c3d-0000-4000-8000-000000000000",pci_bus_id="00000000:1B:00.0",device="nvidia0",modelName="NVIDIA H100 80GB HBM3",Hostname="dgx01",DCGM_FI_DRIVER_VERSION="550.54.15"} 0.061
(output trimmed)

What to look at: Each metric line ends with its current value, labeled by GPU and host. If grep prints nothing, the exporter isn't collecting profiling fields (check its metrics CSV) or isn't running; values that stay at 0 during a dcgmproftester run mean the counters are held by another tool.

13. Scenarios

14. Searchable reference

Item	Meaning

15. Flashcards (click to flip)

16. Exam traps

dcgmproftester is a load generator, not a diagnostic. It doesn't certify GPU health.
"Verify that DCGM Exporter / Grafana shows correct tensor-core metrics" → dcgmproftester12 -t 1004.
gpu-burn = maximum GEMM load + result checking → OK/FAULTY per GPU. Great for burn-in and catching marginal GPUs.
Official health / RMA evidence → dcgmi diag -r 3/4, not gpu-burn or dcgmproftester.
gpu-burn is community software. dcgmproftester and dcgmi ship with DCGM.
The 12 in dcgmproftester12 = the CUDA major version, not a test level.
Field 1004 = tensor active. 1005 = DRAM active. 1002 = SM active. 1001 = GR engine active.
nvidia-smi GPU-Util ≠ real work. Use SM active / tensor active DCGM profiling fields.
No GPU stress tool tests the network. Use nccl-tests / perftest / ClusterKit for that.
Monitor during stress (dmon, throttle reasons, temps, Xids). A run that "passes" but throttles heavily still reveals a cooling or power problem.

17. Practice quiz (22 questions)

Score: 0 / 0