ClusterKit and the Validation Toolbox

What NVIDIA ClusterKit is, when to use it, and how it compares to DCGM, perftest, ibdiagnet, nccl-tests, and HPL. A companion to the cluster bring-up guide.

1. What ClusterKit is

ClusterKit is NVIDIA's multi-node assessment tool for HPC and AI clusters. It ships with the NVIDIA HPC-X software toolkit (HPC-X = NVIDIA's bundle of Open MPI, UCX, and related communication libraries, originally from Mellanox). It's launched by a script, clusterkit.sh, found under $HPCX_CLUSTERKIT_DIR once HPC-X is loaded.

Its job is to run a set of network and node performance tests across many nodes automatically, then tell you which nodes or links are outliers. You get one report for the whole cluster instead of running hundreds of manual pair tests.

perftest is checking one water pipe at a time with a pressure gauge. ClusterKit is a building-wide inspection crew: it tests every pipe pair, every room's pressure, and what happens when all taps run at once, then hands you a list of the leaky ones.

ClusterKit is for fabric- and node-level screening at scale: "Across all N nodes, which node or link is slow or broken?" It works at the MPI / UCX level (point-to-point messaging), not at the NCCL collective level.

2. What ClusterKit tests

NVIDIA's HPC-X documentation describes ClusterKit as testing latency, bandwidth, effective bandwidth, memory bandwidth, GFLOPS per node, and GPU-to-GPU communication over the network.

Test	What it measures	What a bad result points to
Latency (pairwise)	Small-message round-trip time between node pairs	Bad link, wrong routing, a node falling back to a slow path
Bandwidth (pairwise)	Large-message throughput between node pairs	Degraded cable/port (for example running at 200 instead of 400), bad HCA
Effective bandwidth	Bandwidth while many pairs communicate at the same time (shuffled pairings)	Fabric congestion, oversubscribed or failing spine links, routing problems. Things pair tests alone miss.
Memory bandwidth (per node)	Host memory throughput on each node	DIMM problems, wrong BIOS memory settings, a node that differs from the rest
GFLOPS (per node)	CPU compute throughput on each node	Throttling, BIOS power settings, a misbehaving node
GPU-to-GPU (over the network)	Bandwidth and latency between GPUs on different nodes (exercises GPUDirect RDMA)	GPUDirect RDMA not working (for example `nvidia-peermem`), poor GPU–NIC affinity

ClusterKit = "perftest at scale + a congestion test + per-node sanity, with outlier ranking."

3. How it runs and what you get

# Load HPC-X (path and version vary by install)
source /opt/hpcx/hpcx-init.sh && hpcx_load

# Hostfile: one node per line
cat hosts.txt
dgx001
dgx002
...

# Run ClusterKit across those nodes
$HPCX_CLUSTERKIT_DIR/bin/clusterkit.sh --hostfile hosts.txt

# See test selection, GPU, and per-node options for YOUR version
$HPCX_CLUSTERKIT_DIR/bin/clusterkit.sh --help

Flags for picking tests, HCAs, and GPU mode differ between HPC-X releases. For the exam, know what ClusterKit does and when to use it. For real work, check --help on your version.

It uses MPI (HPC-X Open MPI + UCX) to launch on every host. You need passwordless SSH or a scheduler allocation, the same as for mpirun nccl-tests.
Results go into a timestamped output directory (machine-readable files per test plus a summary).
The summary highlights outliers: node pairs with low bandwidth or high latency, and nodes with weak memory or compute numbers. That gives you a list of suspects.
On a large cluster, run it under Slurm (salloc across the nodes, then run the script) so it doesn't collide with user jobs.

4. Where it fits in bring-up

#	Step	Tool	Answers
1	Node GPU health	DCGM `dcgmi diag -r 3`	Is each GPU healthy?
2	Node NVLink	Single-node nccl-tests	Do the 8 GPUs talk at full NVLink speed?
3	Fabric structure	`ibstat`, ibdiagnet, UFM	Are links up at the right speed, with no errors, and the topology as designed?
4	Link performance (spot check)	perftest `ib_write_bw`	Does this one link reach line rate?
5	Fabric and node performance screen	ClusterKit	Across all nodes, which links or nodes are outliers? Does the fabric hold up under simultaneous load?
6	Real communication stack	Multi-node nccl-tests	Do NCCL collectives (what training uses) reach the expected busbw at full scale?
7	Whole system	HPL, real training, burn-in	Stable and fast under sustained, realistic load?

Steps 4 and 5 overlap. perftest is the manual, one-link tool (great for confirming a specific suspect). ClusterKit is the automated, all-nodes tool (great for finding suspects in the first place). Many teams run ClusterKit first, then use perftest to dig into the pairs it flags.

5. Head-to-head comparison

	DCGM (`dcgmi diag`)	perftest (`ib_write_bw`)	ClusterKit	nccl-tests	HPL
Scope	One node (its GPUs)	One link / node pair	All nodes, pairwise + simultaneous	Single node or the whole job's nodes	Whole cluster
Layer	GPU hardware	Raw RDMA verbs	MPI / UCX point-to-point, plus host memory/CPU, plus GPU-RDMA	NCCL collectives over NVLink + IB + GDR	Full application stack
Uses the network?	No	Yes	Yes	Yes (multi-node)	Yes
Main output	PASS/FAIL per GPU and test	Gb/s or µs for one link	Matrix / ranking of pairs and nodes, outliers flagged	algbw / busbw per message size, #wrong	TFLOPS, pass/fail residual
Pinpoints bad part?	Yes (which GPU)	Yes (that link)	Yes (which node or pair)	Partly (needs bisecting)	No
Finds congestion?	No	No	Yes (effective bandwidth)	Indirectly (low busbw at scale)	Indirectly
Reflects training communication?	No	No	Partly (not NCCL)	Yes	Partly
Ships with	DCGM package	perftest / OFED	HPC-X	github NVIDIA/nccl-tests	NVIDIA HPC Benchmarks container
Typical moment	First, every node	Fabric up / debugging a link	After the fabric is up, before NCCL at scale	After ClusterKit/perftest	Final acceptance / burn-in

DCGM = is the GPU healthy? perftest = is this wire healthy? ClusterKit = which wires or nodes across the cluster are weak, and does the fabric hold up under load? nccl-tests = does GPU communication (the way training does it) work at full speed? HPL = does everything hold up under sustained load?

ClusterKit doesn't replace DCGM: it doesn't stress-test GPU memory, power, or thermals. It doesn't replace nccl-tests either: it doesn't exercise NCCL's algorithms, NVLink intra-node collectives, or NCCL environment settings. A clean ClusterKit run with poor nccl-tests results points at NCCL configuration (HCA selection, GDR, topology), not the cables.

6. "Which tool?" picker

Click the question you're trying to answer.

Pick a question above.

7. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

source /opt/hpcx/hpcx-init.sh && hpcx_load && echo $HPCX_CLUSTERKIT_DIR && which mpirunLoad HPC-X and locate ClusterKit

/opt/hpcx/clusterkit
/opt/hpcx/ompi/bin/mpirun

What to look at: hpcx_load itself prints nothing; success means HPCX_CLUSTERKIT_DIR is set and mpirun resolves to the HPC-X Open MPI. An empty variable or a system mpirun path means HPC-X is not loaded, so clusterkit.sh will not be found or will launch with the wrong MPI.

cat hosts.txtHostfile, one node per line

dgx01
dgx02
dgx03
dgx04
dgx05
dgx06
dgx07
dgx08

What to look at: One hostname per line, every node you want screened, and each reachable by passwordless SSH or inside your Slurm allocation. A node missing from the file is simply never tested, and a typo makes the MPI launch fail before any test runs.

$HPCX_CLUSTERKIT_DIR/bin/clusterkit.sh --helpOptions for this ClusterKit version

./clusterkit.sh [options]
Parameters:
  -v|--verbose          Set verbose mode
  -f|--hostfile         File with newline separated hostnames to run tests on.
  -r|--hpcx_dir         Path to HPCX installation root folder (or use env HPCX_DIR)
Options:
  -p|--ppn              Select number of processes per hostname (default: 1)
  -d|--hca_list "string"  Comma separated list of HCAs to use (default: autoselect)
  -t|--transport_list "string"  List of RDMA transports to use (rc,dc,ud) (default: autoselect best)
  -s|--ssh              Use ssh for process launching (default: autoselect)
  -h|--help             Show help message
  -n|--dry-run          Dry run (do nothing, only print)
  -m|--map-by [node|core|socket]
  -U|--unidirectional   Run unidirectional bandwidth tests (default: bidirectional)
  -e|--mapper           shell script that maps local MPI rank to a core and one or more HCAs
  -g|--gpu              Run GPU lat/bw/neighbor tests
  -G|--gpudirect        Run GPU tests with GPU-Direct.
  -x|--exe_opt          Options for clusterkit.
  -i|--mpi_opt          Options for mpirun.
...

What to look at: Use it to see how your release selects HCAs (--hca_list "mlx5_0:1"), GPU tests (-g, -G for GPUDirect) and test subsets (--exe_opt). Flag names change between HPC-X releases, so trust this output over notes written for another version.

salloc -N 8 --exclusiveReserve nodes before a fabric screen

salloc: Granted job allocation 4211
salloc: Waiting for resource configuration
salloc: Nodes dgx[01-08] are ready for job

What to look at: Run ClusterKit inside an exclusive allocation so user jobs cannot share the fabric and skew the numbers. If the allocation stays 'Pending' or lists fewer nodes than requested, some nodes are drained or busy and will be missing from the screen.

$HPCX_CLUSTERKIT_DIR/bin/clusterkit.sh --hostfile hosts.txt --hca_list "mlx5_0:1"Run the multi-node screen

Running ClusterKit on 8 hosts (1 ppn), HCA mlx5_0:1
...
Latency test (0 B, 1024 iters) ................. done
  min 1.92 us   max 2.61 us   bad pairs (>2.1x min): 0
Bandwidth test (32 MB, 16 iters) ............... done
  min 23102 MB/s   max 48655 MB/s   bad pairs (<93% of max): 7
    dgx05 <-> dgx01   23102 MB/s
    dgx05 <-> dgx02   23310 MB/s
    ...
Effective bandwidth test ...................... done
...
Output directory: 20261110_101532/

What to look at: ClusterKit flags latencies more than 2.1x the minimum and bandwidths below 93% of the maximum by default. Every bad pair involving one node (dgx05 here, at roughly half of NDR) is the per-node pattern that points at a degraded port, cable or HCA on that node; the summary format varies by HPC-X version.

cat 20261110_101532/bandwidth.txtPairwise bandwidth matrix (MB/s)

# Bandwidth (MB/s), technology: NDR, 32 MB messages
        dgx01  dgx02  dgx03  dgx04  dgx05  dgx06  dgx07  dgx08
dgx01       -  48602  48577  48655  23102  48590  48611  48549
dgx02   48602      -  48633  48588  23310  48570  48601  48598
dgx03   48577  48633      -  48614  23245  48621  48580  48607
dgx04   48655  48588  48614      -  23188  48599  48612  48575
dgx05   23102  23310  23245  23188      -  23270  23199  23230
...

What to look at: For NDR 4x the pass bar is about 46,060 MB/s (95% of 400 Gb/s) for the minimum pair. A full row and column at about half rate (dgx05) is the 'red plus' pattern of one bad node, while scattered low cells between nodes on different leaves point at spine links or routing.

dcgmi diag -r 3Per-node GPU health diagnostic

Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 3.3.5                                          |
| Driver Version Detected   | 535.161.08                                     |
| GPU Device IDs Detected   | 2330,2330,2330,2330,2330,2330,2330,2330        |
|-----  Deployment  --------+------------------------------------------------|
| Denylist                  | Pass                                           |
| NVML Library              | Pass                                           |
| CUDA Main Library         | Pass                                           |
| Permissions and OS Blocks | Pass                                           |
| Persistence Mode          | Pass                                           |
| Environment Variables     | Pass                                           |
| Page Retirement/Row Remap | Pass                                           |
| Graphics Processes        | Pass                                           |
| Inforom                   | Pass                                           |
+-----  Integration  -------+------------------------------------------------+
| PCIe                      | Pass - All                                     |
+-----  Hardware  ----------+------------------------------------------------+
| GPU Memory                | Pass - All                                     |
| Diagnostic                | Pass - All                                     |
+-----  Stress  ------------+------------------------------------------------+
| Targeted Stress           | Pass - All                                     |
| Targeted Power            | Pass - All                                     |
| Memory Bandwidth          | Pass - All                                     |
+---------------------------+------------------------------------------------+

What to look at: Every row should read Pass (Pass - All for per-GPU tests). A Fail names the GPU and test, for example GPU Memory failing on one GPU after an XID 48, which is a GPU hardware problem that ClusterKit and other network tests cannot detect.

./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8Single-node NVLink all-reduce check

# nThread 1 nGpus 8 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid  41022 on      dgx01 device  0 [0x18] NVIDIA H100 80GB HBM3
...
#  Rank  7 Group  0 Pid  41022 on      dgx01 device  7 [0xdb] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    17.92    0.00    0.00      0    17.55    0.00    0.00      0
...
  1073741824     268435456     float     sum      -1   4012.6  267.59  468.29      0   4008.1  267.89  468.81      0
  8589934592    2147483648     float     sum      -1    31790  270.21  472.86      0    31772  270.36  473.13      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 182.44

What to look at: Look at busbw for the largest messages: a DGX H100 should reach roughly 450+ GB/s over NVLink, and #wrong must be 0. Much lower busbw points at an NVLink or NVSwitch problem on this node, which you fix before any multi-node test.

ibstat mlx5_0Fabric structure: port up at 400

CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0xa088c20300000001
        System image GUID: 0xa088c20300000001
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 57
                LMC: 0
                SM lid: 1
                Capability mask: 0xa751e848
                Port GUID: 0xa088c20300000001
                Link layer: InfiniBand

What to look at: Each compute HCA should be Active, LinkUp, Rate 400. A node ClusterKit flags at half bandwidth will often show Rate 200 here on one of its ports, which is a physical-layer problem to chase with mlxlink and ibdiagnet.

ibdiagnetFabric-wide structural health check

Loading IBDIAGNET from: /usr/lib/x86_64-linux-gnu/ibdiagnet2.1.1
-I- Using port 1 as the local port
...
Summary
-I- Stage                     Warnings   Errors     Comment
-I- Discovery                 0          0
-I- Lids Check                0          0
-I- Links Check               0          0
-I- Subnet Manager            0          0
-I- Port Counters             0          0
-I- Nodes Information         0          0
-I- Speed / Width checks      0          0

-I- You can find detailed errors/warnings in: /var/tmp/ibdiagnet2/ibdiagnet2.log

What to look at: This answers 'is the fabric built correctly' (links, speeds, counters, SM), not 'is it fast under load'. Run it clean (0 warnings, 0 errors) before ClusterKit so performance outliers are not just known structural faults.

ib_write_bw -d mlx5_0 --report_gbits dgx01Confirm one flagged pair with perftest

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 Mtu             : 4096[B]
 Link type       : IB
...
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      5000             196.84             196.52             0.374832
---------------------------------------------------------------------------------------

What to look at: Run this from the flagged node (dgx05) with 'ib_write_bw -d mlx5_0 --report_gbits' started on dgx01 as the server. About 196 Gb/s on a 400G link confirms ClusterKit's finding on that HCA (typically a 2x link); about 390 Gb/s means that rail is fine and you test the next one.

lsmod | grep nvidia_peermemCheck GPUDirect RDMA module

nvidia_peermem         16384  0
ib_core               434176  9 rdma_cm,ib_ipoib,nvidia_peermem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
nvidia              56463360  1037 nvidia_uvm,nvidia_peermem,nvidia_modeset

What to look at: nvidia_peermem must be loaded for GPUDirect RDMA. No output means it is not loaded, so GPU-to-GPU traffic bounces through host memory and ClusterKit's GPU tests (and NCCL) show inter-node GPU bandwidth well below host-to-host bandwidth.

nvidia-smi topo -mGPU-to-NIC affinity matrix

        GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  NIC0  NIC1  NIC2  NIC3  CPU Affinity  NUMA Affinity
GPU0     X    NV18  NV18  NV18  NV18  NV18  NV18  NV18  PIX   SYS   SYS   SYS   0-55,112-167  0
GPU1    NV18   X    NV18  NV18  NV18  NV18  NV18  NV18  SYS   PIX   SYS   SYS   0-55,112-167  0
GPU2    NV18  NV18   X    NV18  NV18  NV18  NV18  NV18  SYS   SYS   PIX   SYS   0-55,112-167  0
GPU3    NV18  NV18  NV18   X    NV18  NV18  NV18  NV18  SYS   SYS   SYS   PIX   0-55,112-167  0
...
NIC Legend:
  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
...

What to look at: Each GPU should have one NIC at PIX (same PCIe switch) and NV18 to every other GPU on a DGX H100. If a GPU's rail NIC shows SYS or NODE instead of PIX, or the GPU-to-NIC mapping is wrong for the job, GPUDirect traffic crosses the CPU and GPU-to-GPU network bandwidth drops.

mpirun -np 64 -N 8 --hostfile hosts.txt -x NCCL_DEBUG=INFO ./build/all_reduce_perf -b 8 -e 16G -f 2 -g 1Multi-node NCCL collectives at scale

dgx01:52110:52110 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp170s0f0np0:192.168.50.11<0>
dgx01:52110:52190 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA
...
#       size         count      type   redop    root     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)
  4294967296    1073741824     float     sum      -1    22688  189.31  370.70      0
  8589934592    2147483648     float     sum      -1    45106  190.44  372.92      0
 17179869184    4294967296     float     sum      -1    90044  190.79  373.61      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 121.07

What to look at: Large-message busbw should approach the fabric's per-GPU rate (roughly 350 to 380 GB/s on 8 x NDR per node) with #wrong 0, and the log should show NET/IB with GDRDMA on all eight rails. If ClusterKit was clean but this shows NET/Socket or low busbw, the problem is NCCL configuration (NCCL_IB_HCA, container device access, GDR), not the cables.

8. Scenarios

9. Exam traps

ClusterKit is part of HPC-X, not DCGM and not NCCL.
"Identify the underperforming nodes/links across the whole cluster" → ClusterKit (automated multi-node screening). "Confirm one specific link" → perftest.
"Detect congestion when all nodes communicate at once" → ClusterKit effective bandwidth (or nccl-tests at scale). Pair tests alone miss it.
"Verify GPU hardware health / stress GPUs" → DCGM, not ClusterKit.
"Verify the collective communication training will use" → nccl-tests, not ClusterKit.
"Check fabric structure: link errors, speeds, topology" → ibdiagnet / UFM (diagnostics), not a bandwidth benchmark.
Order: per-node (DCGM, single-node NCCL) → fabric structure (ibdiagnet) → fabric performance (perftest / ClusterKit) → multi-node nccl-tests → HPL / burn-in.

10. Practice quiz (15 questions)

Score: 0 / 0