nccl-tests

NVIDIA's collective benchmark as a working tool: building it with MPI, how ranks map to processes and GPUs, launching with Slurm srun and Open MPI mpirun, reading the output, what healthy numbers look like on DGX H100, and splitting one run into per-rail or per-node tests with NCCL_TESTS_SPLIT to find the bad link or node. Ends with five hands-on simulated tickets. Complements the NCCL deep dive (how NCCL itself works). Objectives 4.3, 5.1, and 5.2.

1. Where nccl-tests fits

nccl-tests (github.com/NVIDIA/nccl-tests) is a set of small programs, one per NCCL collective, that run the collective on real GPUs over a range of message sizes and report the time, the bandwidth, and whether the results were correct. Because it uses the same NCCL library as training jobs, a good nccl-tests result proves the whole communication path works: GPUs, NVLink and NVSwitch, the NICs, GPUDirect RDMA, the InfiniBand fabric, and the software stack (driver, NCCL, MPI, Slurm) together.

Question	Tool
Is each GPU healthy?	DCGM diagnostics (stress tools)
Is each InfiniBand wire healthy?	perftest (`ib_write_bw`), `mlxlink`, `ibdiagnet`
Do the GPUs communicate correctly and fast, inside a node and across nodes?	nccl-tests
Does the whole system perform and stay stable under real load?	HPL, burn-in, real training

nccl-tests is a timed relay race for the GPUs. It doesn't check each runner's fitness (DCGM does), or measure each lane of the track (perftest does). It tells you whether the whole team hands the baton around fast enough, and splitting the race (section 11) tells you which lane or which team is slow.

MPI or Slurm only starts the processes and helps them find each other. NCCL moves the data over NVLink and InfiniBand itself. So launcher problems show up as wrong rank counts, missing libraries, or startup errors; data-path problems show up as low busbw.

2. Building nccl-tests (with and without MPI)

git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make                                   # single-process build: one process, threads (-t) and GPUs (-g)
make MPI=1 MPI_HOME=/opt/openmpi CUDA_HOME=/usr/local/cuda NCCL_HOME=/opt/nccl
make MPI=1 NVCC_GENCODE="-gencode=arch=compute_90,code=sm_90"   # H100 only: much faster build
ls build/                              # all_reduce_perf, all_gather_perf, ...
ldd build/all_reduce_perf | grep -E "mpi|nccl"                  # prove it links libmpi and the right libnccl
make clean                             # needed before rebuilding with different flags

Variable	Meaning
`MPI=1`	Build with MPI so one test can span many processes and nodes. Without it, each process runs its own private test.
`MPI_HOME`	Where MPI's `include/mpi.h` and `lib/libmpi` live (Open MPI or HPC-X)
`CUDA_HOME`	CUDA toolkit (default `/usr/local/cuda`)
`NCCL_HOME`	NCCL headers and library to build against. If unset, the system NCCL is used.
`NVCC_GENCODE`	GPU architectures to compile for. `sm_90` = Hopper.

make also reads environment variables, so a module that sets NCCL_HOME and MPI_HOME works without repeating them on the command line. MPI=1 you still pass yourself.

make doesn't track flags. If build/ already exists from a non-MPI build, make MPI=1 says there's nothing to do and you keep the old binaries. Run make clean first, then confirm with ldd that libmpi is linked.

A non-MPI binary started by srun or mpirun with 16 tasks runs 16 separate one-rank tests. The output repeats 16 times, every copy says rank 0, and AllReduce busbw is 0.00 (with one rank the factor 2(n−1)/n is zero). It looks like it ran; it tested nothing.

Binary	Collective	busbw factor (n ranks)
`all_reduce_perf`	AllReduce: everyone ends with the sum	2(n−1)/n
`all_gather_perf`	AllGather: everyone gets everyone's piece	(n−1)/n
`reduce_scatter_perf`	ReduceScatter: everyone gets one reduced piece	(n−1)/n
`broadcast_perf` / `reduce_perf`	One root to all / all to one root	1
`alltoall_perf`	Everyone sends a distinct piece to everyone (MoE-style)	(n−1)/n
`sendrecv_perf`	Each rank sends to the next one	1

3. The collectives, visualized

Every test binary runs one collective: an operation that all ranks call together. Pick a tab and press Step to watch 4 GPUs run it, one chunk at a time. Each GPU's buffer is split into 4 chunks. GPU0 starts with 1, 2, 3, 4, GPU1 with 10, 20, 30, 40, GPU2 with 100s, and GPU3 with 1000s. Each GPU writes to its own digit, so a sum shows who contributed: 1011 means GPU0, GPU1, and GPU3 have been added, and 1111 is the full sum.

Cheat sheet: the collectives in one line each

Collective	What happens	In the kitchen	busbw factor
Broadcast	One GPU (the root) copies its data to all the other GPUs.	One chef shares their unique recipe with the whole kitchen.	1
Reduce	All GPUs send their data to one root GPU, which combines (sums) it. Only the root has the result.	All chefs dump their ingredients into the Head Chef's single bowl.	1
AllReduce	All GPUs combine their data, and every GPU gets a copy of the final total.	All chefs mix their spices together, and everyone fills their own bowl with the exact same final blend.	2(n−1)/n
ReduceScatter	All GPUs combine their data, but the total is sliced so each GPU keeps only one distinct chunk.	All chefs mix their spices together, but each chef walks away with only their assigned portion of the blend.	(n−1)/n
AllGather	Each GPU shares its distinct chunk, so every GPU ends up with the complete data set. No combining.	Each chef brings one unique ingredient, and they copy each other's until every chef has the full list.	(n−1)/n
AlltoAll	Each GPU sends a different piece to every other GPU; GPU j collects everyone's piece j.	Every chef sorts their prep into labeled trays and hands each tray to the chef whose station it belongs to.	(n−1)/n
SendRecv	Point to point: each GPU sends its buffer to the next GPU and receives from the previous one.	Each chef passes their finished plate to the next station down the line.	1

AllReduce = ReduceScatter + AllGather. "Reduce" means combine (add). "All" means everyone gets the result. "Scatter" means each gets a different slice; "Gather" means each gets every slice.

The busbw factor is just how much data each GPU had to send, relative to the buffer size. Count the chunks in the animation: in ring AllReduce each GPU sends 2 × (n−1) chunks of size S/n, which is 2(n−1)/n × S. nccl-tests multiplies algbw (S ÷ time) by that factor, so busbw is comparable across collectives and rank counts, and can be compared with the link speed.

The animation uses the ring algorithm because the busbw formulas come from it. Real NCCL also picks tree, or on H100 with NVSwitch, NVLS (the switch does the adding). The result is the same; only the route and speed differ.

4. The flags that matter

Flag	Meaning	Default	Typical use
`-b` / `-e`	Smallest / largest message size (accepts K, M, G)	32M / 32M	`-b 8 -e 8G` for a full sweep; `-b 8G -e 8G` for one quick large size
`-f`	Multiply the size by this factor each step	off	`-f 2` (doubling)
`-i`	Add this many bytes each step instead	1M	Rarely; `-f` is usual
`-g`	GPUs per thread	1	`-g 1` with one rank per GPU; `-g 8` for one process driving a whole node
`-t`	Threads per process	1	Usually left at 1
`-n` / `-w`	Timed iterations / untimed warm-up iterations	20 / 5	Raise for steadier numbers
`-c`	Correctness check iterations (0 turns checking off)	1	Keep on: it fills the `#wrong` column
`-a`	How times from different ranks are combined: 0 rank 0, 1 average, 2 minimum, 3 maximum	1	`-a 3` to make the slowest rank or group visible
`-o` / `-d`	Reduction op (sum, prod, min, max, avg, all) / data type (float, half, bfloat16, ...)	sum / float	`-d bfloat16` to match training
`-r`	Root rank for broadcast and reduce	0
`-z`	Blocking mode	0
`-G`	Launch through CUDA graphs	0	Lower launch overhead on small sizes

begin, end, factor; gPUs per thread; average mode. "-b 8 -e 8G -f 2 -g 1" is the line you'll type most.

5. Ranks, processes, threads, and GPUs

Every GPU in a test is one NCCL rank. The total is:

nranks = processes × threads per process (-t) × GPUs per thread (-g)

Layout	Command shape	Needs MPI build?	Use it for
One process, all 8 GPUs of one node	`./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8`	No	Quick single-node NVLink/NVSwitch check, including on a drained node over ssh
One rank per GPU (the standard)	8 processes per node, `-g 1`	Yes	Everything multi-node; mirrors how training jobs run
One process per node, 8 GPUs each	1 process per node, `-g 8`	Yes	Fewer processes; same NCCL traffic

How a process picks its GPUs: nccl-tests numbers the processes on each node (the local rank) and has local rank L use GPUs starting at L × t × g. That has two consequences:

Every process must see all of the node's GPUs. Under Slurm, request them with --gpus-per-node=8. Per-task binding such as --gpus-per-task=1 can leave local rank 1 asking for a GPU it can't see: Test CUDA failure ... 'invalid device ordinal'.
Too many ranks on one node (for example 32 ranks packed onto the first host) also asks for GPU 8 and up, and fails the same way. Check placement before blaming the GPUs.

The # Using devices block at the top of every run lists each rank's host, GPU index, and bus ID. Read it: it's the quickest way to confirm the layout is what you meant.

6. What MPI does (and doesn't do)

MPI (or the launcher) handles	NCCL handles
Starting one process per rank on the right hosts	Choosing NVLink, P2P, NVLS, or the network for each path
Giving each process its rank number	Moving every byte of the collective (GPUDirect RDMA over InfiniBand)
Sending the NCCL unique ID from rank 0 to everyone so they can form a communicator	Rings, trees, channels, protocols
Combining timings and correctness results at the end	Reporting its choices when `NCCL_DEBUG=INFO` is set

MPI transport settings (UCX, --mca pml ucx, btl) change how MPI's own small messages travel. They rarely change busbw. NCCL_* variables do.
But MPI must deliver those NCCL_* variables and LD_LIBRARY_PATH to every rank (section 7).
NCCL_DEBUG=INFO is the proof of what NCCL actually used: look for NET/IB : Using [0]mlx5_0:1/IB ..., channels via NET/IB/N/GDRDMA between nodes, and via P2P or NVLS inside a node. NET/Socket means it fell back to TCP.

7. Launching with Slurm: srun and sbatch

# Interactive: 2 nodes x 8 GPUs = 16 ranks
srun --mpi=pmix -N 2 --ntasks-per-node=8 --gpus-per-node=8 \
     ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

srun --mpi=list                            # MPI plugins this Slurm supports (look for pmix)
scontrol show config | grep -i mpi         # MpiDefault: what srun uses without --mpi
export SLURM_MPI_TYPE=pmix                 # same effect as --mpi=pmix for later srun calls
srun -N 8 -x dgx06 ...                     # exclude a node; -w dgx[01-04] picks nodes

A batch job for the same test:

#!/bin/bash
#SBATCH --job-name=nccl-allreduce
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --exclusive
#SBATCH --output=%x-%j.out
module load cuda nccl openmpi
srun --mpi=pmix /shared/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

--mpi=pmix starts a PMIx server so the Open MPI ranks can find each other. If the cluster's MpiDefault is none and you leave it off, Open MPI ranks fail at startup with a message saying it "was not built with SLURM's PMI support". The message blames the build; the usual fix is just --mpi=pmix.
--ntasks-per-node=8 gives one rank per GPU; Slurm numbers ranks node by node (ranks 0–7 on the first node, 8–15 on the second), which is what NCCL_TESTS_SPLIT recipes assume.
--gpus-per-node=8 is required: Slurm hides GPUs you didn't ask for, and the test fails with "no CUDA-capable device is detected".
srun passes your whole environment to the tasks by default, so module paths and NCCL_* settings arrive on every node.
Drained nodes take no new jobs, so you can't srun on them. To test a drained node, ssh to it and run the binary directly with -g 8 (or use a reservation).
With Pyxis/Enroot, add --container-image=<image> and --container-mounts=/shared:/shared; the container then supplies CUDA, NCCL, and often a prebuilt nccl-tests.

8. Launching with Open MPI: mpirun

# hostfile: one line per node, with the number of ranks it may run
dgx01 slots=8
dgx02 slots=8
dgx03 slots=8
dgx04 slots=8

mpirun -np 32 --hostfile hosts --map-by ppr:8:node --bind-to numa \
       -x LD_LIBRARY_PATH -x PATH -x NCCL_DEBUG=INFO -x NCCL_IB_HCA \
       ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

mpirun -np 32 --hostfile hosts --map-by ppr:8:node hostname | sort | uniq -c   # check placement first

Option	What it does
`-np N`	Total number of processes (ranks with `-g 1`)
`--hostfile f` / `-H a:8,b:8`	Which hosts, and how many slots each. A hostfile line without `slots=` counts the host's CPU cores; an `-H` host without `:N` counts as 1.
`--map-by ppr:8:node` (or `-N 8`)	Place exactly 8 processes per node
`--bind-to numa`	Keep each process on CPU cores near its GPU and NIC
`-x NAME` / `-x NAME=value`	Export a variable to every rank. Ranks started over ssh get a fresh login environment, not your shell's.
`--mca pml ucx`	MPI's own transport (HPC-X default). Doesn't change NCCL's data path.
`--report-bindings`	Print where each rank landed and what it's bound to

With a hostfile that has no slots=, recent Open MPI treats each host as having as many slots as CPU cores, so -np 32 fills the first host with all 32 ranks. On an 8-GPU node, ranks 8–31 then fail with "invalid device ordinal". Fix the hostfile (slots=8) or add --map-by ppr:8:node.

You exported NCCL_DEBUG=INFO and loaded the NCCL module, yet remote ranks print nothing, or fail with "libnccl.so.2: cannot open shared object file". mpirun over ssh doesn't forward your environment. Pass what the ranks need: -x LD_LIBRARY_PATH -x NCCL_DEBUG and any other NCCL_* settings, including NCCL_TESTS_SPLIT.

	srun	mpirun over ssh
Who places ranks	Slurm (`-N`, `--ntasks-per-node`)	Open MPI (hostfile slots, `--map-by`)
Environment on ranks	Your whole environment by default	A fresh one plus whatever you name with `-x`
Wire-up	`--mpi=pmix`	Built in
GPUs	Only what you request (`--gpus-per-node`)	All GPUs on the host
Respects drained nodes and other jobs	Yes	No: it can land on busy or drained nodes

Inside a Slurm allocation (salloc or an sbatch script) mpirun reads the node list and slots from Slurm, so no hostfile is needed, and its processes usually inherit the job's environment. -x for the NCCL_* variables is still a good habit. On DGX and Base Command clusters, Open MPI often comes as HPC-X (module load hpcx).

9. Reading the output

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid   4108 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
#  ...
#  Rank 15 Group  0 Pid   4249 on      dgx02 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    23.02    0.00    0.00      0    22.91    0.00    0.00      0
  ...
  8589934592    2147483648     float     sum      -1  43312.4  198.32  371.85      0  43098.1  199.31  373.70      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 142.375

Item	What to check
`nThread`, `nGpus`	Your `-t` and `-g`. With the rank list, they confirm the layout.
`Rank` lines	One per GPU: total count, host, device index. A list with only rank 0, repeated per process, means a non-MPI build or broken wire-up.
`Group`	The split group the rank belongs to (all 0 without `NCCL_TESTS_SPLIT`)
`time`, `algbw`, `busbw`	Microseconds per operation; size ÷ time; algbw times the collective's factor. Judge busbw at large sizes.
`#wrong`	Must be 0. Anything else means corrupted data: hardware, not tuning.
out-of-place / in-place	Separate output buffer vs overwrite the input. They should match closely.
`Avg bus bandwidth`	The average over every size, including tiny latency-bound ones. Fine for comparing identical runs; misleading as a health number.

More on algbw vs busbw: NCCL deep dive, section 8.

10. What "good" looks like on DGX H100

Test	Healthy large-message result	What it exercises
One node, 8 GPUs, AllReduce	About 450–480 GB/s busbw (NVLS on)	NVLink, NVSwitch, Fabric Manager
One node, 8 GPUs, AllGather / ReduceScatter	About 350–370 GB/s	NVLink, NVSwitch
Many nodes, 8 GPUs each, AllReduce	About 350–400 GB/s (8 × 400 Gb/s NDR per node)	Everything
Per-rail split: 1 GPU per node, one NIC each	About 45–48 GB/s (400 Gb/s = 50 GB/s raw)	One rail: its NICs, cables, and leaf switch
`ib_write_bw`, one NDR port	About 390 Gb/s	One wire, no GPUs
NCCL fell back to sockets	Single-digit GB/s	Something's wrong: no IB, or `NCCL_IB_DISABLE=1`

Small messages are limited by latency (tens of microseconds), so their busbw is near zero. That's normal.
Exact numbers vary with the NCCL version and fabric. What matters: large-message busbw close to your own baseline, the same on every node, and #wrong 0.
A drop to about half usually means one component at half speed (a link trained at 2x width, or a GPU with half its NVLinks down): collectives run at the pace of the slowest participant.

Large-message busbw within a few percent of the baseline on every node and every rail, #wrong 0, and NCCL_DEBUG=INFO showing NET/IB with GDRDMA between nodes.

11. Splitting ranks: NCCL_TESTS_SPLIT

A normal run puts every GPU in one AllReduce, and a collective runs at the pace of its slowest member. A low number tells you something is slow, not what. A split runs several smaller AllReduces at the same time, each built to use only one kind of path, so you can see which kind is slow.

Each node is a room with 8 desks (GPUs 0–7). Desks in the same room talk over the intercom: NVLink through the NVSwitch chips on the GPU baseboard. Rooms talk only over phone lines, and desk N in every room has its own line N: GPU N's NIC, on network rail N.
No split = one conference call with every desk in the building. If it sounds bad, you can't tell whether an intercom or a phone line is at fault.
DIV 8 = one meeting per room. Only the intercom is used, and every room meets at once.
MOD 8 = one call per phone line: desk 3 of every room joins call 3, and so on. Each call uses only its line; there's one person per room on it, so the intercom is never used.
OR 1 = desk buddies: desks 0+1, 2+3, 4+5, 6+7 in each room. Smaller intercom meetings that narrow a bad room down to a pair of GPUs.

How a rank picks its group

Ranks are numbered node by node: rank = node × 8 + local GPU (node 0 has ranks 0–7, node 1 has 8–15, ...). Each rank applies the operation to its own number to get a color; ranks with the same color form one group.

NCCL_TESTS_SPLIT="DIV 8"     # color = floor(rank / 8)  -> the node number     -> per node (NVLink)
NCCL_TESTS_SPLIT="MOD 8"     # color = rank mod 8       -> the local GPU index -> per rail (network)
NCCL_TESTS_SPLIT="AND 7"     # color = rank AND 7       -> same as MOD 8 when there are 8 GPUs per node
NCCL_TESTS_SPLIT_MASK=0x7    # older spelling of AND 7
NCCL_TESTS_SPLIT="OR 1"      # color = rank OR 1        -> pairs 0+1, 2+3, ... inside each node

Worked example, rank 11 (node 1, GPU 3): DIV 8 gives 1 (the node-1 group), MOD 8 gives 3 (the rail-3 group), OR 1 gives 11 (paired with rank 10).

See the groups

Three nodes, 8 GPUs each. Cells with the same color run one AllReduce together.

What each split tests

Healthy numbers assume DGX H100: NVLink 4 through NVSwitch inside a node, one NDR 400 Gb/s NIC per GPU.

Setting	Groups	Path tested	Healthy busbw (AllReduce, large messages)
none	1 group of all ranks	Everything together	350–400 GB/s
`"DIV 8"`	One per node	Per node: the 8 GPUs of one node over NVLink/NVSwitch only. Every node at once.	About 450–480 GB/s
`"MOD 8"`, `"AND 7"`, `MASK=0x7`	One per rail: GPU N of every node	Per rail: one GPU and one NIC per node, so all traffic crosses that rail. Isolates the network.	About 45–48 GB/s (400 Gb/s = 50 GB/s raw)
`"OR 1"` (same groups as `"DIV 2"`)	GPU pairs inside each node	Pairs over NVLink: narrows a slow node to the pair that holds the bad GPU or link	Compare the pairs with each other; a 2-GPU number differs from the 8-GPU one
`"DIV 16"`	One per pair of nodes	Node pairs: bisect a big cluster in one run	Like a 2-node run

The per-rail number is low on purpose: each group has one 400 Gb/s NIC per node, so 46 GB/s is healthy. Don't compare it with the 370 GB/s of a full run, which spreads traffic over all 8 NICs plus NVLink.

What the result tells you, and what it doesn't

One table for the whole job. Each group times itself, then the times are combined the way -a says: by default the average, so one slow group of eight only dents the number. Add -a 3 (use the slowest time) to make it obvious.
It tells you the kind of path, not which one. A low MOD 8 result means "one rail is slow", not "rail 3 is slow". nccl-tests prints only times, bandwidth, and #wrong; it never reports FEC or link errors. To find the rail or node, follow the isolation playbook: ibstat, mlxlink, and ib_write_bw per HCA for a rail; single-node runs for a node.
busbw uses the group size: in a per-rail split on 8 nodes, n = 8.
A split that leaves groups of one rank (for example "MOD 8" on a single node) gives AllReduce busbw 0: there's nothing to communicate with.
The variable must reach every rank: automatic with srun; with mpirun over ssh add -x NCCL_TESTS_SPLIT (or -x NCCL_TESTS_SPLIT_MASK).

DIV = "everyone in my room" = NVLink. MOD = "same desk in every room" = rails. OR 1 = "my desk buddy".

12. Interactive split calculator

Pick the job size and a split. The calculator lists the groups, what each one exercises, and the matching launch lines. Ranks are numbered node by node, as srun --ntasks-per-node and --map-by ppr:N:node do.

Nodes
Ranks (GPUs) per node
Split
Value

13. Isolation playbook

Confirm the layout. Rank count, hosts, and devices in the # Using devices block. Fix launcher problems before reading any bandwidth.
Confirm the transport. One run with NCCL_DEBUG=INFO: NET/IB with all eight compute HCAs and GDRDMA, not NET/Socket.
Inside the nodes: NCCL_TESTS_SPLIT="DIV 8" across all nodes, with -a 3. If the worst group is low, loop single-node runs (srun -N1 -w $n ... -g 8) to find the node, then nvidia-smi nvlink -s, nvidia-smi topo -m, Fabric Manager, and Xid 74 in dmesg.
Between the nodes: NCCL_TESTS_SPLIT="MOD 8" (or MASK=0x7), with -a 3. If it's low, check every node's HCAs: pdsh -w dgx[01-08] 'ibstat | grep -E "^CA|Rate"' | dshbak -c, then mlxlink on the odd port and ib_write_bw to a neighbor.
Bisect when nothing stands out: halves of the node list with -w, then quarters, or "DIV 16" for pairs.
Remove and prove. Drain the bad node with a reason someone can act on ("mlx5_5 at 2x width"), rerun without it (-x), and save the result as evidence.

Practice the whole playbook on a fault that only bisecting finds: lab ticket ISO-606.

From a slow rail to a fixed cable

Say NCCL_TESTS_SPLIT="MOD 8" with -a 3 comes in well under 45 GB/s while "DIV 8" is healthy. The network is at fault, but the result doesn't say which port.

Find the port. Compare every node: pdsh -w dgx[01-08] 'ibstat | grep -E "^CA|State|Rate"' | dshbak -c. The odd one out (Rate 200 instead of 400, or a port that isn't Active) is your suspect. On a real DGX H100 the compute HCAs aren't numbered mlx5_0–7 in order, so map GPU to HCA with nvidia-smi topo -m and ibdev2netdev. This guide's lab uses mlx5_N for rail N to keep it simple.
Confirm it with link telemetry. mlxlink -d mlx5_3 -p 1 -c shows width, speed, the error counters, and raw and effective BER (FEC corrections live here, not in nccl-tests). mlxlink -d mlx5_3 -p 1 --cable --ddm shows the module's optical power per lane; one weak lane points at a dirty or damaged connector. ib_write_bw -d mlx5_3 to a healthy neighbor puts a number on it.
Drain the node with a reason that names the port, so no job lands on it while you work.
Take the link down from the management side, for example mlxlink -d mlx5_3 -p 1 -a DN, or disable the matching switch port in UFM. Not ip link set dev ib3 down: that only stops the IPoIB interface, and the InfiniBand port stays Active.
Inspect, then clean. Unplug, and look at both the cable's MPO face and the module's receptacle with a fiber scope. Clean only what's dirty (dry click-cleaner first), inspect again, then reconnect until it latches. Treat every unplugged fiber as live: disabling one end doesn't stop the far end from transmitting, so never look into a connector, only at the scope screen.
Prove the fix. mlxlink -a UP, check for Active 4x at NDR, clear the counters and recheck BER after a soak, rerun the MOD 8 split with -a 3, and only then return the node to service. If the problem is still there, swap in a known-good cable or module.

The details of each step are in the mlxlink and physical handling sections of the transceiver guide.

Symptom	Most likely layer	First check
Output repeated per process, all "rank 0"	Non-MPI build or missing PMIx	`ldd` for libmpi; `--mpi=pmix`
"invalid device ordinal"	Rank placement or GPU visibility	`hostname \| sort \| uniq -c` under the launcher; `--gpus-per-node`
"libnccl.so.2: cannot open shared object file"	Environment on remote ranks	`-x LD_LIBRARY_PATH` or load the module in the job
Single-digit GB/s across nodes	Socket fallback	`NCCL_DEBUG=INFO`: NET/Socket? `NCCL_IB_DISABLE`?
About half the expected busbw	One slow participant	Split per node and per rail with `-a 3`
`#wrong` above 0	Hardware or data corruption	Isolate the pair or node; DCGM; link errors

14. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

make MPI=1 MPI_HOME=/opt/openmpi CUDA_HOME=/usr/local/cuda NCCL_HOME=/opt/ncclBuild nccl-tests with MPI support

make -C src build BUILDDIR=/shared/nccl-tests/build
make[1]: Entering directory '/shared/nccl-tests/src'
Compiling  timer.cc                            > /shared/nccl-tests/build/timer.o
Compiling /shared/nccl-tests/build/verifiable/verifiable.o
Compiling  all_reduce.cu                       > /shared/nccl-tests/build/all_reduce.o
Compiling  common.cu                           > /shared/nccl-tests/build/common.o
Linking  /shared/nccl-tests/build/all_reduce.o               > /shared/nccl-tests/build/all_reduce_perf
Compiling  all_gather.cu                       > /shared/nccl-tests/build/all_gather.o
Linking  /shared/nccl-tests/build/all_gather.o               > /shared/nccl-tests/build/all_gather_perf
...
Compiling  sendrecv.cu                         > /shared/nccl-tests/build/sendrecv.o
Linking  /shared/nccl-tests/build/sendrecv.o                 > /shared/nccl-tests/build/sendrecv_perf
make[1]: Leaving directory '/shared/nccl-tests/src'

What to look at: Every collective should end in a Linking line that produces a *_perf binary in build/. If you only see 'Nothing to be done', make reused an old build and MPI=1 did not take effect.

ldd build/all_reduce_perf | grep -E "mpi|nccl"Prove libmpi and the right libnccl

        libnccl.so.2 => /opt/nccl/2.21.5/lib/libnccl.so.2 (0x00007f3a41a00000)
        libmpi.so.40 => /opt/openmpi/4.1.7/lib/libmpi.so.40 (0x00007f3a418c1000)

What to look at: You want a libmpi.so line (MPI=1 worked) and libnccl.so.2 resolving to the NCCL you meant to test. A missing libmpi line means every rank will run its own one-rank test; 'not found' next to libnccl means LD_LIBRARY_PATH is wrong.

make MPI=1Rebuild without make clean

make -C src build BUILDDIR=/shared/nccl-tests/build
make[1]: Entering directory '/shared/nccl-tests/src'
make[1]: Nothing to be done for 'build'.
make[1]: Leaving directory '/shared/nccl-tests/src'

What to look at: 'Nothing to be done' after changing flags means the old non-MPI objects were kept. Run make clean first, then rebuild and confirm libmpi with ldd.

./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8One process, 8 GPUs, NVLink baseline

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 8 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid   4101 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
#  Rank  2 Group  0 Pid   4101 on      dgx01 device  2 [0x52] NVIDIA H100 80GB HBM3
#  Rank  3 Group  0 Pid   4101 on      dgx01 device  3 [0x61] NVIDIA H100 80GB HBM3
#  Rank  4 Group  0 Pid   4101 on      dgx01 device  4 [0x9d] NVIDIA H100 80GB HBM3
#  Rank  5 Group  0 Pid   4101 on      dgx01 device  5 [0xc3] NVIDIA H100 80GB HBM3
#  Rank  6 Group  0 Pid   4101 on      dgx01 device  6 [0xd1] NVIDIA H100 80GB HBM3
#  Rank  7 Group  0 Pid   4101 on      dgx01 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    11.45    0.00    0.00      0    11.51    0.00    0.00      0
          16             4     float     sum      -1    11.50    0.00    0.00      0    11.45    0.00    0.00      0
          32             8     float     sum      -1    11.55    0.00    0.00      0    11.53    0.00    0.00      0
          64            16     float     sum      -1    11.48    0.01    0.01      0    11.49    0.01    0.01      0
  ...
    16777216       4194304     float     sum      -1    72.67  230.88  404.04      0    72.34  231.92  405.86      0
    33554432       8388608     float     sum      -1   134.37  249.72  437.01      0   134.17  250.09  437.66      0
    67108864      16777216     float     sum      -1   255.66  262.50  459.37      0   256.04  262.10  458.68      0
   134217728      33554432     float     sum      -1   501.84  267.45  468.04      0   504.10  266.25  465.95      0
   268435456      67108864     float     sum      -1   986.21  272.19  476.33      0   981.77  273.42  478.48      0
   536870912     134217728     float     sum      -1  1968.84  272.68  477.20      0  1965.89  273.09  477.91      0
  1073741824     268435456     float     sum      -1  3941.89  272.39  476.69      0  3947.80  271.98  475.97      0
  2147483648     536870912     float     sum      -1  7825.19  274.43  480.26      0  7860.40  273.20  478.11      0
  4294967296    1073741824     float     sum      -1  15701.6  273.54  478.69      0  15630.9  274.77  480.85      0
  8589934592    2147483648     float     sum      -1  31203.7  275.29  481.75      0  31156.8  275.70  482.47      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 184.488
#
# Collective test concluded: all_reduce_perf

What to look at: Run on dgx01 itself: one Pid drives all 8 GPUs, and busbw at the 8 GB line should be about 470-480 GB/s on NVLink with NVLS; below about 400 GB/s, look at NVLink, Fabric Manager or P2P settings. Judge health by the large sizes, because the Avg bus bandwidth line averages in the tiny latency-bound sizes.

grep -E "Rank|nThread" slurm-7001.out | head -6Spot 16 separate one-rank tests

# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#  Rank  0 Group  0 Pid   4108 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#  Rank  0 Group  0 Pid   5212 on      dgx02 device  0 [0x1b] NVIDIA H100 80GB HBM3

What to look at: A healthy 2-node job prints one header listing Rank 0 to Rank 15. Many headers that each say Rank 0 on device 0, with busbw 0.00, mean the binary was built without MPI=1, so every process ran alone.

srun -N2 --ntasks-per-node=8 --gpus-per-node=8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1MPI build launched without PMIx

--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.
  ...
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[dgx01:04101] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: dgx01: tasks 0-7: Exited with exit code 1
srun: error: dgx02: tasks 8-15: Exited with exit code 1

What to look at: This message means srun started the ranks without a PMIx server, so Open MPI cannot wire them together. Add --mpi=pmix (or export SLURM_MPI_TYPE=pmix); it is not a rebuild problem.

scontrol show config | grep -i mpiSlurm's default MPI plugin

MpiDefault              = none
MpiParams               = (null)

What to look at: MpiDefault = none means a plain srun starts no PMIx server, so MPI builds need --mpi=pmix. A cluster with MpiDefault = pmix is why the same command works there without the flag.

srun --mpi=listMPI plugins this Slurm supports

MPI plugin types are...
        cray_shasta
        none
        pmi2
        pmix
specific pmix plugin versions available: pmix_v4

What to look at: Look for pmix in the list. If it is missing, Slurm was built without PMIx and an admin has to fix that; no srun flag will help.

srun --mpi=pmix -N 2 --ntasks-per-node=8 --gpus-per-node=8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 116 ranks across 2 nodes

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid   4108 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
#  Rank  2 Group  0 Pid   4115 on      dgx01 device  2 [0x52] NVIDIA H100 80GB HBM3
#  ...
#  Rank  8 Group  0 Pid   5212 on      dgx02 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  ...
#  Rank 15 Group  0 Pid   5261 on      dgx02 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    22.91    0.00    0.00      0    23.01    0.00    0.00      0
          16             4     float     sum      -1    23.00    0.00    0.00      0    22.90    0.00    0.00      0
          32             8     float     sum      -1    23.09    0.00    0.00      0    23.06    0.00    0.00      0
          64            16     float     sum      -1    22.95    0.00    0.01      0    22.99    0.00    0.01      0
  ...
    16777216       4194304     float     sum      -1   107.34  156.31  293.07      0   106.85  157.01  294.40      0
    33554432       8388608     float     sum      -1   192.44  174.36  326.93      0   192.15  174.63  327.42      0
    67108864      16777216     float     sum      -1   359.62  186.61  349.89      0   360.16  186.33  349.37      0
   134217728      33554432     float     sum      -1   699.08  191.99  359.98      0   702.23  191.13  358.37      0
   268435456      67108864     float     sum      -1  1366.88  196.38  368.22      0  1360.73  197.27  369.89      0
   536870912     134217728     float     sum      -1  2721.75  197.25  369.85      0  2717.67  197.55  370.40      0
  1073741824     268435456     float     sum      -1  5442.18  197.30  369.94      0  5450.34  197.00  369.38      0
  2147483648     536870912     float     sum      -1  10796.4  198.91  372.95      0  10844.9  198.02  371.28      0
  4294967296    1073741824     float     sum      -1  21656.2  198.32  371.86      0  21558.8  199.22  373.54      0
  8589934592    2147483648     float     sum      -1  43030.2  199.63  374.30      0  42965.6  199.93  374.86      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 136.907
#
# Collective test concluded: all_reduce_perf

What to look at: One header with Rank 0 to Rank 15 on two hosts proves a single 16-rank communicator. busbw near 370 GB/s at 8 GB is healthy for 8x 400G NDR rails; #wrong must be 0 on every row.

sbatch nccl-allreduce.sbatchSubmit the batch acceptance job

Submitted batch job 7004

What to look at: Note the job ID: with --output=%x-%j.out the results land in nccl-allreduce-7004.out. Check that file's header for 16 ranks and its 8 GB busbw, not just that the job finished.

mpirun -np 32 --hostfile hosts --map-by ppr:8:node hostname | sort | uniq -cCheck rank placement before testing

      8 dgx01
      8 dgx02
      8 dgx03
      8 dgx04

What to look at: Exactly 8 per node is what you want. If all 32 land on dgx01, the hostfile has no slots= and Open MPI counted CPU cores, so the '4-node' test never leaves one node.

mpirun -np 32 --hostfile /shared/hostfiles/rack1 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1Remote ranks missing LD_LIBRARY_PATH

./build/all_reduce_perf: error while loading shared libraries: libnccl.so.2: cannot open shared object file: No such file or directory
./build/all_reduce_perf: error while loading shared libraries: libnccl.so.2: cannot open shared object file: No such file or directory
...
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[41823,1],8]
  Exit code:    127
--------------------------------------------------------------------------

What to look at: Exit code 127 with libnccl.so.2 not found on remote ranks means mpirun over ssh did not pass your module's library path. Add -x LD_LIBRARY_PATH (and -x PATH).

mpirun -np 32 --hostfile /shared/hostfiles/rack1 -x LD_LIBRARY_PATH ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1Too many ranks on one node

dgx01: Test CUDA failure common.cu:1103 'invalid device ordinal'
 .. dgx01 pid 4109: Test failure common.cu:1027
dgx01: Test CUDA failure common.cu:1103 'invalid device ordinal'
 .. dgx01 pid 4110: Test failure common.cu:1027
...
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[41907,1],8]
  Exit code:    2
--------------------------------------------------------------------------

What to look at: Local rank 8 or higher on dgx01 asked for a GPU index that does not exist, because all 32 ranks were placed on one host. Fix placement with --map-by ppr:8:node or slots=8, then check it with hostname | sort | uniq -c.

mpirun -np 32 --hostfile hosts --map-by ppr:8:node --bind-to numa -x LD_LIBRARY_PATH -x PATH -x NCCL_DEBUG=INFO -x NCCL_IB_HCA ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

4-node acceptance run with NCCL INFO

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  ...
#  Rank 31 Group  0 Pid   7483 on      dgx04 device  7 [0xdf] NVIDIA H100 80GB HBM3
dgx01:4101:4101 [0] NCCL INFO Bootstrap : Using bond0:10.0.0.11<0>
dgx01:4101:4101 [0] NCCL INFO NCCL version 2.21.5+cuda12.4
dgx01:4101:4152 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB bond0:10.0.0.11<0>
dgx01:4101:4152 [0] NCCL INFO Using network IB
dgx01:4101:4152 [0] NCCL INFO comm 0x55d1c8a0000 rank 0 nranks 32 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x5c1d0ae2 - Init START
dgx01:4101:4152 [0] NCCL INFO Channel 00/0 : 31[7] -> 0[0] [receive] via NET/IB/0/GDRDMA
dgx01:4101:4152 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM/read
dgx01:4101:4152 [0] NCCL INFO Connected all rings
dgx01:4101:4152 [0] NCCL INFO comm 0x55d1c8a0000 rank 0 nranks 32 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x5c1d0ae2 - Init COMPLETE
...
dgx02:5212:5212 [0] NCCL INFO Bootstrap : Using bond0:10.0.0.12<0>
dgx02:5212:5263 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB bond0:10.0.0.12<0>
dgx02:5212:5263 [0] NCCL INFO Channel 00/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA
dgx02:5212:5263 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] via P2P/CUMEM/read
dgx02:5212:5263 [0] NCCL INFO comm 0x55d1c8a0800 rank 8 nranks 32 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x5c1d0ae2 - Init COMPLETE
...
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    29.88    0.00    0.00      0    30.01    0.00    0.00      0
          16             4     float     sum      -1    30.00    0.00    0.00      0    29.87    0.00    0.00      0
  ...
  1073741824     268435456     float     sum      -1  5644.89  190.21  368.54      0  5653.36  189.93  367.99      0
  2147483648     536870912     float     sum      -1  11192.4  191.87  371.75      0  11242.7  191.01  370.08      0
  4294967296    1073741824     float     sum      -1  22444.4  191.36  370.76      0  22343.4  192.23  372.44      0
  8589934592    2147483648     float     sum      -1  44590.2  192.64  373.24      0  44523.3  192.93  373.80      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 132.556
#
# Collective test concluded: all_reduce_perf

What to look at: Every node should print NET/IB : Using with all eight compute HCAs, and the inter-node channels should say via NET/IB/<n>/GDRDMA. NET/Socket, a single HCA, or NET/IB without /GDRDMA explains low busbw; healthy here is about 370 GB/s at 8 GB with #wrong 0.

srun --mpi=pmix -N8 --ntasks-per-node=8 --gpus-per-node=8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 18-node AllReduce with one bad rail

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid   4108 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
#  ...
#  Rank 63 Group  0 Pid  11927 on      dgx08 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    44.82    0.00    0.00      0    45.02    0.00    0.00      0
          16             4     float     sum      -1    45.00    0.00    0.00      0    44.80    0.00    0.00      0
  ...
   268435456      67108864     float     sum      -1  2874.76   93.38  183.84      0  2861.82   93.80  184.67      0
   536870912     134217728     float     sum      -1  5727.61   93.73  184.54      0  5719.01   93.87  184.82      0
  1073741824     268435456     float     sum      -1  11455.9   93.73  184.53      0  11473.0   93.59  184.25      0
  2147483648     536870912     float     sum      -1  22729.9   94.48  186.00      0  22832.2   94.06  185.17      0
  4294967296    1073741824     float     sum      -1  45596.9   94.19  185.45      0  45391.7   94.62  186.28      0
  8589934592    2147483648     float     sum      -1  90602.8   94.81  186.65      0  90466.9   94.95  186.94      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 68.906
#
# Collective test concluded: all_reduce_perf

What to look at: Read the 8 GB line: about 185 GB/s is half the roughly 370 GB/s baseline, so something is slow. Ignore Avg bus bandwidth for health; it is dragged down by small sizes even on a healthy cluster.

NCCL_TESTS_SPLIT="DIV 8" srun --mpi=pmix -N8 --ntasks-per-node=8 --gpus-per-node=8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 -a 3

Per-node groups: NVLink only

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid   4108 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
#  ...
#  Rank  7 Group  0 Pid   4150 on      dgx01 device  7 [0xdf] NVIDIA H100 80GB HBM3
#  Rank  8 Group  1 Pid   5212 on      dgx02 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  9 Group  1 Pid   5219 on      dgx02 device  1 [0x43] NVIDIA H100 80GB HBM3
#  ...
#  Rank 63 Group  7 Pid  11927 on      dgx08 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    11.95    0.00    0.00      0    12.01    0.00    0.00      0
          16             4     float     sum      -1    12.00    0.00    0.00      0    11.95    0.00    0.00      0
  ...
   268435456      67108864     float     sum      -1  1007.45  266.45  466.29      0  1002.91  267.66  468.40      0
   536870912     134217728     float     sum      -1  2010.99  266.97  467.20      0  2007.97  267.37  467.90      0
  1073741824     268435456     float     sum      -1  4026.01  266.70  466.73      0  4032.05  266.30  466.03      0
  2147483648     536870912     float     sum      -1  7991.93  268.71  470.24      0  8027.90  267.50  468.13      0
  4294967296    1073741824     float     sum      -1  16035.9  267.83  468.71      0  15963.7  269.05  470.83      0
  8589934592    2147483648     float     sum      -1  31867.8  269.55  471.71      0  31820.0  269.95  472.42      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 180.174
#
# Collective test concluded: all_reduce_perf

What to look at: The Group column shows 8 groups, one per node, and -a 3 reports the slowest group's time. About 470 GB/s means every node is healthy inside; a low number here points at one node's NVLink, NVSwitch or Fabric Manager.

NCCL_TESTS_SPLIT_MASK=0x7 srun --mpi=pmix -N8 --ntasks-per-node=8 --gpus-per-node=8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 -a 3

Per-rail groups: find the slow rail

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  1 Pid   4108 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
#  ...
#  Rank  7 Group  7 Pid   4150 on      dgx01 device  7 [0xdf] NVIDIA H100 80GB HBM3
#  Rank  8 Group  0 Pid   5212 on      dgx02 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  9 Group  1 Pid   5219 on      dgx02 device  1 [0x43] NVIDIA H100 80GB HBM3
#  ...
#  Rank 63 Group  7 Pid  11927 on      dgx08 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    27.89    0.00    0.00      0    28.01    0.00    0.00      0
          16             4     float     sum      -1    28.00    0.00    0.00      0    27.88    0.00    0.00      0
  ...
   268435456      67108864     float     sum      -1  20195.3   13.29   23.26      0  20104.4   13.35   23.37      0
   536870912     134217728     float     sum      -1  40524.7   13.25   23.18      0  40463.9   13.27   23.22      0
  1073741824     268435456     float     sum      -1  81345.5   13.20   23.10      0  81467.6   13.18   23.06      0
  2147483648     536870912     float     sum      -1 161690.9   13.28   23.24      0 162418.5   13.22   23.14      0
  4294967296    1073741824     float     sum      -1 324649.8   13.23   23.15      0 323188.9   13.29   23.26      0
  8589934592    2147483648     float     sum      -1 645383.7   13.31   23.29      0 644415.7   13.33   23.33      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 11.227
#
# Collective test concluded: all_reduce_perf

What to look at: Each group is one local GPU index and its NIC on all 8 nodes, so busbw is per rail; healthy is about 46 GB/s for a 400G link. The slowest group at about 23 GB/s means one rail runs at half speed, so compare ibstat Rate across nodes next.

pdsh -w dgx[01-08] 'ibstat | grep -E "^CA|Rate"' | dshbak -cCompare HCA link rates across nodes

----------------
dgx[01-05,07-08]
----------------
CA 'mlx5_0'
        Rate: 400
CA 'mlx5_1'
        Rate: 400
...
CA 'mlx5_5'
        Rate: 400
CA 'mlx5_6'
        Rate: 400
CA 'mlx5_7'
        Rate: 400
CA 'mlx5_8'
        Rate: 200
CA 'mlx5_9'
        Rate: 200
----------------
dgx06
----------------
CA 'mlx5_0'
        Rate: 400
...
CA 'mlx5_4'
        Rate: 400
CA 'mlx5_5'
        Rate: 200
CA 'mlx5_6'
        Rate: 400
CA 'mlx5_7'
        Rate: 400
CA 'mlx5_8'
        Rate: 200
CA 'mlx5_9'
        Rate: 200

What to look at: dshbak -c groups identical output, so a node that differs gets its own block. dgx06 shows mlx5_5 at Rate 200 while every other compute port is 400; mlx5_8 and mlx5_9 are the 200G storage ports and are expected to be 200.

sudo mlxlink -d mlx5_5Link state, speed and width

Operational Info
----------------
State                              : Active
Physical state                     : LinkUp
Speed                              : IB-NDR
Width                              : 2x
FEC                                : Standard_RS-FEC - (544,514)
Loopback Mode                      : No Loopback
Auto Negotiation                   : ON

Supported Info
--------------
Enabled Link Speed                 : 0x000000f0 (NDR,HDR,EDR,FDR)
Supported Cable Speed              : 0x000000f0 (NDR,HDR,EDR,FDR)

Troubleshooting Info
--------------------
Status Opcode                      : 0
Group Opcode                       : N/A
Recommendation                     : No issue was observed

Tool Information
----------------
Firmware Version                   : 28.39.1002
amBER Version                      : 3.4
MFT Version                        : mft 4.26.1-3

What to look at: The link is Active at NDR speed but Width is 2x instead of 4x, so only half the lanes trained and the port runs at 200 Gb/s. The Recommendation line can still say no issue, so read Width yourself; reseat or replace the cable or optic.

ib_write_bw -d mlx5_5 --report_gbits dgx07RDMA bandwidth on one rail

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_5
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x2e QPN 0x0127 PSN 0x3a1f2c RKey 0x1fffbd VAddr 0x007f5c3a200000
 remote address: LID 0x35 QPN 0x0129 PSN 0x8c42e1 RKey 0x1fffbe VAddr 0x007f91d4600000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      5000             196.42             196.18              0.374180
---------------------------------------------------------------------------------------

What to look at: BW average is the number to read: a healthy NDR port gives about 390 Gb/s, so about 196 Gb/s confirms the half-width link on mlx5_5. Run the same test on a good rail (for example mlx5_4) for comparison; ib_write_bw needs a server started on dgx07 first.

15. Hands-on lab: six simulated tickets

A mock Slurm cluster: a login node (login01) and eight DGX H100 nodes (dgx01–dgx08) with 8 GPUs and 8 InfiniBand rails each (GPU N uses HCA mlx5_N), on a shared filesystem. Pick a ticket, start with cat notes.txt, and type help for the commands on that host. Results follow from what you type: the build you made, how ranks are placed, which environment reaches them, the split you chose, and the state of each GPU and link.

NCCL-101: build nccl-tests with MPI and record a single-node baseline (guided; start here).
INC-202: a 2-node job printed 16 separate results. Fix the build and the Slurm launch.
ACC-303: a customer acceptance run with mpirun over ssh: placement, environment, and evidence.
NET-404: 8-node AllReduce at half speed. Split per node and per rail, find the bad port, drain, prove.
NVL-505: 8-node AllReduce slow again, for a different reason. Find the node and fix it.
ISO-606: the whole isolation playbook, end to end. Every per-rail and per-node check looks clean, so you have to bisect.

Outputs are simplified mock-ups modeled on how the real tools behave, not exact copies, and every name, address, and number is a placeholder. The lab skips queue waits: jobs run as soon as you submit them. Up/Down arrows recall history; Tab completes commands, paths, and hosts. Supported shell features: $VAR, export, VAR=value command, pipes to grep, head, tail, wc, sort, uniq, tee, sed, and dshbak; > and >>; ; and &&; simple for loops.

Ticket

Ticket checklist

Answer key for this ticket (spoilers)

One clean path. Some steps fail on purpose so you see the real error; the note says so. Other orders work too: the checklist tracks results, not exact commands.

16. Scenarios

17. Searchable reference

Item	Meaning	When it matters

18. Flashcards (click to flip)

19. Exam traps

No MPI=1, no multi-node test. N processes run N one-rank tests; AllReduce busbw 0.00.
make ignores flag changes. make clean before rebuilding; verify with ldd.
srun needs --mpi=pmix (or SLURM_MPI_TYPE=pmix) for an Open MPI build when MpiDefault is none.
srun needs --gpus-per-node=8: no GPUs requested, no GPUs visible.
mpirun over ssh doesn't forward your environment. -x LD_LIBRARY_PATH, -x NCCL_DEBUG, -x NCCL_TESTS_SPLIT.
Hostfile slots: no slots= means CPU cores, so ranks pack onto the first host. Use slots=8 or --map-by ppr:8:node.
nranks = processes × -t × -g. -g 8 with 8 ranks per node asks for 64 GPUs per node.
MPI starts processes; NCCL moves the data. MPI transport flags rarely change busbw.
Judge busbw at large sizes, not Avg bus bandwidth.
#wrong above 0 is a hardware problem, never a tuning one.
MOD 8 = per rail (network); DIV 8 = per node (NVLink). MASK 0x7 = AND 7 = MOD 8 for 8 GPUs per node.
Split results are one combined table; -a 3 shows the slowest group. They show that a rail is slow, never which one, and never FEC errors; use ibstat and mlxlink for that.
ip link set ibX down doesn't take an InfiniBand port down; it stops only IPoIB. Use mlxlink -a DN or the switch side, and treat every fiber as live.
A healthy per-rail number is about 46 GB/s, not 370. One GPU and one NIC per node.
Drained nodes can't take srun jobs; test them over ssh with -g 8.

20. Practice quiz (26 questions)

Score: 0 / 0