NCCL Deep Dive

The NVIDIA Collective Communications Library: how GPUs talk to each other, how to test it, how to read its logs, and how to fix it.

1. What NCCL is (and isn't)

NCCL (pronounced "nickel") is a library that moves data between GPUs, both inside one server and across servers, using the fastest path available. Deep learning frameworks call it to synchronize GPUs during distributed training.

NCCL is…	NCCL is NOT…
A communication library (not a daemon or service) Used by PyTorch (DDP, FSDP), Megatron-LM, DeepSpeed, JAX, TensorFlow, Horovod Topology-aware: it detects NVLink, PCIe, NICs, and NUMA, and picks the best paths Configured mainly through *`NCCL_` environment variables**	A monitoring tool (that's DCGM) A scheduler (Slurm / Kubernetes) The thing that configures NVSwitch (Fabric Manager) The InfiniBand subnet manager (UFM / OpenSM) MPI. MPI is often used to launch NCCL jobs, but NCCL does the GPU data movement.

NCCL is a smart courier company for GPUs. You say "everyone needs everyone's totals," and it decides which roads to use (NVLink, PCIe, InfiniBand), how many trucks to send in parallel (channels), and which delivery route to follow (ring or tree).

2. The collective operations

A collective is an operation that every GPU in a group takes part in. Learn these five, plus point-to-point:

Operation	Before → After (4 GPUs, each starting with a value)	Where it's used
AllReduce	Each GPU has A, B, C, D → every GPU has A+B+C+D	Gradient sync in data-parallel training (DDP). The most important one.
Broadcast	GPU 0 has X → every GPU has X	Sending initial model weights from one rank
Reduce	Each has A, B, C, D → only one GPU (the root) has the sum	Collecting a total in one place
AllGather	Each has one piece → every GPU has all pieces [A, B, C, D]	FSDP / ZeRO (gather sharded weights)
ReduceScatter	Each has full arrays → arrays are summed, and each GPU keeps one slice of the sum	FSDP / ZeRO (shard gradients)
Send / Recv (point-to-point), AllToAll	Specific GPU-to-GPU transfers. AllToAll = everyone sends a different piece to everyone else.	Pipeline parallelism, Mixture-of-Experts

AllReduce = ReduceScatter + AllGather. That's how the ring algorithm actually implements it.

Background: what a training step does, and what "sharded" means

The table above only makes sense once you know what the GPUs are passing around. One training step has three phases:

Forward pass: push a batch of data through the model's layers to get a prediction and a loss (how wrong it was).
Backward pass: work backwards through the layers to compute a gradient for every weight. The gradient is a number per weight saying "nudge me this way to reduce the loss". Gradients are the same size as the weights.
Optimizer step: use the gradients to update the weights. Optimizers like Adam also keep extra state per weight (running averages). In mixed-precision training, weights, gradients, and optimizer state add up to about 16 bytes per parameter.

With many GPUs, each one works on a different slice of the data. They must still agree on one set of weights, and that agreement is what the collectives are for.

Sharding means splitting something into pieces and giving each GPU only one piece. A shard is one piece. "Sharded weights" means no GPU holds the whole model: with 8 GPUs, each stores 1/8 of every layer's weights.

A study group of 8 shares one huge textbook. Instead of everyone buying the whole book (data parallel), each person keeps one chapter-slice (sharding). When the group is about to study chapter 5, everyone photocopies their part of chapter 5 for everyone else (AllGather). When they finish, they throw the copies away to save shelf space, and keep only their own slice.

Why bother? A 70B-parameter model at 16 bytes per parameter needs about 1.1 TB for weights, gradients, and optimizer state. One H100 has 80 GB. Sharded across 64 GPUs, each holds about 17.5 GB, which fits with room for activations. That's what FSDP (PyTorch Fully Sharded Data Parallel) and ZeRO stage 3 (DeepSpeed) do. The next section explains those names.

Who's who: PyTorch, DDP, ZeRO, DeepSpeed, FSDP, Megatron-LM

These names come up constantly, and they're a mix of libraries (code you install) and techniques (ways of splitting the work). None of them move bytes themselves. Under the hood they all call NCCL to do the collectives.

Name	What it is	Who makes it	The one-line idea
PyTorch	Library (deep learning framework)	Meta / PyTorch Foundation	The most common framework for writing and training models. Its `torch.distributed` module is what calls NCCL.
DDP (DistributedDataParallel)	Technique, built into PyTorch	PyTorch	Every GPU keeps a full copy of the model, works on different data, and AllReduces gradients.
ZeRO (Zero Redundancy Optimizer)	Technique (stages 1, 2, 3)	Microsoft (from the DeepSpeed team)	Stop every GPU storing identical copies. Shard them instead.
DeepSpeed	Library, add-on to PyTorch	Microsoft	The library where ZeRO was first implemented, plus other large-model training tricks.
FSDP (Fully Sharded Data Parallel)	Technique, built into PyTorch	PyTorch (Meta)	PyTorch's own version of ZeRO stage 3. Same idea, different library.
Megatron-LM / Megatron-Core	Library for training very large transformers	NVIDIA	Split the model itself: tensor parallelism (inside each layer) plus pipeline parallelism (across layers). NVIDIA's NeMo framework builds on it.

DDP: the simple, default approach

Imagine 8 GPUs and a model that fits comfortably on one GPU.

Every GPU loads the whole model, and rank 0 broadcasts its weights so all copies start identical.
The training data is split 8 ways. Each GPU runs the forward and backward pass on its own share.
Each GPU now has different gradients. DDP AllReduces them so every GPU has the same averaged gradients.
Every GPU applies the same update, and the 8 copies stay identical. Repeat.

Benefit: 8 GPUs get through the data about 8× faster. Limit: the whole model plus its gradients and optimizer state must fit on one GPU. At about 16 bytes per parameter, an 80 GB H100 tops out at a few billion parameters (less once activations are counted). Beyond that, DDP simply runs out of memory.

ZeRO: notice the waste, then remove it

With DDP on 8 GPUs, there are 8 identical copies of the weights, the gradients, and the optimizer state. Microsoft's ZeRO idea: each GPU keeps only 1/8 of each, and fetches the rest from its peers only when needed. It comes in three stages, each sharding one more thing:

Stage	What gets sharded	Memory saving	Communication vs DDP
ZeRO-1	Optimizer state (the biggest part, about 12 of the 16 bytes)	Large	About the same
ZeRO-2	+ gradients (ReduceScatter instead of AllReduce)	Larger	About the same
ZeRO-3	+ the weights themselves (AllGather each layer when it's needed)	About n×, the full saving	About 1.5×

DeepSpeed is the Microsoft library that implements ZeRO. You wrap a normal PyTorch model with it and pick a stage in a config file.

FSDP: PyTorch's built-in ZeRO-3

FSDP does the same thing as ZeRO-3, built directly into PyTorch, so no extra library is needed. Each layer's weights are sharded. Before a layer runs, it's AllGathered, used, then freed. Its gradients are ReduceScattered so each GPU keeps only its slice. FSDP's sharding options map onto the ZeRO stages: full sharding is ZeRO-3, gradient-and-optimizer-only sharding is close to ZeRO-2. A "hybrid" mode shards inside each node (over fast NVLink) and keeps copies across nodes, which reduces traffic on the slower inter-node network.

DDP, ZeRO, and FSDP are all data parallelism: every GPU still runs the whole model on its own slice of data. ZeRO and FSDP just avoid storing the whole model on every GPU. When people say "FSDP or ZeRO-3", they mean the same strategy from two different libraries.

Megatron-LM: split the model's work, not just its storage

With very large models, sharding storage isn't enough. A single layer can be huge, and running every layer on every GPU is slow. NVIDIA's Megatron-LM splits the computation itself, in two ways:

Tensor parallelism (TP): each big matrix multiplication inside a layer is cut into pieces, and several GPUs work on the same layer at the same time, each computing part of the answer. The parts are combined with an AllReduce (or AllGather + ReduceScatter), several times per layer. That's so much communication that TP is kept inside one node on NVLink, typically TP = 8 on a DGX.
Pipeline parallelism (PP): the layers are split into stages on different groups of GPUs, like an assembly line. Activations move between stages with Send/Recv, which is light enough to cross nodes.

Megatron then adds data parallelism on top (often with ZeRO-style optimizer sharding). Using all three at once is called 3D parallelism, and it's how models with hundreds of billions of parameters are trained on thousands of GPUs. Megatron-Core is the reusable library version, and NVIDIA NeMo is the higher-level framework built on it. Both ship as NGC containers.

Example layout, 1,024 H100s (128 DGX nodes): TP = 8 inside each node, PP = 8 across 8 nodes, and data parallel = 16 copies of that 64-GPU arrangement (8 × 8 × 16 = 1,024).

A restaurant kitchen. DDP: every chef has the whole cookbook and cooks different orders, then they compare notes so everyone learns the same lessons. ZeRO / FSDP: the cookbook is split into pages among the chefs, and when someone needs a page they photocopy it from whoever has it. Tensor parallel: one enormous dish is cooked by 8 chefs at the same station, each chopping part of it, so they must talk constantly and stand close together (same node). Pipeline parallel: an assembly line where one station preps, the next cooks, and the next plates, passing the dish along.

Which one would you pick?

Situation	Usual choice	NCCL collectives you'll see
Model + optimizer fits on one GPU	DDP	AllReduce (plus Broadcast at startup)
Doesn't fit on one GPU, but you want plain PyTorch code	FSDP or DeepSpeed ZeRO-3	AllGather + ReduceScatter
Very large model, hundreds of GPUs or more	Megatron-LM / NeMo (TP + PP + DP)	AllReduce inside nodes (TP), Send/Recv between stages (PP), AllReduce or ReduceScatter/AllGather across copies (DP)
Mixture-of-Experts model	Megatron / DeepSpeed with expert parallelism	Add AllToAll

These are not NCCL settings, and NCCL doesn't know which one you're using. It only sees a stream of AllReduce / AllGather / ReduceScatter / Send / Recv calls. So if an infrastructure question asks "what does FSDP rely on?", the answer is the collectives (AllGather + ReduceScatter). Tuning the fabric (NCCL env vars, InfiniBand, NVLink) helps all of them.

Interactive: watch each operation (4 GPUs)

Pick an operation. Colors show which GPU the data came from. A rainbow cell is a sum of all four GPUs' data.

Before

After

AllReduce: "everyone gets the total"

What it does: every GPU contributes an array of the same size. The arrays are added element by element, and every GPU ends up with the full summed array. Averages are a sum divided by the GPU count.

Real world, data parallel (PyTorch DDP): each of 8 GPUs holds a full copy of a 1B-parameter model and processes its own 1/8 of the batch. After the backward pass, each GPU has different gradients, because it saw different data. If each GPU applied its own gradients, the 8 copies would drift apart into 8 different models. So DDP AllReduces the gradients (about 2 GB in bf16). Every GPU then applies the same averaged update, and the copies stay identical.

DDP doesn't wait for the whole backward pass to finish. It groups gradients into buckets (25 MB by default) and AllReduces each bucket as soon as it's ready, overlapping communication with the rest of the backward pass.
Tensor parallelism (Megatron-LM style) splits each layer's matrices across GPUs. Each GPU computes a partial result, and an AllReduce combines them, twice per transformer layer in the forward pass and again in the backward pass. That's many latency-sensitive AllReduces per step, which is why tensor parallelism is usually kept inside one node on NVLink (tensor-parallel size ≤ 8 on a DGX).
Small uses too: averaging the loss across GPUs for logging, or checking that all ranks agree a step succeeded.

Cost: with the ring algorithm each GPU sends about 2·(n−1)/n × the array size. For the 2 GB example on 8 GPUs that's about 3.5 GB per GPU, every step. This is where busbw's 2(n−1)/n factor comes from (section 8).

Call it: torch.distributed.all_reduce(t) · benchmark: all_reduce_perf

Broadcast: "copy mine to everyone"

What it does: one GPU, the root, has some data. After the operation, every GPU has an identical copy. Nothing is added up.

Real world:

Starting training: if each rank randomly initialized its own weights, the replicas would start different. DDP broadcasts rank 0's parameters and buffers to all ranks when the model is wrapped, so everyone starts from the same point.
Resuming from a checkpoint: rank 0 reads the file from storage and broadcasts it, instead of 512 GPUs all hammering the shared filesystem for the same file.
Inference: with a model split across GPUs, the incoming request (the token IDs) is often broadcast from one rank so every GPU processes the same input.

Cost: each GPU receives the array once (busbw factor 1). It's usually rare (startup, checkpoint load), so it seldom limits training speed.

Call it: torch.distributed.broadcast(t, src=0) · benchmark: broadcast_perf

Reduce: "add it all up, give the total to one GPU"

What it does: like AllReduce, but only the root ends up with the sum. The others don't receive the result.

Real world: collecting a total where only one rank needs it. For example, rank 0 computes the overall validation loss or accuracy to log and decide whether to save a checkpoint. Gradient training almost never uses Reduce, because every GPU needs the result to update its weights.

Cost: about half of AllReduce, since there's no "send the answer back to everyone" half.

Call it: torch.distributed.reduce(t, dst=0) · benchmark: reduce_perf

AllGather: "everyone shares their piece, everyone gets the whole thing"

What it does: each GPU starts with a different piece. After the operation, every GPU has all the pieces concatenated in rank order: [piece0, piece1, piece2, piece3]. Nothing is added; it's pure copying. The output is n times bigger than each input.

Real world, "gather sharded weights" in FSDP / ZeRO-3: each GPU stores only 1/n of every layer's weights. A GPU can't compute with 1/8 of a layer, so:

Just before layer 12's forward pass, all GPUs AllGather layer 12's shards. Now every GPU briefly holds the complete layer 12.
Each GPU runs layer 12 on its own batch.
Each GPU frees the gathered copy and keeps only its own shard, then moves to layer 13 and repeats.
The backward pass needs the full weights again, so FSDP AllGathers each layer a second time on the way back.

Only one or two layers are ever "inflated" at once, so memory stays small. To hide the wait, FSDP prefetches: it starts gathering layer 13 while layer 12 is still computing.

Other uses: collecting every GPU's predictions for evaluation, and sequence-parallel attention, where each GPU holds a slice of a long sequence and gathers the rest.

Cost: each GPU receives the (n−1) pieces it didn't have, so busbw = algbw × (n−1)/n.

Call it: torch.distributed.all_gather_into_tensor(out, t) · benchmark: all_gather_perf

ReduceScatter: "add it all up, but each GPU keeps only its slice of the total"

What it does: each GPU starts with a full-size array. The arrays are summed element by element, like AllReduce. But the result is split into n slices, and GPU i receives only slice i of the sum.

Real world, "shard gradients" in FSDP / ZeRO: during the backward pass, each GPU computes full-size gradients for a layer from its own data. The gradients must be summed across GPUs, as in DDP. But under FSDP, GPU 3 only owns (stores and updates) slice 3 of the weights and optimizer state. It has no use for the summed gradient of slices 0–2 and 4–7. So ReduceScatter sums the gradients and hands each GPU just the slice it owns. Each GPU then runs the optimizer on its own slice only.

Compare with DDP: AllReduce would give every GPU the whole summed gradient, which is wasted memory and traffic when each GPU only updates 1/n of the weights.

Cost: busbw factor (n−1)/n, about half of an AllReduce. That's no accident: an AllReduce is a ReduceScatter followed by an AllGather.

Call it: torch.distributed.reduce_scatter_tensor(out, t) · benchmark: reduce_scatter_perf

Putting it together: one step of DDP vs FSDP

Phase	DDP (full copy per GPU)	FSDP / ZeRO-3 (sharded)
Forward	No communication	AllGather each layer's weights just before using it, then free
Backward	AllReduce gradient buckets	AllGather weights again, then ReduceScatter that layer's gradients
Optimizer	Every GPU updates all weights (identical work)	Each GPU updates only its own shard
Memory per GPU	Whole model + optimizer (about 16 B/param)	About 1/n of that, plus one or two gathered layers
Traffic per step	1× (one AllReduce ≈ RS + AG)	About 1.5× (AG + AG + RS)

FSDP trades about 50% more network traffic for an n-fold memory saving. That's why fast fabrics (NVLink, 400 Gb/s InfiniBand per GPU) matter so much for large-model training: the collectives run constantly, layer by layer.

AllToAll: "everyone sends a different package to everyone"

What it does: each GPU splits its data into n chunks and sends chunk j to GPU j. Afterwards, GPU j holds chunk j from every GPU. Think of it as a matrix transpose spread across GPUs. Unlike AllGather, every destination gets different data.

Real world, Mixture-of-Experts (MoE) models such as Mixtral or DeepSeek: each layer has many "expert" sub-networks, spread across GPUs (expert parallelism). A small router decides, for every token, which expert(s) should process it.

Dispatch AllToAll: each GPU sends each of its tokens to the GPU that hosts that token's chosen expert.
Each GPU runs its experts on the tokens it received.
Combine AllToAll: results are sent back to the GPUs the tokens came from.

This happens in every MoE layer, forward and backward. Every GPU talks to every other GPU at once, and chunk sizes vary with routing. AllToAll therefore stresses the whole fabric (bisection bandwidth) more than ring-friendly collectives do, and it's sensitive to congestion and uneven loads.

Call it: torch.distributed.all_to_all_single(out, t) · benchmark: alltoall_perf

Send / Recv: "GPU A hands something to GPU B"

What it does: point-to-point. One rank sends a buffer, one specific rank receives it. NCCL counts it alongside the collectives because it uses the same communicator and transports.

Real world, pipeline parallelism: a model too deep for one node is cut into stages. For example, layers 1–20 on node A and layers 21–40 on node B.

Forward: node A finishes layer 20 and sends the activations (the intermediate outputs) to node B.
Backward: node B sends the activation gradients back to node A.
The batch is cut into micro-batches so both stages stay busy. Idle time while the pipeline fills and drains is called the bubble.

Traffic is modest compared with gradient sync, so pipeline stages are often placed across nodes, while tensor parallelism stays inside a node. Inference systems also use send/recv, for example to move the KV cache from a "prefill" GPU to a "decode" GPU.

Call it: torch.distributed.send / recv (or isend / irecv) · benchmark: sendrecv_perf

Which parallelism uses which collective

Strategy	What's split	Main collective(s)	Usually placed on
Data parallel (DDP)	The data batch; full model on every GPU	AllReduce (gradients), Broadcast (startup)	Anywhere; across nodes is fine
Sharded data parallel (FSDP / ZeRO-3)	Data and weights / gradients / optimizer state	AllGather (weights), ReduceScatter (gradients)	Across nodes; needs good fabric bandwidth
Tensor parallel	Each layer's matrices	AllReduce (or AllGather + ReduceScatter with sequence parallel)	Inside a node, on NVLink
Pipeline parallel	Groups of layers (stages)	Send / Recv	Across nodes
Expert parallel (MoE)	The experts in each MoE layer	AllToAll	Inside or across nodes; stresses bisection bandwidth

Big training runs combine several strategies. For example, tensor-parallel 8 inside each DGX, pipeline-parallel across a few nodes, and data-parallel or FSDP across the rest. Each strategy gets its own NCCL communicator, so one job runs several kinds of collectives at once.

All- means everyone gets the result. Reduce means add things up. Gather means collect pieces without adding. Scatter means each GPU ends up with a different piece. So ReduceScatter means "add, then hand out slices", and AllGather means "everyone collects every piece".

AllGather and AllReduce both leave every GPU with "everything", but AllGather concatenates different pieces (the output is n× the input) while AllReduce sums same-sized arrays (the output is the same size as the input). If the question mentions sharded weights, it's AllGather. If it mentions averaging gradients across replicas, it's AllReduce.

3. Core vocabulary

Term	Meaning
Rank	The ID of one participant, usually one process driving one GPU. 4 nodes × 8 GPUs = ranks 0–31.
World size / nranks	Total number of ranks (32 in that example).
Local rank	The rank within one node (0–7 on a DGX). It decides which GPU the process uses.
Communicator	A group of ranks that do collectives together. Created once at startup (`ncclCommInitRank`).
Bootstrap	The startup handshake: ranks exchange connection info over normal TCP sockets, on the interface chosen by `NCCL_SOCKET_IFNAME`.
Channel	One parallel lane of communication. NCCL uses many channels at once, each mapped to different GPU links and NICs, so all the hardware stays busy.
Topology	NCCL's internal map of GPUs, NVLink, PCIe switches, NICs, and CPUs. It uses this map to build rings and trees.

4. Transports: how bytes actually move

Transport	Used for	Shows in logs as
P2P (peer-to-peer)	GPU ↔ GPU on the same node, over NVLink/NVSwitch (or PCIe)	`via P2P/IPC`, `P2P/CUMEM`, `P2P/direct pointer`
SHM (shared memory)	Same node when P2P isn't possible: data goes through host memory (slower)	`via SHM`
NET / IB	Across nodes over InfiniBand (or RoCE) using RDMA verbs	`via NET/IB/<n>`
NET / IB + GDRDMA	Across nodes with GPUDirect RDMA: the NIC reads and writes GPU memory directly. This is what you want.	`via NET/IB/<n>/GDRDMA`
NET / Socket	Across nodes over plain TCP. Very slow fallback, used when InfiniBand isn't available or is disabled.	`NET/Socket`
NVLS (NVLink SHARP)	H100 + NVSwitch: the NVSwitch itself performs reductions (in-switch math)	`NVLS`
CollNet / SHARP	InfiniBand switches perform reductions in the network (needs the SHARP plugin and Quantum switches)	`CollNet`

Inside a node, traffic goes over NVLink. Between nodes, it goes over InfiniBand with GPUDirect RDMA. If logs show NET/Socket or SHM where you expected IB or NVLink, that's your performance problem.

5. Algorithms and protocols

Algorithms (the delivery route). Force one with `NCCL_ALGO`

Algorithm	How it works	Best for
Ring	GPUs form a loop and pass chunks around it. Full bandwidth, but latency grows with the number of GPUs.	Large messages (bandwidth-bound)
Tree	Double binary tree across nodes: two complementary trees, so load stays balanced. Latency grows only logarithmically.	Small/medium messages, many nodes (latency-bound)
NVLS / NVLSTree	Uses NVSwitch in-switch reduction (H100 and later)	Intra-node (and combined with trees across nodes) on NVSwitch systems
CollNet	Uses SHARP in-network reduction on InfiniBand switches	Large clusters with SHARP-capable switches

Protocols (how each packet is sent). Force one with `NCCL_PROTO`

Protocol	Trade-off
Simple	Highest bandwidth, higher latency. Used for large messages.
LL (Low Latency)	Lowest latency, lower bandwidth. Used for tiny messages.
LL128	Middle ground: low latency with good bandwidth (needs NVLink-class hardware).

NCCL picks the algorithm and protocol automatically based on message size and topology. Forcing NCCL_ALGO or NCCL_PROTO is for debugging or experiments. It's rarely "the fix," and a leftover forced value is a common cause of poor performance.

6. NCCL on a DGX H100 cluster

8 GPUs per node, all connected through NVSwitch (Fabric Manager must be running).
8 compute NICs per node, one per GPU, 400 Gb/s each (≈ 50 GB/s each, ≈ 400 GB/s per node).
Rail-optimized network: NIC i on every node plugs into leaf switch i, so GPU i talks to GPU i on other nodes in one hop (fabric topology).
GPUDirect RDMA (nvidia-peermem or DMA-BUF) lets each NIC access its GPU's memory directly.
A healthy setup uses all 8 NICs in parallel. If only one is used, GPU 0 becomes the bottleneck. See the GPU 0 bottleneck guide.

7. nccl-tests: running and reading benchmarks

nccl-tests is NVIDIA's benchmark suite (github.com/NVIDIA/nccl-tests). Each binary tests one collective: all_reduce_perf, all_gather_perf, reduce_scatter_perf, broadcast_perf, reduce_perf, alltoall_perf, sendrecv_perf.

For building with MPI, launching with srun or mpirun, every flag, NCCL_TESTS_SPLIT per-rail and per-node tests, and a hands-on lab, see nccl-tests mastery.

Common flags

Flag	Meaning	Typical
`-b`	begin (minimum) message size	`-b 8` (8 bytes)
`-e`	end (maximum) message size	`-e 8G`
`-f`	Multiplication factor between sizes	`-f 2` (doubling)
`-g`	GPUs per thread/process	`-g 1` with MPI (one process per GPU), or `-g 8` for a single process on one node
`-t`	Threads per process	Usually 1
`-n` / `-w`	Iterations / warm-up iterations	Defaults are fine
`-c`	Check results for correctness (1 = on)	On by default
`-o` / `-d`	Reduction op (sum, prod, ...) / data type (float, half, bfloat16, ...)

Single node vs multi node

# Single DGX, one process driving all 8 GPUs (tests NVLink / NVSwitch only)
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8

# 4 DGX nodes = 32 GPUs, one MPI process per GPU (tests NVLink + InfiniBand)
mpirun -np 32 -N 8 --hostfile hosts \
  -x NCCL_DEBUG=INFO -x NCCL_IB_HCA -x LD_LIBRARY_PATH \
  ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

# Same thing under Slurm
srun -N 4 --ntasks-per-node=8 --gpus-per-node=8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Reading the output

#                                                out-of-place                       in-place
#       size     count   type  redop  root   time  algbw  busbw  #wrong     time  algbw  busbw  #wrong
#        (B) (elements)                      (us) (GB/s) (GB/s)            (us) (GB/s) (GB/s)
         8          2   float    sum    -1   45.1   0.00   0.00       0     44.8   0.00   0.00       0
   ...
  8589934592  2147483648 float  sum    -1  46102  186.3  360.9       0    46050  186.5  361.3       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 152.4

Column	Meaning
`size` / `count`	Message size in bytes / number of elements
`time`	Microseconds per operation
`algbw`	Algorithm bandwidth = size ÷ time
`busbw`	Bus bandwidth: algbw corrected for the collective, so you can compare it to hardware link speed. This is the number to judge.
`#wrong`	Correctness errors. Must be 0. Anything else points to a hardware or data-corruption problem.
out-of-place / in-place	Separate output buffer / output overwrites input. They should give similar numbers.
`Avg bus bandwidth`	Average over all sizes. Tiny messages pull it down, so look at the large-message busbw to judge peak performance.

Rough expectations on DGX H100 (large messages, AllReduce busbw)

Setup	Healthy ballpark	If you see much less
Single node, 8 GPUs (NVLink)	Around 450+ GB/s (higher with NVLS)	Fabric Manager down, P2P disabled, NVLink errors
Multi node (8 × 400 Gb/s IB per node)	Roughly 350–400 GB/s (the network ceiling is ~400 GB/s per node)	Only some NICs used, no GDRDMA, falling back to sockets, link down, bad cable

Exact numbers depend on NCCL version, message size, and fabric. Exams usually test the pattern ("far below expected" means a misconfiguration), not exact values.

8. algbw vs busbw

algbw = data size ÷ time. But in AllReduce, each GPU actually has to send and receive about 2× the data (once in the reduce-scatter half, once in the all-gather half), so algbw understates how hard the links are working.

busbw applies a correction factor so the result can be compared directly to the hardware's link bandwidth:

Collective	busbw formula (n = number of ranks)
AllReduce	busbw = algbw × 2(n−1)/n
AllGather, ReduceScatter	busbw = algbw × (n−1)/n
Broadcast, Reduce	busbw = algbw (factor 1)

Example: AllReduce with 32 ranks, algbw = 186 GB/s → busbw = 186 × 2 × 31/32 ≈ 360 GB/s.

Compare busbw (not algbw) against hardware capability. busbw stays roughly the same no matter how many GPUs you add, which makes it the fair "is the fabric healthy?" metric.

9. Environment variables that matter

Group them by purpose, as with nvidia-smi:

Debugging

Variable	What it does
`NCCL_DEBUG=INFO`	Prints the chosen NICs, transports, rings, trees, and versions. First thing to set when troubleshooting. (Levels: VERSION, WARN, INFO, TRACE.)
`NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH`	Limits the debug output to certain subsystems
`NCCL_DEBUG_FILE=/tmp/nccl.%h.%p`	Writes logs to per-host (`%h`) and per-process (`%p`) files
`NCCL_TOPO_DUMP_FILE=topo.xml`	Dumps the topology NCCL detected

Network selection

Variable	What it does
`NCCL_SOCKET_IFNAME`	Which Ethernet interface to use for bootstrap (and for socket transport). For example `=ens1f0`, `=eth` (prefix match), or `=^docker,lo` (`^` = exclude). A wrong value often causes hangs at startup.
`NCCL_IB_HCA`	Which InfiniBand adapters NCCL may use, for example `=mlx5_0,mlx5_1,...`. Set to only one, it funnels all traffic through one NIC.
`NCCL_IB_DISABLE=1`	Turns InfiniBand off, forcing the TCP socket fallback. Huge slowdown. Only for debugging.
`NCCL_NET_GDR_LEVEL`	How close a NIC and GPU must be (PIX, PXB, PHB, SYS) for NCCL to use GPUDirect RDMA
`NCCL_IB_GID_INDEX`	Which GID to use. Needed for RoCE (RDMA over Ethernet), often 3. See RoCE & Spectrum-X.
`NCCL_IB_TIMEOUT`, `NCCL_IB_RETRY_CNT`	InfiniBand timeout and retry tuning for large or flaky fabrics
`NCCL_CROSS_NIC`	Whether rings and trees may use different NICs on different nodes (it matters for rail-optimized fabrics)

Intra-node paths

Variable	What it does
`NCCL_P2P_DISABLE=1`	Disables GPU peer-to-peer (NVLink/PCIe), forcing SHM. Big intra-node slowdown.
`NCCL_P2P_LEVEL`	Maximum topology distance (for example NVL, PIX, PXB, PHB, SYS) at which P2P is used
`NCCL_SHM_DISABLE=1`	Disables the shared-memory transport
`NCCL_NVLS_ENABLE`	Enables or disables NVLink SHARP (H100 + NVSwitch)

Algorithm and tuning

Variable	What it does
`NCCL_ALGO`	Force algorithm(s): Ring, Tree, NVLS, CollNet… Normally leave unset.
`NCCL_PROTO`	Force protocol(s): Simple, LL, LL128. Normally leave unset.
`NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`	Bounds on the number of parallel channels
`NCCL_COLLNET_ENABLE=1`	Enables SHARP / CollNet (requires the plugin and capable switches)
`NCCL_TOPO_FILE`	Supplies a topology XML (common in VMs and clouds where auto-detection is wrong)

Mnemonic for the variable names: NCCL_IB_* = InfiniBand, NCCL_SOCKET_* = Ethernet/TCP, NCCL_P2P_* = GPU-to-GPU inside a node, NCCL_NET_* = generic network/GDR, NCCL_DEBUG* = logging.

10. Reading NCCL_DEBUG=INFO logs

NCCL INFO NCCL version 2.21.5+cuda12.4
NCCL INFO Bootstrap : Using ens1f0:10.0.0.11<0>                    <- bootstrap interface (NCCL_SOCKET_IFNAME)
NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)              <- fine, unless you expected a SHARP / cloud plugin
NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB
                         [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB ; OOB ens1f0
                                                                    <- all 8 HCAs found: good
NCCL INFO Using network IB
NCCL INFO comm 0x55d0 rank 0 nranks 32 cudaDev 0 nvmlDev 0 busId 18000 - Init START
NCCL INFO Channel 00/32 :    0   1   2   3   4   5   6   7   8 ...    <- ring order for channel 0
NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] ...
NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM                 <- intra-node over NVLink: good
NCCL INFO Channel 00/0 : 7[7] -> 8[0] [send] via NET/IB/0/GDRDMA    <- inter-node with GPUDirect RDMA: good
NCCL INFO NVLS multicast support is available on dev 0              <- NVLink SHARP available
NCCL INFO Connected all rings
NCCL INFO Connected all trees
NCCL INFO comm 0x55d0 rank 0 nranks 32 ... - Init COMPLETE          <- initialization succeeded

Red flags in logs

Log line	Meaning
`NET/IB : Using [0]mlx5_0:1/IB` (only one)	Only one HCA in use. Check `NCCL_IB_HCA` and port states.
`NET/Socket : Using ...` / `Using network Socket`	InfiniBand not used: IB disabled, missing drivers, or a container without `/dev/infiniband`.
`via NET/IB/0` without `/GDRDMA`	No GPUDirect RDMA. Data bounces through CPU memory. Check `nvidia-peermem`.
`via SHM` between GPUs on the same node	P2P not used: `NCCL_P2P_DISABLE`, Fabric Manager down, or PCIe ACS/IOMMU blocking P2P.
`NCCL WARN ... unhandled system error`	Usually an OS or resource issue: IB device access, locked-memory limits (`ulimit -l`), or a small `/dev/shm` in containers.
Stuck before `Init COMPLETE`	Bootstrap can't connect: wrong `NCCL_SOCKET_IFNAME`, firewall, or a rank that never started.

11. Troubleshooting playbook

Symptom	Likely causes	Check / fix
Hangs at initialization	Bootstrap on the wrong interface (docker0, a management NIC), firewall, unreachable hosts, mismatched NCCL versions	Set `NCCL_SOCKET_IFNAME` to the right interface. Open ports. Check that every rank started. `NCCL_DEBUG=INFO`.
Multi-node bandwidth very low	Socket fallback, only some HCAs used, no GDRDMA, IB link down or degraded, cabling	Logs (`NET/IB` vs `NET/Socket`, `GDRDMA`), `NCCL_IB_HCA`, `ibstat`, `ibdev2netdev`, `lsmod \| grep nvidia_peermem`
One GPU per node does all the network work	Only one HCA configured or visible	Expose all HCAs (`NCCL_IB_HCA`), bring all ports up, enable GDR
Single-node bandwidth low	P2P disabled, Fabric Manager not running, NVLink errors, PCIe ACS on (bare metal)	`systemctl status nvidia-fabricmanager`, `nvidia-smi topo -m`, `nvidia-smi nvlink -s`, unset `NCCL_P2P_DISABLE`
One node slower than the rest	Bad cable or NIC, throttled GPU, wrong firmware	Run nccl-tests on node subsets (pairs) to isolate the slow one. Check `ibstat` speed and DCGM health.
Fails in containers but works on bare metal	IB devices not passed in, small `/dev/shm`, memlock limit	`--device=/dev/infiniband`, `--cap-add=IPC_LOCK`, `--shm-size=…` or `--ipc=host`, `--ulimit memlock=-1`
PyTorch "NCCL timeout" / watchdog error	One rank crashed or is stuck, ranks are calling collectives in a different order (desync), network failure	Find the first failing rank's logs. Check that all ranks call the same collectives. Look for XID errors.
`#wrong` ≠ 0 in nccl-tests	Data corruption: faulty GPU memory, NVLink, or NIC	DCGM diagnostics, check ECC/XID, isolate by testing subsets

To isolate a bad node, use binary search: run nccl-tests on halves or pairs of nodes, find the slow group, and keep splitting.

12. NCCL in PyTorch and containers

# PyTorch picks NCCL as the backend for GPU training
import torch.distributed as dist
dist.init_process_group(backend="nccl")     # uses MASTER_ADDR, MASTER_PORT, RANK, WORLD_SIZE

# Launch 8 processes per node on 4 nodes
torchrun --nnodes=4 --nproc_per_node=8 \
         --rdzv_backend=c10d --rdzv_endpoint=node01:29500 train.py

TORCH_NCCL_ASYNC_ERROR_HANDLING=1 (older name: NCCL_ASYNC_ERROR_HANDLING): crash cleanly instead of hanging when a collective fails.
The NCCL version usually comes bundled with PyTorch or the NGC container. Check it with python -c "import torch; print(torch.cuda.nccl.version())".
In containers you need: IB devices passed through, enough /dev/shm, IPC_LOCK, and an unlimited memlock limit.

13. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8Single DGX: NVLink/NVSwitch only

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 8 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x18] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid   4101 on      dgx01 device  1 [0x2a] NVIDIA H100 80GB HBM3
#  Rank  2 Group  0 Pid   4101 on      dgx01 device  2 [0x3a] NVIDIA H100 80GB HBM3
#  Rank  3 Group  0 Pid   4101 on      dgx01 device  3 [0x5d] NVIDIA H100 80GB HBM3
#  Rank  4 Group  0 Pid   4101 on      dgx01 device  4 [0x9a] NVIDIA H100 80GB HBM3
#  Rank  5 Group  0 Pid   4101 on      dgx01 device  5 [0xab] NVIDIA H100 80GB HBM3
#  Rank  6 Group  0 Pid   4101 on      dgx01 device  6 [0xba] NVIDIA H100 80GB HBM3
#  Rank  7 Group  0 Pid   4101 on      dgx01 device  7 [0xdb] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    11.45    0.00    0.00      0    11.51    0.00    0.00      0
          16             4     float     sum      -1    11.50    0.00    0.00      0    11.45    0.00    0.00      0
          32             8     float     sum      -1    11.55    0.00    0.00      0    11.53    0.00    0.00      0
          64            16     float     sum      -1    11.48    0.01    0.01      0    11.49    0.01    0.01      0
  ...
    16777216       4194304     float     sum      -1    72.67  230.88  404.04      0    72.34  231.92  405.86      0
    33554432       8388608     float     sum      -1   134.37  249.72  437.01      0   134.17  250.09  437.66      0
    67108864      16777216     float     sum      -1   255.66  262.50  459.37      0   256.04  262.10  458.68      0
   134217728      33554432     float     sum      -1   501.84  267.45  468.04      0   504.10  266.25  465.95      0
   268435456      67108864     float     sum      -1   986.21  272.19  476.33      0   981.77  273.42  478.48      0
   536870912     134217728     float     sum      -1  1968.84  272.68  477.20      0  1965.89  273.09  477.91      0
  1073741824     268435456     float     sum      -1  3941.89  272.39  476.69      0  3947.80  271.98  475.97      0
  2147483648     536870912     float     sum      -1  7825.19  274.43  480.26      0  7860.40  273.20  478.11      0
  4294967296    1073741824     float     sum      -1  15701.6  273.54  478.69      0  15630.9  274.77  480.85      0
  8589934592    2147483648     float     sum      -1  31203.7  275.29  481.75      0  31156.8  275.70  482.47      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 184.488
#
# Collective test concluded: all_reduce_perf

What to look at: This tests only NVLink and NVSwitch inside dgx01. Expect about 470-480 GB/s busbw at the largest sizes; a much lower number points at Fabric Manager, disabled P2P or NVLink errors, not the InfiniBand network.

mpirun -np 32 -N 8 --hostfile hosts -x NCCL_DEBUG=INFO -x NCCL_IB_HCA -x LD_LIBRARY_PATH ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

4 nodes, 32 ranks, with NCCL INFO

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x18] NVIDIA H100 80GB HBM3
#  ...
#  Rank 31 Group  0 Pid   7483 on      dgx04 device  7 [0xdb] NVIDIA H100 80GB HBM3
dgx01:4101:4101 [0] NCCL INFO Bootstrap : Using ens1f0:10.0.0.11<0>
dgx01:4101:4101 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
dgx01:4101:4101 [0] NCCL INFO NCCL version 2.21.5+cuda12.4
dgx01:4101:4152 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB ens1f0:10.0.0.11<0>
dgx01:4101:4152 [0] NCCL INFO Using network IB
dgx01:4101:4152 [0] NCCL INFO comm 0x55d0a2c4e1b0 rank 0 nranks 32 cudaDev 0 nvmlDev 0 busId 18000 commId 0x9e3c71d2a4b05f18 - Init START
dgx01:4101:4152 [0] NCCL INFO Channel 00/32 :    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16 ...
dgx01:4101:4152 [0] NCCL INFO Trees [0] 1/16/-1->0->-1 [1] 1/-1/-1->0->7 [2] ...
dgx01:4101:4152 [0] NCCL INFO Channel 00/0 : 31[7] -> 0[0] [receive] via NET/IB/0/GDRDMA
dgx01:4101:4152 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
dgx01:4101:4152 [0] NCCL INFO NVLS multicast support is available on dev 0
dgx01:4101:4152 [0] NCCL INFO Connected all rings
dgx01:4101:4152 [0] NCCL INFO Connected all trees
dgx01:4101:4152 [0] NCCL INFO comm 0x55d0a2c4e1b0 rank 0 nranks 32 cudaDev 0 nvmlDev 0 busId 18000 commId 0x9e3c71d2a4b05f18 - Init COMPLETE
...
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    44.82    0.00    0.00      0    45.02    0.00    0.00      0
  ...
   536870912     134217728     float     sum      -1  2926.41  183.46  355.45      0  2922.02  183.73  355.98      0
  1073741824     268435456     float     sum      -1  5831.04  184.14  356.78      0  5839.79  183.87  356.24      0
  2147483648     536870912     float     sum      -1  11547.5  185.97  360.32      0  11599.4  185.14  358.70      0
  4294967296    1073741824     float     sum      -1  23142.4  185.59  359.58      0  23038.3  186.43  361.20      0
  8589934592    2147483648     float     sum      -1  45962.9  186.89  362.10      0  45894.0  187.17  362.64      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 122.332
#
# Collective test concluded: all_reduce_perf

What to look at: Check that NET/IB lists all eight mlx5 HCAs, inter-node channels say via NET/IB/<n>/GDRDMA, intra-node channels say via P2P, and you reach Init COMPLETE. busbw around 360 GB/s at 8 GB with #wrong 0 is healthy; NET/Socket or one HCA would cut it sharply.

srun -N 4 --ntasks-per-node=8 --gpus-per-node=8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1Same 32-rank test under Slurm

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x18] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid   4108 on      dgx01 device  1 [0x2a] NVIDIA H100 80GB HBM3
#  ...
#  Rank  8 Group  0 Pid   5212 on      dgx02 device  0 [0x18] NVIDIA H100 80GB HBM3
#  ...
#  Rank 31 Group  0 Pid   7483 on      dgx04 device  7 [0xdb] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    44.82    0.00    0.00      0    45.02    0.00    0.00      0
          16             4     float     sum      -1    45.00    0.00    0.00      0    44.80    0.00    0.00      0
  ...
   268435456      67108864     float     sum      -1  1475.80  181.89  352.42      0  1469.15  182.71  354.01      0
   536870912     134217728     float     sum      -1  2918.45  183.96  356.42      0  2914.07  184.23  356.95      0
  1073741824     268435456     float     sum      -1  5815.06  184.65  357.76      0  5823.78  184.37  357.22      0
  2147483648     536870912     float     sum      -1  11515.7  186.48  361.31      0  11567.5  185.65  359.69      0
  4294967296    1073741824     float     sum      -1  23078.6  186.10  360.57      0  22974.8  186.94  362.20      0
  8589934592    2147483648     float     sum      -1  45836.1  187.41  363.10      0  45767.3  187.69  363.64      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 122.624
#
# Collective test concluded: all_reduce_perf

What to look at: The header should list Rank 0 to Rank 31 across dgx01-dgx04 in one block. If you see many headers that each say Rank 0, srun started independent single-rank tests (missing --mpi=pmix or a non-MPI build) and busbw is meaningless.

mpirun -np 32 -N 8 --hostfile hosts -x NCCL_DEBUG=INFO -x NCCL_DEBUG_SUBSYS=INIT,NET -x LD_LIBRARY_PATH ./build/all_reduce_perf -b 8G -e 8G -g 1 | grep "NCCL INFO" | grep -E "Bootstrap|NCCL version|NET/IB|Using network"

Trim INFO to init and network

dgx01:4101:4101 [0] NCCL INFO Bootstrap : Using ens1f0:10.0.0.11<0>
dgx01:4101:4101 [0] NCCL INFO NCCL version 2.21.5+cuda12.4
dgx01:4101:4152 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB ens1f0:10.0.0.11<0>
dgx01:4101:4152 [0] NCCL INFO Using network IB
dgx02:5212:5212 [0] NCCL INFO Bootstrap : Using ens1f0:10.0.0.12<0>
dgx02:5212:5263 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB ens1f0:10.0.0.12<0>
dgx02:5212:5263 [0] NCCL INFO Using network IB
dgx03:6323:6323 [0] NCCL INFO Bootstrap : Using ens1f0:10.0.0.13<0>
dgx03:6323:6374 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB ens1f0:10.0.0.13<0>
dgx03:6323:6374 [0] NCCL INFO Using network IB
dgx04:7434:7434 [0] NCCL INFO Bootstrap : Using ens1f0:10.0.0.14<0>
dgx04:7434:7485 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB ens1f0:10.0.0.14<0>
dgx04:7434:7485 [0] NCCL INFO Using network IB
...

What to look at: With the output limited to INIT and NET you can check every node quickly: each should bootstrap on ens1f0 and say Using network IB with eight HCAs. A node that says Using network Socket, or lists fewer HCAs, is the one to fix.

mpirun -np 32 -N 8 --hostfile hosts -x NCCL_IB_DISABLE=1 -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Forced socket fallback (debug only)

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x18] NVIDIA H100 80GB HBM3
#  ...
#  Rank 31 Group  0 Pid   7483 on      dgx04 device  7 [0xdb] NVIDIA H100 80GB HBM3
dgx01:4101:4101 [0] NCCL INFO Bootstrap : Using ens1f0:10.0.0.11<0>
dgx01:4101:4101 [0] NCCL INFO NCCL version 2.21.5+cuda12.4
dgx01:4101:4152 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
dgx01:4101:4152 [0] NCCL INFO NET/Socket : Using [0]ens1f0:10.0.0.11<0>
dgx01:4101:4152 [0] NCCL INFO Using network Socket
dgx01:4101:4152 [0] NCCL INFO Channel 00/0 : 31[7] -> 0[0] [receive] via NET/Socket/0
dgx01:4101:4152 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
...
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1   179.28    0.00    0.00      0   180.09    0.00    0.00      0
  ...
  1073741824     268435456     float     sum      -1 201016.9    5.34   10.35      0 201318.4    5.33   10.33      0
  2147483648     536870912     float     sum      -1 399451.6    5.38   10.42      0 401249.1    5.35   10.37      0
  4294967296    1073741824     float     sum      -1 801924.8    5.36   10.38      0 798316.1    5.38   10.42      0
  8589934592    2147483648     float     sum      -1  1594066    5.39   10.44      0  1591675    5.40   10.46      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 4.570
#
# Collective test concluded: all_reduce_perf

What to look at: Using network Socket and via NET/Socket mean InfiniBand is not used, and busbw collapses to roughly 10 GB/s. If you see this without setting NCCL_IB_DISABLE, look for missing IB drivers or a container without /dev/infiniband.

mpirun -np 32 -N 8 --hostfile hosts -x NCCL_DEBUG=INFO -x NCCL_DEBUG_FILE=/tmp/nccl.%h.%p -x LD_LIBRARY_PATH ./build/all_reduce_perf -b 8G -e 8G -g 1 > /dev/null; ls /tmp/nccl.*

Per-host, per-process NCCL log files

/tmp/nccl.dgx01.4101  /tmp/nccl.dgx01.4115  /tmp/nccl.dgx01.4129  /tmp/nccl.dgx01.4143
/tmp/nccl.dgx01.4108  /tmp/nccl.dgx01.4122  /tmp/nccl.dgx01.4136  /tmp/nccl.dgx01.4150

What to look at: %h expands to the host name and %p to the PID, so dgx01 gets one file per local rank (8 files); the other nodes write theirs to their own /tmp. Missing files for a rank usually mean that rank never started.

NCCL_TOPO_DUMP_FILE=topo.xml ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8 > /dev/null; head -n 14 topo.xmlDump the topology NCCL detected

<system version="1">
  <cpu numaid="0" affinity="00000000,00000000,000000ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff" arch="x86_64" vendor="GenuineIntel" familyid="6" modelid="143">
    <pci busid="0000:15:00.0" class="0x060400" vendor="0x1000" device="0xc030" subsystem_vendor="0x1000" subsystem_device="0x100b" link_speed="32.0 GT/s PCIe" link_width="16">
      <pci busid="0000:18:00.0" class="0x030200" vendor="0x10de" device="0x2330" subsystem_vendor="0x10de" subsystem_device="0x16c1" link_speed="32.0 GT/s PCIe" link_width="16">
        <gpu dev="0" sm="90" rank="0" gdr="1">
          <nvlink target="fffffff:ffffffff" count="18" tclass="0x068000"/>
        </gpu>
      </pci>
      <pci busid="0000:17:00.0" class="0x020700" vendor="0x15b3" device="0x1021" subsystem_vendor="0x15b3" subsystem_device="0x0041" link_speed="32.0 GT/s PCIe" link_width="16">
        <nic>
          <net name="mlx5_0" dev="0" speed="400000" port="1" latency="0.000000" guid="0x0a01" maxconn="131072" gdr="1"/>
        </nic>
      </pci>
    </pci>

What to look at: Each GPU should sit under the same PCIe switch as its NIC, with gdr="1" on both and an nvlink entry with count="18" for H100. gdr="0" on a NIC or GPU means NCCL will not use GPUDirect RDMA for it.

ulimit -lLocked-memory limit for RDMA

unlimited

What to look at: RDMA needs to pin memory, so this should print unlimited on hosts and inside containers (docker run --ulimit memlock=-1). A small value such as 64 or 8192 is a classic cause of NCCL WARN ... unhandled system error at init.

lsmod | grep nvidia_peermemGPUDirect RDMA kernel module loaded

nvidia_peermem         16384  0
ib_core               434176  9 rdma_cm,ib_ipoib,nvidia_peermem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
nvidia              56389632  1095 nvidia_uvm,nvidia_peermem,nvidia_modeset

What to look at: An nvidia_peermem line means GPUDirect RDMA can work through the peer-memory path. No output (and no DMA-BUF support) explains logs that show via NET/IB/<n> without /GDRDMA, where data bounces through CPU memory.

systemctl status nvidia-fabricmanagerFabric Manager state for NVSwitch

● nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-11-10 08:14:02 UTC; 2h 51min ago
    Process: 2481 ExecStart=/usr/bin/nv-fabricmanager-start.sh (code=exited, status=0/SUCCESS)
   Main PID: 2490 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 21.6M
        CPU: 4.213s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2490 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

Nov 10 08:13:58 dgx01 systemd[1]: Starting NVIDIA fabric manager service...
Nov 10 08:14:02 dgx01 nv-fabricmanager[2490]: Connected to 1 node.
Nov 10 08:14:02 dgx01 nv-fabricmanager[2490]: Successfully configured all the available NVSwitches to route GPU NVLink traffic. NVLink Peer-to-Peer support will be enabled once the GPUs are successfully registered with the NVLink fabric.
Nov 10 08:14:02 dgx01 systemd[1]: Started NVIDIA fabric manager service.

What to look at: You want active (running) and the 'Successfully configured all the available NVSwitches' message. inactive or failed means NVSwitch routing is not set up, so single-node P2P and NVLink bandwidth fail; it does not affect the InfiniBand path between nodes.

nvidia-smi topo -mGPU, NVLink and NIC topology matrix

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0    X       NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU1    NV18    X       NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU2    NV18    NV18    X       NV18    NV18    NV18    NV18    NV18    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU3    NV18    NV18    NV18    X       NV18    NV18    NV18    NV18    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU4    NV18    NV18    NV18    NV18    X       NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    56-111,168-223  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18    X       NV18    NV18    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    56-111,168-223  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18    X       NV18    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    56-111,168-223  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18    X       SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     56-111,168-223  1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     X       NODE    NODE    NODE    SYS     SYS     SYS     SYS
...

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  ...

What to look at: GPU-to-GPU cells should all be NV18 on a DGX H100, and each GPU should be PIX to exactly one NIC (its rail). A smaller NV number means NVLinks are down; SYS or PHB to every NIC means poor GPU-NIC affinity for GPUDirect RDMA.

nvidia-smi nvlink -sPer-link NVLink state and speed

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-00000000-aaaa-bbbb-cccc-000000000000)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         ...
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-00000001-aaaa-bbbb-cccc-000000000000)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         ...
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
...

What to look at: Each H100 should list 18 links at about 26.562 GB/s. A link shown as <inactive> or missing links on one GPU means degraded NVLink; check dmesg for Xid 74 and nvidia-smi topo -m for NV numbers below 18.

python -c "import torch; print(torch.cuda.nccl.version())"NCCL version bundled with PyTorch

(2, 21, 5)

What to look at: This is the NCCL PyTorch actually loads, which can differ from the system libnccl that nccl-tests used. Mismatched NCCL versions across nodes or containers can cause hangs at init, so compare this tuple on every node.

14. Searchable cheat sheet

Item	What it means / does

15. Exam traps

Cross-node problems → InfiniBand / HCAs / GDRDMA / NCCL_IB_*. Not Fabric Manager, not NVSwitch.
Intra-node problems → NVLink / P2P / Fabric Manager / NCCL_P2P_*.
Hang at startup → usually bootstrap: NCCL_SOCKET_IFNAME or firewall, not the algorithm.
Judge with busbw, not algbw, and look at large-message busbw, not just "Avg bus bandwidth."
NCCL_IB_DISABLE=1 / NCCL_P2P_DISABLE=1 are debugging switches that reduce performance. They're never the "optimization" answer.
Forcing NCCL_ALGO/NCCL_PROTO is rarely the right fix. NCCL auto-tunes, and a stale forced value can cause problems.
NCCL's tree is a double binary tree designed to balance load, so "tree concentrates all traffic at one root" is a distractor.
NCCL is a library, not a service. There's no "NCCL daemon" to restart.
MPI ≠ NCCL: MPI or Slurm often launches the ranks, and NCCL moves the GPU data.
MIG and NCCL: MIG instances don't support the P2P/NVLink that multi-GPU NCCL training needs.
#wrong must be 0. Non-zero means data corruption, a hardware issue.

16. Practice quiz (28 questions)

Score: 0 / 0