Multi-Node NCCL: Why Is GPU 0 Doing All the Network Work?

A study guide that starts from zero and works up to diagnosing this symptom with confidence.

1. The scenario

You run all_reduce_perf across a handful of DGX H100 nodes. Bus bandwidth is far below expectations, and the counters show that on every node only GPU 0 (and the NIC next to it) carries traffic between nodes. The other seven GPUs sit idle while data crosses the network. The hardware checks out. What's misconfigured?

Three clues drive the answer:

Traffic between nodes: this travels over InfiniBand, not NVLink.
GPU 0 on every node: one specific GPU per node is the bottleneck. That suggests a one-path setup, such as only one network card being used.
The hardware checks out: something is set up so that only part of the hardware gets used.

2. Why GPUs need to talk to each other at all

When you train a large AI model on many GPUs (called distributed training), each GPU works on a different slice of the data. After every training step, the GPUs must combine their results (their "gradients") so all of them end up with the same updated model.

That combining step is called all-reduce: every GPU contributes a set of numbers, they're summed, and every GPU receives the total. With 4 nodes × 8 GPUs = 32 GPUs, each training step moves gigabytes of data between them.

If this communication is slow, the expensive GPUs sit idle waiting. In large clusters, the network often decides how fast training runs.

32 accountants each count part of the receipts. At the end of every hour they must share totals so everyone has the same grand total. If they all have to go through one phone line, most of them sit waiting.

3. What NCCL is

NCCL (pronounced "nickel") is the NVIDIA Collective Communications Library. It's the software that actually moves data between GPUs for operations such as:

All-reduce: sum across all GPUs, and everyone gets the result
All-gather: everyone gets everyone's pieces
Reduce-scatter: sum, then each GPU keeps one slice
Broadcast: one GPU sends data to all

PyTorch (torch.distributed with the nccl backend), TensorFlow, Megatron, and others all use NCCL underneath.

NCCL is topology-aware: at startup it detects which GPUs are connected by NVLink, which network cards (NICs) sit closest to which GPU, and which network links exist. It then plans the fastest routes. If it's given a bad picture of the hardware (for example "there is only one NIC"), it plans bad routes.

An "NCCL test" usually means the nccl-tests benchmark suite, for example:

mpirun -np 32 -N 8 ./all_reduce_perf -b 8 -e 8G -f 2 -g 1

It reports bus bandwidth (busbw). On a healthy DGX H100 cluster, cross-node all-reduce busbw should be several hundred GB/s. If it's a fraction of that, something is funneling the traffic.

4. The two networks: NVLink inside a node, InfiniBand between nodes

This is the single most important concept for this question. GPU traffic travels on two completely separate networks:

NVLink + NVSwitch (inside a node)

Connects the 8 GPUs within one DGX server
Very fast: about 900 GB/s per H100 GPU
Managed by Fabric Manager
Used for the intra-node phase of NCCL

InfiniBand (between nodes)

Connects different DGX servers through network switches
On DGX H100, eight 400 Gb/s ConnectX-7 links per node
Configured through NICs (HCAs), drivers, the Subnet Manager, and NCCL environment variables
Used for the inter-node (network) phase of NCCL

"Cross-node" traffic means InfiniBand. Fabric Manager and NVSwitch only handle traffic inside one node, so they can't be responsible for how cross-node traffic is spread. This alone rules out any Fabric Manager explanation.

NVLink is the hallways inside an office building. InfiniBand is the roads between buildings. A traffic jam on the roads between buildings has nothing to do with how the hallways are laid out.

5. How a DGX H100 is wired to the network

A DGX H100 has 8 GPUs and 8 ConnectX-7 InfiniBand adapters for the compute network (400 Gb/s each, 3.2 Tb/s per node in total). It also has separate adapters for storage and management, which aren't part of this question.

Each GPU is paired with its own nearby NIC that sits on the same PCIe switch. This is called GPU–NIC affinity:

GPU	Closest NIC (example name)	Rail (leaf switch)
GPU 0	`mlx5_0`	Rail 0
GPU 1	`mlx5_1`	Rail 1
…	…	…
GPU 7	`mlx5_7` (exact device names vary by system)	Rail 7

In a rail-optimized cluster, NIC 0 on every node plugs into the "rail 0" switch, NIC 1 on every node into "rail 1," and so on. So GPU 3 on node A talks to GPU 3 on node B over its own dedicated lane.

Designed state: 8 GPUs → 8 NICs → 8 parallel lanes out of each node. All of them carry traffic at the same time.

6. GPUDirect RDMA

RDMA (Remote Direct Memory Access) lets one computer's network card write straight into another computer's memory, without involving either CPU.

GPUDirect RDMA goes further: the InfiniBand NIC reads from and writes to GPU memory directly, skipping the trip through CPU system memory.

Without GPUDirect RDMA

GPU memory → copy to CPU RAM → NIC → network → NIC → CPU RAM → copy to GPU memory

Extra copies, more latency, and the CPU and PCIe paths become a bottleneck.

With GPUDirect RDMA

GPU memory → NIC → network → NIC → GPU memory

Direct path, low latency, full bandwidth. Each GPU uses its own neighbor NIC.

It needs a driver piece (typically the nvidia-peermem kernel module, or DMA-BUF support on newer stacks). In NCCL debug logs, a working path shows GDRDMA, for example via NET/IB/3/GDRDMA.

7. Understanding the symptom: healthy vs broken

Healthy

NCCL sees all 8 NICs
It builds many "channels," each using a different GPU and NIC pair
All 8 GPUs send and receive across the network in parallel
High bus bandwidth

Broken (this question)

NCCL only sees or uses one NIC (mlx5_0, next to GPU 0)
NCCL picks the GPU closest to that NIC (GPU 0) to do the network sends
The other 7 GPUs pass their data to GPU 0 over NVLink, then wait
Bandwidth is roughly 1/8 of what it should be

Common ways this happens:

NCCL_IB_HCA is set to a single adapter, for example NCCL_IB_HCA=mlx5_0 left over from an old script.
The other IB ports are down or not configured (cables, Subnet Manager, link state).
GPUDirect RDMA isn't working, so NCCL can't do the efficient "each GPU uses its own NIC" path.

A warehouse has 8 loading docks, one next to each team. Someone locked 7 of them. Now every team carries its boxes over to dock 0, and the dock-0 team does all the shipping while everyone else stands around.

8. Ring vs tree algorithms

NCCL can use different algorithms for all-reduce. You can force one with NCCL_ALGO, but NCCL normally chooses automatically.

Algorithm	How it works	Good for
Ring	GPUs form a circle. Each passes chunks to the next until everyone has the full sum.	Large messages (maximizes bandwidth)
Tree	NCCL uses a double binary tree: two trees whose roles are swapped, so every node is a "leaf" in one and an "inner node" in the other. Data flows up to the root, then back down.	Smaller messages and many nodes (lower latency)

Why tree doesn't explain "GPU 0 on every node does everything":

NCCL's double binary tree is designed specifically to balance the load, so no single root carries everything.
NCCL runs many channels in parallel, each mapped to different GPUs and NICs, with or without tree.
A "tree root" would be one GPU in the whole cluster, not GPU 0 on each node.
Choosing an algorithm changes the pattern of traffic. It doesn't remove network adapters.

9. What MIG is

MIG (Multi-Instance GPU) splits one physical H100 into up to 7 smaller, isolated GPUs ("instances"), each with its own memory and compute. It's used to share one GPU among several small jobs, such as inference or notebooks.

Why MIG doesn't fit this symptom:

MIG is for splitting GPUs for small isolated workloads, not for large multi-node training.
MIG instances don't support peer-to-peer NVLink communication, so NCCL multi-GPU jobs typically fail or misbehave visibly. You wouldn't see clean all-reduce traffic with GPU 0 doing all the networking.
MIG is configured per GPU and would affect all the GPUs it's enabled on. Nothing about it makes "the first GPU on each node" special for network traffic.
You'd easily see it in nvidia-smi (MIG mode: Enabled).

10. Weighing the likely causes

WRONGSomeone set NCCL_ALGO=Tree, so all the traffic piles up at a tree root

NCCL's tree is a double binary tree built to spread load, and it runs over many channels on different NICs. A tree root also wouldn't be "GPU 0 on every node." The algorithm changes the traffic pattern, not how many network adapters are used.

WRONGMIG is turned on, so NCCL can only talk through the first GPU instance

MIG splits GPUs for small isolated workloads and doesn't support the peer-to-peer communication NCCL multi-GPU training needs. It would cause visible failures or unsupported configurations, not neat "GPU 0 handles the network" behavior. It's also unrelated to InfiniBand adapters.

WRONGFabric Manager is steering the between-node NVLink traffic onto GPU 0's NVSwitch ports

This mixes up the two networks. Fabric Manager manages NVSwitch inside a node. Cross-node traffic in a standard DGX H100 cluster goes over InfiniBand, which Fabric Manager doesn't control. "NVLink between nodes" is the trap.

CORRECTNCCL can only reach one InfiniBand NIC per node (and GPUDirect RDMA isn't working across all of them)

A DGX H100 has 8 compute NICs, one paired with each GPU. If NCCL can only use one (the one next to GPU 0), or GPUDirect RDMA isn't working, all cross-node traffic funnels through GPU 0 and its NIC while the other GPUs wait. The fix is to expose all adapters (for example with NCCL_IB_HCA), make sure every IB port is up, and enable GPUDirect RDMA.

11. How you would diagnose and fix it in real life

Turn on NCCL logging and look at which NICs it picks:

export NCCL_DEBUG=INFO
# look for lines like:
#   NET/IB : Using [0]mlx5_0:1/IB              <- only one NIC = problem
#   NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB ... [7]mlx5_7:1/IB   <- healthy
#   Channel 03 : ... via NET/IB/3/GDRDMA       <- GPUDirect RDMA in use

Check environment variables for a restrictive setting:

env | grep NCCL
# bad:  NCCL_IB_HCA=mlx5_0
# good: unset (auto-detect), or list all compute HCAs, e.g.
#       NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7

Check that every IB port is up:

ibstat            # each port should show State: Active, Physical state: LinkUp
ibdev2netdev      # maps mlx5_X devices to interfaces and shows Up/Down

Check GPU–NIC topology:

nvidia-smi topo -m    # shows which NIC sits next to which GPU (PIX/PXB = close)

Check that GPUDirect RDMA is enabled:

lsmod | grep nvidia_peermem    # module should be loaded (or DMA-BUF in use)

Re-run all_reduce_perf and confirm bus bandwidth jumps up and all 8 NICs show traffic.

12. How to spot the right answers on similar questions

"Cross-node," "inter-node," "network phase," "between servers": think InfiniBand / NICs / RDMA / NCCL_IB_*. Rule out Fabric Manager and NVSwitch answers.
"Intra-node," "within a node," "between GPUs on the same server": think NVLink / NVSwitch / Fabric Manager.
One GPU (or one NIC) is overloaded while the others sit idle: think "only one path is configured." Look at NCCL_IB_HCA, downed ports, or missing GPUDirect RDMA.
Low bandwidth but balanced across GPUs: think link speed, cabling, congestion, or algorithm and protocol tuning.
"Inter-node NVLink" in a standard DGX H100 InfiniBand cluster is usually a deliberately mixed-up distractor.
MIG in a large-training question is usually a distractor. MIG is for splitting GPUs, not combining them.

13. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

mpirun -np 32 -N 8 ./all_reduce_perf -b 8 -e 8G -f 2 -g 1The slow 4-node AllReduce

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid   4108 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
#  ...
#  Rank  8 Group  0 Pid   5212 on      dgx02 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  ...
#  Rank 31 Group  0 Pid   7483 on      dgx04 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    31.87    0.00    0.00      0    32.02    0.00    0.00      0
          16             4     float     sum      -1    32.00    0.00    0.00      0    31.86    0.00    0.00      0
          32             8     float     sum      -1    32.13    0.00    0.00      0    32.08    0.00    0.00      0
  ...
    67108864      16777216     float     sum      -1  2822.55   23.78   46.07      0  2826.78   23.74   46.00      0
   134217728      33554432     float     sum      -1  5635.65   23.82   46.14      0  5661.01   23.71   45.94      0
   268435456      67108864     float     sum      -1  11171.9   24.03   46.55      0  11121.7   24.14   46.76      0
   536870912     134217728     float     sum      -1  22401.6   23.97   46.43      0  22368.0   24.00   46.50      0
  1073741824     268435456     float     sum      -1  44950.3   23.89   46.28      0  45017.8   23.85   46.21      0
  2147483648     536870912     float     sum      -1  89331.5   24.04   46.58      0  89733.5   23.93   46.37      0
  4294967296    1073741824     float     sum      -1 179346.9   23.95   46.40      0 178539.9   24.06   46.61      0
  8589934592    2147483648     float     sum      -1 356514.2   24.09   46.68      0 355979.4   24.13   46.75      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 20.929
#
# Collective test concluded: all_reduce_perf

What to look at: 32 ranks in one communicator, but busbw tops out near 46 GB/s at 8 GB, about one 400G link's worth instead of roughly 370 GB/s for eight rails. #wrong is 0, so the data is correct and the problem is that only one network path is in use.

export NCCL_DEBUG=INFO; mpirun -np 32 -N 8 -x NCCL_DEBUG ./all_reduce_perf -b 8G -e 8G -g 1 | grep -E "NET/IB : Using|via NET/IB"

Which HCAs NCCL picked

dgx01:4101:4152 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB bond0:10.0.0.11<0>
dgx02:5212:5263 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB bond0:10.0.0.12<0>
dgx03:6323:6374 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB bond0:10.0.0.13<0>
dgx04:7434:7485 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB bond0:10.0.0.14<0>
dgx01:4101:4152 [0] NCCL INFO Channel 00/0 : 31[7] -> 0[0] [receive] via NET/IB/0/GDRDMA
dgx01:4101:4152 [0] NCCL INFO Channel 01/0 : 31[7] -> 0[0] [receive] via NET/IB/0/GDRDMA
dgx01:4101:4152 [0] NCCL INFO Channel 02/0 : 31[7] -> 0[0] [receive] via NET/IB/0/GDRDMA
dgx01:4101:4152 [0] NCCL INFO Channel 03/0 : 31[7] -> 0[0] [receive] via NET/IB/0/GDRDMA
...

What to look at: Every node lists only [0]mlx5_0, and every network channel goes through NET/IB/0 on GPU 0. A healthy node lists [0]mlx5_0 through [7]mlx5_7, and channels spread across NET/IB/0 to NET/IB/7.

env | grep NCCLFind a restrictive NCCL setting

NCCL_DEBUG=INFO
NCCL_IB_HCA=mlx5_0
NCCL_SOCKET_IFNAME=bond0

What to look at: NCCL_IB_HCA=mlx5_0 restricts NCCL to the single HCA next to GPU 0, which is exactly the symptom. Unset it or list all eight compute HCAs (mlx5_0 to mlx5_7); NCCL_DEBUG and NCCL_SOCKET_IFNAME here are harmless.

ibstatIB port state and rate

CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0xa088c20300000000
        System image GUID: 0xa088c20300000000
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 41
                LMC: 0
                SM lid: 1
                Capability mask: 0xa751e848
                Port GUID: 0xa088c20300000000
                Link layer: InfiniBand
CA 'mlx5_1'
        ...
                State: Active
                Physical state: LinkUp
                Rate: 400
...
CA 'mlx5_7'
        ...
                State: Active
                Physical state: LinkUp
                Rate: 400
...

What to look at: Every compute port should show State: Active, Physical state: LinkUp and Rate: 400 for NDR. Down, Polling or Initializing (no Subnet Manager) on mlx5_1 to mlx5_7 would also leave only one usable HCA.

ibdev2netdevMap mlx5 devices to interfaces

mlx5_0 port 1 ==> ibp24s0 (Up)
mlx5_1 port 1 ==> ibp64s0 (Up)
mlx5_2 port 1 ==> ibp79s0 (Up)
mlx5_3 port 1 ==> ibp94s0 (Up)
mlx5_4 port 1 ==> ibp154s0 (Up)
mlx5_5 port 1 ==> ibp192s0 (Up)
mlx5_6 port 1 ==> ibp206s0 (Up)
mlx5_7 port 1 ==> ibp220s0 (Up)
mlx5_8 port 1 ==> ibp41s0f0 (Up)
mlx5_9 port 1 ==> ibp41s0f1 (Up)
mlx5_10 port 1 ==> enp170s0f0np0 (Up)
mlx5_11 port 1 ==> enp170s0f1np1 (Down)

What to look at: All eight compute HCAs (mlx5_0 to mlx5_7 here) should be (Up). Any (Down) among them removes a rail; mlx5_8 and higher are storage or in-band management ports that NCCL should not use for compute traffic.

nvidia-smi topo -mGPU-to-NIC affinity

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0    X       NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU1    NV18    X       NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU2    NV18    NV18    X       NV18    NV18    NV18    NV18    NV18    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU3    NV18    NV18    NV18    X       NV18    NV18    NV18    NV18    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU4    NV18    NV18    NV18    NV18    X       NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    56-111,168-223  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18    X       NV18    NV18    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    56-111,168-223  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18    X       NV18    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    56-111,168-223  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18    X       SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     56-111,168-223  1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     X       NODE    NODE    NODE    SYS     SYS     SYS     SYS
...

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks
  ...

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  ...

What to look at: The diagonal PIX cells show each GPU has its own NIC on the same PCIe switch: GPU0 with NIC0 (mlx5_0), GPU1 with NIC1, and so on. That is why limiting NCCL to mlx5_0 makes GPU 0 carry all the network traffic.

lsmod | grep nvidia_peermemGPUDirect RDMA module check

nvidia_peermem         16384  0
ib_core               434176  9 rdma_cm,ib_ipoib,nvidia_peermem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
nvidia              56389632  1095 nvidia_uvm,nvidia_peermem,nvidia_modeset

What to look at: A loaded nvidia_peermem (or a stack using DMA-BUF) lets each GPU use its own NIC directly, which shows as /GDRDMA in NCCL channel lines. No output and no DMA-BUF means traffic bounces through CPU memory.

unset NCCL_IB_HCA; mpirun -np 32 -N 8 ./all_reduce_perf -b 8 -e 8G -f 2 -g 1Re-run after exposing all HCAs

# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4101 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  ...
#  Rank 31 Group  0 Pid   7483 on      dgx04 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    29.88    0.00    0.00      0    30.01    0.00    0.00      0
          16             4     float     sum      -1    30.00    0.00    0.00      0    29.87    0.00    0.00      0
  ...
    67108864      16777216     float     sum      -1   378.77  177.18  343.28      0   379.33  176.91  342.77      0
   134217728      33554432     float     sum      -1   730.51  183.73  355.98      0   733.80  182.91  354.39      0
   268435456      67108864     float     sum      -1  1422.39  188.72  365.65      0  1415.99  189.57  367.30      0
   536870912     134217728     float     sum      -1  2826.20  189.96  368.05      0  2821.96  190.25  368.60      0
  1073741824     268435456     float     sum      -1  5644.89  190.21  368.54      0  5653.36  189.93  367.99      0
  2147483648     536870912     float     sum      -1  11192.4  191.87  371.75      0  11242.7  191.01  370.08      0
  4294967296    1073741824     float     sum      -1  22444.4  191.36  370.76      0  22343.4  192.23  372.44      0
  8589934592    2147483648     float     sum      -1  44590.2  192.64  373.24      0  44523.3  192.93  373.80      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 132.556
#
# Collective test concluded: all_reduce_perf

What to look at: busbw at 8 GB should jump to about 370 GB/s, roughly eight times the broken run, with #wrong 0. Confirm with NCCL_DEBUG=INFO that NET/IB now lists all eight HCAs on every node.

14. Self-check quiz

15. Glossary

Term	Meaning
NCCL	NVIDIA Collective Communications Library. Moves data between GPUs for all-reduce and similar operations.
nccl-tests	A benchmark suite (`all_reduce_perf` and others) used to measure NCCL bandwidth.
All-reduce	Sum values across all GPUs and give every GPU the result.
Bus bandwidth (busbw)	The nccl-tests metric for effective communication speed. Compare it to the hardware's expected peak.
Intra-node / inter-node	Inside one server / between servers.
NVLink / NVSwitch	High-speed GPU-to-GPU links and switch chips inside a DGX.
Fabric Manager	The service that configures the NVSwitch fabric. Intra-node only.
InfiniBand (IB)	The high-speed, low-latency network that connects nodes in AI clusters.
HCA / NIC	Host Channel Adapter / network card. On DGX H100 these are NVIDIA ConnectX-7 (`mlx5_X` devices).
GPU–NIC affinity	Each GPU has a NIC physically close to it (same PCIe switch). Using that pair gives the best performance.
Rail-optimized network	NIC i on every node connects to the same leaf switch (rail i).
RDMA	Remote Direct Memory Access: the NIC writes into remote memory without involving the CPU.
GPUDirect RDMA (GDR)	RDMA directly to and from GPU memory. Needs `nvidia-peermem` or DMA-BUF.
`NCCL_IB_HCA`	Environment variable that selects which IB adapters NCCL may use.
`NCCL_ALGO`	Environment variable that forces an algorithm (Ring, Tree, and so on). Normally left unset.
`NCCL_DEBUG=INFO`	Makes NCCL print which NICs, paths, and algorithms it chose.
Channel	One parallel communication path NCCL builds. Many channels spread traffic across GPUs and NICs.
Ring / Tree	NCCL algorithms. Ring favors bandwidth, while tree (double binary tree) favors latency at scale.
MIG	Multi-Instance GPU: splits one GPU into isolated smaller GPUs.