NVLink, NVSwitch & nvbandwidth

NVLink generations and bandwidth math, NVSwitch topology, CUDA peer-to-peer, how to check link status and errors, and how to measure NVLink performance with nvbandwidth, p2pBandwidthLatencyTest, nccl-tests, and DCGM. Focused on DGX H100.

1. What NVLink is

NVLink is NVIDIA's high-speed, direct GPU-to-GPU interconnect. It replaces PCIe for GPU-to-GPU traffic inside a server and delivers several times more bandwidth at lower latency. GPUs can read and write each other's memory directly (peer-to-peer), which NCCL, CUDA, and frameworks rely on for fast tensor and model parallelism.

NVLink = the link itself (lanes between chips).
NVSwitch = a switch chip that connects many GPUs' NVLinks, so every GPU talks to every other GPU at full speed (all-to-all).
Fabric Manager = the host service that configures the NVSwitch fabric (see the Fabric Manager guide).

PCIe is a shared city road with traffic lights. NVLink is a private multi-lane highway between GPUs, and the NVSwitch is a giant interchange where every highway meets, so any GPU can reach any other without slowing down.

NVLink is intra-node on DGX H100 (inside the box). Between nodes, traffic goes over InfiniBand (ConnectX-7, 400 Gb/s per port). Rack-scale systems such as GB200 NVL72 extend NVLink across trays, but for DGX H100: NVLink = inside, IB = between.

2. Generations and bandwidth

Generation	GPU	Links per GPU	Per link (bidirectional)	Total per GPU (bidirectional)	Per direction
NVLink 1	P100	4	40 GB/s	160 GB/s	80 GB/s
NVLink 2	V100	6	50 GB/s	300 GB/s	150 GB/s
NVLink 3	A100	12	50 GB/s	600 GB/s	300 GB/s
NVLink 4	H100 / H200	18	50 GB/s	900 GB/s	450 GB/s
NVLink 5	B200 / GB200	18	100 GB/s	1.8 TB/s	900 GB/s

Compared with PCIe (x16)

PCIe	Per direction	Bidirectional	H100 NVLink 4 is…
Gen4 x16	~32 GB/s	~64 GB/s	~14× faster
Gen5 x16 (H100 host link)	~64 GB/s	~128 GB/s	~7× faster

H100 = 18 × 50 = 900. Eighteen links, fifty GB/s each (bidirectional), 900 GB/s per GPU, which is 450 each way. "~7× PCIe Gen5."

3. Bidirectional vs unidirectional

Number	Means
900 GB/s (H100)	Bidirectional total per GPU: 450 GB/s sending + 450 GB/s receiving at the same time
450 GB/s	Unidirectional: the most one GPU can push in one direction
7.2 TB/s (DGX H100)	8 GPUs × 900 GB/s = total bidirectional NVLink bandwidth of the NVSwitch fabric
~3.6 TB/s	The same total counted one direction only

A copy test from GPU 0 → GPU 1 is unidirectional, so compare it with 450 GB/s, not 900. A measurement of ~370–400 GB/s is healthy, not "less than half the spec." Bidirectional tests (both directions at once) are compared against 900.

Watch the units: GB/s (gigabytes, NVLink/HBM specs) vs Gb/s (gigabits, network specs). 400 Gb/s InfiniBand = 50 GB/s.

4. Bandwidth calculator

Generation

Active links per GPU

GPUs in the node

Data to move (GB)

Achievable efficiency (%)

Measured GPU→GPU (GB/s, optional)

5. NVSwitch and DGX H100 topology

4 NVSwitch chips in DGX H100 (DGX A100 had 6). Every GPU connects to all four.
Each GPU's 18 links are split across the switches (commonly described as 5 + 4 + 4 + 5).
Result: non-blocking all-to-all. Any pair of GPUs gets the full per-GPU bandwidth, and all 8 can communicate at once.
H100-generation NVSwitches support NVLink SHARP (NVLS): in-switch reductions and multicast for collectives.
The NVSwitch fabric only works after Fabric Manager configures it. Without it, CUDA fails with error 802.

nvidia-smi topo -m on a healthy DGX H100 shows NV18 between every GPU pair: "connected through 18 NVLinks." Fewer (for example NV16 or NV12), or SYS/PHB between GPUs, means missing links or a broken fabric.

6. The NVLink family

Variant	What it connects	Example	Fabric Manager?
NVLink via NVSwitch (baseboard)	8 GPUs all-to-all inside one server	DGX/HGX A100, H100, H200, B200	Yes
Direct NVLink (no switch)	GPUs wired straight to each other	HGX 4-GPU boards	No
NVLink bridge	A pair (or small group) of PCIe cards	H100 NVL pairs, workstation GPUs	No
NVLink-C2C	CPU ↔ GPU chip-to-chip (coherent)	Grace Hopper GH200 (900 GB/s CPU-GPU)	n/a
NVLink Switch System / rack-scale NVLink	GPUs across trays/nodes	GB200 NVL72 (72 GPUs in one NVLink domain)	Rack-level fabric management

7. CUDA peer-to-peer over NVLink

P2P lets one GPU access another GPU's memory directly (cudaMemcpyPeer, direct loads/stores in kernels).
CUDA checks cudaDeviceCanAccessPeer() and turns access on with cudaDeviceEnablePeerAccess(). Frameworks and NCCL do this for you.
Two copy mechanisms: copy engines (CE), dedicated DMA hardware used by cudaMemcpy, and SM-based copies, where kernels do the loads and stores. nvbandwidth tests both.
If P2P is unavailable, data bounces through host memory over PCIe, which is much slower.

nvidia-smi topo -p2p r      # P2P READ capability matrix between GPUs (OK = supported)
nvidia-smi topo -p2p w      # P2P WRITE capability
nvidia-smi topo -p2p n      # NVLink P2P capability

In VMs/containers, P2P depends on what's passed through: all GPUs and NVSwitches (bare metal or full passthrough), or a Fabric Manager partition in shared NVSwitch mode. A GPU subset that isn't a valid NVLink partition loses NVLink P2P between partitions.

8. Checking links

nvidia-smi topo -m                    # matrix: NV18 between all GPU pairs = healthy DGX H100
nvidia-smi nvlink -s                  # per-GPU, per-link status and speed (e.g. "Link 0: 26.562 GB/s")
nvidia-smi nvlink -s -i 3             # just GPU 3
nvidia-smi nvlink -c                  # link capabilities (P2P, system memory access, …)
nvidia-smi nvlink -e                  # error counters (CRC, replay, recovery)
nvidia-smi -q | grep -A3 -i fabric    # Fabric State: Completed / Status: Success

The per-link speed shown by nvidia-smi nvlink -s is a per-link, per-direction signalling figure (about 26.5 GB/s per link on H100). It won't equal the 50 GB/s bidirectional marketing number per link. What matters is that all 18 links on every GPU are up and report the same speed. Option names for throughput counters vary between driver versions, so check nvidia-smi nvlink -h.

Healthy DGX H100 checklist: NV18 everywhere in topo -m, 18 active links per GPU in nvlink -s, error counters not growing in nvlink -e, fabric state Completed.

9. nvbandwidth: the dedicated bandwidth tool

nvbandwidth is an open-source NVIDIA tool (github.com/NVIDIA/nvbandwidth) that measures memory-copy bandwidth between GPUs and the host and between GPUs, using both copy engines (CE) and SM kernels. It prints matrices (one value per source/destination pair), which makes a weak GPU or link stand out immediately.

nvbandwidth replaces and extends the old CUDA sample p2pBandwidthLatencyTest / bandwidthTest for node validation. It answers "do the raw copy paths (host↔GPU over PCIe, GPU↔GPU over NVLink) reach their expected bandwidth?" with no NCCL or MPI involved.

Build and run

# Build (needs CUDA toolkit, a C++ compiler, CMake, Boost program_options)
git clone https://github.com/NVIDIA/nvbandwidth.git
cd nvbandwidth && cmake . && make

./nvbandwidth -l                                   # list all testcases
./nvbandwidth                                      # run all testcases
./nvbandwidth -t device_to_device_memcpy_write_ce  # one testcase
./nvbandwidth -t host_to_device_memcpy_ce -b 1024  # 1024 MiB buffer
./nvbandwidth -j > results.json                    # JSON output for automation
./nvbandwidth -h                                   # all options (iterations, verbose, …)

Testcase families

Testcase (examples)	Path tested	Compare with
`host_to_device_memcpy_ce`, `device_to_host_memcpy_ce`	CPU memory ↔ each GPU (PCIe)	PCIe Gen5 x16: ~64 GB/s per direction (expect ~50+ achieved)
`host_to_device_bidirectional_memcpy_ce`	Both directions over PCIe at once	~2× per-direction
`all_to_host_memcpy_ce`, `host_to_all_memcpy_ce`	All GPUs ↔ host simultaneously	Shows CPU/PCIe-switch contention
`device_to_device_memcpy_read_ce` / `_write_ce`	GPU ↔ GPU over NVLink, one direction, copy engine	450 GB/s per direction (H100)
`device_to_device_bidirectional_memcpy_read_ce` / `_write_ce`	GPU ↔ GPU, both directions	900 GB/s total
`device_to_device_memcpy_read_sm` / `_write_sm`	GPU ↔ GPU using SM kernels	Same spec. SM copies can behave differently from CE.
`all_to_one_`, `one_to_all_`	Many GPUs to one, or one to many	Stresses the NVSwitch fabric and one GPU's links
`device_local_copy`	Copy within one GPU's HBM	HBM bandwidth context

Testcase names and defaults can change between nvbandwidth releases. Always run ./nvbandwidth -l. There's also a multi-node build (MPI) for rack-scale NVLink systems. On DGX H100, nvbandwidth is a single-node tool.

10. Reading nvbandwidth output

Illustrative DGX H100 output (values are examples):

Running device_to_device_memcpy_write_ce.
memcpy CE GPU(row) -> GPU(column) bandwidth (GB/s)
           0       1       2       3       4       5       6       7
 0       N/A  389.2   389.5   389.1   389.4   389.0   389.3   389.2
 1     389.3    N/A   389.4   389.2   389.1   389.5   389.0   389.3
 2     389.1  389.4     N/A   389.3   389.2   389.4   389.1   389.0
 3     195.2  194.9   195.1     N/A   195.0   195.3   194.8   195.1
 4     389.4  389.2   389.0   389.1     N/A   389.3   389.5   389.2
 5     389.2  389.3   389.4   389.0   389.1     N/A   389.2   389.4
 6     389.0  389.1   389.3   389.2   389.4   389.0     N/A   389.3
 7     389.3  389.4   389.2   389.1   389.0   389.3   389.2     N/A

SUM device_to_device_memcpy_write_ce 51368.7

What to read	Meaning
Row = source, column = destination	Each cell is one GPU→GPU path
~389 GB/s cells	≈86% of the 450 GB/s per-direction spec: healthy
Row 3 at ~195	GPU 3's outgoing bandwidth is halved to every peer, so the problem is GPU 3's own links (for example about half its NVLinks down), not one pair
One single low cell	A problem specific to that pair/path (rare on NVSwitch systems)
All cells low	System-wide: Fabric Manager/degraded fabric, clocks/power, wrong build, or comparing against the wrong spec
`SUM`	Sum of all cells (handy for comparing nodes)

Matrices make diagnosis fast: a bad row or column points to one GPU. Everything low points to the system (fabric, FM, power/clock settings). Compare every node's matrix against a known-good node.

11. nvbandwidth vs the other tools

	nvbandwidth	`p2pBandwidthLatencyTest`	Single-node nccl-tests	`dcgmi diag`	`nvidia-smi topo/nvlink`
Question	Do raw copy paths (PCIe + NVLink) hit their bandwidth?	P2P bandwidth and latency matrix	Do NCCL collectives over NVLink hit the expected busbw?	Is GPU hardware (incl. PCIe/NVLink) healthy? PASS/FAIL	Are links present, up, error-free?
Layer	CUDA memcpy (CE and SM)	CUDA P2P	NCCL library (what training uses)	Hardware diagnostics	Configuration/status (no load)
Tests host↔GPU (PCIe)?	Yes	No (P2P focus; `bandwidthTest` does H2D)	No	Yes (PCIe test in -r 2+)	Shows PCIe topology
Output	GB/s matrices + SUM, JSON	GB/s and µs matrices	algbw/busbw per size, #wrong	PASS/FAIL/WARN per test	NV18, link states, counters
Best for	Pinpointing a weak GPU/link or PCIe path with numbers	Quick legacy check	Communication performance as seen by training	Health screening / RMA evidence	First look, no load needed
Doesn't tell you	NCCL settings, inter-node	Host paths, NCCL	Which raw path is weak (needs isolating)	Fine-grained bandwidth numbers	Performance under load

topo/nvlink = "are the roads there?" · dcgmi diag = "is the car healthy?" · nvbandwidth = "how fast is each road?" · nccl-tests = "how fast is real delivery traffic?"

12. NVLink errors and monitoring

Signal	Where	Meaning
Xid 74	`dmesg`	NVLink error detected by the GPU
SXid	`dmesg`	NVSwitch error
CRC / replay / recovery counters	`nvidia-smi nvlink -e`, DCGM fields	Bit errors (CRC) cause retransmissions (replay). Recovery = link retraining. Rising counters = marginal link.
Fewer links than expected	`topo -m`, `nvlink -s`	Links down: degraded bandwidth
NVLink bandwidth telemetry	DCGM profiling fields (NVLink TX/RX bytes), DCGM Exporter	Live utilization across the fleet
NVLink health watch	`dcgmi health -s a -c` (includes NVLink)	Flags NVLink errors
Fabric state	`nvidia-smi -q`	Completed/Success vs not initialized

dmesg -T | grep -iE "xid|sxid"
nvidia-smi nvlink -e          # run twice: are the counters increasing?
dcgmi health -s a -c
dcgmi diag -r 2               # includes PCIe/NVLink-related checks

13. How NCCL uses NVLink

Inside a node, NCCL uses the P2P transport over NVLink (logs show via P2P/CUMEM or P2P/IPC).
On H100 NVSwitch systems, NCCL can use NVLS (NVLink SHARP) for AllReduce: the switch does the reduction (NCCL_NVLS_ENABLE).
Single-node all_reduce_perf -g 8 on DGX H100 should show high busbw (in the hundreds of GB/s, depending on NCCL version, message size, and NVLS).
NCCL_P2P_DISABLE=1 forces NCCL off NVLink (through host memory): a huge slowdown, only useful for debugging.
NCCL_P2P_LEVEL controls how "far" apart GPUs can be and still use P2P.

If single-node NCCL shows SHM instead of P2P between GPUs, NVLink P2P isn't working: check Fabric Manager, topo -m, the container/VM GPU visibility, and whether NCCL_P2P_DISABLE is set.

14. Validation workflow (per node)

#	Step	Pass
1	`systemctl status nvidia-fabricmanager`, `nvidia-smi -q` fabric state	Active. Completed/Success.
2	`nvidia-smi topo -m`	NV18 between all pairs
3	`nvidia-smi nvlink -s` / `-e`	18 links up per GPU. Error counters 0 or not increasing.
4	`dcgmi diag -r 2` (or -r 3)	All PASS
5	`nvbandwidth` (D2D CE read/write, bidirectional, H2D/D2H)	Uniform matrices. D2D near the per-direction spec. H2D near the PCIe spec. Matches a known-good node.
6	Single-node `all_reduce_perf -b 8 -e 8G -f 2 -g 8`	Expected busbw, `#wrong 0`, P2P (and NVLS) in logs
7	Then move to multi-node: perftest/ClusterKit → multi-node nccl-tests → HPL	See the bring-up guide

15. "Which tool?" chooser

Pick a question above.

16. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

nvidia-smi topo -p2p rP2P read capability matrix

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7
 GPU0   X       OK      OK      OK      OK      OK      OK      OK
 GPU1   OK      X       OK      OK      OK      OK      OK      OK
 GPU2   OK      OK      X       OK      OK      OK      OK      OK
 GPU3   OK      OK      OK      X       OK      OK      OK      OK
 GPU4   OK      OK      OK      OK      X       OK      OK      OK
 GPU5   OK      OK      OK      OK      OK      X       OK      OK
 GPU6   OK      OK      OK      OK      OK      OK      X       OK
 GPU7   OK      OK      OK      OK      OK      OK      OK      X

Legend:

  X    = Self
  OK   = Status Ok
  CNS  = Chipset not supported
  GNS  = GPU not supported
  TNS  = Topology not supported
  NS   = Not supported
  U    = Unknown

What to look at: Every off-diagonal cell should be OK on a healthy DGX H100. Any NS, TNS or CNS means CUDA peer access is not possible between that pair, so traffic falls back through host memory; the same check with w (write) and n (NVLink) should also be all OK.

nvidia-smi topo -p2p nNVLink P2P capability matrix

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7
 GPU0   X       OK      OK      OK      OK      OK      OK      OK
 GPU1   OK      X       OK      OK      OK      OK      OK      OK
 GPU2   OK      OK      X       OK      OK      OK      OK      OK
 GPU3   OK      OK      OK      X       OK      OK      OK      OK
 GPU4   OK      OK      OK      OK      X       OK      OK      OK
 GPU5   OK      OK      OK      OK      OK      X       OK      OK
 GPU6   OK      OK      OK      OK      OK      OK      X       OK
 GPU7   OK      OK      OK      OK      OK      OK      OK      X

Legend:

  X    = Self
  OK   = Status Ok
  CNS  = Chipset not supported
  GNS  = GPU not supported
  TNS  = Topology not supported
  NS   = Not supported
  U    = Unknown

What to look at: This matrix answers whether each pair can do P2P over NVLink specifically. NS here while -p2p r shows OK means the pair only has a PCIe path, which on DGX H100 points at NVLink or Fabric Manager trouble.

nvidia-smi topo -mGPU/NIC topology and CPU affinity

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    0-55,112-167    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     NODE    NODE    0-55,112-167    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    PIX     NODE    0-55,112-167    0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE    PIX     0-55,112-167    0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC1    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC2    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE
NIC3    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X
...

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_3
  NIC2: mlx5_4
  NIC3: mlx5_5
  ...

What to look at: NV18 between every GPU pair means each H100 reaches every other through all 18 NVLink 4 links via NVSwitch. A lower NV# (for example NV12) or SYS/PHB between two GPUs means links are down or Fabric Manager has not set up the fabric; also check each GPU has a PIX NIC and a sensible CPU/NUMA affinity.

nvidia-smi nvlink -sPer-link NVLink status and speed

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-3f1c2a7e-8b40-4d1a-9e6b-0a2c5d7e9f11)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         ...
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-7a2d9c41-15e3-4b8f-a0c2-6e1f3b9d2a47)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         ...
...

What to look at: Each H100 should list 18 links, all at 26.562 GB/s (the per-direction signalling rate of an NVLink 4 link). A link reported as <inactive> or with a lower speed is a degraded path, which shows up as reduced bandwidth for that GPU in nvbandwidth and NCCL.

nvidia-smi nvlink -s -i 3NVLink status for one GPU

GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-0b9f6e3a-2c71-4d8e-95fa-7d3e1c6b8a05)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: <inactive>
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: <inactive>
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s

What to look at: Two inactive links (5 and 11) on GPU 3 leave it with 16 of 18 links, which lowers its NVLink bandwidth and makes it the slow GPU in collectives. Check nvidia-smi nvlink -e and dmesg for Xid 74, then reset the GPU or open a hardware case if the links stay down.

nvidia-smi nvlink -c -i 0NVLink capabilities per link

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-3f1c2a7e-8b40-4d1a-9e6b-0a2c5d7e9f11)
         Link 0, P2P is supported: true
         Link 0, Access to system memory supported: true
         Link 0, P2P atomics supported: true
         Link 0, System memory atomics supported: true
         Link 0, SLI is supported: true
         Link 0, Link is supported: false
         Link 1, P2P is supported: true
         Link 1, Access to system memory supported: true
         ...
         Link 17, P2P atomics supported: true
         Link 17, System memory atomics supported: true
         Link 17, SLI is supported: true
         Link 17, Link is supported: false

What to look at: P2P is supported: true on every link is what matters for GPU-to-GPU traffic. 'Link is supported' refers to a legacy capability and false is normal on H100; a link missing from the list or P2P false indicates a configuration or hardware problem.

nvidia-smi nvlink -e -i 0NVLink error counters

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-3f1c2a7e-8b40-4d1a-9e6b-0a2c5d7e9f11)
         Link 0: Replay Errors: 0
         Link 0: Recovery Errors: 0
         Link 0: CRC Errors: 0
         Link 1: Replay Errors: 0
         Link 1: Recovery Errors: 0
         Link 1: CRC Errors: 0
         ...
         Link 17: Replay Errors: 0
         Link 17: Recovery Errors: 0
         Link 17: CRC Errors: 0

What to look at: Small historical counts can be noise, so run the command twice and compare. Counters that keep increasing, especially Recovery or CRC errors on one link, indicate a failing link that will cause Xid 74 and slow or failed jobs.

nvidia-smi -q | grep -A3 -i fabricFabric Manager registration state

    Fabric
        State                             : Completed
        Status                            : Success
        CliqueId                          : 0
--
    Fabric
        State                             : Completed
        Status                            : Success
        CliqueId                          : 0
...

What to look at: Every GPU should show State Completed and Status Success. In Progress or Not Started, or a failure status, means the GPU has not registered with the NVLink fabric, so CUDA apps get error 802 (system not yet initialized) until Fabric Manager runs correctly.

systemctl status nvidia-fabricmanagerFabric Manager service state

● nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-11-10 09:12:44 +08; 1h 8min ago
    Process: 2711 ExecStart=/usr/bin/nv-fabricmanager-start.sh (code=exited, status=0/SUCCESS)
   Main PID: 2730 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 21.4M
        CPU: 2.113s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2730 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

Nov 10 09:12:41 dgx01 systemd[1]: Starting NVIDIA fabric manager service...
Nov 10 09:12:44 dgx01 nv-fabricmanager[2730]: Connected to 1 node.
Nov 10 09:12:44 dgx01 nv-fabricmanager[2730]: Successfully configured all the available NVSwitches to route GPU NVLink traffic. NVLink Peer-to-Peer support will be enabled once the GPUs are successfully registered with the NVLink fabric.
Nov 10 09:12:44 dgx01 systemd[1]: Started NVIDIA fabric manager service.

What to look at: You want enabled and active (running) plus the 'Successfully configured all the available NVSwitches' message. A failed state, often with a log line about a driver version mismatch, means the Fabric Manager package does not exactly match the driver version.

./nvbandwidth -lList nvbandwidth testcases

nvbandwidth Version: v0.4
Built from Git version: v0.4

Index, Name:
        Description
=======================
0, host_to_device_memcpy_ce:
        Host to device CE memcpy using cuMemcpyAsync
1, device_to_host_memcpy_ce:
        Device to host CE memcpy using cuMemcpyAsync
2, host_to_device_bidirectional_memcpy_ce:
        A host to device copy is measured while a device to host copy is run simultaneously.
        Only the host to device copy bandwidth is reported.
...
6, device_to_device_memcpy_read_ce:
        Measures bandwidth of cuMemcpyAsync between each pair of accessible peers.
        Read tests launch a copy from the peer device to the target using the target's context.
7, device_to_device_memcpy_write_ce:
        Measures bandwidth of cuMemcpyAsync between each pair of accessible peers.
        Write tests launch a copy from the target device to the peer using the target's context.
...

What to look at: Names ending in _ce use copy engines (cudaMemcpy-style DMA) and names ending in _sm use kernels. host_to_device tests exercise PCIe, while device_to_device tests exercise NVLink.

./nvbandwidthRun every nvbandwidth testcase

nvbandwidth Version: v0.4
Built from Git version: v0.4

NOTE: This tool reports current measured bandwidth on your system.
Additional system-specific tuning may be required to achieve maximal peak bandwidth.

CUDA Runtime Version: 12040
CUDA Driver Version: 12040
Driver Version: 550.54.15

Device 0: NVIDIA H100 80GB HBM3
Device 1: NVIDIA H100 80GB HBM3
...
Device 7: NVIDIA H100 80GB HBM3

Running host_to_device_memcpy_ce.
memcpy CE CPU(row) -> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     55.39     55.41     55.37     55.40     55.38     55.42     55.36     55.40

SUM host_to_device_memcpy_ce 443.13

Running device_to_host_memcpy_ce.
memcpy CE CPU(row) <- GPU(column) bandwidth (GB/s)
...

What to look at: The header confirms the runtime, driver and that all 8 GPUs are visible. Read each testcase's matrix for outliers rather than the absolute SUM; a full run takes several minutes.

./nvbandwidth -t device_to_device_memcpy_write_ceGPU-to-GPU NVLink write bandwidth

Running device_to_device_memcpy_write_ce.
memcpy CE GPU(row) -> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0       N/A    389.47    389.12    389.38    389.05    389.33    389.26    389.41
 1    389.47       N/A    389.38    389.05    389.33    389.26    389.41    389.21
 2    389.12    389.38       N/A    389.33    389.26    389.41    389.21    389.47
 3    389.38    389.05    389.33       N/A    389.41    389.21    389.47    389.12
 4    389.05    389.33    389.26    389.41       N/A    389.47    389.12    389.38
 5    389.33    389.26    389.41    389.21    389.47       N/A    389.38    389.05
 6    389.26    389.41    389.21    389.47    389.12    389.38       N/A    389.33
 7    389.41    389.21    389.47    389.12    389.38    389.05    389.33       N/A

SUM device_to_device_memcpy_write_ce 21796.04

What to look at: On a healthy DGX H100 every off-diagonal cell is close to the same value (about 389 GB/s here). One row or column at roughly half the others, like the GPU 3 example in this guide, means that GPU has lost NVLink links.

./nvbandwidth -t host_to_device_memcpy_ce -b 1024CPU-to-GPU PCIe bandwidth, 1 GiB buffer

Running host_to_device_memcpy_ce.
memcpy CE CPU(row) -> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     55.39     55.41     55.37     55.40     55.38     55.42     55.36     55.40

SUM host_to_device_memcpy_ce 443.13

What to look at: About 55 GB/s per GPU matches PCIe Gen5 x16. A GPU near 26 GB/s has trained at Gen4 or x8, and values that differ by CPU socket usually point at NUMA placement or BIOS settings rather than NVLink.

./nvbandwidth -j > results.json && head -n 22 results.jsonJSON results for automation

{
  "nvbandwidth" :
  {
    "CUDA Runtime Version" : 12040,
    "Driver Version" : "550.54.15",
    "git_version" : "v0.4",
    "testcases" :
    [
      {
        "bandwidth_description" : "memcpy CE CPU(row) -> GPU(column) bandwidth (GB/s)",
        "bandwidth_matrix" :
        [
          [
            "55.39",
            "55.41",
            "55.37",
            ...
          ]
        ],
        "name" : "host_to_device_memcpy_ce",
        "status" : "Passed",
        "sum" : 443.13

What to look at: Automation should check each testcase's status is Passed and compare matrix values against a baseline instead of scraping text. A status of Error or Waived means that test did not produce a valid measurement.

dmesg -T | grep -iE "xid|sxid"NVLink (Xid 74) and NVSwitch (SXid) errors

[Tue Nov 10 13:41:07 2026] NVRM: Xid (PCI:0000:5d:00): 74, pid='<unknown>', name=<unknown>, NVLink: fatal error detected on link 5(0x0, 0x0, 0x10000, 0x0, 0x0, 0x0, 0x0)
[Tue Nov 10 13:41:07 2026] NVRM: Xid (PCI:0000:5d:00): 74, pid='<unknown>', name=<unknown>, NVLink: fatal error detected on link 11(0x0, 0x0, 0x10000, 0x0, 0x0, 0x0, 0x0)
[Tue Nov 10 13:41:08 2026] nvidia-nvswitch2: SXid (PCI:0000:07:00.0): 20034, Fatal, Link 36 LTSSM Fault Up
[Tue Nov 10 13:41:08 2026] nvidia-nvswitch2: SXid (PCI:0000:07:00.0): 20034, Severity 5 Engine instance 36 Sub-engine instance 00

What to look at: No output is the healthy result. Xid 74 on the GPU at bus 5d (GPU 3) plus an SXid on the NVSwitch side, as here, means an NVLink link fault; repeated events after a GPU reset usually need a hardware case.

dcgmi health -s a -cSet all health watches and check

Health monitor systems set successfully.
+---------------------------+----------------------------------------------------------+
| Health Monitor Report                                                                |
+===========================+==========================================================+
| Overall Health            | Healthy                                                  |
+---------------------------+----------------------------------------------------------+

What to look at: Healthy is what you want. Warning or Failure lists the affected GPU and subsystem, for example an NVLink error-count watch on one GPU, which tells you where to look next.

dcgmi diag -r 2Medium DCGM diagnostic

Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 3.3.5                                          |
| Driver Version Detected   | 550.54.15                                      |
| GPU Device IDs Detected   | 2330,2330,2330,2330,2330,2330,2330,2330        |
|-----  Deployment  --------+------------------------------------------------|
| Denylist                  | Pass                                           |
| NVML Library              | Pass                                           |
| CUDA Main Library         | Pass                                           |
| Permissions and OS Blocks | Pass                                           |
| Persistence Mode          | Pass                                           |
| Environment Variables     | Pass                                           |
| Page Retirement/Row Remap | Pass                                           |
| Graphics Processes        | Pass                                           |
| Inforom                   | Pass                                           |
+-----  Integration  -------+------------------------------------------------+
| PCIe                      | Pass - All                                     |
+-----  Hardware  ----------+------------------------------------------------+
| GPU Memory                | Pass - All                                     |
+-----  Stress  ------------+------------------------------------------------+
+---------------------------+------------------------------------------------+

What to look at: All rows should read Pass. A Fail on PCIe (which includes P2P/NVLink bandwidth checks) names the failing GPU; run -r 3 for longer stress tests before returning the node to service.

all_reduce_perf -b 8 -e 8G -f 2 -g 8Single-node NCCL all-reduce over NVLink

# nThread 1 nGpus 8 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid  41231 on      dgx01 device  0 [0x18] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid  41231 on      dgx01 device  1 [0x2a] NVIDIA H100 80GB HBM3
...
#  Rank  7 Group  0 Pid  41231 on      dgx01 device  7 [0xdb] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    22.51    0.00    0.00      0    22.10    0.00    0.00      0
          16             4     float     sum      -1    22.36    0.00    0.00      0    22.29    0.00    0.00      0
...
   268435456      67108864     float     sum      -1   1102.4  243.50  426.13      0   1100.8  243.86  426.75      0
   536870912     134217728     float     sum      -1   2071.3  259.19  453.59      0   2069.9  259.37  453.90      0
  1073741824     268435456     float     sum      -1   4022.6  266.93  467.12      0   4020.1  267.09  467.41      0
  2147483648     536870912     float     sum      -1   7944.2  270.32  473.06      0   7940.8  270.44  473.27      0
  4294967296    1073741824     float     sum      -1  15790.3  272.00  476.00      0  15785.1  272.09  476.16      0
  8589934592    2147483648     float     sum      -1  31466.7  272.99  477.73      0  31461.2  273.03  477.80      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 162.418
#

What to look at: Read busbw at the largest sizes (about 470-480 GB/s on DGX H100 with NVLink SHARP) and confirm #wrong is 0. A much lower plateau, or one that drops sharply when you include a specific GPU, points at a degraded NVLink path or P2P being disabled.

17. Scenarios

18. Searchable reference

Item	Meaning

19. Flashcards (click to flip)

20. Exam traps

H100 NVLink = 900 GB/s bidirectional per GPU (18 links × 50 GB/s), so 450 GB/s per direction. A one-way copy at ~390 GB/s is healthy.
DGX H100 total = 7.2 TB/s bidirectional (8 × 900) across 4 NVSwitches.
A100 = 600 GB/s (12 links). B200 = 1.8 TB/s (NVLink 5).
NVLink ≈ 7× PCIe Gen5 x16 (128 GB/s bidirectional).
NV18 in topo -m = healthy DGX H100 GPU pair.
nvbandwidth = raw copy bandwidth matrices (CE/SM, host↔GPU and GPU↔GPU). nccl-tests = collective performance. DCGM diag = health PASS/FAIL.
A bad row/column in an nvbandwidth matrix → one GPU's links. Everything low → system-wide (FM, fabric, clocks).
Xid 74 = NVLink error. SXid = NVSwitch error. Rising CRC/replay counters = marginal link.
NVLink is intra-node on DGX H100. Inter-node is InfiniBand.
NVSwitch systems need Fabric Manager before any NVLink P2P works.
NVLink-C2C = CPU↔GPU (Grace Hopper), not GPU↔GPU across a switch.
NCCL_P2P_DISABLE=1 disables NVLink use by NCCL: a debugging step, never a performance fix.

21. Practice quiz (25 questions)

Score: 0 / 0