NVIDIA Fabric Manager: Mastery Guide

What it is, why NVSwitch systems can't run without it, the boot sequence, version matching, configuration, failure modes, multi-tenancy, monitoring, and troubleshooting. Focused on DGX/HGX A100 and H100.

1. What Fabric Manager is

NVIDIA Fabric Manager (FM) is a host service (systemd unit nvidia-fabricmanager, daemon nv-fabricmanager) that configures and manages the NVSwitch fabric inside NVSwitch-based servers such as DGX/HGX A100 and DGX/HGX H100.

In those servers, GPUs don't connect to each other directly. Every GPU's NVLinks go to NVSwitch chips, and the switches connect every GPU to every other GPU at full bandwidth. But the switches start out unprogrammed. Something has to:

Discover the GPUs and NVSwitches and check the topology matches what's expected.
Train and bring up the NVLink connections.
Program the NVSwitch routing tables so each GPU's memory addresses are reachable by every other GPU (the "memory fabric").
Tell the driver the fabric is ready, so CUDA can start.
Watch for errors (link, NVSwitch, GPU failures) for the life of the system, and handle them.
Optionally, partition the GPUs into isolated NVLink groups for multi-tenant virtualization.

That something is Fabric Manager.

The NVSwitches are a telephone exchange connecting 8 offices (GPUs). When the building powers on, all the switchboards are blank. Fabric Manager is the operator who checks every line, plugs in all the routes, announces "lines are open," and stays on duty to reroute or report when a line breaks. Until the operator finishes, nobody can make a call (CUDA can't start).

On an NVSwitch system, no Fabric Manager = no usable GPUs for CUDA. nvidia-smi still lists all 8 GPUs (it uses NVML, which doesn't need the fabric), but CUDA apps fail with error 802: "system not yet initialized" (cudaErrorSystemNotReady).

2. Which systems need it

System	NVSwitch?	FM required?
DGX-2 / HGX-2 (V100, 16 GPUs)	Yes (12× gen-1 NVSwitch)	Yes
DGX A100 / HGX A100 8-GPU	Yes (6× gen-2 NVSwitch)	Yes
DGX H100 / HGX H100 8-GPU (and H200)	Yes (4× gen-3 NVSwitch, 18 NVLink 4 links per GPU)	Yes
HGX A100 4-GPU / HGX H100 4-GPU	No (GPUs are directly NVLinked)	No
PCIe GPUs (L40S, H100 PCIe), including NVLink-bridged pairs (H100 NVL)	No	No
Workstations / consumer GPUs	No	No

"No switch, no manager." Fabric Manager exists because of NVSwitch. Direct GPU-to-GPU NVLink (bridges, 4-GPU boards) doesn't need it.

NVSwitch generation names vary between sources (the H100's NVSwitch is often called "3rd-generation NVSwitch" and its NVLink "4th-generation NVLink"). The point that matters: DGX H100 = 4 NVSwitch chips, 7.2 TB/s total intra-node bidirectional bandwidth, 900 GB/s bidirectional per GPU.

3. Architecture

Each H100 has 18 NVLink 4 links spread across the 4 NVSwitches (each GPU connects to every switch). 18 × 50 GB/s = 900 GB/s bidirectional per GPU.
FM uses the NVSwitch driver (part of the NVIDIA kernel driver on HGX; device nodes /dev/nvidia-nvswitch*) and the GPU driver to program the hardware.
FM is a control plane: GPU data traffic never passes through the FM process. It sets up the routes, and the hardware does the forwarding.

FM is control plane only. If FM crashes after setup, the routes are already programmed in hardware, but FM is no longer there to manage errors or new jobs (see section 10 for what happens to CUDA jobs).

Fabric Manager manages NVLink/NVSwitch inside the node. It has nothing to do with InfiniBand. The IB fabric between nodes is managed by the Subnet Manager (OpenSM / UFM). Don't mix them up: "IB ports stuck in Initializing" → Subnet Manager. "CUDA error 802 on HGX" → Fabric Manager.

4. The boot / initialization sequence

#	Step	What you'd see if it fails here
1	Server powers on. BIOS enumerates GPUs and NVSwitches on PCIe.	GPUs/switches missing in `lspci`
2	NVIDIA kernel driver loads (`nvidia.ko`, including NVSwitch support)	`nvidia-smi` fails, `lsmod` missing modules
3	`nvidia-persistenced` keeps the driver initialized	Slow first CUDA calls
4	`nvidia-fabricmanager` starts. Checks the driver version matches its own.	FM fails to start: version mismatch in the log
5	FM discovers GPUs and switches and checks the topology against what it expects for this platform	Topology errors: missing GPU/NVSwitch
6	FM trains NVLinks and programs NVSwitch routing	NVLink training failures, degraded mode
7	FM marks the fabric state "Completed / Success", and the driver lets CUDA initialize	Before this point: CUDA error 802
8	Slurm (`slurmd`), DCGM, and other services start. Jobs can run.	Jobs fail if they start before FM finishes
9	FM stays resident, monitoring and handling errors	SXid errors, degraded operation

FM must finish before CUDA workloads start. On a BCM/Slurm cluster, FM must be enabled in the software image so it starts on every boot. Otherwise a reprovisioned node joins Slurm but every GPU job fails with 802.

5. The version-match rule

The Fabric Manager version must exactly match the NVIDIA driver version, for example driver 535.161.08 ↔ FM 535.161.08. Not just the same branch: the exact same version.

nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1   # 535.161.08
nv-fabricmanager --version                                              # Fabric Manager version is : 535.161.08
dpkg -l | grep -E "nvidia-fabricmanager|nvidia-driver"                  # Debian/Ubuntu (DGX OS)
rpm -qa | grep -E "fabricmanager|nvidia-driver"                         # RHEL-based

After a driver upgrade without a matching FM upgrade, FM refuses to start, so CUDA error 802 appears on every GPU. The fix is to install the FM package that matches the driver (or roll the driver back), not to reinstall CUDA or reboot repeatedly.

"Driver and FM are twins: same birthday, same version." Upgrade them together, and hold or pin both packages so routine updates can't separate them.

6. Install, enable, upgrade

# Install matching FM (Ubuntu example: the package name carries the driver branch)
sudo apt install nvidia-fabricmanager-535          # must match the installed driver version exactly
sudo systemctl enable --now nvidia-fabricmanager
systemctl status nvidia-fabricmanager

# Pin both so they upgrade together
sudo apt-mark hold nvidia-driver-535 nvidia-fabricmanager-535   # names vary by distro / repo

# RHEL-style (module streams)
sudo dnf module install nvidia-driver:535-dkms
sudo dnf install nvidia-fabricmanager-535

Safe upgrade procedure (one node)

Drain the node in Slurm (scontrol update nodename=dgx01 state=drain reason="driver upgrade").
Stop GPU services: systemctl stop nvidia-fabricmanager (and DCGM, persistenced if needed).
Upgrade the driver and FM together to the same version.
Reboot (loads the new kernel modules cleanly).
Verify: versions match, FM active, fabric state Completed, deviceQuery / single-node nccl-tests pass.
Resume the node.

In a BCM cluster, do this in the software image on the head node (driver + FM + enable the service), then reprovision or sync the nodes. Upgrading by hand on each node is overwritten at the next sync.

On DGX OS, Fabric Manager is preinstalled and enabled. On other HGX servers built on a standard Linux distro, you have to install and enable it yourself. That's a common miss on new HGX deployments.

7. Verifying it works

systemctl status nvidia-fabricmanager           # active (running)
journalctl -u nvidia-fabricmanager -b           # this boot's FM messages
sudo tail -n 50 /var/log/fabricmanager.log      # default FM log file

nvidia-smi -q | grep -A 3 -i fabric             # Fabric: State : Completed, Status : Success
nvidia-smi topo -m                              # GPU-GPU entries show NV18 on DGX H100
nvidia-smi nvlink -s                            # per-link status and speed for each GPU
nvidia-smi nvlink -e                            # NVLink error counters

# Prove CUDA and the NVLink data path
./deviceQuery                                    # Result = PASS
./p2pBandwidthLatencyTest                        # high P2P bandwidth between all pairs
all_reduce_perf -b 8 -e 8G -f 2 -g 8             # single-node nccl-tests over NVLink

Check	Healthy	Unhealthy
Fabric state in `nvidia-smi -q`	Completed / Success	Not Started / In Progress / failure status
`topo -m`	`NV18` between all GPU pairs (H100)	Fewer links, or `SYS`/`PHB` (no NVLink)
FM log	Reports all NVSwitches configured successfully	Version mismatch, NVLink training failure, missing devices
CUDA	deviceQuery PASS	Error 802

8. The config file: `fabricmanager.cfg`

Default location: /usr/share/nvidia/nvswitch/fabricmanager.cfg. Restart FM after changes (on a drained node).

Option	Meaning
`LOG_LEVEL`	Log verbosity (0 = critical … 4 = most verbose). Raise it temporarily when debugging.
`LOG_FILE_NAME`	Log path (default `/var/log/fabricmanager.log`)
`LOG_APPEND_TO_LOG`, `LOG_FILE_MAX_SIZE`	Append vs overwrite, size limit
`LOG_USE_SYSLOG`	Send logs to syslog instead of the file
`DAEMONIZE`	Run as a background daemon (1 under systemd)
`FABRIC_MODE`	0 = bare metal / full passthrough (default). 1 = Shared NVSwitch multi-tenancy. 2 = vGPU multi-tenancy.
`FABRIC_MODE_RESTART`	Restart FM while preserving the existing fabric state (used with multi-tenancy so active partitions aren't torn down)
`STATE_FILE_NAME`	Where FM saves state for that restart
`BIND_INTERFACE_IP`, `STARTING_TCP_PORT` / `UNIX_DOMAIN_SOCKET`	Internal communication endpoints
`FM_CMD_BIND_INTERFACE`, `FM_CMD_PORT_NUMBER`	Endpoint for the FM API (used by partition management tools / hypervisor integrations)
`FM_STAY_RESIDENT_ON_FAILURES`	Keep FM running (instead of exiting) if fabric init hits errors
`ACCESS_LINK_FAILURE_MODE`	Policy when a GPU ↔ NVSwitch link fails
`TRUNK_LINK_FAILURE_MODE`	Policy when an NVSwitch ↔ NVSwitch trunk link fails (multi-baseboard systems)
`NVSWITCH_FAILURE_MODE`	Policy when an entire NVSwitch fails
`ABORT_CUDA_JOBS_ON_FM_EXIT`	Whether running CUDA jobs are aborted if FM exits
`TOPOLOGY_FILE_PATH`	Where FM finds its platform topology files

Option names and defaults can shift between FM releases. Check the comments in your installed fabricmanager.cfg and the Fabric Manager User Guide for your driver branch. For the exam, FABRIC_MODE (0/1/2) and the existence of failure-mode policies are what matter.

9. Fabric modes

	Bare metal / full passthrough (mode 0)	Shared NVSwitch multi-tenancy (mode 1)	vGPU multi-tenancy (mode 2)
Who uses the GPUs	The host OS (or one VM given all GPUs + NVSwitches)	Several VMs, each getting a partition of whole GPUs passed through	VMs using NVIDIA vGPU on the GPUs
Where FM runs	The host (or in the full-passthrough VM)	A separate service VM (or the hypervisor host) that owns the NVSwitches	The hypervisor host
NVLink scope	All 8 GPUs	Only within a partition (for example 1×8, 2×4, 4×2, 8×1 GPUs)	Managed by the vGPU stack
Typical use	DGX / training clusters	Cloud providers selling 2/4/8-GPU NVLink VMs	Virtualized enterprise GPU sharing

Shared NVSwitch partitions in practice

FM offers a fixed set of predefined partitions for the platform (GPU groups that are valid NVLink domains).
An orchestrator (hypervisor integration, or the sample fmpm partition-manager tool) activates a partition before starting the VM that gets those GPUs, and deactivates it afterwards.
GPUs in different partitions can't reach each other over NVLink, which gives isolation between tenants.

fmpm -l          # list supported partitions and their state
fmpm -a 1        # activate partition 1
fmpm -d 1        # deactivate partition 1

Don't confuse FM partitions (groups of whole GPUs isolated at the NVLink level, for VMs) with MIG (slicing one GPU into instances). Different layers, different tools.

10. Failures, degraded modes, and FM restarts

Hardware failure types

Failure	What it means	Typical effect
Access link failure	A link between a GPU and an NVSwitch is down	Depending on policy, the GPU is removed from the NVLink fabric or the system runs with reduced bandwidth. Replace or reseat hardware.
Trunk link failure	A link between NVSwitches (between baseboards on multi-baseboard systems) is down	Reduced cross-baseboard bandwidth or a degraded fabric, depending on policy
NVSwitch failure	A whole switch chip fails	The fabric loses a share of its bandwidth, or GPUs/partitions are disabled, depending on policy
GPU failure / fell off bus	Xid 79 etc.	FM may fail init or exclude the GPU. Needs a reset or service.

The *_FAILURE_MODE options decide how FM behaves when hardware is partly broken: keep running degraded (fewer GPUs or less bandwidth) or refuse to bring up the fabric. Either way, a degraded fabric means drain the node and open a hardware case, not "tune NCCL."

When the FM process stops or crashes

The routes already in hardware stay, but FM can no longer manage errors.
New CUDA processes generally can't initialize properly until FM is running and the fabric is healthy again (error 802 or similar).
Running CUDA jobs may be aborted, depending on ABORT_CUDA_JOBS_ON_FM_EXIT.
Restarting FM on a bare-metal node usually means: drain the node, stop GPU jobs, restart FM (or reboot), then verify the fabric state.

GPU resets on NVSwitch systems

A GPU reset also affects the fabric. The safe pattern: stop all GPU workloads → stop FM (and DCGM / persistenced if they hold the GPU) → nvidia-smi -r as supported for that platform → start FM → verify the fabric state. If unsure, reboot the node. That's the cleanest way to reinitialize driver + FM together.

11. Monitoring

Signal	Where	Meaning
SXid errors	`dmesg` / kernel log, prefixed by the NVSwitch driver	NVSwitch errors (the switch equivalent of GPU Xid). Fatal SXids usually need a reset or service.
Xid 74	`dmesg`	NVLink error detected by the GPU (link problem: cable/board/GPU)
Xid 79	`dmesg`	GPU fell off the bus
FM log	`/var/log/fabricmanager.log`, `journalctl -u nvidia-fabricmanager`	Init results, link training, errors FM handled
NVLink counters	`nvidia-smi nvlink -s` / `-e`	Link status and speed, CRC/replay/recovery error counts
Fabric state	`nvidia-smi -q` (Fabric section)	Completed/Success vs not initialized
DCGM	`dcgmi discovery -l`, health watches, NVLink fields, `dcgmi diag`	Cluster-wide NVLink/NVSwitch health. On H100, DCGM's NVSwitch monitoring uses the NSCQ library (`libnvidia-nscq`), which must match the driver too.

dmesg -T | grep -iE "sxid|xid"                  # NVSwitch (SXid) and GPU (Xid) errors
nvidia-smi nvlink -e                            # error counters per link
dcgmi discovery -l                              # GPUs and NVSwitches DCGM can see
dcgmi health -s a -c                            # enable all health watches and check (includes NVLink)
dcgmi diag -r 2                                 # includes NVLink / P2P checks

Xid = GPU errors. SXid = (NV)Switch errors. Xid 74 = NVLink.

12. How FM interacts with other components

Component	Relationship	Gotcha
NVIDIA driver	FM controls the NVSwitches through the driver	Exact version match required
CUDA	CUDA waits for the fabric to be ready	Error 802 if FM isn't done or not running
nvidia-smi / NVML	Shows GPUs even without FM. Shows fabric state and NVLink status.	"nvidia-smi works" ≠ "FM works"
NCCL	Uses NVLink P2P (and NVLS) through the fabric FM set up	A degraded fabric gives low single-node busbw. NVLS needs a healthy FM-managed fabric.
DCGM	Monitors NVLink/NVSwitch health. Doesn't replace FM.	DCGM diag may fail P2P/NVLink tests if the fabric is degraded
Slurm	Runs jobs once the GPUs are usable	Make `slurmd` start after FM, or use a health check that verifies the fabric
BCM	Provisions the driver + FM in the image and enables the service	Upgrade both in the image. Don't fix by hand on nodes.
Containers	FM runs on the host, not in containers	802 inside a container → check FM on the host
MIG	MIG slices a GPU. FM manages NVLink between GPUs.	MIG instances don't use NVLink P2P
InfiniBand / Subnet Manager	Unrelated: SM manages the inter-node IB fabric	Don't use FM to fix IB, or SM to fix NVLink

The two "fabric managers" of a DGX cluster: Fabric Manager = NVLink/NVSwitch inside each node. Subnet Manager (UFM/OpenSM) = InfiniBand between nodes.

13. NVLink SHARP (NVLS) and FM

H100-generation NVSwitches can do in-switch reductions (NVLink SHARP, "NVLS"): the switch adds up data from the GPUs itself, instead of the GPUs passing partial sums around a ring. NCCL can use this for AllReduce inside a node (algorithm NVLS; controlled by NCCL_NVLS_ENABLE), which raises effective AllReduce bandwidth and frees GPU resources.

NVLS relies on multicast features of the NVSwitch fabric that FM and the driver set up.
If the fabric isn't fully healthy, or the software stack is too old, NCCL just doesn't select NVLS (look for NVLS in NCCL_DEBUG=INFO output).

Don't confuse NVLink SHARP (inside the NVSwitch, intra-node) with InfiniBand SHARP (in the Quantum IB switches, inter-node, used through the NCCL SHARP plugin / CollNet).

14. Beyond the box: multi-node NVLink (awareness)

Newer rack-scale systems (for example GB200 NVL72) extend NVLink across trays using external NVLink Switch trays. There, NVLink fabric management happens at the rack level (switch-tray software, an NVLink subnet manager) rather than only through a per-host FM process. Multi-node GPU memory sharing also involves the IMEX service. Even inside a single HGX/DGX B200/B300, the fifth-generation NVLink stack adds packages such as nvlsm (NVLink subnet manager) and libnvsdm. DGX OS 7 requires updating the GPU driver and these NVLink stack packages together, just like the FM-must-match-driver rule. For a DGX H100 / BasePOD exam, focus on the per-node FM model above. Just know that rack-scale NVLink changes who manages the fabric.

15. Symptom troubleshooter

Pick a symptom.

Pick a symptom above.

16. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1Installed NVIDIA driver version

535.161.08

What to look at: This is the version Fabric Manager must match exactly. Write it down and compare it with nv-fabricmanager --version; a different patch number such as 535.183.01 is already a mismatch.

nv-fabricmanager --versionInstalled Fabric Manager version

Fabric Manager version is : 535.161.08

What to look at: The full string must equal the driver version from nvidia-smi, digit for digit. If the driver says 535.183.01 and this says 535.161.08, FM will refuse to start and CUDA jobs fail with error 802.

dpkg -l | grep -E "nvidia-fabricmanager|nvidia-driver"Driver and FM package versions

hi  nvidia-driver-535                     535.161.08-0ubuntu1    amd64        NVIDIA driver metapackage
hi  nvidia-fabricmanager-535              535.161.08-1           amd64        Fabric Manager for NVSwitch based systems.

What to look at: Both packages should show the same version, and the leading hi means installed and held, so apt upgrade cannot move one without the other. A plain ii on only one of them, or two different versions, is how a routine upgrade breaks FM.

systemctl status nvidia-fabricmanagerIs the FM service running?

● nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-11-10 08:01:09 UTC; 3h 12min ago
    Process: 2297 ExecStart=/usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg (code=exited, status=0/SUCCESS)
   Main PID: 2311 (nv-fabricmanage)
      Tasks: 18 (limit: 2473640)
     Memory: 18.6M
        CPU: 1min 2.114s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2311 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

Nov 10 08:00:52 dgx01 systemd[1]: Starting NVIDIA fabric manager service...
Nov 10 08:01:09 dgx01 nv-fabricmanager[2311]: Connected to 1 node.
Nov 10 08:01:09 dgx01 nv-fabricmanager[2311]: Successfully configured all the available NVSwitches to route GPU NVLink traffic. NVLink Peer-to-Peer support will be enabled once the GPUs are successfully registered with the NVLink fabric.
Nov 10 08:01:09 dgx01 systemd[1]: Started NVIDIA fabric manager service.

What to look at: You want Loaded ... enabled and Active: active (running). Active: failed (Result: exit-code), or disabled on a reprovisioned BCM node, means GPU jobs will hit CUDA error 802.

journalctl -u nvidia-fabricmanager -bFM messages for this boot

Nov 10 08:00:52 dgx01 systemd[1]: Starting NVIDIA fabric manager service...
Nov 10 08:01:09 dgx01 nv-fabricmanager[2311]: Connected to 1 node.
Nov 10 08:01:09 dgx01 nv-fabricmanager[2311]: Successfully configured all the available NVSwitches to route GPU NVLink traffic. NVLink Peer-to-Peer support will be enabled once the GPUs are successfully registered with the NVLink fabric.
Nov 10 08:01:09 dgx01 systemd[1]: Started NVIDIA fabric manager service.

What to look at: The key line is Successfully configured all the available NVSwitches. A version mismatch message, followed by nvidia-fabricmanager.service: Failed with result 'exit-code', points straight at a driver/FM upgrade that went out of step.

sudo tail -n 50 /var/log/fabricmanager.logDefault FM log file

[Nov 10 2026 08:01:02] [INFO] [tid 2311] Fabric Manager version 535.161.08 is running with the following configuration options
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Logging level = 4
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Logging file name/path = /var/log/fabricmanager.log
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Append to log file = 1
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Max Log file size = 1024 (MBs)
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Use Syslog file = 0
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Fabric Manager communication ports = 16000
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Fabric Mode = 0
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Fabric Mode Restart = 0
[Nov 10 2026 08:01:02] [INFO] [tid 2311] FM Library communication bind interface = 127.0.0.1
[Nov 10 2026 08:01:02] [INFO] [tid 2311] FM Library communication port number = 6666
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Continue to run when facing failures = 0
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Option when facing GPU to NVSwitch NVLink failure = 0
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Option when facing NVSwitch failure = 0
[Nov 10 2026 08:01:02] [INFO] [tid 2311] Abort CUDA jobs when FM exits = 1
...
[Nov 10 2026 08:01:09] [INFO] [tid 2311] Connected to 1 node.
[Nov 10 2026 08:01:09] [INFO] [tid 2311] Successfully configured all the available NVSwitches to route GPU NVLink traffic. NVLink Peer-to-Peer support will be enabled once the GPUs are successfully registered with the NVLink fabric.

What to look at: The header echoes the fabricmanager.cfg options FM actually loaded, such as Fabric Mode = 0 for bare metal and the failure-mode policies. Look for the final Successfully configured line; version mismatch, NVLink training failure, or missing NVSwitch messages instead of it mean the fabric is not up.

nvidia-smi -q | grep -A 3 -i fabricPer-GPU fabric registration state

    Fabric
        State                             : Completed
        Status                            : Success
--
    Fabric
        State                             : Completed
        Status                            : Success
--
... (same block for all 8 GPUs)

What to look at: Every GPU must show State : Completed and Status : Success. In Progress or Not Started (with Status N/A) means CUDA is not allowed to initialize yet, which is the classic CUDA error 802 situation.

nvidia-smi topo -mGPU-to-GPU link matrix

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity     NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    0-55,112-167     0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    0-55,112-167     0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    0-55,112-167     0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    0-55,112-167     0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    56-111,168-223   1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    56-111,168-223   1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    56-111,168-223   1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      56-111,168-223   1               N/A
(NIC0-NIC11 columns trimmed)

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

What to look at: On a healthy DGX H100 every GPU pair shows NV18 (18 bonded NVLinks through the NVSwitches). A pair showing NV12 or fewer means degraded links, and SYS or PHB between GPUs means NVLink is not being used at all.

nvidia-smi nvlink -sPer-link NVLink status and speed

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5f3a1c2e-0000-4a1b-9c3d-00000000d001)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-5f3a1c2e-0001-4a1b-9c3d-00000000d001)
         Link 0: 26.562 GB/s
...

What to look at: Each H100 should list 18 links (0 to 17), all at the same speed. A link that shows <inactive> or is missing from the list is a down access link, which matches a topo -m entry below NV18.

nvidia-smi nvlink -eNVLink error counters per link

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5f3a1c2e-0000-4a1b-9c3d-00000000d001)
         Link 0: Replay Errors: 0
         Link 0: Recovery Errors: 0
         Link 0: CRC Errors: 0
         Link 1: Replay Errors: 0
         Link 1: Recovery Errors: 0
         Link 1: CRC Errors: 0
         Link 2: Replay Errors: 0
         Link 2: Recovery Errors: 0
         Link 2: CRC Errors: 0
...
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-5f3a1c2e-0005-4a1b-9c3d-00000000d001)
         Link 4: Replay Errors: 0
         Link 4: Recovery Errors: 0
         Link 4: CRC Errors: 0
         Link 5: Replay Errors: 1843
         Link 5: Recovery Errors: 12
         Link 5: CRC Errors: 9021
...

What to look at: Healthy links stay at or near zero. Counters that keep climbing on one link, like GPU 5 link 5 here, mean a marginal link: drain the node and run diagnostics rather than tuning NCCL.

./deviceQueryProve CUDA initializes (no 802)

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 8 CUDA Capable device(s)

Device 0: "NVIDIA H100 80GB HBM3"
  CUDA Driver Version / Runtime Version          12.2 / 12.2
  CUDA Capability Major/Minor version number:    9.0
  Total amount of global memory:                 81008 MBytes (84942979072 bytes)
  (132) Multiprocessors, (128) CUDA Cores/MP:    16896 CUDA Cores
  ...
  Device PCI Domain ID / Bus ID / location ID:   0 / 27 / 0
  ...
> Peer access from NVIDIA H100 80GB HBM3 (GPU0) -> NVIDIA H100 80GB HBM3 (GPU1) : Yes
> Peer access from NVIDIA H100 80GB HBM3 (GPU0) -> NVIDIA H100 80GB HBM3 (GPU2) : Yes
...
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.2, CUDA Runtime Version = 12.2, NumDevs = 8
Result = PASS

What to look at: You need Detected 8 CUDA Capable device(s), peer access Yes between GPUs, and Result = PASS. If FM is not ready you instead get cudaGetDeviceCount returned 802 (system not yet initialized) and Result = FAIL.

./p2pBandwidthLatencyTestGPU-to-GPU P2P bandwidth over NVLink

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA H100 80GB HBM3, pciBusID: 1b, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA H100 80GB HBM3, pciBusID: 43, pciDeviceID: 0, pciDomainID:0
...
P2P Connectivity Matrix
     D\D     0     1     2     3     4     5     6     7
     0     1     1     1     1     1     1     1     1
     1     1     1     1     1     1     1     1     1
...
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D       0       1       2       3       4       5       6       7
     0 2615.18  391.42  391.42  391.42  391.42  391.42  391.42  391.42
     1  391.42 2615.18  391.42  391.42  391.42  391.42  391.42  391.42
     2  391.42  391.42 2615.18  391.42  391.42  391.42  391.42  391.42
     3  391.42  391.42  391.42 2615.18  391.42  391.42  391.42  391.42
     4  391.42  391.42  391.42  391.42 2615.18  391.42  391.42  391.42
     5  391.42  391.42  391.42  391.42  391.42 2615.18  391.42  391.42
     6  391.42  391.42  391.42  391.42  391.42  391.42 2615.18  391.42
     7  391.42  391.42  391.42  391.42  391.42  391.42  391.42 2615.18
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D       0       1       2       3       4       5       6       7
     0 2654.71  744.36  744.36  744.36  744.36  744.36  744.36  744.36
     1  744.36 2654.71  744.36  744.36  744.36  744.36  744.36  744.36
...

What to look at: With P2P enabled every off-diagonal cell should be roughly equal and high (hundreds of GB/s on H100). One row or column far lower than the rest, or P2P connectivity 0, singles out a GPU with failed NVLinks.

all_reduce_perf -b 8 -e 8G -f 2 -g 8Single-node NCCL all-reduce over NVLink

# nThread 1 nGpus 8 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid  48211 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid  48211 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
...
#  Rank  7 Group  0 Pid  48211 on      dgx01 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    41.20    0.00    0.00      0    40.11    0.00    0.00      0
          16             4     float     sum      -1    40.87    0.00    0.00      0    40.52    0.00    0.00      0
...
   536870912     134217728     float     sum      -1   2183.4  245.88  430.29      0   2179.9  246.28  430.99      0
  1073741824     268435456     float     sum      -1   4190.6  256.22  448.39      0   4186.2  256.50  448.87      0
  2147483648     536870912     float     sum      -1   8291.7  258.99  453.24      0   8287.0  259.14  453.49      0
  4294967296    1073741824     float     sum      -1    16502  260.27  455.47      0    16494  260.40  455.70      0
  8589934592    2147483648     float     sum      -1    32950  260.70  456.22      0    32911  261.01  456.77      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 148.912
#

What to look at: Read the busbw column for the largest sizes and check #wrong is 0. A large-message busbw far below the expected NVLink figure (for example about 60% of it) points to a degraded fabric, not an NCCL tuning problem.

fmpm -lList Shared NVSwitch partitions

{
   "numPartitions" : 15,
   "partitionInfo" : [
      {
         "gpuInfo" : [
            { "numEnabledLinks" : 18, "pciBusId" : "", "physicalId" : 1, "uuid" : "" },
            { "numEnabledLinks" : 18, "pciBusId" : "", "physicalId" : 2, "uuid" : "" },
            ...
         ],
         "isActive" : 0,
         "numGpus" : 8,
         "partitionId" : 1
      },
      {
         "gpuInfo" : [ ... 4 GPUs, physicalId 1-4 ... ],
         "isActive" : 1,
         "numGpus" : 4,
         "partitionId" : 2
      },
      ...
   ],
   "version" : 1
}

What to look at: This only works with FABRIC_MODE=1 (Shared NVSwitch). Check numGpus and isActive per partitionId; on H100 the uuid and pciBusId fields are empty, so map GPUs to VMs by physicalId.

dmesg -T | grep -iE "sxid|xid"NVSwitch SXid and GPU Xid errors

[Tue Nov 10 09:14:22 2026] NVRM: Xid (PCI:0000:9d:00): 74, pid='<unknown>', name=<unknown>, NVLink: fatal error detected on link 5(0x0, 0x0, 0x10000, 0x0, 0x0, 0x0, 0x0)
[Tue Nov 10 09:14:22 2026] nvidia-nvswitch2: SXid (PCI:0000:07:00.0): 20034, Fatal, Link 32 LTSSM Fault Up
[Tue Nov 10 09:14:22 2026] nvidia-nvswitch2: SXid (PCI:0000:07:00.0): 20034, Severity 1 Engine instance 32 Sub-engine instance 00
[Tue Nov 10 09:14:22 2026] nvidia-nvswitch2: SXid (PCI:0000:07:00.0): 20034, Data {0x50610002, 0x10100030, 0x00000000, 0x10100030, 0x00000000, 0x00000000, 0x00000000, 0x00000000}

What to look at: No output is the healthy result. Xid 74 is an NVLink error seen by a GPU and SXid lines come from an NVSwitch; the word Fatal means drain, reset or reboot, collect nvidia-bug-report.sh, and escalate if it repeats.

dcgmi discovery -lGPUs and NVSwitches DCGM can see

8 GPUs found.
+--------+----------------------------------------------------------------------+
| GPU ID | Device Information                                                   |
+--------+----------------------------------------------------------------------+
| 0      | Name: NVIDIA H100 80GB HBM3                                          |
|        | PCI Bus ID: 00000000:1B:00.0                                         |
|        | Device UUID: GPU-5f3a1c2e-0000-4a1b-9c3d-00000000d001                |
+--------+----------------------------------------------------------------------+
| 1      | Name: NVIDIA H100 80GB HBM3                                          |
|        | PCI Bus ID: 00000000:43:00.0                                         |
|        | Device UUID: GPU-5f3a1c2e-0001-4a1b-9c3d-00000000d001                |
+--------+----------------------------------------------------------------------+
...
4 NvSwitches found.
+-----------+
| Switch ID |
+-----------+
| 0         |
| 1         |
| 2         |
| 3         |
+-----------+

What to look at: A DGX H100 should report 8 GPUs and 4 NvSwitches. 0 NvSwitches usually means the NSCQ library (libnvidia-nscq) is missing or does not match the driver, so DCGM cannot monitor the switches.

dcgmi health -s a -cEnable all health watches and check

Health monitor systems set successfully.
Health Monitor Report
+----------------------------------------------------------------------------+
| Overall Health: Healthy                                                    |
+============================================================================+

What to look at: Healthy is the pass. Warning or Failure lists the GPU and subsystem underneath, for example NVLink with a count of CRC or replay errors, which is your cue to drain the node and run a deeper diag.

dcgmi diag -r 2Diagnostics including NVLink/P2P checks

Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 3.3.5                                          |
| Driver Version Detected   | 535.161.08                                     |
| GPU Device IDs Detected   | 2330,2330,2330,2330,2330,2330,2330,2330        |
|-----  Deployment  --------+------------------------------------------------|
| Denylist                  | Pass                                           |
| NVML Library              | Pass                                           |
| CUDA Main Library         | Pass                                           |
| Permissions and OS Blocks | Pass                                           |
| Persistence Mode          | Pass                                           |
| Environment Variables     | Pass                                           |
| Page Retirement/Row Remap | Pass                                           |
| Graphics Processes        | Pass                                           |
| Inforom                   | Pass                                           |
+-----  Integration  -------+------------------------------------------------+
| PCIe                      | Pass - All                                     |
+-----  Hardware  ----------+------------------------------------------------+
| GPU Memory                | Pass - All                                     |
+-----  Stress  ------------+------------------------------------------------+
+---------------------------+------------------------------------------------+

What to look at: Every row should read Pass. A Fail on PCIe names the GPU and often an NVLink or P2P bandwidth problem; the Driver Version Detected line is another quick check that the driver matches what FM expects.

17. Scenarios

18. Searchable command reference

Command / item	Purpose

19. Flashcards (click to flip)

20. Exam traps

CUDA error 802 / "system not yet initialized" on DGX/HGX → Fabric Manager not running, not finished, or version ≠ driver.
nvidia-smi shows all 8 GPUs doesn't prove FM is fine. NVML works without the fabric.
FM version must exactly match the driver. Upgrade both together. Hold the packages.
FM is only for NVSwitch systems (8-GPU HGX/DGX). Not needed for PCIe GPUs or 4-GPU direct-NVLink boards.
FM ≠ Subnet Manager. FM = NVLink inside the node. SM/UFM = InfiniBand between nodes.
FM is control plane. It programs routes and monitors. GPU data doesn't flow through the FM process.
FM runs on the host, not in a container. In shared NVSwitch mode it runs in a service VM / the hypervisor.
FABRIC_MODE: 0 = bare metal (DGX default), 1 = shared NVSwitch multi-tenancy, 2 = vGPU.
SXid = NVSwitch error. Xid 74 = NVLink error. Both point to hardware, so drain and investigate.
BCM cluster: install/enable/upgrade FM in the software image, not by SSH on nodes.
Low single-node NCCL busbw plus NVLink errors / fewer than NV18 in topo -m → degraded fabric (hardware), not NCCL tuning.
FM partitions (whole-GPU NVLink groups for VMs) ≠ MIG (slices of one GPU).

21. Practice quiz (25 questions)

Score: 0 / 0