GPU Cluster Error Codes

One place for the error codes and messages an AI-infrastructure admin meets: GPU Xid errors, ECC and row remapping, NVSwitch SXid, NVML / nvidia-smi messages, CUDA runtime errors, NCCL errors, throttle reasons, DCGM results, PCIe/BMC events, and the action ladder that tells you whether to restart, reset, reboot, drain, or RMA.

1. The big picture: where errors come from

Errors surface at every layer, and the layer that reports an error is often not the layer that caused it. A training job dies with a CUDA error; the real story is an Xid in the kernel log, which points at a GPU that fell off the PCIe bus because of a power fault recorded by the BMC.

Layer	Reports	Read with
Application / framework	CUDA errors, NCCL errors, watchdog timeouts	Job logs, `NCCL_DEBUG=INFO`
GPU driver (kernel)	Xid errors, NVSwitch SXid errors	`dmesg -T`, `journalctl -k`
GPU state (NVML)	ECC counts, row remap state, clock-event reasons, retired pages	`nvidia-smi -q`, DCGM
Fabric Manager	NVSwitch fabric state, partition and training errors	FM log, `systemctl status nvidia-fabricmanager`
Network	Port counters, BER, link flaps	`ibdiagnet`, `mlxlink`, UFM
Platform	PSU, fan, thermal, PCIe AER, CPU/DIMM machine checks	BMC SEL, Redfish logs, `dmesg` (AER, MCE, EDAC)

Always read the first error in time order, not the loudest. Xid 45 ("preemptive cleanup") and many CUDA and NCCL errors are consequences. The root cause is usually the earliest Xid, SXid, AER, or BMC event just before them.

A car's dashboard. The "check engine" light (CUDA error in the job) tells you something is wrong. The diagnostic code (Xid) tells you which system. The workshop's records (BMC event log, ECC history) tell you whether it's a one-off or a part that's wearing out.

2. The action ladder

Most exam questions about errors really ask: what's the right next step? Climb only as high as the evidence requires.

Level	Action	Typical triggers
1 App	Fix or restart the application; no hardware action	Isolated Xid 13 / 31 / 43, CUDA out-of-memory, illegal address in one app, no kernel image
2 Config	Fix software or configuration: start a service, match versions, change a mode	CUDA 802 (Fabric Manager), 803/804 (driver or compat mismatch), 46 (compute mode), NVML version mismatch
3 GPU reset	Reset the GPU when no work is on it	Xid 95, row remap pending, Xid 63/94 follow-up, a GPU stuck after Xid 61/62 or 119/120
4 Reboot	Reboot or power-cycle the node	Xid 79 (fallen off the bus), GPU won't reset, driver can't load, pending page retirement on older GPUs
5 Drain + diag	Drain the node, collect evidence, run DCGM diagnostics level 3	Xid 48, rising Xid 92, repeated Xid 74, fatal SXid, repeated 79, AER uncorrectable
6 RMA	Replace the part with evidence	Row remapping failure (Xid 64), a DCGM diagnostic that fails repeatedly, recurring 79 after reseat and power cycle, failed PSU or fan

App → Config → Reset → Reboot → Drain → RMA. "A Clever Robot Rarely Drops Racks."

Two classic wrong answers: jumping straight to RMA for an application error (Xid 13/31 in one job), and "just restarting the job" after a hardware error (Xid 48/79/95) without draining the node and running diagnostics.

3. Where to look

dmesg -T | grep -iE "xid|sxid|nvrm"            # GPU (Xid) and NVSwitch (SXid) driver errors, with timestamps
journalctl -k --since "1 hour ago" | grep -i xid
dmesg -T | grep -iE "aer|pcie bus error"       # PCIe Advanced Error Reporting
dmesg -T | grep -iE "mce|edac|machine check"   # CPU / DIMM errors
nvidia-smi -q -d ECC,ROW_REMAPPER,PAGE_RETIREMENT,PERFORMANCE   # memory health and throttle reasons
nvidia-smi --query-gpu=index,pci.bus_id,serial,uuid --format=csv   # map a PCI address in an Xid line to a GPU
dcgmi health -c                                # DCGM health watches (after dcgmi health -s a)
dcgmi diag -r 3                                # active diagnostics
systemctl status nvidia-fabricmanager ; sudo tail -n 100 /var/log/fabricmanager.log
ipmitool sel elist                             # BMC event log (PSU, fan, thermal)
nvsm show health ; nvsm show alerts            # DGX health summary

An Xid line looks like this (example):

NVRM: Xid (PCI:0000:1b:00): 79, pid=..., name=..., GPU has fallen off the bus.

The PCI address identifies the GPU; the number is the Xid; the text gives the short meaning. Map the PCI address to a GPU index and serial with the nvidia-smi --query-gpu line above. Don't assume nvidia-smi index 0 is the GPU in slot 0 (see GPU ordering & visibility).

4. Xid errors

An Xid is a GPU error report from the NVIDIA kernel driver. Some are about the application, some about the driver or firmware, and some about hardware. The meaning of each number is documented by NVIDIA; the most common ones are below.

Xid	Meaning	Usually	Action
13	Graphics engine exception	Application (out-of-bounds access, bad kernel)	1 App Debug the app; suspect hardware only if it repeats across different apps on the same GPU
31	GPU memory page fault (MMU fault)	Application illegal memory access; rarely driver	1 App Same as 13
32	Invalid or corrupted push buffer stream	Driver, PCIe errors, or hardware	5 Drain + diag if repeated; check AER
43	GPU stopped processing	The application hit a fault and was stopped; GPU usually fine	1 App
45	Preemptive cleanup due to previous errors	A consequence: an app was killed (Ctrl-C, OOM-kill) or an earlier error occurred	Look for the earlier error; alone, informational
48	Double-bit (uncorrectable) ECC error	Hardware memory error	5 Drain + diag; check row remapping; reset; RMA if it recurs or remap fails
61 / 62	Internal micro-controller breakpoint / halt	Driver or GPU firmware	3 GPU reset or reboot; update the driver if it recurs
63	ECC page retirement / row remapping recorded	Memory self-repair in progress	3 GPU reset at a convenient time to apply it
64	ECC page retirement / row remapping failure	No spare resources: memory can't be repaired	6 RMA
72	CE5: Unknown Error (copy engine 5). Neighbouring codes 75–77 are copy engines 6–8.	A copy (DMA) engine fault; not NVLink	1 App restart the application; contact support if it recurs
74	NVLink error	NVLink / NVSwitch hardware or fabric	5 Drain + diag; check SXid and `nvidia-smi nvlink -e`
79	GPU has fallen off the bus	PCIe link, power, thermal, or GPU hardware	4 Reboot (often a full power cycle); 5 drain and investigate; RMA if it recurs
92	High single-bit ECC error rate	Memory degrading	5 Drain + diag if rising; plan replacement
94	Contained ECC error	Uncorrectable error contained to the affected application(s); other work continues	Restart affected apps; 3 GPU reset when idle to remap
95	Uncontained ECC error	All work on the GPU is affected	3 GPU reset required; then investigate
119 / 120	GSP RPC timeout / GSP error	GPU System Processor firmware or driver	3 Reset / 4 reboot; collect a bug report; check driver version
154	GPU recovery action changed (newer drivers)	The driver states what recovery the GPU now needs (for example a GPU reset or node reboot)	Do the stated action; `nvidia-smi -q` shows the required recovery action

Three groups: 13 / 31 / 43 / 45 are usually software. 48 / 63 / 64 / 92 / 94 / 95 are memory (ECC). 74 / 79 and fatal SXids are interconnect or bus. Memorize those groups and the extreme cases: 64 = RMA, 79 = fallen off the bus, 95 = reset.

Details for specific codes (and newer codes added for new GPU generations) are in NVIDIA's Xid documentation. Always check the doc for your driver branch when you meet a code that isn't listed here.

5. ECC, row remapping, and page retirement

Term	Meaning
Correctable (single-bit, SBE)	ECC fixed it. Normal in small numbers. A rising rate (Xid 92) is a warning.
Uncorrectable (double-bit, DBE)	ECC couldn't fix it. Data was lost; the affected work is killed (Xid 48, 94, or 95).
Volatile vs aggregate	Volatile counts reset at driver reload or reboot; aggregate counts persist for the life of the GPU.
DRAM vs SRAM	DRAM = HBM memory; SRAM = on-chip caches and register files. SRAM uncorrectable errors can't be repaired by remapping and are taken more seriously.
Row remapping (A100, H100 and later)	A bad memory row is replaced by a spare row in the same bank. Recorded when the error happens (Xid 63), applied at the next GPU reset.
Page retirement (V100 and earlier)	The bad memory page is blacklisted. Applied at the next driver reload / reboot.

nvidia-smi -q -d ROW_REMAPPER
    Remapped Rows
        Correctable Error                 : 0
        Uncorrectable Error               : 1
        Pending                           : Yes      # a remap is waiting for a GPU reset
        Remapping Failure Occurred        : No       # Yes = no spare rows left: RMA
        Bank Remap Availability Histogram
            Max/High/Partial/Low/None     : ...      # how many spare rows remain per bank
nvidia-smi -q -d ECC                             # volatile and aggregate, SRAM and DRAM, correctable and uncorrectable

Row remap state	Meaning	Action
Pending: No, Failure: No, some remapped rows	Memory repaired itself successfully earlier	Healthy; monitor
Pending: Yes	A remap is recorded but not applied	3 GPU reset (drain first)
Remapping Failure Occurred: Yes	No spare row available in that bank	6 RMA
Histogram mostly Low/None	Spare rows nearly exhausted	Plan replacement

"Uncorrectable ECC error, and the remap is pending" does not mean the GPU is broken. Reset it (with no work on it) so the remap takes effect, then validate. Remap failure is the RMA signal.

6. NVSwitch SXid and Fabric Manager

SXid errors are the NVSwitch equivalent of Xid, written to the kernel log by the NVSwitch driver on HGX/DGX systems. Fatal SXids usually affect the GPUs whose traffic crosses that switch, and often appear alongside Xid 74 on those GPUs.

Signal	Meaning	Action
Non-fatal SXid	Corrected or recoverable switch error	Monitor; investigate if frequent
Fatal SXid	The switch or a port hit an unrecoverable error	5 Drain + diag; reset or reboot per the platform procedure; service if it recurs
CUDA 802 "system not yet initialized"	Fabric Manager isn't running or hasn't finished setting up the NVSwitch fabric	2 Config start FM; versions must match the driver exactly
FM fails to start after a driver update	FM and driver versions differ	2 Config install the matching FM version

Full detail in the Fabric Manager guide and the NVLink guide.

Xid = GPU. SXid = Switch. Xid 74 = the link between them.

7. nvidia-smi / NVML messages

Message	Meaning	Fix
"NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver"	The kernel module isn't loaded	2 Config Kernel updated without a module rebuild, Secure Boot rejecting an unsigned module, or the driver isn't installed. Check `lsmod \| grep nvidia`, `dmesg`.
"Failed to initialize NVML: Driver/library version mismatch"	User-space libraries were updated but the old kernel module is still loaded	4 Reboot (or unload and reload the modules with no GPU users)
"No devices were found"	The driver is loaded but no GPU is usable	Check `lspci \| grep -i nvidia`, `dmesg` for Xid 79 or init failures, power and seating
"Unable to determine the device handle for GPU …: Unknown Error"	A GPU that was there is no longer responding	Look for Xid 79; 4 Reboot / power cycle; investigate
`ERR!` in a field, or "GPU requires reset"	NVML can't read the value / the GPU needs a reset	Check `dmesg`; 3 GPU reset or reboot
Fewer GPUs than expected	A GPU didn't initialize or fell off the bus	`lspci` count vs `nvidia-smi -L` count; check Xids and BMC logs

lspci sees the GPU but nvidia-smi doesn't → driver or GPU-initialization problem. lspci doesn't see it either → PCIe, power, seating, or hardware (and the BMC log).

8. CUDA runtime errors

CUDA errors reach the admin as "my job crashed with …". The number and text map to infrastructure causes. See also the CUDA for infrastructure guide.

Code	Text	Infrastructure cause	Level
2	out of memory	Batch too large, another process holding memory, fragmentation	1 App
3	initialization error	Driver or device not usable by this process (permissions, broken state)	2 Config
35	CUDA driver version is insufficient for CUDA runtime version	Container or app CUDA newer than the driver	2 Config upgrade driver or use forward compatibility
46	all CUDA-capable devices are busy or unavailable	Compute mode EXCLUSIVE_PROCESS already in use, or PROHIBITED; or MIG enabled with no instances	2 Config
100	no CUDA-capable device is detected	Empty `CUDA_VISIBLE_DEVICES`, container without GPUs, driver not loaded	2 Config
209	no kernel image is available for execution on the device	Binary not built for this GPU architecture	1 App
214	uncorrectable ECC error encountered	GPU memory error (see Xid 48/94/95)	5 Drain + diag
220	uncorrectable NVLink error detected	NVLink fault (see Xid 74)	5 Drain + diag
700	an illegal memory access was encountered	Usually the app (with Xid 13/31); hardware only if widespread on one GPU	1 App
719	unspecified launch failure	Kernel crashed; app bug or GPU fault; check Xids	1 / 5
802	system not yet initialized	Fabric Manager not running / not ready on an NVSwitch system	2 Config
803	system has unsupported display driver / cuda driver combination	User-mode driver libraries (for example compat libs in a container) don't match the kernel driver	2 Config
804	forward compatibility was attempted on non supported HW	Compat package used on a GPU that doesn't support it	2 Config
999	unknown error	Often driver state (missing `nvidia-uvm`), or a GPU in a bad state	3 / 4

9. NCCL errors and timeouts

Error	Points to	Next step
`ncclUnhandledCudaError`	A CUDA call failed underneath NCCL	Find the CUDA error and Xid on the failing rank's node
`ncclSystemError`	System call or OS resource problem: sockets, shared memory (`/dev/shm` too small in containers), memlock limits, IB device access	Check container shm / ulimits / device access; `NCCL_DEBUG=INFO`
`ncclInternalError`	NCCL internal failure, often a version or plugin problem	Check NCCL / plugin versions; collect logs
`ncclInvalidArgument` / `ncclInvalidUsage`	Application or framework misuse	Application issue
`ncclRemoteError`	A peer failed or disconnected	Look at the other rank's node: crash, Xid, link down
"NET/IB : Got completion with error 12" (retry exceeded)	RDMA transport retries exhausted: link down, flapping, or a dead peer	`ibstat`, `mlxlink`, port counters, UFM; then the peer node
Framework watchdog: collective operation timeout	A collective didn't finish in time: a hung or crashed rank, a straggler, or a network stall	Find the rank that didn't arrive (its logs, Xids, link)
Logs show `NET/Socket` instead of `NET/IB`	Not an error message, but RDMA isn't in use	IB devices / plugin / container access (see the NCCL guide)

An NCCL error on every rank usually has one root cause on one node. Find the first rank that failed (earliest timestamp, or the one that reports a local error rather than a remote one) and go to that node's kernel log.

10. Clock-event (throttle) reasons

nvidia-smi -q -d PERFORMANCE lists why clocks are reduced. They're not errors, but they explain "slow node, no errors".

Reason	Meaning	Look at
GPU idle	Nothing running	Normal
SW power cap	At the configured power limit	Normal under heavy load; check the limit wasn't lowered
SW thermal slowdown	Driver reducing clocks for temperature	Cooling, airflow, inlet temperature
HW slowdown	Hardware-triggered clock reduction	Power or thermal event: check the next two and the BMC
HW thermal slowdown	Hardware thermal protection	Cooling failure, fan, blocked airflow, liquid-cooling issue
HW power brake slowdown	An external power-brake signal from the system	PSU failure or lost power redundancy: BMC SEL, PSU status

HW power brake on all GPUs of one node at once usually means the node's power (a PSU failed or redundancy was lost), not eight bad GPUs. Check the BMC.

11. DCGM health and diagnostics results

Tool	Result	Meaning
`dcgmi health -c`	Healthy / Warning / Failure per system (PCIe, memory, InfoROM, thermal, power, NVLink)	Passive watches since `dcgmi health -s` was set: what has gone wrong recently
`dcgmi diag -r 1/2/3/4`	Pass / Fail / Warn / Skip per test	Active tests: software, PCIe bandwidth, memory, targeted stress and power, NVLink; higher levels run longer and deeper
Memory test failure	—	GPU memory fault: check ECC and row remapping; RMA evidence
PCIe test failure / low bandwidth	—	Link width or speed reduced, riser or seating, AER errors
Targeted power / stress failure	—	Can't hold power or clocks: cooling or power delivery, or a weak GPU
Software / deployment test failure	—	Driver, permissions, or library issues, not hardware

DCGM details: DCGM deep dive. Stress tools: gpu-burn & dcgmproftester.

12. PCIe, BMC, and platform events

Event	Where	Meaning	Action
PCIe AER corrected errors	`dmesg`	Link noise that was corrected	Monitor; many of them on one device → seating, riser, link speed
PCIe AER uncorrected (non-fatal / fatal)	`dmesg`	Data lost on the PCIe link	5 Drain + diag; often precedes Xid 79
PSU failure / redundancy lost	BMC SEL, Redfish	A power supply failed or lost input	Check input power; replace PSU (often hot-swappable)
Fan failure / low RPM	BMC SEL, sensors	Cooling degraded	Replace fan; check thermals
Thermal trip / over-temperature	BMC SEL, `nvidia-smi`	Component exceeded its limit	Cooling and airflow; data-center inlet temperature
Machine check (MCE) / EDAC	`dmesg`, SEL	CPU or DIMM errors	Identify DIMM; replace if uncorrectable or a storm of correctable

See the Redfish, IPMI & BMC guide and NVSM guide.

13. Resetting a GPU safely

nvidia-smi --query-compute-apps=pid,process_name,gpu_bus_id --format=csv   # anything still using the GPU?
sudo systemctl stop dcgm nvidia-dcgm 2>/dev/null     # monitoring agents hold GPU handles (service name varies)
sudo nvidia-smi -r -i 3                                 # reset GPU 3 (no processes may be using it)
nvidia-smi -q -i 3 -d ROW_REMAPPER                     # Pending should now be No

Drain the node (or at least the GPU) first; a reset kills anything on it.
Monitoring and persistence agents (DCGM, exporters) may need to stop for the reset to proceed.
On NVSwitch systems, follow the platform's reset procedure; Fabric Manager must be running and may need to re-establish the GPU's fabric state.
If the reset fails, or the GPU has fallen off the bus (Xid 79), reboot or power-cycle the node instead.
Validate after any reset: dcgmi diag -r 2 or higher, and a short NCCL test.

14. Collecting evidence and RMA

Before rebooting or resetting, capture state: nvidia-bug-report.sh (creates nvidia-bug-report.log.gz), dmesg -T, nvidia-smi -q.
On DGX: nvsm dump health.
DCGM: dcgmi diag -r 3 -j (JSON) on the suspect GPU.
BMC: ipmitool sel elist or the Redfish log entries.
Identify the part: GPU serial and PCI address, PSU slot, and the system serial.
Drain, replace following the service guide, bring the replacement to the firmware baseline, re-validate, return to service. Full procedure: replacement procedure in the bring-up guide.

Rebooting first destroys evidence: volatile ECC counts reset, and the kernel log of the failure is gone unless it was persisted. Collect, then act.

15. Interactive lookup

Type a code or words from a message (for example 79, 802, remap, mismatch, power brake, error 12).

Code / message	Source	Meaning	Action

16. Triage drill

You get a log line. Pick the right next step on the action ladder.

17. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

dmesg -T | grep -iE "xid|sxid|nvrm"GPU and NVSwitch driver errors

[Tue Nov 10 08:10:31 2026] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  550.54.15  Tue Mar  5 22:23:56 UTC 2024
[Tue Nov 10 18:42:07 2026] NVRM: Xid (PCI:0000:5d:00): 48, pid='<unknown>', name=<unknown>, An uncorrectable double bit error (DBE) has been detected on GPU in the framebuffer at partition 6, subpartition 0.
[Tue Nov 10 18:42:07 2026] NVRM: Xid (PCI:0000:5d:00): 63, pid='<unknown>', name=<unknown>, Row Remapper: New row (0x0000000000a1b2c3) marked for remapping, reset gpu to activate.
[Tue Nov 10 18:42:08 2026] NVRM: Xid (PCI:0000:5d:00): 45, pid=51237, name=python3, Ch 00000008

What to look at: Read the earliest Xid first: Xid 48 (uncorrectable ECC) is the root cause, Xid 63 says a row remap is waiting for a GPU reset, and Xid 45 is only the cleanup of the killed job. The PCI address 0000:5d:00 tells you which GPU it is; an Xid 79 or 64, or any fatal SXid line, would call for a reboot or RMA path instead.

journalctl -k --since "1 hour ago" | grep -i xidRecent Xids from the kernel journal

Nov 10 18:42:07 dgx05 kernel: NVRM: Xid (PCI:0000:5d:00): 48, pid='<unknown>', name=<unknown>, An uncorrectable double bit error (DBE) has been detected on GPU in the framebuffer at partition 6, subpartition 0.
Nov 10 18:42:07 dgx05 kernel: NVRM: Xid (PCI:0000:5d:00): 63, pid='<unknown>', name=<unknown>, Row Remapper: New row (0x0000000000a1b2c3) marked for remapping, reset gpu to activate.
Nov 10 18:42:08 dgx05 kernel: NVRM: Xid (PCI:0000:5d:00): 45, pid=51237, name=python3, Ch 00000008

What to look at: These are the same events as dmesg, with host name and journal timestamps, filtered to a time window. The journal can keep kernel messages across reboots if it is persistent, which is why it is useful evidence; empty output for a window when a job died points you to other layers such as NCCL, the network, or the BMC.

dmesg -T | grep -iE "aer|pcie bus error"PCIe Advanced Error Reporting events

[Tue Nov 10 17:58:31 2026] pcieport 0000:59:01.0: AER: Corrected error message received from 0000:5d:00.0
[Tue Nov 10 17:58:31 2026] nvidia 0000:5d:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[Tue Nov 10 17:58:31 2026] nvidia 0000:5d:00.0:   device [10de:2330] error status/mask=00000001/0000e000
[Tue Nov 10 17:58:31 2026] nvidia 0000:5d:00.0:    [ 0] RxErr                  (First)

What to look at: A single corrected Physical Layer error is link noise that was fixed and is not the cause of an ECC Xid. Watch for severity=Uncorrected (Non-Fatal or Fatal) or a burst of corrected errors on one device, which often comes just before an Xid 79.

dmesg -T | grep -iE "mce|edac|machine check"CPU and DIMM hardware errors

[Tue Nov 10 08:10:12 2026] EDAC MC0: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#0: DEV 0000:7e:0c.0 (INTERRUPT)
[Tue Nov 10 08:10:12 2026] EDAC MC1: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#1: DEV 0000:7e:0d.0 (INTERRUPT)
[Tue Nov 10 08:10:12 2026] EDAC i10nm: v0.0.6

What to look at: Only the boot-time EDAC driver registration lines appear, so the CPUs and DIMMs have reported no errors. Lines such as "mce: [Hardware Error]" or "EDAC MC0: 1 UE memory read error on ... DIMM" would point at host memory or a CPU rather than the GPU.

nvidia-smi -q -d ECC,ROW_REMAPPER,PAGE_RETIREMENT,PERFORMANCEMemory health and throttle reasons

Timestamp                                 : Tue Nov 10 18:50:03 2026
Driver Version                            : 550.54.15
CUDA Version                              : 12.4

Attached GPUs                             : 8
...
GPU 00000000:5D:00.0
    ECC Errors
        Volatile
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 0
            DRAM Uncorrectable            : 1
        Aggregate
            ...
            DRAM Correctable              : 3
            DRAM Uncorrectable            : 1
    Retired Pages
        Single Bit ECC                    : N/A
        Double Bit ECC                    : N/A
        Pending Page Blacklist            : N/A
    Remapped Rows
        Correctable Error                 : 0
        Uncorrectable Error               : 1
        Pending                           : Yes
        Remapping Failure Occurred        : No
        Bank Remap Availability Histogram
            Max                           : 2559 bank(s)
            High                          : 0 bank(s)
            Partial                       : 1 bank(s)
            Low                           : 0 bank(s)
            None                          : 0 bank(s)
    Performance State                     : P0
    Clocks Event Reasons
        Idle                              : Active
        SW Power Cap                      : Not Active
        ...
(other GPUs trimmed; GPU 3 shown)

What to look at: GPU 3 has one volatile DRAM uncorrectable error and Pending: Yes with Remapping Failure Occurred: No, so the fix is to drain and reset the GPU, not RMA. Retired Pages shows N/A because H100 uses row remapping instead of page retirement; Remapping Failure Occurred: Yes or a histogram drifting to Low/None would be the RMA signal.

nvidia-smi --query-gpu=index,pci.bus_id,serial,uuid --format=csvMap Xid PCI address to GPU

index, pci.bus_id, serial, uuid
0, 00000000:18:00.0, 1654000000100, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000000
1, 00000000:2A:00.0, 1654000000101, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000001
2, 00000000:3A:00.0, 1654000000102, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000002
3, 00000000:5D:00.0, 1654000000103, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000003
4, 00000000:9A:00.0, 1654000000104, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000004
5, 00000000:AB:00.0, 1654000000105, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000005
6, 00000000:BA:00.0, 1654000000106, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000006
7, 00000000:DB:00.0, 1654000000107, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000007

What to look at: The Xid said PCI:0000:5d:00, which is the 00000000:5D:00.0 row, so the failing part is GPU index 3 and that row's serial goes on the RMA or support case. Never assume index equals slot; always match by bus ID.

dcgmi health -cDCGM passive health check

Health Monitor Report
+----------------------------------------------------------------------------+
| Group 0          | Overall Health: Failure                                 |
+==================+=========================================================+
| GPU ID: 3        | Failure                                                 |
|                  | Memory system: Failure - Detected 1 volatile            |
|                  | double-bit ECC error(s) in GPU 3.                       |
+------------------+---------------------------------------------------------+

What to look at: Only GPU 3 is listed, which means the other seven GPUs passed every watched subsystem; GPU 3's memory system is in Failure because of the volatile double-bit error. Overall Health: Healthy is the good result, and this check only works after health watches were enabled with dcgmi health -s a.

dcgmi diag -r 3Active level-3 GPU diagnostics

Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 3.3.5                                          |
| Driver Version Detected   | 550.54.15                                      |
| GPU Device IDs Detected   | 2330,2330,2330,2330,2330,2330,2330,2330        |
|-----  Deployment  --------+------------------------------------------------|
| Denylist                  | Pass                                           |
| NVML Library              | Pass                                           |
| CUDA Main Library         | Pass                                           |
| Permissions and OS Blocks | Pass                                           |
| Persistence Mode          | Pass                                           |
| Environment Variables     | Pass                                           |
| Page Retirement/Row Remap | Fail - GPU: 3                                  |
|    Warning                | GPU 3 had uncorrectable memory errors and row  |
|                           | remapping is pending. Reset the GPU.           |
| Graphics Processes        | Pass                                           |
| Inforom                   | Pass                                           |
+-----  Integration  -------+------------------------------------------------+
| PCIe                      | Skip - All                                     |
+-----  Hardware  ----------+------------------------------------------------+
| GPU Memory                | Skip - All                                     |
| Diagnostic                | Skip - All                                     |
+-----  Stress  ------------+------------------------------------------------+
| Targeted Stress           | Skip - All                                     |
| Targeted Power            | Skip - All                                     |
| Memory Bandwidth          | Skip - All                                     |
+---------------------------+------------------------------------------------+

What to look at: A failed deployment check stops the run, so the remaining tests show Skip; the row-remap failure on GPU 3 must be fixed with a GPU reset before the hardware and stress tests can tell you anything. After the reset, a clean run is Pass on every line; a GPU Memory or Targeted Power Fail on the same GPU after a reset is RMA evidence.

systemctl status nvidia-fabricmanager ; sudo tail -n 100 /var/log/fabricmanager.logFabric Manager service state and log

● nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-11-10 08:10:44 UTC; 10h ago
    Process: 2311 ExecStart=/usr/bin/nv-fabricmanager-start.sh (code=exited, status=0/SUCCESS)
   Main PID: 2334 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 21.4M
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2334 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

Nov 10 08:10:40 dgx05 systemd[1]: Starting NVIDIA fabric manager service...
Nov 10 08:10:44 dgx05 systemd[1]: Started NVIDIA fabric manager service.
(output trimmed)
[Nov 10 2026 08:10:41] [INFO] [tid 2334] Fabric Manager version 550.54.15 is running with the following configuration options
[Nov 10 2026 08:10:43] [INFO] [tid 2334] Connected to 1 node.
[Nov 10 2026 08:10:44] [INFO] [tid 2334] Successfully configured all the available NVSwitches to route GPU NVLink traffic. NVLink Peer-to-Peer support will be enabled once the GPUs are successfully registered with the NVLink fabric.

What to look at: Active: active (running) and a Fabric Manager version identical to the driver (550.54.15) is the healthy state, so this incident is not a fabric problem. failed or inactive, or a version mismatch error in the log, explains CUDA error 802 (system not yet initialized) on every GPU.

ipmitool sel elistBMC system event log

   1 | 11/10/2026 | 08:09:58 | Event Logging Disabled SEL_Status | Log area reset/cleared | Asserted
   2 | 11/10/2026 | 08:10:05 | System Event #0x6f | Timestamp Clock Sync | Asserted
   3 | 11/10/2026 | 14:22:17 | Power Supply PSU2_Status | Power Supply AC lost | Asserted
   4 | 11/10/2026 | 14:22:21 | Power Supply PSU2_Status | Power Supply AC lost | Deasserted

What to look at: The only platform event is a four-second AC loss on PSU2 hours before the Xid, which recovered (Deasserted), so power and cooling do not explain the ECC error. A Failure detected or Redundancy Lost entry that stays Asserted, fan Lower Critical, or temperature Upper Critical events would make the platform the suspect.

nvsm show healthDGX health summary

Info
----
Timestamp                     : Tue Nov 10 18:55:31 2026 +0000
Version                       : 24.03.07

Checks
------
Verify installed DIMM memory sticks.......................... Healthy
Number of logical CPU cores [224]............................ Healthy
Number of GPUs [8]........................................... Healthy
GPU link speed [0000:5d:00.0][32GT/s]........................ Healthy
GPU link width [0000:5d:00.0][x16]........................... Healthy
...
Verify GPU row remapping [0000:5d:00.0]...................... Unhealthy
...
Health Summary
--------------
212 out of 213 checks are healthy
1 out of 213 checks are unhealthy
0 out of 213 checks are unknown
0 out of 213 checks are informational

Overall system status is unhealthy

What to look at: Read the Health Summary first, then search the Checks list for Unhealthy lines; here a single GPU row-remapping check on bus 5d fails while GPU count, link speed and width are fine. Wrong counts such as Number of GPUs [7] or a downgraded link such as [16GT/s] or [x8] point to a missing GPU or a PCIe problem.

lsmod | grep nvidiaCheck NVIDIA kernel modules are loaded

nvidia_uvm           4911104  0
nvidia_peermem         16384  0
nvidia_drm            122880  0
nvidia_modeset       1355776  1 nvidia_drm
nvidia              54333440  85 nvidia_uvm,nvidia_peermem,nvidia_modeset
ib_core               ...  (output trimmed)

What to look at: The nvidia, nvidia_uvm, nvidia_modeset and nvidia_peermem modules are loaded, so the kernel side of the driver is present. Empty output means the module is not loaded, which is the cause of "couldn't communicate with the NVIDIA driver" (check dmesg for Secure Boot signature rejections); a missing nvidia_uvm can cause CUDA error 999.

lspci | grep -i nvidiaCount NVIDIA devices on the PCIe bus

05:00.0 Bridge: NVIDIA Corporation GH100 [H100 NVSwitch] (rev a1)
06:00.0 Bridge: NVIDIA Corporation GH100 [H100 NVSwitch] (rev a1)
07:00.0 Bridge: NVIDIA Corporation GH100 [H100 NVSwitch] (rev a1)
08:00.0 Bridge: NVIDIA Corporation GH100 [H100 NVSwitch] (rev a1)
18:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
2a:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
3a:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
5d:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
9a:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
ab:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
ba:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
db:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)

What to look at: A DGX H100 should show 4 NVSwitch bridges and 8 H100 3D controllers. If lspci shows all 8 GPUs but nvidia-smi lists fewer, it is a driver or initialization problem; if lspci also shows fewer, suspect PCIe, power or seating and check the BMC log.

nvidia-smiRun nvidia-smi after a package update

Failed to initialize NVML: Driver/library version mismatch
NVML library version: 550.90.07

What to look at: This means the user-space libraries were upgraded to 550.90.07 while the old kernel module is still loaded. The fix is to drain and reboot (or reload the modules with no GPU users), not to reinstall CUDA or RMA anything.

cat /proc/driver/nvidia/versionShow the loaded kernel module version

NVRM version: NVIDIA UNIX x86_64 Kernel Module  550.54.15  Tue Mar  5 22:23:56 UTC 2024
GCC version:  gcc version 12.3.0 (Ubuntu 12.3.0-1ubuntu1~22.04)

What to look at: The loaded module is 550.54.15, but NVML reported library version 550.90.07, which confirms the version mismatch diagnosis. If this file does not exist at all, the nvidia module is not loaded.

nvidia-smi --query-compute-apps=pid,process_name,gpu_bus_id --format=csvFind processes still holding the GPU

pid, process_name, gpu_bus_id
51290, /usr/bin/python3, 00000000:5D:00.0

What to look at: A leftover python3 process is still attached to bus 5D (GPU 3), so a reset would fail until it is stopped. Header-only output means no compute process holds any GPU, but monitoring agents such as DCGM can still hold handles without appearing here.

sudo nvidia-smi -r -i 3Reset GPU 3 to apply the remap

GPU 00000000:5D:00.0 was successfully reset.
All done.

What to look at: "successfully reset" means the pending row remap could now be applied. "Unable to reset ... being used by some other process" means a job or agent still holds the GPU; if the reset keeps failing or the GPU has fallen off the bus (Xid 79), reboot or power-cycle the node instead.

nvidia-smi -q -i 3 -d ROW_REMAPPERConfirm the remap was applied

==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 19:01:44 2026
Driver Version                            : 550.54.15
CUDA Version                              : 12.4

Attached GPUs                             : 8
GPU 00000000:5D:00.0
    Remapped Rows
        Correctable Error                 : 0
        Uncorrectable Error               : 1
        Pending                           : No
        Remapping Failure Occurred        : No
        Bank Remap Availability Histogram
            Max                           : 2559 bank(s)
            High                          : 0 bank(s)
            Partial                       : 1 bank(s)
            Low                           : 0 bank(s)
            None                          : 0 bank(s)

What to look at: Pending is now No and there is no remapping failure, so the bad row was swapped for a spare; one bank moved from Max to Partial because it used a spare row. Validate with dcgmi diag -r 2 or higher before returning the node to service, and treat Remapping Failure Occurred: Yes as an RMA case.

sudo nvidia-bug-report.shCollect a full driver bug report


nvidia-bug-report.sh will now collect information about your
system and create the file 'nvidia-bug-report.log.gz' in the current
directory.  It may take several seconds to run.  In some
cases, it may hang trying to capture data generated dynamically
by the Linux kernel and/or the NVIDIA kernel module.  While
the bug report log file will be incomplete if this happens, it
may still contain enough data to diagnose your problem.

If nvidia-bug-report.sh hangs, consider running with the --safe-mode
and --extra-system-data command line arguments.

Please include the 'nvidia-bug-report.log.gz' log file when reporting
your bug via the NVIDIA Linux forum (see forums.developer.nvidia.com)
or by sending email to 'linux-bugs@nvidia.com'.

(output trimmed)

Running nvidia-bug-report.sh... complete.

What to look at: "complete" means nvidia-bug-report.log.gz was written to the current directory; attach it to the support case together with the GPU serial. Run it before rebooting, because a reboot clears volatile ECC counts and the kernel log of the failure; if it hangs, rerun with --safe-mode.

sudo nvsm dump healthCollect a DGX health snapshot

Writing output to /tmp/nvsm-health-dgx05-20261110185612.tar.xz
Done.

What to look at: The tarball path is the evidence file NVIDIA Enterprise Support asks for on DGX systems, so keep it with the bug report. An error instead of Done usually means the nvsm services are not running (check systemctl status nvsm).

18. Scenarios

Try to name the cause and next step before opening each one.

19. Flashcards (click to flip)

20. Exam traps

Read the first error, not the last. Xid 45 and most NCCL errors are consequences.
Xid 13 / 31 / 43 alone → usually the application. Don't RMA.
The NVLink Xid is 74, not 72. Xid 72 is a copy-engine error (CE5). If a question asks what "Xid 72 NVLink problem" means, it has the number wrong.
Xid 48 (DBE) → hardware memory error: drain, check row remapping, diagnose.
Row remap Pending: Yes → reset the GPU. Remapping failure (Xid 64) → RMA.
Xid 94 contained (only affected apps) vs Xid 95 uncontained (GPU reset required).
Xid 79 = fallen off the bus: PCIe, power, thermal, hardware. Reboot/power cycle, investigate, RMA if repeated.
Xid 74 = NVLink; SXid = NVSwitch.
CUDA 802 = Fabric Manager not ready. 803/804 = driver / compat mismatch. 46 = compute mode or MIG. 100 = no visible device.
"Driver/library version mismatch" = libraries updated, old module loaded: reboot.
"Couldn't communicate with the NVIDIA driver" = module not loaded (kernel update, Secure Boot).
ncclRemoteError → look at the peer. NET/IB error 12 → link or peer gone.
HW power brake on every GPU of a node → node power / PSU.
Collect evidence before rebooting: nvidia-bug-report.sh, nvsm dump health, DCGM JSON, SEL.
Correctable ECC errors in small numbers are normal; a rising rate (Xid 92) is the warning.

21. Practice quiz (26 questions)

Score: 0 / 0