DCGM Deep Dive

Version : 4.2.3
Build ID : 7
Build Date : 2025-05-20
Build Type : Release
Commit ID : 0000aaaa1111bbbb2222cccc3333dddd4444eeee
Branch Name : rel_dcgm_4_2
CPU Arch : x86_64
Build Platform : Linux 4.15.0-180-generic #189-Ubuntu SMP Wed May 18 14:13:57 UTC 2022 x86_64
CRC : 00000000000000000000000000000000

Hostengine build info:
Version : 4.2.3
Build ID : 7
Build Date : 2025-05-20
Build Type : Release
Commit ID : 0000aaaa1111bbbb2222cccc3333dddd4444eeee
Branch Name : rel_dcgm_4_2
CPU Arch : x86_64
Build Platform : Linux 4.15.0-180-generic #189-Ubuntu SMP Wed May 18 14:13:57 UTC 2022 x86_64
CRC : 00000000000000000000000000000000

What to look at: The client version and the 'Hostengine build info' version should match. If the hostengine block is missing or shows an error, nv-hostengine (the nvidia-dcgm service) isn't running or can't be reached on port 5555.

8 GPUs found.
+--------+----------------------------------------------------------------------+
| GPU ID | Device Information                                                   |
+--------+----------------------------------------------------------------------+
| 0      | Current Name: NVIDIA H100 80GB HBM3                                  |
|        | PCI Bus ID: 00000000:18:00.0                                         |
|        | Device UUID: GPU-7c1e0000-0000-4000-8000-000000000000                |
+--------+----------------------------------------------------------------------+
| 1      | Current Name: NVIDIA H100 80GB HBM3                                  |
|        | PCI Bus ID: 00000000:2A:00.0                                         |
|        | Device UUID: GPU-7c1e0000-0000-4000-8000-000000000001                |
+--------+----------------------------------------------------------------------+
...  (GPUs 2-6 trimmed)
| 7      | Current Name: NVIDIA H100 80GB HBM3                                  |
|        | PCI Bus ID: 00000000:DB:00.0                                         |
|        | Device UUID: GPU-7c1e0000-0000-4000-8000-000000000007                |
+--------+----------------------------------------------------------------------+
4 NvSwitches found.
+-----------+
| Switch ID |
+-----------+
| 0         |
| 1         |
| 2         |
| 3         |
+-----------+

What to look at: A DGX H100 should report 8 GPUs and 4 NvSwitches. Fewer GPUs than installed (for example '7 GPUs found') means a GPU fell off the bus or the driver didn't load it, so check dmesg for Xid 79 before anything else.

+-------------------+------------------------------------------------------------------------------+
| GPU ID: 0         | Topology Information                                                         |
+===================+==============================================================================+
| CPU Core Affinity | 0 - 55, 112 - 167                                                            |
+-------------------+------------------------------------------------------------------------------+
| To GPU 1          | Connected via eighteen NVLinks                                               |
| To GPU 2          | Connected via eighteen NVLinks                                               |
| To GPU 3          | Connected via eighteen NVLinks                                               |
| To GPU 4          | Connected via eighteen NVLinks                                               |
| To GPU 5          | Connected via eighteen NVLinks                                               |
| To GPU 6          | Connected via eighteen NVLinks                                               |
| To GPU 7          | Connected via eighteen NVLinks                                               |
+-------------------+------------------------------------------------------------------------------+

What to look at: On a DGX H100 every GPU pair is connected through NVSwitch with eighteen NVLinks. A pair shown only as connected via PCIe or a CPU-level link means NVLink or Fabric Manager is down, and GPU-to-GPU traffic will be far slower.

+-------------------+----------------------------------------------------------+
| GROUPS                                                                       |
| 2 groups found.                                                              |
+===================+==========================================================+
| Groups            |                                                          |
| -> 0              |                                                          |
|    -> Group ID    | 0                                                        |
|    -> Group Name  | DCGM_ALL_SUPPORTED_GPUS                                  |
|    -> Entities    | GPU 0, GPU 1, GPU 2, GPU 3, GPU 4, GPU 5, GPU 6, GPU 7   |
| -> 1              |                                                          |
|    -> Group ID    | 1                                                        |
|    -> Group Name  | DCGM_ALL_SUPPORTED_NVSWITCHES                            |
|    -> Entities    | SWITCH 0, SWITCH 1, SWITCH 2, SWITCH 3                   |
+-------------------+----------------------------------------------------------+

What to look at: Groups 0 and 1 are built in, so the first group you create gets ID 2. If a later command fails with 'Cannot find group', list the groups and use the ID shown here.

Successfully created group "allgpus" with a group ID of 2

What to look at: Note the printed ID: every later -g option uses it. Without --default the new group is empty, and you add GPUs with 'dcgmi group -g <id> -a 0,1,2,3'.

___________________________________________________________________________________
                                   Long Name          Short Name          Field ID
___________________________________________________________________________________
driver_version                                             DRVER                 1
...
gpu_temp                                                   TMPTR               150
power_usage                                                POWER               155
gpu_utilization                                            GPUTL               203
fb_used                                                    FBUSD               252
xid_errors                                                 XIDER               230
...
gr_engine_active                                           GRACT              1001
sm_active                                                  SMACT              1002
sm_occupancy                                               SMOCC              1003
tensor_active                                              TENSO              1004
dram_active                                                DRAMA              1005
pcie_tx_bytes                                              PCITX              1009
pcie_rx_bytes                                              PCIRX              1010
nvlink_tx_bytes                                            NVLTX              1011
nvlink_rx_bytes                                            NVLRX              1012
(output trimmed)

What to look at: Use this to find the numeric field ID for dmon -e and to match it with the short column name you'll see in dmon output. The 1001-1012 profiling fields are the ones that show real efficiency; 203 (GPUTL) only shows that a kernel was running.

#Entity   GPUTL  FBUSD     TMPTR  POWER
ID        %      MB        C      W
GPU 0     98     71680     58     412.6
GPU 1     98     71680     57     409.3
GPU 2     99     71680     59     415.0
GPU 3     98     71680     58     411.8
GPU 4     0      0         33     71.2
GPU 5     0      0         32     70.8
GPU 6     0      0         33     71.5
GPU 7     0      0         32     70.9
GPU 0     98     71680     58     413.1
...
(runs until Ctrl+C; add -c N to stop after N samples)

What to look at: GPUs 0-3 show 98% utilization, but at about 410 W they're far below the 700 W an H100 draws under full load, which hints the job isn't really busy. Look for one GPU much hotter or drawing much less power than its peers; GPUTL alone can't tell you whether the work is efficient.

#Entity   SMACT  TENSO  DRAMA
ID
GPU 0     0.312  0.081  0.124
GPU 1     0.305  0.079  0.121
GPU 2     0.318  0.084  0.127
GPU 3     0.309  0.080  0.122
GPU 4     0.000  0.000  0.000
GPU 5     0.000  0.000  0.000
GPU 6     0.000  0.000  0.000
GPU 7     0.000  0.000  0.000
GPU 0     0.297  0.077  0.119
...
(10 samples, 1000 ms apart)

What to look at: Profiling fields are ratios from 0 to 1. Here SM active is about 0.3 and tensor active below 0.1 while GPUTL reads 98%, so the GPUs are mostly waiting (often on data loading). A well-fed training job usually shows SMACT above 0.8, and 'N/A' means profiling is paused or another tool holds the counters.

+---------------------------+----------------------------------------------------------+
| Health Monitor Report                                                                |
+===========================+==========================================================+
| Overall Health            | Failure                                                  |
| GPU ID: 3                 | Failure                                                  |
|                           | Memory system: Failure                                   |
|                           |  - A volatile double-bit ECC error (count 1) was         |
|                           |    detected on GPU 3. Drain the GPU and reset it or      |
|                           |    reboot to clear. Pending row remap: yes.              |
+---------------------------+----------------------------------------------------------+

What to look at: Healthy output shows only 'Overall Health | Healthy'. Any Warning or Failure names the GPU and subsystem, and a memory Failure from a double-bit ECC error (Xid 48) is a hardware fault, so drain the node and run a level 3 diag. This check only reports on watches you set first with 'dcgmi health -g 2 -s a'.

Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 4.2.3                                          |
| Driver Version Detected   | 570.133.20                                     |
| GPUs Tested               | 0, 1, 2, 3, 4, 5, 6, 7                         |
|-----  Deployment  --------+------------------------------------------------|
| Software                  | Pass                                           |
+---------------------------+------------------------------------------------+

What to look at: Level 1 only checks the software: driver, libraries, permissions, persistence mode, and conflicting processes. A Fail here usually names a running compute process or a missing library. A Pass says nothing about GPU memory or NVLink hardware.

Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 4.2.3                                          |
| Driver Version Detected   | 570.133.20                                     |
| GPUs Tested               | 0, 1, 2, 3, 4, 5, 6, 7                         |
|-----  Deployment  --------+------------------------------------------------|
| Software                  | Pass                                           |
|-----  Integration  -------+------------------------------------------------|
| PCIe                      | Pass                                           |
| NVBandwidth               | Pass                                           |
| NCCL Tests                | Pass                                           |
|-----  Hardware  ----------+------------------------------------------------|
| Memory                    | Fail - GPU 3                                   |
| Warning                   | GPU 3: A double-bit ECC error was detected     |
|                           | (volatile DBE 1, pending row remap). Memory    |
|                           | test failed.                                   |
| Diagnostic                | Pass                                           |
|-----  Stress  ------------+------------------------------------------------|
| Memory Bandwidth          | Fail - GPU 3                                   |
| Targeted Stress           | Pass                                           |
| Targeted Power            | Pass                                           |
+---------------------------+------------------------------------------------+

What to look at: Read every row: a clean run is Pass on all tests, and any Fail names the GPU plus a Warning line with the reason. Keep this output as support evidence. It takes tens of minutes and loads the GPUs heavily, so run it only on a drained node.

{
  "DCGM GPU Diagnostic" : {
    "driver" : "570.133.20",
    "gpu_ids" : "0",
    "test_level" : 3,
    "version" : "4.2.3",
    "test_categories" : [
      {
        "category" : "Deployment",
        "tests" : [ { "name" : "software", "results" : [ { "entity_id" : 0, "status" : "Pass" } ] } ]
      },
      {
        "category" : "Hardware",
        "tests" : [
          { "name" : "memory", "results" : [ { "entity_id" : 0, "status" : "Pass" } ] },
          { "name" : "diagnostic", "results" : [ { "entity_id" : 0, "status" : "Pass" } ] }
        ]
      },
      {
        "category" : "Stress",
        "tests" : [
          { "name" : "targeted_stress", "results" : [ { "entity_id" : 0, "status" : "Pass" } ] },
          { "name" : "targeted_power", "results" : [ { "entity_id" : 0, "status" : "Pass" } ] }
        ]
      }
      (output trimmed)
    ]
  }
}

What to look at: Scripts should parse the status of every test and treat anything other than Pass or Skip as a failure. A 'Fail' status has a matching warnings entry with the reason; this GPU passed everything.

Policy information
+---------------------------+------------------------------------------------+
| allgpus                   | Policy Information                             |
+===========================+================================================+
| Violation conditions      | Double-bit ECC errors                          |
|                           | PCI errors and replays                         |
|                           | Max temperature threshold - 90                 |
|                           | Max power threshold - 700                      |
|                           | XID errors                                     |
|                           | NVLink errors                                  |
| Isolation mode            | Manual                                         |
| Action on violation       | Reset GPU                                      |
| Validation after action   | System validation (short)                      |
| Validation failure action | None                                           |
+---------------------------+------------------------------------------------+

What to look at: This is what 'dcgmi policy -g 2 --set 1,1 -e -p -x -n -T 90 -P 700' produces: action 1 means reset the GPU, validation 1 means run a short diag afterwards. If you want the evidence kept for a support case, the action should be None (--set 0,0), because a reset clears volatile error state.

Listening for violations.
Timestamp: Tue Nov 10 14:02:11 2026
A XID error has violated policy manager values.
XID error 48 detected on GPU 3 (00000000:5D:00.0)
...
(blocks until Ctrl+C)

What to look at: Each violation prints with the GPU and the condition that fired. Xid 48 (double-bit ECC) or 79 (fell off the bus) point to hardware, while 13 and 31 are usually application bugs. Nothing printed means no violations yet, not that the policy is broken.

Successfully started process watches.
Successfully started recording stats for job123.

What to look at: Both lines must appear before the job starts (typically in the Slurm prolog). After the job ends, 'dcgmi stats -x job123' prints 'Successfully stopped recording stats for job123.' A report for a job that was never started comes back empty.

Successfully retrieved statistics for job: job123.
+------------------------------------------------------------------------------+
| GPU ID: 0                                                                    |
+====================================+=========================================+
|-----  Execution Stats  ------------+-----------------------------------------|
| Start Time                         | Tue Nov 10 13:00:02 2026                |
| End Time                           | Tue Nov 10 13:41:05 2026                |
| Total Execution Time (sec)         | 2463.12                                 |
| No. of Processes                   | 1                                       |
+-----  Performance Stats  ----------+-----------------------------------------+
| Energy Consumed (Joules)           | 1391290                                 |
| Power Usage (Watts)                | Avg: 564.8, Max: 698.1, Min: 88.2       |
| Max GPU Memory Used (bytes)        | 73400320000                             |
| SM Clock (MHz)                     | Avg: 1755, Max: 1980, Min: 1410         |
| Memory Clock (MHz)                 | Avg: 2619, Max: 2619, Min: 2619         |
| SM Utilization (%)                 | Avg: 91, Max: 100, Min: 0               |
| Memory Utilization (%)             | Avg: 38, Max: 61, Min: 0                |
+-----  Event Stats  ----------------+-----------------------------------------+
| Single Bit ECC Errors              | 0                                       |
| Double Bit ECC Errors              | 0                                       |
| PCIe Replay Warnings               | 0                                       |
| Critical XID Errors                | 0                                       |
+-----  Slowdown Stats  -------------+-----------------------------------------+
| Due to - Power (%)                 | 0                                       |
|        - Thermal (%)               | 0                                       |
|        - Reliability (%)           | 0                                       |
|        - Board Limit (%)           | 0                                       |
|        - Low Utilization (%)       | 0                                       |
|        - Sync Boost (%)            | 0                                       |
+-----  Overall Health  -------------+-----------------------------------------+
| Overall Health                     | Healthy                                 |
+------------------------------------+-----------------------------------------+
(the same block repeats for GPUs 1-7)

What to look at: Use Energy Consumed for chargeback and the Event Stats for errors during the job. Non-zero Slowdown percentages (Thermal or Power) or any Double Bit ECC or Critical XID count mean the job ran on a degraded GPU.

+--------------------------+------------------------+------------------------+
| allgpus                  |                        |                        |
| Group of 8 GPUs          | TARGET CONFIGURATION   | CURRENT CONFIGURATION  |
+==========================+========================+========================+
| Sync Boost               | Not Specified          | Disabled               |
| SM Application Clock     | Not Specified          | ****                   |
| Memory Application Clock | Not Specified          | 2619                   |
| ECC Mode                 | Not Specified          | Enabled                |
| Power Limit              | 600                    | 600                    |
| Compute Mode             | Not Specified          | Unrestricted           |
+--------------------------+------------------------+------------------------+
**** Non-homogenous settings across group. Use with -v flag to see details.

What to look at: After 'dcgmi config -g 2 --set -P 600' (which prints 'Configuration successfully set.'), the Power Limit target and current columns should both read 600. If they differ, the setting didn't apply. DCGM re-applies the target after a GPU reset, which a one-off nvidia-smi -pl does not.

+----------------------+
|  NvLink Link Status  |
+----------------------+
GPUs:
    gpuId 0:
        U U U U U U U U U U U U U U U U U U
    gpuId 1:
        U U U U U U U U U U U U U U U U U U
    gpuId 2:
        U U U U U U U U U U U U U U U U U U
    gpuId 3:
        U U U U U U U U U U U U U U U U U U
    gpuId 4:
        U U U U U U U U U U U U U U U U U U
    gpuId 5:
        U U U U U U U U U U U U U U U U U U
    gpuId 6:
        U U U U U U U U U U U U U U U U U U
    gpuId 7:
        U U U U U U U U U U U U U U U U U U
NvSwitches:
    physicalId 8:
        U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U
    physicalId 9:
        U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U
    physicalId 10:
        U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U
    physicalId 11:
        U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U U

Key: Up=U, Down=D, Disabled=X, Not Supported=_

What to look at: Every H100 should show 18 links, all U (Up). A D (Down) on any GPU or NVSwitch port means a failed link: check Fabric Manager and dcgmi nvlink -e for that GPU, and expect slower all-reduce inside the node.

+-----------------------------+------------------------------------------------------+
| GPU ID: 0                   | NVLINK Error Counts                                  |
+-----------------------------+------------------------------------------------------+
|Link 0                       | CRC FLIT Error              => 0                     |
|Link 0                       | CRC Data Error              => 0                     |
|Link 0                       | Replay Error                => 0                     |
|Link 0                       | Recovery Error              => 0                     |
|Link 1                       | CRC FLIT Error              => 0                     |
|Link 1                       | CRC Data Error              => 0                     |
|Link 1                       | Replay Error                => 0                     |
|Link 1                       | Recovery Error              => 0                     |
...
|Link 17                      | CRC FLIT Error              => 0                     |
|Link 17                      | CRC Data Error              => 0                     |
|Link 17                      | Replay Error                => 0                     |
|Link 17                      | Recovery Error              => 0                     |
+-----------------------------+------------------------------------------------------+

What to look at: All counters should be 0 or stay flat over time. Replay or Recovery errors that keep growing on one link point to a marginal NVLink or NVSwitch connection, which usually also shows up as Xid 74.

Successfully paused profiling.

What to look at: While paused, profiling fields (1001-1012) read N/A in dmon and DCGM Exporter, so Nsight Compute can use the hardware counters. Run 'dcgmi profile --resume' (prints 'Successfully resumed profiling.') afterwards, or the dashboards stay blank.