NVSM: NVIDIA System Management

Info
----
Timestamp                        : Tue Nov 10 09:30:12 2026 +0800
Version                          : 24.09.3

Checks
------
Verify installed DIMM memory sticks.......................... Healthy
Number of logical CPU cores [224]............................ Healthy
Number of GPUs [8]........................................... Healthy
GPU link speed [0000:1b:00.0][32GT/s]........................ Healthy
GPU link speed [0000:43:00.0][32GT/s]........................ Healthy
...
GPU link width [0000:1b:00.0][x16]........................... Healthy
GPU link width [0000:43:00.0][x16]........................... Healthy
...
Number of NVSwitches [4]..................................... Healthy
NVSwitch / NVLink link state................................. Healthy
Number of NVMe drives [10]................................... Unhealthy
Verify OS drive RAID-1 volume................................ Healthy
Verify data drive RAID-0 volume.............................. Unhealthy
Root file system usage....................................... Healthy
Check BMC sensor thresholds.................................. Healthy
Verify chassis fan presence.................................. Healthy
Verify PSU presence / status................................. Healthy
...

Health Summary
--------------
204 out of 206 checks are healthy
2 out of 206 checks are unhealthy
0 out of 206 checks are unknown
0 out of 206 checks are informational
Overall system status is Unhealthy

What to look at: Go straight to the Health Summary and the last line: a healthy DGX ends with 'Overall system status is Healthy' and 0 unhealthy checks. Here a missing NVMe drive breaks the RAID-0 data volume; a PCIe link width of x8 instead of x16 or a GPU count below 8 would also show up as Unhealthy.

/systems/localhost/storage/alerts/alert0
Properties:
    system_name = dgx-h100-07
    message_details = NVMe drive nvme5n1 has failed / missing from RAID-0 volume
    component_id = nvme5n1
    description = Storage sub-system is reporting an error
    event_time = 2026-11-10 09:12:44
    recommended_action =
        1. Please run nvsm dump health
        2. Please open a case with NVIDIA Enterprise Support
        3. Attach this notification and the nvsm dump health output file
    severity = Critical
    alert_id = NV-DRIVE-01
Verbs:
    cd
    show

What to look at: Each alert names the component, its severity, and an alert ID such as NV-DRIVE-01, NV-PSU-03, or NV-VOL-01. No output means no active alerts; Critical ones like this failed RAID-0 member need a support case with the dump health bundle attached.

Writing output to /tmp/nvsm-health-dgx-h100-07-20261110093512.tar.xz
Done.

What to look at: The archive path is the file to attach to the NVIDIA Enterprise Support case. Run it while the fault is still present and before rebooting, and handle the bundle carefully because it contains hostnames, IPs, and configuration.

The stress test puts a heavy load on the system and may disrupt running workloads.
Do you want to continue? [y/N]: y

Test Plan Summary
-----------------
CPU      Duration: 300 s
DCGM     Duration: 300 s (dcgmi diag)
GPU      Duration: 300 s
MEMORY   Duration: 300 s
PCIE     Duration: 300 s
STORAGE  Duration: 300 s
...
Running stress test, this may take a while ...
[==============================================] 100%

Stress Test Summary
-------------------
CPU      : PASS
DCGM     : PASS
GPU      : PASS
MEMORY   : PASS
PCIE     : PASS
STORAGE  : FAIL  (nvme5n1: device not found)
...
Overall result: FAIL
Logs: /var/log/nvsm/stress-test/<timestamp>/

What to look at: The end summary gives a PASS or FAIL per component, and any FAIL names the part. It asks for confirmation unless you add --force, and it loads the whole system heavily, so drain the node before running it.

usage: nvsm stress-test [--usage, -h, --help] [--force] [--no-prompt] [<test>...] [DURATION]

NVIDIA System Management Stress Testing

positional arguments:
  <test>      Component test(s) to run, for example CPU, DCGM, GPU,
              MEMORY, PCIE, STORAGE. Default: all supported tests
  DURATION    Duration of each test in seconds

optional arguments:
  --usage, -h, --help  Show this help message and exit
  --force              Run without asking for confirmation
  --no-prompt          Don't prompt during the test
(output trimmed)

What to look at: Use it to see which component tests and options your NVSM version supports before scripting a stress test. NVIDIA's DGX documentation recommends 'sudo nvsm stress-test --force' (about 20 minutes, all components) before production use and after servicing.

nvsm-> cd /systems/localhost
nvsm(/systems/localhost)-> show
/systems/localhost
Targets:
    gpus
    memory
    networkadapters
    processors
    storage
Verbs:
    cd
    show
nvsm(/systems/localhost)-> cd gpus
nvsm(/systems/localhost/gpus)-> show GPU0
/systems/localhost/gpus/GPU0
Properties:
    Inventory_ModelName = NVIDIA H100 80GB HBM3
    Inventory_UUID = GPU-7c1e0a42-0000-4000-8000-000000000000
    Inventory_SerialNumber = 1650000000000
    Connections_PCIeGen = 5
    Connections_PCIeLinkWidth = 16x
    Connections_PCIeLocation = 00000000:1B:00.0
    Power_PowerDraw = 71.32 W
    Stats_ErrorStats_ECCMode = Enabled
    Stats_PerformanceState = P0
    Stats_UtilGPU = 0 %
    Status_Health = OK
Verbs:
    cd
    show
nvsm(/systems/localhost/gpus)-> exit

What to look at: The prompt shows your current path, and show at each level lists child Targets and Properties. On a GPU, check Status_Health = OK and Connections_PCIeLinkWidth = 16x; a lower width or Gen means the GPU trained a degraded PCIe link.

/systems/localhost/gpus
Targets:
    GPU0
    GPU1
    GPU2
    GPU3
    GPU4
    GPU5
    GPU6
    GPU7
    alerts
    policy
Verbs:
    cd
    show

What to look at: A DGX H100 should list GPU0 through GPU7. A missing GPU target matches a failed 'Number of GPUs [8]' health check and usually an Xid 79 in dmesg; show a single GPU (for example /systems/localhost/gpus/GPU5) for its properties.

/systems/localhost/storage/volumes/md0
Properties:
    Status_State = Enabled
    Status_Health = OK
    Name = md0
    Encrypted = False
    VolumeType = RAID-1
    Drives = [ nvme0n1, nvme1n1 ]
    CapacityBytes = 1.7T
    Id = md0
Verbs:
    cd
    show

/systems/localhost/storage/volumes/md1
Properties:
    Status_State = Enabled
    Status_Health = Critical
    Name = md1
    Encrypted = False
    VolumeType = RAID-0
    Drives = [ nvme2n1, nvme3n1, nvme4n1, nvme6n1, nvme7n1, nvme8n1, nvme9n1 ]
    CapacityBytes = 28T
    Id = md1
Verbs:
    cd
    show

What to look at: md0 is the OS RAID-1 mirror and md1 is the /raid RAID-0 cache. md1 is Critical and nvme5n1 is missing from its drive list; because it is RAID-0, the whole /raid volume is lost and must be recreated after the drive is replaced.

/chassis/localhost/power/PSU1
Properties:
    Status_State = Present
    Status_Health = OK
    LastPowerOutputWatts = 1712
    Name = PSU1
    SerialNumber = LABPSU000001
    PowerSupplyType = AC
    Manufacturer = Delta
Verbs:
    cd
    show
...
/chassis/localhost/power/PSU4
Properties:
    Status_State = Present
    Status_Health = Critical
    LastPowerOutputWatts = 0
    Name = PSU4
    SerialNumber = LABPSU000004
    PowerSupplyType = AC
    Manufacturer = Delta
Verbs:
    cd
    show
...

What to look at: Each of the six PSUs should be Present and OK with a similar output wattage. PSU4 is present but Critical at 0 W, which usually means it lost AC input (alert NV-PSU-03) from a dropped PDU feed rather than a failed supply.

/chassis/localhost/thermal/fans/FAN1_F
Properties:
    Status_State = Enabled
    Status_Health = OK
    Name = FAN1_F
    ReadingUnits = RPM
    LowerThresholdNonCritical = 5046.000
    Reading = 11760 RPM
    LowerThresholdCritical = 3596.000
Verbs:
    cd
    show
...
/chassis/localhost/thermal/fans/FAN6_R
Properties:
    Status_State = Enabled
    Status_Health = OK
    Name = FAN6_R
    ReadingUnits = RPM
    LowerThresholdNonCritical = 5046.000
    Reading = 10920 RPM
    LowerThresholdCritical = 3596.000
Verbs:
    cd
    show

What to look at: Compare each Reading with its LowerThresholdCritical and with the other fans. A fan at 0 RPM or below the critical threshold shows Status_Health Critical, and the GPUs behind it will run hot and throttle.

  UNIT                                  LOAD   ACTIVE SUB     DESCRIPTION
  nvsm-api-gateway.service              loaded active running NVSM API Gateway
  nvsm-core.service                     loaded active running NVSM Core
  nvsm-mqtt.service                     loaded active running NVSM MQTT
  nvsm-notifier.service                 loaded active running NVSM Notifier
  nvsm-plugin-environment.service       loaded active running NVSM environment plugin
  nvsm-plugin-gpumonitor.service        loaded active running NVSM GPU monitor plugin
  nvsm-plugin-memory.service            loaded active running NVSM memory plugin
  nvsm-plugin-monitor-storage.service   loaded active running NVSM storage monitor plugin
  nvsm-plugin-pcie.service              loaded active running NVSM PCIe plugin
  nvsm.service                          loaded active exited  NVIDIA System Management
...

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
14 loaded units listed.

What to look at: Every NVSM unit should be loaded and active. A failed or inactive nvsm-core or nvsm-api-gateway is why nvsm commands hang or error out, so restart the services before trusting any health result.

nvidia-bug-report.sh will now collect information about your
system and create the file 'nvidia-bug-report.log.gz' in the current
directory.  It may take several seconds to run.  In some
cases, it may hang trying to capture data generated dynamically
by the Linux kernel and/or the NVIDIA kernel module.  While
the bug report log file will be incomplete if this happens, it
may still contain enough data to diagnose your problem.

If nvidia-bug-report.sh hangs, consider running with the --safe-mode
and --extra-system-data command line arguments.

Please include the 'nvidia-bug-report.log.gz' log file when reporting
your bug via the NVIDIA Linux forum (see forums.developer.nvidia.com)
or by sending email to 'linux-bugs@nvidia.com'.

Running nvidia-bug-report.sh... complete.

What to look at: The result is nvidia-bug-report.log.gz in the current directory, covering the GPU driver side only. For a DGX case NVIDIA support expects nvsm dump health first; send this as well if they ask for it.

   1 | 11/09/2026 | 22:14:03 | Event Logging Disabled #0x01 | Log area reset/cleared | Asserted
   2 | 11/10/2026 | 09:12:41 | Power Supply PSU4_Status | Power Supply AC lost | Asserted
   3 | 11/10/2026 | 09:12:41 | Power Unit #0x02 | Redundancy Lost | Asserted
   4 | 11/10/2026 | 09:31:20 | Power Supply PSU4_Status | Power Supply AC lost | Deasserted

What to look at: Read the events in time order: 'Asserted' means the condition started and 'Deasserted' means it cleared. This works out-of-band even when the OS is hung, so it is the tool to use when NVSM can't run; never put the real BMC password on a shared command line or in notes.

Chassis Power Control: Cycle

What to look at: This line only means the BMC accepted the request. Confirm the node came back with 'chassis power status' or the BMC console, then run nvsm show health once the OS is up.

{
  "@odata.context": "/redfish/v1/$metadata#ComputerSystemCollection.ComputerSystemCollection",
  "@odata.id": "/redfish/v1/Systems",
  "@odata.type": "#ComputerSystemCollection.ComputerSystemCollection",
  "Description": "Collection of Computer Systems",
  "Members": [
    {
      "@odata.id": "/redfish/v1/Systems/DGX"
    }
  ],
  "Members@odata.count": 1,
  "Name": "Systems Collection"
}

What to look at: The Members list gives the system path to query next (here /redfish/v1/Systems/DGX) for power state, health, and logs. A 401 error means wrong credentials; no response means the BMC network is unreachable, not that the OS is down.