HGX Firmware Updates & nvfwupd

What firmware lives on a DGX/HGX system, who manages it (BMC vs HMC), how nvfwupd pushes packages through Redfish, how updates are staged and activated, how to roll out across a fleet without breaking it, and how to detect and recover from a failed update. Written for NCP-AII objective 1.4.

1. The big picture: why firmware matters

A GPU server runs dozens of firmware images: the BIOS, the BMC, every GPU, every NVSwitch, PCIe switches and retimers, NICs, drives, power supplies. NVIDIA and the server vendor test specific combinations of these versions together with specific drivers. Mixing versions across a cluster, or within one node, causes problems that look like hardware faults or software bugs: GPUs missing, NVLink errors, lower bandwidth, nodes behaving differently from their neighbors.

A firmware recipe is like the approved parts list for an aircraft. Each part works on its own, but the airline only flies combinations that were certified together. Swapping in one part from a different list may still fly, until it doesn't.

The exam's angle on objective 1.4: know which tool updates which component, that HGX baseboard firmware goes through the BMC's Redfish service to the HMC, that updates must be activated (often by a power cycle), that you roll out carefully (drain, canary, waves), and how to detect and recover from failures.

2. What firmware is on an HGX system

Location	Component	Notes
Host (motherboard)	BIOS / UEFI	CPU, memory, PCIe settings; Secure Boot
	BMC	Out-of-band management controller for the whole server
	CPLD / FPGA	Board logic, power sequencing
	PSUs	Power supply firmware
HGX baseboard (GPU tray)	GPU firmware (VBIOS) and InfoROM	One per GPU (8 on an HGX H100 8-GPU board)
	NVSwitch firmware	One per NVSwitch chip
	HMC (HGX Management Controller)	The baseboard's own management controller
	ERoT (External Root of Trust)	Security chips that authenticate firmware
	PCIe switches	Connect GPUs, NICs, and drives to the host
	PCIe retimers	Signal conditioning on PCIe Gen5 links
	Baseboard FPGA / CPLD	Baseboard control logic
Add-in devices	ConnectX NICs, BlueField DPUs	Updated from the host with MFT (`mlxfwmanager`, `flint`)
	NVMe drives	`nvme fw-download` / `fw-commit` or vendor tools
	Transceivers	See the transceiver guide

The exact list depends on the platform (DGX H100/H200, HGX from an OEM, B200 systems) and generation. The firmware package's release notes list what it contains.

3. BMC vs HMC: who updates what

The BMC is the server's management controller on the out-of-band (OOB) network. It exposes Redfish.
The HMC (HGX Management Controller) sits on the GPU baseboard and manages the baseboard's components. It is normally not directly reachable; the host BMC talks to it internally and exposes the baseboard's firmware inventory and update service through its own Redfish service.
So HGX firmware updates go: admin host → BMC Redfish → HMC → GPU / NVSwitch / retimer flash.
In the BMC's Redfish firmware inventory, baseboard entries typically have HGX-prefixed names (for example GPU, NVSwitch, retimer, ERoT, and HMC entries). Exact IDs vary by platform.

BMC = the Building Manager for the whole server. HMC = the HGX board's own Manager, reached through the building manager.

GPU firmware on an HGX baseboard is not flashed with nvidia-smi or the GPU driver. It goes through the BMC / HMC path (with NVIDIA's tooling or Redfish). nvidia-smi is for reading the VBIOS version.

4. ERoT and signed firmware

An ERoT (External Root of Trust) is a security chip next to a component (GPU, NVSwitch, HMC, and others). It holds keys and checks that every firmware image is signed by NVIDIA before it's allowed to run, and protects against tampering.

You can only install NVIDIA-signed packages; modified or corrupted images are rejected.
Some platforms enforce anti-rollback: once a security fix is installed, older vulnerable versions may be refused.
The ERoTs themselves have firmware that sometimes needs updating first; release notes state the order.

A "signature verification failed" or "image rejected" result usually means the wrong package for this platform, a corrupted download (check the checksum), or a blocked downgrade. It is not a reason to bypass security.

5. Firmware recipes and release notes

NVIDIA publishes firmware release packages for DGX systems (and HGX OEMs publish theirs) with release notes: included versions, supported drivers, required order, known issues, and activation steps.
The recipe = the full set of tested versions: BIOS, BMC, HGX components, NIC firmware, driver, Fabric Manager, DOCA/OFED.
Driver and Fabric Manager versions must be compatible with the GPU and NVSwitch firmware. Plan the firmware and driver update together.

Every node runs the same recipe; the inventory you collect matches the release notes exactly; driver, Fabric Manager, and NIC firmware are from the same tested set.

6. nvfwupd: what it is and how it works

nvfwupd is NVIDIA's command-line firmware update tool for DGX and HGX systems. It runs on an admin machine (or the host), talks to the target's BMC over Redfish, and can:

Show installed firmware versions on the target.
Show the versions contained in a firmware package (.fwpkg, a PLDM firmware package).
Upload a package and start the update, optionally for selected components only.
Track the update task's progress.
Work against several systems using a configuration file.

nvfwupd is a courier who knows the building. You could carry the package to the building manager yourself (raw Redfish with curl), but the courier knows the right door, fills in the forms, and checks on delivery.

Subcommands and options differ between nvfwupd versions. The commands below show the shape; always check nvfwupd --help and the release notes for the version you use.

7. nvfwupd commands

# Credentials come from environment variables / a secret store; never hard-code them
nvfwupd --help

# 1. What's installed on the target system?
nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS show_version

# 2. Compare installed versions with one or more packages (adds Pkg Version and Up-To-Date columns)
nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS show_version -p nvfw_DGX_<ver>.fwpkg nvfw_DGX-HGX-H100-H200x8_<ver>.fwpkg
nvfwupd show_pkg_content -p <pkg>.fwpkg        # list what's inside a package (no target needed)
nvfwupd -t ... show_version -s                  # staged (installed but not yet active) versions

# 3. Update. -y skips the prompt; -s names a "special update" JSON file that picks the targets
echo '{}' > mb_tray.json                                                     # every component in the package
echo '{"Targets":["/redfish/v1/UpdateService/FirmwareInventory/HostBMC_0"]}' > update_bmc.json
echo '{"Targets":["/redfish/v1/UpdateService/FirmwareInventory/HGX_0"]}' > gpu_tray.json   # whole GPU tray
nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS update_fw -p <pkg>.fwpkg -y -s update_bmc.json

# 4. Run in the background (-b; add -j for JSON) and follow the task
nvfwupd -t ... update_fw -p <pkg>.fwpkg -y -s gpu_tray.json -b
nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS show_update_progress -i <task-id>

# 5. Several systems: a JSON list of {"ip","user","password"} objects, updated one after another
nvfwupd -t targets=fleet.json update_fw -p <pkg>.fwpkg -y -s gpu_tray.json   # no -b or show_update_progress here

The DGX H100/H200 guide documents -t targets=<file> for several systems. Newer nvfwupd releases move multi-system work into the -c config file and can update in parallel, so check the documentation for your version.

Step	Why
`show_version` before	Record the baseline (for comparison and rollback planning)
`show_version -p <pkgs>`	Confirm it's the right package for this platform and what will change (Up-To-Date column)
`update_fw`	Upload and stage; returns a task
`show_update_progress`	Wait for completion; catch failures
Activation	Reboot or power cycle as the release notes say (next section)
`show_version` after	Prove every package component is at the target version (InfoROM rows aren't in the package and stay N/A / No)

8. The same update with raw Redfish

R='curl -s -u $BMC_USER:$BMC_PASS --cacert bmc-ca.pem'
$R https://$BMC/redfish/v1/UpdateService | jq '{MultipartHttpPushUri, HttpPushUri}'
$R https://$BMC/redfish/v1/UpdateService/FirmwareInventory | jq -r '.Members[]."@odata.id"'   # HGX entries included

# Upload the package (multipart push), then follow the returned task
curl -s -u "$BMC_USER:$BMC_PASS" --cacert bmc-ca.pem -X POST "https://$BMC<MultipartHttpPushUri>" \
     -F 'UpdateParameters={"Targets":[]};type=application/json' -F 'UpdateFile=@nvfw_<platform>.fwpkg'
$R https://$BMC/redfish/v1/TaskService/Tasks/<id> | jq '{TaskState, PercentComplete, Messages}'

An empty Targets list means "everything applicable in the package"; listing specific firmware inventory URIs limits the update. See the Redfish guide for the HTTP details (202 Accepted, task monitors, ETags).

9. Hands-on lab: simulated tickets

Two tickets from head01, an admin host on the out-of-band network with nvfwupd 2.1.1, ipmitool, curl, and Slurm. Each simulated DGX H100 has two firmware packages, following the DGX H100/H200 Firmware Update Guide: the motherboard tray (nvfw_DGX_260625.1.0.fwpkg: BMC, BIOS, ERoTs, CPLDs, PCIe retimers and switches, PSUs) and the GPU tray (nvfw_DGX-HGX-H100-H200x8_0014_251020.1.0.fwpkg: HMC, GPUs, NVSwitches, FPGA, HGX ERoTs).

FWU-101: bring dgx01 to release 26.07.5 in the documented order. Compare versions, update and reset the BMC alone, then the motherboard tray, then the GPU tray with the GPUs idle. Wait for the ERoT background copy, then do a chassis power cycle, which activates the BIOS and GPU tray. Commit the backup images, then do an AC power cycle, which activates the ERoT, CPLD, PCIe switch, and retimer firmware. Verify and return the node.
FWU-202: audit dgx02–dgx04 after last week's rollout with one multi-system run. One node has firmware staged but never activated, and one has a GPU tray that never updated. Fix both without killing anyone's job.

Outputs are simplified mock-ups modeled on the NVIDIA guide's examples, not exact copies. Target versions are the published 26.07.5 component versions; installed versions and credentials are lab values (source lab.env). pdu off|on <node> stands in for unplugging all PSUs, either by hand or from a PDU. Each command advances time one step: BMC resets, power cycles, background updates (-b), and ERoT copies take a few steps. Write JSON files with echo '...' > file.json. Variables ($GPU) expand, | jq supports simple paths, Up/Down recall history, and Tab completes files and hosts.

Ticket

Ticket checklist

Answer key for this ticket (spoilers)

One clean path; other orders work too. The checklist tracks results, not exact commands.

10. Staging vs activation: reboot, DC cycle, AC cycle

Most components write the new image to an inactive flash bank (staging) and only run it after a reset (activation). Until then, the inventory may still report the old version, or show the new one as pending.

Activation method	What happens	How
BMC reset	Only the BMC restarts; the host keeps running	Redfish `Manager.Reset`, `ipmitool mc reset cold`
Host reboot (warm)	OS restarts; standby power stays on	`reboot`, Redfish `ComputerSystem.Reset` GracefulRestart
DC power cycle	Host powers off and on through the BMC; standby power remains	`ipmitool chassis power cycle`, Redfish `PowerCycle` / BCM `power reset`
AC power cycle	All power removed, including standby: BMC and HMC restart too	Remove input power (PDU outlets off, or pull cords), wait, restore. Some platforms offer an out-of-band equivalent.

On a DGX H100/H200, the current guide's order is: update the BMC alone, then ipmitool mc reset cold, then the motherboard tray ({}), then the GPU tray (Targets: HGX_0). Wait for "BackgroundCopyStatus": "Completed" on /redfish/v1/Chassis/HGX_ERoT_BMC_0, then run ipmitool chassis power cycle, which activates the BIOS and GPU tray firmware. Commit the new BMC and BIOS images to their backup slots (NvidiaUpdateService.CommitImage). Finally, do an AC power cycle: power the chassis off from the BMC, unplug all power supplies for five to six minutes, then reconnect them. The activation table lists PCIe switch, PCIe retimer, ERoT, and CPLD firmware as AC-cycle components, and an AC cycle activates every updated component. The lab walks through it.

On some HGX platforms and releases, components only activate after a full AC power cycle, because the baseboard and its controllers stay powered during a normal reboot or a DC cycle. "The update completed but the version didn't change" is the classic sign the required activation step wasn't done.

Never cut power during an update. Do the AC cycle only after the update task has completed, and only if the release notes call for it.

11. Firmware that nvfwupd doesn't cover

Component	Tool	Activation
ConnectX NIC / BlueField NIC firmware	`mlxfwmanager --query`, `mlxfwmanager -u -i <pkg>`, `flint -d <dev> -i <img> burn`	`mlxfwreset` or reboot / power cycle
BlueField Arm OS image (BFB)	`bfb-install` over RShim	DPU reboots into the new image
NVMe drives	`nvme fw-download` + `nvme fw-commit`	Commit action / reset
InfiniBand / Ethernet switches	Switch OS image upgrade, UFM for fleets	Switch reboot
Transceivers	MFT / switch tooling for LinkX modules	Link retrain / reseat per notes
OEM server BIOS / BMC (non-DGX HGX)	The OEM's tools or its Redfish service	Per OEM

sudo mst start
sudo mlxfwmanager --query                        # every NVIDIA NIC/DPU: current vs available firmware, PSID
sudo mlxfwmanager -u -i fw-ConnectX7-<ver>.bin  # update from a package / image
sudo mlxfwreset -d /dev/mst/mt4129_pciconf0 reset   # activate without a full reboot where supported

Full detail on NIC, DPU, switch, and module firmware: the networking firmware (MFT) guide.

The NIC's PSID identifies the board variant. Burning firmware for a different PSID is refused by default for a reason; don't force it.

12. Fleet rollout: canary, waves, drain

Read the release notes: versions, order, activation, driver/FM compatibility, known issues.
Inventory everything first (all nodes), so you know the starting point and spot nodes that already drifted.
Drain the nodes you'll update (Slurm scontrol update nodename=... state=drain reason="fw update", Kubernetes kubectl drain, BCM). Never update under running jobs.
Canary: update one node, activate, verify, run diagnostics and a short NCCL test.
Waves: update in batches (for example one rack or one scalable unit at a time) so a bad update never takes the whole cluster down.
Update the OS-side pieces in the same window if the recipe requires: driver, Fabric Manager, DOCA/OFED, through the BCM software image.
Verify and record the inventory for every node; return nodes to service; keep the evidence.

Stagger power: AC-cycling a whole rack at once can trip breakers and stress PDUs. Power-cycle in groups.

13. Interactive update plan builder

Pick what you're updating; the builder writes an ordered plan with the right tools and checks. It's a study aid: the release notes for your platform are the real authority.

Platform

Scheduler

14. Verifying after the update

nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS show_version    # every component at the target version
nvidia-smi --query-gpu=index,name,vbios_version,inforom.img --format=csv   # GPU VBIOS / InfoROM as the driver sees them
nvidia-smi -q | grep -i -A1 "vbios"
sudo mlxfwmanager --query                                             # NIC firmware
sudo dmidecode -t bios | grep -i version                              # BIOS
ipmitool mc info | grep -i "firmware revision"                        # BMC
systemctl status nvidia-fabricmanager ; nvidia-smi nvlink -s          # fabric up after the change
nvsm show health                                                      # DGX health
dcgmi diag -r 2                                                       # quick functional check (r3 for acceptance)

Inventory matches the target recipe on every node; no failed tasks or new SEL/Redfish errors; all 8 GPUs and all NVSwitches present; Fabric Manager running; NVLinks up; DCGM diag passes; performance matches the baseline (single-node NCCL, nvbandwidth).

15. Fault detection and recovery

Symptom	Likely cause	Next step
Task state `Exception` / failed	Wrong package, transfer error, component busy, dependency order	Read the task messages; check the package and order; retry
Update "completed" but version unchanged	Not activated yet	Do the activation step from the release notes (DGX H100: chassis power cycle for BIOS and GPU tray; AC power cycle for ERoT, CPLD, PCIe switch, and retimer). `show_version -s` shows what is staged
GPU or NVSwitch missing after update	Partial update, mismatched baseboard firmware	Keep drained; retry the package; check HGX inventory and health; contact support if it persists
Update service unresponsive	BMC in a bad state	BMC reset (`ipmitool mc reset cold` or Redfish), then retry
Signature / image rejected	Wrong platform package, corrupted file, blocked downgrade	Verify checksum and platform; use the supported version
Fabric Manager fails or CUDA 802 after update	Driver/FM incompatible with the new GPU/NVSwitch firmware, or FM version mismatch	Install the driver and FM versions the recipe specifies
Some nodes behave differently after a rollout	Partial rollout: version drift	Compare inventories across nodes; re-apply to stragglers

Many components keep a backup image (dual-bank flash) and can fall back if the new image fails to boot. A BMC whose primary image is damaged may boot from its secondary bank. Vendor support procedures cover recovery beyond that.

16. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

nvfwupd --helpTool version and command list

nvfwupd version 2.1.1
Usage: nvfwupd [ global options ] <command>
Global options:
    -t --target ip=<BMC IP> user=<BMC login id> password=<BMC password> servertype=<Type of server>
                (or targets=<json file> for several systems, updated serially)
Commands:
     help | version | show_pkg_content -p <pkg> | unpack -p <pkg>
     <Global options...> show_version [-p <pkg> [<pkg> ...]] [-j] [-s]
     <Global options...> update_fw -p <pkg> [-y] [-s <special json>] [-b] [-j]
     <Global options...> show_update_progress -i <task id> [<task id> ...] [-j]
     ...

What to look at: Note the version on the first line, since options differ between nvfwupd releases. Every target command takes the global -t first, then the subcommand.

nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS show_versionInstalled firmware on the target

System Model: DGXH100
Part number: 965-24387-0002-003
Serial number: 1660423000018
BMC IP: dgx01-bmc

Firmware Devices:
AP Name                   Sys Version
-------                   -----------
HostBMC_0                 26.03.17
HostBIOS_0                1.7.4
EROT_BIOS_0               00.04.0055.0000_n00
EROT_BMC_0                00.04.0055.0000_n00
CPLDMB_0                  0.2.1.9
CPLDMID_0                 0.2.1.3
PCIeRetimer_0             2.07.15
PCIeRetimer_1             2.07.15
PCIeSwitch_0              0.0.6
PCIeSwitch_1              1.0.6
PSU_0                     0204
PSU_1                     0204
HGX_FW_BMC_0              HGX-22.10-1-rc80
HGX_FW_ERoT_BMC_0         00.02.0182.0000_n00
HGX_FW_FPGA_0             2.51
HGX_FW_GPU_SXM_1          96.00.D9.00.01
...
HGX_FW_NVSwitch_0         96.10.6D.00.01
HGX_FW_NVSwitch_1         96.10.6D.00.01
...
HGX_FW_PCIeSwitch_0       1.7.5F
HGX_FW_PCIeRetimer_0      2.7.14
HGX_FW_PCIeRetimer_1      2.7.14
...
HGX_InfoROM_NVSwitch_0    G520.0201.00.01
--------------------------------------------------------------------------------
Error Code: 0

What to look at: This is the inventory: one row per component with its running Sys Version. Record it before any change; an expected component row missing, or the command failing to connect, means a BMC or network problem to fix first.

nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS show_version -p nvfw_DGX_260625.1.0.fwpkg nvfw_DGX-HGX-H100-H200x8_0014_251020.1.0.fwpkg

Compare installed vs package versions

System Model: DGXH100
Part number: 965-24387-0002-003
Serial number: 1660423000018
BMC IP: dgx01-bmc

Firmware Devices:
AP Name                   Sys Version           Pkg Version           Up-To-Date
-------                   -----------           -----------           ----------
HostBMC_0                 26.03.17              26.06.24              No
HostBIOS_0                1.7.4                 1.8.0                 No
EROT_BIOS_0               00.04.0055.0000_n00   00.04.0058.0000_n00   No
EROT_BMC_0                00.04.0055.0000_n00   00.04.0058.0000_n00   No
CPLDMB_0                  0.2.1.9               0.2.1.9               Yes
CPLDMID_0                 0.2.1.3               0.2.1.3               Yes
PCIeRetimer_0             2.07.15               2.07.19               No
PCIeRetimer_1             2.07.15               2.07.19               No
PCIeSwitch_0              0.0.6                 0.0.7                 No
PCIeSwitch_1              1.0.6                 1.0.7                 No
PSU_0                     0204                  0205                  No
PSU_1                     0204                  0205                  No
HGX_FW_BMC_0              HGX-22.10-1-rc80      HGX-22.10-1-rc83      No
HGX_FW_ERoT_BMC_0         00.02.0182.0000_n00   00.02.0192.0000_n00   No
HGX_FW_FPGA_0             2.51                  2.53                  No
HGX_FW_GPU_SXM_1          96.00.D9.00.01        96.00.DA.00.0C        No
...
HGX_FW_NVSwitch_0         96.10.6D.00.01        96.10.73.00.01        No
HGX_FW_NVSwitch_1         96.10.6D.00.01        96.10.73.00.01        No
...
HGX_FW_PCIeSwitch_0       1.7.5F                1.9.5F                No
HGX_FW_PCIeRetimer_0      2.7.14                2.7.20                No
HGX_FW_PCIeRetimer_1      2.7.14                2.7.20                No
...
HGX_InfoROM_NVSwitch_0    G520.0201.00.01       N/A                   No
--------------------------------------------------------------------------------
Error Code: 0

What to look at: Up-To-Date No marks what this update will change. The InfoROM row with Pkg Version N/A and No is expected because InfoROM is not in these packages; a system model that does not match the package is the real stop sign.

nvfwupd show_pkg_content -p nvfw_DGX_260625.1.0.fwpkgList components inside a package

Package: nvfw_DGX_260625.1.0.fwpkg
AP Name                   Version
HostBMC_0                 26.06.24
HostBIOS_0                1.8.0
EROT_BIOS_0               00.04.0058.0000_n00
EROT_BMC_0                00.04.0058.0000_n00
CPLDMB_0                  0.2.1.9
CPLDMID_0                 0.2.1.3
PCIeRetimer_0             2.07.19
PCIeRetimer_1             2.07.19
PCIeSwitch_0              0.0.7
PCIeSwitch_1              1.0.7
PSU_0                     0205
PSU_1                     0205

What to look at: This needs no target: it shows what the package contains and at which versions. Use it to confirm you have the motherboard-tray package versus the GPU-tray (HGX) package before writing the special update file.

nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS show_version -sStaged (not yet active) versions

System Model: DGXH100
Part number: 965-24387-0002-003
Serial number: 1660423000018
BMC IP: dgx01-bmc

Firmware Devices:
AP Name                   Sys Version           Staged Version
-------                   -----------           --------------
HostBMC_0                 26.06.24              N/A
HostBIOS_0                1.8.0                 N/A
EROT_BIOS_0               00.04.0055.0000_n00   00.04.0058.0000_n00
EROT_BMC_0                00.04.0055.0000_n00   00.04.0058.0000_n00
CPLDMB_0                  0.2.1.9               N/A
CPLDMID_0                 0.2.1.3               N/A
PCIeRetimer_0             2.07.15               2.07.19
PCIeRetimer_1             2.07.15               2.07.19
PCIeSwitch_0              0.0.6                 0.0.7
PCIeSwitch_1              1.0.6                 1.0.7
PSU_0                     0204                  0205
PSU_1                     0204                  0205
HGX_FW_BMC_0              HGX-22.10-1-rc83      N/A
HGX_FW_ERoT_BMC_0         00.02.0192.0000_n00   N/A
HGX_FW_FPGA_0             2.53                  N/A
HGX_FW_GPU_SXM_1          96.00.DA.00.0C        N/A
...
HGX_FW_NVSwitch_0         96.10.73.00.01        N/A
HGX_FW_NVSwitch_1         96.10.73.00.01        N/A
...
HGX_FW_PCIeSwitch_0       1.9.5F                N/A
HGX_FW_PCIeRetimer_0      2.7.20                N/A
HGX_FW_PCIeRetimer_1      2.7.20                N/A
...
HGX_InfoROM_NVSwitch_0    G520.0201.00.01       N/A
--------------------------------------------------------------------------------
Error Code: 0

What to look at: A version in Staged Version that differs from Sys Version is written but not active. Here the ERoT, PCIe switch, retimer and PSU rows are staged after the chassis power cycle; on DGX H100 those activate only after an AC power cycle.

nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS update_fw -p nvfw_DGX_260625.1.0.fwpkg -y -s update_bmc.jsonUpdate the BMC alone (special file)

FW package: ['nvfw_DGX_260625.1.0.fwpkg']
{"@odata.type": "#UpdateService.v1_11_0.UpdateService", "Messages": [{"Message": "A new task /redfish/v1/TaskService/Tasks/1 was created.", "MessageId": "Task.1.0.New", "Severity": "OK"}, {"Message": "The action UpdateService.MultipartPush was submitted to do firmware update.", "MessageId": "UpdateService.1.0.StartFirmwareUpdate", "Severity": "OK"}]}
 FW update started, Task Id: 1
Wait for Firmware Update to Start...
 Started Updating: HostBMC_0
 TaskState: Running
 PercentComplete: 40
 TaskStatus: OK
 TaskState: Completed
 PercentComplete: 100
 TaskStatus: OK
Firmware update successful!
 Overall Time Taken: 0:36:11
 Refer to the DGX H100/H200 Firmware Update Guide on activation steps for new firmware to take effect.

What to look at: Started Updating should name only HostBMC_0, because update_bmc.json targets it. TaskState Completed with TaskStatus OK is success, but the BMC still runs the old image until ipmitool mc reset cold.

nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS update_fw -p nvfw_DGX-HGX-H100-H200x8_0014_251020.1.0.fwpkg -y -s gpu_tray.json -b

Start GPU tray update in background

FW package: ['nvfw_DGX-HGX-H100-H200x8_0014_251020.1.0.fwpkg']
{"@odata.type": "#UpdateService.v1_11_0.UpdateService", "Messages": [{"Message": "A new task /redfish/v1/TaskService/Tasks/4 was created.", "MessageId": "Task.1.0.New", "Severity": "OK"}, {"Message": "The action UpdateService.MultipartPush was submitted to do firmware update.", "MessageId": "UpdateService.1.0.StartFirmwareUpdate", "Severity": "OK"}]}
 FW update started, Task Id: 4
Exiting; follow it with show_update_progress -i 4

What to look at: With -b the command returns at once; the number to keep is the Task Id. If the GPUs are busy (an nvidia-smi loop or dcgm-exporter), a pre-flight check fails instead, and you should stop that activity rather than skip the check.

nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS show_update_progress -i 4Follow an update task

Task Info for Id: 4
 StartTime: 2026-11-10T09:12:31+00:00
 TaskState: Running
 PercentComplete: 62
 TaskStatus: OK

What to look at: Poll until TaskState Completed and PercentComplete 100. TaskState Exception, or TaskStatus Warning with a message such as VBIOS update failed, GPU busy, means the task did not fully succeed; read the message before any power action.

nvfwupd -t targets=fleet.json update_fw -p nvfw_DGX-HGX-H100-H200x8_0014_251020.1.0.fwpkg -y -s gpu_tray.jsonUpdate several systems one after another

Updating ip address: dgx02-bmc
FW package: ['nvfw_DGX-HGX-H100-H200x8_0014_251020.1.0.fwpkg']
 FW update started, Task Id: 4
...
 TaskState: Completed
 PercentComplete: 100
 TaskStatus: OK
Firmware update successful!
 Overall Time Taken: 0:09:14
Updating ip address: dgx03-bmc
FW package: ['nvfw_DGX-HGX-H100-H200x8_0014_251020.1.0.fwpkg']
 FW update started, Task Id: 6
...
 TaskState: Completed
 PercentComplete: 100
 TaskStatus: OK
Firmware update successful!
 Overall Time Taken: 0:09:20
...

What to look at: Systems in the targets file are updated serially, so check each block for Completed and OK. Every system in the file is touched, so a node still running a Slurm job must not be in the list.

$R https://$BMC/redfish/v1/UpdateService | jq '{MultipartHttpPushUri, HttpPushUri}'Where the BMC accepts uploads

{
  "MultipartHttpPushUri": "/redfish/v1/UpdateService/upload",
  "HttpPushUri": "/redfish/v1/UpdateService"
}

What to look at: MultipartHttpPushUri is the path you POST the .fwpkg to. A null value means this BMC does not support multipart push and you need its other method (HttpPushUri or SimpleUpdate).

$R https://$BMC/redfish/v1/UpdateService/FirmwareInventory | jq -r '.Members[]."@odata.id"'List firmware inventory entries

/redfish/v1/UpdateService/FirmwareInventory/HostBMC_0
/redfish/v1/UpdateService/FirmwareInventory/HostBIOS_0
/redfish/v1/UpdateService/FirmwareInventory/EROT_BIOS_0
/redfish/v1/UpdateService/FirmwareInventory/EROT_BMC_0
/redfish/v1/UpdateService/FirmwareInventory/CPLDMB_0
/redfish/v1/UpdateService/FirmwareInventory/PCIeSwitch_0
/redfish/v1/UpdateService/FirmwareInventory/PSU_0
...
/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_BMC_0
/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_ERoT_BMC_0
/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_GPU_SXM_1
/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_NVSwitch_0
...
/redfish/v1/UpdateService/FirmwareInventory/HGX_0

What to look at: Host components appear as HostBMC_0, HostBIOS_0 and so on, and the HGX baseboard entries as HGX_FW_*; HGX_0 is the whole GPU tray that gpu_tray.json targets. Use these exact URIs in Targets, since a misspelled ID is rejected with PropertyValueNotInList.

curl -s -u "$BMC_USER:$BMC_PASS" --cacert bmc-ca.pem -X POST "https://$BMC/redfish/v1/UpdateService/upload" -F 'UpdateParameters={"Targets":[]};type=application/json' -F 'UpdateFile=@nvfw_DGX_260625.1.0.fwpkg'

Raw Redfish multipart push

{
  "@odata.id": "/redfish/v1/TaskService/Tasks/2",
  "@odata.type": "#Task.v1_4_3.Task",
  "Id": "2",
  "TaskState": "Running",
  "TaskStatus": "OK",
  "Messages": [
    {
      "@odata.type": "#Message.v1_1_1.Message",
      "Message": "The task with Id '2' has started.",
      "MessageArgs": [
        "2"
      ],
      "MessageId": "TaskEvent.1.0.TaskStarted",
      "Resolution": "None.",
      "Severity": "OK"
    }
  ]
}

What to look at: A Task resource with TaskState Running means the upload was accepted; note its Id for polling. An error object such as Base.1.15.InsufficientPrivilege or a 400 about UpdateParameters means nothing was started.

$R https://$BMC/redfish/v1/TaskService/Tasks/2 | jq '{TaskState, PercentComplete, Messages}'Poll a Redfish update task

{
  "TaskState": "Completed",
  "PercentComplete": 100,
  "Messages": [
    {
      "Message": "The task with Id '2' has started.",
      "MessageId": "TaskEvent.1.0.TaskStarted",
      "Severity": "OK"
    },
    {
      "Message": "Image 'nvfw_DGX_260625.1.0.fwpkg' is being transferred to 'HostBIOS_0'.",
      "MessageId": "Update.1.0.TransferringToComponent",
      "Severity": "OK"
    },
    ...
    {
      "Message": "The task with Id '2' has completed.",
      "MessageId": "TaskEvent.1.0.TaskCompletedOK",
      "Severity": "OK"
    }
  ]
}

What to look at: Completed at 100 with TaskCompletedOK is success. TaskState Exception, or a message with Severity Critical such as TaskAborted, means the update failed and the message tells you which component.

curl -s -k -u $BMC_USER:$BMC_PASS https://dgx01-bmc/redfish/v1/Chassis/HGX_ERoT_BMC_0 | jq .Oem.Nvidia.BackgroundCopyStatus

HGX ERoT background copy state

"Completed"

What to look at: After a GPU tray update this must read Completed before the chassis power cycle. InProgress means wait and check again; power-cycling now can interrupt the ERoT copy.

curl -s -k -u $BMC_USER:$BMC_PASS -X POST https://dgx01-bmc/redfish/v1/UpdateService/Actions/Oem/NvidiaUpdateService.CommitImage -H 'Content-Type: application/json' -d '{"Targets":["/redfish/v1/UpdateService/FirmwareInventory/HostBMC_0"]}'

Commit active image to backup slot

{
   "@odata.id": "/redfish/v1/TaskService/Tasks/5",
   "@odata.type": "#Task.v1_4_3.Task",
   "Id": "5",
   "TaskState": "Running",
   "TaskStatus": "OK",
   "Messages": [
      {"Message":"A new task /redfish/v1/TaskService/Tasks/5 was created.","MessageId":"Task.1.0.New","Severity":"OK"},
      {"Message":"ActivateFirmware Action is initiated.","MessageId":"UpdateService.1.0.StartActivateFirmware","Severity":"OK"}
   ]
}

What to look at: CommitImage returns a task; wait for it to reach 100% before committing the next component. Run it only after the new image is active, otherwise the backup slot gets the old firmware.

ipmitool -I lanplus -H dgx01-bmc -U $BMC_USER -E mc reset coldReboot the BMC to load new image

Sent cold reset command to MC

What to look at: Only the BMC restarts; the host keeps running. Expect IPMI and Redfish to be unreachable for a few minutes, so wait for ping before confirming with mc info.

ipmitool mc info | grep -i "firmware revision"Confirm the BMC firmware version

Firmware Revision         : 26.06

What to look at: After the BMC update and cold reset this should show the new major.minor (26.06). Still seeing 26.03 means the BMC has not booted the new image yet, or the reset was skipped.

ipmitool -I lanplus -H dgx01-bmc -U $BMC_USER -E chassis power cycleDC power cycle to activate BIOS/GPU tray

Chassis Power Control: Cycle

What to look at: This activates BIOS and GPU-tray firmware on DGX H100 but not ERoT, CPLD, PCIe switch or retimer firmware, which need an AC power cycle. An error saying Command not supported in present state means the host is off; use chassis power on.

nvidia-smi --query-gpu=index,name,vbios_version,inforom.img --format=csvGPU VBIOS and InfoROM per GPU

index, name, vbios_version, inforom.img
0, NVIDIA H100 80GB HBM3, 96.00.DA.00.0C, G520.0200.00.05
1, NVIDIA H100 80GB HBM3, 96.00.DA.00.0C, G520.0200.00.05
2, NVIDIA H100 80GB HBM3, 96.00.DA.00.0C, G520.0200.00.05
3, NVIDIA H100 80GB HBM3, 96.00.DA.00.0C, G520.0200.00.05
4, NVIDIA H100 80GB HBM3, 96.00.DA.00.0C, G520.0200.00.05
5, NVIDIA H100 80GB HBM3, 96.00.DA.00.0C, G520.0200.00.05
6, NVIDIA H100 80GB HBM3, 96.00.DA.00.0C, G520.0200.00.05
7, NVIDIA H100 80GB HBM3, 96.00.DA.00.0C, G520.0200.00.05

What to look at: All eight GPUs should report the same VBIOS, matching the package version. One GPU on the old VBIOS (96.00.D9.00.01 here) means its update failed or has not been activated by a power cycle.

nvsm show healthDGX system health summary

Info
----
Timestamp                                        : Tue Nov 10 11:42:07 2026
Version                                          : 24.03.06

Checks
------
Verify installed DIMM memory sticks.......................... Healthy
Number of logical CPU cores [224]............................ Healthy
GPU link speed [0000:1b:00.0][32GT/s]........................ Healthy
GPU link width [0000:1b:00.0][x16]........................... Healthy
...
Verify chassis fan presence.................................. Healthy
Verify PSU presence.......................................... Healthy
BMC firmware version [26.06.24].............................. Healthy
...

Health Summary
--------------
205 out of 205 checks are healthy
0 out of 205 checks are unhealthy
0 out of 205 checks are unknown
0 out of 205 checks are informational

Overall system status is Healthy

What to look at: After the update every check should be Healthy and the summary should show 0 unhealthy. Any Unhealthy line, for example a GPU link speed below 32GT/s or a missing PSU, must be fixed before the node goes back into service.

17. Scenarios

18. Searchable reference

Item	Meaning

19. Flashcards (click to flip)

20. Exam traps

HGX baseboard firmware (GPUs, NVSwitches, retimers) goes through the BMC's Redfish to the HMC, not through nvidia-smi.
nvfwupd drives the BMC's Redfish update service; it runs from an admin machine over the OOB network.
Update completed but version unchanged → activation step missing (often an AC power cycle).
DC cycle ≠ AC cycle. A DC cycle through the BMC leaves standby power on; the baseboard controllers keep running.
Never interrupt power during an update.
Drain first, then canary, then waves.
One recipe for the whole cluster, including driver and Fabric Manager versions.
NIC firmware is MFT (mlxfwmanager / flint), not nvfwupd.
Signature rejected → wrong/corrupt package or blocked downgrade; don't bypass security.
Inventory before and after; compare across nodes to find drift.

21. Practice quiz (22 questions)

Score: 0 / 0