DGX H100 Node Recovery with Base Command Manager

A study guide that starts from zero and works up to designing hands-off node recovery with confidence.

1. The scenario

A DGX H100 in a Base Command Manager cluster loses power and comes back up. Nobody is on call. What has to be in place so the node returns to service on its own, with the right software and ready to take jobs?

Three ideas drive the answer:

"Automatically": anything that needs a human after every reboot is wrong.
"Rejoin the cluster": the node needs the correct operating system and configuration, and it has to be ready to do work (accept jobs, use its GPUs).
"Managed by Base Command Manager": answer the way BCM is designed to work, which is central control from a head node.

2. What a cluster is: head node and compute nodes

An AI or HPC (high-performance computing) cluster is a group of computers ("nodes") networked together and run as a single system. There are two main roles:

Role	What it does
Head node (also called the management node)	The "brain." It holds the master copy of the operating system images, hands out network addresses, boots the other machines, monitors health, and runs the cluster management software (BCM). Users usually don't run heavy jobs here.
Compute nodes (also called worker nodes)	The "muscle." These are the machines that run the actual work, such as training AI models. A DGX H100 is a compute node.

The head node is like a restaurant's head chef with the master recipe book. The compute nodes are line cooks. If a line cook goes home sick and comes back, they don't cook from memory. They check the head chef's recipe book so every plate comes out the same.

3. What a DGX H100 is

H100 is an NVIDIA data-center GPU (graphics processing unit) from the "Hopper" generation, built for AI training and inference.

A DGX H100 is a complete server NVIDIA sells with the GPUs already built in. It contains:

8 × H100 GPUs with 80 GB of HBM3 memory each (640 GB of GPU memory in total).
NVSwitch chips with NVLink, a very fast internal fabric that lets all 8 GPUs talk to each other directly, much faster than going over normal PCIe.
ConnectX-7 network cards (InfiniBand/Ethernet) that link it to other DGX nodes in the cluster.
Dual CPUs, system RAM, and local NVMe drives.

Because the GPUs are connected through NVSwitch, a DGX H100 needs a software service called NVIDIA Fabric Manager running before multi-GPU work can happen. This matters for one of the correct answers.

Many DGX nodes together form a DGX BasePOD or DGX SuperPOD. NVIDIA's standard software for managing those clusters is Base Command Manager.

4. What Base Command Manager (BCM) is, and how it thinks

NVIDIA Base Command Manager (BCM), formerly called Bright Cluster Manager, is cluster management software. It runs on the head node and handles:

Provisioning: installing and updating the OS on every compute node over the network.
Configuration: deciding which software, services, and settings each node gets.
Monitoring: health checks, GPU status, alerts.
Workload manager integration: setting up schedulers such as Slurm or Kubernetes.

Tools you'll see: cmsh (the BCM command-line shell), Base View (the web GUI), and cmd/CMDaemon (the daemon that runs on every node and talks to the head node).

The BCM design philosophy: centralized, image-based management

The core idea, which you should memorize:

The head node's software image is the single source of truth. Compute nodes are treated as interchangeable copies of that image. They don't "own" their configuration. They get it from the head node.

A software image is a full directory tree of an operating system (for example /cm/images/dgx-h100-image on the head node), containing the OS, drivers, CUDA, Slurm, Fabric Manager, and config files. You assign an image to a group of nodes called a category (for example "dgx-h100"), and every node in that category gets that image.

Why BCM is designed this way

Benefit	Explanation
Consistency	Every node runs exactly the same software. With 100+ DGX nodes, a single node with a different driver version can break or slow down a distributed training job.
No configuration drift	"Drift" means nodes slowly diverging because someone tweaked one by hand. Syncing from the image on every boot wipes out drift.
Easy node replacement	A motherboard dies? Swap the hardware, register its MAC address, power it on, and it installs itself from the image.
Change once, apply everywhere	Update the image on the head node and every node picks the change up (on reboot, or by pushing the update live).
Known good state	After any reboot, planned or not, a node comes back to the tested, approved configuration.

Think of school computer labs that "reset" to a clean image every time they restart. Whatever a student changed, the machine comes back identical to every other machine in the lab. BCM does the same thing for GPU servers.

5. What PXE boot is

PXE (pronounced "pixie") stands for Preboot eXecution Environment. It's a standard that lets a computer boot over the network instead of only from its own disk.

Normally a computer powers on and loads the OS from its local disk. With PXE, the network card (NIC) itself asks the network "who can boot me?" before anything loads from disk.

How PXE works, step by step

Power on. The BIOS/UEFI firmware is set to try network boot first.
DHCP request. The NIC broadcasts: "I'm MAC address aa:bb:cc..., I need an IP address and boot instructions."
The head node answers. Its DHCP server gives the node an IP address and says where to download a boot program.
TFTP download. The node downloads a small boot loader and kernel from the head node (TFTP is a simple file-transfer protocol).
The node installer runs. In BCM, this small program identifies the node and prepares its disk (more on this next).

PXE is like a new employee who, on arrival each morning, goes to the front desk to collect their badge and today's instructions, instead of relying on yesterday's notes in their pocket.

PXE is what gives the head node a chance to check and fix a node every time it boots. Without network boot, the node would just boot whatever is on its own disk, which might be stale or broken.

6. Why nodes sync from the head node

After PXE, BCM's node installer runs on the compute node. It:

Identifies the node (by MAC address or switch port), so it knows "this is dgx-07, category dgx-h100".
Picks an install mode:
- AUTO (the default): compares the local disk with the head node's image and copies over only the differences (an rsync-style sync). Fast.
- FULL: wipes and repartitions the disk, then copies the entire image. Used for new or corrupted nodes.
- NOSYNC: skips syncing (rarely used; defeats the point).
Hands off to the now-correct local OS, which boots normally.

Why sync on every boot?

An unplanned reboot (power loss, kernel panic, hardware fault, crash) might leave the node in an unknown state:

Files corrupted during the crash
Someone made manual changes before it went down
The image on the head node was updated while the node was offline

Syncing from the head node's image solves all three: whatever happened, the node returns to the known good state and matches every other node. That's why PXE boot plus image sync is half of the answer.

7. Services that must start at boot: Slurm and Fabric Manager

Having the right OS on disk isn't enough. To actually rejoin the cluster and do work, certain background programs ("services" or "daemons") must be running.

Service	What it does	What happens if it doesn't start
`slurmd` (Slurm daemon)	Slurm is the job scheduler: users submit jobs, and Slurm decides which nodes run them. `slurmd` runs on each compute node and talks to the central controller (`slurmctld`).	The scheduler sees the node as DOWN and never sends it jobs. The node is "on" but hasn't rejoined the cluster's workload.
`nvidia-fabricmanager` (Fabric Manager)	Configures and manages the NVSwitch fabric so the 8 GPUs can communicate over NVLink.	On NVSwitch systems like the DGX H100, the GPUs can't be used properly for CUDA work. Multi-GPU training fails.
`cmd` (CMDaemon)	The BCM agent on the node that reports health to the head node and receives configuration.	The head node loses visibility and control of the node.

On Linux, services start at boot through systemd:

systemctl enable slurmd                 # start automatically at every boot
systemctl enable nvidia-fabricmanager
systemctl status slurmd                 # check whether it is running

In BCM you don't run this on each node by hand. You enable the services in the software image or category configuration on the head node (BCM can also manage services for a category through cmsh), and every node picks that up when it syncs. That's the other half of the answer: services that start on boot, paired with PXE boot and image sync.

D gets the right software onto the node. C makes sure that software is running and ready for work. You need both to "rejoin the cluster."

8. The whole recovery flow, start to finish

Unplanned reboot: power blip, crash, or kernel panic.
BIOS/UEFI tries the network first, so the node PXE boots.
The head node answers with DHCP and TFTP and sends the BCM node installer.
The node installer identifies the node and syncs its disk against the software image (AUTO mode).
The node boots its now-correct local OS.
systemd starts the enabled services: nvidia-fabricmanager, slurmd, cmd, and so on.
The node reports to the head node, Slurm marks it IDLE, and it's back in the cluster with zero human action.

9. Weighing the common approaches

WRONGKeep each node's configuration on its own local storage so it never depends on the network

This sounds sensible for a single standalone server, but it goes against BCM's philosophy. If each node keeps its own configuration:

Nodes drift apart over time (configuration drift).
Updates on the head node don't reach the node reliably.
A corrupted local configuration after a crash stays corrupted.

Red flag words: "local," "never depends on the network." In BCM the head node is the source of truth, and depending on it is the point.

WRONGHave an admin re-run bcm-pod-setup by hand whenever a node reboots

Two problems:

"Manually … after each reboot" directly contradicts "automatically rejoin."
bcm-pod-setup is a helper for initial deployment of a DGX POD with BCM. It's a one-time setup step, not a reboot-recovery routine.

Red flag word: "manually."

CORRECTMake the services the node needs (slurmd, Fabric Manager) start on boot

Without slurmd, the scheduler won't send jobs to the node. Without Fabric Manager, the NVSwitch-connected GPUs aren't usable. Enabling them through systemd, configured in the BCM software image, makes the node functionally rejoin the cluster on its own.

CORRECTNetwork-boot (PXE) the node and sync it to the head node's software image every time it starts

This is BCM's core provisioning mechanism. PXE gives the head node control at every boot, and syncing puts the node back into the known good, identical state no matter what caused the reboot.

10. How to spot the right answers on similar questions

Words like "automatic," "unattended," or "after an unplanned reboot": eliminate anything with "manually," "after each reboot run…," or "log in and…".
"Managed by BCM" or "Base Command Manager": favor centralized answers (head node, software image, category, PXE, provisioning). Be suspicious of local or per-node persistence.
"Rejoin the cluster" or "ready for jobs": think in two layers:
1. Correct OS and configuration (provisioning and sync)
2. Correct services running (slurmd, Fabric Manager, cmd)
A DGX or HGX with NVSwitch: if GPUs or NVLink aren't working, think Fabric Manager.
A Slurm node shows DOWN: check whether slurmd is running and can reach slurmctld.

11. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

cmsh -c "softwareimage; list"The image nodes sync from

Name (key)           Path                             Kernel version           Nodes
-------------------- -------------------------------- ------------------------ --------
default-image        /cm/images/default-image         5.15.0-119-generic       0
dgx-h100-image       /cm/images/dgx-h100-image        5.15.0-1063-nvidia       8

What to look at: All DGX nodes should be counted under dgx-h100-image, the single source of truth on the head node. Nodes spread across several images are a sign of configuration drift.

cmsh -c "category use dgx-h100; get softwareimage; get installmode"Category image and install mode

dgx-h100-image
AUTO

What to look at: AUTO is what you want for hands-off recovery: every boot does a differential sync against the image. FULL here would wipe and reinstall the disk on every reboot, and NOSYNC would skip the sync and let drift survive.

tail -f /var/log/node-installerWatch a rebooted node resync

Nov 10 19:28:02 bcm-head01 node-installer[2201]: dgx-07: Node-installer started (version 10.24.x)
Nov 10 19:28:05 bcm-head01 node-installer[2201]: dgx-07: Node identified by MAC 0C:42:A1:3E:00:07 on BOOTIF
Nov 10 19:28:06 bcm-head01 node-installer[2201]: dgx-07: Category dgx-h100, software image dgx-h100-image
Nov 10 19:28:06 bcm-head01 node-installer[2201]: dgx-07: Installmode is: AUTO
Nov 10 19:28:08 bcm-head01 node-installer[2201]: dgx-07: Partition layout matches disk setup, keeping local filesystems
Nov 10 19:28:09 bcm-head01 node-installer[2201]: dgx-07: Synchronizing /cm/images/dgx-h100-image to local disk (AUTO)
Nov 10 19:30:44 bcm-head01 node-installer[2201]: dgx-07: Rsync completed
Nov 10 19:30:49 bcm-head01 node-installer[2201]: dgx-07: Calling init, switching to local root

What to look at: After an unplanned reboot you should see the node identified by its MAC, Installmode AUTO, and a completed sync. If no lines appear for the node, it didn't PXE boot, so check the BIOS boot order (network first) and the provisioning network.

systemctl enable slurmdStart slurmd at every boot

Created symlink /etc/systemd/system/multi-user.target.wants/slurmd.service → /lib/systemd/system/slurmd.service.

What to look at: The symlink line means slurmd will start at boot. No output means it was already enabled. In BCM, make this change in the software image or through the slurmclient role, not by hand on one node, or the next sync can undo it.

systemctl enable nvidia-fabricmanagerStart Fabric Manager at every boot

Created symlink /etc/systemd/system/multi-user.target.wants/nvidia-fabricmanager.service → /lib/systemd/system/nvidia-fabricmanager.service.

What to look at: Same check as for slurmd. "Unit file nvidia-fabricmanager.service does not exist" means the Fabric Manager package is missing from the image, which on a DGX H100 leaves the NVSwitch fabric unconfigured.

systemctl status slurmdIs slurmd running after boot?

● slurmd.service - Slurm node daemon
     Loaded: loaded (/lib/systemd/system/slurmd.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-11-10 19:31:02 CST; 4min 12s ago
   Main PID: 3121 (slurmd)
      Tasks: 2
     Memory: 6.1M
        CPU: 84ms
     CGroup: /system.slice/slurmd.service
             └─3121 /cm/shared/apps/slurm/current/sbin/slurmd -D -s

Nov 10 19:31:02 dgx-07 systemd[1]: Started Slurm node daemon.
Nov 10 19:31:02 dgx-07 slurmd[3121]: slurmd: slurmd version 23.02.7 started
Nov 10 19:31:02 dgx-07 slurmd[3121]: slurmd: CPUs=224 Boards=1 Sockets=2 Cores=56 Threads=2 Memory=2063731 TmpDisk=1795364 Uptime=95

What to look at: You want "enabled" on the Loaded line and "active (running)" on the Active line. "disabled" means it won't come back after the next reboot, and "inactive (dead)" or "failed" means the node will show DOWN in Slurm.

systemctl status nvidia-fabricmanagerIs the NVSwitch fabric configured?

● nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-11-10 19:30:58 CST; 4min 16s ago
    Process: 2840 ExecStart=/usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg (code=exited, status=0/SUCCESS)
   Main PID: 2851 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 21.4M
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2851 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

Nov 10 19:30:50 dgx-07 systemd[1]: Starting NVIDIA fabric manager service...
Nov 10 19:30:58 dgx-07 nv-fabricmanager[2851]: Connected to 1 node.
Nov 10 19:30:58 dgx-07 nv-fabricmanager[2851]: Successfully configured all the available NVSwitches to route GPU NVLink traffic.
Nov 10 19:30:58 dgx-07 systemd[1]: Started NVIDIA fabric manager service.

What to look at: Look for active (running) and the "Successfully configured all the available NVSwitches" message. A failed state, often with a driver/Fabric Manager version mismatch message, means multi-GPU CUDA jobs on this node will fail.

cmsh -c "category use dgx-h100; services; list"Services BCM starts on the category

Service (key)              Monitored  Autostart
-------------------------- ---------- ----------
nvidia-fabricmanager       yes        yes
nvidia-persistenced        yes        yes

What to look at: Autostart yes means CMDaemon starts the service on every node in the category, and Monitored yes means it restarts the service if it dies. A required service missing from this list, or set to Autostart no, has to be fixed centrally here or in the image, not on each node.

cmsh -c "device; status"Node back UP in BCM

bcm-head01 ............... [   UP   ]
dgx-01 ................... [   UP   ]
dgx-02 ................... [   UP   ]
...
dgx-06 ................... [   UP   ]
dgx-07 ................... [   UP   ]
dgx-08 ................... [   UP   ]

What to look at: UP means CMDaemon on the node is talking to the head node again. A node stuck in INSTALLING or DOWN long after power came back points to PXE, provisioning, or the cmd service.

sinfoNode back IDLE in Slurm

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
batch*       up   infinite      8   idle dgx-[01-08]

What to look at: The recovered node should be idle (or alloc/mix once it has jobs). down* with an asterisk means slurmctld can't reach slurmd on that node, so check systemctl status slurmd there.

scontrol show node dgx-07Why Slurm marked a node DOWN

NodeName=dgx-07 Arch=x86_64 CoresPerSocket=56
   CPUAlloc=0 CPUEfctv=224 CPUTot=224 CPULoad=0.00
   Gres=gpu:8
   NodeAddr=dgx-07 NodeHostName=dgx-07 Version=23.02.7
   RealMemory=2063000 AllocMem=0 FreeMem=N/A Sockets=2 Boards=1
   State=DOWN* ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=batch
   BootTime=None SlurmdStartTime=None
   LastBusyTime=2026-11-10T19:02:11 ResumeAfterTime=None
   CfgTRES=cpu=224,mem=2063000M,billing=224,gres/gpu=8
   Reason=Not responding [slurm@2026-11-10T19:08:40]

What to look at: State=DOWN* with Reason=Not responding means slurmd isn't running or can't reach slurmctld, which is the classic "booted but never rejoined" symptom. Once slurmd is running, the node normally returns to IDLE by itself if ReturnToService allows it; otherwise run scontrol update nodename=dgx-07 state=resume.

12. Self-check quiz

13. Glossary

Term	Meaning
BCM	NVIDIA Base Command Manager (formerly Bright Cluster Manager): cluster provisioning and management software.
Head node	The central management server that holds the software images and runs BCM.
Compute node	A worker machine that runs jobs (for example a DGX H100).
Software image	A master OS directory tree on the head node that compute nodes are provisioned from.
Category	A BCM group of nodes that share an image and settings.
Provisioning	Installing or syncing an OS image onto a node over the network.
PXE	Preboot eXecution Environment: booting from the network using DHCP and TFTP.
DHCP	The protocol that hands out IP addresses (and, for PXE, boot information).
TFTP	Trivial File Transfer Protocol, used to download the boot loader and kernel.
Node installer	The BCM program that runs after PXE to identify the node and sync its disk.
AUTO / FULL install	Sync only the differences / wipe the disk and copy the whole image.
Configuration drift	Nodes gradually becoming different from each other because of local changes.
DGX H100	An NVIDIA server with 8 H100 GPUs linked by NVSwitch/NVLink.
NVLink / NVSwitch	NVIDIA's high-speed GPU-to-GPU link, and the switch chips that connect all the GPUs.
Fabric Manager	The NVIDIA service (`nvidia-fabricmanager`) that configures the NVSwitch fabric. Required on DGX/HGX H100.
Slurm	A job scheduler. `slurmctld` is the controller and `slurmd` is the daemon on each compute node.
systemd	The Linux service manager. `systemctl enable X` makes X start at boot.
CMDaemon (`cmd`)	The BCM agent running on each node that talks to the head node.