Base Command Manager (BCM)

How NVIDIA Base Command Manager (formerly Bright Cluster Manager) runs an AI cluster: the head node and CMDaemon, cmsh and its commit workflow, categories, node overrides, roles and configuration overlays, software images and imageupdate, PXE provisioning and install modes, head-node high availability with fencing, Slurm with Enroot and Pyxis, and MIG through cmsh. Ends with five hands-on simulated tickets. Goes deeper than the BCM parts of the bring-up guide. Objectives 3.1, 3.2, and 3.3.

1. What BCM is and where it fits

Base Command Manager is the control plane for bare-metal nodes. One or two head nodes hold the cluster's configuration in a database, serve the OS images, PXE-boot and provision every compute node, and set up the workload manager. You change the cluster by changing that configuration; BCM pushes it to the nodes.

Piece	What it does
Head node	Runs everything below. Two head nodes give high availability (one active, one passive).
CMDaemon	The BCM service on every node. On the head it owns the configuration database (MySQL); on compute nodes it applies roles and reports health.
cmsh / Base View	The shell and the web GUI. Both are clients of the same CMDaemon API, so anything you do in one shows in the other.
Software images	Full root filesystems under `/cm/images/<name>` on the head node. Nodes copy one to local disk at boot.
Node installer	A small OS the node PXE boots first. It identifies the node, partitions the disk if needed, syncs the image, then hands over to the real OS.
Shared storage	`/cm/shared` (software for all nodes, for example Slurm) and `/home`, exported by the head node or a NAS.

BCM is a franchise headquarters. The head node keeps the master recipe book (images) and the staff rules (categories and roles). Each restaurant (node) gets a fresh copy of the book when it opens and updates when headquarters pushes changes. A cook who scribbles in their own copy (editing a node by hand) loses the notes at the next update.

Change the image or the category, then push. Hand edits on a node are temporary: the next sync or reprovision undoes them. This one rule answers many exam questions.

Objective	What it asks	Sections here
3.1	Install BCM, configure it, verify HA	Installing, cmsh, HA
3.2	Install the OS (on nodes: through images)	Software images, provisioning
3.3	Categories, interfaces, Slurm, Enroot, Pyxis	Categories, roles and overlays, networks, Slurm

2. Installing the head node and the license

Boot the BCM installer (ISO over BMC virtual media or USB) on the first head node.
Pick networks. A typical DGX layout: internalnet (provisioning and management, nodes PXE boot here), externalnet (site network, the way admins reach the head node), ipminet (BMCs), and ibnet (IPoIB on the compute fabric). The head node needs an interface on internalnet and one on externalnet.
Disk layout, time, DNS, admin password, then the installer lays down the head-node OS, CMDaemon, the database, and a default software image.
License: request-license on the head node, then check with cmsh -c "main; licenseinfo". The license is tied to the head node MAC addresses (both, for HA) and caps the node count.
Reference configuration: on DGX BasePOD/SuperPOD, run bcm-pod-setup once to create the DGX networks, categories, and images.
Add the nodes (next sections), then the second head node and HA, then the workload manager with cm-wlm-setup.

The license is bound to the head nodes' MAC addresses. Add the second head node's MAC when you request the license for HA, or the passive head can't take over legally. Replacing a head node's NIC can also require a new license.

3. cmsh: modes, objects, and the commit workflow

cmsh works like a filesystem of objects. You enter a mode (device, category, softwareimage, configurationoverlay, network, partition, main, wlm, ...), pick an object with use, then read or change its properties.

[bcm-head01]% device
[bcm-head01->device]% list                         # every node, MAC, category, IP, status
[bcm-head01->device]% use dgx05
[bcm-head01->device[dgx05]]% show                  # all properties
[bcm-head01->device[dgx05]]% get category
[bcm-head01->device[dgx05]]% set category dgx-h100
[bcm-head01->device*[dgx05*]]% commit              # the * means uncommitted changes
[bcm-head01->device[dgx05]]% exit                  # up one level
[bcm-head01->device]% foreach -n dgx05..dgx08 (set category dgx-h100; commit)

Command	Meaning
`set` / `append` / `removefrom`	Replace a value / add to a list / remove from a list. `set` on a list replaces the whole list.
`clear <prop>`	Remove a node's own value so it inherits from its category again.
`commit`	Save the staged changes to CMDaemon. Nothing happens on the cluster until you commit.
`refresh`	Throw away uncommitted changes in this mode.
`foreach -n a..b (cmds)`, `-c category`	Run commands on many objects. Node ranges like `dgx05..dgx08` work in most commands with `-n`.
`cmsh -c "mode; cmd; cmd"`	Non-interactive: good for scripts. Include `commit` in the string, or the change is lost when cmsh exits.

set is writing in pencil; commit is handing the form in. A * in the prompt is the pencil mark that hasn't been handed in yet.

A change without commit never reached CMDaemon. If you then reboot the node, it provisions with the old settings. Quitting cmsh with uncommitted changes discards them (cmsh asks first).

4. Categories, nodes, and inheritance

A category is a template for a group of identical nodes: software image, install mode, disk layout, kernel parameters, roles, filesystem mounts, and more. Every node belongs to exactly one category. Make one category per hardware or software type, for example dgx-h100 and dgx-b200.

Nodes inherit every setting from their category, and a node can override one. show tells you which values come from the category.
Booleans are different: a node's Boolean and its category's are combined with OR. A Boolean that's true in the category can't be turned off on one node.
Overrides are how one-off differences hide. A node that was pointed at a test image months ago keeps that image silently while its category moves on. clear <property> puts it back on inheritance.

category; clone default dgx-h100; commit          # new category from an existing one
category; use dgx-h100; set softwareimage dgx-os-6; commit
device; use dgx03; show                           # look for values that are NOT inherited
device; use dgx03; clear softwareimage; commit    # back to the category's image

Category = the default, node = the exception. Look for exceptions first when one node behaves differently from the rest.

5. Roles and configuration overlays (priorities)

A role makes a node run a service: slurmclient, slurmserver, slurmsubmit, slurmaccounting, boot, storage, and so on. You can assign a role in three places, and when the same role comes from more than one place, the highest priority wins:

Assigned on	Priority	Notes
Category	250 (fixed)	Applies to all nodes of the category.
Configuration overlay	500 by default, settable 0–1000 (not 250 or 750). −1 means the overlay is ignored.	A named bundle of roles applied to a list of nodes and/or categories. This is what `cm-wlm-setup` creates.
Node	750 (fixed)	Beats categories and default overlays. Handy for a test, dangerous to leave behind.

Overlays that cm-wlm-setup creates for Slurm (names can differ by version):

Overlay	Priority	Role, usually on
`slurm-server`	500	slurmserver on the head node(s): slurmctld
`slurm-client`	500	slurmclient on CPU node categories: slurmd
`slurm-client-gpu`	450	slurmclient on GPU categories, with GPU settings (GRES)
`slurm-submit`	500	slurmsubmit: login and submit hosts
`slurm-accounting`	500	slurmaccounting: slurmdbd
`wlm-headnode-submit`	600	Lets the head node submit jobs

configurationoverlay; list
configurationoverlay; use slurm-client-gpu; append categories dgx-b200; commit
configurationoverlay; use slurm-client-gpu; roles; use slurmclient; show
device; use dgx10; roles; list                    # a node-level role here beats the overlay
device; imageupdate -c dgx-b200 -w                # role changes reach nodes on the next sync

To add new GPU nodes to Slurm, add their category to the GPU client overlay. Don't assign slurmclient node by node: those node-level roles (priority 750) then ignore every later overlay change.

set categories dgx-b200 replaces the overlay's category list, so the H100 nodes silently drop out of Slurm. Use append.

6. Networks and node interfaces

BCM networks are objects (network mode) with a base address, netmask, domain, and type. Node interfaces point at them. A DGX node usually has:

Interface	Network	Purpose
`BOOTIF` (provisioning)	internalnet	PXE boot, image sync, management. Its MAC identifies the node.
`ipmi0` (BMC)	ipminet	Power control and the console. Needed for `power` commands and HA fencing.
`ib0`/`ibp…` (IPoIB)	ibnet	IP over the compute fabric, for storage or tools that need IP. NCCL itself uses RDMA verbs, not IPoIB.
bonded or BlueField ports	storage or in-band networks	Site dependent

device; use dgx01; interfaces; list
device; use dgx01; interfaces; add physical ib0; set network ibnet; set ip 10.149.0.11; commit
device; use dgx01; set mac 0c:42:a1:3e:00:01; commit      # after a motherboard or NIC swap

A node is identified at PXE boot by the MAC of its boot interface, or by the switch port it's cabled to if you configure that. After a motherboard swap the MAC changes, so update the node's MAC or BCM treats it as an unknown new node. See BCM node recovery.

7. Software images: change, preview, push

An image is a directory tree on the head node. You change it in place, then push the changes to running nodes or let them pick them up on the next boot.

softwareimage; clone dgx-os-6 dgx-os-6-test; commit          # copy for testing
cm-chroot-sw-img /cm/images/dgx-os-6-test                     # chroot with /proc, /sys, /dev mounted
  apt update && apt install <package>                          # inside the image
  exit
device; use dgx01; set softwareimage dgx-os-6-test; commit    # one canary node tries it
device; use dgx01; imageupdate                                # dry run
device; use dgx01; synclog                                    # what would change
device; use dgx01; imageupdate -w                             # do it, no reboot
device; use dgx01; clear softwareimage; commit                # canary back to the category

Tool	Direction	Notes
`imageupdate`	image → running node	A dry run by default. Read `synclog`, then repeat with `-w` to write. Use `-n` for nodes, `-c` for a category. Excludes runtime paths (for example `/proc`, logs) through exclude lists.
`grabimage`	running node → image	Pulls a node's changes back into its image (also dry run until `-w`). Use when you tuned a node by hand and want to keep it.
`cm-chroot-sw-img`	edit the image	Use it instead of plain `chroot`: it mounts the special filesystems package scripts need. Services don't start inside a chroot; that's expected.
Reboot	image → node	AUTO install mode syncs the image at boot. Required for kernel changes.

The flow is clone → change in chroot → canary node → dry run → synclog → -w → category. Driver and kernel changes need a reboot; package and config changes can usually go live with imageupdate -w.

Installing a package on a node over ssh "works" until the next sync. It's not a fix. Installing it on the head node instead of in the image (forgetting the chroot) is the other common slip: compute nodes never see it.

8. Provisioning and install modes

The node PXE boots from the head node on internalnet and loads the node installer.
The installer identifies the node (MAC or switch port) and asks CMDaemon for its category, image, disk setup, and install mode.
It checks the disk against the category's disk setup and, depending on the install mode, partitions it and/or syncs the image (rsync from /cm/images/<image>).
It switches root to the local disk and the node boots its real OS; CMDaemon starts the node's roles.

Install mode	Disk	Image sync	Use it for
AUTO (default)	Checked; repartitioned (FULL) only if the layout doesn't match	Incremental (only differences)	Normal boots
FULL	Repartitioned and reformatted: local data is erased	Complete	New nodes, corrupted disks, layout changes
MAIN	Untouched	None: the node stops in maintenance mode	Inspecting a node's disk from the installer
NOSYNC	Checked (repartitioned if it doesn't match)	Skipped	Rare; boots whatever is already on disk
SKIP	Not checked	Skipped	Booting a known-good local install as is

Where the mode comes from:

installmode on the node, inherited from its category by default (shown as AUTO from the category).
nextinstallmode: one boot only, then cleared. The safe way to force a FULL reinstall once: device; use dgx07; set nextinstallmode FULL; commit; reboot.
New nodes BCM hasn't seen before use newnodeinstallmode (FULL by default). Choosing an install mode from the PXE boot menu defaults to AUTO.

tail -f /var/log/node-installer          # on the head node: every node's installer messages
device; status                           # UP / DOWN / INSTALLING / INSTALLER_CALLINGINIT ...

NOSYNC is not a "protect my data" switch: if the disk layout doesn't match, the installer still repartitions. To keep data on a node, fix the disk setup or use MAIN to inspect first. And a FULL install on a node that holds local scratch data (for example /raid) erases it.

AUTO = "only what changed", FULL = "start over", MAIN = "stop and let me look", NOSYNC / SKIP = "don't copy".

9. Head-node high availability

Two head nodes, one active and one passive. The active one owns the shared IP addresses, runs CMDaemon in active mode, slurmctld, and the provisioning services; the passive one mirrors the database and waits.

Setting it up

Install and license the cluster on head01 (with both heads' MACs in the license).
Run cmha-setup (an interactive wizard): it clones head01 to head02, sets up database replication (MySQL), and configures shared IPs: one on internalnet (nodes use it to reach "the head node") and one on externalnet (users and admins use it). Each head keeps its own static external IP too.
Shared storage for /cm/shared and /home, so both heads serve the same files: an NFS server or NAS (BCM 11 dropped the old shared-disk option). The active head mounts it and re-exports it.
Power control on both heads (BMC credentials, powercontrol ipmi0 or similar), so each one can power off the other.

How failover decides

The passive head checks the active one (failoverping, ping, mysql, status).
If the active head looks dead, the passive one asks for a quorum: the compute nodes vote, and more than half must agree the active head is unreachable.
Then it fences the old head: powers it off through its BMC (STONITH, "shoot the other node in the head") before taking the shared IPs. Without working power control it won't take over, because two active heads (split brain) would corrupt the database and provisioning.
You can turn automatic failover off (disableautomaticfailover in the failover settings) and rely on manual cmha makeactive.

cmha status                    # four checks per direction, all [ OK ]; * marks the active head
bcm-head01* -> bcm-head02       failoverping  [  OK  ]
bcm-head01* -> bcm-head02       mysql         [  OK  ]
...
cmha makeactive                # on the PASSIVE head: take over (manual failover)
cmha dbreclone bcm-head02      # re-copy the database to a head whose replica is broken
cmsh -c "device; power status -n bcm-head01,bcm-head02"    # fencing works?
echo c > /proc/sysrq-trigger   # on the active head: a real crash test (lab or maintenance window only)

Two pilots in one cockpit. The co-pilot takes the controls only after confirming the captain is incapacitated (quorum) and after making sure the captain's hands are off the yoke (fencing). If the co-pilot can't confirm the second part, grabbing the yoke anyway would leave two people steering.

Verifying HA means: cmha status all OK, power control working for both heads, a manual failover with makeactive, proof that cmsh, provisioning, and Slurm work from the new active head, and a failback. A makeactive test alone doesn't prove automatic failover, because it skips fencing.

cmha makeactive runs on the head that should become active (the passive one), not on the current active head. If the database replicas disagree after an outage, re-clone with cmha dbreclone rather than forcing a failover.

10. Slurm, Enroot, and Pyxis through BCM

cm-wlm-setup (wizard or --setup with options) installs Slurm into /cm/shared, creates the overlays above, writes slurm.conf and gres.conf, and starts the services. BCM manages those files: change them through cmsh (or the wizard), not by hand, or the next role update rewrites them.
GPUs as GRES: the slurmclient role's gpuautodetect setting writes Slurm's AutoDetect in gres.conf. nvml asks the driver for GPU details, but slurm.conf still needs the GPU count, so the role's genericresources must define the GPUs (cm-wlm-setup creates them in the slurm-client-gpu overlay). bcm uses what BCM detected and needs nothing else; none means you describe the GPUs by hand. Check with scontrol show node dgx01 | grep Gres (expect gpu:8).
Enroot (container runtime that unpacks images into unprivileged sandboxes) and Pyxis (the Slurm SPANK plugin that adds --container-image to srun) go into the compute image; Pyxis is listed in Slurm's plugstack.conf. NGC credentials live in ~/.config/enroot/.credentials.
Prove it: sinfo shows the nodes idle, srun -N2 --gpus-per-node=8 nvidia-smi -L lists 16 GPUs, and a container job works: srun --container-image=nvcr.io#nvidia/pytorch:<tag> nvidia-smi -L.

Slurm commands on the head node need the Slurm environment module: module load slurm (BCM sets it up for root by default). If sinfo says "command not found", that's the reason, not a broken Slurm.

Day-to-day Slurm commands

Need	Command	Remember
Node and partition state	`sinfo`, `sinfo -R`	`-R` lists the reason each node is drained or down
What's queued or running	`squeue`, `squeue -O jobid,name,state,tres-per-node`	`tres-per-node` shows the GPUs each job asked for (older guides use the format letter `%b`)
Everything about one job	`scontrol show job <id>`	The `TRES` and `TresPerNode` lines show the GPUs requested and allocated, plus the nodes
Cancel a job	`scancel <id>`	`scancel -u alice` cancels all of a user's jobs. Not `scontrol cancel` or `skill`
Take a node out of service	`scontrol update nodename=dgx01 state=drain reason="..."`	Running jobs finish; nothing new starts. `state=resume` puts it back

Setting	What it does
`SelectType=select/cons_tres` (slurm.conf)	cons_tres = consumable trackable resources. CPUs, memory, and GRES such as GPUs are allocated individually, so several jobs can share a node, and GPU options like `--gpus-per-task` work
`GresTypes=gpu` plus `gres.conf`	Declares the GPUs as GRES (generic resources); `AutoDetect=nvml` finds them
`ConstrainDevices=yes` (cgroup.conf)	The job's cgroup can open only the GPUs it was allocated. Slurm also sets `CUDA_VISIBLE_DEVICES`, so the job sees exactly those GPUs without any manual binding

More: NGC with Enroot and Pyxis, BCM disk setup and node storage.

11. MIG through BCM

MIG (Multi-Instance GPU) splits one GPU into up to seven isolated instances, each with its own SMs, memory, and cache, so small jobs stop wasting a whole H100. BCM can manage MIG from cmsh, which keeps the layout in CMDaemon like any other setting. All of it runs in device mode on a node (device use dgx04):

Check: mig status shows Active and Pending per GPU. mig profiles [gpu] lists what the GPU supports. H100 80GB: 1g.10gb (up to 7), 1g.20gb (4), 2g.20gb (3), 3g.40gb (2), 4g.40gb (1), 7g.80gb (1). The name is GPU slices (out of 7) and memory.
Enable: mig enable 0-1 (no list = every GPU) sets the pending state; mig disable undoes it. A100 needs a node reboot to activate it; H100 and newer don't. The GPU must be idle: a running job leaves it pending with "In use by another client", so drain the node first.
Store the layout: gpusettings; add nvidia 0-1; set migprofiles 7* 1g.10gb; commit. N* repeats the next profile, so 2* 1g.10gb 3g.40gb is two 1g.10gb and one 3g.40gb.
Create the instances: mig apply, then check with mig show, or nvidia-smi -L on the node (each MIG device has its own MIG- UUID).
Tell Slurm: with AutoDetect=nvml (not nvidia, which doesn't see MIG), Slurm counts each MIG device as a GPU. Two GPUs split 7 ways plus six whole ones is 20, not 8, so the node needs its own slurmclient overlay with that count. Slurm doesn't repartition MIG itself and reads the devices when slurmd starts, so push the role (imageupdate -w) after the layout changes.

Persistence traps. mig apply --profile 1g.10gb creates instances right away but isn't stored: they're gone at the next reboot. Only a profile set in gpusettings persists. On H100, MIG mode itself isn't kept across reboots either (A100 keeps it in the InfoROM); BCM turns it back on at boot when gpusettings has MIG profiles. Overallocating isn't an error: profiles that don't fit are left out, so read mig show.

Use one MIG manager. If the DGX's own MIG manager service or the GPU Operator's nvidia-mig-manager handles MIG, BCM leaves it alone; running two at once isn't supported. Source: BCM Administrator Manual, "MIG Configuration"; NVIDIA MIG User Guide; Slurm GRES docs.

12. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

cmsh -c "main; licenseinfo"Check the BCM license

License info
-------------------------------- ------------------------------------------
Licensed to                      Example Corp AI Cluster
Version                          10.0
Edition                          Advanced
Type                             Commercial
Serial                           <redacted>
Start time                       Mon Nov 2 00:00:00 2026
End time                         Tue Nov 2 23:59:59 2027
Licensed nodes                   64
Node count                       10
Licensed GPUs                    512
Accelerator count                64
MAC address / Cloud ID           52:54:00:AA:00:01|52:54:00:AA:00:02

What to look at: Check that End time is in the future and that Node count stays below Licensed nodes. For an HA pair, both head-node MACs must appear on the MAC line, or the license breaks after a failover.

[bcm-head01->device]% listAll nodes, MACs, categories, IPs

Type             Hostname (key)   MAC                Category         Ip               Network          Status
---------------- ---------------- ------------------ ---------------- ---------------- ---------------- ----------------
HeadNode         bcm-head01       52:54:00:AA:00:01                   10.141.255.254   internalnet      [   UP   ]
HeadNode         bcm-head02       52:54:00:AA:00:02                   10.141.255.253   internalnet      [   UP   ]
PhysicalNode     dgx01            0C:42:A1:3E:00:01  dgx-h100         10.141.0.1       internalnet      [   UP   ]
PhysicalNode     dgx02            0C:42:A1:3E:00:02  dgx-h100         10.141.0.2       internalnet      [   UP   ]
PhysicalNode     dgx03            0C:42:A1:3E:00:03  dgx-h100         10.141.0.3       internalnet      [   UP   ]
PhysicalNode     dgx04            0C:42:A1:3E:00:04  dgx-h100         10.141.0.4       internalnet      [   UP   ]
PhysicalNode     dgx05            0C:42:A1:3E:00:05  default          10.141.0.5       internalnet      [  DOWN  ]
PhysicalNode     dgx06            0C:42:A1:3E:00:06  default          10.141.0.6       internalnet      [  DOWN  ]
PhysicalNode     dgx07            0C:42:A1:3E:00:07  default          10.141.0.7       internalnet      [  DOWN  ]
PhysicalNode     dgx08            0C:42:A1:3E:00:08  default          10.141.0.8       internalnet      [  DOWN  ]

What to look at: Check the Category column first: dgx05 to dgx08 are still in default, so they would boot the default image. A MAC that doesn't match the hardware means the node installer can't identify the node.

[bcm-head01->device[dgx05]]% showAll properties of one node

Parameter                        Value
-------------------------------- ------------------------------------------------
Activation                       Mon, 09 Nov 2026 14:02:11 CST
BMC settings                     <submode>
Category                         default
Disk setup                       <inherited from category>
Hostname                         dgx05
Install mode                     AUTO (default)
Interfaces                       <3 in submode>
Ip                               10.141.0.5
MAC                              0C:42:A1:3E:00:05
Network                          internalnet
Next install mode
Power control                    ipmi0
Provisioning interface           BOOTIF
Roles                            <0 in submode>
Software image                   default-image (default)
Type                             PhysicalNode
...

What to look at: Values with the category name in parentheses are inherited; a plain value with no parentheses is a node-level override. Here Category is still default, so Software image resolves to default-image instead of dgx-os-6.

cmsh -c "category; list"Categories and their images

Name (key)               Software image           Nodes
------------------------ ------------------------ --------
default                  default-image            0
dgx-h100                 dgx-os-6                 8

What to look at: Every DGX should be counted under dgx-h100 with software image dgx-os-6. Nodes still counted under default, or a category pointing at a test image, explain nodes that boot the wrong OS.

cmsh -c "configurationoverlay; list"Overlays, priorities, and roles

Name (key)           Priority   All head nodes Nodes                  Categories             Roles
-------------------- ---------- -------------- ---------------------- ---------------------- ----------------
slurm-accounting     500        no             bcm-head01                                    slurmaccounting
slurm-client-gpu     450        no                                    dgx-h100               slurmclient
slurm-server         500        yes                                                          slurmserver
slurm-submit         500        yes            bcm-head01,bcm-head02                         slurmsubmit

What to look at: Check that slurm-client-gpu lists every GPU category; a missing category, such as dgx-b200, means those nodes get no slurmclient role and never appear in sinfo. A priority of -1 means the overlay is ignored.

cmsh -c "device; use dgx01; interfaces; list"A node's network interfaces

Type         Network device name    IP               Network
------------ ---------------------- ---------------- ----------------
bmc          ipmi0                  10.148.0.1       ipminet
physical     BOOTIF [prov]          10.141.0.1       internalnet
physical     ib0                    10.149.0.11      ibnet

What to look at: BOOTIF marked [prov] is the provisioning interface, and its MAC is the node's identity. Missing ipmi0 means no power control; a missing ib interface means no IPoIB address on the compute fabric.

cmsh -c "softwareimage; list"Software images on the head node

Name (key)           Path                             Kernel version           Nodes
-------------------- -------------------------------- ------------------------ --------
default-image        /cm/images/default-image         5.15.0-119-generic       0
dgx-os-6             /cm/images/dgx-os-6              5.15.0-1063-nvidia       7
dgx-os-6-test        /cm/images/dgx-os-6-test         5.15.0-1063-nvidia       1

What to look at: Check the Nodes count: here one node is still on the test image, which usually means a forgotten node-level softwareimage override. The kernel version should be an -nvidia kernel for DGX nodes.

cm-chroot-sw-img /cm/images/dgx-os-6-testEnter a software image as chroot

mounted /cm/images/dgx-os-6-test/dev
mounted /cm/images/dgx-os-6-test/dev/pts
mounted /cm/images/dgx-os-6-test/proc
mounted /cm/images/dgx-os-6-test/sys
mounted /cm/images/dgx-os-6-test/run

Type "exit" to exit the chroot.
root@bcm-head01:/# apt install openssh-server
...
Setting up openssh-server (1:8.9p1-3ubuntu0.10) ...
invoke-rc.d: policy-rc.d denied execution of restart.
root@bcm-head01:/# exit
umounted /cm/images/dgx-os-6-test/run
...

What to look at: The mount lines prove you are inside the image, not on the head node's own OS. The policy-rc.d denied message is normal in a chroot; installing without the chroot would only change the head node.

[bcm-head01->device[dgx01]]% imageupdateDry run of image-to-node sync

Performing dry run (use synclog command to review result, then pass -w to perform real update)...
Tue Nov 10 19:10:12 2026 [notice] bcm-head01: Provisioning started: sending bcm-head01:/cm/images/dgx-os-6-test to dgx01:/, mode UPDATE, dry run = yes, no data changes!
Tue Nov 10 19:10:31 2026 [notice] bcm-head01: Provisioning completed: sent bcm-head01:/cm/images/dgx-os-6-test to dgx01:/, mode UPDATE, dry run = yes, no data changes!
imageupdate [ COMPLETED ]

What to look at: "dry run = yes" means nothing changed on the node. If an update finishes in seconds and nodes are unchanged, this line shows you forgot -w.

[bcm-head01->device[dgx01]]% synclogWhat the dry run would change

Tue Nov 10 19:10:12 2026 - Starting rsync daemon based provisioning. Mode is UPDATE.
...
>f.st...... usr/sbin/sshd
>f.st...... usr/lib/openssh/sftp-server
>f..t...... usr/share/doc/openssh-server/changelog.Debian.gz
>f.st...... var/lib/dpkg/status
*deleting   var/cache/apt/archives/partial/
...
sent 2.41M bytes  received 88.17K bytes  2.50M bytes/sec
total size is 21.67G  speedup is 8,690.31 (DRY RUN)
Tue Nov 10 19:10:31 2026 - Rsync completed.

What to look at: Lines starting with >f are files that would be updated and *deleting are files that would be removed from the node. An unexpected long list of deletions (for example under /raid or /home) means the exclude lists are wrong.

[bcm-head01->device[dgx01]]% imageupdate -wReal sync to a running node

Tue Nov 10 19:12:03 2026 [notice] bcm-head01: Provisioning started: sending bcm-head01:/cm/images/dgx-os-6-test to dgx01:/, mode UPDATE, dry run = no
Tue Nov 10 19:13:41 2026 [notice] bcm-head01: Provisioning completed: sent bcm-head01:/cm/images/dgx-os-6-test to dgx01:/, mode UPDATE, dry run = no
imageupdate -w [ COMPLETED ]

What to look at: "dry run = no" with COMPLETED means the files are on the node without a reboot. A new kernel or driver in the image still needs a reboot to take effect.

tail -f /var/log/node-installerHead-node log of node installs

Nov 10 19:20:02 bcm-head01 node-installer[2201]: dgx07: Node-installer started (version 10.24.x)
Nov 10 19:20:05 bcm-head01 node-installer[2201]: dgx07: Node identified by MAC 0C:42:A1:3E:00:07 on BOOTIF
Nov 10 19:20:06 bcm-head01 node-installer[2201]: dgx07: Category dgx-h100, software image dgx-os-6
Nov 10 19:20:06 bcm-head01 node-installer[2201]: dgx07: Installmode is: FULL (nextinstallmode)
Nov 10 19:20:09 bcm-head01 node-installer[2201]: dgx07: Setting up disks, all data on local disks will be erased
Nov 10 19:20:41 bcm-head01 node-installer[2201]: dgx07: Mounting disks
Nov 10 19:20:42 bcm-head01 node-installer[2201]: dgx07: Synchronizing /cm/images/dgx-os-6 to local disk (FULL)
Nov 10 19:31:58 bcm-head01 node-installer[2201]: dgx07: Rsync completed
Nov 10 19:32:03 bcm-head01 node-installer[2201]: dgx07: Calling init, switching to local root

What to look at: Check the Installmode line: FULL means the disks were repartitioned and local data was erased, while AUTO normally just syncs differences. A node that never gets past identification usually has the wrong MAC registered.

cmsh -c "device; status"UP/DOWN/INSTALLING per node

bcm-head01 ............... [   UP   ]
bcm-head02 ............... [   UP   ]
dgx01 .................... [   UP   ]
dgx02 .................... [   UP   ]
dgx03 .................... [   UP   ]
dgx04 .................... [   UP   ]
dgx05 .................... [   UP   ]
dgx06 .................... [   UP   ]
dgx07 .................... [ INSTALLING ] (node installer started)
dgx08 .................... [  DOWN  ]

What to look at: Every node should end up UP. INSTALLING is normal during provisioning but should not last; DOWN after a power-on points to PXE, BMC, or cabling problems.

cmha statusHead-node HA health

Node Status: running in active mode

bcm-head01* -> bcm-head02
  failoverping  [  OK  ]
  mysql         [  OK  ]
  ping          [  OK  ]
  status        [  OK  ]

bcm-head02 -> bcm-head01*
  failoverping  [  OK  ]
  mysql         [  OK  ]
  ping          [  OK  ]
  status        [  OK  ]

What to look at: The * marks the active head, and all eight checks should be OK. A FAILED mysql means a broken database replica (fix with cmha dbreclone). All OK does not prove fencing works, so check power status as well.

cmha makeactiveManual failover, on the passive head

root@bcm-head02:~# cmha makeactive
Proceeding will initiate a failover sequence which will make this node
(bcm-head02) the active master.

Are you sure ? [Y/N]
Y
Your session ended because: CMDaemon failover, no longer master
bcm-head02 became active master, reconnecting your cmsh ...

What to look at: Run it on the head that should become active; on the already active head it refuses. Afterwards, cmha status should show the * on bcm-head02.

cmsh -c "device; power status -n bcm-head01,bcm-head02"Prove fencing (power control) works

ipmi0 .................... [   ON    ] bcm-head02
none ..................... [ FAILED  ] bcm-head01 (no power control configured)

What to look at: Both heads must show ON or OFF through a real power control such as ipmi0. A FAILED head can't be fenced, so the survivor refuses to take over during a crash, to avoid split brain.

scontrol show node dgx01Full Slurm view of one node

NodeName=dgx01 Arch=x86_64 CoresPerSocket=56
   CPUAlloc=0 CPUEfctv=224 CPUTot=224 CPULoad=0.31
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=gpu:8
   NodeAddr=dgx01 NodeHostName=dgx01 Version=23.02.7
   OS=Linux 5.15.0-1063-nvidia #69-Ubuntu SMP Fri Sep 27 14:44:28 UTC 2026
   RealMemory=2063000 AllocMem=0 FreeMem=2011832 Sockets=2 Boards=1
   State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=defq
   BootTime=2026-11-10T08:12:44 SlurmdStartTime=2026-11-10T08:14:02
   LastBusyTime=2026-11-10T17:30:15 ResumeAfterTime=None
   CfgTRES=cpu=224,mem=2063000M,billing=224,gres/gpu=8
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 AveWatts=0

What to look at: Gres should read gpu:8 on a whole-GPU DGX H100; Gres=(null) means the slurmclient role defines no GPUs. A Reason= line appears only when the node is drained or down, and tells you why.

sinfoSlurm partitions and node states

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite      7   idle dgx[01-03,05-08]
defq*        up   infinite      1   drng dgx04

What to look at: Healthy nodes show idle (or alloc/mix when running jobs). drain or drng means an admin or Slurm took the node out of service, down* means slurmd isn't responding, and a node missing from the list has no slurmclient role.

srun -N2 --gpus-per-node=8 nvidia-smi -LProve a 2-node GPU job runs

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-3f9a1c2e-0b4d-4e71-9a55-1c0d2e3f4a01)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-3f9a1c2e-0b4d-4e71-9a55-1c0d2e3f4a02)
...
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-3f9a1c2e-0b4d-4e71-9a55-1c0d2e3f4a08)
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-7c2e4b10-5a6f-4d3c-8e21-9b0a1f2e3d01)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-7c2e4b10-5a6f-4d3c-8e21-9b0a1f2e3d02)
...
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-7c2e4b10-5a6f-4d3c-8e21-9b0a1f2e3d08)

What to look at: Expect 16 lines, eight per node, with two distinct UUID sets. Fewer GPUs means a Gres or cgroup problem, and an error like "Requested node configuration is not available" means the nodes lack GRES in Slurm.

nvidia-smi -LMIG devices on a node

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5b1e2a3c-4d5e-6f70-8192-a3b4c5d6e7f0)
  MIG 1g.10gb     Device  0: (UUID: MIG-1a2b3c4d-5e6f-5a7b-8c9d-0e1f2a3b4c50)
  MIG 1g.10gb     Device  1: (UUID: MIG-1a2b3c4d-5e6f-5a7b-8c9d-0e1f2a3b4c51)
  ...
  MIG 1g.10gb     Device  6: (UUID: MIG-1a2b3c4d-5e6f-5a7b-8c9d-0e1f2a3b4c56)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-6c2f3b4d-5e6f-7081-92a3-b4c5d6e7f801)
  MIG 1g.10gb     Device  0: (UUID: MIG-2b3c4d5e-6f70-5b8c-9d0e-1f2a3b4c5d60)
  ...
  MIG 1g.10gb     Device  6: (UUID: MIG-2b3c4d5e-6f70-5b8c-9d0e-1f2a3b4c5d66)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-7d304c5e-6f70-8192-a3b4-c5d6e7f80912)
...
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-ce8593a5-b6c7-d8e9-f0a1-b2c3d4e5f607)

What to look at: Count the MIG lines: seven under GPU 0 and GPU 1 plus six whole GPUs is 20 devices, which is the GRES count Slurm needs. If MIG lines are missing after a reboot, the layout was applied with --profile and never stored in gpusettings.

13. Hands-on lab: simulated tickets

Five tickets on a simulated BCM cluster: two head nodes, DGX H100 nodes dgx01–dgx04, and whatever the ticket adds. Each one practices a part of objectives 3.1–3.3:

ONB-101: move new nodes into a category with commit, provision them, read the node-installer log, see them join Slurm.
IMG-202: patch a software image in a chroot, dry-run and push imageupdate, and find the hidden node override.
SLM-303: fix Slurm membership through a configuration overlay, without breaking other nodes, and beat a node-level role.
HA-404: verify head-node HA properly: health, fencing, failover, failback.
MIG-505: drain a node, enable MIG on two GPUs, store the layout so it survives a reboot, and give Slurm the new GPU count through an overlay.

Outputs are simplified mock-ups modeled on how BCM behaves, not exact copies, and every name, address, and version is a placeholder. cmsh works interactively (type cmsh, then quit) or with cmsh -c "mode; cmd; cmd". The lab's commit without a selected object commits every changed object in the mode; on a real cluster, check help commit for your version or commit inside foreach. Up/Down recall history; Tab completes.

Ticket

Ticket checklist

Answer key for this ticket (spoilers)

One clean path; other orders work too. The checklist tracks results, not exact commands.

14. Scenarios

15. Searchable reference

Item	Meaning

16. Flashcards (click to flip)

17. Exam traps in one list

No commit, no change. A * in the cmsh prompt means staged, not saved. cmsh -c needs commit in the string.
Change the image or category, not the node. Hand edits on a node vanish at the next sync.
imageupdate is a dry run until -w. Read synclog in between.
grabimage is the reverse: node → image.
Use cm-chroot-sw-img, not the head node's own apt, to install into an image.
Node overrides hide. One node that ignores category changes usually has its own value: show, then clear.
Priorities: category 250 < overlay 500 (default) < node 750. −1 disables an overlay. slurm-client-gpu is 450.
set replaces a list; append adds. Replacing an overlay's categories drops nodes out of Slurm.
AUTO = sync differences (FULL if the layout doesn't match). FULL erases local disks. nextinstallmode lasts one boot.
NOSYNC doesn't protect data; MAIN stops in maintenance without touching the disk.
Node identity is the boot MAC (or switch port). Update it after a motherboard swap.
HA needs shared IPs, shared storage (NAS/NFS), database replication, and power control on both heads.
No fencing, no automatic failover. The passive head won't take over if it can't power off the active one. Quorum: more than half the nodes must agree.
cmha makeactive runs on the passive head. cmha dbreclone fixes a broken database replica.
A makeactive test isn't a full HA test: it skips fencing. Verify power control and fail back too.
GPU GRES comes from the slurmclient role's gpuautodetect; check scontrol show node | grep Gres.
Enroot and Pyxis live in the compute image; Pyxis adds --container-image to srun.
MIG persists only from gpusettings (set migprofiles, commit, mig apply). mig apply --profile is lost at reboot.
MIG needs idle GPUs; H100 needs no reboot to enable it, A100 does. MIG devices change Slurm's GPU count: set it in an overlay for that node.

18. Practice quiz (28 questions)

Score: 0 / 0