Cluster Bring-Up and Validation

From pallets on the data-center floor to "ready for users": every phase, every test, what each test proves, and how to troubleshoot when something fails. Organized against the official NCP-AII exam blueprint: every objective has its own section, and a tracker shows your coverage weighted by domain.

NCP-AII exam blueprint: what's tested and where to study it

This guide is organized around the official NVIDIA-Certified Professional: AI Infrastructure (NCP-AII) exam blueprint. The exam checks that you can deploy, configure, and validate NVIDIA AI infrastructure: GPU servers, networking, storage, and the cluster software that runs on them. Responsibilities listed on the page are infrastructure deployment, platform and cluster management, testing and performance validation, troubleshooting and optimization, and monitoring, security, and scalability.

Objective tracker

Every official objective, with the sections that cover it. Tick objectives as you study them. Progress is saved in this browser only.

Context: what an "AI factory" is

An AI factory is NVIDIA's name for a data center built to produce AI: trained models and inference tokens, measured by output such as time-to-train and tokens per second. It's a full stack (facility, compute, fabrics, storage, management, software) built from validated reference architectures like DGX BasePOD and DGX SuperPOD, not a product you install. This guide covers the build and validate (Day 1) stage of its lifecycle.

The design side lives in the AI factory architecture guide: BasePOD vs SuperPOD vs Enterprise RAs, scalable units, the four networks, rail-optimized topology, sizing math, and the lifecycle and productivity metrics.

1. The big picture: the 3 golden rules

2. The layer model: what can break, and what tests it

3. The four networks of an AI cluster 1.2

Many bring-up questions test whether you know which network is involved. A DGX SuperPOD has four separate fabrics: compute (GPU-to-GPU, NCCL), storage (datasets and checkpoints), in-band management (SSH, PXE, Slurm, NFS home directories), and out-of-band (BMCs). Hardware per network, BasePOD and GB200 differences, and a traffic-mapping exercise are in AI factory architecture: the four networks.

0Physical install, power, cooling, cabling 1.11.51.8

• Site readiness: rack power capacity (a DGX H100 draws about 10.2 kW at maximum), cooling, floor loading, PDUs, redundant power feeds.
• Racking and power-on: confirm every PSU is green and no fans have failed.
• Cabling against the cabling plan: each DGX's compute ports go to the correct leaf switch (rail-optimized: port/rail i on every node goes to leaf i), plus the storage, in-band, and BMC cables.
• Label everything. Most "mysterious" fabric problems later turn out to be swapped or bad cables.

1Firmware and BMC / out-of-band 1.31.4

• Configure BMC IP addresses and credentials on the out-of-band network. Test remote power and console with ipmitool or Redfish.
• Bring all firmware to one validated set: BIOS/UEFI, BMC, GPU VBIOS, NVSwitch, ConnectX NIC firmware, and SSD firmware. On DGX, NVIDIA provides a firmware update container/tool. Mismatched firmware between nodes causes inconsistent, hard-to-debug behavior.
• BIOS settings: boot order (PXE on the provisioning NIC first for BCM-managed nodes), and performance settings recommended by NVIDIA.

ipmitool -I lanplus -H <bmc-ip> -U admin -P *** chassis power status
ipmitool -I lanplus -H <bmc-ip> -U admin -P *** sdr          # sensors: temps, fans, PSUs
nvsm show health                                              # DGX: overall system health
nvsm show firmware                                            # DGX: component firmware versions (if available)

2Head node, BCM, provisioning 3.13.23.3

• Install Base Command Manager on the head node (usually two head nodes for high availability).
• Define networks, software images (DGX OS based), and categories (for example dgx-h100). Register nodes by MAC address or switch port.
• Nodes PXE boot from the head node, the node installer syncs the image to local disk, and the node boots.
• The centralized model applies: change the image or category, not individual nodes (see the earlier BCM guides).

cmsh -c "device; list"                    # all nodes
cmsh -c "device; status"                  # UP / DOWN / installing
cmsh -c "softwareimage; list"
cmsh -c "category; list"

2b. bcm-pod-setup: one-time BasePOD configuration

After BCM is installed on the head node, NVIDIA provides bcm-pod-setup to apply the DGX BasePOD / SuperPOD reference configuration in one pass, instead of configuring everything by hand in cmsh. It sets up the cluster-wide basics: networks (internal/provisioning, InfiniBand compute, IPMI/BMC), DGX node categories and software images, kernel parameters, and related settings.

3Software stack on compute nodes 3.43.5

Turning each feature on: install, enable, prove

Exam questions often give a feature and ask for the missing step, or describe a step and ask what's still missing. Each row is the minimum sequence; the linked guide has the detail. On BCM, packages and modprobe files go in the software image and kernel parameters go in the category.

4InfiniBand fabric bring-up 1.84.44.5

The Subnet Manager (SM)

An InfiniBand fabric needs a Subnet Manager to assign addresses (LIDs) and compute routes. Until an SM runs, ports stay in state Initializing and no traffic flows. In NVIDIA clusters the SM is usually run by UFM (Unified Fabric Manager) or on the switches. OpenSM is the open-source option.

Checks

ibstat                       # per port: State Active? Physical state LinkUp? Rate 400?
ibstatus                     # compact version
ibdev2netdev                 # map mlx5_X devices to interfaces and Up/Down
sminfo                       # is a subnet manager present?
iblinkinfo                   # every link in the fabric with speed/width
ibnetdiscover                # discover topology (compare with cabling plan)
ibdiagnet                    # full fabric diagnostic: errors, speeds, topology, bad links
ibqueryerrors                # ports with error counters above threshold
mlxlink -d mlx5_0            # deep link diagnostics: BER, cable info (needs mst)

5Single-node validation (GPU health) 1.61.71.104.1

5a. Quick sanity with nvidia-smi

nvidia-smi -L                               # expect 8 GPUs on a DGX H100
nvidia-smi --query-gpu=index,name,driver_version,persistence_mode,ecc.mode.current,mig.mode.current,\
pstate,temperature.gpu,power.limit --format=csv
nvidia-smi -q -d ECC,ROW_REMAPPER           # no uncorrectable errors, no pending remaps
nvidia-smi topo -m                          # GPUs NV18 to each other; each GPU PIX to its NIC
nvidia-smi nvlink -s                        # all links up
dmesg | grep -i xid                         # no XID errors

5b. DCGM diagnostics: the main per-node hardware test

dcgmi discovery -l                          # DCGM sees all GPUs (and NVSwitches)
dcgmi diag -r 3                             # long diag: stress, memory, PCIe/NVLink bandwidth, power
dcgmi diag -r 4                             # extended, for burn-in / acceptance
dcgmi diag -r 3 -j > /tmp/$(hostname)-diag.json    # save results as evidence

Run these on every node (Slurm or pdsh can run them in parallel across nodes). Collect the results.

5c. DGX-specific health

nvsm show health                            # DGX System Management: overall health
nvsm dump health                            # collect a health/log bundle (for NVIDIA support)

Full coverage of NVSM commands, alerts, the object tree, and how NVSM compares to DCGM and the BMC: the NVSM guide.

6Single-node NCCL (NVLink) 4.3

For raw NVLink and PCIe copy bandwidth per GPU pair (nvbandwidth), link status, and NVLink errors, see the NVLink guide.

# One process, all 8 GPUs, NVLink/NVSwitch only
./all_reduce_perf -b 8 -e 8G -f 2 -g 8
./all_gather_perf -b 8 -e 8G -f 2 -g 8
./reduce_scatter_perf -b 8 -e 8G -f 2 -g 8

This uses no network at all, so it isolates NVLink, NVSwitch, Fabric Manager, and GPU P2P.

7Raw network tests (perftest) 4.44.10

Before involving NCCL, test the InfiniBand links themselves with the perftest tools. These measure the network directly, with no GPU or NCCL in the way.

# On node A (server side), for HCA mlx5_0
ib_write_bw -d mlx5_0 --report_gbits
# On node B (client side), pointing at node A
ib_write_bw -d mlx5_0 --report_gbits nodeA

# Latency
ib_write_lat -d mlx5_0 nodeA

# With GPU memory (GPUDirect RDMA path), if perftest was built with CUDA
ib_write_bw -d mlx5_0 --use_cuda=0 --report_gbits nodeA

Repeat for each rail (mlx5_0…mlx5_7) and across many node pairs. Scripts or vendor tools such as NVIDIA ClusterKit (part of HPC-X) can run all-pairs tests automatically. See the ClusterKit guide.

8Multi-node NCCL (the fabric end-to-end) 4.104.11

Now test the full communication path: GPU → NVLink → NIC → InfiniBand → NIC → GPU, through NCCL, exactly as training jobs use it.

Scale up step by step

1. Pairs of nodes (2 × 8 = 16 GPUs): test many different pairs.
2. Groups: a rack, or one scalable unit (SU).
3. The full cluster.

# 2 nodes, Open MPI style
mpirun -np 16 -N 8 --host nodeA:8,nodeB:8 \
   -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH \
   ./all_reduce_perf -b 8 -e 8G -f 2 -g 1

# Full cluster under Slurm (32 nodes)
srun -N 32 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
   ./all_reduce_perf -b 8 -e 16G -f 2 -g 1

# Also test other collectives used by real workloads
srun ... ./all_gather_perf ...  ;  srun ... ./alltoall_perf ...

Check the logs

• NET/IB : Using [0]mlx5_0 … [7]mlx5_7: all 8 compute HCAs
• via NET/IB/x/GDRDMA: GPUDirect RDMA in use
• Not NET/Socket, and not only mlx5_0
• Init COMPLETE on all ranks

9Storage validation 1.114.145.5

df -h ; mount | grep -E "lustre|nfs|gpfs|wekafs"     # filesystems mounted everywhere?
fio --name=seqread --rw=read --bs=1M --size=10G --numjobs=8 --direct=1 --directory=/mnt/data
fio --name=seqwrite --rw=write --bs=1M --size=10G --numjobs=8 --direct=1 --directory=/mnt/data
/usr/local/cuda/gds/tools/gdscheck -p                  # GPUDirect Storage support/config
/usr/local/cuda/gds/tools/gdsio -f /mnt/data/f -d 0 -w 8 -s 10G -i 1M -x 0 -I 0    # GDS read test

• Test one node, then many nodes at once (aggregate throughput is what training at scale needs).
• Check checkpoint write speed. Large models write huge checkpoints.
• Local NVMe RAID (/raid on DGX) works as a fast cache. Test it too.

10Scheduler and containers 3.33.53.6

sinfo                                   # all nodes idle (not down / drain)
scontrol show node dgx01                # GPUs (Gres=gpu:8), state, reason
srun -N 32 hostname                     # can a job reach every node?
srun -N 2 --gpus-per-node=8 nvidia-smi -L
munge -n | ssh dgx01 unmunge            # munge auth works across nodes (same key, synced time)
srun --container-image=nvcr.io#nvidia/pytorch:24.05-py3 -N 2 --gpus-per-node=8 \
     python -c "import torch; print(torch.cuda.device_count())"     # pyxis/enroot container test

• Verify the GRES GPU config (gres.conf) and cgroup GPU isolation.
• Set up a prolog/epilog with quick health checks (dcgmi diag -r 1 or dcgmi health -c) that drain a node if the check fails.
• Make sure containers can see InfiniBand devices (multi-node NCCL inside containers).

Kubernetes and NVIDIA Run:ai (the other scheduler path)

Slurm is the classic choice for training clusters, but many AI factories also (or instead) run Kubernetes, especially for inference, interactive notebooks, and mixed teams. NVIDIA Run:ai (NVIDIA acquired Run:ai in 2024) is a GPU orchestration layer that runs on top of Kubernetes. It adds the things plain Kubernetes is weak at for AI: team quotas, fair sharing of idle GPUs, queueing, gang scheduling of distributed jobs, and fractional GPUs.

The stack, bottom to top:

1. Kubernetes on the GPU nodes (BCM can deploy Kubernetes as well as Slurm).
2. NVIDIA GPU Operator: installs and manages the driver (or uses the host's), container toolkit, device plugin (which advertises nvidia.com/gpu), MIG manager, and DCGM exporter on every GPU node.
3. NVIDIA Network Operator (for multi-node training): exposes InfiniBand / RoCE devices to pods so NCCL can use RDMA.
4. Run:ai: a cluster component (installed with Helm) that connects to the Run:ai control plane (hosted or self-hosted). Admins create departments and projects with GPU quotas; users submit work through the UI, the runai CLI, or the API.

# Validate the Kubernetes GPU layer before Run:ai
kubectl get pods -n gpu-operator                 # all GPU Operator pods Running / Completed
kubectl describe node dgx01 | grep nvidia.com/gpu   # node advertises 8 GPUs (Capacity / Allocatable)

# One-GPU smoke test pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: { name: gpu-smoke }
spec:
  restartPolicy: Never
  containers:
  - name: cuda
    image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
    command: ["nvidia-smi"]
    resources: { limits: { nvidia.com/gpu: 1 } }
EOF
kubectl logs gpu-smoke                           # should print one GPU

Then validate Run:ai itself: the cluster shows as connected in the Run:ai UI, nodes and GPUs appear in its inventory, a test project with a quota can run a 1-GPU job, a distributed multi-node job starts all its workers together and gets RDMA (check NCCL_DEBUG=INFO shows NET/IB), and an over-quota job is preempted when the quota owner submits work.

Full detail: the dedicated NVIDIA Run:ai guide (quotas, reclaim, gang scheduling, fractions vs MIG, CLI, troubleshooting).

11Application benchmarks and burn-in 4.24.124.13

# Example idea: HPL in NVIDIA's HPC-Benchmarks container across 4 nodes
srun -N 4 --ntasks-per-node=8 --gpus-per-node=8 \
     --container-image=nvcr.io#nvidia/hpc-benchmarks:<tag> \
     ./hpl.sh --dat /path/HPL.dat

12Monitoring, baselines, handover

• Monitoring: DCGM Exporter → Prometheus → Grafana for GPUs, UFM for the IB fabric, BCM monitoring for nodes, and alerting on XIDs, ECC, temperature, and link errors.
• Baselines: save "golden" DCGM results, nccl-tests busbw (single-node and multi-node), perftest numbers, HPL results, and storage throughput.
• Ongoing health: Slurm prolog/epilog checks, periodic DCGM diagnostics on drained nodes, periodic nccl-tests sweeps.
• Documentation: firmware and software versions, topology, cabling map, test results, known issues.

Domain 1 deep dive: System and Server Bring-Up 31%

NVIDIA's summary of this domain: end-to-end hardware deployment, meaning rack, power, BMC, security, firmware, physical installation, initial server and network setup, cable validation, and hardware verification, so that everything is deployed, configured, and working before AI operations scale up.

1.1 The sequence of events for deployment and validation

The exam expects you to know the order and the gate between steps: you don't move on until the current step passes.

1.2 Network topologies for AI factories

This objective is covered in the AI factory architecture guide:

• The four networks: compute, storage, in-band, out-of-band; east-west vs north-south.
• Compute fabric topology: rail-optimized, non-blocking fat tree, PXN, and why a swapped cable slows NCCL.
• Scalable units and sizing math: leaf and spine counts, cables, transceivers.
• GB200 NVL72: a 72-GPU NVLink domain per rack.

For bring-up, the topology turns into one check: cabling must match the plan, port i of every node on leaf i, verified with ibnetdiscover / ibdiagnet before performance tests (phase 4).

1.3 Initial configuration of BMC, OOB, and TPM

BMC and OOB first-time setup

1. Connect each BMC port to the OOB switch (the dedicated BMC port, not a shared host NIC, on GPU servers).
2. Set addressing: DHCP reservations, or static from the host or BIOS setup screen.
3. Change default credentials, create named admin accounts, and set roles.
4. Set time (NTP), hostname / DNS, and a syslog or event destination.
5. Install trusted TLS certificates for Redfish and the web UI; prefer Redfish over IPMI-over-LAN, and disable IPMI-over-LAN if it isn't needed.
6. Test remote power, the remote console (KVM / Serial over LAN), sensors, and the event log.
7. Register BMC addresses and credentials in BCM so it can power-control nodes.

# In-band, from the host OS (first-time network config of the local BMC)
sudo ipmitool lan print 1
sudo ipmitool lan set 1 ipsrc static
sudo ipmitool lan set 1 ipaddr 10.0.100.11
sudo ipmitool lan set 1 netmask 255.255.255.0
sudo ipmitool lan set 1 defgw ipaddr 10.0.100.1
sudo ipmitool user list 1
sudo ipmitool user set password <user-id>           # prompts for the new password

# Out-of-band checks with Redfish
curl -s -u $BMC_USER:$BMC_PASS --cacert bmc-ca.pem https://$BMC/redfish/v1/Managers/<bmc-id>/EthernetInterfaces
curl -s -u $BMC_USER:$BMC_PASS --cacert bmc-ca.pem https://$BMC/redfish/v1/Systems/<system-id> | jq .PowerState

TPM (Trusted Platform Module)

A TPM is a small security chip (TPM 2.0 on current servers) that stores keys and measurements of the boot process. It enables measured boot (each boot stage is hashed into the TPM's registers), attestation (proving to a remote party that the server booted known-good firmware and software), and sealing of keys such as disk-encryption keys so they're only released to a trusted boot.

More depth: Redfish, IPMI & BMC.

1.4 Firmware upgrades (including HGX) and fault detection

# Example: NVIDIA's firmware tool against a DGX/HGX BMC (syntax varies by version: check --help)
nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS show_version
nvfwupd -t ip=$BMC user=$BMC_USER password=$BMC_PASS update_fw -p <package>.fwpkg

# NIC firmware on the host
sudo mst start
sudo mlxfwmanager --query                       # all Mellanox/NVIDIA devices, current vs available FW
sudo mlxfwmanager -u -i <fw-package>.bin       # update (then reset or power cycle to activate)

Best practice

• Use one validated firmware recipe (the versions NVIDIA or the OEM tested together) for every node. Read the release notes for order and dependencies.
• Update a canary node first, validate it, then roll out in batches (drained nodes only).
• Don't interrupt power during an update. Some components only activate after a full AC power cycle or a BMC/host cold reset; do what the release notes say.
• After the update: re-read every version, check the event log, run nvidia-smi, nvsm show health (DGX), and a short DCGM diag.

Fault detection during and after updates

• Redfish task ends in Exception / failed state, or the component's Status.Health is Warning/Critical.
• SEL or Redfish log entries for the component; GPU missing from nvidia-smi; NVSwitch errors; NICs missing from lspci.
• Version mismatch between nodes after a partial rollout: compare inventories and re-apply.
• Recovery: retry the update, reset the BMC (ipmitool mc reset cold), or use the component's recovery / backup image if it has dual-bank firmware.

Deeper: HGX firmware & nvfwupd guide (BMC vs HMC, activation, rollout plan builder) and the networking firmware (MFT) guide (NICs, DPUs, switches).

1.5 Validate power and cooling parameters

• Capacity: a DGX H100 can draw about 10.2 kW at maximum, and has multiple redundant PSUs. Rack PDUs, circuits, and feeds must cover the maximum draw with redundancy (A and B feeds).
• Cooling: front-to-back airflow, hot/cold aisle containment, blanking panels, inlet temperature within the system's operating range. Rack-scale systems such as GB200 NVL72 are liquid cooled: check coolant distribution units (CDUs), flow, temperatures, and leak detection.
• Check under load, not just at idle: HPL and burn-in are also power and cooling tests.

# BMC side
ipmitool -I lanplus -H $BMC -U $BMC_USER -E sdr type "Power Supply"   # every PSU present and OK
ipmitool -I lanplus -H $BMC -U $BMC_USER -E sdr type Temperature      # inlet / exhaust / CPU temps
ipmitool -I lanplus -H $BMC -U $BMC_USER -E sdr type Fan
ipmitool -I lanplus -H $BMC -U $BMC_USER -E dcmi power reading        # current system power

# GPU side
nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCE
nvidia-smi --query-gpu=index,power.draw,power.limit,temperature.gpu,temperature.memory,clocks_event_reasons.active --format=csv -l 5
dcgmi dmon -e 150,140,155 -d 1000                                     # GPU temp, memory temp, power usage

1.6 Install GPU-based servers (SMI)

Here "SMI" refers to the NVIDIA System Management Interface, nvidia-smi: you install the server and then prove, with nvidia-smi, that the OS and driver see the GPUs correctly.

1. Rack the server on the correct rails. GPU servers are very heavy (a DGX H100 is well over 100 kg); use a lift and follow the site's safety rules.
2. Connect power (both feeds), the OOB BMC port, management, compute, and storage networks.
3. Power on, configure the BMC, update firmware, set BIOS settings, install or provision the OS and driver.
4. Verify with nvidia-smi:

nvidia-smi                                   # driver version, all GPUs, no errors
nvidia-smi -L                                # GPU count and UUIDs
nvidia-smi --query-gpu=index,name,pci.bus_id,serial,vbios_version,persistence_mode,ecc.mode.current,mig.mode.current,power.limit --format=csv
nvidia-smi topo -m                           # NVLink between GPUs, GPU↔NIC affinity
nvidia-smi nvlink -s                         # every NVLink active
systemctl is-active nvidia-persistenced nvidia-fabricmanager

More: nvidia-smi deep dive.

1.7 Validate installed hardware

Compare what the OS and BMC see against the bill of materials. Missing or downgraded parts are easier to fix now than after users arrive.

lscpu | grep -E "Model name|Socket|NUMA"          # CPUs and NUMA nodes
free -g ; sudo dmidecode -t memory | grep -c "Size: [0-9]"   # memory total and populated DIMMs
lspci | grep -i nvidia                            # GPUs and NVSwitches
lspci -d 15b3:                                    # NVIDIA networking (ConnectX, BlueField)
sudo lspci -vv -s <bus:dev.fn> | grep -E "LnkCap|LnkSta"   # PCIe link speed/width: LnkSta should match LnkCap
nvme list                                         # NVMe drives, models, firmware
ibstat | grep -E "CA '|State|Rate"                # HCAs, port state, rate
ibdev2netdev                                      # HCA to interface map
sudo nvsm show health                             # DGX: overall health and inventory checks
curl -s -u $BMC_USER:$BMC_PASS --cacert bmc-ca.pem https://$BMC/redfish/v1/Systems/<system-id> | jq '{Model, ProcessorSummary, MemorySummary}'

1.8 Describe and validate cable types and transceivers

• Form factors: OSFP (DGX H100 and Quantum-2 switches use twin-port OSFP, 2 × 400G per cage), QSFP112 / QSFP56. Fiber connectors: MPO (parallel) or LC (duplex).
• 400G NDR uses PAM4 signaling and requires RS-FEC. Healthy links have a low raw (pre-FEC) BER and zero effective (post-FEC) errors.
• Validate: cable type and part number match the plan (mlxlink -m), link at full speed and width (ibstat, iblinkinfo), BER and error counters clean under load (mlxlink -c, ibdiagnet).

Full detail, including a BER calculator: InfiniBand physical layer.

1.9 Install physical GPUs

On DGX and HGX systems the GPUs are SXM modules on a baseboard that's serviced as a unit. On PCIe GPU servers (NVIDIA-Certified Systems from OEMs), you may install cards yourself:

1. Check the server's qualified configuration: which GPU models, how many, in which slots, with which risers and power cables.
2. Power off, remove AC power, use ESD protection.
3. Install in x16 slots (PCIe Gen5 for H100-class), balanced across CPU sockets, ideally on the same PCIe switch or root complex as the NIC each GPU will use.
4. Connect the auxiliary power cable the card needs (for example the 16-pin 12V-2x6 / 12VHPWR connector) fully seated; don't bend it sharply near the connector.
5. Install NVLink bridges on paired cards that support them (for example H100 NVL pairs).
6. Data-center GPUs are passively cooled: the server's fans and airflow (and fan profile) must be the ones specified for GPUs; fill empty slots with blanks.
7. BIOS: enable above-4G decoding and resizable/large BAR support as required; set slot speed to auto/Gen5.
8. Verify: lspci (all cards, full link width and speed), nvidia-smi, nvidia-smi topo -m, then a DCGM diag.

1.10 Validate hardware operation for workloads

"Hardware works" means more than "it's detected". Load it the way workloads will:

See phase 5, gpu-burn & dcgmproftester, and the DCGM deep dive.

1.11 Configure initial parameters for third-party storage

Most AI clusters use partner storage (parallel filesystems or high-performance NFS). Initial configuration on the compute side:

1. Network: the storage fabric NICs, IP addressing, MTU (jumbo frames, 9000, end to end on Ethernet), and RDMA settings (for RoCE: lossless or congestion-controlled configuration on switches and NICs).
2. Client software: install the vendor's client (for example a Lustre client, a GPFS / Storage Scale client, a WEKA client), or use NFS with recommended options (for example nconnect for multiple TCP connections, or NFS over RDMA).
3. Mounts: mount options from the vendor's guide, mounted on every node. In BCM, define the mount in the node category (fsmounts) so every node gets it, instead of editing /etc/fstab by hand.
4. GPUDirect Storage (if planned): IOMMU off/pt and ACS disabled, install nvidia-gds, load nvidia_fs, configure /etc/cufile.json for the storage NICs and filesystem, and check with gdscheck -p that the filesystem shows a real GDS mode, not just compat (full procedure).
5. Test a single node, then many nodes at once (phase 9).

# Examples (vendor guides give the real options)
sudo mount -t nfs -o vers=4.1,nconnect=16,rsize=1048576,wsize=1048576 storage01:/datasets /mnt/datasets
sudo mount -t lustre 10.10.0.10@o2ib:/aifs /mnt/aifs         # Lustre over InfiniBand (LNet o2ib)
lnetctl net show                                              # Lustre networking status

# BCM: add the mount to the category, not to individual nodes
cmsh -c "category use dgx-h100; fsmounts; add /mnt/datasets; set device storage01:/datasets; set filesystem nfs; set mountoptions vers=4.1,nconnect=16; commit"

/usr/local/cuda/gds/tools/gdscheck -p                         # GDS: drivers, filesystem support, config

Deeper: Lustre & third-party storage guide, shared storage, GPUDirect Storage.

Domain 2 deep dive: Physical Layer Management 5%

Configuring and maintaining the physical resources: GPU and BlueField DPU networks, cables and transceivers, and GPU partitioning with MIG.

2.1 Configure and manage a BlueField network platform

BlueField is NVIDIA's DPU (data processing unit): a ConnectX network adapter plus its own Arm CPU cores, memory, and accelerators, running its own Linux OS. BlueField-3 also ships as a SuperNIC for Spectrum-X east-west traffic. On a DGX H100, BlueField-3 DPUs typically serve storage and in-band networking, while ConnectX-7 adapters serve the compute rails.

# On the host
sudo mst start ; sudo mst status -v                  # find the device (for example /dev/mst/mt41692_pciconf0 for BlueField-3)
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q | grep -E "LINK_TYPE|INTERNAL_CPU"   # port types and DPU/NIC mode settings
sudo flint -d /dev/mst/mt41692_pciconf0 q            # firmware version and PSID
systemctl status rshim                               # RShim service
sudo bfb-install --bfb <image>.bfb --rshim rshim0    # install a BFB image onto the DPU

# Reach the DPU's Arm OS
ssh ubuntu@192.168.100.2                             # over tmfifo_net0 (host side is typically 192.168.100.1)
# or the RShim console, or the DPU's OOB management port

# On the DPU (Arm)
cat /etc/mlnx-release                                # installed BFB / DOCA version
sudo ovs-vsctl show                                  # DPU-mode switching (OVS bridges and ports)

2.2 Configure MIG (AI and HPC)

MIG (Multi-Instance GPU) splits one A100/H100-class GPU into up to 7 isolated instances, each with its own SMs, memory, and cache.

sudo nvidia-smi -i 0 -mig 1                  # enable MIG mode on GPU 0 (GPU must be idle; may need a GPU reset)
nvidia-smi mig -lgip                         # list GPU instance profiles (for example 1g.10gb, 3g.40gb, 7g.80gb on H100 80GB)
sudo nvidia-smi mig -cgi 9,9 -C              # create two GPU instances (profile ID 9 = 3g.40gb on H100 80GB) plus compute instances
nvidia-smi mig -lgi ; nvidia-smi mig -lci    # list GPU / compute instances
nvidia-smi -L                                # MIG devices now listed with their own UUIDs
sudo nvidia-smi mig -dci ; sudo nvidia-smi mig -dgi   # delete compute, then GPU instances
sudo nvidia-smi -i 0 -mig 0                  # disable MIG

• Profile IDs differ by GPU model; always list them with -lgip first.
• MIG configurations aren't automatically recreated at boot. Use a declarative tool (for example nvidia-mig-parted, the Kubernetes GPU Operator's MIG manager, or BCM configuration) to make them persistent.
• Slurm: GRES can auto-detect MIG devices (AutoDetect=nvml in gres.conf). Kubernetes: the GPU Operator exposes MIG with a single or mixed strategy.
• Inside MIG instances there's no NVLink P2P between instances, so MIG is a poor fit for multi-GPU training.

Full command reference: nvidia-smi MIG.

Domain 3 deep dive: Control Plane Installation and Configuration 19%

Installing and configuring operating systems, cluster managers, drivers, container tools, and management software.

3.1 Install BCM, configure, and verify HA

1. Install the head node from the Base Command Manager installer (ISO via BMC virtual media or USB): choose networks (internal/provisioning, external, IPMI/BMC, InfiniBand), disk layout, and admin settings.
2. Install the license (for example request-license, then verify with cmsh -c "main; licenseinfo").
3. Apply the DGX BasePOD/SuperPOD reference configuration once with bcm-pod-setup where applicable.
4. Configure high availability: add a second head node and run the HA setup wizard (cmha-setup). HA uses a shared virtual IP that moves to the active head node, replicates the BCM database, and needs shared storage for directories both head nodes serve (for example /cm/shared and /home).
5. Verify HA: cmha status shows both head nodes healthy; perform a controlled failover (for example cmha makeactive on the passive node) and check that provisioning, cmsh, and Slurm control keep working.

cmha status                         # HA state: active / passive, checks OK?
cmha makeactive                     # (run on the passive head node) take over as active
cmsh -c "device; status"            # nodes still UP after failover
cmsh -c "main; licenseinfo"

3.2 Install the OS

cmsh -c "softwareimage; list"
cmsh -c "softwareimage; clone dgx-os-image dgx-os-image-test; commit"   # test changes on a copy
cm-chroot-sw-img /cm/images/dgx-os-image-test                         # enter the image to install packages

• After install: apply updates, pin (hold) the kernel version your driver and OFED/DOCA builds support, blacklist nouveau, set time sync.
• Keep OS, kernel, driver, and DOCA versions identical across nodes (BCM images make this automatic).

DGX OS: releases and install paths

The major number follows the Ubuntu release. Minor releases (7.1, 7.2…) add backward-compatible NVIDIA features and fixes. Upgrades are cumulative: apt always installs the latest packages in the repositories, which can be newer than the latest ISO, and you can't stop at a chosen minor version. The GPU driver stays on its current branch until that branch reaches end of support.

Reimaging from the ISO

1. Get the ISO from the NVIDIA Enterprise Support Download Center (needs a support account) and check it: md5sum DGXOS-7.x.x-<date>-amd64.iso against the value in the release announcement.
2. Boot it:
   • Remotely through the BMC: Remote Control → Launch KVM → Browse File (the ISO) → Start Media, then Power → Hard Reset.
   • USB: at least 16 GB. Write it as a raw image: sudo dd if=<iso> bs=2048 of=/dev/sdX (check the device with lsblk first; dd erases it). On Windows, Rufus in DD Image mode. A plain file copy won't boot. Press F11 at the NVIDIA logo for the boot menu.
3. Pick a GRUB option (table below). The install runs unattended for about 15–60 minutes. It also updates the ConnectX firmware (up to 5 minutes per card); no other system firmware is updated.
4. The system reboots into the first-boot wizard.

First boot (DGX servers)

The wizard runs on a display: a monitor or the BMC's KVM, not a serial console.

1. Accept the EULA; choose language, keyboard, time zone, and the UTC clock.
2. Create the admin user (lowercase). The same username becomes the BMC and GRUB username: the default BMC admin account is disabled and replaced. The BMC won't accept sysadmin as a username.
3. Set the BMC password (13–20 characters) and a GRUB password (at least 8; leaving it blank disables GRUB protection). If you chose encrypted root, set its passphrase.
4. Pick the primary network interface for in-band management (eno3 on DGX H100/H200/B200). Don't pick the USB/Redfish interface (bmc_redfish0 or similar). Then DHCP or static, name servers, and hostname.
5. Before the first reboot, wait for the data RAID to finish building: sudo systemctl status nvidia-raid-config until "Finished NVIDIA RAID Configuration" (about 10 minutes).
6. Update packages right away (the ISO is a snapshot): sudo apt update && sudo apt full-upgrade, then reboot.

Ubuntu + the DGX Software Stack

Install Ubuntu 24.04 Server (OS on the RAID 1 pair, and don't install Docker as a snap: Docker CE comes with the DGX packages), then add NVIDIA's repositories and the DGX metapackages:

curl https://repo.download.nvidia.com/baseos/ubuntu/noble/x86_64/dgx-repo-files.tgz | sudo tar xzf - -C /
sudo apt update && sudo apt upgrade
sudo apt install nvidia-system-core nvidia-system-utils nvidia-system-extra   # configs, tools (NVSM), Docker CE
sudo apt install nvidia-mig-manager nvdebug dgx-server-grub nvidia-grub-params
sudo apt install -y linux-tools-generic nvidia-peermem-loader
sudo apt install nvidia-driver-580-open libnvidia-nscq nvidia-modprobe \
     nvidia-fabricmanager datacenter-gpu-manager-4-cuda13 nv-persistence-mode    # DGX H100/H200
sudo systemctl enable nvidia-fabricmanager nvidia-persistenced nvidia-dcgm
sudo reboot                                                                      # then nvidia-smi: all GPUs, Persistence-M On

Verify an install or upgrade

cat /etc/dgx-release            # DGX_SWBUILD_VERSION = ISO installed; each DGX_OTA_VERSION line = a network upgrade since
uname -r                        # kernel at least the release notes' minimum
nvidia-smi                      # all GPUs, expected driver branch
systemctl status nvidia-fabricmanager nvidia-persistenced nvidia-dcgm
df -h /raid                     # data RAID mounted (unless you kept it unmounted on purpose)
sudo dcgmi diag -r 2            # quick functional check before handing the node back

Upgrading DGX OS

Before any upgrade: read the release notes; drain the node (Slurm/Kubernetes); upgrade one or two systems first and test your applications; use the BMC console rather than SSH if you can (with DHCP the IP can change at reboot, VPNs drop, and Docker's default 172.17.0.0/16 bridge makes a DGX on that subnet unreachable); confirm the node reaches the Ubuntu and NVIDIA repositories (or use a local mirror).

# Package upgrade
sudo apt update
sudo apt full-upgrade -s        # dry run: review what will change
sudo apt full-upgrade           # GRUB prompts: keep the current config and the default install devices
sudo reboot                     # required for a new driver or kernel

# Release upgrade, DGX OS 6 → 7
sudo apt update && sudo apt -y full-upgrade        # to the latest 6.x; reboot if prompted
sudo apt install -y nvidia-release-upgrade
sudoedit /etc/update-manager/release-upgrades.d/nvidia.cfg   # optional: DriverBranch=...
sudo nvidia-release-upgrade                          # sudo -E behind a proxy; reboot when prompted

• Config-file prompts: take the maintainer's /etc/apt/sources.list.d/dgx.list; keep your local /etc/ssh/sshd_config.
• Over SSH, the upgrader asks to continue and starts a spare sshd on port 1022 for recovery.
• Third-party repositories are disabled (Canonical and DGX ones are kept); re-enable them afterwards.
• MLNX_OFED is replaced by DOCA OFED during 6 → 7. Check applications that depend on MOFED.
• Packages with no DGX OS 7 candidate stop the upgrade; the release notes list what was removed.

3.3 Install the cluster: categories, interfaces, Slurm / Enroot / Pyxis

Categories and nodes

cmsh
% category clone default dgx-h100            # one category per node type
% category use dgx-h100
% set softwareimage dgx-os-image
% commit
% device use dgx01
% set category dgx-h100
% commit

Interfaces

% device use dgx01
% interfaces
% add physical ib0                           # an InfiniBand interface (IPoIB) on node dgx01
% set network ibnet
% set ip 10.149.0.11
% commit
% list                                       # check bootif (provisioning), BMC (ipmi0 / rf0), ib interfaces

Node identity: BCM recognizes nodes by MAC address of the boot interface (or by switch port). Replacing a motherboard or NIC changes the MAC, so update the node's record.

Slurm, Enroot, Pyxis

• Slurm: BCM's workload-manager setup wizard (cm-wlm-setup) installs Slurm, assigns the server role to the head node(s) and the client role to the compute category, and generates the configuration, including GPU GRES.
• Enroot turns container images into unprivileged sandboxes; Pyxis is the Slurm SPANK plugin that adds --container-image to srun. Both go into the software image; Pyxis is enabled in Slurm's plugstack.conf.
• NGC credentials for Enroot: ~/.config/enroot/.credentials with machine nvcr.io login $oauthtoken password <NGC API key>.

sinfo ; scontrol show node dgx01 | grep -E "Gres|State"
srun -N 2 --gpus-per-node=8 --container-image=nvcr.io#nvidia/pytorch:<tag> nvidia-smi -L

More: Base Command Manager mastery (cmsh, overlays, images, HA, with a lab), NGC with Enroot and Pyxis, BCM node recovery.

3.4 Install, update, and remove NVIDIA GPU and DOCA drivers

GPU driver

# Install from NVIDIA's repository (Ubuntu example; package names depend on branch and distro)
sudo apt install nvidia-driver-<branch>-server-open nvidia-fabricmanager-<branch>
sudo systemctl enable --now nvidia-fabricmanager nvidia-persistenced

# Update: drain the node, stop GPU users and services, upgrade driver AND Fabric Manager together, reboot
sudo systemctl stop nvidia-dcgm nvidia-fabricmanager nvidia-persistenced
sudo apt install --only-upgrade 'nvidia-*<branch>*'
sudo reboot

# Remove
sudo apt purge 'nvidia-*' 'libnvidia-*' ; sudo apt autoremove     # package installs
sudo /usr/bin/nvidia-uninstall                                     # .run installs

# Verify
nvidia-smi ; cat /proc/driver/nvidia/version ; systemctl is-active nvidia-fabricmanager

• Use one install method per system: mixing a .run installer with packages causes conflicts.
• Fabric Manager must match the driver version exactly on NVSwitch systems. Upgrade them together.
• Hold the packages (apt-mark hold) so a routine OS update doesn't change the driver unexpectedly.
• Open vs proprietary kernel modules: recent data-center GPUs support (and newer drivers default to) the open kernel modules; follow the release notes.

DOCA drivers (networking)

DOCA-Host is the current host package for ConnectX and BlueField drivers and tools, replacing standalone MLNX_OFED. It comes in profiles (for example doca-ofed for the OFED-equivalent drivers and tools).

sudo /usr/sbin/ofed_uninstall.sh              # remove an old MLNX_OFED first, if present
sudo apt install doca-ofed                    # after adding the DOCA repository for your distro
sudo /etc/init.d/openibd restart              # reload the RDMA stack (or reboot)
ofed_info -s ; ibstat ; ibv_devinfo
lsmod | grep -E "mlx5_core|ib_core|nvidia_peermem"
sudo apt remove 'doca-*' ; sudo apt autoremove    # removal

Deep dive with a lab: GPU drivers, DOCA-Host, and the Container Toolkit.

3.5 Install the NVIDIA Container Toolkit

sudo apt install nvidia-container-toolkit      # after adding NVIDIA's container toolkit repository
sudo nvidia-ctk runtime configure --runtime=docker      # writes the nvidia runtime into /etc/docker/daemon.json
sudo systemctl restart docker
sudo nvidia-ctk runtime configure --runtime=containerd  # for containerd (Kubernetes) instead
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml   # Container Device Interface spec (Podman, CDI-aware runtimes)
nvidia-ctk --version

The toolkit doesn't contain a GPU driver: it makes the host's driver and GPUs available inside containers. In Kubernetes, the GPU Operator can install and configure it for you.

3.6 Use NVIDIA GPUs with Docker

docker run --rm --gpus all nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi    # all GPUs
docker run --rm --gpus 2 ...                                                         # any two GPUs
docker run --rm --gpus '"device=0,3"' ...                                            # specific GPUs (index or UUID)
docker run --rm --gpus all -e NVIDIA_VISIBLE_DEVICES=0 ...                           # alternative selection

# Realistic training container on one node, with InfiniBand access for multi-node NCCL
docker run --rm -it --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  --device=/dev/infiniband --cap-add=IPC_LOCK --network=host \
  -v /mnt/datasets:/data nvcr.io/nvidia/pytorch:<tag>

echo "$NGC_API_KEY" | docker login nvcr.io -u '$oauthtoken' --password-stdin           # pull from NGC

3.7 Install the NGC CLI on hosts

# Download the NGC CLI for Linux (from the NGC website / documentation), then:
unzip ngccli_linux.zip
chmod u+x ngc-cli/ngc
echo 'export PATH="$PATH:$HOME/ngc-cli"' >> ~/.bashrc
ngc config set                       # prompts for API key, org, team, output format
ngc config current
ngc registry image list nvidia/pytorch
ngc registry model download-version nvidia/<model>:<version>

The NGC CLI manages NGC resources (images, models, resources, orgs). Pulling containers is still done with Docker, Enroot, or the cluster's runtime. More: NGC CLI.

Domain 4 deep dive: Cluster Test and Verification 33%

Validating cluster health and readiness through stress testing, benchmarking, cable integrity checks, firmware validation, and bandwidth verification, ending with burn-in.

4.1 Single-node stress test

dcgmi diag -r 3                       # hardware diagnostics incl. stress, memory, PCIe/NVLink, power
dcgmi diag -r 4                       # extended (longer) for acceptance and burn-in
./gpu_burn -d 3600                    # 1 hour of sustained compute (double precision with -d)
dcgmproftester12 --no-dcgm-validation -t 1004 -d 300   # tensor-core load (field 1004); name varies with CUDA version
nvidia-smi dmon -s pucvmet            # watch power, utilization, clocks, violations, memory, ECC, temps

See gpu-burn & dcgmproftester and the DCGM deep dive.

4.2 Execute HPL

• Run NVIDIA's HPC-Benchmarks NGC container (hpl.sh with an HPL.dat), single node first, then multi-node.
• Size N to fill most of GPU memory, pick NB for the GPU, and choose P × Q = number of GPUs.
• Pass: residual check PASSED, performance (Rmax) close to the expected fraction of peak, and consistent per node.

Everything else: HPL mastery (sizing calculator, output, tuning).

4.3 Single-node NCCL, including verifying the NVLink Switch

systemctl status nvidia-fabricmanager           # must be active on NVSwitch systems
nvidia-smi topo -m                              # GPU pairs show NV18 (H100 SXM: 18 NVLinks per GPU)
nvidia-smi nvlink -s                            # every link up at the same speed
nvidia-smi nvlink -e                            # NVLink error counters (should not grow)
nvidia-smi -q | grep -i -A3 fabric              # fabric registration state on newer drivers / platforms
./all_reduce_perf -b 8 -e 8G -f 2 -g 8          # single-node NCCL across all 8 GPUs (NVLink/NVSwitch only)
NCCL_DEBUG=INFO ./all_reduce_perf -b 1G -e 1G -g 8 2>&1 | grep -E "via P2P|NVLS|SHM"   # P2P / NVLS in use, not SHM

See phase 6, NVLink & nvbandwidth, Fabric Manager, nccl-tests mastery.

4.4 Validate cables by verifying signal quality

sudo mlxlink -d mlx5_0 -m        # module / cable info: type, vendor, part number, length, temperature, firmware
sudo mlxlink -d mlx5_0 -c        # counters: raw (pre-FEC) BER, effective (post-FEC) BER, symbol errors
sudo mlxlink -d mlx5_0 --pc      # clear counters, then run traffic (perftest / NCCL) and check again
sudo mlxlink -d mlx5_0 -e        # eye opening (signal margin) where supported
ibdiagnet                        # fabric-wide: error counters, link speed/width, cable info, BER warnings
ibqueryerrors                    # ports whose error counters exceed thresholds

Full detail: InfiniBand physical layer, BER, and FEC, and the transceivers, cables & signal quality guide (form factors, eye diagrams, DDM, test duration).

4.5 Confirm cabling is correct

• At scale, use NVIDIA's UFM Cable Validation Tool (CVT). A collector container on the management network deploys agents to every switch (and optionally hosts) and validates each link against the P2P cabling plan (or a .topo/.dot file). It reports Wrong-neighbor, Wrong-port, Extra-cable, Media unplugged, flapping, BER, and power/temperature anomalies, and separates non-cable issues (AdminDown, negotiation fail, unreachable device). Details: CVT section.
• Discover the real topology (ibnetdiscover, iblinkinfo, UFM topology view) and compare it with the cabling plan: every node port connects to the planned switch and port. Good for spot checks and small fabrics.
• Rail check: on every node, HCA i (mlx5_i) must connect to leaf / rail i. A swapped pair still links up, but breaks rail alignment and slows NCCL.
• Ethernet networks: use LLDP neighbor information (on hosts and switches) to confirm which switch port each NIC is plugged into.
• Count links: every expected link present, none missing between leaf and spine (missing uplinks show up only at scale).

iblinkinfo | grep dgx05                 # which switch ports node dgx05's HCAs reach
ibnetdiscover > fabric.topo             # save, then diff against the expected topology
ibdiagnet                               # reports topology, speeds, and mismatches; can compare with a reference topology (options vary)
sudo lldptool -t -n -i eth0             # Ethernet: neighbor switch and port (or lldpctl)

4.6 Confirm firmware and software on switches

ibswitches                              # all IB switches in the fabric (GUID, LID, description)
sudo flint -d lid-<switch-LID> q       # in-band firmware query of an IB switch (needs MFT with in-band access)
# On the switch CLI (depends on switch OS):
show version                            # MLNX-OS / Onyx style
nv show system                          # NVOS / Cumulus style

For a fleet, use UFM's inventory to list every switch's model, OS, and firmware. All switches of the same role should run the same validated version, matching the cluster's firmware recipe.

Deeper: InfiniBand and Ethernet switch upgrades in the networking firmware guide.

4.7 Confirm firmware and software on BlueField-3

# Host side: NIC firmware of the DPU
sudo mlxfwmanager --query
sudo flint -d /dev/mst/mt41692_pciconf0 q

# DPU side (Arm OS): BFB image, DOCA runtime, drivers
ssh ubuntu@192.168.100.2 'cat /etc/mlnx-release; ofed_info -s'

# Its BMC (BlueField-3): Redfish firmware inventory on the DPU BMC address
curl -s -u $BMC_USER:$BMC_PASS --cacert bmc-ca.pem https://$DPU_BMC/redfish/v1/UpdateService/FirmwareInventory

4.8 Confirm firmware on transceivers

• Optical transceivers have their own firmware. Mismatched or old module firmware can cause link flaps or high BER.
• Host side: mlxlink -d mlx5_X -m shows module vendor, part number, and firmware information for each port.
• Fabric-wide: ibdiagnet collects cable/module information for every link; UFM shows cable inventory.
• Switch side: the switch OS's transceiver commands show the modules plugged into each switch port.
• Compare against the qualified module list and firmware versions; update through the vendor-supported tool where needed.
• More: transceiver firmware in the signal-quality guide.

4.9 Run ClusterKit for a multifaceted node assessment

ClusterKit (part of HPC-X) runs a battery of tests across many nodes at once: GPU and CPU checks, memory bandwidth, and pairwise and simultaneous network bandwidth and latency. It points to outlier nodes and links quickly.

Details, output, and how it compares with DCGM, perftest, and nccl-tests: ClusterKit vs other tools.

4.10 Run NCCL to verify east-west fabric bandwidth

Multi-node all_reduce_perf (and all_gather_perf, reduce_scatter_perf, alltoall_perf) through Slurm or MPI, scaling from pairs to racks to the whole cluster. Check NET/IB with all HCAs and GDRDMA in the logs. See phase 8, the NCCL deep dive, and the GPU0 bottleneck case.

4.11 NCCL burn-in

The goal is stability over time at full fabric load, not a single good number.

# Repeated full-cluster runs with correctness checking, logged for later comparison
for i in $(seq 1 48); do
  srun -N 32 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
    ./all_reduce_perf -b 1G -e 16G -f 2 -g 1 -n 200 -c 1 | tee -a nccl-burnin.log
done
grep -E "Avg bus bandwidth|#wrong" nccl-burnin.log | tail
ibqueryerrors                                   # did any link's error counters grow during the burn-in?

4.12 HPL burn-in

• Run HPL repeatedly (or with a long run time) on all nodes at the same time, so the whole facility runs at maximum power and heat. This is also the real test of power and cooling.
• Check every run: residual PASSED, Gflops per node consistent, no node drops out, no throttling, no Xids/ECC errors.
• Watch rack power draw and inlet temperatures during the run.

See HPL validation strategy and the burn-in guide (sequence, monitoring, and a lab for NCCL, HPL, and NeMo).

4.13 NeMo burn-in

NeMo is NVIDIA's framework for training and customizing large language and multimodal models. A NeMo burn-in runs a real LLM pretraining workload (for example a GPT- or Llama-style model) across the cluster for hours, usually with synthetic ("mock") data so storage isn't the bottleneck, then optionally with real data and checkpointing.

# Sketch: launch from the NeMo NGC container under Slurm (recipes and arguments differ between NeMo versions)
srun -N 16 --ntasks-per-node=8 --gpus-per-node=8 \
     --container-image=nvcr.io#nvidia/nemo:<tag> --container-mounts=/mnt/ckpt:/ckpt \
     python <nemo-pretraining-script-or-recipe> <model-size and parallelism options> <mock-data option>

In depth: NeMo burn-in (parallelism and throughput math with a calculator, log fields, checkpoints and the resume test, failure signatures, and a NeMo practice set).

4.14 Test storage

See phase 9 and benchmarking with fio.

Domain 5 deep dive: Troubleshooting and Optimization 12%

Detecting, analyzing, and resolving hardware faults and performance bottlenecks: root cause, component replacement, server and storage tuning.

5.1 5.2 Identify and troubleshoot hardware faults; identify faulty cards, GPUs, and PSUs

Xid codes worth knowing

All codes, ECC/row-remap states, CUDA/NCCL/NVML messages, and the restart → reset → reboot → drain → RMA ladder: error codes guide.

Find the physical part

nvidia-smi --query-gpu=index,pci.bus_id,serial,uuid --format=csv   # map a failing GPU to its serial and PCI address
ipmitool -I lanplus -H $BMC -U $BMC_USER -E chassis identify 120    # blink the locator LED for 2 minutes
sudo nvsm show gpus                                                 # DGX: GPU inventory and health

On DGX/HGX, use the platform's service documentation to map PCI bus IDs / GPU indexes to physical module positions. The nvidia-smi index isn't guaranteed to match the slot label; use serial numbers or bus IDs (see GPU ordering & visibility).

5.3 Replace faulty cards, GPUs, and power supplies

1. Collect evidence for the RMA: nvidia-bug-report.sh, nvsm dump health (DGX), DCGM diag JSON, SEL, serial numbers.
2. Drain the node so no new jobs land on it: scontrol update nodename=dgx07 state=drain reason="GPU4 Xid79 RMA" (or cordon/drain in Kubernetes).
3. Wait for running jobs to finish (or requeue them), then power down through the BMC or BCM.
4. Replace the part following the service guide. Hot-swappable parts such as redundant PSUs and fans on many servers can be replaced while running; GPUs, GPU baseboards, NICs, and DIMMs need the system powered off. ESD protection always.
5. Bring the new part to the firmware baseline (replacement parts often ship with different firmware).
6. Update identity if it changed: a new motherboard or boot NIC means a new MAC address for BCM provisioning; a new BMC may need its network and credentials set again.
7. Re-validate: nvidia-smi, dcgmi diag -r 3, single-node NCCL, and a multi-node test with a known-good peer.
8. Return to service: scontrol update nodename=dgx07 state=resume, and record what was replaced.

5.4 Performance optimization for AMD and Intel servers

The GPUs do the math, but the host CPU, memory, and PCIe topology feed them. Tuning focuses on predictable performance, NUMA locality, and PCIe/RDMA efficiency.

OS-level tuning (both)

sudo cpupower frequency-set -g performance            # CPU frequency governor
sudo tuned-adm profile throughput-performance          # or the profile the vendor recommends
numactl -H ; nvidia-smi topo -m                        # which NUMA node / CPU cores each GPU and NIC is near
srun --cpu-bind=cores --gpus-per-task=1 ...            # bind each rank to cores near its GPU
sudo mlnx_tune                                          # NVIDIA networking tuning report / profiles
cat /sys/kernel/mm/transparent_hugepage/enabled        # follow framework / vendor guidance

Deeper: AMD vs Intel server tuning guide (knob translator, ACS/IOMMU, NUMA pinning, locality checker).

5.5 Storage optimization

• Use the right network: storage traffic on the storage fabric with RDMA (IB, RoCE, NFS over RDMA) or many parallel TCP connections (NFS nconnect), jumbo frames on Ethernet.
• Parallel filesystems: stripe large files across many storage targets (for example Lustre lfs setstripe -c -1 on directories with large files); avoid millions of tiny files.
• Local caching: stage datasets to the local NVMe RAID (/raid) or use a caching layer, so every epoch doesn't hit shared storage.
• GPUDirect Storage: DMA directly between storage and GPU memory, bypassing CPU bounce buffers.
• Data format and loader: fewer, larger files (sharded record formats), enough data-loader workers, prefetching, pinned memory.
• Checkpoints: asynchronous or distributed checkpointing; spread writes across storage targets.
• Measure before and after with fio / IOR / mdtest, and confirm GPUs are no longer waiting on input (DCGM activity metrics or Nsight Systems timelines).

Deeper: Lustre & third-party storage guide, filesystems and tuning, local caching, spotting input-bound jobs.

Master test matrix: which tool proves what

DCGM vs nccl-tests vs perftest vs HPL

Troubleshooting method: isolate and bisect

1. Pin down the symptom: hang, error, slow, or wrong results? All nodes or some? Inside one node or across nodes?
2. Go down the layers: if multi-node NCCL is slow, check single-node NCCL. If that's fine, it's the network. Then perftest, then ibstat.
3. Bisect: split the node list in half, test each half, and repeat until you find the bad node or link.
4. Compare with a known-good: diff firmware, driver, settings, and test results against a healthy node.
5. Read the logs: NCCL_DEBUG=INFO, dmesg (XIDs), DCGM health, ibdiagnet reports, Slurm logs.
6. Fix centrally: in BCM, fix the image or category rather than patching one node by hand. Drain the node before hardware work. RMA failing parts with DCGM or nvsm evidence.

Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Type             Hostname (key)   MAC                Category         Ip               Network          Status
---------------- ---------------- ------------------ ---------------- ---------------- ---------------- ----------------
HeadNode         bcm-head01       52:54:00:AA:00:01                   10.141.255.254   internalnet      [   UP   ]
PhysicalNode     dgx01            0C:42:A1:3E:00:01  dgx-h100         10.141.0.1       internalnet      [   UP   ]
PhysicalNode     dgx02            0C:42:A1:3E:00:02  dgx-h100         10.141.0.2       internalnet      [   UP   ]
PhysicalNode     dgx03            0C:42:A1:3E:00:03  dgx-h100         10.141.0.3       internalnet      [   UP   ]
PhysicalNode     dgx04            0C:42:A1:3E:00:04  dgx-h100         10.141.0.4       internalnet      [   UP   ]
PhysicalNode     dgx05            0C:42:A1:3E:00:05  dgx-h100         10.141.0.5       internalnet      [   UP   ]
PhysicalNode     dgx06            0C:42:A1:3E:00:06  dgx-h100         10.141.0.6       internalnet      [   UP   ]
PhysicalNode     dgx07            0C:42:A1:3E:00:07  dgx-h100         10.141.0.7       internalnet      [   UP   ]
PhysicalNode     dgx08            0C:42:A1:3E:00:08  default          10.141.0.8       internalnet      [  DOWN  ]

Tue Nov 10 19:52:11 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   31C    P0             71W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  |   00000000:43:00.0 Off |                    0 |
| N/A   30C    P0             69W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
...
|   7  NVIDIA H100 80GB HBM3          On  |   00000000:DF:00.0 Off |                    0 |
| N/A   32C    P0             72W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

OFED-internal-25.04-0.6.1:

nvidia_peermem         16384  0
nvidia              11563008  152 nvidia_uvm,nvidia_peermem,nvidia_modeset
ib_core               520192  9 rdma_cm,ib_ipoib,nvidia_peermem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx5_core            2629632  1 mlx5_ib
mlxfw                  36864  1 mlx5_core

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Aug_20_01:58:59_PM_PDT_2025
Cuda compilation tools, release 13.0, V13.0.88
Build cuda_13.0.r13.0/compiler.36424714_0

CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.43.1014
        Hardware version: 0
        Node GUID: 0x946dae0300a1b2c0
        System image GUID: 0x946dae0300a1b2c0
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 17
                LMC: 0
                SM lid: 1
                Capability mask: 0xa751e848
                Port GUID: 0x946dae0300a1b2c0
                Link layer: InfiniBand
CA 'mlx5_1'
        CA type: MT4129
        ...
                State: Initializing
                Physical state: LinkUp
                Rate: 400
...

mlx5_0 port 1 ==> ibp24s0 (Up)
mlx5_1 port 1 ==> ibp64s0 (Down)
mlx5_2 port 1 ==> ibp79s0 (Up)
mlx5_3 port 1 ==> ibp94s0 (Up)
mlx5_4 port 1 ==> ibp154s0 (Up)
mlx5_5 port 1 ==> ibp192s0 (Up)
mlx5_6 port 1 ==> ibp206s0 (Up)
mlx5_7 port 1 ==> ibp220s0 (Up)
...

index, name, driver_version, persistence_mode, ecc.mode.current, mig.mode.current, pstate, temperature.gpu, power.limit [W]
0, NVIDIA H100 80GB HBM3, 580.95.05, Enabled, Enabled, Disabled, P0, 31, 700.00 W
1, NVIDIA H100 80GB HBM3, 580.95.05, Enabled, Enabled, Disabled, P0, 30, 700.00 W
2, NVIDIA H100 80GB HBM3, 580.95.05, Enabled, Enabled, Disabled, P0, 32, 700.00 W
3, NVIDIA H100 80GB HBM3, 580.95.05, Enabled, Enabled, Disabled, P0, 31, 700.00 W
4, NVIDIA H100 80GB HBM3, 580.95.05, Enabled, Enabled, Disabled, P0, 29, 700.00 W
5, NVIDIA H100 80GB HBM3, 580.95.05, Enabled, Enabled, Enabled, P0, 30, 700.00 W
6, NVIDIA H100 80GB HBM3, 580.95.05, Enabled, Enabled, Disabled, P0, 31, 700.00 W
7, NVIDIA H100 80GB HBM3, 580.95.05, Enabled, Enabled, Disabled, P0, 33, 700.00 W

==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 19:53:40 2026
Driver Version                            : 580.95.05
CUDA Version                              : 13.0

Attached GPUs                             : 8
GPU 00000000:1B:00.0
    ECC Mode
        Current                           : Enabled
        Pending                           : Enabled
    ECC Errors
        Volatile
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 0
            DRAM Uncorrectable            : 0
        Aggregate
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 12
            DRAM Uncorrectable            : 0
            SRAM Threshold Exceeded       : No
    Remapped Rows
        Correctable Error                 : 0
        Uncorrectable Error               : 0
        Pending                           : No
        Remapping Failure Occurred        : No
        Bank Remap Availability Histogram
            Max                           : 2560 bank(s)
            High                          : 0 bank(s)
            Partial                       : 0 bank(s)
            Low                           : 0 bank(s)
            None                          : 0 bank(s)
GPU 00000000:43:00.0
...

        GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  NIC0  NIC1  NIC2  NIC3  NIC4  NIC5  NIC6  NIC7  CPU Affinity    NUMA Affinity  GPU NUMA ID
GPU0    X     NV18  NV18  NV18  NV18  NV18  NV18  NV18  PIX   NODE  NODE  NODE  SYS   SYS   SYS   SYS   0-55,112-167    0              N/A
GPU1    NV18  X     NV18  NV18  NV18  NV18  NV18  NV18  NODE  PIX   NODE  NODE  SYS   SYS   SYS   SYS   0-55,112-167    0              N/A
GPU2    NV18  NV18  X     NV18  NV18  NV18  NV18  NV18  NODE  NODE  PIX   NODE  SYS   SYS   SYS   SYS   0-55,112-167    0              N/A
GPU3    NV18  NV18  NV18  X     NV18  NV18  NV18  NV18  NODE  NODE  NODE  PIX   SYS   SYS   SYS   SYS   0-55,112-167    0              N/A
GPU4    NV18  NV18  NV18  NV18  X     NV18  NV18  NV18  SYS   SYS   SYS   SYS   PIX   NODE  NODE  NODE  56-111,168-223  1              N/A
GPU5    NV18  NV18  NV18  NV18  NV18  X     NV18  NV18  SYS   SYS   SYS   SYS   NODE  PIX   NODE  NODE  56-111,168-223  1              N/A
GPU6    NV18  NV18  NV18  NV18  NV18  NV18  X     NV18  SYS   SYS   SYS   SYS   NODE  NODE  PIX   NODE  56-111,168-223  1              N/A
GPU7    NV18  NV18  NV18  NV18  NV18  NV18  NV18  X     SYS   SYS   SYS   SYS   NODE  NODE  NODE  PIX   56-111,168-223  1              N/A
NIC0    PIX   NODE  NODE  NODE  SYS   SYS   SYS   SYS   X     NODE  NODE  NODE  SYS   SYS   SYS   SYS
NIC1    NODE  PIX   NODE  NODE  SYS   SYS   SYS   SYS   NODE  X     NODE  NODE  SYS   SYS   SYS   SYS
NIC2    NODE  NODE  PIX   NODE  SYS   SYS   SYS   SYS   NODE  NODE  X     NODE  SYS   SYS   SYS   SYS
NIC3    NODE  NODE  NODE  PIX   SYS   SYS   SYS   SYS   NODE  NODE  NODE  X     SYS   SYS   SYS   SYS
NIC4    SYS   SYS   SYS   SYS   PIX   NODE  NODE  NODE  SYS   SYS   SYS   SYS   X     NODE  NODE  NODE
NIC5    SYS   SYS   SYS   SYS   NODE  PIX   NODE  NODE  SYS   SYS   SYS   SYS   NODE  X     NODE  NODE
NIC6    SYS   SYS   SYS   SYS   NODE  NODE  PIX   NODE  SYS   SYS   SYS   SYS   NODE  NODE  X     NODE
NIC7    SYS   SYS   SYS   SYS   NODE  NODE  NODE  PIX   SYS   SYS   SYS   SYS   NODE  NODE  NODE  X

Legend:
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:
  NIC0: mlx5_0
  NIC1: mlx5_1
  ...

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-3f9a1c2e-0b4d-4e71-9a55-1c0d2e3f4a01)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         ...
         Link 17: 26.562 GB/s
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-3f9a1c2e-0b4d-4e71-9a55-1c0d2e3f4a02)
         Link 0: 26.562 GB/s
         ...
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-3f9a1c2e-0b4d-4e71-9a55-1c0d2e3f4a08)
         Link 0: 26.562 GB/s
         ...
         Link 5: <inactive>
         ...
         Link 17: 26.562 GB/s

Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 4.2.3                                          |
| Driver Version Detected   | 580.95.05                                      |
| GPU Device IDs Detected   | 2330,2330,2330,2330,2330,2330,2330,2330        |
|-----  Deployment  --------+------------------------------------------------|
| software                  | Pass                                           |
+-----  Integration  -------+------------------------------------------------+
| pcie                      | Pass - All                                     |
+-----  Hardware  ----------+------------------------------------------------+
| memory                    | Pass - All                                     |
| diagnostic                | Pass - All                                     |
| nvbandwidth               | Pass - All                                     |
+-----  Stress  ------------+------------------------------------------------+
| targeted_stress           | Pass - All                                     |
| targeted_power            | Pass - All                                     |
| memory_bandwidth          | Pass - All                                     |
+---------------------------+------------------------------------------------+

# nThread 1 nGpus 8 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid  41210 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid  41210 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
...
#  Rank  7 Group  0 Pid  41210 on      dgx01 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    24.91    0.00    0.00      0    24.33    0.00    0.00      0
          16             4     float     sum      -1    24.60    0.00    0.00      0    24.48    0.00    0.00      0
...
    67108864      16777216     float     sum      -1    357.2  187.88  328.79      0    355.9  188.56  329.98      0
   134217728      33554432     float     sum      -1    621.4  215.99  377.99      0    619.8  216.55  378.96      0
...
  4294967296    1073741824     float     sum      -1    16091  266.92  467.11      0    16077  267.15  467.51      0
  8589934592    2147483648     float     sum      -1    32042  268.08  469.15      0    32019  268.28  469.49      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 171.094
#

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x11 QPN 0x0128 PSN 0x5a3c1e RKey 0x1fff00 VAddr 0x007f3a2c000000
 remote address: LID 0x2b QPN 0x0129 PSN 0x7b2d4f RKey 0x1fff01 VAddr 0x007f91e8000000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      5000             392.45             392.31               0.748270
---------------------------------------------------------------------------------------

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
batch*       up   infinite      7   idle dgx[01-07]
batch*       up   infinite      1  down* dgx08
   Gres=gpu:8
   State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A

PSU0_Status      | 8Ah | ok  | 10.0 | Presence detected
PSU1_Status      | 8Bh | ok  | 10.1 | Presence detected
PSU2_Status      | 8Ch | ok  | 10.2 | Presence detected
PSU3_Status      | 8Dh | ok  | 10.3 | Presence detected
PSU4_Status      | 8Eh | ok  | 10.4 | Presence detected, Power Supply AC lost
PSU5_Status      | 8Fh | ok  | 10.5 | Presence detected

05:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
06:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
07:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
08:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
1b:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
43:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
52:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
61:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
9d:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
c3:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
d1:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)
df:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] (rev a1)

                LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM not supported
                LnkSta: Speed 32GT/s, Width x16
                LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+

DGX_NAME="DGX Server"
DGX_PRETTY_NAME="NVIDIA DGX Server"
DGX_SWBUILD_DATE="2025-06-12-09-41-17"
DGX_SWBUILD_VERSION="7.1.0"
DGX_COMMIT_ID="4b1d9e7"
DGX_PLATFORM="DGX Server for DGX H100"
DGX_SERIAL_NUMBER="<redacted>"

DGX_OTA_VERSION="7.2.0"
DGX_OTA_DATE="Thu Aug 21 10:14:52 AM UTC 2025"
DGX_OTA_VERSION="7.3.1"
DGX_OTA_DATE="Mon Nov  2 03:22:10 PM UTC 2026"

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  linux-image-6.8.0-1032-nvidia linux-modules-6.8.0-1032-nvidia
The following packages will be upgraded:
  datacenter-gpu-manager-4-core datacenter-gpu-manager-4-cuda13 libnvidia-nscq
  nvidia-driver-580-open nvidia-fabricmanager nvidia-system-core ...
37 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Inst nvidia-fabricmanager [580.82.07-1] (580.95.05-1 NVIDIA CUDA:cuda [amd64])
Inst nvidia-driver-580-open [580.82.07-0ubuntu1] (580.95.05-0ubuntu1 NVIDIA CUDA:cuda [amd64])
...
Conf nvidia-driver-580-open (580.95.05-0ubuntu1 NVIDIA CUDA:cuda [amd64])

Checking for a new Ubuntu release
...
Continue running under SSH?

This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.

If you continue, an additional ssh daemon will be started at port
'1022'.
Do you want to continue?

Continue [yN] y
...
Do you want to start the upgrade?

5 installed packages are no longer supported by Canonical. You can
still get support from the community.

48 packages are going to be removed. 412 new packages are going to be
installed. 2187 packages are going to be upgraded.

You have to download a total of 3,912 M. This download will take
about 6 minutes with your connection.

Continue [yN]  Details [d] y
...
System upgrade is complete.

Restart required

To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.

Continue [yN] y

Operational Info
----------------
State                              : Active
Physical state                     : LinkUp
Speed                              : IB-NDR
Width                              : 4x
FEC                                : Interleaved_Standard_RS_FEC_PLR - (544,514)
Loopback Mode                      : No Loopback
Auto Negotiation                   : ON

Troubleshooting Info
--------------------
Status Opcode                      : 0
Group Opcode                       : N/A
Recommendation                     : No issue was observed

Physical Counters and BER Info
------------------------------
Time Since Last Clear [Min]        : 1442.6
Effective Physical Errors          : 0
Effective Physical BER             : 15E-255
Raw Physical Errors Per Lane       : 102,98,131,87
Raw Physical BER                   : 3E-12
Link Down Counter                  : 0
Link Error Recovery Counter        : 0

Problem scenarios (44). Click to expand

Scenarios 27–42 map to specific blueprint objectives (shown in the LAYER line).

For each one, try to name the layer and cause before you open it.

Searchable command reference

Exam heuristics and traps

Practice quiz (73 questions)

Filter by domain: All domains1. System and Server Bring-Up (31%)2. Physical Layer Management (5%)3. Control Plane (19%)4. Cluster Test and Verification (33%)5. Troubleshooting and Optimization (12%)