Cluster burn-in: NCCL, HPL, and NeMo

How to burn in a new AI cluster: what separates a burn-in from a validation run, the node-scaling sequence NVIDIA's HPL burn-in tooling uses, what to monitor, and how to run and judge NCCL, HPL, and NeMo burn-ins (LLM pretraining and the BERT question-answering fine-tune). Ends with three hands-on simulated tickets where a burn-in finds a bad cable, a failed fan, and a GPU that falls off the bus. Goes deeper than the burn-in parts of the bring-up guide. Objectives 4.11, 4.12, and 4.13.

1. Validation vs burn-in

A validation run answers "does it work right now?": one run of a test, compared with an expected number. A burn-in answers "does it keep working under sustained full load?": the same kinds of tests, repeated back to back for hours, while you watch for anything that drifts or fails.

	Validation	Burn-in
Length	Minutes	Hours to days
Question	Is the number right?	Does the number stay right, and does nothing break?
Catches	Misconfiguration, missing drivers, wrong cabling, a dead part	Marginal parts: links whose errors grow with heat, GPUs and HBM that fail when hot, fans and PSUs that give out, intermittent hangs
Judged by	One result vs the baseline	The trend over every iteration, plus logs and counters

Validation is a test drive around the block. Burn-in is driving the new car across the country with the A/C on full: the radiator hose that was fine for ten minutes splits at hour six. Hardware failure rates follow a "bathtub curve": many early failures ("infant mortality"), then a long quiet period. Burn-in pushes the early failures into the week before handover instead of the first week of production.

Burn-in comes after the cluster passes validation. If single-node and multi-node validation numbers aren't right yet, a burn-in only tells you the same thing slowly.

The three burn-ins load different parts of the system, which is why the blueprint lists all three:

Burn-in	What it stresses most	Typical failure it finds
NCCL (4.11)	NVLink and the InfiniBand/RoCE fabric: every NIC, cable, transceiver, switch port	A transceiver whose bit error rate climbs as it warms up; a link that flaps; NCCL timeouts
HPL (4.12)	GPU math, HBM, power delivery, and cooling: the highest sustained power draw	Thermal throttling, a failed fan, PSU trips, HBM ECC errors, a node 5–10% slower than its peers
NeMo (4.13)	Everything at once, the way production uses it: mixed math, overlapping collectives, containers, scheduler, storage for checkpoints	Hangs and timeouts under real communication patterns, a GPU falling off the bus, slow checkpoint storage, software stack mismatches

2. Before you start: plan and pass criteria

Baselines from validation: single-node and full-cluster numbers for each test (for example AllReduce busbw at large sizes, HPL TFLOPS per node, NeMo step time). The burn-in is judged against these.
Freeze and record versions: firmware, driver, CUDA, NCCL, DOCA/OFED, container tags, BCM image. A version change during the burn-in invalidates the comparison.
Monitoring on before the first job: DCGM (exporter or dcgmi), kernel logs for Xids, fabric counters cleared, BMC sensors, rack power.
Write the pass criteria down before you look at results: duration, allowed variation from baseline, zero tolerance items (Xid, ECC uncorrectable, link down, hang).
Decide the failure policy: drain the node, repair, retest it alone, and rerun the stage it failed in (the clock restarts for that node).

Burn-in	Typical duration	Notes
NCCL	12–24 hours	Long enough for every link to reach full temperature and for slow error growth to show
HPL	Several hours per scale step; the full-cluster stage often runs overnight or longer	Run it with the whole facility loaded, so power and cooling are tested together
NeMo	Several hours to a day of training	Long enough for many checkpoints and thousands of steps

These durations are common practice and come from vendor and third-party acceptance plans, not from a published NVIDIA number. On the exam, look for the idea (sustained, repeated, hours) rather than an exact figure. The real duration is whatever the acceptance plan for that cluster says.

Write criteria you can check mechanically, for example: "every iteration completes; AllReduce busbw at 8–16 GB stays within 5% of baseline for the full 24 hours; #wrong = 0; no new Xid, no uncorrectable ECC, no row-remap failure; no link-down events; no node drained".

3. The scaling sequence: one node, then up to the whole cluster

NVIDIA's HPL Burn-In Test tooling (published on GitHub) structures the run as a sequence, and the same idea works for NCCL and NeMo. Each step only adds nodes that already passed the step before, so a bad node is found while it's alone instead of slowing a 64-node job.

Single-node runs on every node. Compare nodes with each other: the outlier (slowest, hottest, throttling) is fixed now.
Powers of two: 2, 4, 8, ... nodes, up to half the cluster. Per-node performance should stay similar at each size; a drop at one size points at the fabric between those nodes.
Two jobs at once, each on half the cluster. Both halves under load together: shared switches, power, and cooling at full use.
One job on all nodes. The final full-scale run.

A validation run is one pass through the sequence. A burn-in repeats validation runs back to back for the planned duration.

Burn-in plan builder

Nodes in the cluster
Repetitions of the sequence (burn-in)

1 → 2 → 4 → … → N/2 → two N/2 at once → N. Small first, so the bad node is found when it's alone.

4. What to watch while it runs

Signal	How	Bad sign
Test result per iteration	The job log: busbw, TFLOPS, step time	A downward trend, or a sudden drop that doesn't recover
Correctness	nccl-tests `#wrong`; HPL residual check; NeMo loss	Any `#wrong` above 0; residual FAILED; loss NaN or spikes
GPU errors	`dmesg -T \| grep -i xid`, `dcgmi health -c`, `nvidia-smi -q -d ECC,ROW_REMAPPER`	Xid 48/63/64/94/95 (memory), 74 (NVLink), 79 (fell off the bus); uncorrectable ECC; remap failure
Throttling	`nvidia-smi -q -d PERFORMANCE,TEMPERATURE`, DCGM fields	HW Slowdown or HW Thermal Slowdown active; one GPU much hotter than its neighbors
Fabric	`mlxlink -d mlx5_N -c`, `ibqueryerrors`, `ibdiagnet` before vs after	Effective BER rising, Link Down counter above 0, symbol errors growing
Platform	BMC sensors (`ipmitool sdr`, Redfish), rack PDUs	A fan at 0 RPM, a PSU alarm, inlet temperature over spec

dcgmi health -s a              # turn on all health watches (once per node or group)
dcgmi health -c                # check: anything new since the watches started?
dcgmi dmon -e 150,155,203,1002 -d 5000   # temperature, power, utilization, SM activity every 5 s
pdsh -w dgx[01-32] "dmesg -T | grep -i xid"        # run the same check on every node
ibdiagnet -pc                  # clear fabric counters before the run; run ibdiagnet again after

A burn-in that "finished" isn't a pass until the logs and counters are clean too. A run can complete with the right average while one link retrained twice and an HBM row was remapped.

5. NCCL burn-in (4.11)

Run nccl-tests at large message sizes on every node over and over, and check each iteration. AllReduce is the core; add AllGather, ReduceScatter, and AlltoAll so the patterns real training uses are covered.

# Repeat the test for the burn-in window (here: 24 h) and keep every iteration's output
end=$(( $(date +%s) + 24*3600 ))
while [ $(date +%s) -lt $end ]; do
  srun -N 32 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
       ./build/all_reduce_perf -b 8G -e 16G -f 2 -g 1 -n 100 -c 1 | tee -a nccl-burnin.log
done
grep -E "Avg bus bandwidth|#wrong|WARN|error" nccl-burnin.log

# Alternative: one long-running process; -N 0 repeats the test until the job's time limit
srun ... ./build/all_reduce_perf -b 8G -e 16G -f 2 -g 1 -N 0

-c 1 checks results (the default), so a silent data error shows as #wrong.
Judge the large-size busbw over time, not the single "Avg bus bandwidth" line.
When the aggregate drops, split it: NCCL_TESTS_SPLIT="MOD 8" runs one test per rail, DIV 8 one per node (see nccl-tests: splitting). Then use mlxlink on that rail across the nodes to find the link.

Every iteration completes (no hang, no NCCL timeout); #wrong = 0; large-size busbw stays within a small band of the baseline for the whole run (no trend down); no Xid; no new link errors or link-down events.

A rail that degrades only after hours (BER climbing as a transceiver heats up) is exactly what a 10-minute validation misses. Comparing the first hour with the last hour of the log is the point of the burn-in.

6. HPL burn-in (4.12)

HPL drives the GPUs and HBM at their highest sustained power, so it's the real test of power delivery and cooling. Use one well-chosen HPL.dat configuration (not a tuning sweep) and the NVIDIA HPC-Benchmarks container.

# Single node (repeat on every node, compare the nodes)
srun -N 1 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix -w dgx03 \
     --container-image=nvcr.io#nvidia/hpc-benchmarks:<tag> \
     hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-8GPUs.dat

# Scale up following the sequence: 2, 4, ... N/2 nodes, two N/2 jobs at once, then all N
srun -N 16 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
     --container-image=nvcr.io#nvidia/hpc-benchmarks:<tag> hpl.sh --dat /dat/HPL-128GPUs.dat
grep -E "WR|PASSED|FAILED" hpl-*.out

Every run must say PASSED for the residual check. A FAILED residual means wrong math: hardware, not tuning.
Compare nodes with each other: in the single-node step, a node 5% or more below the median is the outlier. Look at its throttle reasons and temperatures first.
Per-node TFLOPS should hold as you scale; some efficiency loss is normal at large scale, a sudden drop at one size isn't.
Watch rack power and inlet temperatures: HPL on every node at once is the moment the facility runs at its peak.

All runs complete with residual PASSED; single-node results are within a few percent of each other and the reference; scaled runs keep per-node performance close to the single-node number; no throttling, no Xids or ECC errors, no PSU or fan alarms.

7. NeMo burn-in: LLM pretraining (4.13)

NeMo burn-in trains a real large language model the way customers will: tensor, pipeline, and data parallelism with overlapping NCCL collectives, mixed precision math, Slurm and containers, and checkpoint writes. Synthetic (mock) data is the usual choice, so a slow dataset doesn't hide a compute or fabric problem.

It's the last of the three burn-ins. HPL proved the GPUs, power, and cooling; NCCL proved the fabric; NeMo proves they work together, under the real mix of compute and communication, with the software stack and storage customers will use.

What NeMo catches that HPL and NCCL miss	Why
Hangs and timeouts that only appear when collectives overlap	`nccl-tests` runs one collective at a time; training runs tensor-parallel, pipeline, and data-parallel traffic at once, overlapped with compute
Faults in kernels HPL doesn't run	Transformer Engine BF16/FP8 matrix multiplies, attention, and layer norms, at the power draw of a real job
Software stack problems	Container versus driver, NCCL network settings inside the container, Pyxis/Enroot, Slurm GPU binding
Checkpoint storage	Every node writes its share of a large checkpoint at the same moment, every N steps
Stragglers	Training is synchronous: every step waits for the slowest GPU, so one throttling GPU slows the whole job

HPL tests the engine, NCCL tests the roads, NeMo tests rush hour.

7.1 The pieces you'll hear about

Piece	What it is	Where you see it
NeMo Framework container	The NGC image with NeMo, Megatron Core, Transformer Engine, PyTorch, and NCCL at matched versions. Tags are `YY.MM`.	`nvcr.io/nvidia/nemo:25.07`
Megatron Core	NVIDIA's library that implements the parallelism (TP, PP, CP, EP), the distributed optimizer, and distributed checkpoints	`MegatronStrategy(tensor_model_parallel_size=..., ...)`
Transformer Engine	BF16 and FP8 kernels for Hopper and Blackwell tensor cores	Precision setting: `bf16-mixed` or an FP8 recipe
Recipes	Ready-made configs per model and size: model, parallelism, optimizer, data, trainer, logging	`--factory llama3_8b`, `llama3_70b`, ...
MockDataModule	Synthetic token data, the recipes' default. The 8B recipe uses sequence length 8192, micro batch 1, global batch 512 unless you override them.	`data=MockDataModule(seq_length=8192, ...)`
NeMo-Run	The launcher: runs the same recipe locally, on Slurm (`SlurmExecutor`), or on Kubernetes	`NEMORUN_HOME`, `run.executor=...`
`nemo` CLI	Runs a recipe with `component.parameter=value` overrides; `--dryrun` prints the resolved config without running	`nemo llm pretrain --factory llama3_8b trainer.max_steps=2000`
Performance scripts	The configs behind NVIDIA's published performance tables, one per model, submitted to Slurm through NeMo-Run	`scripts/performance/llm/pretrain_llama3_8b.py`

Versions: NeMo 1.x used YAML configs, Hydra overrides, and training scripts such as megatron_gpt_pretraining.py. NeMo 2.x uses Python recipes, NeMo-Run, and the nemo CLI. Newer releases move the LLM training recipes and performance scripts to Megatron Bridge. The exam tests the ideas (mock data, parallelism, throughput, checkpoints, failure reading), not one release's flags.

7.2 Running it, step by step

Freeze the stack. Record the container tag, driver, NCCL, and firmware, and don't change them during the burn-in (see the plan).
Stage the container once on shared storage (an Enroot .sqsh file), so 64 nodes don't each pull a multi-gigabyte image from NGC at the same moment.
Dry run. --dryrun prints the resolved recipe (model, parallelism, batch sizes, steps, checkpoint settings) without using a GPU. Catch config mistakes here, not 40 minutes into a 64-node job.
Single-node smoke test on every node, a short run of about 100 steps. Compare step time across nodes; a node 5% or more slower than the median is the outlier to fix first.
Scale with the usual sequence: 2, 4, ... N/2, two N/2 jobs at once, then N. Per-GPU throughput should stay close at each size.
The long run on all nodes for the planned hours, with checkpoints and monitoring on.
Resume test. Stop the job after a checkpoint and restart it unchanged; it must continue from that step (7.5).

# Stage the container once on shared storage
enroot import -o /lustre/images/nemo-25.07.sqsh docker://nvcr.io#nvidia/nemo:25.07

# Preview the resolved config, then a short single-node run (the CLI starts one worker per local GPU)
nemo llm pretrain --factory llama3_8b --dryrun
srun -N 1 -w dgx03 --gpus-per-node=8 --container-image=/lustre/images/nemo-25.07.sqsh \
     nemo llm pretrain -y --factory llama3_8b trainer.max_steps=100     # -y skips the confirm prompt

# Overrides are component.parameter=value; constructor arguments go in the factory string
nemo llm pretrain -y --factory "llama3_8b(num_nodes=8)" trainer.max_steps=5000 data.global_batch_size=128

# Multi-node: NeMo-Run's Slurm executor writes and submits the sbatch job.
# NVIDIA's performance scripts do this for the published configs (flags vary by release; check --help):
python -m scripts.performance.llm.pretrain_llama3_8b --account <acct> --partition <part> \
     --gpu h100 --compute_dtype bf16 --num_gpus 64 --gpus_per_node 8 --max_steps 200 --dryrun

The performance scripts need NEMORUN_HOME set (where NeMo-Run keeps job directories and logs). Their defaults come from the recommended_model_configs CSV files, the same configs as the published tables, and they default to 100 steps: enough to measure throughput, not a burn-in. For the burn-in, raise the step count.

Environment variable	What it does
`CUDA_DEVICE_MAX_CONNECTIONS=1`	Required on H100 for tensor-parallel communication overlap; the recipes set it. A hand-written job that leaves it out runs, but slower.
`NCCL_DEBUG=INFO` (or `WARN` for long runs)	Prints the transport NCCL picked (`NET/IB` versus `NET/Socket`), the topology, and errors. INFO on a 24-hour run makes huge logs.
`NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`	Which InfiniBand adapters carry data, and which Ethernet interface is used for setup
`NVTE_FWD_LAYERNORM_SM_MARGIN` / `NVTE_BWD_LAYERNORM_SM_MARGIN`	Keep a few SMs free (for example 16) for data-parallel communication overlap

7.3 Parallelism and batch math

A model too big or too slow for one GPU is split several ways at once. Each split has its own traffic pattern, and that decides where it should run.

Split	What it divides	Traffic	Where it belongs
TP, tensor parallel	Each layer's weight matrices across GPUs	AllReduce / AllGather / ReduceScatter inside every layer, every micro-batch: the heaviest	Inside the NVLink domain (TP ≤ 8 on a DGX H100)
SP, sequence parallel	Activations TP doesn't split (layer norm, dropout), along the sequence	Part of TP's collectives	Always paired with TP
PP, pipeline parallel	Groups of layers into stages	Point-to-point activation sends between neighbouring stages	Across nodes is fine
VP, virtual pipeline	Each pipeline rank holds several smaller chunks (interleaved schedule)	More point-to-point sends, less idle "bubble" time	With PP
CP, context parallel	Each sample's sequence (its tokens)	Attention keys and values exchanged between CP ranks	Long sequences (8K and up)
EP, expert parallel	The experts of a mixture-of-experts (MoE) model	All-to-all token routing	Inside the NVLink domain where possible
DP, data parallel	The batch: each replica trains on different samples	Gradient reduction once per step (ReduceScatter + AllGather with the distributed optimizer), overlapped with the backward pass	Across nodes; this is what grows with cluster size

The formulas
GPUs = TP × PP × CP × DP, so DP = GPUs ÷ (TP × PP × CP)
Gradient accumulation GA = GBS ÷ (MBS × DP), and it must be a whole number (GBS = global batch size, MBS = micro batch size)
Tokens per step = GBS × sequence length
Tokens/s/GPU = GBS × sequence length ÷ (step time × GPUs)
Model TFLOPS/GPU ≈ 6 × parameters × tokens/s/GPU, plus attention. NeMo's logged figure includes attention, which adds about 20% for Llama 3 8B at 8K tokens.
MFU (model FLOPs utilization) = model TFLOPS/GPU ÷ peak dense TFLOPS. For H100 SXM, peak is about 989 TFLOPS in BF16 and 1,979 in FP8.

Worked example 1, the NVIDIA reference (Llama 3 8B, DGX H100, FP8, 8 GPUs, TP 1, PP 1, CP 2, GBS 128, MBS 1, sequence 8192):

DP = 8 ÷ 2 = 4, and GA = 128 ÷ (1 × 4) = 32, which matches the table's GA of 32.
Each step processes 128 × 8192 = 1,048,576 tokens.
At the published 14,340 tokens/s/GPU, a step takes 1,048,576 ÷ (14,340 × 8) ≈ 9.1 s.
MFU = 830 ÷ 1,979 ≈ 42%.

Worked example 2, the lab's burn-in (the same model in BF16 on 64 GPUs, TP 1, PP 1, CP 2, GBS 128, step time 2.1 s):

DP = 64 ÷ 2 = 32, and GA = 128 ÷ 32 = 4.
Throughput is 1,048,576 ÷ (2.1 × 64) ≈ 7,800 tokens/s/GPU.
The 6 × parameters estimate gives 6 × 8.03 × 10⁹ × 7,800 ≈ 376 TFLOPS. Adding attention gives about 450, which is what the lab's log shows. That's about 46% of 989.

Worked example 3 (Llama 3 70B on 64 GPUs, TP 4, PP 8, CP 1, GBS 128): DP = 64 ÷ 32 = 2, and GA = 128 ÷ 2 = 64. Each tensor-parallel group of 4 GPUs sits inside one node, and the 8 pipeline stages span nodes.

Changing the node count changes DP, and with a fixed GBS, also GA. Per-GPU throughput is only comparable between runs with the same TP, PP, and CP, GBS, sequence length, and precision. If TP is larger than the GPUs in a node, tensor-parallel traffic crosses InfiniBand and throughput collapses. That's a config error, not a hardware fault.

NeMo throughput calculator

Nodes
GPUs per node
TP (tensor parallel)
PP (pipeline parallel)
CP (context parallel)
Global batch size (GBS)
Micro batch size (MBS)
Sequence length (tokens)
Step time (s, from train_step_timing)
Model parameters (billions)
Peak dense TFLOPS per GPU

7.4 Reading the log and judging performance

Training epoch 0, iteration 99/4999 | lr: 2.97e-05 | global_batch_size: 128 | global_step: 99 | reduced_train_loss: 8.912 | train_step_timing in s: 2.104 | consumed_samples: 12800
[NeMo I ... nemo_logging:393] Global Checkpoint Save : Rank: 0 : Iteration: 499 : Start time: ... : Save duration: 37.882s

Field	Meaning	Healthy in a burn-in
`train_step_timing in s`	Wall time for one optimizer step: the main burn-in metric	Flat for the whole run, apart from checkpoint steps
`reduced_train_loss`	Loss averaged across data-parallel ranks	Falls, then flattens; never NaN or Inf; the same curve every run
`global_step` / iteration	Steps completed (the iteration counter starts at 0)	Reaches `max_steps`
`consumed_samples`	Samples trained so far: steps completed × GBS	Continues after a resume instead of restarting at 0
`lr`	Learning rate (warmup, then decay)	Explains loss changes early on: not a hardware signal
Tokens/s/GPU, TFLOPS/GPU	Throughput, when the throughput callbacks are enabled (for example `log_tflops_per_sec_per_gpu` in the performance scripts); otherwise compute them from step time	Near the reference, stable
`Global Checkpoint Save ... Save duration`	How long the checkpoint write took	The same each time

The reference. NVIDIA publishes per-GPU throughput for each NeMo release (the "Performance Summary" in the NeMo docs). A few DGX H100 rows, all FP8, sequence length 8192:

Model	Container	GPUs	GBS	TP / PP / CP / VP	Tokens/s/GPU	TFLOPS/GPU
Llama 3 8B	25.07	8	128	1 / 1 / 2 / 1	14,340	830
Llama 3 8B	25.04	8	128	1 / 1 / 2 / 1	13,812	800
Llama 3 8B	24.12	8	128	1 / 1 / 2 / 1	14,064	814
Llama 3 70B	25.07	64	128	4 / 8 / 1 / 5	1,663	801
Llama 3.1 405B	25.07	1,024	512	8 / 8 / 2 / 8	320	840

Compare like with like: the same container, precision, GPU count, GBS, sequence length, and parallelism. A BF16 burn-in at 7,800 tokens/s/GPU is not "46% below" an FP8 table row; it's a different measurement.
The same config moves a few percent between containers (13,812 to 14,340 above), so pin the container tag before setting a baseline.
The two-step approach works well. First run the performance script with the reference config and compare it with the published row; that's the validation. Then run the long burn-in and judge it against your own first hour; that's the drift check.

7.5 Checkpoints and the resume test

A checkpoint is the largest storage burst of the burn-in. With a distributed checkpoint, every rank writes its own shard in parallel, so a few hundred GPUs hit the file system at the same moment, every N steps.

Size: with mixed precision and Adam, a checkpoint holds the weights, FP32 master weights, and two optimizer moments, about 14 bytes per parameter. That's roughly 110 GB for Llama 3 8B and about 1 TB for 70B.
Bandwidth = size ÷ save duration. 112 GB in 38 s ≈ 2.9 GB/s. A save duration that grows during the run points at storage (filling up, a degraded target, competing jobs).
Layout: one directory per checkpoint, named with step and consumed samples (.../checkpoints/model_name=0--step=499-consumed_samples=64000.0/), holding context/ (config, tokenizer) and weights/ (*.distcp shards, one set per rank).

Setting	What it controls
`ModelCheckpoint(every_n_train_steps=500, save_top_k=..., save_last=True)`	How often to save and how many to keep (`log.ckpt.*` in CLI overrides)
`MegatronStrategy(ckpt_async_save=True)`	Write in the background, so training doesn't stall for the whole save
`AutoResume(resume_if_exists=True, resume_ignore_no_checkpoint=True)`	On restart, continue from the newest checkpoint in the log directory; start fresh if there is none
Local checkpointing (NVIDIA Resiliency Extension)	Saves to node-local NVMe with replication to peers: faster saves, so you can checkpoint more often

The resume test.

Let the job pass a checkpoint, for example step 1000.
Stop it with scancel.
Resubmit it unchanged, with the same log and checkpoint directory.

It passes when:

The log says it restored from step=999 (counting starts at 0).
global_step and consumed_samples continue from there.
The loss carries on from about the same value.

A loss back near its starting value (around 11 to 12 for a 128K vocabulary) means the job started over, so resume isn't configured or is pointed at the wrong directory.

7.6 Failure signatures

In synchronous training, every rank waits for every other rank at each collective. One dead GPU makes all the others time out, and one slow GPU makes the whole job slow. Read the first error by time, not the most common one. To find the node, use node index = rank ÷ 8 (integer) and local GPU = rank mod 8, with 8 GPUs per node in node-list order. Rank 37 on dgx[01-08] is node index 4, the fifth node (dgx05), GPU 5.

What you see	Most likely	Do next
One rank: `CUDA error` (and an Xid in `dmesg`), then `Watchdog caught collective operation timeout` on many ranks	The GPU of the first failing rank	Map the rank to a node, check `dmesg` for the Xid, drain, then `dcgmi diag -r 3`
Watchdog timeouts on all ranks with no error before them	A collective stuck in the network (one link or rail) or one stuck process	`NCCL_DEBUG=INFO` on a rerun, `nccl-tests` on node subsets and per rail, link counters, `ibdiagnet`
Never reaches step 1: stalls during NCCL setup, or `NET/IB : No device found`	NCCL network config inside the container (IB devices not visible, wrong `NCCL_IB_HCA` / `NCCL_SOCKET_IFNAME`, memlock limits)	Run `nccl-tests` in the same container on the same nodes
Runs, but the log shows `NET/Socket` instead of `NET/IB`, and step time is far below reference	NCCL fell back to TCP sockets	Fix the InfiniBand visibility or settings; don't burn in on sockets
`CUDA out of memory` in the first steps on every node	Configuration: MBS or sequence too large, too little TP/PP/CP, recompute off	Fix the recipe; this is not a hardware fault
`CUDA out of memory` on one node only, same config	That node has less free memory: a leftover process, MIG enabled, a different GPU model	`nvidia-smi` on that node: processes, MIG mode, memory total
Steady but 10 to 30% below the like-for-like reference everywhere	Stack or config: wrong precision, missing `CUDA_DEVICE_MAX_CONNECTIONS=1`, sockets instead of InfiniBand, clocks or power capped	Compare the config with the reference; check the NCCL transport and `nvidia-smi -q -d PERFORMANCE`
Step time creeps up over hours	One component degrading under heat: a throttling GPU (everyone waits for it) or a link with rising errors	Straggler report, throttle reasons per GPU, link error counters over time, inlet temperatures
A regular spike every N steps	The checkpoint save: expected	Only a problem if the spikes grow (storage)
Loss goes NaN or jumps, with a known recipe and mock data	If it follows one node across reruns: silent data corruption, a hardware fault. If it happens on any set of nodes: config (learning rate, precision).	Rerun without the suspect node; `dcgmi diag -r 3` or `-r 4` on it
`CUDA driver version is insufficient for CUDA runtime version`	Container newer than the host driver supports	Pick a container that matches the driver (NGC support matrix) or update the driver as planned
Slurm kills it at the time limit	Planning: `--time` shorter than the steps need	Size steps to the time limit, or rely on checkpoints and AutoResume and resubmit
`srun: error: dgxNN: tasks ...: Exited with exit code 1` on every node	A symptom: every task exits once one fails	Scroll up to the first error

7.7 Straggler detection and other tools

Straggler detection (from the NVIDIA Resiliency Extension, added as a callback). Every GPU gets a performance score relative to the others and to its own past; the default report interval is 300 s. A score below 0.7 marks a straggler, and by default the job stops when one is found. The report lists the worst ranks:

GPU relative performance:
 Worst performing 5/512 ranks:
  Rank=76 Node=dgx10 Score=0.62
  Rank=77 Node=dgx10 Score=0.94
  ...

Fault tolerance and in-job restart (also from the Resiliency Extension) detect hangs and restart the workload. For a burn-in, the hang itself is the finding: record it before letting anything restart.
Nsight Systems: the performance scripts can profile a few steps (-en / --enable_nsys). Use it to answer "why is it slow", not during the long run.
Outside the job: DCGM, dmesg Xids, nvidia-smi throttle reasons, link counters, BMC sensors (section 4).

7.8 What to watch, and when it passes

Watch	Healthy	Problem
Step time / throughput (tokens/s, TFLOPS per GPU)	Flat, close to the reference for that model and scale	Gradual increase (throttling, a degrading link), random spikes between checkpoints
Checkpoint steps	A predictable spike every N steps while the checkpoint writes	Spikes that grow, or checkpoint writes that fail (storage)
Loss	Finite, behaving the same way every run	NaN, sudden jumps
Job	Runs to the planned step count	NCCL watchdog timeout, a hang, a rank exiting

Checklist

Stack frozen and recorded; container staged on shared storage.
Dry run reviewed: TP inside a node, GA a whole number, mock data, checkpoint interval, step count sized to the time limit.
Single-node runs on every node, within a few percent of each other.
Scaling sequence passed; reference config compared with the published row like for like.
Long run complete: flat step time, finite loss, steady checkpoint durations, no Xids, ECC errors, or link errors.
Resume test passed.
Results archived: config, container tag, logs, step-time trend, checkpoint durations.

Training runs for the full period without hangs, NCCL timeouts, or node failures; step time and throughput are stable and near the like-for-like reference; loss is finite; checkpoints save in steady time, and a restart from one continues where it left off.

7.9 NeMo practice set ( questions)

8. NeMo question answering: the BERT SQuAD fine-tune

A smaller, well-known NeMo workload is extractive question answering: fine-tune a BERT-style model on SQuAD (Stanford Question Answering Dataset) and score it. Because the expected accuracy is published, it checks that the stack produces the right results, not just fast ones. It's a quick end-to-end check that complements the long pretraining run.

# NeMo 1.x example script and dataset helper
python NeMo/examples/nlp/question_answering/get_squad.py --destDir /data/squad   # train-v1.1.json, dev-v1.1.json
srun -N 1 --ntasks-per-node=8 --gpus-per-node=8 --container-image=nvcr.io#nvidia/nemo:<tag> \
  python NeMo/examples/nlp/question_answering/question_answering.py \
    do_training=true \
    model.train_ds.file=/data/squad/v1.1/train-v1.1.json \
    model.validation_ds.file=/data/squad/v1.1/dev-v1.1.json \
    model.language_model.pretrained_model_name=bert-large-uncased \
    trainer.devices=8 trainer.precision=16 trainer.max_epochs=2

Metric	Meaning	Reference (BERT-large, SQuAD v1.1)
EM (exact match)	Percentage of answers exactly equal to a reference answer	Roughly 84
F1	Word-overlap score between the predicted and reference answers	Roughly 91

Older releases used question_answering_squad.py launched with python -m torch.distributed.launch --nproc_per_node=8; NGC also hosts pretrained QA checkpoints (for example qa_squadv1_1_bertlarge) you can evaluate instead of training.
SQuAD v2.0 adds unanswerable questions, so its scores are lower; compare against the right version.

The fine-tune completes on all GPUs, throughput is consistent run to run, and EM/F1 land near the published reference for that model and SQuAD version. A run that's fast but scores far lower points at a broken stack (wrong precision settings, data, or silent errors).

The reference numbers are the original BERT paper's ballpark for BERT-large on SQuAD v1.1 and vary a little with the recipe and number of epochs. Use the NeMo model card for your checkpoint when you need an exact target.

9. When something fails mid burn-in

Keep the evidence: job log, the failing rank or node, dmesg, DCGM health, link counters, BMC event log. Run nvidia-bug-report.sh or nvsm dump health if you'll open a support case.
Drain the node in Slurm with a reason, so the rest of the burn-in continues without it: scontrol update nodename=dgx05 state=drain reason="burnin: Xid 79 GPU5".
Repair (reseat or replace the part), then retest the node alone: dcgmi diag -r 3 (or -r 4), a single-node HPL or NCCL run.
Resume it and rerun the stage it failed in, so the repaired node gets its full burn-in time.
Report: what ran, for how long, the results against the criteria, every failure with its fix. That report is the acceptance evidence.

Rerunning a failed burn-in without changing anything is not a fix. If it passes the second time, you've only learned the fault is intermittent: the node still needs the diagnosis.

10. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

dcgmi health -s a && dcgmi health -cEnable all health watches, then check

Health monitor systems set successfully.
+---------------------------+----------------------------------------------------------+
| Health Monitor Report                                                                |
+===========================+==========================================================+
| Overall Health            | Healthy                                                  |
+---------------------------+----------------------------------------------------------+

What to look at: Set the watches once before the burn-in starts, then re-check during and after it. Anything other than Healthy (for example a Memory or PCIe Warning naming a GPU) is a zero-tolerance failure for that node, even if the job finished.

dcgmi dmon -e 150,155,203,1002 -d 5000Sample temp, power, util, SM activity

#Entity   TMPTR  POWER    GPUTL  SMACT
ID        C      W        %
GPU 0     63     688.2    100    0.982
GPU 1     64     691.5    100    0.980
GPU 2     62     686.9    100    0.983
GPU 3     64     690.3    100    0.979
GPU 4     63     689.7    100    0.981
GPU 5     88     612.4    100    0.904
GPU 6     63     690.8    100    0.982
GPU 7     62     687.1    100    0.981
...
(one block every 5 s until Ctrl+C)

What to look at: Under HPL or NeMo load every GPU should sit at a similar temperature and close to its 700 W power limit. GPU 5 here is 25 C hotter and drawing less power, the signature of thermal throttling from a failed fan or blocked airflow.

pdsh -w dgx[01-32] "dmesg -T | grep -i xid"Search every node's kernel log for Xids

dgx05: [Tue Nov 10 13:47:05 2026] NVRM: Xid (PCI:0000:c3:00): 79, pid=48122, name=python, GPU has fallen off the bus.
dgx05: [Tue Nov 10 13:47:05 2026] NVRM: GPU 0000:c3:00.0: GPU has fallen off the bus.

What to look at: No output from a node means no Xids, which is what you want. Any line is prefixed with the node name: Xid 79 means the GPU fell off the bus, 74 is NVLink, and 48, 63, 64, 94, or 95 are memory errors, all of which fail the burn-in for that node.

ibdiagnet -pcClear fabric counters and check the fabric

Loading IBDIAGNET from: /usr/lib/x86_64-linux-gnu/ibdiagnet2.1.1
-------------------------------------------------
ibdiagnet v2.13.0 (build 2401151405)
...
Discovery
-I- Discovering ... 299 nodes (42 Switch(es) & 257 CA(s)) discovered.
-I- Fabric Discover finished successfully

Lids Check
-I- Lids Check finished successfully

Links Check
-I- Links Check finished successfully
...
Port Counters
-I- Ports counters were cleared
...
Summary
-I- Stage                     Warnings   Errors     Comment
-I- Discovery                 0          0
-I- Lids Check                0          0
-I- Links Check               0          0
-I- Subnet Manager            0          0
-I- Port Counters             0          0
-I- Nodes Information         0          0
-I- Speed / Width checks      0          0
-I- Virtualization            0          0
-I- Partition Keys            0          0
-I- Alias GUIDs               0          0

-I- You can find detailed errors/warnings in: /var/tmp/ibdiagnet2/ibdiagnet2.log

What to look at: Check that the discovered CA count matches the fabric (32 nodes x 8 compute HCAs plus the head node here) and that every stage shows 0 errors. Run ibdiagnet again after the burn-in: counters that were 0 at the start and are now non-zero point to the links that degraded under load.

srun -N 32 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix ./build/all_reduce_perf -b 8G -e 16G -f 2 -g 1 -n 100 -c 1 | tee -a nccl-burnin.log

One NCCL AllReduce burn-in iteration, 256 GPUs

# nThread 1 nGpus 1 minBytes 8589934592 maxBytes 17179869184 step: 2(factor) warmup iters: 5 iters: 100 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank   0 Group  0 Pid  41220 on      dgx01 device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank   1 Group  0 Pid  41221 on      dgx01 device  1 [0x43] NVIDIA H100 80GB HBM3
#  Rank   2 Group  0 Pid  41222 on      dgx01 device  2 [0x52] NVIDIA H100 80GB HBM3
#  ...
#  Rank 255 Group  0 Pid  40987 on      dgx32 device  7 [0xdf] NVIDIA H100 80GB HBM3
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
  8589934592    2147483648     float     sum      -1    45890  187.19  372.91      0    45837  187.40  373.34      0
 17179869184    4294967296     float     sum      -1    91635  187.48  373.50      0    91581  187.59  373.72      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 373.37 
#

What to look at: Check that #wrong is 0 in both the out-of-place and in-place columns and that large-size busbw matches the validation baseline (about 372 GB/s here). Any #wrong above 0 is data corruption, which means hardware, never tuning.

grep -E "Avg bus bandwidth|#wrong|WARN|error" nccl-burnin.logPull the per-iteration results from the log

#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
# Avg bus bandwidth    : 373.37
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
# Avg bus bandwidth    : 372.84
...
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
# Avg bus bandwidth    : 372.95
dgx14:48211:48390 [3] NCCL WARN NET/IB : Got async event : port error
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
# Avg bus bandwidth    : 331.08
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
# Avg bus bandwidth    : 344.61
(output trimmed)

What to look at: Read the Avg bus bandwidth values top to bottom as a trend. Here they hold near 373 GB/s, then drop to 331-345 right after an IB port error on dgx14 GPU 3. That fails the burn-in even if the 24-hour average looks close to baseline; split the test by rail to find the link.

srun ... ./build/all_reduce_perf -b 8G -e 16G -f 2 -g 1 -N 0Repeat AllReduce cycles until the time limit

# nThread 1 nGpus 1 minBytes 8589934592 maxBytes 17179869184 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 run cycles: 0
...
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
  8589934592    2147483648     float     sum      -1    45893  187.17  372.88      0    45854  187.33  373.20      0
 17179869184    4294967296     float     sum      -1    91644  187.46  373.46      0    91608  187.54  373.61      0
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
  8589934592    2147483648     float     sum      -1    45909  187.11  372.75      0    45865  187.29  373.11      0
 17179869184    4294967296     float     sum      -1    91662  187.43  373.39      0    91615  187.52  373.58      0
...  (the table repeats once per cycle)
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
  8589934592    2147483648     float     sum      -1    45903  187.13  372.80      0    45860  187.31  373.15      0
 17179869184    4294967296     float     sum      -1    91649  187.45  373.44      0    91610  187.53  373.60      0
slurmstepd: error: *** STEP 5102.0 ON dgx01 CANCELLED AT 2026-11-11T09:00:04 DUE TO TIME LIMIT ***

What to look at: With -N 0 there's no final summary: each cycle prints its own table until Slurm cancels the step at the time limit, so the cancel message is expected. Compare the busbw of the first and last cycles, and treat any #wrong above 0 or a missing cycle (a hang) as a failure.

mlxlink -d mlx5_3 -cShow link state and BER counters

Operational Info
----------------
State                              : Active
Physical state                     : LinkUp
Speed                              : IB-NDR
Width                              : 4x
FEC                                : Interleaved_Standard_RS_FEC_PLR - (544,514)
Loopback Mode                      : No Loopback
Auto Negotiation                   : ON

Supported Info
--------------
Enabled Link Speed (Ext.)          : 0x00000020 (NDR)
Supported Cable Speed (Ext.)       : 0x00000030 (NDR,HDR)

Troubleshooting Info
--------------------
Status Opcode                      : 0
Group Opcode                       : N/A
Recommendation                     : No issue was observed

Tool Information
----------------
Firmware Version                   : 28.39.1002
amBER Version                      : 2.22
MFT Version                        : mft 4.26.1-3

Physical Counters and BER Info
------------------------------
Time Since Last Clear [Min]        : 1440.2
Effective Physical Errors          : 41877
Effective Physical BER             : 3E-10
Raw Physical Errors Per Lane       : 1023441,998213,1100432,1302214
Raw Physical BER                   : 2E-5
Link Down Counter                  : 2
Link Error Recovery Counter        : 57

What to look at: A healthy NDR link reads Active, 4x, IB-NDR, with Effective Physical Errors 0 and an Effective BER around 1E-15 or lower. Here the effective BER is 3E-10 and the link went down twice during the run, so this cable or optic is the fault even though the link is up now.

mlxlink -d mlx5_3 -mShow transceiver module info

Operational Info
----------------
State                              : Active
Physical state                     : LinkUp
Speed                              : IB-NDR
Width                              : 4x
...
Module Info
-----------
Identifier                         : OSFP
Compliance                         : N/A
Cable Technology                   : N/A
Cable Type                         : Optical Module (separated)
OUI                                : Other
Vendor Name                        : NVIDIA
Vendor Part Number                 : MMA4Z00-NS
Vendor Serial Number               : MT0000LAB0003
Rev                                : A6
Wavelength [nm]                    : 850
Transfer Distance [m]              : 50
FW Version                         : 47.120.0026
Digital Diagnostic Monitoring      : Yes
Power Class                        : 8
CDR RX                             : ON,ON,ON,ON
CDR TX                             : ON,ON,ON,ON
Temperature [C]                    : 71 [-5..75]
Voltage [mV]                       : 3277.4 [2970..3630]
Bias Current [mA]                  : 8.2,8.2,8.3,8.2 [2..15]
Rx Power Current [dBm]             : -7.9,-2.1,-2.0,-2.2 [-10.0..5.0]
Tx Power Current [dBm]             : 0.8,0.9,0.8,0.8 [-8.2..5.0]

What to look at: Compare each reading with the range in brackets and with the other lanes. Lane 1's Rx power (-7.9 dBm against about -2 on the others) and a module at 71 C, near its 75 C limit, point to a dirty connector or failing optic that gets worse as it heats up.

srun -N 1 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix -w dgx03 --container-image=nvcr.io#nvidia/hpc-benchmarks:<tag> hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-8GPUs.dat

Single-node HPL run on dgx03

pyxis: importing docker image: nvcr.io#nvidia/hpc-benchmarks:24.03
pyxis: imported docker image: nvcr.io#nvidia/hpc-benchmarks:24.03
...
================================================================================
HPL-NVIDIA 24.3.0  -- NVIDIA accelerated HPL benchmark -- NVIDIA
================================================================================
...
N        :  270336
NB       :    1024
PMAP     : Column-major process mapping
P        :       2
Q        :       4
...
Prog= 2.27%     N_left= 267264  Time= 0.71    Time_left= 30.62  iGF= 425307.84  GF= 425307.84  iGF_per= 53163.48  GF_per= 53163.48
...
================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR01C2R4      270336  1024     2     4              30.55              4.312e+05
--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        3.21564841e-03 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

End of Tests.
================================================================================

What to look at: Look at two things: the Gflops value on the WR line (about 4.3e+05 for an 8-GPU H100 node here) and PASSED on the residual line. A FAILED residual means wrong math and points to hardware; a PASSED run that is 5% or more below the other nodes is an outlier to fix before scaling up.

srun -N 16 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix --container-image=nvcr.io#nvidia/hpc-benchmarks:<tag> hpl.sh --dat /dat/HPL-128GPUs.dat

HPL across 16 nodes (128 GPUs)

...
N        : 1081344
NB       :    1024
P        :       8
Q        :      16
...
================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR01C2R4     1081344  1024     8    16             128.67              6.551e+06
--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        2.87411093e-03 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.

What to look at: Divide the Gflops by the node count to get per-node performance: 4.09e+05 here, about 95% of the single-node number, which is normal at this scale. A sudden drop at one step of the sequence points at the fabric between the nodes that were just added, or at a slow node in the job.

grep -E "WR|PASSED|FAILED" hpl-*.outCompare HPL results across node logs

hpl-dgx01.out:WR01C2R4      270336  1024     2     4              30.55              4.312e+05
hpl-dgx01.out:||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        3.20564841e-03 ...... PASSED
hpl-dgx02.out:WR01C2R4      270336  1024     2     4              30.64              4.298e+05
hpl-dgx02.out:||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        3.21564841e-03 ...... PASSED
hpl-dgx03.out:WR01C2R4      270336  1024     2     4              34.74              3.791e+05
hpl-dgx03.out:||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        3.22564841e-03 ...... PASSED
hpl-dgx04.out:WR01C2R4      270336  1024     2     4              30.48              4.321e+05
hpl-dgx04.out:||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        3.23564841e-03 ...... PASSED
hpl-dgx05.out:WR01C2R4      270336  1024     2     4              30.59              4.305e+05
hpl-dgx05.out:||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        3.24564841e-03 ...... PASSED
hpl-dgx06.out:WR01C2R4      270336  1024     2     4              30.51              4.317e+05
hpl-dgx06.out:||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        3.25564841e-03 ...... PASSED
hpl-dgx07.out:WR01C2R4      270336  1024     2     4              30.62              4.301e+05
hpl-dgx07.out:||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        3.26564841e-03 ...... PASSED
hpl-dgx08.out:WR01C2R4      270336  1024     2     4              30.57              4.309e+05
hpl-dgx08.out:||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        3.27564841e-03 ...... PASSED

What to look at: Every node must say PASSED, and the Gflops should be within a few percent of each other. dgx03 passed but is about 12% slower than its peers, so check its throttle reasons and fans before any multi-node run includes it.

nvidia-smi -q -d PERFORMANCE,TEMPERATUREShow throttle reasons and GPU temperatures

==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 11:42:08 2026
Driver Version                            : 550.54.15
CUDA Version                              : 12.4

Attached GPUs                             : 8
GPU 00000000:1B:00.0
    Temperature
        GPU Current Temp                  : 62 C
        GPU T.Limit Temp                  : 25 C
        GPU Shutdown T.Limit Temp         : -5 C
        GPU Slowdown T.Limit Temp         : -2 C
        GPU Max Operating T.Limit Temp    : 0 C
        Memory Current Temp               : 55 C
    Performance State                     : P0
    Clocks Event Reasons
        Idle                              : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
        SW Thermal Slowdown               : Not Active
...
GPU 00000000:C3:00.0
    Temperature
        GPU Current Temp                  : 88 C
        GPU T.Limit Temp                  : -3 C
        GPU Shutdown T.Limit Temp         : -5 C
        GPU Slowdown T.Limit Temp         : -2 C
        GPU Max Operating T.Limit Temp    : 0 C
        Memory Current Temp               : 81 C
    Performance State                     : P0
    Clocks Event Reasons
        Idle                              : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Active
            HW Thermal Slowdown           : Active
        SW Thermal Slowdown               : Active
...

What to look at: GPU T.Limit Temp is the headroom below the maximum operating temperature: a large positive number is good, 0 or below means the GPU is at its limit, and past the Slowdown value (-2 here) the hardware throttles. GPU C3 (GPU 5) is at -3 and HW Thermal Slowdown Active, which explains a slow HPL node.

ipmitool sdr type fanRead fan sensors from the BMC

FAN1_F           | 30h | ok  | 29.1 | 11760 RPM
FAN1_R           | 31h | ok  | 29.1 | 10920 RPM
FAN2_F           | 32h | ok  | 29.2 | 11760 RPM
FAN2_R           | 33h | ok  | 29.2 | 10920 RPM
FAN3_F           | 34h | ok  | 29.3 | 11900 RPM
FAN3_R           | 35h | ok  | 29.3 | 11060 RPM
FAN4_F           | 36h | ok  | 29.4 | 11760 RPM
FAN4_R           | 37h | ok  | 29.4 | 10920 RPM
FAN5_F           | 38h | ok  | 29.5 | 11620 RPM
FAN5_R           | 39h | ok  | 29.5 | 10780 RPM
FAN6_F           | 3Ah | cr  | 29.6 | 0 RPM
FAN6_R           | 3Bh | cr  | 29.6 | 0 RPM

What to look at: The third column is the sensor status: ok is normal, while cr (critical) or nr (non-recoverable) needs action. FAN6 reads 0 RPM with status cr, so that fan module has failed and is starving the GPUs behind it of airflow.

srun -N 8 --ntasks-per-node=8 --gpus-per-node=8 --container-image=nvcr.io#nvidia/nemo:<tag> --container-mounts=/lustre/ckpt:/ckpt nemo llm pretrain -y --factory llama3_8b trainer.num_nodes=8 trainer.max_steps=5000

NeMo Llama 3 8B pretraining burn-in

pyxis: imported docker image: nvcr.io#nvidia/nemo:24.07
...
[NeMo I 2026-11-10 10:02:11 nemo_logging:393] Experiments will be logged at /ckpt/llama3_8b/default
GPU available: True (cuda), used: True
...
[NeMo I 2026-11-10 10:02:47 nemo_logging:393] Rank 0 has data parallel group : [0, 2, 4, ..., 62]
[NeMo I 2026-11-10 10:03:30 nemo_logging:393] Padded vocab_size: 128256, original vocab_size: 128256, dummy tokens: 0.
...
Training epoch 0, iteration 99/4999 | lr: 2.97e-05 | global_batch_size: 128 | global_step: 99 | reduced_train_loss: 8.912 | train_step_timing in s: 2.104 | consumed_samples: 12800
Training epoch 0, iteration 100/4999 | lr: 3e-05 | global_batch_size: 128 | global_step: 100 | reduced_train_loss: 8.874 | train_step_timing in s: 2.101 | consumed_samples: 12928
...
Training epoch 0, iteration 499/4999 | lr: 0.0001497 | global_batch_size: 128 | global_step: 499 | reduced_train_loss: 6.215 | train_step_timing in s: 2.098 | consumed_samples: 64000
[NeMo I 2026-11-10 10:21:19 nemo_logging:393] Global Checkpoint Save : Rank: 0 : Iteration: 499 : Start time: 1794053679.112s : Save duration: 37.882s
Training epoch 0, iteration 500/4999 | lr: 0.00015 | global_batch_size: 128 | global_step: 500 | reduced_train_loss: 6.207 | train_step_timing in s: 2.103 | consumed_samples: 64128
...
Training epoch 0, iteration 4999/4999 | lr: 0.0003 | global_batch_size: 128 | global_step: 4999 | reduced_train_loss: 3.402 | train_step_timing in s: 2.102 | consumed_samples: 640000
`Trainer.fit` stopped: `max_steps=5000` reached.

What to look at: train_step_timing should stay flat (about 2.1 s here) for all 5000 steps, with a slower step only at each checkpoint save, and the save duration should not grow. Loss must stay finite; a NaN loss, a creeping step time, or an NCCL watchdog timeout fails the burn-in.

python NeMo/examples/nlp/question_answering/get_squad.py --destDir /data/squadDownload the SQuAD v1.1 and v2.0 datasets

Downloading: https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
Downloading: https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
Downloading: https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json
Downloading: https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json

$ ls /data/squad/*
/data/squad/v1.1:
dev-v1.1.json  train-v1.1.json

/data/squad/v2.0:
dev-v2.0.json  train-v2.0.json

What to look at: Confirm both versions landed in separate v1.1 and v2.0 directories, and point the fine-tune at the v1.1 files when you compare against the BERT-large v1.1 reference (about 84 EM and 91 F1). A download error usually means the node has no outbound internet access; stage the files from another host.

srun -N 1 --ntasks-per-node=8 --gpus-per-node=8 --container-image=nvcr.io#nvidia/nemo:<tag> python NeMo/examples/nlp/question_answering/question_answering.py do_training=true model.train_ds.file=/data/squad/v1.1/train-v1.1.json model.validation_ds.file=/data/squad/v1.1/dev-v1.1.json model.language_model.pretrained_model_name=bert-large-uncased trainer.devices=8 trainer.precision=16 trainer.max_epochs=2

BERT-large SQuAD v1.1 fine-tune on one node

[NeMo I 2026-11-10 15:01:42 exp_manager:396] Experiments will be logged at /results/QA/2026-11-10_15-01-42
...
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/8
...
Epoch 0: 100%|##########| 1386/1386 [21:12<00:00, 1.09it/s, loss=1.21, v_num=1-42]
Validating: 100%|##########| 166/166 [01:12<00:00, 2.29it/s]
[NeMo I 2026-11-10 15:24:08 qa_model:186] val exact match 83.12
[NeMo I 2026-11-10 15:24:08 qa_model:187] val f1 90.31
Epoch 1: 100%|##########| 1386/1386 [21:09<00:00, 1.09it/s, loss=0.78, v_num=1-42]
Validating: 100%|##########| 166/166 [01:11<00:00, 2.33it/s]
[NeMo I 2026-11-10 15:46:31 qa_model:186] val exact match 84.07
[NeMo I 2026-11-10 15:46:31 qa_model:187] val f1 90.92
`Trainer.fit` stopped: `max_epochs=2` reached.

What to look at: The final exact match and F1 should land near the published BERT-large SQuAD v1.1 reference (about 84 and 91). A run that finishes quickly but scores far lower, for example EM 12, means the stack is producing wrong results: check the data files, precision settings, and the log for errors.

scontrol update nodename=dgx05 state=drain reason="burnin: Xid 79 GPU5" && sinfo -RDrain a failed node with a reason

REASON               USER      TIMESTAMP           NODELIST
burnin: Xid 79 GPU5  root      2026-11-10T14:05:33 dgx05

What to look at: scontrol prints nothing on success, so sinfo -R is the proof the drain took effect. Slurm refuses a drain without a reason, and the reason is what the next shift sees, so name the fault and the part.

dcgmi diag -r 3Retest the repaired node's GPUs

Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 3.3.5                                          |
| Driver Version Detected   | 550.54.15                                      |
| GPU Device IDs Detected   | 2330,2330,2330,2330,2330,2330,2330,2330        |
|-----  Deployment  --------+------------------------------------------------|
| Denylist                  | Pass                                           |
| NVML Library              | Pass                                           |
| CUDA Main Library         | Pass                                           |
| Permissions and OS Blocks | Pass                                           |
| Persistence Mode          | Pass                                           |
| Environment Variables     | Pass                                           |
| Page Retirement/Row Remap | Pass                                           |
| Graphics Processes        | Pass                                           |
| Inforom                   | Pass                                           |
+-----  Integration  -------+------------------------------------------------+
| PCIe                      | Pass - All                                     |
+-----  Hardware  ----------+------------------------------------------------+
| GPU Memory                | Pass - All                                     |
| Diagnostic                | Pass - All                                     |
+-----  Stress  ------------+------------------------------------------------+
| Targeted Stress           | Pass - All                                     |
| Targeted Power            | Pass - All                                     |
| Memory Bandwidth          | Pass - All                                     |
| EUD Test                  | Skip - All                                     |
+---------------------------+------------------------------------------------+

What to look at: All eight GPU device IDs must be detected and every test must read Pass - All before you resume the node; Skip on EUD is normal when that plugin isn't installed. A Fail names the GPU, and after a repair the node then reruns the burn-in stage it failed in.

11. Hands-on lab: simulated tickets

Three tickets on a simulated 8-node DGX H100 cluster, run from the head node in ~/burnin. Each one is a burn-in stage that finds a different hidden fault:

BRN-101 (4.11): a 24-hour NCCL burn-in whose average looks fine. Read the trend, isolate the rail, find the link, repair it, rerun.
HPL-202 (4.12): the full single-node → N/2 → two halves → full sequence, with an outlier node to fix first.
NMO-303 (4.13): a NeMo pretraining job dies with NCCL timeouts. Map the rank to a node, prove the fault, repair, rerun, then check question-answering accuracy.

Outputs are simplified mock-ups modeled on the real tools, not exact copies, and every name and number is a placeholder. Jobs finish instantly. dctech <node> <part> is a lab-only command that stands in for a datacenter technician. Up/Down recall history; Tab completes.

Ticket

Ticket checklist

Answer key for this ticket (spoilers)

One clean path; other orders work too. The checklist tracks results, not exact commands.

12. Scenarios

13. Searchable reference

Item	Meaning

14. Flashcards (click to flip)

15. Exam traps in one list

Validation = one run; burn-in = repeated runs over hours. Burn-in comes after validation passes.
Judge the trend, not the average. A 24-hour average a few percent low can hide a link that failed at hour 7.
Finishing isn't passing. Check Xids, ECC and row remaps, link counters, and throttling too.
Order: single node on every node → 2, 4, … N/2 → two N/2 jobs at once → all N. Fix outliers before scaling.
Two halves at once loads shared switches, power, and cooling together; one full job is the last step.
NCCL burn-in: large sizes, many iterations, -c 1, several collectives. -N 0 repeats until the time limit.
Any #wrong above 0 or residual FAILED is hardware, never tuning.
HPL burn-in is the power and cooling test. A PASSED but 10% slow node is still a failure: look at throttle reasons and fans.
NeMo burn-in uses mock data so storage doesn't hide compute or fabric problems; checkpoint spikes every N steps are expected.
Rank → node: rank ÷ 8 = node index, rank mod 8 = GPU (8 ranks per node, node-list order).
The rank that errors first is the suspect; the many ranks with watchdog timeouts were waiting for it.
Xid 79 = GPU fell off the bus. Drain, reseat or replace, dcgmi diag -r 3, then rerun.
NeMo QA = BERT on SQuAD, scored by EM and F1 (about 84 / 91 for BERT-large on v1.1). It checks correctness, not just speed.
Keep TP inside the NVLink domain (≤ 8 GPUs on a DGX H100); PP and DP cross nodes.
DP = GPUs ÷ (TP × PP × CP); GA = GBS ÷ (MBS × DP); tokens/s/GPU = GBS × seq ÷ (step time × GPUs).
Compare throughput like for like: same container, precision, GPU count, GBS, sequence length, and parallelism. NVIDIA's published tables are FP8.
OOM on every node at step 1 is config; OOM on one node only is that node.
NET/Socket in the NCCL log means no InfiniBand: the job runs, slowly, and the burn-in proves nothing about the fabric.
A resume that shows a loss near its starting value restarted from scratch. Resume must continue global_step and consumed_samples.
Straggler score below 0.7 = straggler; one slow GPU slows the whole synchronous job.
Drain with a reason before repair, resume after proof, and rerun the stage the node failed in.
A rerun that passes without a fix proves nothing: the fault is intermittent and still there.

16. Practice quiz (20 questions)

Score: 0 / 0