GPU drivers, DOCA-Host, and the Container Toolkit

The node software stack below every workload: installing, upgrading, and removing the NVIDIA GPU driver with Fabric Manager and nvidia-peermem, replacing MLNX_OFED with DOCA-Host (DOCA-OFED), and giving containers GPUs with the NVIDIA Container Toolkit for Docker, containerd, and Podman (CDI). Includes the version-matching rules, doing all of it inside BCM images, and a troubleshooting table of the real error messages. Ends with three hands-on simulated tickets. Goes deeper than the bring-up guide. Objectives 3.4, 3.5, and 3.6.

1. The stack, bottom to top

Layer	Packages / services	Must match
Kernel	`linux-image-…`, `linux-headers-…`	Headers must match the running kernel (`uname -r`), because DKMS builds the GPU and network modules against them
Network driver	DOCA-OFED (from DOCA-Host), formerly MLNX_OFED: `mlx5_core`, `mlx5_ib`, `ib_core`	The kernel; NIC firmware within the supported range
GPU driver	`nvidia-driver-<branch>-server-open`: `nvidia`, `nvidia_uvm`, `nvidia_modeset`, `nvidia_drm`, `nvidia_peermem`	The kernel; `nvidia_peermem` must be built against the installed network stack's `ib_core`
NVSwitch control	`nvidia-fabricmanager-<branch>` (HGX/DGX with NVSwitch); on newer systems also NVLink Subnet Manager pieces	Exactly the same version as the driver
GPU services	`nvidia-persistenced`, DCGM (`nvidia-dcgm`)	Driver branch supported by that DCGM release
Containers	NVIDIA Container Toolkit (`nvidia-container-toolkit`, `nvidia-ctk`), Docker / containerd / CRI-O / Podman	Works with any recent driver; the container's CUDA must be supported by the host driver

The stack is a set of nesting adapters. The kernel is the wall socket; DKMS shapes the GPU and network plugs to fit it. nvidia_peermem is a little bridge plug that joins the GPU adapter to the network adapter. Replace the network adapter, and the bridge no longer fits until it's reshaped (rebuilt).

Install order on a fresh node: kernel and headers → DOCA-OFED → GPU driver + Fabric Manager → Container Toolkit. Installing the network stack before the GPU driver means nvidia_peermem is built against the right ib_core the first time.

2. GPU driver: install, upgrade, remove (3.4)

Install

apt install -y linux-headers-$(uname -r)                      # DKMS needs them
apt install -y nvidia-driver-550-server-open nvidia-fabricmanager-550
apt-mark hold nvidia-driver-550-server-open nvidia-fabricmanager-550
systemctl enable --now nvidia-fabricmanager nvidia-persistenced
reboot                                                         # cleanest way to load new modules
nvidia-smi                                                     # driver version, 8 GPUs
nvidia-smi -q | grep -A2 Fabric                                # State: Completed, Status: Success
systemctl status nvidia-fabricmanager
lsmod | grep nvidia_peermem                                    # GPUDirect RDMA module loaded

Open kernel modules (-open packages) are the recommended flavor for Hopper and required for Blackwell; the proprietary flavor still exists for older GPUs. Don't mix the two.
Server branches (-server) are the data-center builds with long support. Pick a branch your CUDA, DCGM, and NCCL versions support.
Don't mix install methods: a .run installer on top of distribution packages (or the reverse) leaves two sets of libraries and breaks on the next update.
Blacklist nouveau (the packages do this) and keep persistence mode on (nvidia-persistenced), so the driver stays initialized between jobs.

Upgrade

Drain the node and make sure no process uses the GPUs (nvidia-smi shows no processes).
Stop GPU services: systemctl stop nvidia-fabricmanager nvidia-persistenced nvidia-dcgm (and DCGM exporter, any monitoring agent holding the GPUs).
Install the new driver and the matching Fabric Manager together. If they're held, apt-mark unhold first (a new branch is a different package name, so the old hold doesn't block it).
Hold the new packages, reboot, and verify as above. Until the reboot, nvidia-smi says Driver/library version mismatch: the new libraries are installed but the old module is loaded. That's expected, not a failure.

Remove

systemctl stop nvidia-fabricmanager nvidia-persistenced nvidia-dcgm
apt purge '^nvidia-.*' '^libnvidia-.*'        # packages (check the list before confirming)
nvidia-uninstall                              # only if it was installed with the .run file
reboot

Fabric Manager must be exactly the driver version. Upgrading only the driver leaves Fabric Manager failing to start ("interface version … don't match with driver version"), and CUDA apps on NVSwitch systems fail with error 802, "system not yet initialized". Upgrade both, hold both.

On a new kernel, DKMS rebuilds the GPU modules automatically if the matching headers are installed. If they aren't, the node boots with no NVIDIA module ("NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver"). Pin the kernel version along with the driver.

3. DOCA-Host and DOCA-OFED (3.4)

DOCA-Host is NVIDIA's host-side networking software package for ConnectX and BlueField. It replaces the standalone MLNX_OFED: its doca-ofed profile installs the same kind of stack (drivers, ibstat, perftest, firmware tools integration). DOCA-Host ships as a small repository package; apt then installs a profile:

Profile	Contents
`doca-ofed`	The MLNX_OFED replacement: kernel modules, RDMA user space, tools. The usual choice for GPU compute nodes.
`doca-roce` / `doca-networking`	Smaller or networking-focused subsets
`doca-all`	Everything, including DOCA libraries for BlueField development
`*-userspace` variants, `doca-kernel`	User space only (for containers or inbox kernel drivers) / kernel modules only

# Install (Ubuntu); kernel headers for the running kernel first
apt install -y linux-headers-$(uname -r)
dpkg -i doca-host_<version>_<distro>_amd64.deb       # adds the local DOCA repository
apt-get update
apt install -y doca-ofed mlnx-fw-updater               # mlnx-fw-updater: NIC firmware from the bundle
/etc/init.d/openibd restart                            # or reboot

# Verify
ofed_info -s                                           # OFED-internal-24.10-… with DOCA-OFED (MLNX_OFED_LINUX-… before)
ibstat | grep -E "State|Rate"                          # Active, 400
ibv_devinfo | grep -E "hca_id|state"
lsmod | grep -E "mlx5_core|nvidia_peermem"

Upgrade or replace

From MLNX_OFED: remove it first with /usr/sbin/ofed_uninstall.sh --force, then install DOCA-Host. The two can't be installed on top of each other.
From an older DOCA: NVIDIA's procedure is to remove the DOCA packages first (a loop over dpkg --list entries matching doca, flexio, dpa-…, purging each), run ofed_uninstall.sh --force if present, apt-get autoremove, reboot, then install the new repo package and profile. A plain apt upgrade <profile> isn't enough when extra DOCA packages were added.
After any network stack change, rebuild nvidia_peermem: it was compiled against the old ib_core. Reinstall the driver's DKMS package (apt install --reinstall nvidia-dkms-<branch>-server-open) or force it (dkms install --force nvidia/<version>). dkms autoinstall isn't enough: it skips modules already installed for the running kernel. Symptom if you forget: modprobe nvidia-peermem fails, and dmesg says "disagrees about version of symbol ib_register_peer_memory_client". NCCL then runs without GPUDirect RDMA and multi-node busbw drops.

New network stack → rebuild peermem. GPUDirect RDMA is the bridge between the two drivers; change one side and rebuild the bridge.

4. NVIDIA Container Toolkit (3.5)

Containers don't ship the driver. The toolkit's job is to inject the host's driver libraries and GPU device nodes into a container at start time, so the CUDA in the image can talk to the host driver.

# Repository (once) and install
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt-get update
apt install -y nvidia-container-toolkit     # pulls nvidia-container-toolkit-base, libnvidia-container-tools, libnvidia-container1
# pin all four with NVIDIA_CONTAINER_TOOLKIT_VERSION=... when the fleet must match

Runtime	Configure	Then
Docker	`nvidia-ctk runtime configure --runtime=docker` (edits `/etc/docker/daemon.json`)	`systemctl restart docker`
containerd (Kubernetes)	`nvidia-ctk runtime configure --runtime=containerd` (writes a drop-in under `/etc/containerd/conf.d/`)	`systemctl restart containerd`
CRI-O	`nvidia-ctk runtime configure --runtime=crio`	`systemctl restart crio`
Podman, or any CDI-aware runtime	CDI: `nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml`	`nvidia-ctk cdi list`; regenerate after a driver change or GPU swap
Rootless Docker	`nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json`, plus `nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place`	Restart the user's Docker daemon

CDI (Container Device Interface) is a standard file that lists devices (nvidia.com/gpu=0 … =all) and what to mount for each. Podman uses it natively; Docker 25+ and containerd can use it too. It doesn't need a special runtime, just the spec file.

On Kubernetes, the NVIDIA GPU Operator installs the driver (optionally), the toolkit, the device plugin, and DCGM exporter as pods, instead of you installing them on each node. Same pieces, managed for you.

5. Using GPUs with Docker (3.6)

docker run --rm --gpus all ubuntu nvidia-smi -L                       # every GPU
docker run --rm --gpus 2 ubuntu nvidia-smi -L                         # any two GPUs
docker run --rm --gpus '"device=2,3"' ubuntu nvidia-smi -L            # GPUs 2 and 3 (note the quoting)
docker run --rm --gpus device=GPU-5f1c2a2e-... ubuntu nvidia-smi -L   # by UUID
docker run --rm --gpus '"device=0:2"' ubuntu nvidia-smi -L           # MIG: device 2 on GPU 0 (GPU:MIG index)
docker run --rm --gpus '"device=0:0,0:1"' ubuntu nvidia-smi -L       # MIG: two devices on GPU 0
docker run --rm --gpus device=MIG-1c2b9e4f-... ubuntu nvidia-smi -L   # MIG: by MIG UUID
docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=2,3 ubuntu nvidia-smi -L   # older style
docker run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L     # CDI (Docker 25+ with CDI enabled)
podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L     # Podman: CDI only

# Multi-GPU / multi-node training containers usually also need:
docker run --rm --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  --network host --device /dev/infiniband --cap-add IPC_LOCK nvcr.io/nvidia/pytorch:<tag> ...

Inside the container, the selected GPUs are renumbered from 0. GPU 3 on the host may be GPU 1 in the container; UUIDs don't change.
NVIDIA_DRIVER_CAPABILITIES controls which driver libraries are injected (compute,utility is the default; all adds graphics and video).
--ipc=host or a large --shm-size: NCCL and PyTorch data loaders use shared memory, and the 64 MB default breaks them. /dev/infiniband plus IPC_LOCK and unlimited memlock are needed for RDMA from a container.
The image's CUDA version must be supported by the host driver (or use CUDA forward compatibility packages on data-center GPUs). See CUDA for infrastructure.

--gpus device=2,3 without the inner double quotes doesn't do what it looks like: the comma splits the flag's value, so Docker rejects it. Write --gpus '"device=2,3"'.

MIG devices in Docker use the same --gpus flag; there's no separate MIG flag. Name a MIG device as GPU:MIG ('"device=0:2"' is MIG device 2 on GPU 0) or by its MIG- UUID. Both come from nvidia-smi -L on the host:

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c2a2e-...)
  MIG 3g.40gb     Device  0: (UUID: MIG-8a41d0c3-...)
  MIG 2g.20gb     Device  1: (UUID: MIG-4e7f2b19-...)
  MIG 1g.10gb     Device  2: (UUID: MIG-1c2b9e4f-...)    <- device=0:2

The second number is the MIG device index in that list, not the profile size or the GPU instance ID. MIG must already be enabled and the instances created on the host (nvidia-smi mig); a container can't create them. The MIG UUID is the stable choice, because indices change when instances are recreated. See nvidia-smi: MIG.

6. Troubleshooting: the real error messages

Message	Meaning	Fix
`NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver`	No NVIDIA kernel module loaded: not installed, not built for this kernel (DKMS, missing headers), or nouveau loaded	Install headers, rebuild (`dkms autoinstall`), reboot; check `dkms status`
`Failed to initialize NVML: Driver/library version mismatch`	Libraries upgraded, old module still loaded	Reboot (or stop everything using the GPUs and reload the modules)
CUDA error 802 `system not yet initialized`	NVSwitch fabric not up: Fabric Manager not running or version mismatch	Match FM to the driver, `systemctl start nvidia-fabricmanager`, check `nvidia-smi -q` Fabric State
FM log: `… interface version … don't match with driver version`	Fabric Manager and driver versions differ	Install the FM package of the same version; hold both
`modprobe: ERROR: could not insert 'nvidia_peermem': Invalid argument` + dmesg "disagrees about version of symbol"	peermem built against a different ib_core (network stack changed)	Reinstall the driver's DKMS package, or `dkms install --force nvidia/<version>` (`autoinstall` skips it)
`could not select device driver "" with capabilities: [[gpu]]`	Docker doesn't know about GPUs: toolkit missing, or Docker not configured or not restarted	Install the toolkit, `nvidia-ctk runtime configure --runtime=docker`, restart Docker
`unknown or invalid runtime name: nvidia`	`--runtime=nvidia` used but daemon.json has no nvidia runtime (or Docker not restarted)	Same as above, or use `--gpus`
`nvidia-container-cli: initialization error: nvml error: driver not loaded`	The host has no working driver	Fix the host driver first; check `nvidia-smi` on the host
`Failed to initialize NVML: Unknown Error` inside a long-running container	Known issue with the systemd cgroup driver: a `systemctl daemon-reload` can remove the container's GPU device access	Use CDI, or follow the toolkit's documented workaround (for example creating `/dev/char` symlinks with `nvidia-ctk system create-dev-char-symlinks`); restart the container
`CUDA driver version is insufficient for CUDA runtime version`	The container's CUDA is newer than the host driver supports	Older image tag, newer driver, or CUDA forward compatibility
Multi-node NCCL busbw low, log shows no GPUDirect RDMA	`nvidia_peermem` not loaded	`modprobe nvidia-peermem`; rebuild if it fails; make it load at boot

7. Doing it in a BCM software image

On a BCM cluster you don't install any of this on nodes one by one. You install it into the software image, then reboot the nodes onto it (driver and kernel module changes need a reboot; user-space-only changes like the toolkit can go out with imageupdate -w).

softwareimage; clone dgx-os-6 dgx-os-6-r550; commit               # test on a copy
cm-chroot-sw-img /cm/images/dgx-os-6-r550
  apt install -y nvidia-driver-550-server-open nvidia-fabricmanager-550
  apt-mark hold nvidia-driver-550-server-open nvidia-fabricmanager-550
  dkms status                                                     # built for the IMAGE's kernel?
  exit
device; use dgx01; set softwareimage dgx-os-6-r550; commit; reboot  # canary node
# verify on dgx01, then point the category at the new image and reboot the rest in batches

Inside a chroot, uname -r reports the head node's kernel, not the image's. DKMS must build the modules for the image's kernel version (install that kernel's headers in the image, and check dkms status lists it). A driver built only for the head node's kernel leaves every node booting without a GPU driver.

More: BCM software images, Fabric Manager, NGC and containers.

8. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

nvidia-smiDriver loaded, all 8 GPUs visible

Tue Nov 10 09:21:07 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.08             Driver Version: 550.127.08     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:18:00.0 Off |                    0 |
| N/A   31C    P0             69W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  |   00000000:2A:00.0 Off |                    0 |
| N/A   32C    P0             70W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
...   (GPUs 2-6 trimmed)
|   7  NVIDIA H100 80GB HBM3          On  |   00000000:DB:00.0 Off |                    0 |
| N/A   33C    P0             72W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

What to look at: Check that the header shows the driver branch you installed (550.127.08, CUDA 12.4) and that eight H100 rows are listed. A 'couldn't communicate with the NVIDIA driver' error or 'Driver/library version mismatch' instead of the table means the module isn't loaded or a reboot is still pending.

nvidia-smi -q | grep -A2 FabricNVSwitch fabric state per GPU

    Fabric
        State                             : Completed
        Status                            : Success
--
    Fabric
        State                             : Completed
        Status                            : Success
--
...   (same block for each of the 8 GPUs)

What to look at: Every GPU must report State Completed and Status Success. 'In Progress' or 'Not Started' with Status N/A means Fabric Manager isn't running or doesn't match the driver, and CUDA jobs will fail with error 802 'system not yet initialized'.

systemctl status nvidia-fabricmanagerFabric Manager service health

● nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-11-10 09:12:44 UTC; 8min ago
    Process: 2481 ExecStart=/usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg (code=exited, status=0/SUCCESS)
   Main PID: 2490 (nv-fabricmanage)
      Tasks: 19 (limit: 629145)
     Memory: 18.4M
        CPU: 2.104s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2490 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

Nov 10 09:12:41 dgx01 systemd[1]: Starting NVIDIA fabric manager service...
Nov 10 09:12:44 dgx01 nv-fabricmanager[2490]: Connected to 1 node.
Nov 10 09:12:44 dgx01 nv-fabricmanager[2490]: Successfully configured all the available NVSwitches to route GPU NVLink traffic. NVLink Peer-to-Peer support will be enabled once the GPUs are successfully registered with the NVLink fabric.
Nov 10 09:12:44 dgx01 systemd[1]: Started NVIDIA fabric manager service.

What to look at: You want 'active (running)' and the 'Successfully configured all the available NVSwitches' line. 'failed (Result: exit-code)' with a message that the fabric manager interface version doesn't match the driver version means the FM package and the driver are on different versions.

lsmod | grep nvidia_peermemGPUDirect RDMA module loaded

nvidia_peermem         16384  0
ib_core               434176  9 rdma_cm,ib_ipoib,nvidia_peermem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
nvidia              56832000  263 nvidia_uvm,nvidia_peermem,nvidia_modeset

What to look at: nvidia_peermem must appear as its own module and in ib_core's 'used by' list. No output means NCCL will run without GPUDirect RDMA, which shows up as a large multi-node bandwidth drop rather than an error.

dpkg -l | grep -E 'nvidia-driver|fabricmanager'Installed driver and FM versions

hi  nvidia-driver-550-server-open           550.127.08-0ubuntu0.22.04.1  amd64        NVIDIA driver (open kernel) metapackage
hi  nvidia-fabricmanager-550                 550.127.08-1                 amd64        Fabric Manager for NVSwitch based systems.

What to look at: The version columns for the driver and Fabric Manager must match exactly (550.127.08 here), and 'hi' means installed and held. A different FM version, or 'ii' when your runbook requires holds, is the problem to fix.

cat /proc/driver/nvidia/versionVersion of the loaded kernel module

NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  550.127.08  Release Build  (dvs-builder@U16-I3-B03-4-3)  Fri Oct 25 03:21:46 UTC 2024
GCC version:  gcc version 12.3.0 (Ubuntu 12.3.0-1ubuntu1~22.04)

What to look at: This is the module actually running in the kernel, not the package on disk. If it shows an older version (for example 535.183.01) than dpkg -l, you get 'Driver/library version mismatch' until you reboot.

dkms statusWhich kernel each module is built for

mlnx-ofed-kernel/24.10.OFED.24.10.1.1.4.1, 5.15.0-1063-nvidia, x86_64: installed
nvidia/550.127.08, 5.15.0-1063-nvidia, x86_64: installed

What to look at: Each module should say 'installed' for the kernel that uname -r reports (5.15.0-1063-nvidia). A missing entry for the running kernel, or 'added'/'built' instead of 'installed', means the module wasn't built for it, usually because linux-headers-$(uname -r) is missing.

ofed_info -sInstalled OFED / DOCA-OFED version

OFED-internal-24.10-1.1.4:

What to look at: With DOCA-OFED the string starts with OFED-internal-<version>. If it still says MLNX_OFED_LINUX-5.8-3.0.7.0, the old stack is still installed and must be removed with ofed_uninstall.sh --force before installing doca-ofed.

ibstat | grep -E "State|Rate"InfiniBand port state and speed

                State: Active
                Physical state: LinkUp
                Rate: 400
                State: Active
                Physical state: LinkUp
                Rate: 400
                State: Active
                Physical state: LinkUp
                Rate: 400
...   (mlx5_3 .. mlx5_6 trimmed: same three lines each)
                State: Active
                Physical state: LinkUp
                Rate: 400

What to look at: Every compute HCA should show State Active, Physical state LinkUp, and Rate 400 (NDR). 'Initializing' means no subnet manager has configured the port, 'Down'/'Polling' points to cabling or the switch, and a lower rate means a degraded link.

ibv_devinfo | grep -E "hca_id|state"Verbs-level view of each HCA port

hca_id: mlx5_0
                        state:                  PORT_ACTIVE (4)
hca_id: mlx5_1
                        state:                  PORT_ACTIVE (4)
hca_id: mlx5_2
                        state:                  PORT_ACTIVE (4)
...
hca_id: mlx5_7
                        state:                  PORT_ACTIVE (4)

What to look at: Each hca_id should be followed by PORT_ACTIVE (4). PORT_INIT (2) or PORT_DOWN (1) means RDMA traffic can't use that port yet.

apt install -y nvidia-container-toolkitInstall the NVIDIA Container Toolkit

Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
  libnvidia-container-tools libnvidia-container1 nvidia-container-toolkit
  nvidia-container-toolkit-base
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
...
Setting up libnvidia-container1:amd64 (1.17.3-1) ...
Setting up libnvidia-container-tools (1.17.3-1) ...
Setting up nvidia-container-toolkit-base (1.17.3-1) ...
Setting up nvidia-container-toolkit (1.17.3-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...

What to look at: All four packages should install at the same version (1.17.3-1 here). 'E: Unable to locate package nvidia-container-toolkit' means the NVIDIA repository list or its signing key wasn't added before apt-get update.

nvidia-ctk runtime configure --runtime=dockerRegister the nvidia runtime with Docker

INFO[0000] Loading config from /etc/docker/daemon.json
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that docker daemon be restarted.

What to look at: This only edits /etc/docker/daemon.json; Docker doesn't pick it up until you run systemctl restart docker. Skipping the restart gives 'could not select device driver "" with capabilities: [[gpu]]' or 'unknown or invalid runtime name: nvidia'.

nvidia-ctk runtime configure --runtime=containerdSame for containerd (Kubernetes nodes)

INFO[0000] Using config version 2
INFO[0000] Using CRI runtime plugin name "io.containerd.grpc.v1.cri"
INFO[0000] Wrote updated config to /etc/containerd/config.toml
INFO[0000] It is recommended that containerd daemon be restarted.

What to look at: The target file is /etc/containerd/config.toml (for --runtime=crio it's the CRI-O config instead). Restart containerd afterwards, or pods requesting GPUs will still start on runc without GPU access.

docker info | grep -i runtimeConfirm Docker sees the nvidia runtime

 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc

What to look at: nvidia must appear in the Runtimes list; the default staying runc is normal because --gpus works through the toolkit hook. If nvidia is missing, the daemon.json change wasn't made or Docker wasn't restarted.

nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yamlWrite the CDI spec for GPUs

INFO[0000] Using /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.550.127.08
INFO[0000] Auto-detected mode as 'nvml'
INFO[0000] Selecting /dev/nvidia0 as /dev/nvidia0
INFO[0000] Selecting /dev/nvidia1 as /dev/nvidia1
...
INFO[0000] Selecting /dev/nvidia7 as /dev/nvidia7
INFO[0000] Selecting /dev/nvidiactl as /dev/nvidiactl
INFO[0000] Selecting /dev/nvidia-uvm as /dev/nvidia-uvm
INFO[0000] Selecting /usr/bin/nvidia-smi as /usr/bin/nvidia-smi
...
INFO[0000] Generated CDI spec with version 0.5.0

What to look at: The spec should reference the current driver's libraries (550.127.08) and all eight /dev/nvidiaN nodes. The spec records driver file paths, so regenerate it after every driver upgrade or CDI containers fail to start.

nvidia-ctk cdi listDevice names CDI runtimes can request

INFO[0000] Found 9 CDI devices
nvidia.com/gpu=0
nvidia.com/gpu=1
nvidia.com/gpu=2
nvidia.com/gpu=3
nvidia.com/gpu=4
nvidia.com/gpu=5
nvidia.com/gpu=6
nvidia.com/gpu=7
nvidia.com/gpu=all

What to look at: These names are what --device nvidia.com/gpu=... accepts. 'Found 0 CDI devices' means no spec exists in /etc/cdi or /var/run/cdi, and Podman will report 'unresolvable CDI devices'.

docker run --rm --gpus all ubuntu nvidia-smi -LContainer sees every GPU

Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
de44b265507a: Pull complete
Digest: sha256:3f85b7caad41a95462cf5b787d8a04604c8262cdcdf9a472b8c52ef83375fe15
Status: Downloaded newer image for ubuntu:latest
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c0a2e-7b3d-4c11-9e02-1a0b0c0d0e00)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c1a2e-7b3d-4c11-9e02-1a0b0c0d0e01)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c2a2e-7b3d-4c11-9e02-1a0b0c0d0e02)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c3a2e-7b3d-4c11-9e02-1a0b0c0d0e03)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c4a2e-7b3d-4c11-9e02-1a0b0c0d0e04)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c5a2e-7b3d-4c11-9e02-1a0b0c0d0e05)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c6a2e-7b3d-4c11-9e02-1a0b0c0d0e06)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c7a2e-7b3d-4c11-9e02-1a0b0c0d0e07)

What to look at: Eight GPU lines prove the toolkit, Docker config, and driver are wired together, even in an image with no NVIDIA software, because nvidia-smi is injected from the host. 'could not select device driver "" with capabilities: [[gpu]]' means Docker has no GPU support configured.

docker run --rm --gpus '"device=2,3"' ubuntu nvidia-smi -LSpecific GPUs; renumbered inside container

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c2a2e-7b3d-4c11-9e02-1a0b0c0d0e02)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c3a2e-7b3d-4c11-9e02-1a0b0c0d0e03)

What to look at: Host GPUs 2 and 3 appear as GPU 0 and GPU 1 inside the container; the UUIDs prove which physical GPUs you got. Without the inner double quotes the comma splits the value and Docker rejects the --gpus argument.

nvidia-smi -LMIG device indexes for --gpus device=0:N

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c0a2e-7b3d-4c11-9e02-1a0b0c0d0e00)
  MIG 3g.40gb     Device  0: (UUID: MIG-8a41d0c3-55e1-5b0a-9c3e-2f7d1e0a4b10)
  MIG 2g.20gb     Device  1: (UUID: MIG-4e7f2b19-0c6a-5d2e-8f41-7a3b9c1d2e11)
  MIG 1g.10gb     Device  2: (UUID: MIG-1c2b9e4f-6d3a-5e7b-a0c4-3e8f2a1b5c12)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c1a2e-7b3d-4c11-9e02-1a0b0c0d0e01)
...

What to look at: The 'Device N' number under each GPU is what --gpus '"device=0:2"' refers to (GPU 0, MIG device 2), and the MIG-... UUID works too. If no MIG lines appear, MIG mode is off and the GPU:MIG form fails with 'unknown device'.

podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -LPodman GPU access through CDI

Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
...
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c0a2e-7b3d-4c11-9e02-1a0b0c0d0e00)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c1a2e-7b3d-4c11-9e02-1a0b0c0d0e01)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c2a2e-7b3d-4c11-9e02-1a0b0c0d0e02)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c3a2e-7b3d-4c11-9e02-1a0b0c0d0e03)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c4a2e-7b3d-4c11-9e02-1a0b0c0d0e04)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c5a2e-7b3d-4c11-9e02-1a0b0c0d0e05)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c6a2e-7b3d-4c11-9e02-1a0b0c0d0e06)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-5f1c7a2e-7b3d-4c11-9e02-1a0b0c0d0e07)

What to look at: Eight GPU lines mean the CDI spec resolved. 'Error: setting up CDI devices: unresolvable CDI devices nvidia.com/gpu=all' means you still need nvidia-ctk cdi generate, and --gpus on Podman 3.x silently gives the container no GPUs.

9. Hands-on lab: simulated tickets

Three tickets on one simulated DGX H100 node, dgx01, already drained:

DRV-101: a driver branch upgrade with Fabric Manager, holds, reboot, and proof.
DOCA-202: replace MLNX_OFED with DOCA-OFED, then repair the GPUDirect RDMA module the swap breaks.
CTK-303: make docker run --gpus work, select specific GPUs, and give Podman GPUs through CDI.

Outputs are simplified mock-ups modeled on the real tools, not exact copies; versions are placeholders. Reboots are instant. $(uname -r) is expanded in commands. Up/Down recall history; Tab completes.

Ticket

Ticket checklist

Answer key for this ticket (spoilers)

One clean path; other orders work too. The checklist tracks results, not exact commands.

10. Scenarios

11. Searchable reference

Item	Meaning

12. Flashcards (click to flip)

13. Exam traps in one list

Fabric Manager = driver version, exactly. Mismatch → FM won't start → CUDA error 802 on NVSwitch systems.
Upgrade driver and FM together, hold both (apt-mark hold), reboot, verify.
Stop FM, persistenced, DCGM, and workloads before swapping modules.
"Driver/library version mismatch" after an upgrade = old module still loaded: reboot.
Kernel headers must match uname -r, or DKMS can't build the GPU and DOCA modules.
Don't mix .run and package installs. -open modules for Hopper and newer.
DOCA-OFED replaces MLNX_OFED. Uninstall MLNX_OFED first (ofed_uninstall.sh --force).
DOCA install: dpkg -i doca-host_….deb → apt-get update → apt install doca-ofed (+ mlnx-fw-updater).
Upgrading DOCA: remove the old packages first; apt upgrade of the profile isn't enough with extras installed.
New network stack → rebuild nvidia_peermem (DKMS). Install the network stack before the GPU driver.
Toolkit alone does nothing for Docker: nvidia-ctk runtime configure --runtime=docker, then restart Docker.
Podman uses CDI: nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml, then --device nvidia.com/gpu=all.
--gpus '"device=2,3"' needs the double quoting; GPUs are renumbered from 0 inside.
MIG in Docker: --gpus '"device=0:2"' (GPU 0, MIG device 2 from nvidia-smi -L) or device=MIG-<uuid>. There's no --mig flag, and the MIG instances must already exist on the host.
Containers don't contain the driver; the toolkit injects the host's. The image's CUDA must be supported by the host driver.
In a BCM image, uname -r is the head node's kernel. Build modules for the image's kernel, and reboot nodes for driver changes.

14. Practice quiz (22 questions)

Score: 0 / 0