CUDA for AI Infrastructure

What CUDA is, how its layers fit together, version compatibility, environment variables, errors, and how it connects to the driver, NCCL, containers, Slurm, MIG, and Fabric Manager. Written for admins, not kernel programmers.

1. What CUDA is (in plain words)

CUDA (Compute Unified Device Architecture) is NVIDIA's platform for running general-purpose computation on GPUs. It's several things at once:

A programming model: how you describe work for thousands of GPU threads.
A compiler and toolkit: nvcc, headers, debuggers, profilers.
A set of runtime libraries: the CUDA runtime, plus math and AI libraries (cuBLAS, cuDNN, NCCL…).
A driver interface: the part of the NVIDIA driver that actually talks to the GPU.

Almost everything in AI infrastructure sits on CUDA: PyTorch, TensorFlow, JAX, TensorRT, Triton, NCCL, and RAPIDS. When a training job "uses the GPU," it's calling CUDA libraries that call the CUDA driver.

The GPU is a huge factory floor with thousands of workers. CUDA is the management system: the language for writing work orders (kernels), the dispatchers who hand them out (the runtime and driver), and the specialist teams (cuBLAS for matrix math, cuDNN for neural-net layers, NCCL for shipping goods between factories).

2. The minimum programming model an admin needs

You won't write CUDA code for this exam, but these words come up in docs, logs, and questions:

Term	Meaning
Host / device	Host = the CPU and its RAM. Device = the GPU and its memory (HBM).
Kernel	A function that runs on the GPU, launched by the host.
Thread → block → grid	A kernel runs as a grid of blocks, and each block has many threads.
Warp	A group of 32 threads that execute together.
SM (Streaming Multiprocessor)	The GPU's compute unit. Blocks are scheduled onto SMs. The H100 SXM has 132 SMs.
Tensor Cores	Specialized matrix-math units inside each SM (4th gen on H100, with the Transformer Engine for FP8).
Memory hierarchy	Registers → shared memory / L1 (per SM) → L2 cache (50 MB on H100) → HBM (80 GB, 3.35 TB/s).
Context	A process's "session" on a GPU: its memory allocations, loaded code, and so on. Creating one costs time and some GPU memory.
Stream	A queue of GPU work. Different streams can run concurrently (for example, overlapping compute with communication).
Unified Memory	Memory that both CPU and GPU can access, with the driver migrating pages on demand (needs the `nvidia-uvm` module).
Pinned (page-locked) memory	Host memory that can't be swapped out, so the GPU can DMA to and from it quickly.

The flow of nearly every GPU job: copy data host → device, launch kernels, copy results device → host, and (in distributed training) exchange data between GPUs with NCCL. Bottlenecks happen in any of these steps.

3. The CUDA software stack, layer by layer

Layer	File / package	Comes from	Role
Frameworks	PyTorch, etc.	pip/conda, NGC containers	User code
CUDA-X libraries	`libcublas`, `libcudnn`, `libnccl`	Toolkit / separate packages / bundled in frameworks	Optimized math, DL layers, communication
Runtime API	`libcudart.so`	CUDA Toolkit (or bundled with the app)	Convenient high-level API
Driver API	`libcuda.so`	NVIDIA driver	Low-level API. Talks to the kernel driver.
Kernel driver	`nvidia.ko`, `nvidia-uvm.ko`	NVIDIA driver	Hardware access, memory management
Management	`libnvidia-ml.so` (NVML)	NVIDIA driver	Monitoring and configuration (nvidia-smi, DCGM, Slurm autodetect)

4. Driver vs toolkit vs runtime: the #1 confusion

NVIDIA Driver

Installed once per machine (kernel modules + libcuda.so + NVML)
Version like 550.54.15 or 535.x
Decides the maximum CUDA version the machine supports (shown as "CUDA Version" in nvidia-smi)
In containers, it comes from the host

CUDA Toolkit / Runtime

Can have many versions side by side (/usr/local/cuda-12.2, -12.4…)
Version like 12.4. Check with nvcc --version
Needed to build CUDA code. Its runtime (libcudart) is needed to run it.
In containers, it lives inside the image

nvidia-smi                 # Driver Version: 550.54.15   CUDA Version: 12.4   <- max CUDA the DRIVER supports
nvcc --version             # Cuda compilation tools, release 12.2              <- installed TOOLKIT
cat /proc/driver/nvidia/version     # kernel driver version
python -c "import torch; print(torch.version.cuda)"   # CUDA runtime PyTorch was built with

The "CUDA Version" in nvidia-smi is not the installed toolkit. It's the highest CUDA version the driver can run. You can have toolkit 12.2 installed while nvidia-smi shows 12.4. That's normal and fine.

"The driver sets the ceiling. The toolkit picks a floor below it." Apps built with a toolkit version ≤ the driver's CUDA version just work.

5. Version compatibility

Type	Rule	Example
Backward compatibility (always)	A newer driver runs apps built with an older toolkit.	Driver supports 12.4 → an app built with CUDA 11.8 runs fine.
Minor version compatibility (CUDA 11+)	Within a major version, an app built with a newer minor (12.4) can run on a driver from an older minor of the same major (12.0-era driver), as long as it doesn't need newer driver features. Some features (for example PTX JIT of newer PTX) may not work.	App built with 12.4 on a driver that reports 12.2: usually works.
Forward compatibility (data-center GPUs)	Install the `cuda-compat` package to run a newer major CUDA on an older driver. Supported on data-center GPUs (H100, A100…), not on GeForce.	Older long-term-support driver on DGX plus a newer CUDA 12.x container, using the compat libs.
Insufficient driver	The app needs a newer CUDA than the driver supports, and there's no compat path.	Error: "CUDA driver version is insufficient for CUDA runtime version" (code 35).

The safe production approach: pick a validated driver branch (often a long-term-support branch on DGX), make sure its CUDA ceiling ≥ the CUDA your containers and frameworks were built with, or use forward compatibility on data-center GPUs. Upgrading the driver is the usual fix for "insufficient driver."

You can't fix "driver insufficient" by installing a newer toolkit. The toolkit doesn't replace the driver's libcuda.so. Upgrade the driver, or use the forward-compat package on supported GPUs.

6. Compatibility checker

A simplified model of the rules above, for practice.

Driver's max CUDA ("CUDA Version" in nvidia-smi)

CUDA version the app/container was built with

GPU type

cuda-compat (forward compat) package installed?

7. Compute capability, PTX, and JIT

Compute capability (CC)

A version number for the GPU hardware's feature set, not the CUDA software version.

GPU	Architecture	Compute capability	nvcc target
V100	Volta	7.0	`sm_70`
A100	Ampere	8.0	`sm_80`
L40S / L4	Ada Lovelace	8.9	`sm_89`
H100 / H200	Hopper	9.0	`sm_90` (`sm_90a` for Hopper-specific features)
B200	Blackwell	10.0	`sm_100`

nvidia-smi --query-gpu=name,compute_cap --format=csv    # e.g. NVIDIA H100 80GB HBM3, 9.0

SASS, PTX, and fat binaries

SASS = real machine code for one specific architecture (for example sm_90). Fastest to start, but only runs on that architecture family.
PTX = an intermediate "virtual assembly." The driver can JIT-compile PTX into SASS for whatever GPU it finds, including newer GPUs.
Fat binary = contains SASS for several architectures plus PTX as a fallback. Frameworks like PyTorch ship these.

nvcc -gencode arch=compute_90,code=sm_90 \
     -gencode arch=compute_90,code=compute_90 app.cu    # SASS for H100 + PTX for future GPUs

If a binary has no SASS for your GPU and no compatible PTX, you get "no kernel image is available for execution on the device." The fix is a build that targets your compute capability (for example, a newer PyTorch / NGC container for H100).

JIT compilation happens at startup and is cached in ~/.nv/ComputeCache (see CUDA_CACHE_PATH). A cold cache can make the first run noticeably slower.

8. Kernel modules and device files

Kernel module	Purpose	If missing
`nvidia`	Core GPU driver	No GPU access at all
`nvidia_uvm`	Unified Virtual Memory. Required by the CUDA runtime.	CUDA fails (often "unknown error" / code 999) while nvidia-smi may still work
`nvidia_modeset`, `nvidia_drm`	Display / DRM	Mostly irrelevant on headless servers
`nvidia_peermem`	GPUDirect RDMA with InfiniBand NICs	NCCL works but without GDRDMA (slower multi-node)

Device file	What it is
`/dev/nvidia0` … `/dev/nvidia7`	One per GPU
`/dev/nvidiactl`	Control device
`/dev/nvidia-uvm`, `/dev/nvidia-uvm-tools`	Unified memory
`/dev/nvidia-caps/*`	Capabilities (used for MIG access)
`/dev/nvidia-nvswitch*`	NVSwitch devices (used by Fabric Manager)

lsmod | grep nvidia                    # which modules are loaded
sudo modprobe nvidia                   # load the core kernel driver (nvidia-smi fails without it)
ls -l /dev/nvidia*                     # device files present?
sudo modprobe nvidia-uvm               # load UVM if missing
sudo nvidia-modprobe -u -c=0           # helper that creates the UVM device nodes

The NVIDIA Container Toolkit and Slurm's cgroup GPU isolation work by controlling access to these /dev/nvidia* files.

9. CUDA environment variables

Grouped by purpose. The starred ones are the most tested.

Device selection and ordering

Variable	What it does	Values / example
★ `CUDA_VISIBLE_DEVICES`	Which GPUs the process can see. Renumbers them from 0. Ignored by nvidia-smi.	`0,1` · `GPU-uuid` · `MIG-uuid` · empty = none
★ `CUDA_DEVICE_ORDER`	How CUDA numbers devices	`FASTEST_FIRST` (default) · `PCI_BUS_ID` (matches nvidia-smi/NVML)

Debugging

Variable	What it does	Values
★ `CUDA_LAUNCH_BLOCKING`	Makes kernel launches synchronous, so errors are reported at the line that caused them (not later). Much slower, so use it for debugging only.	`1`
`CUDA_ENABLE_COREDUMP_ON_EXCEPTION`	Write a GPU core dump when a kernel faults	`1`
`CUDA_COREDUMP_FILE`	Where to write it	path

JIT and module loading

Variable	What it does	Values
`CUDA_CACHE_PATH`	Location of the JIT compile cache	default `~/.nv/ComputeCache`
`CUDA_CACHE_MAXSIZE`	Max cache size in bytes	number
`CUDA_CACHE_DISABLE`	Disable the JIT cache	`1`
`CUDA_FORCE_PTX_JIT`	Ignore embedded SASS and JIT from PTX (testing forward compatibility)	`1`
`CUDA_MODULE_LOADING`	Load GPU code lazily (on first use) or eagerly. `LAZY` reduces startup time and memory (the default in recent CUDA 12).	`LAZY` · `EAGER`

Performance and concurrency

Variable	What it does	Notes
`CUDA_DEVICE_MAX_CONNECTIONS`	Number of hardware work queues between host and GPU	Megatron-LM sets `1` to enforce ordering that helps overlap communication and compute

MPS (Multi-Process Service)

Variable	What it does
`CUDA_MPS_PIPE_DIRECTORY`	Where clients find the MPS control daemon
`CUDA_MPS_LOG_DIRECTORY`	Where MPS writes logs
`CUDA_MPS_ACTIVE_THREAD_PERCENTAGE`	Cap the share of SMs a client may use
`CUDA_MPS_PINNED_DEVICE_MEM_LIMIT`	Cap a client's GPU memory

Paths (conventions, not read by the CUDA runtime itself)

Variable	Purpose
`PATH`	Include `/usr/local/cuda/bin` to find `nvcc`
`LD_LIBRARY_PATH`	Include `/usr/local/cuda/lib64` (and cuda-compat if used) to find the libraries
`CUDA_HOME` / `CUDA_PATH`	Used by build systems (PyTorch extensions, CMake) to locate the toolkit

Container toolkit (not CUDA itself, but closely related)

Variable	What it does
★ `NVIDIA_VISIBLE_DEVICES`	Which GPUs are mounted into the container (`all`, `none`, `void`, indices, UUIDs)
`NVIDIA_DRIVER_CAPABILITIES`	Which driver libraries are mounted (`compute`, `utility`, `video`, `graphics`, `all`). `compute,utility` = CUDA + nvidia-smi.
`NVIDIA_REQUIRE_CUDA`	The image declares a minimum CUDA (for example `cuda>=12.4`). The toolkit refuses to start if the host driver is too old.
`NVIDIA_DISABLE_REQUIRE`	Skip that check (for example when relying on forward compatibility)

Framework-level (built on CUDA)

Variable	What it does
`PYTORCH_CUDA_ALLOC_CONF`	Tunes PyTorch's GPU memory allocator (for example `expandable_segments:True` to reduce fragmentation OOMs)
`NCCL_*`	NCCL communication settings (see the NCCL guide)
`TORCH_NCCL_ASYNC_ERROR_HANDLING`	Fail fast on NCCL errors

Group by question: Which GPUs? → CUDA_VISIBLE_DEVICES / NVIDIA_VISIBLE_DEVICES. What order? → CUDA_DEVICE_ORDER. Where did it crash? → CUDA_LAUNCH_BLOCKING=1. Slow first start? → JIT cache / CUDA_MODULE_LOADING. Sharing a GPU? → CUDA_MPS_*.

	Time-slicing (default)	MPS (Multi-Process Service)	MIG (Multi-Instance GPU)
How it works	Processes take turns. The GPU context-switches between them.	A daemon merges multiple processes' work into one context, so kernels from different processes run at the same time.	The GPU is partitioned in hardware into up to 7 instances, each with its own SMs, memory, and cache.
Concurrency	No (one at a time)	Yes	Yes (separate instances)
Memory isolation	Separate contexts	Limited (memory limits are optional)	Strong (dedicated memory)
Fault isolation	Yes	Weak: one client's fault can affect others	Strong
Performance (QoS) predictability	Low	Medium	High
Setup	Nothing	`nvidia-cuda-mps-control -d` (often with compute mode EXCLUSIVE_PROCESS)	`nvidia-smi -mig 1` + `nvidia-smi mig -cgi … -C`
Best for	Dev / light sharing	Many small cooperating processes (for example MPI ranks, small inference) that underuse the GPU	Multi-tenant inference, guaranteed isolation

# MPS quick start
sudo nvidia-smi -i 0 -c EXCLUSIVE_PROCESS
export CUDA_MPS_PIPE_DIRECTORY=/tmp/mps-pipe CUDA_MPS_LOG_DIRECTORY=/tmp/mps-log
nvidia-cuda-mps-control -d                 # start the daemon
echo quit | nvidia-cuda-mps-control        # stop it

Need hard isolation (memory + faults + QoS) → MIG. Need concurrent sharing among trusted processes → MPS. Neither is for large multi-GPU training, which wants whole GPUs with NVLink.

GPUs in virtual machines

Everything above assumes the OS runs directly on the server. On a hypervisor (for example VMware ESXi), a VM gets a GPU in one of two ways: passthrough (a whole GPU to one VM, with the normal driver in the guest) or NVIDIA vGPU (a slice of a GPU per VM, time-sliced or MIG-backed, with a vGPU Manager on the host, a matching guest driver, and a license). Inside the VM, CUDA, containers, and the compatibility rules on this page work exactly as on bare metal.

The full topic, including esxcli, vGPU profiles and licensing, install and upgrade flows, KVM, and multi-GPU/RDMA in VMs, is in the dedicated GPUs in Virtual Machines guide.

11. The GPUDirect family

Technology	Path	Needs	Used by
GPUDirect P2P	GPU ↔ GPU in the same node (NVLink or PCIe)	Driver support; NVSwitch + Fabric Manager on DGX	NCCL intra-node, `cudaMemcpyPeer`
GPUDirect RDMA	NIC ↔ GPU memory directly (across nodes)	`nvidia-peermem` or DMA-BUF, OFED, ConnectX	NCCL inter-node (`GDRDMA`), MPI
GPUDirect Storage (GDS)	Storage (NVMe / network FS) ↔ GPU memory directly	`nvidia-fs` driver, supported filesystem; cuFile API	Data loading, checkpointing

All GPUDirect technologies do the same basic thing: skip the extra copy through CPU memory. That cuts latency, frees the CPU, and raises bandwidth.

12. How CUDA interacts with every other component

Component	Relationship to CUDA	Typical gotcha
NVIDIA driver	Provides `libcuda.so` and the kernel modules. Sets the max CUDA version.	Driver too old for the app or container → error 35
NVML / nvidia-smi / DCGM	A separate management path (NVML), not the CUDA runtime	nvidia-smi works while CUDA fails (for example UVM missing, Fabric Manager down). nvidia-smi ignores `CUDA_VISIBLE_DEVICES`. Different default ordering.
Fabric Manager	Must initialize NVSwitch before CUDA can use the GPUs on DGX/HGX	FM not running or version ≠ driver → CUDA error 802 "system not yet initialized"
nvidia-persistenced	Keeps the driver state loaded between CUDA processes	Off → every job's first CUDA call is slow
NCCL	A CUDA library. Uses CUDA streams, P2P, and GPUDirect RDMA.	Must match a compatible CUDA. Picks GPUs via `cuda:local_rank` within `CUDA_VISIBLE_DEVICES`.
cuDNN / cuBLAS	CUDA libraries for DL layers and matrix math	Version mismatches with frameworks. Use NGC containers to get a tested combination.
PyTorch / TF	Bundle their own CUDA runtime + libraries (pip wheels or containers)	`torch.version.cuda` must be ≤ the driver's CUDA (or use compat). `torch.cuda.is_available()` = False means a driver or visibility problem.
Containers (Container Toolkit)	Mount the host driver (`libcuda`, NVML, `/dev/nvidia*`) into images that carry the CUDA runtime	No `--gpus` → no GPUs. Host driver too old for the image's CUDA.
Slurm	Allocates GPUs and sets `CUDA_VISIBLE_DEVICES`. cgroups restrict `/dev/nvidia*`.	Ordering mismatch → set `CUDA_DEVICE_ORDER=PCI_BUS_ID`. Enable `ConstrainDevices`.
Kubernetes	The device plugin / GPU Operator installs the driver and toolkit and allocates GPUs to pods	Pods without `nvidia.com/gpu` limits get no GPU
MIG	Each instance appears to CUDA as a separate device (select with `MIG-uuid`)	No P2P between MIG instances. One CUDA device per MIG instance per process.
MPS	Runs between CUDA clients and the GPU to allow concurrent kernels	Weak fault isolation
BCM	Puts the driver, CUDA, FM, DCGM, and so on into the software image	Install or upgrade CUDA/driver in the image, not by hand on nodes
InfiniBand / OFED	GPUDirect RDMA links CUDA memory to the NIC	`nvidia-peermem` must be loaded

13. CUDA in containers

The driver always comes from the host. The image carries only the CUDA runtime and libraries.
The host driver's max CUDA must be ≥ the image's CUDA, or forward compatibility must be available (data-center GPUs; NGC images can use the compat libs automatically on supported drivers).
Start with GPU access: docker run --gpus all …, Kubernetes nvidia.com/gpu limits, or Slurm + pyxis/enroot (srun --container-image=…).
Multi-node in containers also needs IB devices, IPC_LOCK, memlock, and shm (see the NCCL guide).
NGC containers are the recommended way to get a tested combination of CUDA + cuDNN + NCCL + framework.

14. Common CUDA errors and what they mean

Error (code)	Typical meaning	Check / fix
no CUDA-capable device is detected (100)	CUDA sees no GPUs	`CUDA_VISIBLE_DEVICES` empty or wrong, container started without GPUs, driver not loaded
CUDA driver version is insufficient for CUDA runtime version (35)	App or container CUDA is newer than the driver supports	Upgrade the driver, or use forward compat (data-center GPUs)
system not yet initialized (802)	On NVSwitch systems, the fabric isn't ready	Start Fabric Manager, and make sure its version matches the driver
forward compatibility was attempted on non supported HW (804)	The compat package was used on an unsupported GPU	Remove the compat libs from the path, or upgrade the driver
out of memory (2)	GPU memory exhausted	Smaller batch, check other processes (`nvidia-smi`), fragmentation (`PYTORCH_CUDA_ALLOC_CONF`)
an illegal memory access was encountered (700)	A kernel accessed bad memory (usually an app bug). XID 13/31 in dmesg.	`CUDA_LAUNCH_BLOCKING=1`, `compute-sanitizer`. If it's widespread across apps on one GPU, suspect hardware.
uncorrectable ECC error encountered (214)	Hardware memory error	Drain the node, check `nvidia-smi -q -d ECC,ROW_REMAPPER`, run DCGM diag, reset the GPU, RMA if needed
all CUDA-capable devices are busy or unavailable (46)	Compute mode is EXCLUSIVE_PROCESS (already in use) or PROHIBITED	`nvidia-smi -q -d COMPUTE`. Change the mode or wait.
no kernel image is available for execution on the device (209)	The binary wasn't built for this GPU's compute capability and has no usable PTX	Use a build or container that supports sm_90 (H100)
unknown error (999)	Often driver state: missing `nvidia-uvm`, a GPU in a bad state	`lsmod \| grep uvm`, `dmesg` for XIDs, GPU reset or reboot

More codes (Xid, ECC, NCCL, NVML messages) and what action each needs: error codes guide.

nvidia-smi working doesn't prove CUDA works. nvidia-smi uses NVML, while CUDA needs libcuda, nvidia-uvm, and (on DGX) Fabric Manager. Always test with a real CUDA program (deviceQuery, torch.cuda.is_available()).

15. Tools and samples for testing

Tool	Purpose
`nvcc --version`	Installed toolkit version
`deviceQuery` (cuda-samples)	Lists CUDA devices, compute capability, memory, PCI IDs. "Result = PASS" proves CUDA works.
`bandwidthTest` (cuda-samples)	Host↔device and device↔device memory copy bandwidth
`p2pBandwidthLatencyTest` (cuda-samples)	GPU↔GPU P2P bandwidth and latency matrix (checks NVLink inside a node)
`compute-sanitizer`	Finds memory errors and race conditions in CUDA code
`cuda-gdb`	Debugger for CUDA
Nsight Systems (`nsys`)	Timeline profiler: CPU, GPU, NCCL, memory copies. Where is time spent?
Nsight Compute (`ncu`)	Detailed per-kernel profiler. Both are covered in the Nsight guide.
`python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"`	Quick framework-level CUDA check
`nvidia-smi`, DCGM	Management and monitoring (NVML path)
nccl-tests	Communication benchmarks (built on CUDA + NCCL)

16. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

nvidia-smiDriver version and max supported CUDA

Tue Nov 10 10:02:11 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:18:00.0 Off |                    0 |
| N/A   33C    P0             71W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  |   00000000:2A:00.0 Off |                    0 |
| N/A   31C    P0             69W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
...

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

What to look at: The CUDA Version in the header is the newest CUDA runtime this driver can run, not the installed toolkit. Check every GPU is listed with Persistence-M On and 0 uncorrected ECC; a missing GPU or an error banner here means a driver or hardware problem, not a CUDA one.

nvcc --versionInstalled CUDA toolkit version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:02:13_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

What to look at: The release line is the toolkit version, which may legitimately be lower than the driver's CUDA Version (12.2 toolkit on a 12.4 driver is fine). A toolkit newer than the driver supports, without cuda-compat, produces error 35 at runtime; 'nvcc: command not found' usually means /usr/local/cuda/bin is not on PATH.

cat /proc/driver/nvidia/versionLoaded kernel driver version

NVRM version: NVIDIA UNIX x86_64 Kernel Module  550.54.15  Tue Mar  5 22:23:56 UTC 2024
GCC version:  gcc version 12.3.0 (Ubuntu 12.3.0-1ubuntu1~22.04)

What to look at: This is the version of the module actually loaded in the kernel, which must match the user-space libraries (libcuda, NVML). If this file is missing the nvidia module is not loaded; if it differs from the nvidia-smi Driver Version you get 'Driver/library version mismatch' until you reload the module or reboot.

python -c "import torch; print(torch.version.cuda)"CUDA runtime PyTorch was built with

12.1

What to look at: This is the CUDA runtime bundled inside the PyTorch wheel, independent of nvcc. It only has to be less than or equal to the driver's CUDA Version (12.4 here); a value above it fails with 'driver too old' (error 35).

python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"Prove the CUDA path works

True 8

What to look at: True and the expected GPU count prove libcuda, nvidia-uvm and the device nodes all work. False with a working nvidia-smi points at a missing nvidia-uvm module, missing /dev/nvidia-uvm, Fabric Manager not running (error 802), or a driver too old for the runtime.

nvidia-smi --query-gpu=name,compute_cap --format=csvGPU model and compute capability

name, compute_cap
NVIDIA H100 80GB HBM3, 9.0
NVIDIA H100 80GB HBM3, 9.0
NVIDIA H100 80GB HBM3, 9.0
NVIDIA H100 80GB HBM3, 9.0
NVIDIA H100 80GB HBM3, 9.0
NVIDIA H100 80GB HBM3, 9.0
NVIDIA H100 80GB HBM3, 9.0
NVIDIA H100 80GB HBM3, 9.0

What to look at: Compute capability 9.0 means binaries need sm_90 SASS or PTX that can JIT to it. An application built only for sm_80 with no PTX fails on H100 with error 209 (no kernel image is available for execution on the device).

nvcc -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 app.cu && cuobjdump --list-elf --list-ptx a.out

Confirm SASS and PTX are embedded

ELF file    1: a.1.sm_90.cubin
PTX file    1: a.1.sm_90.ptx

What to look at: nvcc itself prints nothing on success; cuobjdump shows what got embedded. You want an sm_90 ELF (native SASS for H100) plus a PTX entry for forward compatibility; a binary with only sm_80 ELF and no PTX cannot run on Hopper.

lsmod | grep nvidiaWhich NVIDIA kernel modules are loaded

nvidia_uvm           1777664  4
nvidia_peermem         16384  0
nvidia_fs             262144  0
nvidia_drm            122880  0
nvidia_modeset       1515520  1 nvidia_drm
nvidia              56692736  118 nvidia_uvm,nvidia_peermem,nvidia_fs,nvidia_modeset
ib_core               454656  9 rdma_cm,ib_ipoib,nvidia_peermem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
drm_kms_helper        315392  4 ast,nvidia_drm
drm                   622592  6 drm_kms_helper,ast,nvidia,drm_ttm_helper,nvidia_drm,ttm

What to look at: nvidia must be present for nvidia-smi, nvidia_uvm for any CUDA program, nvidia_peermem for GPUDirect RDMA and nvidia_fs for GPUDirect Storage. A missing nvidia_uvm line is the classic 'nvidia-smi works but CUDA fails with error 999 or 100' case.

ls -l /dev/nvidia*GPU device nodes present

crw-rw-rw- 1 root root 195,   0 Nov 10 09:12 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Nov 10 09:12 /dev/nvidia1
...
crw-rw-rw- 1 root root 195,   7 Nov 10 09:12 /dev/nvidia7
crw-rw-rw- 1 root root 195, 255 Nov 10 09:12 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Nov 10 09:12 /dev/nvidia-modeset
crw-rw-rw- 1 root root 236,   0 Nov 10 09:12 /dev/nvidia-nvswitch0
crw-rw-rw- 1 root root 236,   1 Nov 10 09:12 /dev/nvidia-nvswitch1
crw-rw-rw- 1 root root 236,   2 Nov 10 09:12 /dev/nvidia-nvswitch2
crw-rw-rw- 1 root root 236,   3 Nov 10 09:12 /dev/nvidia-nvswitch3
crw-rw-rw- 1 root root 236, 255 Nov 10 09:12 /dev/nvidia-nvswitchctl
crw-rw-rw- 1 root root 508,   0 Nov 10 09:12 /dev/nvidia-uvm
crw-rw-rw- 1 root root 508,   1 Nov 10 09:12 /dev/nvidia-uvm-tools

/dev/nvidia-caps:
total 0
cr-------- 1 root root 511, 1 Nov 10 09:12 nvidia-cap1
cr--r--r-- 1 root root 511, 2 Nov 10 09:12 nvidia-cap2

What to look at: You need one /dev/nvidiaN per GPU, /dev/nvidiactl and /dev/nvidia-uvm; on DGX H100 the four nvidia-nvswitch nodes should also exist. If /dev/nvidia-uvm is missing, load nvidia-uvm or run nvidia-modprobe -u -c=0; a container that lacks these nodes was started without GPU access.

sudo nvidia-smi -i 0 -c EXCLUSIVE_PROCESSSet compute mode for MPS

Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:18:00.0.
All done.

What to look at: EXCLUSIVE_PROCESS allows a single process (the MPS server) per GPU. If you forget to set it back to DEFAULT, later jobs fail with error 46 'all CUDA-capable devices are busy or unavailable'.

nvidia-cuda-mps-control -d && tail -n 4 /tmp/mps-log/control.logStart the MPS control daemon

[2026-11-10 10:15:02.113 Control 41877] Starting control daemon using socket /tmp/mps-pipe/control
[2026-11-10 10:15:02.114 Control 41877] To connect CUDA applications to this daemon, set env CUDA_MPS_PIPE_DIRECTORY=/tmp/mps-pipe
[2026-11-10 10:15:09.502 Control 41877] Accepting connection...
[2026-11-10 10:15:09.503 Control 41877] NEW SERVER 41921: Ready

What to look at: The daemon is silent on start, so read control.log: the socket path must match the CUDA_MPS_PIPE_DIRECTORY your clients use. Clients that do not appear here are not going through MPS, usually because their pipe directory differs or the daemon runs as another user.

nvidia-smi -q -d COMPUTECurrent compute mode per GPU

==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 10:20:31 2026
Driver Version                            : 550.54.15
CUDA Version                              : 12.4

Attached GPUs                             : 8
GPU 00000000:18:00.0
    Compute Mode                          : Exclusive_Process

GPU 00000000:2A:00.0
    Compute Mode                          : Default

GPU 00000000:3A:00.0
    Compute Mode                          : Default
...

What to look at: Default lets many processes share a GPU; Exclusive_Process allows only one, and Prohibited allows none. A GPU left in Exclusive_Process or Prohibited explains error 46 for a second job on the same GPU.

nvidia-smi -q -d ECC,ROW_REMAPPER -i 0ECC errors and row remapping

==============NVSMI LOG==============

Attached GPUs                             : 8
GPU 00000000:18:00.0
    ECC Mode
        Current                           : Enabled
        Pending                           : Enabled
    ECC Errors
        Volatile
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 0
            DRAM Uncorrectable            : 0
        Aggregate
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 3
            DRAM Uncorrectable            : 0
            SRAM Threshold Exceeded       : No
    Remapped Rows
        Correctable Error                 : 0
        Uncorrectable Error               : 0
        Pending                           : No
        Remapping Failure Occurred        : No
        Bank Remap Availability Histogram
            Max                           : 2560 bank(s)
            High                          : 0 bank(s)
            Partial                       : 0 bank(s)
            Low                           : 0 bank(s)
            None                          : 0 bank(s)

What to look at: A few aggregate correctable DRAM errors are normal. Any uncorrectable count, Pending: Yes (reset the GPU to apply the remap) or Remapping Failure Occurred: Yes means drain the node, and a failure means RMA.

sudo nvidia-smi -i 0 -mig 1Enable MIG mode on a GPU

Enabled MIG Mode for GPU 00000000:18:00.0
All done.

What to look at: Success prints 'Enabled MIG Mode'. If any process holds the GPU (including DCGM or another nvidia-smi) you instead get a warning that MIG is 'pending enable', and it only takes effect after those processes stop or the GPU is reset.

sudo nvidia-smi mig -i 0 -cgi 9,9 -CCreate two 3g.40gb MIG instances

Successfully created GPU instance ID  2 on GPU  0 using profile MIG 3g.40gb (ID  9)
Successfully created compute instance ID  0 on GPU  0 GPU instance ID  2 using profile MIG 3g.40gb (ID  2)
Successfully created GPU instance ID  1 on GPU  0 using profile MIG 3g.40gb (ID  9)
Successfully created compute instance ID  0 on GPU  0 GPU instance ID  1 using profile MIG 3g.40gb (ID  2)

What to look at: Each GPU instance needs a compute instance (-C) or CUDA cannot use it. 'Insufficient Resources' means the requested profiles do not fit the remaining slices; on H100 80GB, profile 9 is 3g.40gb and two of them use 6 of 7 compute slices.

./deviceQueryCUDA sample: prove CUDA works

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 8 CUDA Capable device(s)

Device 0: "NVIDIA H100 80GB HBM3"
  CUDA Driver Version / Runtime Version          12.4 / 12.2
  CUDA Capability Major/Minor version number:    9.0
  Total amount of global memory:                 81079 MBytes (85017493504 bytes)
  (132) Multiprocessors, (128) CUDA Cores/MP:    16896 CUDA Cores
  GPU Max Clock rate:                            1980 MHz (1.98 GHz)
  Memory Clock rate:                             2619 Mhz
  Memory Bus Width:                              5120-bit
  L2 Cache Size:                                 52428800 bytes
  ...
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 24 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
...
> Peer access from NVIDIA H100 80GB HBM3 (GPU0) -> NVIDIA H100 80GB HBM3 (GPU1) : Yes
> Peer access from NVIDIA H100 80GB HBM3 (GPU0) -> NVIDIA H100 80GB HBM3 (GPU2) : Yes
...

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.4, CUDA Runtime Version = 12.2, NumDevs = 8
Result = PASS

What to look at: Result = PASS with the right device count proves the full CUDA stack works, which nvidia-smi alone does not. 'Result = FAIL' with an error such as 'cudaGetDeviceCount returned 802' (Fabric Manager not ready) or 35 (driver too old) tells you which layer is broken.

./bandwidthTestCUDA sample: memory copy bandwidth

[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: NVIDIA H100 80GB HBM3
 Quick Mode

 Host to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)        Bandwidth(GB/s)
   32000000                     55.2

 Device to Host Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)        Bandwidth(GB/s)
   32000000                     55.4

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)        Bandwidth(GB/s)
   32000000                     2185.3

Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

What to look at: Host-device numbers around 50-55 GB/s match PCIe Gen5 x16. A value near 25 GB/s suggests the link trained at Gen4 or x8, and roughly half the expected value on only some GPUs often means a CPU/NUMA placement or PCIe problem.

./p2pBandwidthLatencyTestCUDA sample: GPU-to-GPU P2P matrix

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA H100 80GB HBM3, pciBusID: 18, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA H100 80GB HBM3, pciBusID: 2a, pciDeviceID: 0, pciDomainID:0
...
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
...
P2P Connectivity Matrix
     D\D     0     1     2     3     4     5     6     7
     0       1     1     1     1     1     1     1     1
     1       1     1     1     1     1     1     1     1
...
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 2471.34 391.88 392.10 391.95 392.04 391.77 392.21 391.90
     1 391.92 2475.10 392.03 391.86 392.12 391.98 391.81 392.07
...
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3      4      5      6      7
     0   2.41   2.63   2.61   2.64   2.62   2.60   2.63   2.62
...
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

What to look at: Every off-diagonal pair should show a 1 in the connectivity matrix and similar NVLink-class bandwidth (hundreds of GB/s). A 0, or one row far lower than the others (PCIe-class tens of GB/s), means P2P is disabled or an NVLink path is degraded for that GPU.

compute-sanitizer ./appCatch out-of-bounds memory errors

========= COMPUTE-SANITIZER
========= Invalid __global__ write of size 4 bytes
=========     at vecAdd(const float *, const float *, float *, int)+0x1d0 in app.cu:14
=========     by thread (255,0,0) in block (4095,0,0)
=========     Address 0x7f2a3e400000 is out of bounds
=========     and is 0 bytes after the nearest allocation at 0x7f2a3e000000 of size 4194304 bytes
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     ...
=========
========= ERROR SUMMARY: 1 error

What to look at: 'ERROR SUMMARY: 0 errors' is what you want. Any invalid read or write points at an application bug (the same thing that shows up as error 700 'illegal memory access' in production), not at the driver or hardware.

17. Troubleshooting scenarios

18. Searchable reference

Item	Meaning

19. Exam traps

nvidia-smi "CUDA Version" = the driver's maximum supported CUDA, not the installed toolkit (nvcc --version).
"Insufficient driver" → upgrade the driver (or forward compat). A new toolkit won't help.
Forward compatibility (cuda-compat) is for data-center GPUs, not GeForce.
Containers don't contain the driver. It comes from the host through the Container Toolkit.
Compute capability ≠ CUDA version. H100 = CC 9.0 (sm_90). A100 = 8.0.
CUDA_VISIBLE_DEVICES = visibility + renumbering. CUDA_DEVICE_ORDER = ordering. NVIDIA_VISIBLE_DEVICES = containers.
For pulling and running NGC containers (nvcr.io login, Enroot/Pyxis, Apptainer, Kubernetes), see NGC mastery.
CUDA_LAUNCH_BLOCKING=1 = debugging (synchronous launches), never a performance fix.
Error 802 on DGX/HGX → Fabric Manager.
nvidia-uvm missing → CUDA fails while nvidia-smi still works.
MIG vs MPS: MIG = hardware isolation. MPS = concurrent sharing with weak isolation.
GPUDirect: P2P (GPU↔GPU), RDMA (NIC↔GPU, nvidia-peermem), Storage (disk↔GPU, nvidia-fs).
"no kernel image is available" → the binary wasn't built for the GPU architecture. Use a build that supports sm_90.
In BCM clusters, install or upgrade CUDA and the driver in the software image.

20. Practice quiz (25 questions)

Score: 0 / 0