GPU Ordering and Visibility

Why "GPU 0" can mean different GPUs to different tools, and the environment variables that control numbering and visibility.

1. Warm-up: why does "GPU 0" mean different GPUs?

A user reports that the GPU their PyTorch job calls cuda:0 isn't the one nvidia-smi shows as GPU 0, so they're watching the wrong card. Which setting makes every tool agree on the numbering?

WRONGSet NVIDIA_VISIBLE_DEVICES=all

This controls which GPUs a container gets (through the NVIDIA Container Toolkit). "all" passes in every GPU, but it says nothing about order. It's the right topic area but answers a different question: visibility, not ordering.

CORRECTSet CUDA_DEVICE_ORDER=PCI_BUS_ID

By default CUDA uses FASTEST_FIRST ordering. NVML, which powers nvidia-smi, DCGM, and Slurm's AutoDetect=nvml, orders GPUs by PCI bus ID. Setting CUDA_DEVICE_ORDER=PCI_BUS_ID makes CUDA use the same order, so "GPU 0" means the same physical GPU everywhere.

WRONGFix the order in Slurm or DCGM configuration

Neither one controls how CUDA numbers devices. Slurm's GPU configuration (gres.conf / slurm.conf) decides which GPUs a job gets, and DCGM only monitors. Be suspicious of any plausible-sounding SLURM_*_ORDER or DCGM_*_ORDER variable: no such variables exist.

Two separate questions: "In what ORDER are GPUs numbered?" → CUDA_DEVICE_ORDER. "WHICH GPUs can I see?" → CUDA_VISIBLE_DEVICES (for a process) or NVIDIA_VISIBLE_DEVICES (for a container).

"ORDER fixes the numbering. VISIBLE picks the subset." And "PCI_BUS_ID = speak NVML's language."

2. The 4 ways to identify a GPU

Identifier	Example	Stable?	Notes
Index	`0` … `7`	No	Depends on the enumeration order, and changes with visibility filtering. The source of all the confusion.
PCI bus ID	`00000000:18:00.0`	Yes (per slot)	The physical slot address. NVML sorts by this.
UUID	`GPU-4b1e7a2c-…`	Yes (per GPU)	Unique to that physical GPU, so it follows the GPU if it moves. Best for automation.
MIG UUID	`MIG-9f3c…`	While the instance exists	Identifies one MIG instance. Used in `CUDA_VISIBLE_DEVICES`.

nvidia-smi --query-gpu=index,pci.bus_id,uuid,name --format=csv
nvidia-smi -L       # GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-...)

The index is "the third person in line." It depends on how the line was formed. The PCI bus ID is "the person in seat 18," and the UUID is "the person whose passport number is X." Seat numbers and passport numbers don't change when the line is reshuffled.

3. Ordering: `CUDA_DEVICE_ORDER`

Value	Ordering	Matches nvidia-smi?
`FASTEST_FIRST` (default)	CUDA uses a heuristic to put the "fastest" device first, then the rest	Not guaranteed
`PCI_BUS_ID`	Ascending PCI bus ID, the same rule NVML uses	Yes

export CUDA_DEVICE_ORDER=PCI_BUS_ID   # put in job scripts, module files, or the cluster's default environment

Which tools use which order:

Tool	Library	Default ordering
nvidia-smi	NVML	PCI bus ID
DCGM	NVML	PCI bus ID
Slurm with `AutoDetect=nvml`	NVML	PCI bus ID
CUDA apps (PyTorch, TensorFlow, nccl-tests…)	CUDA runtime	FASTEST_FIRST unless you set PCI_BUS_ID

On a DGX H100 all 8 GPUs are identical, so the two orders often happen to match. Don't rely on luck. Mixed or heterogeneous nodes (for example a workstation with two different GPU models) are where they visibly diverge. Setting PCI_BUS_ID makes the match guaranteed rather than accidental.

4. Visibility: `CUDA_VISIBLE_DEVICES` (and renumbering)

CUDA_VISIBLE_DEVICES tells the CUDA runtime which GPUs a process may use. It accepts indices, UUIDs, or MIG UUIDs.

CUDA_VISIBLE_DEVICES=2,3 python train.py           # by index
CUDA_VISIBLE_DEVICES=GPU-4b1e...,GPU-77ac... ./app  # by UUID (robust)
CUDA_VISIBLE_DEVICES=MIG-9f3c... ./infer            # a MIG instance
CUDA_VISIBLE_DEVICES= ./app                         # empty = no GPUs visible
CUDA_VISIBLE_DEVICES=-1 ./app                       # invalid index = no GPUs (common trick)

The renumbering rule (very commonly tested)

Visible GPUs are renumbered starting at 0 inside the process, in the order you list them.

Setting	Physical GPUs used	What the app calls them
`CUDA_VISIBLE_DEVICES=2,3`	2 and 3	`cuda:0` and `cuda:1`
`CUDA_VISIBLE_DEVICES=5`	5	`cuda:0`
`CUDA_VISIBLE_DEVICES=3,1`	3 and 1	`cuda:0` = physical 3, `cuda:1` = physical 1

nvidia-smi ignores CUDA_VISIBLE_DEVICES. It uses NVML, not the CUDA runtime, so it still shows all GPUs with their NVML indices. If your app reports "cuda:0" busy while nvidia-smi shows GPU 2 busy, that's expected renumbering, not a bug.

The indices in CUDA_VISIBLE_DEVICES are interpreted using CUDA's ordering. If CUDA's order differs from NVML's, then "2,3" may not be the GPUs nvidia-smi calls 2 and 3. That's exactly why CUDA_DEVICE_ORDER=PCI_BUS_ID matters when Slurm hands out indices. UUIDs avoid the problem entirely.

5. Containers: `NVIDIA_VISIBLE_DEVICES`

Used by the NVIDIA Container Toolkit to decide which GPU devices are mounted into the container.

Value	Meaning
`all`	All GPUs passed into the container
`0,1` or UUIDs	Only those GPUs
`none`	No GPUs, but the driver libraries are still mounted
`void` / empty / unset	The NVIDIA runtime does nothing (behaves like a normal container)

docker run --gpus all ...                    # Docker flag, equivalent idea
docker run --gpus '"device=0,1"' ...
docker run -e NVIDIA_VISIBLE_DEVICES=2,3 --runtime=nvidia ...
# Related: NVIDIA_DRIVER_CAPABILITIES=compute,utility  (which driver features are exposed)

Layers: NVIDIA_VISIBLE_DEVICES decides which GPUs enter the container. Inside it, CUDA_VISIBLE_DEVICES can narrow further, and CUDA_DEVICE_ORDER decides how CUDA numbers what's left.

6. Slurm: GRES, cgroups, and what it sets

Configuration

# slurm.conf
GresTypes=gpu
NodeName=dgx[01-32] Gres=gpu:h100:8 ...

# gres.conf: either auto-detect via NVML ...
AutoDetect=nvml
# ... or list devices explicitly
NodeName=dgx[01-32] Name=gpu Type=h100 File=/dev/nvidia[0-7]

# cgroup.conf: actually restrict jobs to their GPUs
ConstrainDevices=yes

Requesting GPUs

srun --gpus-per-node=2 ...      # or --gres=gpu:2, or --gpus=2
sbatch --gres=gpu:h100:8 job.sh

What Slurm sets for the job

Variable	Meaning
`CUDA_VISIBLE_DEVICES`	The GPUs the job may use, as CUDA should see them. With cgroup device constraint on, the job usually sees its GPUs numbered from 0.
`SLURM_JOB_GPUS` / `SLURM_STEP_GPUS`	The global GPU indices allocated on the node (useful for logging which physical GPUs were used)
`GPU_DEVICE_ORDINAL`	Also set by Slurm, for other GPU runtimes

Slurm uses NVML (PCI bus) ordering to pick GPUs, then tells CUDA which ones to use through CUDA_VISIBLE_DEVICES. If CUDA interprets indices in a different order (FASTEST_FIRST), the job can land on GPUs Slurm didn't intend. Fix: CUDA_DEVICE_ORDER=PCI_BUS_ID in the job environment. Also enable ConstrainDevices=yes so jobs physically can't touch other jobs' GPUs.

Related binding options: --gpu-bind=closest (bind tasks to their nearest GPU), --cpu-bind, and Cores= in gres.conf for CPU/NUMA affinity, which matters for performance on DGX.

7. Kubernetes, MIG, and other layers

Kubernetes: the NVIDIA device plugin (deployed by the GPU Operator) allocates GPUs to pods (resources.limits: nvidia.com/gpu: 2) and passes them in through the container toolkit (it sets NVIDIA_VISIBLE_DEVICES or uses CDI). Don't hand-set GPU indices in pods.
MIG: each MIG instance has a MIG-… UUID. Select it with CUDA_VISIBLE_DEVICES=MIG-… (or the device plugin's MIG strategy). A CUDA process sees one MIG instance per device.
DCGM / nvidia-smi: report in NVML (PCI bus) order. Use them as the reference.
NCCL / PyTorch: each rank usually picks cuda:local_rank, which is relative to whatever CUDA_VISIBLE_DEVICES exposes.

8. The big comparison table

Variable / setting	Controls	Read by	Typical values
`CUDA_DEVICE_ORDER`	Order of CUDA device numbering	CUDA runtime	`FASTEST_FIRST` (default), `PCI_BUS_ID`
`CUDA_VISIBLE_DEVICES`	Which GPUs a process sees (and renumbers from 0)	CUDA runtime (not nvidia-smi)	`0,1`, `GPU-uuid`, `MIG-uuid`, empty
`NVIDIA_VISIBLE_DEVICES`	Which GPUs are mounted into a container	NVIDIA Container Toolkit	`all`, `none`, `void`, indices, UUIDs
`NVIDIA_DRIVER_CAPABILITIES`	Which driver features are exposed to a container	NVIDIA Container Toolkit	`compute,utility`, `all`
`gres.conf` / `AutoDetect=nvml`	How Slurm discovers GPUs	Slurm	nvml, or explicit `File=/dev/nvidia[0-7]`
`cgroup.conf ConstrainDevices=yes`	Hard isolation of GPUs per job	Slurm	yes
`SLURM_JOB_GPUS`	Global indices allocated (informational)	Set by Slurm	`4,5`
`nvidia-smi -i`	Which GPU nvidia-smi acts on	NVML	Index, UUID, PCI bus ID

Invented variables to reject on sight: SLURM_GPU_ORDER, DCGM_DEVICE_ORDER, NVIDIA_DEVICE_ORDER, NCCL_GPU_ORDER. The real ordering variable is CUDA_DEVICE_ORDER.

9. Simulator: what does my app see?

A pretend 8-GPU node where CUDA's default order differs from PCI order (as can happen on mixed systems). Change the settings and see the mapping.

CUDA_DEVICE_ORDER

CUDA_VISIBLE_DEVICES (comma list, blank = unset)

10. How to verify ordering on a real node

# 1. NVML's view (the reference): index → PCI bus ID → UUID
nvidia-smi --query-gpu=index,pci.bus_id,uuid --format=csv

# 2. CUDA's view: print each CUDA device's PCI bus ID
#    (CUDA's deviceQuery sample prints PCI bus/device IDs; or in PyTorch:)
python - <<'EOF'
import torch
for i in range(torch.cuda.device_count()):
    p = torch.cuda.get_device_properties(i)
    print(i, p.name, getattr(p, "pci_bus_id", "n/a"), getattr(p, "uuid", "n/a"))
EOF

# 3. Compare with and without the fix
CUDA_DEVICE_ORDER=PCI_BUS_ID python check.py

# 4. Inside a Slurm job
srun --gpus-per-node=2 bash -c 'echo CVD=$CUDA_VISIBLE_DEVICES JOB_GPUS=$SLURM_JOB_GPUS; nvidia-smi -L'

Which device properties PyTorch exposes varies by version. The deviceQuery CUDA sample always prints PCI IDs.

11. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

nvidia-smi --query-gpu=index,pci.bus_id,uuid,name --format=csvMap NVML index to bus ID and UUID

index, pci.bus_id, uuid, name
0, 00000000:18:00.0, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000000, NVIDIA H100 80GB HBM3
1, 00000000:2A:00.0, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000001, NVIDIA H100 80GB HBM3
2, 00000000:3A:00.0, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000002, NVIDIA H100 80GB HBM3
3, 00000000:5D:00.0, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000003, NVIDIA H100 80GB HBM3
4, 00000000:9A:00.0, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000004, NVIDIA H100 80GB HBM3
5, 00000000:AB:00.0, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000005, NVIDIA H100 80GB HBM3
6, 00000000:BA:00.0, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000006, NVIDIA H100 80GB HBM3
7, 00000000:DB:00.0, GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000007, NVIDIA H100 80GB HBM3

What to look at: This is the reference view: NVML numbers GPUs in ascending PCI bus ID order, so index 0 is bus 18 and index 7 is bus DB. If the bus IDs are not in ascending order down the list, or a GPU is missing, something other than plain NVML ordering is going on and you should note the UUIDs before trusting any index.

nvidia-smi -LList GPUs with UUIDs

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000000)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000001)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000002)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000003)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000004)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000005)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000006)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000007)

What to look at: Each line pairs an NVML index with the GPU's permanent UUID, which is the identifier to use in automation and in CUDA_VISIBLE_DEVICES. When MIG is enabled, indented MIG lines with MIG- UUIDs appear under their parent GPU; no MIG lines means no MIG instances exist.

python - <<'EOF'
import torch
for i in range(torch.cuda.device_count()):
    p = torch.cuda.get_device_properties(i)
    print(i, p.name, getattr(p, "pci_bus_id", "n/a"), getattr(p, "uuid", "n/a"))
EOF

Print CUDA's view of each device

0 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000000
1 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000001
2 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000002
3 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000003
4 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000004
5 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000005
6 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000006
7 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000007

What to look at: Compare each CUDA index's UUID (printed without the GPU- prefix) with nvidia-smi -L; on a DGX H100 with identical GPUs they usually line up even with the default FASTEST_FIRST order. If CUDA index 0 shows a UUID that nvidia-smi lists under a different index, CUDA and NVML disagree and you need CUDA_DEVICE_ORDER=PCI_BUS_ID. This PyTorch build has no pci_bus_id property, so it prints n/a.

CUDA_DEVICE_ORDER=PCI_BUS_ID python check.pyCUDA enumeration forced to PCI order

0 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000000
1 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000001
2 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000002
3 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000003
4 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000004
5 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000005
6 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000006
7 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000007

What to look at: With PCI_BUS_ID set, CUDA index N must have the same UUID as nvidia-smi GPU N; here the order is identical, which is now guaranteed rather than luck. Any mismatch at this point would mean the variable did not reach the process, for example because it was set in the wrong shell or stripped by a container or launcher.

CUDA_VISIBLE_DEVICES=2,3 python check.pyExpose only physical GPUs 2 and 3

0 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000002
1 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000003

What to look at: Only two devices are visible and they are renumbered from 0: cuda:0 has GPU 2's UUID (ending ...0002) and cuda:1 has GPU 3's. Seeing 8 devices here means the variable was not exported to the process.

CUDA_VISIBLE_DEVICES=3,1 python check.pyListed order sets the CUDA numbering

0 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000003
1 NVIDIA H100 80GB HBM3 n/a 1a2b3c4d-5e6f-7a8b-9c0d-000000000001

What to look at: The order you list is the order CUDA uses, so cuda:0 is physical GPU 3 and cuda:1 is physical GPU 1. Do not assume visible devices are re-sorted by index.

CUDA_VISIBLE_DEVICES=MIG-9f3c0000-1111-5222-8333-444455556666 python check.pyRun on one specific MIG instance

0 NVIDIA H100 80GB HBM3 MIG 3g.40gb n/a 9f3c0000-1111-5222-8333-444455556666

What to look at: The process sees exactly one device, cuda:0, and its name includes the MIG profile (3g.40gb). If you see no devices, the MIG UUID is wrong or that instance was destroyed, so take a fresh UUID from nvidia-smi -L.

CUDA_VISIBLE_DEVICES= python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"Empty value hides every GPU

False 0

What to look at: An empty CUDA_VISIBLE_DEVICES (or an invalid one such as -1) hides every GPU, so CUDA reports none available. A job that unexpectedly prints False 0 on a GPU node often inherited an empty value from a launcher or script; CUDA programs report this as error 100, no CUDA-capable device is detected.

CUDA_VISIBLE_DEVICES=2,3 nvidia-smi -Lnvidia-smi ignores CUDA_VISIBLE_DEVICES

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000000)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000001)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000002)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000003)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000004)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000005)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000006)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000007)

What to look at: All 8 GPUs are still listed with their NVML indices, because nvidia-smi uses NVML rather than the CUDA runtime. That is expected; it is why an app's cuda:0 can show up as GPU 2 in nvidia-smi.

docker run --rm --gpus all nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -LPass every GPU into a container

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000000)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000001)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000002)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000003)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000004)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000005)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000006)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000007)

What to look at: With --gpus all the container toolkit mounts all 8 GPUs, and the UUIDs match the host. Fewer GPUs here than on the host means the container was started with a narrower device list.

docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=2,3 nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -LMount only GPUs 2 and 3

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000002)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000003)

What to look at: Only two GPUs are mounted, and inside the container they are numbered 0 and 1; the UUIDs (ending ...0002 and ...0003) prove they are host GPUs 2 and 3. Always confirm by UUID, because the in-container index no longer matches the host index.

docker run --rm nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smiContainer started without GPU access

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown.

What to look at: Without --gpus or the NVIDIA runtime the toolkit injects nothing, so even nvidia-smi is missing from the container. This is a visibility problem: fix it with --gpus all or NVIDIA_VISIBLE_DEVICES, not with CUDA_DEVICE_ORDER.

srun --gpus-per-node=2 bash -c 'echo CVD=$CUDA_VISIBLE_DEVICES JOB_GPUS=$SLURM_JOB_GPUS; nvidia-smi -L'Show what Slurm gives a job

CVD=0,1 JOB_GPUS=4,5
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000004)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000005)

What to look at: SLURM_JOB_GPUS shows the node-global GPUs (4 and 5), while CUDA_VISIBLE_DEVICES and nvidia-smi inside the cgroup number them 0 and 1; the UUIDs confirm they are physical GPUs 4 and 5. If nvidia-smi inside the job lists all 8 GPUs, ConstrainDevices=yes is not in effect and the job could touch other jobs' GPUs.

sbatch --gres=gpu:h100:8 job.shSubmit a job requesting 8 H100s

Submitted batch job 41237

What to look at: A job ID means Slurm accepted the request. "Invalid generic resource (gres) specification" means the GPU type or count does not match the node's Gres= definition in slurm.conf and gres.conf.

sudo nvidia-smi -i GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000002 -rReset a GPU by UUID

GPU 00000000:3A:00.0 was successfully reset.
All done.

What to look at: Targeting by UUID guarantees you reset the intended physical GPU; the reply echoes its bus ID (3A), which you can cross-check against the index mapping. An "Unable to reset" message means processes or agents still hold that GPU.

12. Scenarios

A job's log says it used cuda:0, but nvidia-smi shows activity on GPU 5. Bug?

Probably not. Slurm (or the user) set CUDA_VISIBLE_DEVICES=5, so the process renumbers it as cuda:0. nvidia-smi ignores that variable and shows NVML index 5. Check SLURM_JOB_GPUS to confirm.

Two jobs on the same node both end up on the same physical GPU.

Likely causes: the CUDA ordering doesn't match NVML (Slurm handed out indices in PCI order, CUDA interpreted them in FASTEST_FIRST order), and/or ConstrainDevices isn't enabled, so nothing stops a job from using the wrong GPU. Fix: set CUDA_DEVICE_ORDER=PCI_BUS_ID cluster-wide and enable ConstrainDevices=yes in cgroup.conf.

A monitoring script maps DCGM alerts on "GPU 3" to the wrong job.

DCGM uses NVML indices. The job's CUDA indices are renumbered. Correlate by UUID or PCI bus ID (or by SLURM_JOB_GPUS) instead of local indices.

A container shows no GPUs even though the host has 8.

NVIDIA_VISIBLE_DEVICES is unset or void, or the container wasn't started with the NVIDIA runtime / --gpus. Set --gpus all or NVIDIA_VISIBLE_DEVICES=all. Note this fixes visibility, not ordering.

A user wants to run on one specific MIG slice.

Get the MIG UUID from nvidia-smi -L, then CUDA_VISIBLE_DEVICES=MIG-xxxx ./app. (In Slurm or Kubernetes, request the MIG resource instead and let the scheduler set it.)

Automation that resets "GPU 2" sometimes resets the wrong GPU after maintenance.

Indices aren't stable identities. Target by UUID or PCI bus ID: nvidia-smi -i GPU-xxxx -r.

13. Exam traps

Ordering question → CUDA_DEVICE_ORDER=PCI_BUS_ID. Visibility question → CUDA_VISIBLE_DEVICES or NVIDIA_VISIBLE_DEVICES.
NVIDIA_VISIBLE_DEVICES = containers (Container Toolkit). CUDA_VISIBLE_DEVICES = processes (CUDA runtime).
CUDA's default order is FASTEST_FIRST. NVML (nvidia-smi, DCGM, Slurm NVML autodetect) uses PCI bus order.
nvidia-smi ignores CUDA_VISIBLE_DEVICES.
CUDA_VISIBLE_DEVICES renumbers from 0 in the order listed.
For stable identification, use UUIDs (or PCI bus IDs), not indices.
Slurm isolation = ConstrainDevices=yes (cgroups). Discovery = gres.conf (AutoDetect=nvml).
Made-up variables (SLURM_GPU_ORDER, DCGM_DEVICE_ORDER) are distractors.

14. Practice quiz (15 questions)

Score: 0 / 0