nvidia-smi Deep Dive

So many flags. This guide gives you a mental map so you can work out a command instead of memorizing hundreds of them.

1. What nvidia-smi is (and isn't)

nvidia-smi (NVIDIA System Management Interface) is the command-line tool that ships with the NVIDIA driver. It talks to the GPUs through NVML (NVIDIA Management Library) to show GPU state and change GPU settings on one machine.

nvidia-smi is for…	nvidia-smi is NOT for…
Seeing GPU status: utilization, memory, temperature, power, processes Changing GPU settings: power limit, clocks, ECC, compute mode, persistence Setting up MIG (creating and destroying instances) Showing topology: how GPUs, NICs, and CPUs connect Resetting a GPU	Fleet-wide continuous monitoring, health policies, stress diagnostics (use DCGM) Disks, storage, partitions (BCM / Linux tools) InfiniBand / NCCL network config Configuring the NVSwitch fabric (Fabric Manager) Scheduling jobs (Slurm / Kubernetes)

nvidia-smi is the dashboard and control panel of one car. You can read the speedometer and fuel gauge, and you can change some settings. DCGM is the fleet-management system for a whole taxi company.

2. The secret: every command follows one grammar

Stop trying to memorize every command. Almost every nvidia-smi command is built from these four parts:

nvidia-smi [WHICH GPU] [WHAT TO DO] [HOW TO SHOW IT] [HOW OFTEN]

Part	Flags	Example
WHICH GPU	`-i <index \| UUID \| PCI bus ID>`. Leave it out to mean all GPUs.	`-i 0`, `-i 0,3`
WHAT TO DO	Query (`-q`, `--query-gpu=`), a setting (`-pl`, `-pm`, `-e`, `-c`, `-lgc`, `-r`, `-mig`), or a subcommand (`dmon`, `pmon`, `topo`, `nvlink`, `mig`)	`-q -d POWER`, `-pl 500`
HOW TO SHOW IT	`--format=csv[,noheader][,nounits]`, `-x` (XML), `-f file`	`--format=csv,noheader`
HOW OFTEN	`-l <seconds>` or `-lms <milliseconds>` loop	`-l 5`

Decode some real commands with the grammar

nvidia-smi -i 2 -q -d TEMPERATURE
  # GPU 2 · full query, only the temperature section

nvidia-smi --query-gpu=index,utilization.gpu,memory.used --format=csv,noheader -l 2
  # all GPUs · these 3 fields · CSV without header · every 2 seconds

sudo nvidia-smi -i 0 -pl 500
  # GPU 0 · set power limit to 500 W

nvidia-smi dmon -s pu -i 0,1 -d 1
  # subcommand dmon · power + utilization · GPUs 0 and 1 · every 1 second

When you see an unfamiliar command, split it into which GPU + what action + format + frequency. Most exam variations only swap one part.

3. Reading the default output, field by field

Running plain nvidia-smi prints a table. Exams love asking what each field means.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15         Driver Version: 550.54.15      CUDA Version: 12.4          |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:18:00.0 Off |                    0 |
| N/A   34C    P0             72W /  700W |       4MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Field	Meaning
Driver Version	The installed NVIDIA driver. Newer drivers also print the versions on their own with `nvidia-smi --version` (NVIDIA-SMI, NVML, driver, and CUDA versions).
CUDA Version	The highest CUDA version this driver supports, not the CUDA toolkit you have installed. (Check the toolkit with `nvcc --version`.) A classic trap.
Persistence-M	Persistence mode On/Off. On keeps the driver loaded even with no GPU users, so programs start faster. On DGX it's handled by the `nvidia-persistenced` daemon.
Bus-Id	PCI address of the GPU. You can use it with `-i`.
Disp.A	Display active (whether a monitor is driven). Always Off on data-center GPUs.
Volatile Uncorr. ECC	Count of uncorrectable ECC memory errors since the driver last loaded. Anything above 0 deserves attention.
Fan	Fan speed %. N/A on passively cooled data-center GPUs (the chassis fans cool them).
Temp	GPU core temperature (°C).
Perf	Performance state, P0 (maximum performance) to P12 (minimum). Idle GPUs often sit at P8 or similar. A busy GPU should be at P0.
Pwr:Usage/Cap	Current power draw / power limit. The H100 SXM maximum is 700 W.
Memory-Usage	GPU memory used / total. Some memory is always reserved by the driver.
GPU-Util	Percent of time during the sample period that at least one kernel was running. 100% does not mean all the cores are busy.
Compute M.	Compute mode: Default, Exclusive_Process, or Prohibited.
MIG M.	MIG mode Enabled/Disabled.
Processes	Processes using the GPU. Inside containers this often says "No running processes found" because the container can't see the host's process IDs.

4. The 6 flag buckets

Every flag belongs to one of these six buckets. Learn the buckets first, then the flags inside each one.

1. Query

-q -d -L --query-gpu --query-compute-apps
"Tell me about the GPU"

2. Select & format

-i --format -x -f
"Which GPU, and in what shape"

3. Monitor

-l -lms dmon pmon
"Keep watching"

4. Settings (root)

-pm -pl -e -c -ac/-rac -lgc/-rgc -lmc/-rmc -r -am
"Change the GPU"

5. Topology

topo -m nvlink -s nvlink -c nvlink -e
"How things are connected"

6. MIG

-mig 1 mig -lgip mig -cgi … -C mig -lgi/-lci mig -dci/-dgi
"Split the GPU"

5. Bucket 1: Querying

Three ways to ask, from simple to scriptable

Command	Output	Use when
`nvidia-smi`	The summary table	Quick human glance
`nvidia-smi -L`	One line per GPU with its UUID (and MIG devices)	Need GPU names and UUIDs
`nvidia-smi -q`	Everything, very long	Deep investigation
`nvidia-smi -q -d SECTION`	Only the chosen section(s) of `-q`	Focused investigation
`nvidia-smi --query-gpu=FIELDS --format=csv`	Just the fields you asked for, as CSV	Scripts, logging, monitoring

`-d` sections (used with `-q`)

-d means display only these sections. It only makes sense together with -q. You can list several, separated by commas.

Section	Shows
`MEMORY`	Memory total/used/free
`UTILIZATION`	GPU, memory, encoder, decoder utilization
`ECC`	Corrected/uncorrected ECC errors, volatile and aggregate
`TEMPERATURE`	Current temperature plus slowdown/shutdown thresholds
`POWER`	Draw, limit, default/min/max limits
`CLOCK`	Current, application, and max clocks
`SUPPORTED_CLOCKS`	Valid clock combinations (needed before `-ac`)
`PERFORMANCE`	P-state and clock event / throttle reasons
`COMPUTE`	Compute mode
`PIDS`	Processes using the GPU
`ROW_REMAPPER`	Memory row remapping status (A100/H100 memory health)
`PAGE_RETIREMENT`	Retired memory pages (older GPUs)
`ACCOUNTING`	Per-process accounting data (if accounting mode is on)

nvidia-smi -q -d TEMPERATURE,POWER          # two sections
nvidia-smi -i 0 -q -d PERFORMANCE           # why is GPU 0 throttling?
nvidia-smi -q -d ECC,ROW_REMAPPER           # memory health

`--query-gpu`: the scripting workhorse

You pick exact fields and must give --format. List all available fields with nvidia-smi --help-query-gpu.

Common field	Meaning
`index`, `name`, `uuid`, `pci.bus_id`	Identity
`driver_version`	Driver
`temperature.gpu`	Temperature
`utilization.gpu`, `utilization.memory`	Utilization %
`memory.total`, `memory.used`, `memory.free`	Memory
`power.draw`, `power.limit`	Power
`clocks.sm`, `clocks.mem`, `clocks.max.sm`	Clocks
`pstate`	P-state
`persistence_mode`, `compute_mode`, `mig.mode.current`	Modes
`ecc.errors.uncorrected.volatile.total`	Uncorrectable ECC since driver load
`clocks_event_reasons.active` (older drivers: `clocks_throttle_reasons.active`)	Why clocks are reduced

nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw \
           --format=csv
# index, name, temperature.gpu, utilization.gpu [%], memory.used [MiB], memory.total [MiB], power.draw [W]
# 0, NVIDIA H100 80GB HBM3, 34, 0 %, 4 MiB, 81559 MiB, 72.10 W

nvidia-smi --query-gpu=index,utilization.gpu --format=csv,noheader,nounits
# 0, 0          <- no header row, no "%" or "MiB": easiest for scripts

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
# which processes are using GPU memory

--format=csv options: noheader drops the title row, and nounits drops "%", "MiB", "W". Combine them like csv,noheader,nounits.

6. Bucket 2: Selecting GPUs and output format

Flag	Meaning	Example
`-i, --id=`	Target GPU(s) by index, UUID, or PCI bus ID	`-i 0` · `-i 0,2` · `-i GPU-4b1e…` · `-i 00000000:18:00.0`
`--format=csv`	CSV output (required with `--query-*`)	`--format=csv,noheader,nounits`
`-x`	XML output of `-q`	`nvidia-smi -q -x`
`-f FILE`	Write output to a file	`-f /tmp/gpu.log`

GPU index numbering can change between reboots or differ from CUDA_VISIBLE_DEVICES ordering. For reliable automation, target GPUs by UUID or PCI bus ID.

7. Bucket 3: Live monitoring

Loop any query

nvidia-smi -l 5                       # redraw the table every 5 seconds
nvidia-smi --query-gpu=index,utilization.gpu,power.draw --format=csv -l 1    # log every second
nvidia-smi --query-gpu=power.draw --format=csv -lms 200                      # every 200 ms
watch -n 1 nvidia-smi                 # Linux 'watch' alternative

`nvidia-smi dmon`: device monitor (one line per GPU per interval)

Pick metric groups with -s and a combination of letters:

Letter	Group	Columns
`p`	Power / temperature	pwr, gtemp, mtemp
`u`	Utilization	sm %, mem %, enc %, dec %
`c`	Clocks	mclk, pclk
`v`	Power / thermal violations	pviol, tviol
`m`	Memory	fb (used), bar1
`e`	ECC and PCIe replay errors	sbecc, dbecc, pci
`t`	PCIe throughput	rxpci, txpci

Mnemonic: "p-u-c-v-m-e-t", read as "Please Use Care, Very Much Every Time."

nvidia-smi dmon                       # default metric set
nvidia-smi dmon -s pucvmet            # everything
nvidia-smi dmon -s u -i 0 -d 2 -c 10  # utilization, GPU 0, every 2 s, 10 samples then exit
#  -s select metrics · -i GPU · -d delay (seconds) · -c count

`nvidia-smi pmon`: process monitor (one line per process)

nvidia-smi pmon -s um                 # per-process sm%, mem% (u) and memory used (m)
nvidia-smi pmon -i 0 -d 1             # processes on GPU 0, every second

dmon = device view (per GPU). pmon = process view (per process). Both are good for quick live checks on one node. For fleet monitoring, use DCGM.

8. Bucket 4: Changing settings

All of these need root (sudo). Most don't survive a reboot, so automate them (systemd unit, BCM config, or dcgmi config) if you need them to stick.

Flag	What it does	Values / notes
`-pm 0\|1`	Persistence Mode	1 = keep the driver loaded with no clients (faster job startup). Preferred modern approach: the `nvidia-persistenced` service.
`-pl WATTS`	Power Limit	Must be within min/max (see `-q -d POWER`). Lower it to save power or heat, at some performance cost.
`-e 0\|1`	ECC on/off	Takes effect after a reboot or GPU reset. Keep ECC on in data centers.
`-c MODE`	Compute mode	`0`/DEFAULT = many processes can share the GPU · `2`/PROHIBITED = no compute · `3`/EXCLUSIVE_PROCESS = only one process at a time. (1 = old EXCLUSIVE_THREAD, deprecated.)
`-ac MEM,GFX` / `-rac`	Set / reset application clocks	Pick values from `-q -d SUPPORTED_CLOCKS`. `-rac` = reset application clocks.
`-lgc MIN,MAX` / `-rgc`	Lock / reset GPU clocks	Pin SM clocks to a range, for example for consistent benchmarks.
`-lmc MIN,MAX` / `-rmc`	Lock / reset memory clocks	Same idea for memory clocks.
`-r` (`--gpu-reset`)	Reset the GPU	The GPU must be idle (no processes). Use after ECC errors, pending row remaps, or MIG mode changes. On NVSwitch systems, related GPUs may have to be reset together.
`-mig 0\|1`	Enable/disable MIG mode	May need a GPU reset to take effect. Then create instances with `nvidia-smi mig`.
`-am 0\|1`	Accounting mode	Record per-process GPU usage. View with `--query-accounted-apps=…`, clear with `-caa`.

Naming patterns that make these easier

Two letters = what + thing: -pm (persistence mode), -pl (power limit), -am (accounting mode), -ac (application clocks).
Clock family = l/r + g/m + c: lock or reset · gpu or memory · clocks, giving -lgc -rgc -lmc -rmc.
Leading "r" = reset: -rac, -rgc, -rmc, and -r on its own resets the whole GPU.
Single letters: -e ECC, -c compute mode, -r reset, -i id, -q query, -d display, -l loop, -x XML, -f file, -L list.

sudo nvidia-smi -pm 1                     # persistence on, all GPUs
sudo nvidia-smi -i 0 -pl 600              # GPU 0 power limit 600 W
sudo nvidia-smi -c 3                      # EXCLUSIVE_PROCESS on all GPUs
sudo nvidia-smi -i 0 -e 1                 # enable ECC (then reset/reboot)
sudo nvidia-smi -lgc 1980,1980            # lock SM clock for a stable benchmark
sudo nvidia-smi -rgc                      # undo the lock
sudo nvidia-smi -i 3 -r                   # reset idle GPU 3

9. Bucket 5: Topology and NVLink

`nvidia-smi topo -m`: the connection matrix

Shows how each pair of GPUs (and NICs) is connected, plus which CPU cores and NUMA node are closest to each GPU. This matters for NCCL performance and for pinning processes.

        GPU0  GPU1  ...  NIC0  NIC1  CPU Affinity  NUMA Affinity
GPU0     X    NV18  ...  PIX   SYS   0-55          0
GPU1    NV18   X    ...  SYS   PIX   0-55          0
NIC0    PIX   SYS   ...   X    SYS

Legend, from closest / fastest to farthest / slowest:

Code	Meaning
`X`	Itself
`NV#`	Connected by # bonded NVLinks (fastest). On a DGX H100, GPU pairs show `NV18` through NVSwitch.
`PIX`	Through at most a single PCIe bridge (very close, for example a GPU and its paired NIC)
`PXB`	Through multiple PCIe bridges, without crossing the CPU's host bridge
`PHB`	Through a PCIe host bridge (via the CPU)
`NODE`	Across PCIe host bridges within the same NUMA node
`SYS`	Across the CPU-to-CPU interconnect between NUMA nodes (farthest, slowest)

Mnemonic, closest to farthest: X → NV → PIX → PXB → PHB → NODE → SYS.

Why NV# beats every PCIe code

The codes fall into two families. NV# means the two GPUs talk over NVLink, a dedicated GPU-to-GPU link. Every other code (PIX through SYS) means traffic goes over PCIe. Those codes only describe how many hops the PCIe path takes. So the fastest PCIe path is still slower than any NVLink path.

Path	Rough bandwidth (H100 era)	What the data crosses
`NV18`	~900 GB/s total (18 links × ~50 GB/s, both directions)	NVLink → NVSwitch → NVLink. No PCIe, no CPU.
`PIX`	~64 GB/s each way (PCIe Gen5 x16)	One PCIe switch
`PXB`	Same link speed, more switch hops	Several PCIe switches, still not the CPU
`PHB`	Same link speed, plus CPU root-complex latency	Up into the CPU's PCIe host bridge and back down
`NODE`	Lower; crosses between root complexes	Between host bridges inside one CPU socket / NUMA node
`SYS`	Lowest; shares the socket link with everything else	Across the CPU-to-CPU link (UPI / Infinity Fabric) to the other socket

Moving down the list, each step adds latency and shares more bandwidth with other devices. Peer-to-peer transfers and GPUDirect RDMA also become less likely to work well. At SYS they are often disabled, and data is bounced through CPU memory instead.

What the number in NV# means

The number is the count of NVLinks between that pair. More links means more bandwidth, so NV18 beats NV12, which beats NV2.
DGX H100 / HGX H100: each H100 has 18 NVLink 4 links, all wired to NVSwitch chips. Because the switch gives every pair the full set, every GPU pair shows NV18, even though there's no direct cable between them.
DGX A100: each A100 has 12 NVLink 3 links through NVSwitch, so pairs show NV12.
Older systems with no NVSwitch (DGX-1 hybrid cube mesh) or PCIe cards joined with NVLink bridges: some pairs show NV1/NV2 or a bridged count, while unbridged pairs fall back to a PCIe code such as NODE or SYS. A mixed matrix like this is normal there.

Reading the GPU-to-NIC columns

The same codes apply between GPUs and NICs. On a DGX H100, each GPU's compute NIC sits behind the same PCIe switch as that GPU, so the pair shows PIX. That closeness is what makes GPUDirect RDMA fast: the NIC reads GPU memory without touching the CPU. A GPU-to-NIC entry of SYS means that NIC belongs to the other socket. Using it for that GPU's traffic costs bandwidth.

NCCL uses these same names as thresholds. NCCL_P2P_LEVEL sets how far apart two GPUs may be for direct peer-to-peer. NCCL_NET_GDR_LEVEL sets how far a GPU may be from a NIC for GPUDirect RDMA. Values include NVL, PIX, PXB, PHB and SYS. If topo -m shows the paths are farther apart than the threshold, NCCL falls back to a slower route. See the NCCL deep dive.

Expected vs suspicious on a DGX H100

You see	Meaning
All GPU pairs `NV18`	Healthy. NVSwitch fabric is up.
Some GPU pairs show `SYS` / `NODE` instead of `NV18`	NVLink isn't in use for those pairs. Check that Fabric Manager is running, then `nvidia-smi nvlink -s` for down links.
GPU pair shows `NV` with a lower count (e.g. `NV16`)	Some links are down or degraded. Check `nvidia-smi nvlink -s` and `nvidia-smi nvlink -e`.
Each GPU's paired NIC shows `PIX`	Healthy. GPUDirect RDMA path is local.

nvidia-smi topo -m              # the matrix
nvidia-smi topo -p2p n          # can each pair do peer-to-peer over NVLink? (r/w/a/p for read/write/atomics/prop)
nvidia-smi nvlink -s            # per-link speed; "inactive" links lower the NV# count

Exam trap: PIX is the best PCIe code, which makes it a tempting answer for "fastest". But it's still PCIe. Whenever an NV# option appears, it wins. And between two NV options, the larger number wins.

Read the codes as a walk away from the GPU: NVLink (no walk), PIX (one switch), PXB (several switches), PHB (into the CPU), NODE (across the CPU), SYS (to the other CPU).

NVLink

nvidia-smi nvlink -s              # status: each link's speed (or inactive)
nvidia-smi nvlink -s -i 0         # just GPU 0
nvidia-smi nvlink -c              # capabilities per link
nvidia-smi nvlink -e              # error counters (flag set varies by driver version)

nvidia-smi can show NVLink status. Fabric Manager must be running on NVSwitch systems (DGX/HGX H100) for NVLink to work at all.

10. Bucket 6: MIG (Multi-Instance GPU)

MIG splits one H100 into up to 7 isolated GPU instances. Setup is a strict sequence, and the order is commonly tested. In virtualized setups, a MIG slice can also back a VM's vGPU (see GPUs in Virtual Machines).

Enable MIG mode on the GPU (it must be idle; a reset may be needed):
```
sudo nvidia-smi -i 0 -mig 1
```

List available profiles (GPU instance profiles):

sudo nvidia-smi mig -lgip         # e.g. 1g.10gb, 2g.20gb, 3g.40gb, 4g.40gb, 7g.80gb

Create GPU instances, plus default compute instances inside them with -C:

sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C   # two 3g.40gb slices (profile IDs also work)

Or in two steps: create the GPU instance, then the compute instance inside it with -cci:

sudo nvidia-smi mig -i 0 -cgi 9          # GPU instance only (profile ID 9 = 3g.40gb)
sudo nvidia-smi mig -lcip                # compute-instance profiles
sudo nvidia-smi mig -cci 0 -gi 1         # compute instance (CI profile 0) in GPU instance 1

Verify:

nvidia-smi -L                     # shows MIG devices with MIG-UUIDs
sudo nvidia-smi mig -lgi          # list GPU instances
sudo nvidia-smi mig -lci          # list compute instances

Tear down in reverse order: compute instances first, then GPU instances, then disable MIG:

sudo nvidia-smi mig -dci          # destroy compute instances
sudo nvidia-smi mig -dgi          # destroy GPU instances
sudo nvidia-smi -i 0 -mig 0       # disable MIG mode

Profile name	Reading it
`1g.10gb`	1/7 of the compute slices, 10 GB memory (smallest; up to 7 per H100 80GB)
`3g.40gb`	3 compute slices, 40 GB
`7g.80gb`	The whole GPU as one instance

MIG sub-commands: -lgip = list gpu-instance profiles · -cgi = create gpu instance · -cci = create compute instance · -C (with -cgi) = also create the default compute instance · -lgi/-lci = list GPU / compute instances · -dgi/-dci = destroy them.

You can't destroy a GPU instance while compute instances still exist inside it. Remove the compute instances (-dci) first. Also, MIG instances don't do peer-to-peer NVLink, so MIG is for small, isolated workloads, not multi-GPU training.

11. ECC, row remapping, throttling: health clues

Clue	Where to see it	What it means / action
Corrected (single-bit) ECC errors	`-q -d ECC`	Fixed automatically. Watch the trend.
Uncorrected (double-bit) ECC errors	Default table "Volatile Uncorr. ECC", `-q -d ECC`	Serious. Drain the node, investigate, and run DCGM diag.
Volatile vs aggregate ECC	`-q -d ECC`	Volatile = since the last driver load or reset. Aggregate = lifetime of the GPU.
Row remapping pending	`-q -d ROW_REMAPPER`	The GPU wants to swap a bad memory row for a spare. Reset the GPU (`-r`) so it takes effect.
Row remapping failure	`-q -d ROW_REMAPPER`	No spare rows left. The GPU likely needs replacing (RMA).
Clocks lowered (throttling)	`-q -d PERFORMANCE` ("Clocks Event Reasons", formerly "Throttle Reasons")	Reasons include Idle, SW Power Cap (hit the power limit), HW Slowdown, HW Thermal Slowdown, SW Thermal Slowdown, and HW Power Brake.
Performance state stuck at P8 under load	Default table "Perf"	The GPU isn't ramping up. Check clocks, power, and whether work is really reaching the GPU.

12. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

nvidia-smiSummary table of all GPUs

Tue Nov 10 19:04:12 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15         Driver Version: 550.54.15      CUDA Version: 12.4          |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:18:00.0 Off |                    0 |
| N/A   34C    P0             72W /  700W |       4MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  |   00000000:2A:00.0 Off |                    0 |
| N/A   33C    P0             70W /  700W |       4MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
...                                    (GPUs 2-6 trimmed)
|   7  NVIDIA H100 80GB HBM3          On  |   00000000:DB:00.0 Off |                    0 |
| N/A   35C    P0             73W /  700W |       4MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

What to look at: Check that the driver and CUDA versions match your baseline, all 8 GPUs appear, Volatile Uncorr. ECC is 0 and Persistence-M is On. Remember that CUDA Version is the newest CUDA the driver supports, not the installed toolkit; a nonzero ECC count, a missing GPU, or ERR! in any field needs investigation.

nvidia-smi -LList GPUs with UUIDs

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000000)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000001)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000002)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000003)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000004)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000005)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000006)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000007)

What to look at: You should see 8 lines, one per H100, each with a unique GPU- UUID. Fewer than 8 lines means a GPU failed to initialize or fell off the bus, so check dmesg for Xid 79 and compare with lspci.

nvidia-smi -q -d TEMPERATURE,POWERTemperature and power sections only

==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 19:04:12 2026
Driver Version                            : 550.54.15
CUDA Version                              : 12.4

Attached GPUs                             : 8
GPU 00000000:18:00.0
    Temperature
        GPU Current Temp                  : 34 C
        GPU T.Limit Temp                  : 53 C
        GPU Shutdown T.Limit Temp         : -8 C
        GPU Slowdown T.Limit Temp         : -2 C
        GPU Max Operating T.Limit Temp    : 0 C
        GPU Target Temperature            : N/A
        Memory Current Temp               : 42 C
        Memory Max Operating T.Limit Temp : 0 C
    GPU Power Readings
        Power Draw                        : 71.52 W
        Current Power Limit               : 700.00 W
        Requested Power Limit             : 700.00 W
        Default Power Limit               : 700.00 W
        Min Power Limit                   : 200.00 W
        Max Power Limit                   : 700.00 W
    Power Samples
        Duration                          : 2.38 sec
        Number of Samples                 : 119
        Max                               : 72.10 W
        Min                               : 71.02 W
        Avg                               : 71.48 W
    Module Power Readings
        Power Draw                        : N/A

(GPUs 00000000:2A:00.0 to 00000000:DB:00.0 trimmed)

What to look at: On H100 drivers the thresholds are shown as T.Limit margins: GPU T.Limit Temp is how many degrees remain before slowdown, so a value near 0 or negative means the GPU is about to throttle. Under POWER, a Current Power Limit below the 700 W default means someone lowered it with -pl, and any new -pl value must fall between Min and Max Power Limit.

nvidia-smi -i 0 -q -d PERFORMANCEP-state and clock event reasons

==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 19:04:12 2026
Driver Version                            : 550.54.15
CUDA Version                              : 12.4

Attached GPUs                             : 8
GPU 00000000:18:00.0
    Performance State                     : P0
    Clocks Event Reasons
        Idle                              : Not Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active
    Sparse Operation Mode                 : N/A

What to look at: Under a heavy job, P0 with only SW Power Cap Active is normal: the GPU is running at its configured power limit. HW Slowdown, HW Thermal Slowdown or HW Power Brake Slowdown showing Active points to a cooling or power-delivery problem; power brake on all 8 GPUs at once usually means a PSU problem on the node.

nvidia-smi -q -d ECC,ROW_REMAPPERECC counts and row-remap health

==============NVSMI LOG==============

Timestamp                                 : Tue Nov 10 19:04:12 2026
Driver Version                            : 550.54.15
CUDA Version                              : 12.4

Attached GPUs                             : 8
GPU 00000000:18:00.0
    ECC Mode
        Current                           : Enabled
        Pending                           : Enabled
    ECC Errors
        Volatile
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 0
            DRAM Uncorrectable            : 0
        Aggregate
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 2
            DRAM Uncorrectable            : 0
            SRAM Threshold Exceeded       : No
    Remapped Rows
        Correctable Error                 : 0
        Uncorrectable Error               : 0
        Pending                           : No
        Remapping Failure Occurred        : No
        Bank Remap Availability Histogram
            Max                           : 2560 bank(s)
            High                          : 0 bank(s)
            Partial                       : 0 bank(s)
            Low                           : 0 bank(s)
            None                          : 0 bank(s)

(GPUs 00000000:2A:00.0 to 00000000:DB:00.0 trimmed)

What to look at: This GPU is healthy: ECC is enabled, there are no uncorrectable errors, a couple of lifetime (Aggregate) correctable errors, and no pending remaps. Pending: Yes means you must drain and reset the GPU so the remap takes effect, while Remapping Failure Occurred: Yes means no spare rows are left and the GPU needs RMA.

nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw --format=csvSelected GPU fields as CSV

index, name, temperature.gpu, utilization.gpu [%], memory.used [MiB], memory.total [MiB], power.draw [W]
0, NVIDIA H100 80GB HBM3, 61, 100 %, 71230 MiB, 81559 MiB, 688.41 W
1, NVIDIA H100 80GB HBM3, 60, 100 %, 71230 MiB, 81559 MiB, 679.87 W
2, NVIDIA H100 80GB HBM3, 62, 100 %, 71230 MiB, 81559 MiB, 691.02 W
3, NVIDIA H100 80GB HBM3, 59, 99 %, 71230 MiB, 81559 MiB, 673.55 W
4, NVIDIA H100 80GB HBM3, 33, 0 %, 4 MiB, 81559 MiB, 74.14 W
5, NVIDIA H100 80GB HBM3, 34, 0 %, 4 MiB, 81559 MiB, 75.15 W
6, NVIDIA H100 80GB HBM3, 33, 0 %, 4 MiB, 81559 MiB, 76.16 W
7, NVIDIA H100 80GB HBM3, 34, 0 %, 4 MiB, 81559 MiB, 77.17 W

What to look at: The header row names each field and its unit, and each GPU gets one row; here a job is running on GPUs 0-3 while 4-7 are idle. If you forget --format=csv the command errors out, and a GPU with high utilization but low power draw may not be getting real work.

nvidia-smi --query-gpu=index,utilization.gpu --format=csv,noheader,nounitsScript-friendly CSV, no header or units

What to look at: There is no header line and no % sign, so a script can split each line on the comma directly. If you see a header or units here, the noheader or nounits option was left out of --format.

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csvProcesses using GPU memory

pid, process_name, used_memory [MiB]
51234, /usr/bin/python3, 71230 MiB
51235, /usr/bin/python3, 71230 MiB
51236, /usr/bin/python3, 71230 MiB
51237, /usr/bin/python3, 71230 MiB

What to look at: Each row is a process holding GPU memory, with its PID and how much memory it uses. Inside a container this list is often empty even while GPUs are busy, because the container cannot see host PIDs; a PID still listed after a job ended is a leftover process that will block a GPU reset.

nvidia-smi dmon -s pucvmetLive per-GPU metrics, all groups

# gpu    pwr  gtemp  mtemp     sm    mem    enc    dec    jpg    ofa   mclk   pclk  pviol  tviol     fb   bar1   ccpm  sbecc  dbecc    pci  rxpci  txpci
# Idx      W      C      C      %      %      %      %      %      %    MHz    MHz      %   bool     MB     MB     MB   errs   errs   errs   MB/s   MB/s
    0    688     61     70    100     58      0      0      0      0   2619   1980      3      0  71876      5      0      0      0      0   1203    987
    1    684     60     70    100     58      0      0      0      0   2619   1980      0      0  71876      5      0      0      0      0   1203    987
    2    680     61     70    100     58      0      0      0      0   2619   1980      5      0  71876      5      0      0      0      0   1203    987
    3    676     60     70     99     58      0      0      0      0   2619   1980      0      0  71876      5      0      0      0      0   1203    987
    4     74     33     40      0      0      0      0      0      0   2619    345      0      0      0      1      0      0      0      0      0      0
    5     75     33     40      0      0      0      0      0      0   2619    345      0      0      0      1      0      0      0      0      0      0
    6     76     33     40      0      0      0      0      0      0   2619    345      0      0      0      1      0      0      0      0      0      0
    7     77     33     40      0      0      0      0      0      0   2619    345      0      0      0      1      0      0      0      0      0      0
^C

What to look at: Each interval prints one line per GPU: busy GPUs 0-3 show sm 100 % at the 1980 MHz max SM clock, and idle GPUs drop to 345 MHz. Watch pviol and tviol (power and thermal violations), and sbecc, dbecc and pci (ECC and PCIe replay errors); a nonzero dbecc or a steadily rising pci count is a hardware warning.

nvidia-smi pmon -s umLive per-process utilization and memory

# gpu         pid   type     sm    mem    enc    dec    jpg    ofa     fb   ccpm    command
# Idx           #    C/G      %      %      %      %      %      %     MB     MB    name
    0       51234      C     98     57      -      -      -      -  71230      0    python3
    1       51235      C     97     57      -      -      -      -  71230      0    python3
    2       51236      C     96     57      -      -      -      -  71230      0    python3
    3       51237      C     95     57      -      -      -      -  71230      0    python3
    4           -      -      -      -      -      -      -      -      -      -    -
    5           -      -      -      -      -      -      -      -      -      -    -
    6           -      -      -      -      -      -      -      -      -      -    -
    7           -      -      -      -      -      -      -      -      -      -    -
^C

What to look at: Each line is one process on one GPU, with type C for compute, its SM and memory utilization, and framebuffer memory in MB. A GPU showing only dashes has no processes; two unexpected PIDs on the same GPU suggest jobs are sharing a GPU they should not.

sudo nvidia-smi -pm 1Enable persistence mode on all GPUs

Persistence mode is already Enabled for GPU 00000000:18:00.0.
Persistence mode is already Enabled for GPU 00000000:2A:00.0.
Persistence mode is already Enabled for GPU 00000000:3A:00.0.
Persistence mode is already Enabled for GPU 00000000:5D:00.0.
Persistence mode is already Enabled for GPU 00000000:9A:00.0.
Persistence mode is already Enabled for GPU 00000000:AB:00.0.
Persistence mode is already Enabled for GPU 00000000:BA:00.0.
Persistence mode is already Enabled for GPU 00000000:DB:00.0.
All done.

What to look at: On a DGX the nvidia-persistenced service already keeps persistence on, so nvidia-smi reports it is already enabled; on a fresh node you would see "Enabled persistence mode for GPU ..." instead. If Persistence-M keeps showing Off after a reboot, the persistenced service is not running, because nvidia-smi settings do not survive a reboot.

sudo nvidia-smi -i 0 -pl 600Set GPU 0 power limit to 600 W

Power limit for GPU 00000000:18:00.0 was set to 600.00 W from 700.00 W.

All done.

What to look at: The message confirms the old and new limits for that GPU's bus ID. A value outside the Min/Max Power Limit range is rejected with an error, and running without sudo fails with "Insufficient Permissions".

sudo nvidia-smi -c 3Set EXCLUSIVE_PROCESS compute mode

Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:18:00.0.
Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:2A:00.0.
Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:3A:00.0.
Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:5D:00.0.
Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:9A:00.0.
Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:AB:00.0.
Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:BA:00.0.
Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:DB:00.0.
All done.

What to look at: Every GPU now shows E. Process in the Compute M. column of the default table, so only one process can use each GPU. If a second process then fails with CUDA error 46 (devices busy or unavailable), that is this setting working, not a fault.

sudo nvidia-smi -lgc 1980,1980Lock SM clocks for benchmarking

GPU clocks set to "(gpuClkMin 1980, gpuClkMax 1980)" for GPU 00000000:18:00.0
GPU clocks set to "(gpuClkMin 1980, gpuClkMax 1980)" for GPU 00000000:2A:00.0
GPU clocks set to "(gpuClkMin 1980, gpuClkMax 1980)" for GPU 00000000:3A:00.0
GPU clocks set to "(gpuClkMin 1980, gpuClkMax 1980)" for GPU 00000000:5D:00.0
GPU clocks set to "(gpuClkMin 1980, gpuClkMax 1980)" for GPU 00000000:9A:00.0
GPU clocks set to "(gpuClkMin 1980, gpuClkMax 1980)" for GPU 00000000:AB:00.0
GPU clocks set to "(gpuClkMin 1980, gpuClkMax 1980)" for GPU 00000000:BA:00.0
GPU clocks set to "(gpuClkMin 1980, gpuClkMax 1980)" for GPU 00000000:DB:00.0

All done.

What to look at: Each GPU's SM clock range is now pinned at 1980 MHz, so benchmark runs are repeatable. Undo it with sudo nvidia-smi -rgc afterwards; if clocks still drop below 1980 MHz under load, check PERFORMANCE for power or thermal clock event reasons.

sudo nvidia-smi -i 3 -rReset an idle GPU

Unable to reset GPU 00000000:5D:00.0 because it's being used by some other process (e.g. CUDA application, graphics application like X server, monitoring application like other instance of nvidia-smi). Please first kill all processes using the device and retry the command or reboot the system to make GPU reset operational.

(after stopping the job and the nvidia-dcgm service, the same command prints:)
GPU 00000000:5D:00.0 was successfully reset.
All done.

What to look at: A reset only works on an idle GPU, so the first attempt fails while a job or monitoring agent holds it. "was successfully reset" is the good result; if the reset keeps failing with nothing running, reboot the node instead.

nvidia-smi topo -mGPU/NIC/CPU connection matrix

        GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  NIC0  NIC1  NIC2  NIC3  NIC4  NIC5  NIC6  NIC7  NIC8  NIC9  NIC10 NIC11 CPU Affinity      NUMA Affinity  GPU NUMA ID
GPU0     X    NV18  NV18  NV18  NV18  NV18  NV18  NV18  PIX   NODE  NODE  NODE  NODE  NODE  SYS   SYS   SYS   SYS   SYS   SYS   0-55,112-167      0              N/A
GPU1    NV18   X    NV18  NV18  NV18  NV18  NV18  NV18  NODE  NODE  NODE  PIX   NODE  NODE  SYS   SYS   SYS   SYS   SYS   SYS   0-55,112-167      0              N/A
GPU2    NV18  NV18   X    NV18  NV18  NV18  NV18  NV18  NODE  NODE  NODE  NODE  PIX   NODE  SYS   SYS   SYS   SYS   SYS   SYS   0-55,112-167      0              N/A
GPU3    NV18  NV18  NV18   X    NV18  NV18  NV18  NV18  NODE  NODE  NODE  NODE  NODE  PIX   SYS   SYS   SYS   SYS   SYS   SYS   0-55,112-167      0              N/A
GPU4    NV18  NV18  NV18  NV18   X    NV18  NV18  NV18  SYS   SYS   SYS   SYS   SYS   SYS   PIX   NODE  NODE  NODE  NODE  NODE  56-111,168-223    1              N/A
GPU5    NV18  NV18  NV18  NV18  NV18   X    NV18  NV18  SYS   SYS   SYS   SYS   SYS   SYS   NODE  NODE  NODE  PIX   NODE  NODE  56-111,168-223    1              N/A
GPU6    NV18  NV18  NV18  NV18  NV18  NV18   X    NV18  SYS   SYS   SYS   SYS   SYS   SYS   NODE  NODE  NODE  NODE  PIX   NODE  56-111,168-223    1              N/A
GPU7    NV18  NV18  NV18  NV18  NV18  NV18  NV18   X    SYS   SYS   SYS   SYS   SYS   SYS   NODE  NODE  NODE  NODE  NODE  PIX   56-111,168-223    1              N/A
(NIC0-NIC11 rows trimmed)

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9
  NIC10: mlx5_10
  NIC11: mlx5_11

What to look at: On a healthy DGX H100 every GPU pair reads NV18, each GPU has exactly one PIX NIC (its own compute HCA), and GPUs 0-3 sit on NUMA node 0 while 4-7 sit on NUMA node 1. A GPU pair showing SYS, NODE or a lower count such as NV16 means NVLink is not fully up, so check Fabric Manager and nvidia-smi nvlink -s.

nvidia-smi nvlink -s -i 0NVLink status per link, GPU 0

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-1a2b3c4d-5e6f-7a8b-9c0d-000000000000)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s

What to look at: An H100 should list 18 links, all at the same speed. A link shown as <inactive> lowers the NV# count in topo -m and needs Fabric Manager, nvlink -e error counters, and Xid 74 checks.

sudo nvidia-smi mig -lgipList MIG GPU-instance profiles

+-----------------------------------------------------------------------------+
| GPU instance profiles:                                                      |
| GPU   Name             ID    Instances   Memory     P2P    SM    DEC   ENC  |
|                              Free/Total   GiB              CE    JPEG  OFA  |
|=============================================================================|
|   0  MIG 1g.10gb       19      7/7       9.75       No     16    1     0    |
|                                                            1     1     0    |
+-----------------------------------------------------------------------------+
|   0  MIG 1g.10gb+me    20      1/1       9.75       No     16    1     0    |
|                                                            1     1     1    |
+-----------------------------------------------------------------------------+
|   0  MIG 1g.20gb       15      4/4       19.62      No     26    1     0    |
|                                                            1     1     0    |
+-----------------------------------------------------------------------------+
|   0  MIG 2g.20gb       14      3/3       19.62      No     32    2     0    |
|                                                            2     2     0    |
+-----------------------------------------------------------------------------+
|   0  MIG 3g.40gb        9      2/2       39.50      No     60    3     0    |
|                                                            3     3     0    |
+-----------------------------------------------------------------------------+
|   0  MIG 4g.40gb        5      1/1       39.50      No     64    4     0    |
|                                                            4     4     0    |
+-----------------------------------------------------------------------------+
|   0  MIG 7g.80gb        0      1/1       79.25      No     132   7     0    |
|                                                            8     7     1    |
+-----------------------------------------------------------------------------+

What to look at: Each row is a profile you can create, with its ID, how many are still free, memory, and SM count; for example 3g.40gb is profile ID 9 with 2 free. If the command reports no MIG-enabled devices, MIG mode is not active yet (run -mig 1 and reset the GPU), and Free 0 means existing instances already use that space.

sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -CCreate two 3g.40gb instances

Successfully created GPU instance ID  2 on GPU  0 using profile MIG 3g.40gb (ID  9)
Successfully created compute instance ID  0 on GPU  0 GPU instance ID  2 using profile MIG 3g.40gb (ID  2)
Successfully created GPU instance ID  1 on GPU  0 using profile MIG 3g.40gb (ID  9)
Successfully created compute instance ID  0 on GPU  0 GPU instance ID  1 using profile MIG 3g.40gb (ID  2)

What to look at: With -C, every GPU instance line is followed by a matching compute instance line; without -C you only get GPU instances and CUDA cannot use them yet. "Insufficient Resources" means the requested profiles do not fit in what is still free.

sudo nvidia-smi mig -dci && sudo nvidia-smi mig -dgiTear down MIG instances in order

Successfully destroyed compute instance ID  0 from GPU  0 GPU instance ID  1
Successfully destroyed compute instance ID  0 from GPU  0 GPU instance ID  2
Successfully destroyed GPU instance ID  1 from GPU  0
Successfully destroyed GPU instance ID  2 from GPU  0

What to look at: Compute instances are destroyed first, then GPU instances, and only after that do you disable MIG with -mig 0. Running -dgi first fails with an "In use by another client" error because compute instances still exist inside the GPU instance.

13. "Which command?" scenarios

Log GPU utilization and memory to CSV every 5 seconds for a script:
nvidia-smi --query-gpu=timestamp,index,utilization.gpu,memory.used --format=csv -l 5 -f gpu.csv

Why is GPU 2 running slower than the others?
nvidia-smi -i 2 -q -d PERFORMANCE,TEMPERATURE,POWER,CLOCK (check throttle reasons)

Which process is eating GPU memory?
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv or nvidia-smi pmon

Make sure only one process can use each GPU:
sudo nvidia-smi -c 3 (EXCLUSIVE_PROCESS)

Reduce power draw to fit a rack's power budget:
sudo nvidia-smi -pl <watts> (for a persistent, group-wide setting, use dcgmi config or a boot-time service)

Stable, repeatable benchmark results:
sudo nvidia-smi -lgc <min>,<max>, then -rgc afterwards

Which NIC is closest to GPU 5? Which CPU cores should my process use?
nvidia-smi topo -m (look for PIX, and the CPU Affinity column)

The first CUDA call takes several seconds on an idle node:
Enable persistence: nvidia-persistenced or sudo nvidia-smi -pm 1

Split one H100 into 7 small GPUs for inference:
sudo nvidia-smi -i 0 -mig 1 → sudo nvidia-smi mig -cgi 1g.10gb,1g.10gb,1g.10gb,1g.10gb,1g.10gb,1g.10gb,1g.10gb -C

Row remapping shows "pending":
Drain, then sudo nvidia-smi -i <id> -r (or reboot), then recheck with -q -d ROW_REMAPPER

Get machine-readable full details for a tool:
nvidia-smi -q -x (XML)

14. Searchable cheat sheet

Command	What it does

15. Exam traps

"CUDA Version" in the header = the maximum CUDA version the driver supports, not the installed toolkit.
GPU-Util 100% = a kernel was running all the time. It doesn't mean the GPU is used efficiently (use DCGM profiling metrics for that).
-d without -q doesn't make sense. -d filters the sections of -q. (In dmon, though, -d means delay. Context matters!)
--query-gpu needs --format=csv.
Settings need root and most reset on reboot.
ECC changes need a reboot or reset to take effect.
-r GPU reset needs an idle GPU. It fails if processes are still running.
MIG teardown order: compute instances, then GPU instances, then disable MIG.
Compute mode numbers: 0 Default, 2 Prohibited, 3 Exclusive_Process. (1 is deprecated.)
nvidia-smi ≠ DCGM: no stress diagnostics, no fleet health policies, no per-job stats across the cluster.
nvidia-smi can't configure the NVSwitch fabric (Fabric Manager), can't touch disks, and can't fix InfiniBand.
Topology order: NV# (best) → PIX → PXB → PHB → NODE → SYS (worst).
Perf states: P0 = maximum performance, higher numbers = lower power and performance.

16. Practice quiz (20 questions)

Score: 0 / 0