GPUs in Virtual Machines

Running GPU workloads on a hypervisor instead of bare metal: what hypervisors are, the four ways to give a VM a GPU (passthrough, time-sliced vGPU, MIG-backed vGPU, and the bare-metal alternative), VMware ESXi and esxcli in depth, vGPU licensing and versioning, KVM equivalents, multi-GPU and RDMA inside VMs, and the nuances that trip people up.

1. The big picture: bare metal vs VMs vs containers

There are three common ways to run a GPU workload on a server:

	Bare metal	Virtual machines (VMs)	Containers
What runs on the hardware	One OS (for example DGX OS / Ubuntu)	A hypervisor, which runs several guest OSes	One OS; containers share its kernel
Isolation between tenants	None (one OS)	Strong: each VM has its own kernel and driver	Process-level (shared kernel)
GPU driver lives in	The host OS	The guest (plus a host component for vGPU)	The host OS (containers borrow it)
Performance	Best	Close to bare metal for GPU compute; some overhead and limits on scale-out	Same as bare metal
Typical use	Large-scale training (DGX SuperPOD)	Enterprise IT standardized on VMware, multi-tenant clusters, inference and dev environments, security boundaries	Almost everything, often inside bare metal or VMs

Containers aren't an alternative to VMs: a common enterprise setup is containers inside VMs (for example Kubernetes nodes that are VMs on vSphere). NVIDIA AI Enterprise is supported on bare metal and on hypervisors such as VMware vSphere.

Bare metal is a detached house: one family, all the space, no walls to share. VMs are an apartment building: the building manager (hypervisor) gives each tenant (VM) its own locked apartment with its own kitchen (kernel and driver). A GPU is like a big oven: you can give a whole oven to one apartment (passthrough), or install a smart oven that several apartments book in turns or in separate compartments (vGPU).

Large multi-node training is usually bare metal, because it needs every GPU, NVLink, and InfiniBand at full speed. VMs win when you need strong tenant isolation, sharing of GPUs across many smaller workloads, or the enterprise's existing virtualization operations (live migration, HA, standard tooling).

2. Hypervisor basics and vocabulary

Term	Meaning
Hypervisor	Software that creates and runs VMs, sharing the physical CPU, memory, and devices between them
Type 1 (bare-metal) hypervisor	Installed directly on the server: VMware ESXi, KVM-based (Red Hat, Ubuntu, Nutanix AHV, Proxmox), Microsoft Hyper-V. Used in data centers.
Type 2 (hosted) hypervisor	Runs as an app on a normal OS (VirtualBox, VMware Workstation). Desktop use, not data-center GPU work.
Host	The physical server running the hypervisor
Guest	The OS inside a VM
vCPU	A virtual CPU core given to a VM
IOMMU (Intel VT-d / AMD-Vi)	Hardware that safely maps a physical device's memory access into one VM. Required for passthrough.
SR-IOV	Single Root I/O Virtualization: one PCIe device presents several "virtual functions" (VFs) that can be assigned to VMs. Used by NICs, and by vGPU on recent NVIDIA GPUs.
Live migration (VMware: vMotion)	Moving a running VM to another host without downtime

3. VMware vSphere building blocks

Component	What it is	How you use it
ESXi	VMware's Type 1 hypervisor, installed on each server	DCUI (the console menu), ESXi Shell / SSH, `esxcli`, Host Client web UI
vCenter Server	Central management for many ESXi hosts: clusters, vMotion, HA, DRS	The vSphere Client web UI, APIs, PowerCLI
Datastore	Storage where VM files and uploaded packages live (VMFS, NFS, vSAN), under `/vmfs/volumes/<datastore>/`	Upload driver bundles here before installing
VIB (vSphere Installation Bundle)	ESXi's software package format, like a .deb or .rpm. NVIDIA's vGPU Manager ships as a VIB or as a component inside an offline bundle (.zip).	`esxcli software vib ...`, or vSphere Lifecycle Manager
Maintenance mode	A host state where no VMs run, so it can be patched or rebooted	`esxcli system maintenanceMode set` or vCenter
vSphere Lifecycle Manager (vLCM)	Cluster-wide image management: ESXi version + vendor add-ons + extra components such as the NVIDIA driver	Add the NVIDIA component to the cluster image, then remediate hosts one at a time
DRS / HA	Automatic VM placement and load balancing / restarting VMs after a host failure	Both interact with GPU VMs; passthrough VMs are pinned to their host

SSH and the ESXi Shell are disabled by default on ESXi. Enable them temporarily (DCUI or vSphere Client → host → Configure → Services) when you need esxcli locally, and turn them off again afterwards.

4. Four ways to give a VM a GPU

	Passthrough (DirectPath I/O)	Time-sliced vGPU	MIG-backed vGPU	Bare metal (for contrast)
VM gets	One or more whole physical GPUs	A slice of GPU memory; compute shared in turns	A MIG instance: dedicated SMs, memory, cache	All GPUs, no VM
GPUs shared?	No	Yes, several VMs per GPU	Yes, up to 7 per GPU (H100/A100)	N/A
Isolation between VMs	Full (separate GPU)	Memory isolated; compute time-shared, so a busy neighbor affects you	Hardware isolation of compute, memory, and faults	N/A
Host software	None from NVIDIA	NVIDIA vGPU Manager	NVIDIA vGPU Manager + MIG mode on the GPU	Driver on the host OS
License	Not a vGPU license (normal data-center driver; NVIDIA AI Enterprise subscription for its software)	vGPU license required	vGPU license required	Optional (NVIDIA AI Enterprise)
Live migration	No	Supported (with requirements)	Depends on release and hypervisor	N/A
Best for	Training in VMs, multi-GPU jobs, maximum performance	Many small inference / dev / notebook VMs	Multi-tenant inference needing guaranteed QoS	Large-scale training

Passthrough = Private whole GPU. Time-sliced = Takes turns. MIG-backed = Mini GPUs carved in hardware.

5. Host prerequisites: BIOS, IOMMU, SR-IOV, large BARs

Before any of this works, the server's firmware (BIOS/UEFI) must allow it:

BIOS setting	Needed for	Why
Intel VT-x / AMD-V (CPU virtualization)	Running VMs at all	Lets the CPU run guest OSes efficiently
Intel VT-d / AMD-Vi (IOMMU)	Passthrough	Maps the device's memory access into one VM safely
SR-IOV	vGPU on Ampere and later (A100, H100), SR-IOV NICs	Recent NVIDIA GPUs present each vGPU as an SR-IOV virtual function
Above 4G decoding / large MMIO	Passthrough and vGPU with large-memory GPUs	Modern GPUs expose very large memory windows (BARs) that don't fit below 4 GB of address space

If a GPU doesn't appear as passthrough-capable, or vGPU types don't show up, check the BIOS first (VT-d/IOMMU, SR-IOV, above-4G decoding). No amount of guest configuration fixes a disabled IOMMU.

6. Passthrough (DirectPath I/O) in depth

With passthrough, ESXi hands the physical PCIe device to one VM. The guest loads the normal NVIDIA data-center driver, exactly as on bare metal, and the hypervisor stays out of the data path.

Enable passthrough for the device: vSphere Client → host → Configure → Hardware → PCI Devices → select the GPU → toggle passthrough. (Some ESXi versions need a host reboot afterwards.)
Add the device to the VM: Edit Settings → Add New Device → PCI Device → pick the GPU. Newer vSphere versions offer Dynamic DirectPath I/O, which selects a matching device by type rather than a fixed PCI address.
Reserve all guest memory: a VM with a passthrough device must have its full memory reserved (the device can DMA into any guest page, so pages can't be swapped).
Use EFI firmware for the VM, not legacy BIOS.

Enable 64-bit MMIO in the VM's advanced settings:

pciPassthru.use64bitMMIO = "TRUE"
pciPassthru.64bitMMIOSizeGB = "<size>"   # large enough for all passed-through GPUs' BARs

Install the guest driver (and CUDA, containers) inside the VM, then check nvidia-smi.

Sizing guidance for 64bitMMIOSizeGB differs by VMware release and GPU count. A common approach is to take the total memory of the passed-through GPUs and round up to the next power of two, but check the current VMware and NVIDIA guidance for your versions.

Passthrough = bare-metal performance and features for one VM, at the cost of flexibility: no vMotion, no sharing, full memory reservation, and the VM is tied to that host. If the VM won't boot or the GPU won't initialize, the usual suspects are BIOS IOMMU/above-4G settings, legacy BIOS firmware on the VM, or missing 64-bit MMIO settings.

7. NVIDIA vGPU in depth: components and profiles

The pieces

Component	Where	Role
NVIDIA Virtual GPU Manager	Hypervisor host (ESXi: a VIB / component)	Owns the physical GPU, creates vGPUs, schedules them, enforces memory slices. Makes `nvidia-smi` work on the host.
vGPU guest driver	Each VM	Looks like a normal NVIDIA driver to applications; talks to its vGPU
License	Served by the NVIDIA License System, checked out by each guest	Unlocks full vGPU capability in the guest
vGPU profile (vGPU type)	Chosen per VM in vCenter	Defines the slice: memory size, series (compute vs graphics), time-sliced or MIG-backed

Reading a profile name

Profile names follow a pattern: <GPU>-<memory in GB><series letter>, with MIG-backed profiles also showing the compute slice count.

Example	Meaning
`H100-80C`	Whole H100 memory (80 GB), C = compute series: one VM gets the full GPU through vGPU
`H100-20C`	20 GB compute slice, time-sliced: up to 4 such VMs share one 80 GB GPU
`A100-3-20C`	MIG-backed: 3 compute slices with 20 GB (the 3g.20gb MIG profile)

Series letter	Intended for
C	Compute: AI, data science, HPC. The series used with NVIDIA AI Enterprise.
Q	Virtual workstations (professional graphics plus compute)
B	Virtual PCs (knowledge-worker desktops)
A	Virtual applications (published apps)

Exact profile names, which series a given GPU supports, and how they appear in vCenter (for example with a grid_ prefix) vary by GPU and vGPU release. The pattern above is what to learn. On the host, nvidia-smi vgpu -s lists supported types and nvidia-smi vgpu -c lists types that can be created right now.

The number in a profile is GPU memory, not compute share. A time-sliced -20C VM still competes for compute time with its neighbors. If guaranteed compute is needed, use MIG-backed profiles (or whole-GPU profiles / passthrough).

8. Time-sliced vs MIG-backed vGPU

	Time-sliced vGPU	MIG-backed vGPU
How compute is shared	VMs take turns on all of the GPU's compute engines	Each VM owns a fixed set of SMs; they run truly in parallel
Memory	Fixed slice per VM (from the profile)	Dedicated memory per MIG instance
Noisy neighbor	Possible: a busy VM can slow others, depending on the scheduler	Isolated in hardware
Scheduler choices	Best effort (default: idle time goes to whoever needs it), equal share, fixed share (predictable, even when others are idle)	Not needed: hardware partitions
Setup on the host	Install vGPU Manager; assign profiles	Also enable MIG mode on the GPU on the host, then use MIG-backed profiles
Fits	Bursty dev, notebook, and light inference VMs	Production multi-tenant inference with QoS needs

Same rule as on bare metal (see GPU sharing): time-slicing shares a GPU cheaply but without performance guarantees; MIG gives hardware isolation. vGPU simply delivers either one to VMs instead of processes.

Whether different time-sliced profile sizes can mix on one physical GPU, and how the scheduling policy is set, depends on the vGPU release. Older releases required all time-sliced vGPUs on one GPU to use the same profile.

9. vGPU licensing

vGPU guests check out a license from the NVIDIA License System:

CLS (Cloud License Service): hosted by NVIDIA.
DLS (Delegated License Service): a virtual appliance you run on-premises, for sites without internet access.

Each guest is pointed at the license service with a client configuration token file, and the guest's licensing service (nvidia-gridd on Linux) checks out and renews the license.

# Inside a Linux vGPU guest
sudo cp client_configuration_token_*.tok /etc/nvidia/ClientConfigToken/
sudo systemctl restart nvidia-gridd
nvidia-smi -q | grep -i -A2 license        # expect a licensed status

An unlicensed vGPU guest still shows a GPU in nvidia-smi, but runs with reduced capability after a grace period. "The GPU is there but performance is restricted" in a vGPU VM points to licensing. Passthrough doesn't use vGPU licensing.

Token paths, service names, and grace-period behavior are release-specific. Check the vGPU licensing guide for your release.

10. Versions and compatibility

Must match / be compatible	Why
vGPU Manager ↔ ESXi version	The VIB is built for specific ESXi releases. After an ESXi upgrade, the NVIDIA component may need upgrading too.
vGPU Manager ↔ guest driver	Use guest drivers from the same vGPU release (or a combination listed as compatible). A mismatch can leave the vGPU unusable in the guest.
Guest driver ↔ CUDA in the guest	Same rules as bare metal: the driver must support the CUDA version your containers use. See CUDA compatibility.
GPU ↔ vGPU release	Each release lists supported GPUs and hypervisors

Upgrade in a planned order: check the support matrix, upgrade the host vGPU Manager (in maintenance mode, host by host), then upgrade guest drivers to the matching release. Don't let guests drift onto a newer branch than the host supports.

11. esxcli: structure and access

esxcli is the command line for managing an ESXi host. It's organized as a tree of namespaces, then a command, then options:

esxcli <namespace> [<sub-namespace> ...] <command> [--option value ...]

esxcli software vib list                     # namespace "software", sub-namespace "vib", command "list"
esxcli system maintenanceMode set --enable true
esxcli --help                                # list top-level namespaces
esxcli software vib --help                   # list commands in a namespace
esxcli --formatter=csv software vib list     # machine-readable output (also: xml, keyvalue)

Ways to run it

Method	When
SSH / ESXi Shell on the host	Most common for one host. Enable SSH temporarily.
DCUI → troubleshooting options → ESXi Shell	Console access when the network is broken
Remotely, with VMware's CLI tooling or PowerCLI (`Get-EsxCli -V2`)	Scripting across many hosts without SSH

esxcli is to an ESXi host what systemctl + apt + ip + lspci are to a Linux server, all under one command with a consistent structure.

esxcli manages the hypervisor host, not the guest OSes. Installing a driver inside a VM is done with the guest's tools (for example apt or the NVIDIA .run installer), not esxcli.

12. esxcli command tour by namespace

system: host identity, maintenance mode, reboot

esxcli system version get                       # ESXi version and build
esxcli system hostname get
esxcli system maintenanceMode get               # Enabled / Disabled
esxcli system maintenanceMode set --enable true # enter (VMs must be migrated or powered off first)
esxcli system maintenanceMode set --enable false
esxcli system shutdown reboot --reason "vGPU Manager install"   # host must be in maintenance mode

software: VIBs, components, acceptance level

esxcli software vib list | grep -i nvidia       # is the NVIDIA VIB installed? which version?
esxcli software vib get -n <vib-name>           # details of one VIB
esxcli software vib install -v /vmfs/volumes/<datastore>/<file>.vib      # install a single VIB
esxcli software vib install -d /vmfs/volumes/<datastore>/<bundle>.zip    # install from an offline bundle (depot)
esxcli software vib update  -d /vmfs/volumes/<datastore>/<bundle>.zip    # upgrade in place
esxcli software vib remove  -n <vib-name>       # remove
esxcli software vib install -d <bundle>.zip --dry-run   # show what would change
esxcli software component list                  # ESXi 7+: software components
esxcli software acceptance get                  # host acceptance level

Acceptance levels control which signed packages a host accepts, from most to least trusted: VMwareCertified, VMwareAccepted, PartnerSupported, CommunitySupported. A VIB below the host's level is rejected. Fix that with a properly signed package, not by bypassing signature checks.

hardware and graphics: is the GPU there, and how is it used?

esxcli hardware pci list | grep -i -A3 nvidia    # physical GPUs on the PCI bus
esxcli hardware platform get                     # server vendor and model
esxcli graphics device list                      # GPUs as the graphics stack sees them
esxcli graphics vm list                          # which VMs are using GPUs
esxcli graphics host get                         # default graphics type and assignment policy
esxcli graphics host set --default-type SharedPassthru                       # "Shared Direct": needed for vGPU
esxcli graphics host set --shared-passthru-assignment-policy Performance     # spread vGPUs across GPUs (or Consolidation)
vmkload_mod -l | grep nvidia                     # is the NVIDIA kernel module loaded?
nvidia-smi                                       # works on the host once the vGPU Manager is installed
nvidia-smi vgpu                                  # running vGPUs per GPU
nvidia-smi vgpu -q                               # detailed vGPU info
nvidia-smi vgpu -s / -c                          # supported / currently creatable vGPU types

Assignment policy	Behavior
Performance (spread)	Place each new vGPU on the least-loaded GPU, so VMs get more GPU time
Consolidation (group)	Fill one GPU before using the next, leaving whole GPUs free for large profiles

network and storage

esxcli network nic list                         # physical NICs (vmnic0...), link, speed
esxcli network ip interface ipv4 get            # VMkernel interface IPs
esxcli network vswitch standard list            # standard virtual switches
esxcli network firewall get
esxcli storage filesystem list                  # datastores and mount points (/vmfs/volumes/...)
esxcli storage core device list                 # physical / LUN devices

vm: last-resort control of running VMs

esxcli vm process list                           # running VMs with their World ID
esxcli vm process kill --type soft  --world-id <id>   # try soft first, then hard, then force

Exact option names can differ slightly between ESXi versions. Run esxcli <namespace> --help on your host to confirm.

13. vGPU Manager install and upgrade on ESXi

First install on one host

Check the support matrix: GPU, ESXi version, vGPU release.
Confirm BIOS settings (VT-d, SR-IOV for Ampere and later, above-4G decoding).
Upload the NVIDIA bundle to a datastore (/vmfs/volumes/<datastore>/).
Migrate or power off VMs, then esxcli system maintenanceMode set --enable true.
esxcli software vib install -d /vmfs/volumes/<datastore>/<bundle>.zip (or -v for a single VIB).
esxcli system shutdown reboot --reason "NVIDIA vGPU Manager"
After boot, verify: esxcli software vib list | grep -i nvidia, vmkload_mod -l | grep nvidia, and nvidia-smi.
Set esxcli graphics host set --default-type SharedPassthru (restart the host graphics service or reboot for it to take effect), and enable MIG mode on the GPU if you plan to use MIG-backed profiles.
Exit maintenance mode, then in vCenter add a vGPU profile to each VM (Edit Settings → Add New Device → PCI Device → NVIDIA GRID vGPU / profile).
In each guest: install the matching vGPU guest driver, configure licensing, verify with nvidia-smi.

Clusters and upgrades

For many hosts, add the NVIDIA component to the vSphere Lifecycle Manager cluster image and remediate hosts one at a time. vLCM handles maintenance mode and VM migration for you.
For a manual upgrade: maintenance mode → esxcli software vib update -d ... (or remove the old VIB, then install the new one, if the release notes say so) → reboot → verify → exit maintenance mode → upgrade guest drivers.
After an ESXi upgrade, re-check that the NVIDIA component still matches the new ESXi version.

The host-side flow is always: maintenance mode → install or update → reboot → verify (VIB list, module loaded, nvidia-smi) → graphics type → exit maintenance mode. Then guests: matching driver → license → verify.

14. Inside the guest VM

Once the VM sees a GPU, everything inside works like bare metal: the same CUDA, containers, NGC images, and NVIDIA AI Enterprise software.

lspci | grep -i nvidia                 # the VM sees an NVIDIA device
nvidia-smi                             # vGPU: shows the profile name (for example H100-20C) and its memory
nvidia-smi -q | grep -i -A2 license    # vGPU only: licensed?
docker run --rm --gpus all nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi   # containers work as usual

Guest topic	Passthrough	vGPU
Driver	Standard NVIDIA data-center driver	vGPU guest driver from the matching release
License	No vGPU license	Required (token + `nvidia-gridd`)
`nvidia-smi` shows	The physical GPU (for example H100 80GB HBM3)	The vGPU profile and its memory slice
Kernel updates	Driver must be rebuilt for the new kernel (DKMS or reinstall)	Same

Some host-level operations aren't available from inside a vGPU guest. For example, MIG mode and many clock or power settings are controlled on the host, not in the VM.

15. Logs and troubleshooting tools

Where	What to look for
`/var/log/vmkernel.log` (ESXi host)	NVIDIA kernel module messages (often prefixed `NVRM`), device and passthrough errors
`/var/log/hostd.log` (ESXi host)	Host agent errors, such as VM power-on failures
`vmware.log` in the VM's folder on the datastore	Why one VM failed to power on (for example no vGPU capacity, MMIO, or reservation errors)
`nvidia-bug-report.sh` (host or guest)	NVIDIA's diagnostic bundle for support cases
`vm-support` (ESXi host)	VMware's diagnostic bundle
Guest: `dmesg`, `journalctl -u nvidia-gridd`	Guest driver errors and Xids; licensing problems

grep -i nvrm /var/log/vmkernel.log | tail -20        # host-side NVIDIA messages
grep -i -E "vgpu|nvidia|mmio" /vmfs/volumes/<datastore>/<vm>/vmware.log

16. KVM and other hypervisors

KVM-based platforms (Red Hat, Ubuntu, Nutanix AHV, Proxmox, OpenShift Virtualization) use the same ideas with Linux tools:

Concept	VMware ESXi	KVM / Linux host
Host CLI	`esxcli`, vCenter	`virsh` (libvirt), `lspci`, distro package manager
Enable IOMMU	BIOS	BIOS + kernel arguments, for example `intel_iommu=on iommu=pt`
Passthrough driver	Toggle passthrough in vCenter	Bind the GPU to `vfio-pci`; `virsh nodedev-detach`
vGPU host component	vGPU Manager VIB	vGPU Manager package for the Linux host
Create vGPUs	Pick a profile in vCenter	Mediated devices (`mdevctl`) on older setups; SR-IOV virtual functions on recent GPUs and releases
Enable GPU virtual functions (Ampere and later)	Handled by the host software	`/usr/lib/nvidia/sriov-manage -e <domain:bus:slot.fn>` (or `ALL`). VFs are lost after a host reboot or driver reload, so run it again or automate it. Then choose a profile per VF (`creatable_vgpu_types` → `current_vgpu_type`, or mdev on older frameworks).

BIOS prerequisites for Ampere-and-later vGPU: VT-d/IOMMU, SR-IOV, and ARI (Alternative Routing ID Interpretation). NVIDIA says to enable GPU VFs only with sriov-manage, not by writing sriov_numvfs yourself. A symptom of forgetting it on A100/A40/A10: nvidia-smi on the host shows 100% utilization on an idle GPU.

lspci -nnk -d 10de:                    # NVIDIA devices and which kernel driver owns them (nvidia vs vfio-pci)
dmesg | grep -i -e DMAR -e IOMMU       # IOMMU enabled?
virsh nodedev-list --cap pci           # PCI devices libvirt can assign

The exact vGPU creation method on KVM has changed across vGPU releases and kernels. Follow the release's KVM guide; the concepts (host manager, profiles, guest driver, license) stay the same.

17. Multi-GPU, NVLink/NVSwitch, and RDMA in VMs

Several GPUs in one VM

Passthrough several GPUs to one VM: works, but the VM needs enough 64-bit MMIO space for all of them.
vGPU: a VM can have more than one vGPU, generally only with full-memory profiles. Peer-to-peer between them has extra restrictions.

HGX systems with NVSwitch

On 8-GPU HGX/DGX systems, NVLink goes through NVSwitch chips, and Fabric Manager must program them (see Fabric Manager). In virtualized setups, NVIDIA documents several models for where Fabric Manager runs:

Model	GPUs and NVSwitches go to	Fabric Manager runs in
Full passthrough	All GPUs and NVSwitches to one VM	That VM
Shared NVSwitch multi-tenancy	GPU groups to several VMs; NVSwitches to a service VM	A dedicated service VM
vGPU multi-tenancy	vGPUs to VMs	The hypervisor host

Which models a given hypervisor and release supports varies. Check the Fabric Manager user guide and the vGPU release notes before designing an HGX VM layout.

RDMA, GPUDirect, and NCCL across VMs

For multi-node training between VMs, give each VM direct access to the network adapter: passthrough of the ConnectX NIC or an SR-IOV virtual function. Paravirtual networking adds latency and usually can't do GPUDirect RDMA.
Place the GPU and its NIC on the same NUMA node / PCIe root where possible, and give the VM matching vCPUs and memory (vNUMA).
A VM often hides the real PCIe topology, so NCCL can't see which NIC is close to which GPU. NCCL accepts a topology description file (NCCL_TOPO_FILE) for this case. See the NCCL deep dive.

Single-node GPU compute in a VM is close to bare metal. Scale-out training in VMs needs extra work: NIC passthrough or SR-IOV, NUMA alignment, and sometimes a topology file. That's a big reason large training clusters stay bare metal.

18. Nuances and limitations

Topic	What to know
vMotion / live migration	Not possible with passthrough. Supported for vGPU with requirements (compatible hosts, same profile available, release support). DRS can't freely move passthrough VMs.
Memory reservation	Passthrough (and vGPU) VMs typically need their memory fully reserved; this reduces host overcommit
Snapshots and suspend	Restricted for VMs with passthrough devices; vGPU supports more, depending on release
NUMA and CPU placement	Align the VM's vCPUs and memory with the GPU's NUMA node; misalignment costs bandwidth (same as CPU affinity on bare metal)
MIG changes	MIG mode and partitions for MIG-backed vGPU are configured on the host; changing them disrupts the VMs on that GPU
Monitoring	Host `nvidia-smi` / `nvidia-smi vgpu` see all vGPUs; DCGM and in-guest tools see only that VM's view
Firmware and BMC	Still managed out-of-band per physical server (see Redfish & BMC); the hypervisor doesn't replace the BMC
Kubernetes on VMs	The NVIDIA GPU Operator runs inside the Kubernetes node VMs; with vGPU it installs the vGPU guest driver and licensing. See Kubernetes and Run:ai.

19. Mode selector

Each VM needs
Isolation between tenants
Live migration (vMotion) required?
Scale

20. esxcli command builder

Task

21. "Which tool?" chooser

Pick a question above.

22. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

nvidia-smi -q | grep -i -A2 licenseGuest vGPU license status

    vGPU Software Licensed Product
        Product Name                      : NVIDIA Virtual Compute Server
        License Status                    : Licensed (Expiry: 2026-11-11 10:15:2 GMT)

What to look at: Inside a vGPU VM you want 'Licensed' with a future expiry and the product that matches the profile series (Virtual Compute Server for C profiles). 'Unlicensed (Restricted)' means the token or license server isn't reachable, and performance is throttled after the grace period.

journalctl -u nvidia-griddGuest licensing service log

Nov 10 10:14:58 ml-vm01 systemd[1]: Starting NVIDIA Grid Daemon...
Nov 10 10:14:58 ml-vm01 nvidia-gridd[1187]: Started (1187)
Nov 10 10:14:58 ml-vm01 systemd[1]: Started NVIDIA Grid Daemon.
Nov 10 10:14:59 ml-vm01 nvidia-gridd[1187]: Service provider detection complete.
Nov 10 10:14:59 ml-vm01 nvidia-gridd[1187]: Shutdown (1187)
Nov 10 10:15:00 ml-vm01 nvidia-gridd[1203]: Started (1203)
Nov 10 10:15:01 ml-vm01 nvidia-gridd[1203]: Acquiring license. (Info: 10.20.0.50; NVIDIA Virtual Compute Server)
Nov 10 10:15:02 ml-vm01 nvidia-gridd[1203]: License acquired successfully. (Info: 10.20.0.50, NVIDIA Virtual Compute Server; Expiry: 2026-11-11 10:15:2 GMT)

What to look at: Look for 'License acquired successfully' with the DLS or CLS address. 'Failed to acquire license' or errors about the client configuration token mean the .tok file is missing from /etc/nvidia/ClientConfigToken/ or the VM can't reach the license server.

esxcli system version getESXi version and build

   Product: VMware ESXi
   Version: 8.0.3
   Build: Releasebuild-24022510
   Update: 3
   Patch: 0

What to look at: Match Version/Update against the vGPU release's supported hypervisor list before installing the vGPU Manager bundle. Using the bundle for the wrong ESXi major version (7.0 versus 8.0) fails at install or the module won't load.

esxcli system maintenanceMode getIs the host in maintenance mode?

Enabled

What to look at: It must say Enabled before installing or removing the vGPU Manager VIB and rebooting. Disabled means VMs may still be running, and the install either fails or leaves the host needing a reboot under live workloads.

esxcli software vib list | grep -i nvdInstalled NVIDIA vGPU Manager VIBs

NVD-VMware_ESXi_8.0.0_Driver   550.127.06-1OEM.800.1.0.20613240   NVD      VMwareAccepted    2026-10-02    host
nvdgpumgmtdaemon               550.127.06-1OEM.700.1.0.15843807   NVD      VMwareAccepted    2026-10-02    host

What to look at: Both the host driver VIB and the GPU management daemon should show the same driver version. That version must match the guest driver's vGPU release (17.x: 550.127.06 host, 550.127.05 guest). Current VIBs use the vendor and name prefix NVD, so the guide's grep -i nvidia can return nothing even when the VIB is installed.

esxcli software vib install -d /vmfs/volumes/<datastore>/<bundle>.zipInstall the vGPU Manager offline bundle

Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   VIBs Installed: NVD_bootbank_NVD-VMware_ESXi_8.0.0_Driver_550.127.06-1OEM.800.1.0.20613240, NVD_bootbank_nvdgpumgmtdaemon_550.127.06-1OEM.700.1.0.15843807
   VIBs Removed:
   VIBs Skipped:
   Reboot Required: true
   DPU Results:

What to look at: 'Reboot Required: true' means the NVIDIA module isn't loaded until you reboot, so nvidia-smi fails until then. An error saying the VIB requires a different ESXi version, or about acceptance level, means you have the wrong bundle or the host acceptance level is too strict.

esxcli hardware pci list | grep -i -A3 nvidiaPhysical GPUs and their owner

   Vendor Name: NVIDIA Corporation
   Device Name: GH100 [H100 PCIe]
   Configured Owner: VMkernel
   Current Owner: VMkernel
--
   Vendor Name: NVIDIA Corporation
   Device Name: GH100 [H100 PCIe]
   Configured Owner: VMkernel
   Current Owner: VMkernel
--
   Vendor Name: NVIDIA Corporation
   Device Name: GH100 [H100 PCIe]
   Configured Owner: VMkernel
   Current Owner: VMkernel
--
   Vendor Name: NVIDIA Corporation
   Device Name: GH100 [H100 PCIe]
   Configured Owner: VMkernel
   Current Owner: VMkernel

What to look at: For vGPU the owner must be VMkernel (the vGPU Manager drives the GPU). 'VM Passthru' means the GPU is set up for DirectPath I/O passthrough and can't host vGPUs.

esxcli graphics device listGPUs as the graphics stack sees them

0000:17:00.0
   Vendor Name: NVIDIA Corporation
   Device Name: GH100 [H100 PCIe]
   Module Name: nvidia
   Graphics Type: Shared Direct
   Memory Size in KB: 83517440
   Number of VMs: 2
0000:65:00.0
   Vendor Name: NVIDIA Corporation
   Device Name: GH100 [H100 PCIe]
   Module Name: nvidia
   Graphics Type: Shared Direct
   Memory Size in KB: 83517440
   Number of VMs: 1
...

What to look at: Graphics Type must be Shared Direct for vGPU, and Number of VMs shows how many vGPU VMs each GPU hosts. 'Shared' (vSGA) means the default type is wrong, and 0 KB of memory points to a driver problem.

esxcli graphics host getDefault graphics type and placement policy

   Default Graphics Type: SharedPassthru
   Shared Passthru Assignment Policy: Performance

What to look at: SharedPassthru (Shared Direct) is required for vGPU. Performance spreads vGPUs across GPUs while Consolidation fills one GPU first; a value of Shared means the set command wasn't run, or the host graphics service wasn't restarted after it.

vmkload_mod -l | grep nvidiaNVIDIA module loaded on ESXi

nvidia                   3    49848

What to look at: A line for nvidia means the vGPU Manager kernel module is loaded. No output after install means the host hasn't been rebooted or the module failed to load, so check vmkernel.log for NVRM errors.

nvidia-smi # on the ESXi hostHost view of GPUs and vGPU VMs

Tue Nov 10 09:21:07 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.06          Driver Version: 550.127.06   CUDA Version: Not Supported |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 PCIe               On  |   00000000:17:00.0 Off |                    0 |
| N/A   41C    P0            118W /  350W |   41472MiB /  81559MiB |     37%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 PCIe               On  |   00000000:65:00.0 Off |                    0 |
| N/A   39C    P0             96W /  350W |   20736MiB /  81559MiB |     22%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
...   (GPUs 2-3 trimmed)

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   2102345   C+G    ml-vm01                                     20480MiB |
|    0   N/A  N/A   2102512   C+G    ml-vm02                                     20480MiB |
|    1   N/A  N/A   2103077   C+G    ml-vm03                                     20480MiB |
+-----------------------------------------------------------------------------------------+

What to look at: On the hypervisor the driver is the vGPU Manager (550.127.06) and CUDA Version reads Not Supported, which is normal. The process list shows VM names, one per vGPU; nvidia-smi failing here means the VIB isn't installed or the host wasn't rebooted.

nvidia-smi vgpuRunning vGPUs per physical GPU

Tue Nov 10 10:12:44 2026
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.06             Driver Version: 550.127.06                |
|---------------------------------+------------------------------+------------+
| GPU  Name                       | Bus-Id                       | GPU-Util   |
|      vGPU ID     Name           | VM ID     VM Name            | vGPU-Util  |
|=================================+==============================+============|
|   0  NVIDIA H100 PCIe           | 00000000:17:00.0             |  37%       |
|      3251634213  NVIDIA H100... | 2102345   ml-vm01            |     19%    |
|      3251634214  NVIDIA H100... | 2102512   ml-vm02            |     18%    |
+---------------------------------+------------------------------+------------+
|   1  NVIDIA H100 PCIe           | 00000000:65:00.0             |  22%       |
|      3251634215  NVIDIA H100... | 2103077   ml-vm03            |     22%    |
+---------------------------------+------------------------------+------------+
|   2  NVIDIA H100 PCIe           | 00000000:CA:00.0             |   0%       |
|      No vGPUs running                                                      |
+---------------------------------+------------------------------+------------+
...

What to look at: Each running vGPU is listed under its physical GPU with its VM name and utilization. A VM you expect that isn't listed never got its vGPU, so check vmware.log for that VM.

nvidia-smi vgpu -qDetailed per-vGPU info incl. license

GPU 00000000:17:00.0
    Active vGPUs                          : 2
    vGPU ID                               : 3251634213
        VM ID                             : 2102345
        VM Name                           : ml-vm01
        vGPU Name                         : NVIDIA H100-20C
        vGPU Type                         : 1053
        vGPU UUID                         : 4f2a9c1e-6b3d-11ef-8a7c-0c42a1b2c3d4
        Guest Driver Version              : 550.127.05
        License Status                    : Licensed (Expiry: 2026-11-11 10:15:2 GMT)
        GPU Instance ID                   : N/A
        Accounting Mode                   : Disabled
        ECC Mode                          : Enabled
        Accounting Buffer Size            : 4000
        Frame Rate Limit                  : N/A
        PCI
            Bus Id                        : 00000000:02:00.0
        FB Memory Usage
            Total                         : 20480 MiB
            Used                          : 6144 MiB
            Free                          : 14336 MiB
        Utilization
            Gpu                           : 19 %
            Memory                        : 12 %
    vGPU ID                               : 3251634214
        VM ID                             : 2102512
        VM Name                           : ml-vm02
...

What to look at: Guest Driver Version should be from the same vGPU release as the host, and License Status should be Licensed for each VM. GPU Instance ID N/A means the vGPU is time-sliced; a MIG-backed profile shows a real instance ID.

nvidia-smi vgpu -svGPU types this GPU supports

GPU 00000000:17:00.0
    NVIDIA H100-1-10C
    NVIDIA H100-1-20C
    NVIDIA H100-2-20C
    NVIDIA H100-3-40C
    NVIDIA H100-4-40C
    NVIDIA H100-7-80C
    NVIDIA H100-4C
    NVIDIA H100-5C
    NVIDIA H100-8C
    NVIDIA H100-10C
    NVIDIA H100-16C
    NVIDIA H100-20C
    NVIDIA H100-40C
    NVIDIA H100-80C
...

What to look at: This is everything the GPU model supports: the H100-<slices>-<GB>C names are MIG-backed and the H100-<GB>C names are time-sliced. It doesn't tell you what can be created right now; use -c for that.

nvidia-smi vgpu -cvGPU types creatable right now

GPU 00000000:17:00.0
    NVIDIA H100-20C
GPU 00000000:65:00.0
    NVIDIA H100-20C
GPU 00000000:CA:00.0
    NVIDIA H100-4C
    NVIDIA H100-5C
    NVIDIA H100-8C
    NVIDIA H100-10C
    NVIDIA H100-16C
    NVIDIA H100-20C
    NVIDIA H100-40C
    NVIDIA H100-80C
...

What to look at: A GPU that already hosts H100-20C vGPUs can only create more H100-20C because all time-sliced vGPUs on a GPU share one size, while an empty GPU offers every size. If a GPU lists nothing, it's full; with MIG mode on, only MIG-backed types appear.

esxcli vm process listRunning VMs and their World IDs

ml-vm01
   World ID: 2102345
   Process ID: 0
   VMX Cartel ID: 2102344
   UUID: 42 1a 7c 3e 9b 2d 4f 61-8a 0c 5e 7f 1b 3d 9a 2c
   Display Name: ml-vm01
   Config File: /vmfs/volumes/6530a1b2-5f7c9e10-3d4a-0c42a1b2c3d4/ml-vm01/ml-vm01.vmx

ml-vm02
   World ID: 2102512
   Process ID: 0
   VMX Cartel ID: 2102511
   UUID: 42 1a 9e 5b 3c 7d 1f 82-4b 6e 0a 2c 8d 5f 3e 7a
   Display Name: ml-vm02
   Config File: /vmfs/volumes/6530a1b2-5f7c9e10-3d4a-0c42a1b2c3d4/ml-vm02/ml-vm02.vmx
...

What to look at: The World ID is what esxcli vm process kill --world-id needs (try --type soft first). It also matches the VM ID column in nvidia-smi vgpu, so you can tie a vGPU back to its VM.

nvidia-smi # inside the vGPU VMGuest sees its vGPU profile

Tue Nov 10 09:21:07 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05             Driver Version: 550.127.05     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100-20C                On  |   00000000:02:00.0 Off |                    0 |
| N/A   N/A    P0              N/A /  N/A |       0MiB /  20480MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

What to look at: The GPU name is the profile (NVIDIA H100-20C) and total memory is the profile's slice (20480MiB), not the full 80 GB. Temperature and power show N/A in a vGPU guest by design, and the guest driver must come from the same vGPU release as the host.

grep -i nvrm /var/log/vmkernel.log | tail -20Host-side NVIDIA kernel messages

2026-11-10T09:58:31.204Z In(182) vmkernel: cpu8:2097652)NVRM: loading NVIDIA UNIX x86_64 Kernel Module  550.127.06  Fri Oct 25 01:14:27 UTC 2024
2026-11-10T09:58:33.917Z In(182) vmkernel: cpu8:2097652)NVRM: GPU at PCI:0000:17:00: GPU-9c1e4f2a-6b3d-4a7c-8e21-0c42a1b2c3d4
2026-11-10T09:58:33.918Z In(182) vmkernel: cpu8:2097652)NVRM: GPU Board Serial Number: 1650000000001
2026-11-10T09:58:34.402Z In(182) vmkernel: cpu8:2097652)NVRM: GPU at PCI:0000:65:00: GPU-2d7a8b3c-1e5f-4c9a-b6d0-0c42a1b2c3d5
...
2026-11-10T10:12:05.661Z In(182) vmkernel: cpu21:2102350)NVRM: vGPU: VM ml-vm01: guest driver loaded, version 550.127.05

What to look at: Healthy logs show the module loading with the vGPU Manager version and each GPU being found. Lines with 'Xid', 'GPU has fallen off the bus', or a guest driver version incompatibility message are the ones to act on.

lspci -nnk -d 10de:KVM: which driver owns each GPU

41:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 PCIe] [10de:2331] (rev a1)
        Subsystem: NVIDIA Corporation GH100 [H100 PCIe] [10de:1626]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
81:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 PCIe] [10de:2331] (rev a1)
        Subsystem: NVIDIA Corporation GH100 [H100 PCIe] [10de:1626]
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

What to look at: For passthrough the GPU must show 'Kernel driver in use: vfio-pci'. For KVM vGPU it is owned by the host nvidia (vGPU Manager) driver, and nouveau in use means the host driver isn't installed or nouveau wasn't blacklisted.

dmesg | grep -i -e DMAR -e IOMMUKVM: IOMMU enabled for passthrough

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-5.15.0-126-generic root=/dev/mapper/vg0-root ro intel_iommu=on iommu=pt
[    0.011873] ACPI: DMAR 0x000000006F3B2000 000250 (v01 INTEL  INTEL ID 00000001 INTL 20091013)
[    0.452101] DMAR: IOMMU enabled
[    1.204587] DMAR: Host address width 52
[    1.204593] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0
...
[    2.318842] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    2.981230] iommu: Default domain type: Passthrough (set via kernel command line)

What to look at: You need 'DMAR: IOMMU enabled' and the intel_iommu=on iommu=pt kernel arguments. No DMAR lines at all means VT-d is off in the BIOS, and vfio-pci passthrough won't work.

23. Scenarios

24. Searchable reference

Item	Meaning

25. Flashcards (click to flip)

26. Exam traps

esxcli manages the ESXi host, not the guest OS. Guest drivers are installed inside the VM with the guest's tools.
The vGPU Manager is a host VIB: maintenance mode → esxcli software vib install → reboot → verify.
Passthrough = no NVIDIA host software, no vGPU license, no vMotion, full memory reservation, EFI + 64-bit MMIO for big GPUs.
vGPU = host vGPU Manager + matching guest driver + license.
Profile number = memory (GB). Letter C = compute (NVIDIA AI Enterprise).
Time-sliced vGPU shares compute in turns; MIG-backed gives hardware isolation.
vGPU needs the host default graphics type Shared Direct (SharedPassthru).
Ampere and later vGPU needs SR-IOV in the BIOS; passthrough needs VT-d/IOMMU; big GPUs need above-4G decoding.
GPU visible but performance restricted in a vGPU VM → license.
vGPU VMs fail after an ESXi upgrade → the NVIDIA component doesn't match the new ESXi version.
Host nvidia-smi works only after the vGPU Manager is installed; nvidia-smi vgpu shows the running vGPUs.
HGX with NVSwitch in VMs still needs Fabric Manager, running where the chosen virtualization model puts it.
Multi-node training in VMs needs NIC passthrough or SR-IOV for RDMA; NCCL may need a topology file.
Large-scale training clusters are usually bare metal; VMs are for isolation, sharing, and enterprise operations.

27. Practice quiz (26 questions)

Score: 0 / 0