NVIDIA Run:ai

GPU orchestration on Kubernetes: the stack underneath, the control plane and cluster components, departments, projects and node pools, guaranteed quota vs over-quota, fairness, preemption and reclaim, gang scheduling, GPU fractions vs MIG, workloads and the runai CLI, installation and validation, and troubleshooting Pending and preempted work.

1. The big picture: what Run:ai adds to Kubernetes

NVIDIA Run:ai is a GPU orchestration and scheduling platform that runs on top of Kubernetes. NVIDIA acquired Run:ai in 2024, and its scheduler core has since been open-sourced as the KAI Scheduler. Plain Kubernetes schedules pods one at a time against static resource requests. That works for web services but is a poor fit for a shared pool of expensive GPUs used by many teams. Run:ai adds:

Guaranteed quotas per department and project, so each team is always entitled to its share.
Over-quota use of idle GPUs, so nothing sits unused while someone has work waiting.
Reclaim and preemption: when a team needs its guaranteed GPUs back, borrowed GPUs are taken back from preemptible work.
Fairness between teams competing for the idle pool.
Gang scheduling of distributed jobs: all workers start together or none do.
GPU fractions: several small workloads (notebooks, light inference) share one GPU.
Queueing, workload management (training, inference, interactive workspaces), dashboards, and access control.

A shared office building. Each team has a guaranteed number of desks (quota). Empty desks can be borrowed by any team that needs more (over-quota). When a desk's owning team shows up, the borrower has to move (reclaim / preemption). A team that books a whole meeting room for 16 people gets either all 16 seats or waits; half a room is useless (gang scheduling). Some desks are hot-desks shared by several part-time people (GPU fractions).

2. Where it sits for the NCP-AII exam

Run:ai is not named as its own objective in the NCP-AII blueprint, whose control-plane domain is about BCM, the OS, Slurm with Enroot and Pyxis, drivers, the Container Toolkit, Docker, and the NGC CLI. It still shows up as context in AI-factory scenarios: "teams share a Kubernetes GPU pool with guaranteed quotas", "borrow idle GPUs", "fractional GPUs for notebooks". It also depends on everything the blueprint does test: healthy GPUs, drivers, the container stack, and RDMA. See the blueprint tracker.

Recognize when a scenario points at Run:ai, know the stack it needs underneath, and validate bottom-up. Most Run:ai "failures" in practice are GPU Operator, driver, quota, or node pool problems.

3. The stack underneath

Layer	What it provides	If it's broken
GPU Operator	Driver (or uses the host's), container toolkit, device plugin advertising `nvidia.com/gpu`, GPU Feature Discovery labels, MIG manager, DCGM exporter	Nodes show 0 GPUs; Run:ai has nothing to schedule
Network Operator	Exposes InfiniBand / RoCE devices to pods (RDMA shared device plugin or SR-IOV), with the networking drivers	Distributed jobs run but NCCL falls back to TCP sockets: very slow
Training operators	Job types for distributed frameworks (PyTorch, MPI, TensorFlow, and so on)	Distributed workload types fail to create
Prometheus	Metrics that Run:ai dashboards and some scheduling features use	Missing dashboards and metrics
Ingress + certificates	Access to workspaces, inference endpoints, and (self-hosted) the control plane	Can't reach notebooks or endpoints
Storage	Storage classes / PVCs / NFS for datasets and checkpoints	Workloads Pending on volumes, or slow I/O

Base Command Manager can deploy Kubernetes on the nodes it provisions (alongside or instead of Slurm), and recent versions offer to set up the GPU Operator and Run:ai as part of that. Exact options depend on the BCM version.

4. Architecture: control plane and cluster

Part	Where it runs	Role
Control plane	NVIDIA-hosted (SaaS) or self-hosted on a Kubernetes cluster you run (including air-gapped)	UI, API, users and RBAC, organization (departments, projects, quotas), workload records, analytics
Cluster components	On each GPU Kubernetes cluster, installed with Helm (usually in the `runai` namespace)	The Run:ai scheduler, controllers, and agents that sync with the control plane and run workloads

One control plane can manage several clusters.
Each Run:ai project maps to a Kubernetes namespace on the cluster (commonly named runai-<project>).
Run:ai workloads are scheduled by the Run:ai scheduler, not the default kube-scheduler. Pods submitted outside Run:ai still use the default scheduler and are not governed by Run:ai quotas.

If someone deploys GPU pods with plain kubectl into a namespace without using the Run:ai scheduler, those GPUs are invisible to Run:ai's quota accounting and can starve projects. Submit through Run:ai, or set the pod's scheduler to Run:ai's.

5. Organization: departments, projects, node pools

Object	What it is	Example
Department	A group of projects (a business unit). Can have its own quota that caps or guarantees the sum of its projects.	"Research", "Product AI"
Project	The unit that owns quota and runs workloads; maps to a namespace. Users are given access to projects.	"team-vision", "team-llm"
Node pool	A group of nodes, selected by a node label, usually by GPU type or purpose. Quotas are set per project per node pool.	"h100-pool", "l40s-inference"

Quota is not just a number per team: it's GPUs per team per node pool. A project with 16 GPUs of H100 quota and 0 of L40S quota can't get guaranteed L40S GPUs. If a workload asks for a node pool its project has no quota in, it can only run over-quota (if allowed), or it waits.

6. Quota, over-quota, and fairness

Concept	Meaning
Quota (deserved / guaranteed)	GPUs the project is entitled to. If it has demand up to this level, it gets them, reclaiming from others if needed.
Over-quota	Using idle GPUs beyond the quota. Allowed when enabled for the project and GPUs are free. Over-quota work is preemptible.
Over-quota weight / priority	How idle GPUs are shared among several projects that all want more than their quota. Higher weight = larger share of the idle pool.
Limit (max)	An optional hard ceiling on how many GPUs a project can use, even if GPUs are idle.
Fairness	The scheduler continuously moves the allocation toward each project's fair share: quota first, then a weighted split of the rest.

Quota = what you're owed. Over-quota = what you borrow. Limit = what you can never exceed. Borrowed GPUs can be called back.

Setting quotas whose sum exceeds the GPUs in a node pool (over-subscription) is possible in some setups, but then "guaranteed" can't be honored for everyone at once. For guarantees to mean anything, the quotas in a pool should add up to no more than its GPUs.

7. Interactive quota and reclaim simulator

A node pool with three projects. Set each project's quota, how many GPUs of work it has waiting (demand), and its over-quota weight. The simulator gives each project its quota first (up to its demand), then shares the idle GPUs by weight. Over-quota GPUs are the ones that will be reclaimed if their owner's demand grows. This is a simplified model of the real scheduler.

GPUs in the node pool

Over-quota allowed

Project	Quota	Demand	Weight

8. Preemption, priority, and reclaim

Reclaim: a project below its quota submits work, and the GPUs it's owed are busy with other projects' over-quota work. The scheduler preempts that over-quota work to give the owner its quota.
Priority within a project: higher-priority workloads can preempt lower-priority ones in the same project.
Preemptible vs non-preemptible: by default, training workloads are preemptible and can run over-quota. Interactive workspaces and inference are typically non-preemptible and therefore run only within quota (exact defaults depend on the version and admin settings).
A preempted training workload goes back to Pending and restarts when resources are available. It loses any progress since its last checkpoint.

Over-quota is borrowed time. Anything that runs over-quota must tolerate being stopped: write checkpoints and make the job resumable. Interactive and serving work, which can't tolerate interruption, should stay within guaranteed quota.

"My training job keeps getting killed when another team starts work" is usually not a bug. It was running over-quota and was reclaimed by the GPUs' owner. The fix is more quota, checkpointing, or accepting the borrow.

9. Gang scheduling and distributed training

A distributed job (for example 4 workers × 8 GPUs) needs all its workers running before any can make progress: NCCL's first collective waits for every rank. If a plain scheduler starts 3 of 4 workers, those 24 GPUs sit idle, holding resources another job could use, and can deadlock with other partially started jobs.

Gang scheduling starts all pods of the job together or none of them. Run:ai does this for its distributed workload types (PyTorch, MPI, TensorFlow, and so on, through the corresponding training operators).

# Distributed PyTorch: 2 workers x 8 GPUs (CLI v2 style; check runai --help for your version)
runai training pytorch submit llm-ft -p team-llm -i nvcr.io/nvidia/pytorch:<tag> --workers 2 -g 8

# Older v1 CLI style
runai submit-dist pytorch llm-ft --workers=2 -g 8 -i nvcr.io/nvidia/pytorch:<tag>

For multi-node performance the pods also need the fabric: the Network Operator must expose RDMA devices, and the pods need the right resources and memory limits. Verify with NCCL_DEBUG=INFO that NCCL uses NET/IB and GPUDirect RDMA, not sockets. Run:ai schedules the gang; it doesn't make the fabric fast.

A distributed job that stays Pending while a smaller job runs may be waiting for enough GPUs to be free simultaneously. Partially started workers are not the expected behavior with gang scheduling.

10. Placement: bin-packing vs spreading

Strategy	What it does	Good for
Bin-pack (consolidate)	Fill nodes tightly before using new ones	Keeping whole nodes free for big multi-GPU / multi-node jobs; reducing fragmentation
Spread	Distribute workloads across nodes	Inference and resilience, avoiding hot spots

Placement can be set per node pool. Fragmentation is the classic problem: a pool has 8 free GPUs, but spread one-or-two per node across many nodes, so an 8-GPU job on one node can't start. Bin-packing and consolidation reduce it.

Topology matters within a node too: an 8-GPU job wants all 8 GPUs on the same NVSwitch baseboard, and a multi-node job wants nodes on the same leaf or rail group where possible. Use node pools and node labels to steer big training jobs to the right hardware.

11. GPU fractions vs MIG vs whole GPUs

	Whole GPU	MIG instance	Run:ai GPU fraction
How it's split	Not split	Hardware partitions with fixed profiles (for example 1g.10gb … 7g.80gb on H100)	Software: GPU memory is divided by request; compute is time-shared
Isolation	Full	Strong: dedicated SMs, memory, and cache slices; fault isolation	Memory is enforced per workload; compute and faults are shared
Sizes	1, 2, … GPUs	Only the GPU's supported profiles	Any portion or memory amount (for example 0.5 GPU, or 10 GB)
Changing the layout	N/A	Reconfigure MIG (GPU must be idle)	Dynamic, per workload
Best for	Training, large inference	Multi-tenant inference needing guaranteed, isolated performance	Notebooks, development, light inference, bursty workloads

runai workspace submit nb1 -p team-vision -i <notebook-image> --gpu-portion-request 0.5    # half a GPU (v2 style)
runai workspace submit nb2 -p team-vision -i <notebook-image> --gpu-memory-request 10G     # by memory (v2 style)
runai submit nb1 -i <image> -g 0.5 --interactive                                         # v1 style fraction

Fractions share compute: two half-GPU notebooks on one GPU each get the whole GPU's compute when the other is idle, and compete when both are busy. MIG gives each instance a guaranteed, isolated slice. "Strict isolation and predictable performance per tenant" points to MIG. "Pack many small notebooks onto few GPUs" points to fractions.

Run:ai also offers dynamic fractions (a request plus a higher limit) and GPU memory swap (spilling idle workloads' GPU memory to CPU memory so more fit). These are optimization features; know the concepts, not the flags.

12. MIG on Kubernetes: the GPU Operator's MIG Manager

On Kubernetes, you don't run nvidia-smi mig by hand. The GPU Operator's MIG Manager (the nvidia-mig-manager DaemonSet) does it for you. You put a node label on the node naming the layout you want, and MIG Manager turns MIG mode on and creates the instances using mig-parted. The device plugin then advertises the slices to the scheduler, and Run:ai or plain Kubernetes places pods on them.

Piece	What it does	Where you set or read it
MIG strategy	How the device plugin and GPU Feature Discovery present MIG devices. It doesn't turn MIG on.	ClusterPolicy `spec.mig.strategy` (Helm `--set mig.strategy=single`). It's cluster-wide.
MIG Manager	Watches the node label; stops GPU pods, enables MIG mode, creates the instances, restarts the GPU pods	Pods `nvidia-mig-manager-*` in the `gpu-operator` namespace; read their logs
Layout request	Names one layout from the config	Node label `nvidia.com/mig.config`, for example `all-1g.10gb`, `all-balanced`, `all-disabled` (the default)
Progress	Where MIG Manager is up to	Node label `nvidia.com/mig.config.state`: `pending` → (`rebooting`) → `success` or `failed`
Named layouts	Which profiles go on which GPUs	Generated from the hardware at startup (GPU Operator v26.3.0 and newer). Older drivers such as the 535 branch use the static `default-mig-parted-config` ConfigMap. You can supply your own ConfigMap: its data key must be `config.yaml`, and you point the ClusterPolicy at it with `spec.migManager.config.name`.

single vs mixed

Strategy	What pods request	Rules
`single`	`nvidia.com/gpu: 1` gets one MIG device. Pick the size with a node selector on `nvidia.com/gpu.product`, for example `NVIDIA-H100-80GB-HBM3-MIG-1g.10gb`.	Every GPU on the node has MIG on with one profile, or every GPU has MIG off. An H100 node at `all-1g.10gb` shows `nvidia.com/gpu: 56`.
`mixed`	One resource per profile: `nvidia.com/mig-1g.10gb`, `nvidia.com/mig-3g.40gb`…; GPUs with MIG off stay `nvidia.com/gpu`	Needed when a node mixes profiles or leaves MIG off on some GPUs

kubectl get clusterpolicies.nvidia.com/cluster-policy -o jsonpath='{.spec.mig.strategy}'
kubectl cordon gpu-node2 && kubectl drain gpu-node2 --ignore-daemonsets     # no user workloads on the GPUs
kubectl label nodes gpu-node2 nvidia.com/mig.config=all-1g.10gb --overwrite
kubectl get node gpu-node2 -L nvidia.com/mig.config,nvidia.com/mig.config.state   # pending → success
kubectl logs -n gpu-operator nvidia-mig-manager-<pod>                            # "MIG configuration applied successfully"
kubectl describe node gpu-node2 | grep -A8 Allocatable                            # nvidia.com/gpu: 56
kubectl exec -n gpu-operator nvidia-driver-daemonset-<pod> -- nvidia-smi -L        # MIG 1g.10gb Device 0..6 per GPU
kubectl uncordon gpu-node2
kubectl label nodes gpu-node2 nvidia.com/mig.config=all-disabled --overwrite     # later: back to whole GPUs

The label is the interface. Changing nvidia.com/mig.config is what triggers a reconfiguration. Kubernetes refuses to change a label that already exists (and every MIG-capable node starts at all-disabled) unless you add --overwrite. Re-applying the same value changes nothing, so there's nothing for MIG Manager to react to. To retry a failed apply, restart the MIG Manager pod on that node: it applies the current label when it starts.

MIG Manager needs idle GPUs. It stops the operator's own GPU pods (device plugin, GPU Feature Discovery, DCGM exporter), but not your workloads. A pod still running on the GPUs makes the apply fail. Cordon and drain first. With drivers preinstalled on the host, it also stops the host services listed in the default-gpu-clients ConfigMap (NVSM, DCGM…). In some clouds it has to reboot the node (WITH_REBOOT=true; the state shows rebooting).

The strategy is cluster-wide. Switching from single to mixed renames the resources on every MIG node: a pod that asked for nvidia.com/gpu: 1 on a 1g.10gb node now has to ask for nvidia.com/mig-1g.10gb: 1. Plan the change with the teams that use MIG.

Symptom	Likely cause	Fix
`error: 'nvidia.com/mig.config' already has a value`	Missing `--overwrite`	Add `--overwrite`
State `failed`; log says the GPU is in use	A workload was still on the GPUs	Drain the node, then restart its MIG Manager pod
State `failed` right after you switch to a custom config	The ConfigMap key isn't `config.yaml`, the layout name isn't in it, or `migManager.config.name` points elsewhere	Fix the ConfigMap and the ClusterPolicy, then restart MIG Manager on the node
Device plugin crash-loops after a mixed layout	`single` strategy with more than one MIG profile on the node	Patch the strategy to `mixed`
Pod Pending: `Insufficient nvidia.com/gpu` on a mixed node	Under `mixed`, MIG slices are `nvidia.com/mig-<profile>`	Request the profile resource. You have to delete and recreate the pod: Kubernetes doesn't let you change a pod's resources in place.

Use one MIG manager per node. On a BCM-managed cluster, let either BCM or the GPU Operator own MIG, not both (see MIG through BCM). For the nvidia-smi mig commands underneath, see the nvidia-smi guide. Source: NVIDIA GPU Operator docs, "GPU Operator with MIG".

13. Workload types and assets

Workload	Use	Typical behavior
Workspace (interactive)	Jupyter, VS Code, a shell for development	Runs until stopped; usually non-preemptible, within quota; often uses fractions
Training (standard)	Single-pod training or batch work	Runs to completion; preemptible; can use over-quota
Training (distributed)	Multi-pod PyTorch / MPI / TF jobs	Gang-scheduled; preemptible
Inference	Model serving (for example NIM or Triton containers) with endpoints and autoscaling	Highest priority; non-preemptible; scales replicas

Assets are reusable building blocks admins and users define once and attach to workloads: environments (container image, tools, command), compute resources (GPU count, fraction, CPU, memory), data sources (PVC, NFS, Git, S3), credentials (for example a registry pull secret for nvcr.io, using a placeholder like $NGC_API_KEY), and templates.

Image pull failures from nvcr.io look like scheduling problems in a UI but are not: the pod was scheduled and then couldn't pull. Check the events for ErrImagePull / ImagePullBackOff and the registry credential (see the NGC guide).

14. The runai CLI

The CLI changed shape between versions. Newer versions (v2) use noun-verb commands by workload type; older versions (v1) used runai submit, runai list, and so on. Know both shapes; check runai --help on your cluster for exact flags.

# v2 style
runai login                                    # authenticate (browser or token)
runai cluster list ; runai cluster set <name>
runai project list ; runai project set team-llm
runai training submit t1 -i nvcr.io/nvidia/pytorch:<tag> -g 1
runai training list ; runai training describe t1 ; runai training logs t1
runai training delete t1
runai workspace submit nb1 -i <image> --gpu-portion-request 0.5
runai training pytorch submit d1 -i <image> --workers 2 -g 8
runai workload list                             # all workload types

# v1 style (older clusters)
runai config project team-llm
runai submit t1 -i <image> -g 1
runai submit nb1 -i <image> -g 0.5 --interactive
runai submit-dist pytorch d1 --workers=2 -g 8 -i <image>
runai list jobs ; runai describe job t1 ; runai logs t1 ; runai delete job t1

Underneath, it's all Kubernetes, so kubectl works for investigation:

kubectl get pods -n runai-team-llm -o wide        # the project's pods and their nodes
kubectl describe pod <pod> -n runai-team-llm      # Events: scheduler reasons, image pulls, volumes
kubectl get pods -n runai                          # Run:ai cluster components healthy?
kubectl get nodes --show-labels | grep -i pool     # node pool labels (label name set by the admin)

15. Access control

Users sign in through the control plane, typically with SSO (SAML or OIDC) against the company identity provider.
Roles (for example system administrator, department administrator, research manager, researcher/ML engineer, viewer) are granted on a scope: the whole system, a cluster, a department, or a project.
A researcher can submit workloads only in projects they have access to.
Applications (service accounts with client credentials) are used for automation through the API.

Role names and the exact permission sets vary between versions. The concept to know: role + scope, granted per user or group.

16. Installation and prerequisites

Kubernetes (or OpenShift) cluster on the GPU nodes, at a version the Run:ai release supports.
NVIDIA GPU Operator installed and healthy; every GPU node advertises nvidia.com/gpu.
Network Operator if you run multi-node training over InfiniBand or RoCE.
Prometheus (commonly kube-prometheus-stack), an ingress controller, a DNS name and TLS certificate, and storage classes. Inference features may need extra components (for example Knative); distributed training needs the training operators.
Control plane: use the hosted one, or install the self-hosted control plane with Helm.
Create the cluster in the control plane UI; it generates the Helm command for the cluster components, then install it on the GPU cluster:

helm repo add runai <runai-helm-repo-url> ; helm repo update
helm upgrade -i runai-cluster runai/runai-cluster -n runai --create-namespace \
  --set controlPlane.url=runai.example.com \
  --set cluster.url=gpu-cluster.example.com \
  --set controlPlane.clientSecret=$RUNAI_CLIENT_SECRET \
  --set cluster.uid=$RUNAI_CLUSTER_UID          # values come from the UI-generated command; placeholders here

Treat the cluster's client secret like any other credential: keep it out of scripts and repositories, and pass it from a secret store.

Create node pools (by node label), departments and projects, set quotas per node pool, and grant users access.

17. Validation checklist

Hardware and node layer: per-node DCGM diagnostics, NVLink/NCCL single-node tests, and the fabric tests from the bring-up guide. Run:ai doesn't change these.
Kubernetes GPU layer: kubectl get pods -n gpu-operator all healthy; kubectl describe node shows the right nvidia.com/gpu count; a smoke-test pod runs nvidia-smi.
RDMA in pods (multi-node): a two-pod NCCL test shows NET/IB and the expected bus bandwidth.
Run:ai components: kubectl get pods -n runai healthy; the cluster shows Connected in the UI; nodes, node pools, and GPU counts are correct.
Functional tests:
- A 1-GPU training workload in a test project runs and completes.
- A fractional workspace (for example 0.5 GPU) starts, and two of them share one GPU.
- A distributed workload starts all workers together (gang) and uses RDMA.
- Over-quota: project A fills idle GPUs over its quota; project B submits within its quota and A's over-quota work is preempted (reclaim works).
- RBAC: a researcher can submit only to their projects.
- Dashboards show GPU allocation and utilization.

Every layer below Run:ai passes its own tests; the cluster is Connected with the correct GPU inventory; quotas, over-quota, reclaim, fractions, and gang scheduling behave as designed; distributed workloads get RDMA; access is limited to the right projects.

18. Monitoring and metrics

Run:ai dashboards show allocation (GPUs assigned to workloads) per cluster, node pool, department, and project, plus utilization (how busy those GPUs actually are), pending workloads, and history.
GPU utilization and memory metrics come from DCGM through the GPU Operator's DCGM exporter and Prometheus.
The useful operational signal is the gap between allocation and utilization: GPUs allocated but idle (for example notebooks left open) are capacity wasted. Policies such as idle timeouts for workspaces address this.

Allocated is not the same as utilized. A dashboard showing 100% of GPUs allocated and 15% utilization means GPUs are reserved but idle. That's a policy or workload problem, not a hardware one.

19. Run:ai vs Slurm vs plain Kubernetes

	Slurm	Plain Kubernetes	Run:ai on Kubernetes
Work unit	Batch jobs (`sbatch`, `srun`)	Pods, Jobs, Deployments	Workloads (workspace, training, distributed, inference) in projects
Containers	Optional (Enroot/Pyxis)	Always	Always
Sharing between teams	Partitions, QOS, fair-share	Static namespace ResourceQuotas	Guaranteed quota + over-quota + reclaim + weights
Multi-node jobs	Native	Needs operators; no gang by default	Gang-scheduled distributed workloads
Fractional GPUs	MIG via GRES; sharding options	MIG or time-slicing via device plugin	Fractions by portion or memory, plus MIG
Inference serving	Not its focus	Yes	Yes, with autoscaling

Slurm = HPC queue. Kubernetes = container platform. Run:ai = a GPU-aware brain on top of Kubernetes that knows about teams, borrowing, and gangs.

20. Troubleshooting: why is it Pending?

Start with the workload's details (runai training describe / runai describe job, or the UI event log) and the pod events (kubectl describe pod). The scheduler usually states its reason.

Reason	Clue	Fix
Not enough free GPUs, and the project is at or above quota	Unschedulable; project's allocated GPUs ≥ its quota	Wait, allow over-quota, raise quota, or reduce the request
Work is non-preemptible, so it can't go over quota	Workspace or inference Pending while GPUs are idle	Give the project enough quota, or use a preemptible training workload
Wrong or empty node pool	Requested pool has no free GPUs or no quota for this project	Choose the right pool, or add quota in that pool
Request bigger than any node	Asking for 16 GPUs in one pod on 8-GPU nodes	Use a distributed workload (2 × 8) instead
Fragmentation	Enough GPUs in total, but not on one node	Bin-pack placement; wait for consolidation; drain small work
Gang can't fit	A distributed job waits for N × GPUs simultaneously	Free capacity, reduce workers, or schedule in a quieter window
Nodes advertise 0 GPUs	`nvidia.com/gpu` Allocatable 0; GPU Operator pods failing	Fix the GPU Operator or driver, or the GPU itself (DCGM)
Volume can't mount	PVC Pending, mount errors in events	Fix the storage class / PVC / NFS export
Not actually a scheduling problem	`ImagePullBackOff`, `CrashLoopBackOff`	Registry credentials, image tag, or the application itself

21. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

runai training pytorch submit llm-ft -p team-llm -i nvcr.io/nvidia/pytorch:<tag> --workers 2 -g 8Submit distributed PyTorch (CLI v2)

Creating distributed training workload llm-ft...
To track the workload's status, run 'runai training pytorch list'

What to look at: The CLI only confirms the request was accepted; the gang-scheduled pods (master plus workers) may still be Pending. Check the workload list or describe output next, and an 'unknown command' error means this cluster's CLI is v1-style (runai submit-dist pytorch).

runai workspace submit nb1 -p team-vision -i <notebook-image> --gpu-portion-request 0.5Half-GPU interactive workspace (v2)

Creating workspace nb1...
To track the workload's status, run 'runai workspace list'

What to look at: Once running, the workload list shows GPU Alloc. 0.50 for nb1. If it stays Pending, the project's quota or the node pool has no GPU that can still fit a 0.5 portion.

runai submit nb1 -i <image> -g 0.5 --interactiveSame fractional job, v1 CLI

The job 'nb1' has been submitted successfully
You can run `runai describe job nb1 -p team-vision` to check the job status

What to look at: This is the v1 confirmation wording; the project comes from runai config project. On clusters 2.18 and newer, v1 is unsupported, and the v2 equivalent is runai workspace submit with --gpu-portion-request.

kubectl get clusterpolicies.nvidia.com/cluster-policy -o jsonpath='{.spec.mig.strategy}'GPU Operator MIG strategy

single

What to look at: single means every MIG slice on a node is advertised as plain nvidia.com/gpu, so the node must use one profile type. With mixed, each profile becomes its own resource such as nvidia.com/mig-1g.10gb, and a mixed layout under single makes the device plugin crash.

kubectl cordon gpu-node2 && kubectl drain gpu-node2 --ignore-daemonsetsFree the node's GPUs safely

node/gpu-node2 cordoned
node/gpu-node2 already cordoned
Warning: ignoring DaemonSet-managed Pods: gpu-operator/gpu-feature-discovery-m4p9d, gpu-operator/nvidia-dcgm-exporter-m4p9d, gpu-operator/nvidia-device-plugin-daemonset-m4p9d, gpu-operator/nvidia-driver-daemonset-m4p9d, gpu-operator/nvidia-mig-manager-m4p9d
evicting pod default/train-llm-7-q2r8c
pod/train-llm-7-q2r8c evicted
node/gpu-node2 drained

What to look at: The operator DaemonSet pods stay (that's what --ignore-daemonsets means) and every user pod should be evicted before 'drained'. Without the flag, drain stops with 'cannot delete DaemonSet-managed Pods', and if a GPU job is left running MIG Manager's apply fails with the GPU in use.

kubectl label nodes gpu-node2 nvidia.com/mig.config=all-1g.10gb --overwriteRequest a MIG layout from MIG Manager

node/gpu-node2 labeled

What to look at: 'labeled' means the value changed and MIG Manager will react. 'error: ... already has a value (all-disabled), and --overwrite is false' means you forgot --overwrite, and 'not labeled' means the value was already set, so nothing is triggered.

kubectl get node gpu-node2 -L nvidia.com/mig.config,nvidia.com/mig.config.stateWatch the MIG apply state

NAME        STATUS                     ROLES    AGE   VERSION   MIG.CONFIG     MIG.CONFIG.STATE
gpu-node2   Ready,SchedulingDisabled   <none>   31d   v1.31.4   all-1g.10gb    pending

$ kubectl get node gpu-node2 -L nvidia.com/mig.config,nvidia.com/mig.config.state
NAME        STATUS                     ROLES    AGE   VERSION   MIG.CONFIG     MIG.CONFIG.STATE
gpu-node2   Ready,SchedulingDisabled   <none>   31d   v1.31.4   all-1g.10gb    success

What to look at: The state goes pending and then success once the layout is applied. failed means you need the MIG Manager log, and SchedulingDisabled is expected until you uncordon.

kubectl logs -n gpu-operator nvidia-mig-manager-m4p9dMIG Manager reconfiguration log

Waiting for change to 'nvidia.com/mig.config' label
Updating to MIG config: all-1g.10gb
Getting current value of the 'nvidia.com/gpu.deploy.device-plugin' node label
Current value of 'nvidia.com/gpu.deploy.device-plugin=true'
Asserting that the requested configuration is present in the configuration file
Selected MIG configuration is valid
Getting current value of the 'nvidia.com/mig.config.state' node label
Current value of 'nvidia.com/mig.config.state=success'
Checking if the selected MIG config is currently applied or not
Persisting all-1g.10gb to /etc/systemd/system/nvidia-mig-manager.service.d/override.conf
Changing the 'nvidia.com/mig.config.state' node label to 'pending'
Shutting down all GPU clients in Kubernetes by disabling their component-specific nodeSelector labels
Waiting for the device-plugin to shutdown
Waiting for gpu-feature-discovery to shutdown
Waiting for dcgm-exporter to shutdown
Applying the selected MIG config to the node
time="2026-11-10T09:41:22Z" level=info msg="Parsing config file..."
time="2026-11-10T09:41:22Z" level=info msg="Selecting specific MIG config..."
time="2026-11-10T09:41:22Z" level=info msg="Running apply-start hook"
time="2026-11-10T09:41:24Z" level=info msg="Applying MIG mode change..."
time="2026-11-10T09:41:31Z" level=info msg="Applying MIG device configuration..."
time="2026-11-10T09:41:38Z" level=info msg="Running apply-exit hook"
MIG configuration applied successfully
Restarting validator pod to re-run all validations
Restarting all GPU clients previously shutdown in Kubernetes by reenabling their component-specific nodeSelector labels
Changing the 'nvidia.com/mig.config.state' node label to 'success'

What to look at: The key lines are 'MIG configuration applied successfully' and the final state change to success. On failure, the line before 'failed' names the cause, such as a profile not in the ConfigMap or 'In use by another client' when a pod still holds a GPU.

kubectl describe node gpu-node2 | grep -A8 AllocatableGPU resources the node now advertises

Allocatable:
  cpu:                224
  ephemeral-storage:  1699505060Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             2113302Mi
  nvidia.com/gpu:     56
  pods:               110
System Info:

What to look at: With the single strategy, 8 H100s x 7 slices of 1g.10gb gives nvidia.com/gpu: 56. Still seeing 8 means the layout hasn't applied yet, and 0 or a missing line means the device plugin is down.

kubectl exec -n gpu-operator nvidia-driver-daemonset-m4p9d -- nvidia-smi -LMIG devices as the driver sees them

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-3e8a51c0-4f2b-4d7e-a1c9-0b5d6e7f8a00)
  MIG 1g.10gb     Device  0: (UUID: MIG-7d2c0e41-91a3-5b6c-8d0e-2f4a6b8c0d10)
  MIG 1g.10gb     Device  1: (UUID: MIG-7d2c1e41-91a3-5b6c-8d0e-2f4a6b8c0d11)
  MIG 1g.10gb     Device  2: (UUID: MIG-7d2c2e41-91a3-5b6c-8d0e-2f4a6b8c0d12)
  MIG 1g.10gb     Device  3: (UUID: MIG-7d2c3e41-91a3-5b6c-8d0e-2f4a6b8c0d13)
  MIG 1g.10gb     Device  4: (UUID: MIG-7d2c4e41-91a3-5b6c-8d0e-2f4a6b8c0d14)
  MIG 1g.10gb     Device  5: (UUID: MIG-7d2c5e41-91a3-5b6c-8d0e-2f4a6b8c0d15)
  MIG 1g.10gb     Device  6: (UUID: MIG-7d2c6e41-91a3-5b6c-8d0e-2f4a6b8c0d16)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-3e8a51c1-4f2b-4d7e-a1c9-0b5d6e7f8a01)
  MIG 1g.10gb     Device  0: (UUID: MIG-a19f0b72-03c4-5e8d-9f1a-3b5c7d9e1f20)
  MIG 1g.10gb     Device  1: (UUID: MIG-a19f1b72-03c4-5e8d-9f1a-3b5c7d9e1f21)
  MIG 1g.10gb     Device  2: (UUID: MIG-a19f2b72-03c4-5e8d-9f1a-3b5c7d9e1f22)
  MIG 1g.10gb     Device  3: (UUID: MIG-a19f3b72-03c4-5e8d-9f1a-3b5c7d9e1f23)
  MIG 1g.10gb     Device  4: (UUID: MIG-a19f4b72-03c4-5e8d-9f1a-3b5c7d9e1f24)
  MIG 1g.10gb     Device  5: (UUID: MIG-a19f5b72-03c4-5e8d-9f1a-3b5c7d9e1f25)
  MIG 1g.10gb     Device  6: (UUID: MIG-a19f6b72-03c4-5e8d-9f1a-3b5c7d9e1f26)
...   (GPUs 2-7: seven 1g.10gb devices each)

What to look at: Each GPU should list seven 'MIG 1g.10gb Device 0..6' lines. GPUs with no MIG lines are still whole GPUs, which means the apply skipped them or failed.

runai loginAuthenticate the Run:ai CLI

Opening the login page in your browser: https://runai.example.com/auth/realms/runai/...
Waiting for authentication to complete...
Logged in successfully

What to look at: Once you see the success message, the CLI caches a token for the control plane URL in its config. A browser that never opens on a headless host, or a login timeout, means you should use the terminal credential login or check the control plane URL.

runai project list ; runai project set team-llmList projects, set the default

Project       Department    GPU Quota   Allocated GPUs   Cluster
────────────────────────────────────────────────────────────────────────────────────
team-llm      research      32          16               dgx-cluster
team-vision   research      8           2.5              dgx-cluster

Project team-llm configured successfully

What to look at: Quota is the project's guaranteed GPUs; allocated can go above it when over-quota is allowed, and those extra workloads can be preempted. If your project is missing, you have no access to it and every submit will fail.

runai training submit t1 -i nvcr.io/nvidia/pytorch:<tag> -g 1Submit a single-GPU training workload

Creating training t1...
To track the workload's status, run 'runai training list'

What to look at: Submission success doesn't mean it is scheduled; check the list for Running versus Pending. An error about a missing project means no default project is set, so pass -p or run runai project set.

runai training listStatus of training workloads

Workload  Type      Status   Project   Running/Req.Pods  GPU Alloc.
────────────────────────────────────────────────────────────────────────────────────
llm-ft    Training  Running  team-llm  2/2               16.00
t1        Training  Pending  team-llm  0/1               0.00

What to look at: Running/Req.Pods shows how many pods actually started versus requested; 0/1 with Pending means the scheduler couldn't place it. GPU Alloc. shows the GPUs really allocated, which for a fractional workload is a value like 0.50.

runai training describe t1Why a workload is Pending

Workload Summary
  Name:             t1
  Namespace:        runai-team-llm
  Type:             Training
  Project:          team-llm
  Status:           Pending
  Created:          2026-11-10T09:58:02Z

Compute Resources
  GPU devices:      1
  Node pools:       default

Pods
  Name      Status    Node     Image
  t1-0-0    Pending   <none>   nvcr.io/nvidia/pytorch:<tag>

Events
  Time        Type      Reason              Message
  09:58:03    Warning   Unschedulable       Unable to schedule pod: NonPreemptibleOverQuota: project team-llm would exceed its quota of 32 GPUs
  09:58:03    Normal    PodGroupCreated     Pod group pg-t1-0-6f1c2b created

What to look at: Read the Events section first; the scheduler states its reason in plain words. Quota or over-quota messages mean you should wait or free GPUs in the project, while a node-pool or 'no nodes with enough GPUs' message means the request can't fit anywhere.

runai list jobsv1 job list

Showing jobs for project team-llm
NAME    STATUS   AGE  NODE   IMAGE                             TYPE   PROJECT   USER   GPUs Allocated (Requested)  PODs Running (Pending)  SERVICE URL(S)
llm-ft  Running  42m  dgx03  nvcr.io/nvidia/pytorch:24.05-py3  Train  team-llm  alice  16 (16)                     2 (0)
t1      Pending  1m          nvcr.io/nvidia/pytorch:24.05-py3  Train  team-llm  alice  0 (1)                       0 (1)

What to look at: Compare allocated with requested in the GPUs column; 0 (1) and a Pending pod means the job is waiting on the scheduler. The header line tells you which project is being shown, so a missing job may just be in another project.

kubectl get pods -n runai-team-llm -o wideProject pods and where they run

NAME                 READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
llm-ft-master-0      1/1     Running   0          42m   10.244.3.17    dgx03    <none>           <none>
llm-ft-worker-0      1/1     Running   0          42m   10.244.4.22    dgx04    <none>           <none>
nb1-0-0              1/1     Running   0          18m   10.244.1.9     dgx01    <none>           <none>
t1-0-0               0/1     Pending   0          1m    <none>         <none>   <none>           <none>

What to look at: Each Run:ai project maps to the runai-<project> namespace. A Pending pod with NODE <none> was never scheduled, so describe it; ImagePullBackOff or CrashLoopBackOff point to the image or the container instead.

kubectl describe pod t1-0-0 -n runai-team-llmPod events from the Run:ai scheduler

Name:             t1-0-0
Namespace:        runai-team-llm
Priority:         0
Service Account:  default
Node:             <none>
Labels:           project=team-llm
                  runai/queue=team-llm
                  workloadKind=TrainingWorkload
Status:           Pending
...
    Limits:
      nvidia.com/gpu:  1
...
Events:
  Type     Reason            Age   From              Message
  ----     ------            ----  ----              -------
  Warning  Unschedulable     58s   runai-scheduler   Unable to schedule pod: NonPreemptibleOverQuota: project team-llm would exceed its quota of 32 GPUs

What to look at: Pods in Run:ai projects are scheduled by runai-scheduler, not default-scheduler, so its messages are in the From column. 'Insufficient nvidia.com/gpu' means no node has free GPUs of that resource type, and image or volume errors appear here too.

kubectl get pods -n runaiRun:ai cluster components healthy

NAME                                         READY   STATUS    RESTARTS   AGE
accessrule-controller-6b9f7c8d5-x2k4p        1/1     Running   0          31d
cluster-api-7d5c9b6f4-q8w2n                  1/1     Running   0          31d
cluster-sync-5f8d7c6b9-m3j7r                 1/1     Running   0          31d
metrics-exporter-6c7b8d9f5-t4v6z             1/1     Running   0          31d
nodepool-controller-8d9c7b6f5-k2l8s          1/1     Running   0          31d
pod-grouper-5b6c7d8f9-p9q3w                  1/1     Running   0          31d
prometheus-runai-0                           2/2     Running   0          31d
queue-controller-7c8d9f6b5-h5n2x             1/1     Running   0          31d
researcher-service-6f7c8d9b5-r7t4y           1/1     Running   0          31d
runai-admission-controller-5d6f7c8b9-w3e6u   1/1     Running   0          31d
runai-agent-8b9c7d6f5-z1x9c                  1/1     Running   0          31d
runai-operator-6d7c8b9f5-a4s8d               1/1     Running   0          31d
runai-scheduler-default-7f8d9c6b5-f6g2h      1/1     Running   0          31d
status-updater-5c6d7f8b9-j8k1l               1/1     Running   0          31d
workload-controller-6b7c8d9f5-n5m3b          1/1     Running   0          31d
...

What to look at: Everything should be Running and fully Ready. A crashing runai-scheduler means nothing gets scheduled, and a failing runai-agent or cluster-sync shows the cluster as disconnected in the control plane UI.

helm upgrade -i runai-cluster runai/runai-cluster -n runai --create-namespace --set controlPlane.url=runai.example.com --set cluster.url=gpu-cluster.example.com --set controlPlane.clientSecret=$RUNAI_CLIENT_SECRET --set cluster.uid=$RUNAI_CLUSTER_UID

Install or upgrade Run:ai cluster components

Release "runai-cluster" does not exist. Installing it now.
NAME: runai-cluster
LAST DEPLOYED: Tue Nov 10 08:15:43 2026
NAMESPACE: runai
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
...

What to look at: upgrade -i installs on the first run and upgrades afterwards; REVISION increments on each upgrade. STATUS deployed only means Helm applied the objects, so confirm the pods in the runai namespace and that the cluster shows Connected in the UI.

22. Hands-on lab: simulated tickets

Two tickets on a simulated Kubernetes cluster with the NVIDIA GPU Operator and MIG Manager: three DGX H100 nodes (gpu-node1–gpu-node3, 8 GPUs each), driven with kubectl from the admin host kadmin:

MIG-610: turn gpu-node2 into 56 1g.10gb slices for inference. Check the MIG strategy, drain the node the right way, request the layout with the node label, watch mig.config.state, verify, and run a test pod on a slice.
MIG-720: a colleague's custom mixed layout on gpu-node3 is stuck at failed, and a notebook is Pending. Read the MIG Manager log, fix the ConfigMap, switch the strategy to mixed, retry the apply, and fix the notebook's resource request.

Outputs are simplified mock-ups modeled on kubectl, the GPU Operator's MIG Manager, and nvidia-smi, not exact copies. Each command advances time one step; MIG Manager takes a couple of steps to apply a layout. There's no editor: change files with sed -i 's/old/new/' <file>. Up/Down recall history; Tab completes commands, nodes, pod names, and files.

Ticket

Ticket checklist

Answer key for this ticket (spoilers)

One clean path; other orders work too. The checklist tracks results, not exact commands.

23. Scenarios

Try to name the cause before you open each one.

24. Searchable reference

Item	Meaning

25. Flashcards (click to flip)

26. Exam traps

MIG Manager does the MIG reconfiguration, not the strategy setting. Change the nvidia.com/mig.config node label (with --overwrite), on idle GPUs, and watch mig.config.state.
Run:ai runs on Kubernetes. It isn't a Slurm plugin, a monitoring tool, or a replacement for the GPU Operator.
Validate bottom-up: GPUs and fabric → GPU Operator (nvidia.com/gpu) → RDMA in pods → Run:ai.
Quota is per project per node pool.
Over-quota work is preemptible and is reclaimed when the GPUs' owner needs its quota.
Interactive and inference workloads usually stay within quota (non-preemptible). Pending workspaces with idle GPUs often mean "out of quota", not "out of GPUs".
Gang scheduling: distributed jobs start all workers or none.
Fractions are software sharing (memory split, compute time-shared). MIG is hardware isolation.
A project is a namespace on the cluster.
Run:ai's scheduler governs Run:ai workloads; pods using the default scheduler bypass its quotas.
Allocated ≠ utilized. Idle allocations are a policy problem.
Multi-node speed comes from the fabric, not from Run:ai: check NCCL uses NET/IB.
ImagePullBackOff is not a scheduling failure.

27. Practice quiz (28 questions)

Score: 0 / 0