AMD vs Intel GPU Server Tuning

How the host CPU platform affects a GPU server: NUMA layout (AMD NPS vs Intel SNC), socket links (xGMI vs UPI), power and C-state settings, IOMMU and PCIe ACS, OS tuning, and pinning work to the right CPU cores. Written for NCP-AII objective 5.4.

1. The big picture: why the CPU matters to GPUs

In a GPU server the GPUs do the math, but the host CPUs still feed them: they launch kernels, run the data loader, copy data from storage into host memory and across PCIe, service NIC interrupts, and run NCCL's proxy threads. If those CPU-side paths are slow, far away, or jittery, the GPUs wait.

Three things decide how well the host feeds the GPUs:

Topology: which CPU cores and which memory are physically close to each GPU and NIC (NUMA).
Firmware settings (BIOS): how the CPU exposes NUMA, how it manages power and frequency, how PCIe traffic is routed.
OS settings and placement: CPU governor, idle states, IOMMU mode, interrupt placement, and binding each process to the cores near its GPU.

GPUs are the kitchen's ovens; the CPUs are the prep cooks. If the prep cook for oven 3 works at the far end of the building and takes naps between orders, oven 3 sits empty no matter how hot it is.

Objective 5.4 asks you to optimize AMD- and Intel-based servers. The ideas are the same on both; the names differ. Learn the concept, then the AMD name and the Intel name (section 7 maps them). For DGX systems, NVIDIA ships tested settings: change them only when NVIDIA documentation tells you to. For OEM servers, follow the server vendor's GPU tuning guide.

2. NUMA in one page

NUMA (Non-Uniform Memory Access): each CPU socket (or part of a socket) has its own memory controllers and its own PCIe lanes. Accessing memory or a device attached to your node is fast; reaching one attached to another node crosses an internal link and is slower.

This is an illustrative layout, not a specific product's map. Real GPU-to-NUMA mappings come from nvidia-smi topo -m on the system.

A process driving GPU 5 should run on node 1's cores and allocate from node 1's memory. Otherwise every host-to-device copy crosses the socket link.
The OS reports NUMA nodes; numactl -H shows them with a distance matrix (10 = local; larger = farther).
BIOS options can split one socket into several NUMA nodes (AMD NPS, Intel SNC). More nodes means tighter locality but smaller memory per node.

NUMA = "Near Usually Means Advantage." Keep the process, its memory, its GPU, and its NIC on the same node.

3. AMD EPYC architecture

AMD EPYC processors (Rome 7002, Milan 7003, Genoa 9004, Turin 9005) are built from chiplets:

CCDs (Core Complex Dies) hold the cores and L3 cache. Each CCD has one or more CCXs (core complexes) sharing an L3.
A central I/O die holds the memory controllers (8 channels on Rome/Milan, 12 on Genoa/Turin), the PCIe lanes, and the Infinity Fabric that connects everything.
Two sockets connect with xGMI links (socket-to-socket Infinity Fabric).

Because the I/O die is divided into quadrants, memory and PCIe devices attached to one quadrant are slightly closer to some cores than others. The NPS setting decides how much of that the OS sees.

Setting	NUMA nodes per socket	Memory interleave	When
NPS0	1 for the whole 2-socket system	Across both sockets	Rare; not typical for GPU servers
NPS1	1 per socket	All channels in the socket	Simple default; best when apps aren't NUMA-aware
NPS2	2 per socket	Half the channels each	Middle ground
NPS4	4 per socket	One quadrant each	Best locality when processes are pinned per GPU/NIC

A separate option, LLC as NUMA node (L3 as NUMA), exposes each L3 cache domain as its own NUMA node. It's for very cache-sensitive, carefully pinned workloads and is not a common GPU-server choice.

NPS4 gives each GPU a "home" quadrant with its own cores and memory, so a pinned process per GPU sees the lowest latency. It only helps if you actually pin; unpinned workloads can do worse with more, smaller NUMA nodes.

4. AMD BIOS knobs

Knob	What it does	Typical GPU-server direction
NPS (NUMA nodes per socket)	Splits each socket into 1, 2, or 4 NUMA nodes	Per platform guide; NPS4 for pinned per-GPU work, NPS1 for simplicity
SMT	Two hardware threads per core	Usually on; some HPC benchmarks turn it off
Determinism slider	Performance determinism: every CPU behaves identically (repeatable). Power determinism: each CPU boosts as far as its own silicon allows (faster, varies part to part)	Power for maximum throughput; Performance when node-to-node consistency matters
cTDP / package power limit	Configurable thermal design power	Max supported value if cooling allows
APBDIS + fixed SOC P-state	Disables Algorithm Performance Boost on the I/O die so the Infinity Fabric stays at its highest P-state	APBDIS=1, P0: steadier memory/PCIe latency
DF C-states	Lets the data fabric sleep when idle	Disabled for lower I/O latency
Global C-state control	Core idle states	Per guide; deep states save power but add wake-up latency
xGMI link width / DLWM	Dynamic Link Width Management narrows the socket links when idle	Force maximum width (x16) and max speed for cross-socket bandwidth
IOMMU	DMA remapping for devices	Enabled with OS `iommu=pt`, or as the platform guide says
Preferred I/O	Gives one PCIe device (for example the NIC) priority on the I/O die	Used on some platforms for the high-speed NIC
Relaxed Ordering	Lets PCIe writes complete out of order	Often enabled for NIC/GPU throughput where supported
PCIe ACS	Forces peer-to-peer traffic up through the root complex	Disabled on bare-metal GPU nodes (see section 9)

AMD: "Nice Systems Don't Ask Daily Gains": NPS, SMT, Determinism, APBDIS, DF C-states, Global C-state.

5. Intel Xeon architecture

Intel Xeon Scalable processors (Ice Lake 3rd gen, Sapphire Rapids 4th gen, Emerald Rapids 5th gen, Granite Rapids Xeon 6) put cores on a mesh. Each core has a slice of the shared L3 cache (LLC). Larger parts are built from several tiles joined by EMIB bridges, but the OS sees them as one socket unless you enable SNC.

UPI (Ultra Path Interconnect) links connect sockets (up to 4 UPI links on Sapphire Rapids).
SNC (Sub-NUMA Clustering) splits each socket into 2 (SNC2) or 4 (SNC4) NUMA nodes, each with nearby cores, LLC slices, and memory controllers. It's the Intel equivalent of AMD NPS.
Uncore = everything outside the cores: mesh, LLC, memory controllers, PCIe. Its frequency affects memory and I/O bandwidth.

SNC setting	NUMA nodes per socket	Notes
Off (default)	1	Simplest; one node per socket
SNC2	2	Lower local latency when pinned
SNC4	4	Available on the largest parts; finest locality

6. Intel BIOS knobs

Knob	What it does	Typical GPU-server direction
SNC	Sub-NUMA Clustering	Per platform guide (often off on GPU servers)
Hyper-Threading	Two threads per core	Usually on
Turbo Boost	Opportunistic higher frequency	On
C-states (C1E, C6)	Core idle states	C1E and C6 often disabled for latency-sensitive nodes
EPB (Energy Performance Bias)	Hint between power saving and performance	Performance
HWP (Hardware P-states, Speed Shift)	CPU manages its own frequency	Enabled, with a performance-oriented EPP
Uncore frequency scaling	Lets mesh/LLC frequency drop when idle	Set to maximum (or disable scaling) for steady memory/I/O bandwidth
UPI link power management (L0p/L1)	Lets socket links sleep	Disabled for cross-socket bandwidth
VT-d	Intel's IOMMU	Enabled with `iommu=pt`, or as the platform guide says
Intel SST (Speed Select)	Trades core count for frequency profiles	Leave default unless the vendor guide says otherwise
LLC / XPT prefetch	Memory prefetchers	Vendor guide
PCIe ACS	Peer-to-peer redirection	Disabled on bare-metal GPU nodes

Intel: "Sam Has Two Cats Eating Under Velvet": SNC, Hyper-Threading, Turbo, C-states, EPB, Uncore, VT-d.

7. Interactive AMD ↔ Intel knob translator

Same idea, different names. Filter by a concept or either vendor's term.

Concept	AMD EPYC	Intel Xeon	Typical GPU-server direction

8. What DGX systems use (and why you leave them alone)

System	Host CPUs
DGX A100	2× AMD EPYC 7742 (64 cores each, 128 total)
DGX H100 / H200	2× Intel Xeon Platinum 8480C (56 cores each, 112 total)
DGX B200	2× Intel Xeon Platinum 8570 (56 cores each, 112 total)

DGX systems ship with BIOS settings NVIDIA validated for that platform, and DGX OS applies matching OS settings. Changing BIOS options on a DGX without NVIDIA guidance can hurt performance or support.
For NVIDIA-Certified / OEM HGX servers, the server vendor publishes a GPU or AI tuning guide. Apply it consistently on every node, then validate.
Firmware updates can reset BIOS settings to defaults. Re-check the settings after any BIOS update (see the HGX firmware guide).
Use Redfish BIOS attributes to read and compare settings across many nodes (see the Redfish guide).

"Tune the DGX BIOS for more speed" is usually the wrong answer. The right answer is to keep the validated settings identical across nodes and fix placement (pinning) and OS configuration.

9. PCIe: IOMMU, ACS, MRRS, link checks

IOMMU

The IOMMU (AMD-Vi / Intel VT-d) translates device DMA addresses. It's required for virtualization and device passthrough, but full translation adds overhead to every DMA on bare metal.

iommu=pt (passthrough) on the kernel command line: the IOMMU is on, but host devices use identity mapping, so there's little overhead.
Intel also needs intel_iommu=on on many distros to enable it; AMD uses amd_iommu=on (often on by default).
Check with cat /proc/cmdline and dmesg | grep -i -e iommu -e dmar.

ACS (Access Control Services)

ACS on PCIe switches and root ports forces peer-to-peer traffic (GPU to GPU, GPU to NIC) to travel up to the root complex for checking, instead of going directly through the PCIe switch. That breaks or slows GPUDirect P2P and GPUDirect RDMA.

# Look for ACS control flags on bridges; "SrcValid+" means ACS redirection is on
sudo lspci -vvv | grep -i acsctl
# ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- ...   (disabled: good for bare-metal P2P)

On bare-metal GPU nodes, disable ACS in the BIOS (or with setpci at boot if the vendor documents it).
In virtualized setups, ACS provides isolation between devices assigned to different VMs, so it's kept on there. See the virtualization guide.

MRRS and MPS

MRRS (Max Read Request Size) and MPS (Max Payload Size) set PCIe transaction sizes. NIC tuning guides often raise MRRS (for example to 4096) for throughput. lspci -vv shows them under DevCtl.

Link speed and width

sudo lspci -vv -s <bus:dev.fn> | grep -e LnkCap -e LnkSta
# LnkSta: Speed 32GT/s, Width x16   (Gen5 x16: as expected)
nvidia-smi -q | grep -A3 -i "link"   # current vs max PCIe generation and width

A GPU reporting a lower link generation at idle can be normal power saving; check under load. A link that stays at reduced width (x8 instead of x16) under load points to a seating, riser, or retimer issue.

10. OS-level tuning

Area	Command / setting	Purpose
Tuned profiles	`tuned-adm profile throughput-performance` (or `latency-performance` / `network-latency`); `tuned-adm active`	Applies a bundle of CPU, power, and kernel settings
CPU governor	`cpupower frequency-set -g performance`; `cpupower frequency-info`	Keep cores at high frequency
Idle states	`cpupower idle-info`; `cpupower idle-set -D <latency-us>`	Limit deep C-states that add wake-up latency
Kernel cmdline	`iommu=pt`, `intel_iommu=on` / `amd_iommu=on`, optional C-state limits such as `processor.max_cstate=1`	Boot-time platform behavior
Automatic NUMA balancing	`sysctl kernel.numa_balancing=0`	Stops the kernel migrating pages of already-pinned processes
Transparent huge pages	`cat /sys/kernel/mm/transparent_hugepage/enabled`	Follow the workload's guidance (`madvise` or `always`)
IRQ placement	`irqbalance`, or pin NIC IRQs to local cores (`set_irq_affinity_cpulist.sh`, `mlnx_tune`)	Interrupts handled near the NIC
NIC profile	`mlnx_tune -p HIGH_THROUGHPUT`	Applies NVIDIA networking's host tuning profile and reports issues
GPU persistence	`nvidia-persistenced` running	Avoids driver re-initialization delays

In a BCM cluster, put kernel parameters and service settings in the software image or the node category so every node gets them identically. Hand-tuning one node creates drift.

BIOS settings are the building's wiring; OS settings are the thermostat and light switches. Get the wiring right once per platform, then set the switches identically on every floor through one central panel (the image).

11. CPU, GPU, and NIC locality

nvidia-smi topo -m is the starting point. It prints how GPUs and NICs connect to each other and which CPU cores and NUMA node are local to each GPU.

nvidia-smi topo -m
#        GPU0  GPU1 ... NIC0  NIC1 ...  CPU Affinity   NUMA Affinity
# GPU0    X    NV18     PIX   SYS       0-55           0
# GPU4   NV18  NV18     SYS   PIX       56-111         1
# Legend: X = self, NV# = NVLink, PIX = same PCIe switch, PXB = multiple PCIe switches,
#         PHB = through the CPU's host bridge, NODE = across host bridges in one NUMA node,
#         SYS = across the socket link (slowest)

Output is illustrative. Check the real values on your system.

For GPUDirect RDMA, each GPU should have a NIC at PIX or PXB distance. SYS between a GPU and its NIC means traffic crosses sockets.
Bind the process for GPU n to the cores and memory listed in its CPU Affinity and NUMA Affinity columns.
NCCL reads the topology itself to choose NICs, but it can't fix a process running on the wrong socket.

# Run a process near GPU 4 (NUMA node 1 in this example)
CUDA_VISIBLE_DEVICES=4 numactl --cpunodebind=1 --membind=1 ./train.py
# Graphical/text topology of the whole host
lstopo --of console

Topology letters from near to far: NV → PIX → PXB → PHB → NODE → SYS. "Nice People Pick Paths Near Sockets."

12. Interactive locality checker

Pick an example layout, a GPU, and the NUMA node your process runs on. The example maps are simplified teaching layouts, not real product maps.

Layout

GPU

Process NUMA node

Scheduler

13. Pinning in Slurm and Kubernetes

Slurm

# gres.conf: tell Slurm which cores are local to each GPU (or let NVML detect it)
AutoDetect=nvml
# or explicitly:
Name=gpu File=/dev/nvidia0 Cores=0-13
Name=gpu File=/dev/nvidia4 Cores=56-69

# Job side: bind tasks to cores and GPUs
srun --gpus-per-task=1 --cpus-per-task=14 --cpu-bind=cores --gpu-bind=closest ./train

slurm.conf needs a task plugin that does binding, for example TaskPlugin=task/affinity,task/cgroup.
--gpu-bind=closest picks GPUs near the task's CPUs.

Kubernetes

# kubelet configuration
cpuManagerPolicy: static
topologyManagerPolicy: single-numa-node   # or best-effort / restricted
memoryManagerPolicy: Static               # optional, for NUMA-local memory

The Topology Manager aligns CPUs, memory, and devices (GPUs from the device plugin, NICs from the Network Operator) to one NUMA node.
Static CPU pinning only applies to Guaranteed QoS pods (requests equal limits) with whole-number CPU requests.

single-numa-node rejects a pod that can't fit in one NUMA node. With NPS4 or SNC, nodes are small; an 8-GPU pod won't fit in one. Choose the policy to match the pod shapes.

14. Verification commands

Command	Shows
`lscpu`	Sockets, cores, threads per core (SMT/HT), NUMA nodes and their CPU lists, CPU model
`numactl -H`	NUMA nodes, memory per node, distance matrix
`lstopo` (hwloc)	Full host topology: caches, cores, PCIe devices per node
`nvidia-smi topo -m`	GPU/NIC connections and CPU/NUMA affinity
`cpupower frequency-info`	Governor and frequency limits
`cpupower idle-info`	Available and enabled C-states
`turbostat`	Actual frequency and C-state residency per core
`cat /proc/cmdline`	Kernel parameters in effect
`sysctl kernel.numa_balancing`	Automatic NUMA balancing on/off
`tuned-adm active`	Active tuned profile
`lspci -vvv \| grep -i acsctl`	ACS state on bridges
`nvbandwidth`	Host↔device and device↔device bandwidth per GPU
STREAM	Host memory bandwidth (per NUMA node when pinned)
`mlnx_tune -r`	Report-only check of host tuning for NVIDIA NICs

All nodes show the same NUMA layout, tuned profile, governor, kernel parameters, and BIOS settings; host-to-device bandwidth is similar for every GPU and every node; ACS is off on bare-metal P2P paths; GPU-to-NIC paths are PIX/PXB.

15. Symptoms of a badly tuned host

Symptom	Likely host cause	Check
Some GPUs show lower host-to-device bandwidth	Process running on the far socket; degraded PCIe link	`nvbandwidth` with and without `numactl`; `lspci LnkSta`
GPUDirect P2P/RDMA slow or failing on bare metal	ACS enabled; IOMMU in full translation	`lspci \| grep ACSCtl`; `/proc/cmdline`
Run-to-run jitter, latency spikes in small NCCL messages	Deep C-states, powersave governor, fabric power states	`turbostat`, `cpupower`, BIOS power settings
One node consistently slower than identical neighbors	BIOS or OS settings differ (drift after a BIOS update or manual change)	Compare BIOS attributes (Redfish), `tuned-adm active`, cmdline
Data loader can't keep GPUs busy	Too few cores per GPU, cores on the wrong node, SMT off with thread-hungry loaders	`top`/`htop`, pinning, `lscpu`
Kubernetes pod stuck with a topology affinity error	`single-numa-node` can't fit the request	Pod events; reduce request or change policy
Memory bandwidth lower than expected	Unbalanced DIMM population, NUMA split mismatched to pinning	STREAM per node; `numactl -H` memory per node

16. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

numactl -HNUMA nodes, memory and distances

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 ... 54 55 112 113 114 ... 166 167
node 0 size: 1031782 MB
node 0 free: 1007214 MB
node 1 cpus: 56 57 58 59 60 61 62 63 ... 110 111 168 169 170 ... 222 223
node 1 size: 1032169 MB
node 1 free: 1010655 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

What to look at: A DGX H100 with SNC off shows 2 nodes with about 1 TB each and a distance of 10 local versus 21 remote. Unequal memory per node suggests unbalanced or failed DIMMs, and a different node count than its neighbours means the BIOS NUMA settings have drifted.

numactl --hardwareAMD EPYC host in NPS4 mode

available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 ... 15 128 129 ... 143
node 0 size: 128803 MB
node 0 free: 121344 MB
node 1 cpus: 16 17 18 19 ... 31 144 145 ... 159
node 1 size: 129021 MB
node 1 free: 124018 MB
node 2 cpus: 32 33 34 35 ... 47 160 161 ... 175
node 2 size: 129021 MB
node 2 free: 123987 MB
node 3 cpus: 48 49 50 51 ... 63 176 177 ... 191
node 3 size: 129009 MB
node 3 free: 124102 MB
node 4 cpus: 64 65 66 67 ... 79 192 193 ... 207
node 4 size: 129021 MB
node 4 free: 123870 MB
node 5 cpus: 80 81 82 83 ... 95 208 209 ... 223
node 5 size: 129021 MB
node 5 free: 124233 MB
node 6 cpus: 96 97 98 99 ... 111 224 225 ... 239
node 6 size: 129021 MB
node 6 free: 124011 MB
node 7 cpus: 112 113 114 115 ... 127 240 241 ... 255
node 7 size: 128997 MB
node 7 free: 123954 MB
node distances:
node   0   1   2   3   4   5   6   7
  0:  10  12  12  12  32  32  32  32
  1:  12  10  12  12  32  32  32  32
  2:  12  12  10  12  32  32  32  32
  3:  12  12  12  10  32  32  32  32
  4:  32  32  32  32  10  12  12  12
  5:  32  32  32  32  12  10  12  12
  6:  32  32  32  32  12  12  10  12
  7:  32  32  32  32  12  12  12  10

What to look at: On a 2-socket AMD EPYC 7742 host (DGX A100 class) NPS4 gives 8 NUMA nodes of 16 cores each: 10 is local, 12 is another quadrant in the same socket and 32 crosses xGMI to the other socket. Seeing only 2 nodes means NPS1, and unpinned jobs on NPS4 often land on remote quadrants.

cat /proc/cmdlineKernel parameters in effect

BOOT_IMAGE=/boot/vmlinuz-5.15.0-1042-nvidia root=UUID=3f6a2c1e-7b4d-4e2a-9c51-0d8e2f7a6b10 ro intel_iommu=on iommu=pt nvme_core.multipath=n rd.driver.blacklist=nouveau nouveau.modeset=0 transparent_hugepage=madvise quiet splash

What to look at: Look for iommu=pt with intel_iommu=on (or amd_iommu=on): the IOMMU stays on but host devices use passthrough mapping. An IOMMU in full translation mode (no iommu=pt) can slow or break GPUDirect P2P and RDMA, and parameters that differ between nodes explain one slow node.

dmesg | grep -i -e iommu -e dmarIOMMU mode at boot

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1042-nvidia root=UUID=3f6a2c1e-7b4d-4e2a-9c51-0d8e2f7a6b10 ro intel_iommu=on iommu=pt ...
[    0.000000] DMAR: IOMMU enabled
[    0.012418] ACPI: DMAR 0x000000006A1B2000 0002E8 (v01 INTEL  EDK2     00000002      01000013)
[    1.204117] DMAR: Host address width 46
[    1.204119] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0
...
[    2.311420] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    2.402118] iommu: Default domain type: Passthrough (set via kernel command line)
[    2.413005] pci 0000:00:00.0: Adding to iommu group 0
[    2.413082] pci 0000:00:01.0: Adding to iommu group 1
...

What to look at: 'Default domain type: Passthrough' confirms iommu=pt took effect. 'Default domain type: Translated' means every DMA is remapped, which adds overhead and can break peer-to-peer paths on bare metal.

sudo lspci -vvv | grep -i acsctlACS state on PCIe bridges

                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
...

What to look at: All flags ending in minus means ACS is off, which is what bare-metal GPUDirect P2P and RDMA need. SrcValid+ or ReqRedir+ on bridges between GPUs and NICs forces traffic up to the root complex, giving slow or failing P2P.

sudo lspci -vv -s 18:00.0 | grep -e LnkCap -e LnkStaPCIe link speed and width for GPU0

LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM not supported
LnkSta: Speed 32GT/s (ok), Width x16 (ok)
LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+

What to look at: Compare LnkSta with LnkCap: 32GT/s x16 is PCIe Gen5 x16. 'Speed 16GT/s (downgraded)' or 'Width x8 (downgraded)' under load means a bad riser, slot or retimer; an idle GPU may legitimately show a lower speed for power saving, so check while it is busy.

nvidia-smi -q -i 0 | grep -A3 -i "link"Current vs max PCIe generation and width

    GPU Link Info
        PCIe Generation
            Max                               : 5
            Current                           : 5
--
        Link Width
            Max                               : 16x
            Current                           : 16x
--
        Link Detection                        : N/A
...

What to look at: Current should match Max (generation 5, 16x) while the GPU is under load. Current generation 1 at idle is normal power saving, but Current below Max during a job, or a width below 16x, indicates a degraded PCIe link.

tuned-adm activeActive tuned profile

Current active profile: throughput-performance

What to look at: Compute nodes should run throughput-performance or a latency profile. balanced or powersave, or a profile that differs from the other nodes, explains lower clocks and run-to-run jitter.

cpupower frequency-infoCPU governor and frequency limits

analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 800 MHz - 3.80 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 800 MHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 3.00 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes

What to look at: You want the governor to be performance and boost Active: yes. The governor 'powersave' or a capped upper limit below the hardware maximum leads to slow data loading and latency spikes in small NCCL messages.

cpupower idle-infoAvailable and enabled C-states

CPUidle driver: intel_idle
CPUidle governor: menu
analyzing CPU 0:

Number of idle states: 4
Available idle states: POLL C1 C1E C6
POLL:
Flags/Description: CPUIDLE CORE POLL IDLE
Latency: 0
Usage: 1203
Duration: 2201134
C1:
Flags/Description: MWAIT 0x00
Latency: 1
Usage: 4120331
Duration: 912004118
C1E:
Flags/Description: MWAIT 0x01
Latency: 2
Usage: 2210887
Duration: 1830114220
C6:
Flags/Description: MWAIT 0x20
Latency: 290
Usage: 118220
Duration: 402118337

What to look at: Latency is the wake-up cost in microseconds; C6 at 290 us is what causes jitter for latency-sensitive work. A state marked (DISABLED) has been turned off with cpupower idle-set -D or the BIOS; check this matches the vendor's tuning guide.

sysctl kernel.numa_balancingAutomatic NUMA balancing on or off

kernel.numa_balancing = 1

What to look at: 1 means the kernel migrates pages between NUMA nodes on its own, which works against jobs that are already pinned with numactl or Slurm binding. Pinned HPC nodes usually set it to 0.

cat /sys/kernel/mm/transparent_hugepage/enabledTransparent huge pages mode

always [madvise] never

What to look at: The bracketed value is the active mode; madvise here means only applications that ask get huge pages. Follow the workload's guidance, and treat a value that differs from the rest of the cluster as configuration drift.

sudo mlnx_tune -rReport-only NIC host tuning check

2026-11-10 16:02:11,113 INFO Collecting node information
2026-11-10 16:02:11,114 INFO Collecting OS information
2026-11-10 16:02:11,121 INFO Collecting CPU information
...
Mellanox Technologies - System Report

Operation System Status
UBUNTU22.04
5.15.0-1042-nvidia

CPU Status
GenuineIntel Intel(R) Xeon(R) Platinum 8480C Sapphire Rapids
OK: Frequency 3000.0MHz

Hyper Threading Status
ACTIVE

IRQ Balancer Status
ACTIVE

Driver Status
OK: MLNX_OFED_LINUX-23.10-1.1.9.0 (OFED-23.10-1.1.9)

ConnectX-7 Device Status on PCI 1a:00.0
FW version 28.39.1002
OK: PCI Width x16
OK: PCI Speed 32GT/s
PCI Max Payload Size 256
PCI Max Read Request 4096
Local CPUs list [0-55,112-167]

mlx5_0 (Port 1) Status
Link Type ib
OK: Link status Up
Speed NDR
MTU 4096
...

What to look at: -r only reports, so it is safe on a production node. Look for Warning lines (for example a low CPU frequency, a NIC below x16 or 32GT/s, or a small Max Read Request) and check the Local CPUs list matches the GPU it serves.

nvidia-smi topo -mGPU/NIC links and CPU/NUMA affinity

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    0-55,112-167    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     NODE    NODE    0-55,112-167    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    PIX     NODE    0-55,112-167    0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE    PIX     0-55,112-167    0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC1    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC2    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE
NIC3    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X
...

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_3
  NIC2: mlx5_4
  NIC3: mlx5_5
  ...

What to look at: Use the CPU Affinity and NUMA Affinity columns to pin each GPU's process to its local socket: GPUs 0-3 belong to NUMA node 0 and GPUs 4-7 to node 1. Each GPU should have a PIX (or PXB) NIC; a GPU whose nearest NIC is SYS sends RDMA traffic across the socket link.

lstopo --of consoleFull host topology from hwloc

Machine (2015GB total)
  Package L#0
    NUMANode L#0 (P#0 1007GB)
    L3 L#0 (105MB)
      L2 L#0 (2048KB) + L1d L#0 (48KB) + L1i L#0 (32KB) + Core L#0
        PU L#0 (P#0)
        PU L#1 (P#112)
      L2 L#1 (2048KB) + L1d L#1 (48KB) + L1i L#1 (32KB) + Core L#1
        PU L#2 (P#1)
        PU L#3 (P#113)
      ...
    HostBridge
      PCIBridge
        PCIBridge
          PCIBridge
            PCI 18:00.0 (3D)
              CoProc(CUDA) "cuda0"
              GPU(NVML) "nvml0"
          PCIBridge
            PCI 1a:00.0 (InfiniBand)
              Net "ibp26s0"
              OpenFabrics "mlx5_0"
    ...
  Package L#1
    NUMANode L#1 (P#1 1008GB)
    L3 L#1 (105MB)
    ...

What to look at: lstopo shows which package and NUMA node each GPU (cuda0, nvml0) and NIC (mlx5_0) hangs off, and that the GPU and its NIC share a PCIe switch. A GPU and its paired NIC under different packages is a cabling or slot-placement problem.

lscpuSockets, cores, threads and NUMA CPU lists

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 57 bits virtual
  Byte Order:            Little Endian
CPU(s):                  224
  On-line CPU(s) list:   0-223
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) Platinum 8480C
    CPU family:          6
    Model:               143
    Thread(s) per core:  2
    Core(s) per socket:  56
    Socket(s):           2
    Stepping:            8
    CPU max MHz:         3800.0000
    CPU min MHz:         800.0000
    BogoMIPS:            4000.00
Virtualization features:
  Virtualization:        VT-x
Caches (sum of all):
  L1d:                   5.3 MiB (112 instances)
  L1i:                   3.5 MiB (112 instances)
  L2:                    224 MiB (112 instances)
  L3:                    210 MiB (2 instances)
NUMA:
  NUMA node(s):          2
  NUMA node0 CPU(s):     0-55,112-167
  NUMA node1 CPU(s):     56-111,168-223
...

What to look at: A DGX H100 should show 2 sockets of Xeon Platinum 8480C, 56 cores each, 2 threads per core (224 CPUs) and 2 NUMA nodes. Thread(s) per core of 1 means SMT was disabled, and 4 or 8 NUMA nodes means SNC is on; both should match the rest of the cluster.

sudo turbostat --quiet --interval 5 -n 1 --show Package,Core,CPU,Avg_MHz,Busy%,Bzy_MHz,C6%,PkgWattActual frequency and C-state residency

Package Core    CPU     Avg_MHz Busy%   Bzy_MHz C6%     PkgWatt
-       -       -       2711    90.12   3008    4.31    671.22
0       0       0       2995    99.61   3007    0.00    338.14
0       0       112     12      0.41    2998    0.00
0       1       1       2990    99.43   3007    0.00
0       1       113     9       0.30    3001    0.00
...
1       0       56      2988    99.35   3008    0.00    333.08
1       0       168     15      0.50    3004    0.00
...

What to look at: cpupower shows configuration; turbostat shows behavior. Busy cores should run near the expected all-core frequency in Bzy_MHz with little C6% residency; low Bzy_MHz under load or large C6% on busy cores points at power-saving BIOS settings or thermal or power capping.

17. Scenarios

18. Searchable reference

Item	Meaning

19. Flashcards (click to flip)

20. Exam traps

NPS (AMD) = SNC (Intel): both split a socket into more NUMA nodes. xGMI (AMD) = UPI (Intel): socket links.
More NUMA nodes only helps if work is pinned.
ACS off for bare-metal GPUDirect P2P/RDMA; ACS on for VM isolation.
iommu=pt keeps the IOMMU enabled but avoids translation overhead for host devices.
SYS in nvidia-smi topo -m = crosses the socket link; worst path for GPU-to-NIC.
Use numactl --cpunodebind/--membind, Slurm --cpu-bind + gres.conf Cores, or the Kubernetes Topology Manager to keep work local.
DGX: keep NVIDIA's validated BIOS settings; OEM: follow the vendor's GPU tuning guide.
Consistency beats cleverness: identical settings on every node, applied through the image.
BIOS updates can reset settings; re-verify after firmware work.
Deep C-states and powersave governors cause latency jitter, not usually lower peak bandwidth.

21. Practice quiz (24 questions)

Score: 0 / 0