AI Factory Architecture
What an AI factory is, the NVIDIA reference designs it's built from, the four networks, rail-optimized fat-tree topology, sizing math for switches, cables and power, and the software stack on top. It ends with a hands-on design lab with an answer key. Numbers come from NVIDIA's DGX SuperPOD (H100 and GB200), DGX BasePOD, and Enterprise Reference Architecture documents.
1. What an "AI factory" is (and isn't)
AI factory is NVIDIA's name for a data center built to produce AI. A traditional data center stores data and runs many unrelated applications. An AI factory has one main job: turn data and electricity into intelligence. That means trained models (training), and answers or tokens generated by those models (inference).
A car factory takes in steel and power and ships cars. It's judged on output per hour and cost per car, and one broken machine on the line slows everything. An AI factory takes in data and power and ships trained models and tokens. It's judged on time-to-train, tokens per second, and performance per watt. And one slow GPU node can slow a whole synchronous training job.
It's a full stack, not one product
| Layer | What's in it | Study guide |
|---|---|---|
| Facility | Power, cooling (liquid for dense racks), floor space, cabling | Power and cooling below; bring-up phase 0 |
| Compute | GPU systems: DGX / HGX H100, H200, B200, and rack-scale systems like GB200 NVL72 | DGX H100 specs, NVLink, Fabric Manager |
| Compute fabric | InfiniBand (Quantum) or Ethernet (Spectrum-X) linking GPUs across nodes | Topology below; InfiniBand, RoCE & Spectrum-X |
| Storage | High-performance parallel storage, NFS user storage, local NVMe cache, GPUDirect Storage | Lustre & third-party storage, DGX & cluster storage |
| Management | BMCs, provisioning, cluster management, fabric management, monitoring | Software stack below |
| Communication software | CUDA, NCCL, GPUDirect RDMA | CUDA, NCCL |
| Workloads | Schedulers, NGC containers, NVIDIA AI Enterprise, NeMo / Megatron training, NIM inference microservices | NGC, Run:ai |
An AI factory is only as productive as its weakest layer. Large training jobs are synchronous: every step waits for the slowest GPU, link, or node. One bad cable or throttling node lowers the output of the whole factory.
"AI factory" is not a specific NVIDIA product, switch, or software package you install. If a question asks what makes up an AI factory, think full stack: accelerated compute + high-speed fabric + storage + management + software, built from validated reference architectures and measured by output (training and inference throughput).
2. Reference architectures: BasePOD, SuperPOD, Enterprise RAs
NVIDIA publishes validated blueprints (reference architectures, "RAs") so nobody has to design every layer from scratch. Know the three families and what separates them.
| DGX BasePOD | DGX SuperPOD | Enterprise Reference Architectures | |
|---|---|---|---|
| What it is | A flexible reference design, driven mainly by NVIDIA-certified storage partners, who can customize it | NVIDIA's turnkey product: a specific bill of materials, installation services, support, and guaranteed performance | Prescriptive designs for partner-built NVIDIA-Certified servers (HGX, PCIe GPUs, NVL72) on Spectrum-X Ethernet |
| Must contain | DGX systems + NVIDIA networking + NVIDIA software including Base Command Manager + certified third-party storage | Exactly the RA's components: DGX, NVIDIA networking only, certified storage, BCM (or Mission Control), Slurm | Certified servers in a defined node pattern, Spectrum-X networking, NVIDIA AI Enterprise |
| Size | Current RA: 2–8 DGX nodes (B200, H200, or H100); "from two nodes to dozens" | Built in scalable units. Size isn't what makes it a SuperPOD. | Design points such as 32, 64, and 128 nodes (HGX AI Factory) or 4–32 nodes (RTX PRO AI Factory) |
| Networks | 3: InfiniBand compute; Ethernet for management and storage (or a separate IB storage fabric); OOB | 4 separate fabrics: compute, storage, in-band, OOB | Small designs share one fabric for north-south and east-west; larger ones separate them |
What turns a SuperPOD into "just" a BasePOD
NVIDIA's SuperPOD FAQ lists changes that generally disqualify a system from being a DGX SuperPOD. In many cases the result is still a BasePOD:
- Another cluster manager or scheduler (for example Kubernetes instead of Slurm)
- Another OS (for example Red Hat instead of DGX OS / Ubuntu)
- Storage that isn't SuperPOD-certified
- A changed compute, storage, or in-band fabric topology, or non-NVIDIA networking
Drop a core component (your own cluster manager, uncertified storage, another vendor's switches) and it's neither: it's a large custom DGX cluster.
A SuperPOD isn't defined by node count. A 1-SU system with nodes still being installed can be a SuperPOD. A cluster with hundreds of DGX systems can fail to be one if it deviates from the RA.
Enterprise RA node names: C-G-N-B
Enterprise RAs describe each server pattern as CPUs - GPUs - NICs - east-west bandwidth per GPU (Gb/s):
| Pattern | Meaning |
|---|---|
2-8-5-200 | 2 CPU sockets, 8 GPUs, 5 NICs (1 north-south + 4 east-west), 200 Gb/s east-west per GPU. A PCIe-GPU design, for example RTX PRO 6000 Blackwell Server Edition. |
2-8-9-400 | 2 CPUs, 8 GPUs, 9 NICs (1 north-south + 8 east-west, one per GPU), 400 Gb/s per GPU |
2-8-9-800 | The HGX B300 pattern: one 800 Gb/s east-west NIC per GPU |
2-4-3-200 | 2 CPUs, 4 GPUs, 3 NICs (1 north-south + 2 east-west), 200 Gb/s per GPU |
"Count GPUs, NICs, Bandwidth." In the N field, one NIC is north-south and the rest are east-west. 8 GPUs with 4 east-west NICs means two GPUs share each NIC, which is why the per-GPU figure halves.
3. The scalable unit (SU)
A SuperPOD isn't designed node by node. It's built from identical blocks called scalable units, each with its own racks, leaf switches and cabling, and all of them attach to a common spine (and core, for very large systems).
| Platform | One SU | Scale |
|---|---|---|
| DGX H100 SuperPOD | 32 DGX H100 = 256 GPUs and 8 compute leaf switches (one per rail). In practice 31 nodes in a 1-SU build: one node's leaf ports are given to UFM connectivity. | Tested configuration up to 4 SUs (127 nodes, 1,016 GPUs). The RA describes larger designs with a core tier. |
| DGX GB200 SuperPOD | 8 DGX GB200 NVL72 racks = 576 GPUs, about 1.2 MW | Up to 16 SUs in the RA's spine-leaf-group design (128 racks, 9,216 GPUs) |
- Why blocks: every SU is cabled, validated, and behaves the same. Cable plans, bills of materials and test results repeat; failures stay inside a known domain.
- Partial SUs: if you deploy fewer nodes, NVIDIA's guidance is to still build the fabric for the full SU (leaf switches and leaf-spine cables) and leave the unused node ports empty. Routing stays balanced and adding nodes later doesn't change the fabric.
- No sharing: SU racks are engineered for the SU; NVIDIA says not to put unrelated IT equipment in their empty space.
- No VLAN split: a SuperPOD can't be logically split between SUs with VLANs. It can only be split physically into completely separate compute fabrics.
"32 nodes, 8 leaves, 256 GPUs" for H100. "8 racks, 576 GPUs, 1.2 MW" for GB200.
4. The four networks 1.2
Many questions test whether you know which network is involved. A DGX SuperPOD has four separate fabrics. Inside each node or rack there's also the NVLink domain, which is "scale-up" rather than a network you cable between nodes.
| Network | Direction | Carries | DGX H100 SuperPOD hardware |
|---|---|---|---|
| Compute fabric | East-west (GPU ↔ GPU between nodes) | NCCL collectives for training: nothing else | 8 × ConnectX-7 NDR 400 Gb/s per node (4 twin-port OSFP cages), Quantum-2 QM9700 switches, rail-optimized non-blocking fat tree |
| Storage fabric | East-west (nodes ↔ storage) | Datasets in, checkpoints out, from the high-performance storage | Separate InfiniBand fabric (QM9700). Storage side 1:1; DGX side slightly oversubscribed (about 4:3). Designed for more than 40 GB/s per node. |
| In-band management | North-south (users, admin, internet, provisioning) | SSH, PXE provisioning, Slurm control, monitoring, NFS home directories, internet access | Ethernet (SN4600C), ports on the dual-port ConnectX-7 adapters |
| Out-of-band (OOB) | Isolated management | BMC access: power control, remote console, sensors, firmware | 1 GbE Ethernet (SN2201) to every BMC, switch management port, PDU; on GB200 also BlueField BMCs and NVLink switch tray management |
Compute fabric = training traffic. Storage fabric = data. In-band = OS, admin, users. Out-of-band = the "lights-out" BMC network that works even when the OS is down.
East-west = traffic between servers inside the data center (GPU-to-GPU, node-to-storage). North-south = traffic in and out of the cluster (users, internet, external services). Training traffic is east-west on the compute fabric, never the management network.
Two kinds of storage, on two networks
- High-performance storage (HPS), the "inner ring": a certified parallel filesystem on the storage fabric for datasets and checkpoints. NVIDIA's SuperPOD FAQ lists DDN, Dell PowerScale, IBM Storage Scale, NetApp, Pure Storage, WEKA, and VAST.
- User storage: an NFS share for
$HOMEand similar, served over the in-band network. An object store or data lake ("outer ring") is the customer's responsibility.
How the designs differ
| Design | Network layout |
|---|---|
| DGX BasePOD (2–8 nodes) | 3 networks: InfiniBand compute, one Ethernet fabric for in-band management and storage (or a separate IB storage fabric), OOB |
| DGX H100 SuperPOD | 4 separate fabrics, storage on InfiniBand |
| DGX GB200 SuperPOD | InfiniBand compute; storage and in-band share one Ethernet fabric (SN5600, storage over RoCE, separated with VXLAN); OOB |
| Enterprise RAs | Spectrum-X Ethernet everywhere; small clusters (for example up to 16 RTX PRO nodes) share one fabric for north-south and east-west, larger ones separate them |
5. Compute fabric topology: rails and fat trees 1.2
- Rail-optimized: NIC i (the one next to GPU i) on every node in the SU connects to leaf switch i. Each leaf is one "rail". GPU 3 on node 1 reaches GPU 3 on any other node in the SU through one leaf: one hop. Traffic between rails, or between SUs, goes through the spine.
- Why it fits AI traffic: NCCL builds rings and trees per rail, so most collective traffic stays on the same rail. When GPU 3 needs to reach GPU 5 on another node, NCCL's PXN moves the data over NVLink to the local GPU 5 first, then out on rail 5, which avoids the spine.
- Fat tree: leaf → spine (→ core in very large systems). Every leaf connects to every spine, so there are many equal paths between any two leaves.
- Non-blocking (1:1): each leaf has as much uplink bandwidth as downlink (on a 64-port QM9700: 32 ports down to nodes, 32 up to spines). Any half of the nodes can talk to the other half at full speed. Oversubscribed fabrics (for example 2:1) are cheaper but slow all-to-all traffic. SuperPOD compute fabrics are always non-blocking; storage fabrics may be slightly oversubscribed.
- Managed fabric: InfiniBand needs a Subnet Manager; in a SuperPOD that's UFM in high availability. Quantum switches add SHARP (in-network reductions) and adaptive routing.
- Ethernet version: Spectrum-X uses the same rail-optimized leaf-spine idea with RoCE, adaptive routing, and SuperNICs. Details in RoCE & Spectrum-X.
Rail-optimized means port i of every node to leaf i, not "all 8 ports of a node to the same leaf" (that's top-of-rack thinking). A swapped pair of cables on one node silently pushes that node's traffic through the spine: NCCL still works, just slower.
ibnetdiscover / ibdiagnet topology checks against the cabling plan catch it.InfiniBand cable length limits apply to the whole signal path. NVIDIA's FAQ says you can't extend them with patch panels or extra top-of-rack switches: patch panels add signal loss, and extra switches add latency.
6. Sizing math: switches, cables, transceivers 1.2
The building blocks
- QM9700 (Quantum-2): 32 OSFP cages = 64 NDR 400 Gb/s ports. Each cage takes a twin-port transceiver (2 × 400G).
- DGX H100: 4 OSFP cages = 8 compute ports, one per GPU.
- Leaf (non-blocking): 32 ports down (one per node in the SU), 32 ports up.
The H100 SuperPOD compute fabric table
From NVIDIA's DGX SuperPOD H100 reference architecture:
| SUs | Nodes | GPUs | Leaf switches | Spine switches | Node + UFM cables | Spine-leaf cables |
|---|---|---|---|---|---|---|
| 1 | 31 | 248 | 8 | 4 | 252 | 256 |
| 2 | 63 | 504 | 16 | 8 | 508 | 512 |
| 3 | 95 | 760 | 24 | 16 | 764 | 768 |
| 4 | 127 | 1,016 | 32 | 16 | 1,020 | 1,024 |
Deriving it yourself
| Quantity | Rule | 1 SU | 4 SUs |
|---|---|---|---|
| Nodes | 32 × SUs − 1 (one node's slot used for UFM) | 31 | 127 |
| GPUs | 8 × nodes | 248 | 1,016 |
| Leaf switches | 8 (rails) × SUs | 8 | 32 |
| Node cables | 8 × nodes + 4 UFM links | 252 | 1,020 |
| Leaf uplinks = spine-leaf cables | 32 × leaves | 256 | 1,024 |
| Spine switches | At least uplinks ÷ 64, rounded up to a count that divides 32 evenly (see below) | 4 | 16 |
Why 3 SUs need 16 spines, not 12: 768 uplinks ÷ 64 = 12 spines would be enough ports, but each leaf has to spread its 32 uplinks evenly over every spine so all paths are equal. 32 ÷ 12 isn't a whole number. The next spine count that divides 32 is 16 (2 links from each leaf to each spine), which is also exactly what 4 SUs need. That's a reading of the table; the RA itself just lists the counts.
Transceivers (rough count)
With twin-port OSFP transceivers everywhere, every 2 ports in use need one transceiver, at each end of the link. For 1 SU: node side 31 × 4 = 124, leaf side (252 + 256) ÷ 2 = 254, spine side 256 ÷ 2 = 128. Real bills of materials vary: short runs may use copper cables, UFM uses its own adapters, and cable types depend on distance.
One SU at a glance (DGX H100)
- GPUs: 32 × 8 = 256; GPU memory 32 × 640 GB = 20.48 TB
- Compute links into the leaves: 32 × 8 = 256 × 400 Gb/s
- FP8 compute: 32 × 32 PF ≈ 1 exaFLOP (sparse)
- Power (DGX only): 32 × ~10.2 kW ≈ ~326 kW
The full 4-SU H100 SuperPOD bill of materials lists 127 DGX H100, 48 compute QM9700 switches (32 leaf + 16 spine), 16 storage QM9700, 8 SN4600C in-band switches, 8 SN2201 OOB switches, 4 UFM appliances, 5 management servers, and 38 racks.
7. Power, cooling, and racks 1.5
| Fact | Value |
|---|---|
| DGX H100 maximum power, size | ~10.2 kW, 8U |
| DGX H100 per air-cooled rack | NVIDIA's FAQ: 2 is the practical maximum in a typical air-cooled data center; 3–4 only where the facility is built for that power and cooling density |
| Airflow | Front-to-back; SuperPOD components pull cool air from the DGX bezel side |
| Power feeds | Several DGX per rack needs three-phase power (or many single-phase circuits), with redundant A/B feeds |
| Racks | EIA-310, 19-inch, at least 600 × 1100 mm |
| GB200 NVL72 rack | ~120 kW, hybrid liquid + air cooling |
| GB200 power shelves | 8 per rack, each 6 × 5.5 kW PSUs = 33 kW |
| GB200 SU | ~1.2 MW |
| Facility standard | GB200 RA: Uptime Tier 3 (or equivalent): concurrent maintainability, no single point of failure |
For dense GPU racks the facility usually limits the design before the network does. NVIDIA's FAQ makes the point that scaling comes from adding SUs to a common spine, not from cramming more nodes into each rack.
A rear-door heat exchanger (chilled water in the rack door) raises air-cooled density where plumbing allows it. Rack-scale systems need coolant distribution units (CDUs), flow and temperature monitoring, and leak detection. Bring-up checks for power and cooling are in bring-up objective 1.5.
8. Rack-scale: GB200 NVL72 SuperPOD
With GB200 NVL72, the unit of compute becomes the rack. The NVLink domain spans all 72 GPUs in the rack, so the boundary between "scale-up" (NVLink) and "scale-out" (InfiniBand) moves from the node to the rack.
| Part | Detail |
|---|---|
| Compute trays | 18 per rack. Each has 2 GB200 Superchips (each = 1 Grace CPU + 2 Blackwell GPUs) = 4 GPUs per tray, joined by NVLink-C2C so one OS runs per tray. |
| NVLink switch trays | 9 per rack, 2 NVLink switch chips each (18 chips). Each GPU has 18 NVLink 5 links, one to each switch chip: 1.8 TB/s per GPU, all 72 GPUs in one NVLink domain. |
| Compute fabric NICs | 4 × ConnectX-7 per tray (NDR 400 Gb/s InfiniBand): one rail per GPU in the tray, so 4 rails |
| Storage and in-band NICs | 2 × BlueField-3 per tray (2 × 200 Gb/s), on the shared Ethernet storage/in-band fabric |
| Local storage | 4 × 3.84 TB E1.S NVMe (RAID 0) + 1 × 1.92 TB M.2 for the OS, per tray |
| Management software | NVIDIA Mission Control, which includes BCM technology, Run:ai, and NMX for the NVLink switch trays |
| Scale-out | The RA's spine-leaf-group design: 4 groups per SU (one per rail), 32 leaf and 24 spine switches per SU, up to 16 SUs |
On DGX H100, NVLink stops at the node (8 GPUs) and everything between nodes is InfiniBand. On GB200 NVL72, NVLink covers the whole rack (72 GPUs). Don't carry one platform's boundary over to the other.
9. The software stack, layer by layer
| Layer | What runs there | Study guide |
|---|---|---|
| Hardware management | BMC, Redfish, IPMI on the OOB network; firmware tools | Redfish & BMC, HGX firmware, MFT |
| Cluster management | Base Command Manager (BCM): provisioning, images, categories, monitoring. GB200/B200 SuperPODs use Mission Control, which builds on BCM and adds automated failure detection and recovery, health checks, and checkpoint-based job recovery. | Bring-up phase 2, BCM node recovery |
| Fabric management | UFM for InfiniBand (Subnet Manager, telemetry), NetQ for Ethernet, NMX for NVLink switch trays, Fabric Manager inside HGX nodes | InfiniBand, Fabric Manager |
| Scheduling | Slurm with Enroot and Pyxis is the designated scheduler in a SuperPOD; Kubernetes with Run:ai for MLOps-style platforms | Bring-up phase 10, Run:ai |
| Node health | DCGM, NVSM (DGX), nvidia-smi | DCGM, NVSM |
| Communication | CUDA, NCCL, GPUDirect RDMA and Storage | NCCL |
| Applications | NGC containers, NVIDIA AI Enterprise, NeMo, NIM | NGC |
Control plane servers in the designs: the DGX BasePOD RA uses 5 x86 management servers: 2 BCM head nodes in high availability and 3 for services such as Slurm login nodes or Kubernetes control plane. The 4-SU H100 SuperPOD also lists 5 management servers plus 4 UFM appliances.
10. Lifecycle and productivity metrics
| Stage | What happens | Proof it's done |
|---|---|---|
| Design (Day 0) | Pick a reference architecture; size power, cooling, networks, storage. This guide. | Bill of materials, cabling plan, rack layout |
| Build and validate (Day 1) | Install, cable, firmware, provision, then test bottom-up | Every phase's pass criteria met; golden baselines recorded (bring-up guide) |
| Operate (Day 2) | Monitoring, health checks, failed-node repair, firmware and driver updates, capacity changes | Utilization, uptime, and job throughput stay near baseline |
| Produce | Users train models and serve inference | Time-to-train, tokens per second, cost and energy per token |
How a factory's output is measured
- Time-to-train and tokens per second (inference throughput): the factory's output.
- Performance per watt and cost per token: power is often the scarcest input.
- Model FLOPs utilization (MFU): the model FLOPs a training job actually achieves divided by the hardware's peak. Communication stalls, stragglers and input bottlenecks all lower it.
- Goodput: the share of time spent on useful training progress, after subtracting failures, restarts, and work redone from the last checkpoint. This is why Mission Control emphasizes automatic failure detection, moving work off unhealthy nodes, and fast restart from checkpoints.
Validation (DCGM, NCCL tests, HPL, ClusterKit, burn-in) is how you prove a new factory can reach its expected output before users arrive, and the baselines it records are what Day-2 operations compare against.
11. Fabric calculator (DGX H100 SuperPOD pattern)
12. Sample outputs
What the commands in this guide print. Expand a command to see a typical result and what to check in it.
Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.
ibnetdiscover -p | grep dgx03Port-to-leaf map for one node
CA 301 1 0xa088c20300f1e301 4x NDR - SW 11 3 0xfc6a1c0300a1b201 ( 'dgx03 mlx5_0' - 'su1-leaf01' )
CA 302 1 0xa088c20300f1e302 4x NDR - SW 12 3 0xfc6a1c0300a1b202 ( 'dgx03 mlx5_3' - 'su1-leaf02' )
CA 303 1 0xa088c20300f1e303 4x NDR - SW 13 3 0xfc6a1c0300a1b203 ( 'dgx03 mlx5_4' - 'su1-leaf03' )
CA 304 1 0xa088c20300f1e304 4x NDR - SW 14 3 0xfc6a1c0300a1b204 ( 'dgx03 mlx5_5' - 'su1-leaf04' )
CA 305 1 0xa088c20300f1e305 4x NDR - SW 16 3 0xfc6a1c0300a1b206 ( 'dgx03 mlx5_6' - 'su1-leaf06' )
CA 306 1 0xa088c20300f1e306 4x NDR - SW 15 3 0xfc6a1c0300a1b205 ( 'dgx03 mlx5_9' - 'su1-leaf05' )
CA 307 1 0xa088c20300f1e307 4x NDR - SW 17 3 0xfc6a1c0300a1b207 ( 'dgx03 mlx5_10' - 'su1-leaf07' )
CA 308 1 0xa088c20300f1e308 4x NDR - SW 18 3 0xfc6a1c0300a1b208 ( 'dgx03 mlx5_11' - 'su1-leaf08' )What to look at: On a rail-optimized fabric rail i (the NIC next to GPU i) must land on leaf i, so the eight lines should map to su1-leaf01 through su1-leaf08 in order. Here rails 5 and 6 (mlx5_6 and mlx5_9) land on leaf06 and leaf05: the links are up at full NDR speed, but traffic for those GPUs now crosses the spine.
ibdiagnetFabric-wide health and topology check
Loading IBDIAGNET from: /usr/lib/x86_64-linux-gnu/ibdiagnet1.5.7
-W- Topology file is not specified.
Reports regarding cluster links will use direct routes.
-I- Using port 1 as the local port
-I- Discovering ... 280 nodes (24 Switches & 256 CA-s) discovered.
-I- Fabric Discover finished successfully
-I- Discovering ... 280 nodes (24 Switches & 256 CA-s) discovered.
...
-I- Links Check finished successfully
-I- Subnet Manager Check finished successfully
...
Summary
-I- Stage Warnings Errors Comment
-I- Discovery 0 0
-I- Lids Check 0 0
-I- Links Check 0 0
-I- Subnet Manager 0 0
-I- Port Counters 2 0
-I- Nodes Information 0 0
-I- Speed / Width checks 0 0
-I- Virtualization 0 0
-I- Partition Keys 0 0
-I- Temperature Sensing 0 0
-I- You can find detailed errors/warnings in: /var/tmp/ibdiagnet2/ibdiagnet2.logWhat to look at: Check the discovered switch and CA counts against the design (one SU of 32 DGX nodes has 256 compute HCA ports) and look for non-zero Errors. ibdiagnet passing does not prove the cabling is rail-correct; run it with the cabling plan as a topology file, or compare ibnetdiscover output, to catch swapped cables.
13. Design lab: five tickets with an answer key
Each ticket is a design or audit task you might get on Day 0 or Day 1. Fill in your answer and press Check. Answer key shows the worked solution. Your answers aren't saved.
DES-1001: Size the compute fabric for 2 SUs
The customer bought a 2-SU DGX H100 SuperPOD. Procurement needs the compute fabric counts.
2 SUs × 32 = 64 node slots, minus 1 for UFM = 63 nodes × 8 = 504 GPUs. One leaf per rail per SU: 8 × 2 = 16 leaves. Each leaf has 32 uplinks: 16 × 32 = 512 spine-leaf cables. 512 ÷ 64 = 8 spines, and 8 divides 32 evenly (4 links from each leaf to each spine): 8 spines.
DES-1002: Review a 3-SU quote
A partner's quote for a 3-SU DGX H100 SuperPOD lists 24 leaf and 12 spine QM9700 switches. "768 uplinks ÷ 64 ports = 12," the note says. What do you tell them?
16 spines, which is also what NVIDIA's RA table lists for 3 SUs. Port count isn't the only constraint: every leaf connects to every spine with the same number of links so all paths are equal. With 16 spines, each leaf sends 2 links to each. A bonus: 16 is also the 4-SU spine count, so growing to 4 SUs needs no new spines. Compute fabrics in a SuperPOD are always non-blocking, which rules out the last option.
DES-1003: Cabling audit before handover
Multi-node NCCL on SU 1 is about 15% below baseline, but only in jobs that include one particular node. Single-node NCCL is fine everywhere. The port-to-leaf map from ibnetdiscover, for four of the nodes:
On dgx-03, port 5 lands on leaf 6 and port 6 on leaf 5. Every other node has port i on leaf i. The links are up at full speed, so nothing errors out: GPU 5's traffic to GPU 5 on other nodes now goes leaf 6 → spine → leaf 5 instead of one hop through leaf 5. Swap the two cables and rerun multi-node NCCL. Rail checks like this belong in bring-up before any performance testing (rail alignment guide, InfiniBand diagnostics).
DES-1004: Which network carries it?
The firewall team needs a traffic map. Assign each flow to a network.
Compute: only GPU-to-GPU collectives (NCCL). Storage: dataset reads and checkpoint writes to the high-performance filesystem. In-band: everything admins and users do through the OS, including PXE provisioning, SSH, Slurm control traffic, and NFS home directories (user storage is on in-band, not the storage fabric). Out-of-band: anything that talks to a BMC, such as Redfish power control and serial-over-LAN consoles, which must work even with the OS down.
DES-1005: Will it fit in this data hall?
A colocation hall offers air-cooled racks with 22 kW usable each, front-to-back airflow, no liquid cooling. The customer wants one SU of DGX H100 (31 nodes) now and asks about GB200 NVL72 next year.
22 ÷ 10.2 = 2.16, so 2 DGX H100 per rack, which matches NVIDIA's "2 is the practical maximum" for typical air-cooled halls. 31 ÷ 2 = 15.5, so 16 racks for the DGX nodes alone, before switches, management servers, and storage. A GB200 NVL72 is one rack of about 120 kW with direct liquid cooling: power can't be "borrowed" from neighbouring racks, and an NVL72 can't be split across racks because its NVLink domain is cabled inside one rack. The hall needs a liquid-cooling and power upgrade first.
14. Scenarios
15. Exam traps
- "AI factory" is a concept for the full stack, not a product, switch, or software package.
- SuperPOD = turnkey product with a fixed bill of materials and guaranteed performance. BasePOD = flexible reference design. Size defines neither.
- Swapping Slurm for Kubernetes, DGX OS for another OS, certified storage for uncertified, or changing a fabric topology generally makes a SuperPOD a BasePOD.
- H100 SU = 32 nodes / 256 GPUs, 8 leaf switches; a 1-SU build has 31 nodes because of UFM. 4 SUs = 127 nodes.
- GB200 SU = 8 NVL72 racks / 576 GPUs, about 1.2 MW.
- Rail-optimized: port i of every node to leaf i. Not all ports of a node to one leaf.
- Compute fabric = non-blocking. A storage fabric may be slightly oversubscribed.
- NFS home directories ride the in-band network; the high-performance filesystem rides the storage fabric.
- BMCs live on the out-of-band network, which must work when the OS is down.
- BasePOD has 3 networks (management and storage can share Ethernet); SuperPOD H100 has 4 separate fabrics.
- C-G-N-B: 2-8-5-200 is 2 CPUs, 8 GPUs, 5 NICs (1 north-south + 4 east-west), 200 Gb/s east-west per GPU.
- Air-cooled DGX H100 racks: about 2 nodes per rack in practice. GB200 NVL72: about 120 kW per rack, liquid cooled.
- Don't extend InfiniBand cable reach with patch panels or extra top-of-rack switches.
16. Flashcards (click to flip)
17. Practice quiz
Score: 0 / 0