RoCE & Spectrum-X

RDMA over Converged Ethernet from first principles: RoCEv2, GIDs, lossless Ethernet (PFC, ECN, DCQCN), and how to configure and verify it on hosts and switches. Then NVIDIA Spectrum-X, the Ethernet platform built for AI (Spectrum-4 switches + BlueField-3 SuperNICs, adaptive routing, congestion control), and how it compares with InfiniBand.

1. The big picture: RDMA on two kinds of network

GPU clusters move huge amounts of data between nodes. To do that fast, they use RDMA (Remote Direct Memory Access): one machine's network card writes straight into another machine's memory (or GPU memory) without involving either CPU. RDMA can run on two physical network types:

	InfiniBand (IB)	RoCE (RDMA over Converged Ethernet)
Network	A dedicated fabric with its own switches, addressing, and a Subnet Manager	Ethernet: standard Ethernet switches, IP addressing, routing
Lossless by design?	Yes, through credit-based flow control in the link layer	No. Ethernet drops packets when buffers fill, so RoCE needs extra configuration (PFC, ECN) or NIC-side features to behave well.
NVIDIA products	Quantum switches (for example Quantum-2 NDR 400 Gb/s), ConnectX HCAs	Spectrum switches, ConnectX NICs, BlueField SuperNICs / DPUs. Spectrum-X is the AI-tuned combination.
Programming interface	The same RDMA verbs API (`libibverbs`, `mlx5` devices). NCCL, MPI/UCX, and perftest run on both.

InfiniBand is a private railway built only for freight trains: no collisions, strict signalling. RoCE runs the same freight trains on the public road network (Ethernet). It's cheaper and familiar, but you need traffic rules (PFC, ECN, QoS) or a smart traffic-control system (Spectrum-X) to stop traffic jams and dropped cargo.

Spectrum-X isn't a different protocol from RoCE. It's NVIDIA's end-to-end Ethernet platform for AI that runs RoCE, adding adaptive routing, congestion control, and isolation by having the switches and SuperNICs work together. Learn RoCE first, and Spectrum-X then makes sense as "RoCE, fixed for AI traffic."

2. RDMA in five minutes

Term	Meaning
Verbs	The RDMA API (`libibverbs`). Same for IB and RoCE.
QP (Queue Pair)	A send queue + receive queue: the RDMA "connection." NCCL opens one or more QPs per peer.
RC (Reliable Connected)	The QP type NCCL uses. The NIC guarantees in-order delivery and retransmits on loss.
Memory registration	Pinning a buffer so the NIC can DMA it. Needs a large `memlock` limit.
GPUDirect RDMA	The NIC reads and writes GPU memory directly (through `nvidia_peermem` or DMA-BUF), skipping host memory
RDMA write / read / send	One-sided (write, read) or two-sided (send/receive) operations

On a host, a RoCE port looks just like an IB port to software: an mlx5_X device. The difference shows in ibv_devinfo as link_layer: Ethernet (IB shows InfiniBand).

3. RoCEv1 vs RoCEv2

	RoCEv1	RoCEv2 (what everyone uses)
Encapsulation	IB transport directly in an Ethernet frame (EtherType 0x8915)	IB transport inside UDP/IP, destination UDP port 4791
Routable?	No, single Layer-2 domain only	Yes, across IP subnets (Layer 3), so it works in leaf-spine fabrics
Congestion signal	-	Uses IP ECN bits, which enables DCQCN
Load balancing	-	ECMP hashes on the 5-tuple; the UDP source port varies per QP to add entropy

| Ethernet | IP | UDP (dst 4791) | IB BTH | payload (RDMA data) | ICRC | FCS |

"v2 = UDP = routable. Port 4791."

4. GIDs and the GID index

Each RDMA port has a GID table (Global Identifier). On RoCE, the GIDs are derived from the interface's IP addresses, with separate entries for RoCEv1 and RoCEv2. Applications choose an entry by its GID index.

$ show_gids
DEV     PORT  INDEX  GID                                      IPv4         VER  DEV
mlx5_0  1     0      fe80:0000:0000:0000:ba3f:d2ff:fe12:3456               v1   ens1f0np0
mlx5_0  1     1      fe80:0000:0000:0000:ba3f:d2ff:fe12:3456               v2   ens1f0np0
mlx5_0  1     2      0000:0000:0000:0000:0000:ffff:0a00:000b  10.0.0.11    v1   ens1f0np0
mlx5_0  1     3      0000:0000:0000:0000:0000:ffff:0a00:000b  10.0.0.11    v2   ens1f0np0

You want the RoCEv2 entry for the routable IPv4 (or global IPv6) address. In the common layout above, that's index 3, which is why you so often see NCCL_IB_GID_INDEX=3 and ib_write_bw -x 3.

Index 3 is a convention, not a law. The table changes if the interface has extra IPs, IPv6 globals, VLANs, or no IP at all (then there's no IPv4 GID). Always confirm with show_gids. A wrong index, such as a link-local or v1 entry, causes connection failures or traffic that doesn't route across subnets.

Recent NCCL versions can auto-select a suitable RoCEv2 GID, so the variable is often optional. Setting it explicitly is still common in runbooks. rdma_cm-based tools (-R in perftest) pick the GID from the IP address you connect to.

5. Why RoCE wants lossless Ethernet

Ethernet switches drop packets when a queue overflows. TCP handles this gracefully, with selective retransmits and per-flow backoff.
Classic RoCE NICs use go-back-N recovery on RC QPs. One lost packet can force retransmission of everything sent after it, so even a tiny drop rate destroys throughput and adds big tail latency.
AI collectives are synchronous: one slow flow delays the whole all-reduce, and therefore every GPU in the job.

The traditional answer is a lossless traffic class, built from two mechanisms that work together:

Mechanism	What it does	Analogy
PFC (Priority Flow Control)	Emergency brake: a switch tells its upstream neighbour "stop sending priority 3 for a moment" before its buffer overflows	A red light at the motorway on-ramp
ECN + DCQCN	Early warning: switches mark packets when queues grow, and the senders slow down before PFC is needed	A "congestion ahead, slow down" sign

ECN is the primary tool; PFC is the safety net. A well-tuned fabric rarely sends pause frames. Lots of PFC pauses means congestion control isn't keeping up.

Newer NICs (ConnectX-6 Dx and later, BlueField-3) have better loss recovery (selective repeat), and Spectrum-X relies on NIC-side congestion control and reordering. That's why some deployments describe themselves as "lossy" or "PFC-light." For the exam, know the classic model: PFC + ECN/DCQCN on a dedicated RoCE priority.

6. PFC: pausing one traffic class

Standard: IEEE 802.1Qbb. Classic Ethernet PAUSE (802.3x) stops all traffic on a link; PFC pauses just one of 8 priorities.
RoCE traffic is placed on one priority, conventionally priority 3, and only that priority is lossless. Everything else (TCP, storage, management) stays normal and lossy.
The switch needs buffer headroom: data still in flight after the pause is sent. Longer cables and higher speeds need more headroom.

PFC's dark side

Problem	What happens	Mitigation
Head-of-line blocking	A pause stops all flows in that priority on the link, including ones not heading to the congested port	ECN/DCQCN so PFC rarely triggers; adaptive routing
Congestion spreading	Pauses propagate hop by hop back toward the senders (a "pause tree")	Same: early ECN marking
PFC storm	A faulty NIC or host keeps sending pause frames forever and freezes part of the fabric	PFC watchdog on switches: detects stuck queues and drops or ignores pauses
Deadlock	Cyclic buffer dependencies (for example routing loops during failures) where everyone waits on everyone	Loop-free topologies, watchdog

PFC must match end to end: same priority, enabled on every NIC and switch port in the path. One port without PFC on priority 3 becomes a drop point, and RoCE performance collapses for every flow through it.

7. ECN and DCQCN: slowing senders down early

A switch queue passes a threshold, and the switch sets the ECN CE (Congestion Experienced) bits in the IP header of passing RoCE packets.
The receiving NIC (the "notification point," NP) sees CE marks and sends a CNP (Congestion Notification Packet) back to the sender.
The sending NIC (the "reaction point," RP) cuts that QP's rate, then gradually increases it again. That's the DCQCN algorithm (Data Center Quantized Congestion Notification).

Sender NIC (RP) ──data──▶ Switch (CP: marks ECN) ──data+CE──▶ Receiver NIC (NP)
        ▲                                                        │
        └──────────────────── CNP ◀──────────────────────────────┘

Counter (in `/sys/class/infiniband/mlx5_0/ports/1/hw_counters/`)	Meaning
`np_ecn_marked_roce_packets`	Received packets carrying ECN marks (this NIC is a receiver on congested paths)
`np_cnp_sent`	CNPs this NIC sent back to senders
`rp_cnp_handled`	CNPs this NIC received and acted on (it was told to slow down)
`out_of_sequence`, `packet_seq_err`	Signs of packet loss / retransmission. Should be ~0 on a healthy lossless fabric.

Counter names can differ slightly between NIC generations and driver versions. List the directory on your system. Some ECN marking is normal under load; rising sequence errors are not.

"Switch marks, receiver tells (CNP), sender slows." CP → NP → RP.

8. QoS plumbing: DSCP, priorities, traffic classes

To put RoCE on "priority 3," every device must agree on how to classify packets:

Field	Where	Notes
DSCP (6 bits)	IP header	Survives routing, so it's the preferred trust mode for RoCEv2 in L3 fabrics. A common RoCE marking is DSCP 26, with CNPs at DSCP 48.
PCP (3 bits)	802.1Q VLAN tag	Only exists with VLAN tags and is lost at routed hops. "Trust PCP" is the older L2 approach.
Switch priority / TC	Inside the device	DSCP or PCP maps to an internal priority (for example 3), which maps to a queue with PFC/ECN enabled

Traffic class vs DSCP (a classic calculation)

The IP ToS byte = DSCP (upper 6 bits) + ECN (lower 2 bits). Tools such as NCCL_IB_TC and cma_roce_tos take the whole ToS / traffic-class byte, so:

traffic_class = DSCP × 4          # shift left 2 bits
DSCP 26  →  TC 104                # NCCL_IB_TC=104
DSCP 24  →  TC 96

Setting NCCL_IB_TC=26 when you meant DSCP 26 marks packets with DSCP 6, which lands in the wrong (lossy) queue. Multiply by 4.

DSCP 26 / priority 3 and CNP DSCP 48 are common NVIDIA defaults (for example in the Cumulus Linux RoCE profile), but your site may use other values. What matters is that hosts and switches agree.

9. Host configuration and verification

# Which RDMA device is which Ethernet interface? Is it up?
ibdev2netdev                       # mlx5_0 port 1 ==> ens1f0np0 (Up)
ibv_devinfo -d mlx5_0 | grep -E 'state|link_layer|active_mtu'
#   state: PORT_ACTIVE   link_layer: Ethernet   active_mtu: 4096

# GID table
show_gids mlx5_0

# QoS on the NIC (mlnx_qos from MLNX_OFED / DOCA-OFED)
mlnx_qos -i ens1f0np0                          # show current trust / PFC / ETS
mlnx_qos -i ens1f0np0 --trust dscp             # classify by DSCP
mlnx_qos -i ens1f0np0 --pfc 0,0,0,1,0,0,0,0    # PFC on priority 3 only

# Default ToS for rdma_cm connections (DSCP 26 → 104)
cma_roce_tos -d mlx5_0 -t 104

# MTU: Ethernet MTU must fit the RoCE MTU (4096) plus headers → use jumbo frames
ip link set ens1f0np0 mtu 9000

# Counters
ethtool -S ens1f0np0 | grep -Ei 'prio3|pause|discard'
ls /sys/class/infiniband/mlx5_0/ports/1/hw_counters/

Bandwidth test (perftest)

# server (node A)
ib_write_bw -d mlx5_0 -x 3 --report_gbits -F
# client (node B)
ib_write_bw -d mlx5_0 -x 3 --report_gbits -F 10.0.0.11
#  -x 3  = GID index (RoCEv2 IPv4)   or use -R to let rdma_cm choose from the IP
#  Expect close to line rate (for example ~390 Gb/s on a 400 GbE port)

The RoCE MTU (path MTU, typically 4096) is separate from the Ethernet MTU. The Ethernet MTU must be large enough to carry it, so set 9000 (jumbo) end to end on hosts and switches. If one hop is still at 1500, RoCE falls back to a 1024-byte MTU and loses bandwidth.

On DGX systems and NVIDIA reference designs, much of this is preset by DGX OS, the network operator, or BCM. The commands above are how you verify it.

10. Switch side: Cumulus Linux and NetQ

NVIDIA Spectrum switches commonly run Cumulus Linux (configured with the NVUE CLI, nv) or SONiC. Cumulus has a single-command RoCE profile:

nv set qos roce                 # apply the lossless RoCE profile (PFC + ECN + DSCP mapping)
nv config apply
nv show qos roce                # verify: mode, DSCP → priority mapping, PFC/ECN settings
nv show interface swp1 counters # per-port counters (drops, pause frames)

Tool	What it is
Cumulus Linux / NVUE	Switch OS and its CLI/API
NetQ	Fabric-wide telemetry and validation: checks RoCE/QoS config consistency, shows congestion, PFC and ECN counters, and topology across all switches
NVIDIA Air	A cloud-hosted digital twin of the network for testing configurations before deploying them
LLDP	Neighbour discovery: which host port connects to which switch port (the Ethernet counterpart to `ibnetdiscover`)

The exact NVUE syntax (for example lossless vs lossy RoCE modes) depends on your Cumulus release. Check nv set qos roce -h.

11. NCCL over RoCE

NCCL treats RoCE as its IB transport (verbs), so the NCCL_IB_* variables apply:

Variable	RoCE use
`NCCL_IB_HCA`	Which `mlx5_X` devices to use (for example the 8 compute NICs, excluding storage/management NICs)
`NCCL_IB_GID_INDEX`	RoCEv2 GID entry (often 3; confirm with `show_gids`)
`NCCL_IB_TC`	Traffic class = DSCP × 4 (for example 104 for DSCP 26), so NCCL traffic lands in the lossless queue
`NCCL_IB_QPS_PER_CONNECTION`	More QPs per peer → more distinct UDP source ports → better ECMP spreading on non-adaptive fabrics
`NCCL_SOCKET_IFNAME`	Interface for NCCL's TCP bootstrap (not the data path)
`NCCL_NET_GDR_LEVEL`	When to use GPUDirect RDMA based on GPU-NIC distance

NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE ... [7]mlx5_7:1/RoCE
NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA

In NCCL_DEBUG=INFO output, /RoCE after each device confirms Ethernet RDMA (IB shows /IB), and GDRDMA confirms GPUDirect RDMA. Seeing NET/Socket instead means NCCL fell back to TCP, which is very slow.

For the full NCCL variable and debugging guide, see NCCL deep dive. For the "only one NIC used" pattern, see the GPU 0 bottleneck guide.

12. The AI traffic problem: elephant flows and ECMP

Standard Ethernet fabrics balance traffic with ECMP (Equal-Cost Multi-Path): the switch hashes each flow's 5-tuple and sends the whole flow down one of the equal paths. That works well for normal data-centre traffic (millions of small flows), but AI training traffic is different:

Typical cloud traffic	AI training traffic
Many small, random flows	Few, huge "elephant" flows (collectives between GPUs)
High entropy, so hashing spreads them evenly	Low entropy: a few flows hash unevenly, and two elephants can collide on one link while others sit idle
Average throughput matters	Tail latency matters: the slowest flow sets the pace of every all-reduce

With plain ECMP, AI jobs on Ethernet often get far less than the fabric's theoretical bandwidth because of hash collisions, plus congestion that PFC spreads around. Fixing this is the main reason Spectrum-X exists.

13. Spectrum-X: what it is

NVIDIA Spectrum-X is an end-to-end Ethernet networking platform for AI. The key word is end-to-end: the switch and the NIC are designed to cooperate, so it's more than a switch model.

Component	Role
Spectrum-4 switch (for example SN5600: 64 × 800 GbE, 51.2 Tb/s)	The Ethernet switch ASIC. Performs adaptive routing and provides congestion telemetry.
BlueField-3 SuperNIC (up to 400 Gb/s RoCE per GPU)	The host NIC for GPU-to-GPU (east-west) traffic. Reorders sprayed packets, runs Spectrum-X congestion control.
Software: Cumulus Linux or SONiC, NetQ, DOCA, NVIDIA Air	Configuration, telemetry, validation, the NIC programming framework, and the digital twin

Ordinary Ethernet is a motorway where every car picks a lane at the on-ramp and must stay in it. Spectrum-X is a motorway where the road tells each car which lane is empty right now, cars may change lanes at every junction, and the car park at the destination (the SuperNIC) puts the cars back in order before anyone notices.

NVIDIA has since extended the family with 800 Gb/s generations (for example "Spectrum-X800" with ConnectX-8 SuperNICs) and multi-site variants. Product names and speeds evolve, but the architecture (switch + SuperNIC co-design) stays the same. NVIDIA's headline claim is substantially higher effective AI network performance than traditional Ethernet. Treat specific multipliers as marketing figures.

14. Spectrum-X features

RoCE adaptive routing (per-packet load balancing)

Instead of pinning a flow to one hashed path, the Spectrum-4 switch sends each packet down the least-congested available path, based on real-time queue state.
Packets now arrive out of order. The SuperNIC places each one directly at its correct memory location (direct data placement), so the application sees ordered data.
Result: near-full use of all fabric links even with a handful of elephant flows. No hash collisions.

Adaptive routing needs both ends. Packet spraying without a NIC that handles out-of-order arrival would cause retransmissions. That's why Spectrum-X is defined as switch + SuperNIC together, not just a fast switch.

Spectrum-X congestion control

Switches send in-band telemetry about congestion to the SuperNICs, which run programmable congestion control to meter each flow precisely.
The goal is to stop congestion before buffers fill, so the fabric relies far less on PFC and avoids congestion spreading to innocent flows.

Performance isolation (noise isolation)

In multi-tenant AI clouds, one job's burst shouldn't slow a neighbour's job. Adaptive routing plus congestion control keep each workload's performance predictable, close to what it would get on a dedicated fabric.

Problem on standard RoCE	Spectrum-X answer
ECMP hash collisions on elephant flows	Per-packet adaptive routing + SuperNIC reordering
Congestion spreading, PFC pause trees	Telemetry-driven congestion control at the SuperNIC
Noisy neighbours in multi-tenant clusters	Performance isolation
Hard-to-see fabric problems	NetQ telemetry, NVIDIA Air digital twin

Spectrum-X = "AR + CC + Isolation": Adaptive Routing, Congestion Control, performance Isolation. All three need the switch and the SuperNIC together.

15. SuperNIC vs DPU vs ConnectX

	ConnectX NIC (for example ConnectX-7)	BlueField-3 SuperNIC	BlueField-3 DPU
Main job	High-speed RDMA NIC (IB or Ethernet)	East-west GPU traffic in Spectrum-X: adaptive-routing reordering, congestion control	North-south and infrastructure: storage, security, networking offload, isolation from the host OS
Typical count	One per GPU in DGX H100 (as compute NICs)	One per GPU in Spectrum-X compute fabrics	One or two per server
Arm cores / services	No	Yes, but focused on networking; lower power than a full DPU	Yes: runs its own OS and services (DOCA)

SuperNIC and DPU are both BlueField-3, but they're different products for different roles. "Use a DPU for every GPU's compute traffic" or "a SuperNIC is for storage and security offload" are mix-ups.

DGX H100 compute networking uses ConnectX-7 (InfiniBand or Ethernet), plus BlueField-3 DPUs for storage/management. Spectrum-X reference architectures pair GPU servers with BlueField-3 SuperNICs. Check the specific reference architecture you're studying.

16. AI Ethernet fabric design

An AI Ethernet fabric follows the same blueprint as an InfiniBand SuperPOD: separate compute, storage, in-band, and out-of-band networks, and a rail-optimized, non-blocking leaf-spine compute fabric (GPU n on every server to leaf n). Those design rules, plus Enterprise RA node patterns such as 2-8-5-200, are covered in the AI factory architecture guide (networks, topology, Enterprise RAs). What's specific to Ethernet:

Consistent configuration on every port: MTU 9000, the same DSCP → priority mapping, PFC on the same priority, ECN thresholds. Validate it with NetQ.
Don't mix GPU collectives with storage on the same lossless class unless the design says so.
Adaptive routing and SuperNICs instead of per-flow ECMP hashing (Spectrum-X features).

17. InfiniBand vs RoCE vs Spectrum-X

	InfiniBand (Quantum-2)	Standard RoCE	Spectrum-X
Losslessness	Native credit-based flow control	PFC + ECN/DCQCN (needs careful tuning)	Switch telemetry + SuperNIC congestion control (less PFC dependence)
Load balancing	Adaptive routing (with the Subnet Manager's routing engines)	ECMP hashing (flow-based)	Per-packet adaptive routing + NIC reordering
Management	Subnet Manager (OpenSM / UFM), LIDs, GUIDs	Standard IP routing (BGP/OSPF), no SM	Standard IP + NetQ / Cumulus
In-network compute	SHARP (in-switch reductions)	No	No SHARP; relies on bandwidth and congestion handling
Strengths	Lowest latency, most mature for largest HPC/AI training clusters	Familiar Ethernet, cheap, works with existing tooling	Ethernet operations model with near-IB AI performance; multi-tenant AI clouds
Typical home	DGX SuperPOD, HPC	General data centres, smaller GPU clusters	AI clouds and enterprises standardized on Ethernet

The exam framing: InfiniBand = purpose-built, lossless, SHARP, Subnet Manager. Spectrum-X = Ethernet built for AI, using RoCE with adaptive routing and congestion control through switch + SuperNIC co-design. Plain RoCE = RDMA on Ethernet that depends on PFC/ECN tuning and suffers from ECMP collisions on AI traffic.

18. Which tools work on which fabric?

Tool	InfiniBand	RoCE / Spectrum-X	Notes
`ibstat`, `ibv_devinfo`, `ibdev2netdev`	✔	✔	Host-side verbs tools: link layer, state, rate
`ib_write_bw` / `ib_read_lat` (perftest)	✔	✔ (with `-x <gid>` or `-R`)	The RDMA bandwidth/latency test on both
nccl-tests	✔	✔	Look for `/RoCE` in the NCCL log
`sminfo`, `smpquery`, `ibnetdiscover`, `ibdiagnet`, `perfquery`	✔	✘	Need IB management packets (SMPs) and a Subnet Manager, which don't exist on Ethernet
`show_gids`, `mlnx_qos`, `cma_roce_tos`	-	✔	RoCE-specific host configuration
`ethtool -S`, `hw_counters`	Partial	✔	Pause frames, per-priority counters, ECN/CNP counters
LLDP, NetQ, NVUE (`nv show`)	-	✔	Topology, config validation, switch counters
UFM	✔	-	InfiniBand fabric manager

"Run sminfo to find the master Subnet Manager on the RoCE fabric" is a trap. Ethernet has no Subnet Manager. See InfiniBand diagnostics for the IB-only tools.

19. RoCE config checker

Describe a host or port configuration. The checker flags what would break or slow RoCE, and computes the traffic class from your DSCP.

Trust mode
PFC enabled on
ECN marking
Ethernet MTU (all hops)
GID index used
DSCP for RoCE
NCCL_IB_TC set to
Fabric type
NCCL_IB_QPS_PER_CONNECTION

20. "What should I check?" chooser

Pick a symptom above.

21. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

show_gids mlx5_0GID table: index, IP, RoCE version

DEV     PORT  INDEX  GID                                      IPv4         VER  DEV
---     ----  -----  ---                                      ------------ ---  ---
mlx5_0  1     0      fe80:0000:0000:0000:ba3f:d2ff:fe00:0001               v1   ens1f0np0
mlx5_0  1     1      fe80:0000:0000:0000:ba3f:d2ff:fe00:0001               v2   ens1f0np0
mlx5_0  1     2      0000:0000:0000:0000:0000:ffff:0a00:000b  10.0.0.11    v1   ens1f0np0
mlx5_0  1     3      0000:0000:0000:0000:0000:ffff:0a00:000b  10.0.0.11    v2   ens1f0np0
n_gids_found=4

What to look at: Pick the entry that is v2 (RoCEv2, routable UDP 4791) and carries the interface's IPv4 address; that index (often 3) is what NCCL_IB_GID_INDEX and ib_write_bw -x need. If no v2 entry has an IP, the interface has no address yet or is down, and using a v1 or link-local index breaks routed traffic.

ibdev2netdevMap mlx5_X to Ethernet interfaces

mlx5_0 port 1 ==> ens1f0np0 (Up)
mlx5_1 port 1 ==> ens2f0np0 (Up)
mlx5_2 port 1 ==> ens3f0np0 (Up)
mlx5_3 port 1 ==> ens4f0np0 (Up)
mlx5_4 port 1 ==> ens5f0np0 (Down)
mlx5_5 port 1 ==> ens6f0np0 (Up)
...

What to look at: Every RDMA device used by NCCL should map to an interface that is (Up). A (Down) entry like mlx5_4 here means that rail is unusable until the link or switch port is fixed.

ibv_devinfo -d mlx5_0Verbs view: state, MTU, link layer

hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.1002
        node_guid:                      b83f:d203:0000:0001
        sys_image_guid:                 b83f:d203:0000:0001
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000999
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

What to look at: link_layer: Ethernet means this RDMA port is RoCE, so sm_lid and port_lid are 0 by design (there is no Subnet Manager). active_mtu should be 4096; 1024 means some hop still has a 1500-byte Ethernet MTU, and PORT_DOWN means the Ethernet link itself is down.

mlnx_qos -i ens1f0np0NIC trust, PFC and ETS settings

DCBX mode: OS controlled
Priority trust state: dscp
dscp2prio mapping:
        prio:0 dscp:07,06,05,04,03,02,01,00,
        prio:1 dscp:15,14,13,12,11,10,09,08,
        prio:2 dscp:23,22,21,20,19,18,17,16,
        prio:3 dscp:31,30,29,28,27,26,25,24,
        prio:4 dscp:39,38,37,36,35,34,33,32,
        prio:5 dscp:47,46,45,44,43,42,41,40,
        prio:6 dscp:55,54,53,52,51,50,49,48,
        prio:7 dscp:63,62,61,60,59,58,57,56,
default priority:
Receive buffer size (bytes): 130944,130944,0,0,0,0,0,0,
Cable len: 7
PFC configuration:
        priority    0   1   2   3   4   5   6   7
        enabled     0   0   0   1   0   0   0   0
        buffer      0   0   0   1   0   0   0   0
tc: 0 ratelimit: unlimited, tsa: vendor
         priority:  1
tc: 1 ratelimit: unlimited, tsa: vendor
         priority:  0
tc: 2 ratelimit: unlimited, tsa: vendor
         priority:  2
tc: 3 ratelimit: unlimited, tsa: vendor
         priority:  3
...

What to look at: For a typical RoCE setup you want 'Priority trust state: dscp', DSCP 26 landing in prio 3, and PFC enabled only on priority 3. Trust state pcp when the switch marks by DSCP, PFC enabled on no priority (or all of them), or a mismatch with the switch's PFC priority all lead to drops or pause storms.

mlnx_qos -i ens1f0np0 --pfc 0,0,0,1,0,0,0,0Enable PFC on priority 3 only

DCBX mode: OS controlled
Priority trust state: dscp
...
PFC configuration:
        priority    0   1   2   3   4   5   6   7
        enabled     0   0   0   1   0   0   0   0
        buffer      0   0   0   1   0   0   0   0
...

What to look at: mlnx_qos prints the resulting configuration after the change, so confirm the 'enabled' row has a 1 only under priority 3. The setting must match the switch (nv show qos roce pfc-priority), and it is not persistent across reboots unless your provisioning reapplies it.

cma_roce_tos -d mlx5_0 -t 104; cma_roce_tos -d mlx5_0Default ToS for rdma_cm connections

What to look at: The set command is silent; reading it back should print 104, which is DSCP 26 shifted left two bits (26 x 4). A value of 0 means rdma_cm traffic leaves unmarked and lands in the default lossy class instead of the PFC-protected queue.

ip link set ens1f0np0 mtu 9000; ip link show ens1f0np0Set and verify jumbo MTU

6: ens1f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether b8:3f:d2:00:00:01 brd ff:ff:ff:ff:ff:ff

What to look at: Check 'mtu 9000' and 'state UP'. The jumbo MTU must be set end to end on hosts and switch ports; any hop left at 1500 forces a 1024-byte RoCE MTU and costs bandwidth.

ethtool -S ens1f0np0 | grep -Ei 'prio3|pause|discard'Per-priority, pause and discard counters

     rx_discards_phy: 0
     tx_discards_phy: 0
     rx_prio3_bytes: 918273645512
     rx_prio3_packets: 214477003
     rx_prio3_discards: 0
     tx_prio3_bytes: 920115733120
     tx_prio3_packets: 214908871
     rx_prio3_pause: 18422
     rx_prio3_pause_duration: 52031
     tx_prio3_pause: 4107
     tx_prio3_pause_duration: 11893
     rx_prio3_pause_transition: 9211
     tx_pause_storm_warning_events: 0
     tx_pause_storm_error_events: 0

What to look at: RDMA traffic should show up in the prio3 byte counters, and a moderate number of prio3 pause frames under load is normal for a lossless class. Nonzero rx_discards_phy or prio3 discards mean drops in the lossless class (misconfigured PFC or buffers), and pause storm events mean a stuck receiver is flooding pauses.

ls /sys/class/infiniband/mlx5_0/ports/1/hw_counters/List RoCE hardware counters

duplicate_request          np_cnp_sent                 resp_local_length_error
implied_nak_seq_err        np_ecn_marked_roce_packets  resp_remote_access_errors
lifespan                   out_of_buffer               rnr_nak_retry_err
local_ack_timeout_err      out_of_sequence             roce_adp_retrans
packet_seq_err             req_cqe_error               roce_adp_retrans_to
req_cqe_flush_error        req_remote_access_errors    roce_slow_restart
req_remote_invalid_request resp_cqe_error              roce_slow_restart_cnps
resp_cqe_flush_error       rp_cnp_handled              rp_cnp_ignored
rx_atomic_requests         rx_read_requests            rx_write_requests

What to look at: These per-port files are the RoCE congestion and loss counters; the exam-relevant ones are np_ecn_marked_roce_packets, np_cnp_sent, rp_cnp_handled, out_of_sequence and packet_seq_err. If the directory is missing, the driver or kernel does not expose hardware counters for this device.

cd /sys/class/infiniband/mlx5_0/ports/1/hw_counters && grep . np_ecn_marked_roce_packets np_cnp_sent rp_cnp_handled out_of_sequence packet_seq_err

Read ECN/CNP and loss counters

np_ecn_marked_roce_packets:1488210
np_cnp_sent:1487904
rp_cnp_handled:1502337
out_of_sequence:0
packet_seq_err:0

What to look at: ECN marks with matching CNPs sent and handled mean DCQCN is working and senders are slowing down as designed. out_of_sequence or packet_seq_err climbing means packets are being lost and retransmitted, usually PFC not protecting the RoCE priority; all CNP counters at 0 under heavy congestion means ECN is not enabled on the switch.

ib_write_bw -d mlx5_0 -x 3 --report_gbits -F 10.0.0.11RoCE RDMA write bandwidth (client)

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0108 PSN 0x5a1b2c RKey 0x1fff00 VAddr 0x007f2a4c000000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:12
 remote address: LID 0000 QPN 0x0109 PSN 0x3c4d5e RKey 0x1fff01 VAddr 0x007f9b10000000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:11
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      5000             391.42             391.18             0.746117
---------------------------------------------------------------------------------------

What to look at: Start 'ib_write_bw -d mlx5_0 -x 3 --report_gbits -F' on the server (10.0.0.11) first. Link type Ethernet, GID index 3, Mtu 4096 and BW average near 390 Gb/s on a 400 GbE port is a pass; Mtu 1024 or much lower bandwidth points at MTU, PFC or a degraded link, and a hang at connection usually means the wrong GID index.

nv set qos roce; nv config applyApply the Cumulus RoCE QoS profile

Warning: The following files will be overwritten:
  /etc/cumulus/datapath/qos/qos_features.conf
  /etc/mlx/datapath/qos/qos_infra.conf
Are you sure? [y/N] y
applied [rev_id: 14]

What to look at: 'applied [rev_id: N]' confirms NVUE committed the change; nothing takes effect until 'nv config apply'. An error during apply, or answering N, leaves the previous QoS configuration running.

nv show qos roceVerify switch RoCE QoS settings

                    operational  applied   description
------------------  -----------  --------  ------------------------------------------------
enable                           on        Turn the feature 'on' or 'off'.
mode                lossless     lossless  Roce Mode
cable-length        100          100       Cable Length(in meters) for Roce Lossless Config
congestion-control
  congestion-mode   ECN                    Congestion config mode
  enabled-tc        0,3                    Congestion config enabled Traffic Class
  max-threshold     1.43 MB                Congestion config max-threshold
  min-threshold     146.48 KB              Congestion config min-threshold
  probability       100
lldp-app-tlv
  priority          3                      switch-priority of roce
  protocol-id       4791                   L4 port number
  selector          UDP                    L4 protocol
pfc
  pfc-priority      3                      switch-prio on which PFC is enabled
  rx-enabled        enabled                PFC Rx Enabled status
  tx-enabled        enabled                PFC Tx Enabled status
trust
  trust-mode        pcp,dscp               Trust Setting on the port for packet classification

RoCE PCP/DSCP->SP mapping configurations
===========================================
    pcp  dscp                     switch-prio
--  ---  -----------------------  -----------
0   0    0,1,2,3,4,5,6,7          0
...
3   3    24,25,26,27,28,29,30,31  3
...

What to look at: Check mode lossless, ECN congestion control on TC 3, PFC on priority 3 with rx and tx enabled, and DSCP 26 mapped to switch priority 3. A PFC priority that differs from the hosts' mlnx_qos setting, or ECN missing, explains drops, retransmits or CNP counters stuck at 0.

nv show interface swp1 countersSwitch port drops and pause counters

                     operational
-------------------  -----------
carrier-transitions  2
in-bytes             918.27 GB
in-drops             0
in-errors            0
in-pkts              214477122
out-bytes            920.12 GB
out-drops            0
out-errors           0
out-pkts             214908990
in-pfc-frames        4107
out-pfc-frames       18422
...

What to look at: Compare with the host side: drops and errors should be 0, and carrier-transitions should not keep increasing (that would mean a flapping link). Large or climbing out-drops on the RoCE port mean the lossless class is not protected, and PFC frame counts far higher than other ports point at a congested or slow receiver.

NCCL_DEBUG=INFO NCCL_IB_GID_INDEX=3 NCCL_IB_TC=104 mpirun -np 16 -H dgx01:8,dgx02:8 ./all_reduce_perf -b 8 -e 8G -f 2 -g 1

Confirm NCCL uses RoCE with GPUDirect

dgx01:2211:2211 [0] NCCL INFO Bootstrap : Using enp170s0f0np0:192.168.50.11<0>
dgx01:2211:2211 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [4]mlx5_4:1/RoCE [5]mlx5_5:1/RoCE [6]mlx5_6:1/RoCE [7]mlx5_7:1/RoCE [RO]; OOB enp170s0f0np0:192.168.50.11<0>
dgx01:2211:2211 [0] NCCL INFO Using network IB
dgx01:2211:2290 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA
dgx01:2211:2290 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA
...
#       size         count      type   redop    root     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)
   536870912     134217728     float     sum      -1   2913.4  184.27  345.51      0
  1073741824     268435456     float     sum      -1   5788.1  185.51  347.83      0
  8589934592    2147483648     float     sum      -1    45902  187.14  350.88      0
# Avg bus bandwidth    : 112.37

What to look at: /RoCE after each mlx5 device confirms RDMA over Ethernet (InfiniBand shows /IB), GDRDMA confirms GPUDirect RDMA, and all eight rails should be listed. 'NET/Socket' instead of NET/IB means NCCL fell back to TCP, and a missing GDRDMA tag or low large-message busbw points at nvidia-peermem, GID index or QoS settings.

22. Scenarios

23. Searchable reference

Item	Meaning

24. Flashcards (click to flip)

25. Exam traps

Spectrum-X runs RoCE. It's an Ethernet platform, not a new protocol and not InfiniBand.
Spectrum-X = Spectrum-4 switches + BlueField-3 SuperNICs (+ software). The switch alone isn't Spectrum-X.
Adaptive routing sprays packets; the SuperNIC reorders them. Without it, out-of-order packets hurt.
RoCEv2 = UDP port 4791, routable. RoCEv1 is Layer-2 only.
The GID index is often 3, but confirm with show_gids.
PFC = 802.1Qbb, per-priority pause (RoCE usually on priority 3). Plain PAUSE (802.3x) stops everything.
ECN/DCQCN is the main congestion tool; PFC is the safety net. Lots of pause frames = trouble.
CNPs flow from receiver to sender. The switch only marks.
NCCL_IB_TC = DSCP × 4 (DSCP 26 → 104).
Trust DSCP (not PCP) in routed RoCEv2 fabrics.
MTU 9000 end to end. One 1500-byte hop shrinks RoCE's MTU.
sminfo / ibnetdiscover / ibdiagnet are InfiniBand-only. Use LLDP / NetQ on Ethernet.
SHARP in-network reductions are an InfiniBand feature.
SuperNIC ≠ DPU: SuperNIC for GPU east-west traffic, DPU for infrastructure / north-south offload.
NCCL uses its IB transport for RoCE (NET/IB with /RoCE devices). NET/Socket means a TCP fallback.

26. Practice quiz (24 questions)

Score: 0 / 0