InfiniBand Fabric & Diagnostics
How an IB fabric works (Subnet Manager, LIDs, GUIDs, port states), and exactly what ibstat, sminfo, smpquery, ibnetdiscover, ibsim, perfquery, ibdiagnet and friends do, with the scenarios the exam asks about.
1. InfiniBand in plain words
InfiniBand (IB) is a high-bandwidth, low-latency network built for HPC and AI. In a DGX H100 cluster, each node's 8 compute ConnectX-7 adapters (one per GPU, 400 Gb/s NDR each) connect to Quantum-2 IB switches. That's the compute fabric NCCL uses between nodes.
| Term | Meaning |
|---|---|
| HCA (Host Channel Adapter) | The IB network card in a server (ConnectX-7). Shows as mlx5_0, mlx5_1… |
| Switch | Forwards packets using forwarding tables the SM programs (Quantum-2 for NDR) |
| Subnet | All HCAs and switches managed by one (master) Subnet Manager |
| Subnet Manager (SM) | The "brain": discovers the fabric, assigns addresses, computes routes, programs switches |
| RDMA | Remote Direct Memory Access: one machine reads/writes another's memory without involving its CPU. The basis of IB performance and GPUDirect RDMA. |
sminfo, ibnetdiscover, and ibdiagnet don't apply. See RoCE & Spectrum-X.1b. The layers: from cable to NCCL
The InfiniBand spec defines a layered stack, like TCP/IP. It doesn't number its layers, but they line up with the OSI model's layers 1 to 4, with software above, so the table shows the OSI numbers as a guide. Each layer has its own job, header, and tools, so knowing which layer a symptom lives in tells you which tool to reach for.
| OSI layer | IB layer | Job | Key ideas | Check it with |
|---|---|---|---|---|
| 5–7 | Upper layers (software) | What applications use | The verbs API (libibverbs, rdma-core); registered memory with keys; consumers like NCCL, MPI/UCX, IPoIB, NVMe-oF, GPUDirect RDMA | nccl-tests, NCCL_DEBUG=INFO ("NET/IB"), ibv_devinfo |
| 4 | Transport | End-to-end delivery between two queue pairs (QPs), in the HCA's hardware | BTH header: destination QP, packet sequence number, opcode (Send, RDMA Write, RDMA Read). Service types: RC (reliable connected, what NCCL uses: acks and retransmits in hardware), UC, UD. QP0/QP1 carry management (section 6). | ib_write_bw, ib_read_lat (perftest) |
| 3 | Network | Routing between subnets | GRH header with 128-bit GIDs. Optional inside one subnet, which is the usual cluster case. | ibv_devinfo -v (GID table) |
| 2 | Link | Hop-by-hop delivery inside one subnet, lossless | LRH header with source/destination LIDs; switches forward by LID using tables the SM programs. Credit-based flow control: a port only sends when the next hop has buffer space, so IB doesn't drop packets for congestion. Virtual lanes (VLs) share one cable; VL15 is reserved for SM traffic. Logical port state: Init → Armed → Active. | ibstat (State, Base lid), perfquery (PortXmitWait = waiting for credits), ibtracert |
| 1 | Physical | Bits on the wire | Lanes and width (4x), signaling rate (NDR: 100 Gb/s per lane, 400 Gb/s per 4x port), cables and optics, FEC and BER. Physical state: Polling → LinkUp. | mlxlink, ibstat (Rate, Physical state), ibdiagnet (section 5b) |
One IB packet carries a header per layer, in order:
| LRH (link: LIDs, VL) | GRH (network: GIDs, optional) | BTH (transport: QP, PSN) | payload | ICRC | VCRC |The two checksums map to the layers too: the ICRC (invariant) covers the fields that never change and is checked end to end; the VCRC (variant) is recomputed on every hop because the link header changes there.
mlxlink: link up, right width and speed, BER OK) → link (ibstat: State Active with a LID, so the SM did its job) → transport (ib_write_bw between two nodes) → application (nccl-tests). A failure at one layer makes every layer above it fail too, so start at the bottom.2. Identities: GUID, LID, GID
| ID | Size | Assigned by | Persistent? | Used for |
|---|---|---|---|---|
| GUID (node GUID, port GUID, system image GUID) | 64-bit | Burned in by the manufacturer | Yes (hardware identity) | Uniquely identifying a device/port, like a MAC address |
| LID (Local Identifier) | 16-bit | The Subnet Manager | Usually kept, but can change | Routing inside the subnet. Most diag tools take a LID as the target. |
| GID | 128-bit | Subnet prefix + port GUID | Yes | Global addressing (like IPv6), routing across subnets |
Base lid: 0 (or 65535, the unassigned value 0xffff, on mlx5 adapters) means you haven't been given an address, so there's no working SM.3. The Subnet Manager: master, standby, priority
What the SM does
- Sweeps (discovers) the fabric using direct-routed management packets.
- Assigns LIDs to every HCA port and switch.
- Computes routes (routing engines such as min-hop, up/down, ftree for fat trees, adaptive routing on Quantum switches).
- Programs switch forwarding tables (LFTs) and moves ports to Active.
- Runs the Subnet Administrator (SA): a database that apps query for path records (how to reach LID X).
- Handles partitions (PKeys), QoS, and re-sweeps when links change (traps from switches).
Where it runs
- OpenSM on a host (the open-source SM,
opensmservice; config/etc/opensm/opensm.conf, log/var/log/opensm.log). - NVIDIA UFM (Unified Fabric Manager) on a dedicated server or appliance. Uses an SM engine underneath, adds monitoring, a GUI, and telemetry. The usual choice for DGX SuperPOD/BasePOD.
- An embedded SM on a managed switch (fine for small fabrics).
Master election: several SMs, one boss
For redundancy you run more than one SM. They negotiate, and exactly one becomes MASTER while the others are STANDBY. Standbys poll the master, and if it disappears, a standby takes over (handover / failover).
| Rule | Detail |
|---|---|
| 1. Highest priority wins | SM priority is 0–15, and 15 is highest. Set in OpenSM with sm_priority in opensm.conf or opensm -p 15. |
| 2. Tie-break: lowest GUID wins | If priorities are equal, the SM with the lowest port GUID becomes master. |
| SM states | 0 NOTACTIVE, 1 DISCOVERING, 2 STANDBY, 3 MASTER (shown by sminfo) |
sm_priority only when it starts, so restart each SM after editing it, the standby first, and only while the other SM is running. A running master hands over to a higher-priority SM it discovers. Confirm with sminfo on both LIDs. Practise it in lab ticket CHG-2350.4. Port states and physical states
ibstat shows two separate things for each port. Learn both columns.
| Physical state (the wire) | Meaning |
|---|---|
| LinkUp | Electrical/optical link established with the partner. Cable OK. |
| Polling | Trying to find a partner: no cable, bad cable, far end down, or a mismatch |
| Disabled | Port administratively disabled (for example with ibportstate disable) |
| Sleep / LinkErrorRecovery | Low power / recovering from errors |
| Logical port state (the SM's view) | Meaning |
|---|---|
| Down | No physical link |
| Initializing (Init) | The link is physically up, but the SM hasn't configured it (no SM, SM can't reach it, or the SM hasn't swept yet) |
| Armed | Configured, about to go active (brief) |
| Active | Fully configured. Can carry traffic. |
5. Speeds and widths
| Name | Per lane (approx) | 4x port | Typical |
|---|---|---|---|
| EDR | 25 Gb/s | 100 Gb/s | ConnectX-5 |
| HDR | 50 Gb/s | 200 Gb/s | ConnectX-6, DGX A100 |
| NDR | 100 Gb/s | 400 Gb/s | ConnectX-7, DGX H100, Quantum-2 |
| XDR | 200 Gb/s | 800 Gb/s | Next generation (ConnectX-8, Quantum-X800) |
ibstat showing Rate: 200 on an NDR fabric → the link came up degraded (fewer lanes or lower speed): a bad cable/transceiver, dirty optics, or a port problem. Check with iblinkinfo, mlxlink, and ibdiagnet. It's not a software tuning issue.5b. Physical layer: 400G cables, BER, and FEC
Most "slow or flaky link" problems are physical: a cable, an optic, a connector, or a port. This section covers what connects the ports, how link quality is measured (BER), and how to read mlxlink to find a bad cable.
Cable and optic types
| Type | What it is | Reach at 400G (rough) | Notes |
|---|---|---|---|
| DAC (Direct Attach Copper), passive | A copper cable with the connectors molded on each end. No electronics in the signal path. | Only a few meters | Cheapest, lowest power, lowest latency. Used inside a rack (server to top-of-rack switch, or switch to switch). Thick, heavy, and has a minimum bend radius. |
| ACC (Active Copper Cable) | Copper with small signal-boosting chips in the connectors | A bit longer than a DAC | Bridges the gap between DAC and optics; thinner cable. |
| AOC (Active Optical Cable) | Fiber with the optics built permanently into both ends | Tens of meters | Fixed length; no separate transceivers to clean. |
| Transceivers + fiber | Pluggable optical modules (multimode for shorter runs, single-mode for longer) plus separate fiber patch cables | Tens of meters (multimode) to hundreds of meters or more (single-mode) | Needed between rows and to spine switches. Connectors must be clean; dirt is a classic cause of errors. |
NDR ConnectX-7 and Quantum-2 ports use the OSFP form factor (some adapters use QSFP112). Switch-side "twin-port" OSFP cages carry two 400G links, so splitter cables (one 800G end to two 400G ends) are common. Always use cables rated for the port speed: an older HDR-rated cable in an NDR port typically links at 200, not 400.
For form factors (OSFP, QSFP112, flat-top vs finned-top), fiber and connectors, the eye diagram, module diagnostics (DDM), and transceiver firmware, see the transceivers, cables & signal quality guide.
Why 400G links always have errors, and why that's OK
NDR runs each lane at about 100 Gb/s using PAM4 signaling: four voltage levels per symbol instead of two. That doubles the data per symbol but shrinks the gap between levels, so noise flips bits far more easily. At these speeds some bit errors are normal. The link fixes them with FEC (Forward Error Correction, Reed-Solomon codes): the sender adds redundant data, and the receiver uses it to repair corrupted bits on the fly, without retransmitting.
The BER metrics
BER (Bit Error Rate) = errored bits ÷ total bits received. It's written in scientific notation: 1E-12 means one bad bit per trillion. Smaller is better. Because of FEC there are two BERs to read:
| Metric (mlxlink naming) | Measured | What it tells you | Healthy |
|---|---|---|---|
| Raw Physical BER (also per-lane raw errors) | Before FEC | How noisy the physical channel (cable, optic, connector, port) really is | Low and stable, similar on every lane, and well below the FEC's correction limit |
| Effective Physical BER / Effective Physical Errors | After FEC | Errors the FEC couldn't fix. These become corrupted packets that are dropped and resent. | Zero (or essentially zero) |
| Symbol errors / Symbol BER | After FEC, at the symbol level | Similar to effective errors; reported on newer firmware | Zero |
| Link Down Counter / Link Error Recovery Counter | Events | The link dropped, or had to retrain | Not increasing |
Interactive: what a BER means at line rate
Notice how even a "small" raw BER means many errors per second at 400 Gb/s. That's why FEC is mandatory, and why the post-FEC number is the one that must be zero.
Reading mlxlink
mlxlink is part of MFT (NVIDIA Firmware Tools). Start the tools with mst start; device names come from mst status, or you can use the mlx5_X RDMA name.
mlxlink -d mlx5_0 # link summary: state, speed, width, FEC mode in use
mlxlink -d mlx5_0 -m # module / cable info: type (copper or optical), length, vendor, part number;
# for optics also temperature, TX/RX optical power per lane
mlxlink -d mlx5_0 -c # counters: raw BER, effective BER, per-lane raw errors, link down / recovery
mlxlink -d mlx5_0 --pc # clear the counters (start a fresh measurement window)
mlxlink -d mlx5_0 -e # eye opening per lane (signal quality margin)Counters only mean something over a time window and under traffic. Clear them, run load (for example ib_write_bw on that link, or an nccl-tests run), then read them again. mlxlink can also query switch ports remotely, and recent ibdiagnet versions can collect physical-layer and BER data for every port in the fabric in one run.
-m shows their type, length, and part number, but there's no optical power or temperature to read. Those diagnostics exist only for active cables and optics.Troubleshooting flow: is it the cable, the optic, or the port?
- Find suspects:
ibdiagnet/ibqueryerrors/ UFM alerts, links at the wrong rate iniblinkinfo, or a slow rail in nccl-tests. - Check the cable is right:
mlxlink -mshows the type and part number. Is it rated for 400G, and within its length limit? - Measure:
mlxlink --pc, run traffic, thenmlxlink -c. Look for effective errors, high raw BER, or one bad lane. For optics, compare RX power per lane. - Cheapest fixes first: reseat both ends (a DAC that isn't fully latched is common). For optics, inspect and clean the fiber connectors. Check the DAC isn't kinked or bent past its bend radius.
- Isolate: move the cable to a known-good port, or put a known-good cable in the suspect port. If the errors follow the cable, replace the cable. If they stay with the port, suspect the port, the HCA, or the switch (and its firmware).
- Confirm: clear counters, re-test under load, check the rate is back to 400 with zero effective errors, then return the node to service. While waiting for a part, drain the node or disable the port (
ibportstate ... disable) so jobs don't run on a flaky link.
Active at Rate: 400 can still be bad. Speed and state only prove the link trained. Errors under load (effective BER, symbol errors, rising link-down or recovery counters) are what reveal a marginal cable. And the fix for a high-BER link is physical (reseat, clean, replace), never an NCCL setting.400G Ethernet (RoCE / Spectrum-X) uses the same PAM4 + FEC physics and the same cable families. mlxlink works on ConnectX Ethernet ports too, and the Linux tools are ethtool -m <if> (module info), ethtool --show-fec <if> (FEC mode), and ethtool -S <if> (look for the FEC and PHY error counters). See the RoCE & Spectrum-X guide.
6. How the tools talk: SMPs, SA, PMA
IB management uses special packets called MADs (Management Datagrams). Knowing the three kinds explains which tool needs what:
| Kind | Goes to | Tools |
|---|---|---|
| SMP (Subnet Management Packet), on QP0 / VL15 | Directly to a device's management agent. Works even with direct routing (hop by hop) before LIDs exist. | smpquery, sminfo, ibnetdiscover, ibroute, ibportstate, the SM itself |
| SA query (GMP on QP1) | The master SM's Subnet Administrator database | saquery (and every RDMA app asking for path records) |
| PMA (Performance Management Agent, GMP) | Port counters on any device | perfquery, ibqueryerrors |
infiniband-diags package (part of rdma-core / MLNX_OFED / DOCA-OFED) and generally need root (access to /dev/infiniband/umad*). Use -C mlx5_X -P 1 to choose which local HCA/port sends the query.7. The toolbox: every command explained
| Tool | Scope | What it does | Typical use |
|---|---|---|---|
ibstat | Local HCAs only | Shows each local CA and port: State, Physical state, Rate, Base lid, SM lid, Port GUID, Link layer | "Is my port up, at what speed, and does it see an SM?" |
ibstatus | Local | Similar summary (reads sysfs) | Quick port check |
ibv_devinfo | Local | Verbs-level device info (from rdma-core) | "Can RDMA verbs see the device?" |
ibdev2netdev | Local | Maps mlx5_X ↔ network interface (ib0, ibp...) | NCCL_IB_HCA / IPoIB mapping |
sminfo | Fabric (one SM) | Queries the SMInfo attribute: SM LID, GUID, priority, state (MASTER/STANDBY), activity count. With no target, it asks the master SM the local port knows. | "Who is the master SM? Is it alive? What priority?" |
smpquery | Any node by LID/GUID/direct route | Sends an SMP to read one attribute: nodeinfo, nodedesc, portinfo, switchinfo, pkeys, sl2vl, vlarb, guids | "Tell me details about LID X", for example its hostname (nodedesc) or port speed (portinfo) |
ibnetdiscover | Whole fabric | Walks the fabric and prints the topology: every switch, HCA, and link (with GUIDs, LIDs, ports, node descriptions) | "Map the fabric / verify cabling matches the design" |
ibhosts / ibswitches | Whole fabric | List just the HCAs / switches (built on discovery) | Inventory |
iblinkinfo | Whole fabric | Every link with its width/speed and state, switch port by switch port | Find links at the wrong speed or width |
perfquery | One port | Reads port counters: SymbolErrorCounter, LinkDownedCounter, PortRcvErrors, PortXmitDiscards, data/packet counts (-x extended, -R reset) | "Is this link taking errors?" |
ibqueryerrors | Whole fabric | Scans error counters across all ports and reports those above thresholds | Fabric-wide error sweep |
ibroute | One switch | Dumps a switch's forwarding table (LFT) | Routing debug |
ibtracert | Path | Traces the hop-by-hop path between two LIDs | "Which switches does traffic from A to B cross?" |
ibping | Two nodes | IB-level ping (needs ibping -S on the target) | Basic reachability |
saquery | SA (master SM) | Queries the SA database: nodes, path records, multicast groups | "What does the SM think exists?" |
ibportstate | One port | Query or change a port: enable/disable/reset, speed/width | Bounce a flapping port, disable a bad link |
ibaddr | Local / target | Shows the LID and GID of a port | Address lookup |
ibdiagnet | Whole fabric | NVIDIA's comprehensive fabric diagnostic: discovery, duplicate GUIDs/LIDs, link speed/width, error counters, routing checks, cable info, report files | Bring-up acceptance and full fabric health |
mlxlink | One port (MFT) | Physical-layer detail: eye/BER, cable/transceiver info, FEC | Diagnose a bad cable or optic |
flint / mlxfwmanager | Local (MFT) | Query/burn HCA firmware (see the MFT guide) | Firmware consistency |
ib_write_bw etc. (perftest) | Two nodes | RDMA bandwidth/latency benchmarks | Prove a link reaches line rate |
ibsim | Simulated fabric | An InfiniBand fabric simulator: emulates a fabric from a topology file so management software (OpenSM, diags) can be tested without real hardware | Development/testing. Not used to diagnose a real fabric. |
opensm | Fabric | The Subnet Manager daemon itself | Run/configure an SM |
8. The confusing four (plus ibsim)
ibstat | sminfo | smpquery | ibnetdiscover | ibsim | |
|---|---|---|---|---|---|
| Looks at | Your own HCA ports | The Subnet Manager | One attribute of one node | The whole fabric's topology | A fake fabric |
| Needs the fabric? | No (local) | Yes | Yes | Yes | No (simulated) |
| Answers | "Is my port Active, at 400, with a LID, and which LID is the SM?" | "Who is master SM? GUID, LID, priority, state?" | "What's the name / port info / node info of LID X?" | "What's connected to what?" | "Can I test SM software without hardware?" |
| Master SM? | Gives the SM lid only (no state/priority/name) | Yes: the authoritative answer | Turns the SM's LID into a hostname (nodedesc) | Shows nodes but doesn't say which runs the master SM | No |
sminfo gives its LID/GUID/priority/state, then smpquery nodedesc <LID> tells you which machine it is. That pairing is the answer pattern: sminfo + smpquery.9. Workflow: find the master SM
Step 1: what does my port see?
$ ibstat mlx5_0
CA 'mlx5_0'
CA type: MT4129
Number of ports: 1
Firmware version: 28.39.1002
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 57
LMC: 0
SM lid: 1
Port GUID: 0xb83fd20300a1b2c4
Link layer: InfiniBandPort is Active at 400 and has LID 57. The SM it's using lives at LID 1. That's a pointer, but not proof of state or priority.
Step 2: ask the SM itself
$ sminfo
sminfo: sm lid 1 sm guid 0x0002c90300ab1234, activity count 4183027 priority 15 state 3 SMINFO_MASTER- sm lid 1 / sm guid 0x…1234: the master's address and hardware ID
- priority 15: highest possible
- state 3 SMINFO_MASTER: confirmed master
- activity count: increases while the SM is alive. Run
sminfotwice: if the count stops changing, the SM is hung.
Step 3: which machine is LID 1?
$ smpquery nodedesc 1
Node Description:.........ufm-01 HCA-1
$ smpquery nodeinfo 1 # node type, GUIDs, port count
$ smpquery portinfo 1 1 # LID, state, speed/width of port 1 on LID 1The master SM runs on ufm-01.
Step 4: find the standby SMs
$ saquery -s # SA query: ports advertising the IsSM capability (i.e. running an SM) → their LIDs
$ sminfo 12 # query the SM at LID 12 directly
sminfo: sm lid 12 sm guid 0x0002c90300ab5678, activity count 900311 priority 14 state 2 SMINFO_STANDBY
$ smpquery nodedesc 12
Node Description:.........ufm-02 HCA-1saquery options differ a bit between infiniband-diags versions (check saquery --help). The universal method: get candidate LIDs (from your design, UFM, or discovery), run sminfo <lid> on each, and look for STANDBY vs MASTER.Step 5 (optional): change who's master
# OpenSM: /etc/opensm/opensm.conf
sm_priority 15 # on the intended master
sm_priority 14 # on the standby
# or start with: opensm -p 15
# UFM: set SM priority / HA in the UFM configurationibstat (SM lid) → sminfo (confirm MASTER, GUID, priority, alive) → smpquery nodedesc (hostname).Practice this on a live mock fabric in the hands-on lab below: start with ticket SM-1001.
9b. Hands-on lab: six simulated fabric tickets
A mock fabric you work like the real thing: three Quantum-2 switches (leaf01, leaf02, spine01), a head node, two OpenSM servers (sm01, sm02), and four DGX nodes (two compute HCAs each, to keep output short). Pick a ticket, start with cat notes.txt, and type help for the commands. State carries through, as on a real fabric:
- If the master SM dies, existing ports stay Active. The standby takes over after a few polls.
- A rebooted or retrained port sits in Init until an SM sweeps it.
- OpenSM reads
opensm.confonly when it starts. - The SM with the highest priority wins the election. A tie goes to the lower GUID.
- SM-1001: find the master SM, prove it's alive, and find the standby (guided; start here).
- INC-2214: after a power maintenance every port is stuck in Init, and nothing answers
sminfo. - INC-2240: training jobs lost their partition and all-reduce got slower after a patch night.
- CHG-2301: a planned failover to patch the master, which isn't safe as written.
- INC-2333: a rail running at half width, and an uplink that keeps flapping. You fix them with the DC tech (
dc reseat|clean|replace). - CHG-2350: move the master SM to sm02 for good by changing priorities, without ever leaving the fabric without an SM. Watch for the tie.
grep, head, tail, wc, sort, and uniq; > redirects; ; and &&; pdsh -w dgx[01-04] "cmd".Answer key for this ticket (spoilers)
One clean path. Other orders work too: the checklist tracks results, not exact commands.
10. More workflows
A. Bring-up: are all ports healthy?
ibstat # on every node: all 8 compute ports Active, LinkUp, Rate 400
iblinkinfo # fabric-wide: any link not 4X NDR?
ibdiagnet # full report: errors, speeds, duplicate GUIDs/LIDs, routing
ibqueryerrors # counters above thresholdB. Verify cabling matches the design
ibnetdiscover > fabric.topo # current topology
ibhosts; ibswitches # inventory: expected number of HCAs and switches?
# compare against the planned topology (ibdiagnet can also compare to a reference topology)C. Investigate one suspicious link
perfquery <lid> <port> # SymbolErrorCounter, LinkDownedCounter, PortRcvErrors...
perfquery -x <lid> <port> # extended 64-bit data counters
perfquery -R <lid> <port> # reset counters, then watch whether they grow
smpquery portinfo <lid> <port> # speed/width actually negotiated
mlxlink -d mlx5_0 -m # physical layer, cable/module info (MFT)
ibportstate <lid> <port> reset # bounce the port (careful in production)D. Trace a path between two nodes
ibtracert 57 112 # hop-by-hop path from LID 57 to LID 112
ibroute <switch-lid> # that switch's forwarding table11. Port-state decoder
12. "Which tool?" chooser
13. Sample outputs
What the commands in this guide print. Expand a command to see a typical result and what to check in it.
mlxlink -d mlx5_0Physical link summary and recommendation
Operational Info
----------------
State : Active
Physical state : LinkUp
Speed : IB-NDR
Width : 4x
FEC : Standard_RS-FEC - (544,514)
Loopback Mode : No Loopback
Auto Negotiation : ON
Supported Info
--------------
Enabled Link Speed : 0x00000080 (NDR)
Supported Cable Speed : 0x000000fe (NDR,HDR,EDR,FDR,FDR10,QDR,SDR)
Troubleshooting Info
--------------------
Status Opcode : 0
Group Opcode : N/A
Recommendation : No issue was observed
Tool Information
----------------
Firmware Version : 28.39.1002
amBER Version : 3.2
MFT Version : mft 4.26.1-3What to look at: Healthy NDR is State Active, Physical state LinkUp, Speed IB-NDR, Width 4x and an RS-FEC mode. Width 2x or 1x, a lower speed such as IB-HDR, Physical state Polling, or a Recommendation other than 'No issue was observed' points at the cable, optic or port.
mlxlink -d mlx5_0 -mModule/cable identity and optical power
Module Info
-----------
Identifier : OSFP
Compliance : 400GBASE-SR4
Cable Technology : 850 nm VCSEL
Cable Type : Optical Module (separated)
OUI : Nvidia
Vendor Name : NVIDIA
Vendor Part Number : MMA4Z00-NS400
Vendor Serial Number : MT2301X00001
Rev : A6
Wavelength [nm] : 850
Transfer Distance [m] : 50
FW Version : 47.171.0
Digital Diagnostic Monitoring : Yes
Power Class : 8.5 W max
Temperature [C] : 51 [-5..75]
Voltage [mV] : 3285 [3135..3465]
Bias Current [mA] : 7.412,7.388,7.450,7.401 [5.5..10]
Rx Power Current [dBm] : 0.912,1.104,0.807,1.011 [-7.6..4]
Tx Power Current [dBm] : 1.203,1.287,1.115,1.198 [-5.6..4]
Module State : Ready state
Rx LOS [per lane] : 0,0,0,0
Tx Fault [per lane] : 0,0,0,0What to look at: Check the module type (copper DAC vs optical), part number and firmware against the recipe, and compare the per-lane Rx power values. A lane reading around -40 dBm (no light) or far below its neighbours means a dirty or damaged fiber or a failing far-end laser, and passive DACs show no power or temperature at all.
mlxlink -d mlx5_0 -cRaw and effective BER counters
Physical Counters and BER Info
------------------------------
Time Since Last Clear [Min] : 412.3
Effective Physical Errors : 0
Effective Physical BER : 15E-255
Raw Physical Errors Per Lane : 498210,476931,521877,503466
Raw Physical BER : 2E-10
Link Down Counter : 0
Link Error Recovery Counter : 0
Symbol Errors : 0
Symbol BER : 15E-255What to look at: Effective Physical Errors must be 0 (15E-255 is how mlxlink prints 'no post-FEC errors'), and the raw errors should be similar on every lane. Nonzero effective errors, a climbing Link Down Counter, or one lane with far more raw errors than the others means a marginal cable or optic; clear with --pc and re-read under traffic before judging.
ibstat mlx5_0Local HCA port state, rate, LIDs
CA 'mlx5_0'
CA type: MT4129
Number of ports: 1
Firmware version: 28.39.1002
Hardware version: 0
Node GUID: 0xb83fd20300a1b2c4
System image GUID: 0xb83fd20300a1b2c4
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 57
LMC: 0
SM lid: 1
Capability mask: 0xa751e848
Port GUID: 0xb83fd20300a1b2c4
Link layer: InfiniBandWhat to look at: You want State Active, Physical state LinkUp, Rate 400 and a nonzero Base lid and SM lid. State Initializing with LinkUp means the cable is fine but no Subnet Manager has configured the port, Physical state Polling means no signal from the far end, and Rate 200 on an NDR fabric means a degraded link.
sminfoIdentify the master Subnet Manager
sminfo: sm lid 1 sm guid 0x0002c90300ab1234, activity count 4183027 priority 15 state 3 SMINFO_MASTERWhat to look at: State 3 SMINFO_MASTER at the expected priority confirms the master SM, and the activity count should increase if you run it again a few seconds later. A timeout ('_do_madrpc: recv failed') or an activity count that never changes means the SM is dead or unreachable.
smpquery nodedesc 1Hostname behind a LID
Node Description:.........ufm-01 HCA-1What to look at: Use it right after sminfo to turn the SM's LID into a machine name. If the description is an unexpected host (for example a DGX node instead of the UFM or SM server), a rogue SM has taken over the fabric.
smpquery portinfo 57 1Negotiated width/speed of one port
# Port info: Lid 57 port 1
Mkey:............................<not displayed>
GidPrefix:.......................0xfe80000000000000
Lid:.............................57
SMLid:...........................1
CapMask:.........................0xa751e848
LocalPort:.......................1
LinkWidthEnabled:................1X or 4X
LinkWidthSupported:..............1X or 4X
LinkWidthActive:.................4X
LinkState:.......................Active
PhysLinkState:...................LinkUp
LinkDownDefState:................Polling
LMC:.............................0
NeighborMTU:.....................4096
OperVLs:.........................VL0-7
...
LinkSpeedExtActive:..............106.25 Gbps
LinkSpeedExtSupported:...........14.0625 Gbps or 25.78125 Gbps or 53.125 Gbps or 106.25 Gbps
LinkSpeedExtEnabled:.............14.0625 Gbps or 25.78125 Gbps or 53.125 Gbps or 106.25 GbpsWhat to look at: For NDR the meaningful fields are LinkWidthActive 4X and LinkSpeedExtActive 106.25 Gbps per lane; the legacy LinkSpeedActive field is not the NDR speed. LinkWidthActive 2X or 1X, or a lower LinkSpeedExtActive, confirms a degraded link on that specific port.
saquery -sList every port running an SM
PortInfoRecord dump:
EndPortLid...............1
PortNum..................1
CapabilityMask...........0xa751e84a
LinkState................Active
PortInfoRecord dump:
EndPortLid...............12
PortNum..................1
CapabilityMask...........0xa751e84a
LinkState................ActiveWhat to look at: Each record is a port advertising the IsSM capability, so you should see exactly your intended master and standby (here LIDs 1 and 12); check each with sminfo <lid> to see MASTER vs STANDBY. An extra LID here means an unplanned SM is running, and a single record means there is no standby.
ibstatusQuick sysfs port summary, all HCAs
Infiniband device 'mlx5_0' port 1 status:
default gid: fe80:0000:0000:0000:b83f:d203:00a1:b310
base lid: 0x3b
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 400 Gb/sec (4X NDR)
link_layer: InfiniBand
Infiniband device 'mlx5_1' port 1 status:
default gid: fe80:0000:0000:0000:b83f:d203:00a1:b311
base lid: 0x3c
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 200 Gb/sec (2X NDR)
link_layer: InfiniBand
...What to look at: Every compute port should read 4: ACTIVE, 5: LinkUp and 400 Gb/sec (4X NDR); LIDs here are hex (0x3b = 59). The mlx5_1 entry above (run on dgx02) shows the classic degraded link: 200 Gb/sec (2X NDR) means two lanes failed to train, which is a cable or optic problem, not tuning.
ibv_devinfoVerbs-level view of the HCA
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 28.39.1002
node_guid: b83f:d203:00a1:b2c4
sys_image_guid: b83f:d203:00a1:b2c4
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000999
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid: 57
port_lmc: 0x00
link_layer: InfiniBand
...What to look at: This proves libibverbs (what NCCL and perftest use) can see and open the device: look for PORT_ACTIVE, a nonzero port_lid and link_layer InfiniBand. 'No IB devices found' or PORT_DOWN / PORT_INIT means RDMA applications will fail on this HCA even if the driver is loaded.
ibdev2netdevMap mlx5_X devices to netdevs
mlx5_0 port 1 ==> ibp24s0 (Up)
mlx5_1 port 1 ==> ibp41s0 (Up)
mlx5_2 port 1 ==> ibp64s0 (Up)
mlx5_3 port 1 ==> ibp79s0 (Up)
mlx5_4 port 1 ==> ibp94s0 (Up)
mlx5_5 port 1 ==> ibp154s0 (Up)
mlx5_6 port 1 ==> ibp192s0 (Down)
mlx5_7 port 1 ==> ibp206s0 (Up)
...What to look at: Use it to match RDMA device names (for NCCL_IB_HCA) to the IPoIB interface names. Any compute rail showing (Down), like mlx5_6 here, will be missing from NCCL and should be checked with ibstat and mlxlink.
ibnetdiscoverDiscover and print fabric topology
#
# Topology file: generated on Tue Nov 10 10:14:07 2026
#
# Initiated from node b83fd20300a1b2c4 port b83fd20300a1b2c4
vendid=0x2c9
devid=0xd2f2
sysimgguid=0xfc6a1c0300e5a100
switchguid=0xfc6a1c0300e5a100(fc6a1c0300e5a100)
Switch 64 "S-fc6a1c0300e5a100" # "leaf01" enhanced port 0 lid 2 lmc 0
[1] "H-0002c90300ab1234"[1](2c90300ab1234) # "ufm-01 HCA-1" lid 1 4xNDR
[3] "H-b83fd20300a1b2c4"[1](b83fd20300a1b2c4) # "dgx01 mlx5_0" lid 57 4xNDR
[4] "H-b83fd20300a1b310"[1](b83fd20300a1b310) # "dgx02 mlx5_0" lid 59 4xNDR
[33] "S-fc6a1c0300e5b100"[1] # "spine01" lid 4 4xNDR
[34] "S-fc6a1c0300e5b100"[2] # "spine01" lid 4 4xNDR
...
vendid=0x2c9
devid=0x1021
sysimgguid=0xb83fd20300a1b2c4
caguid=0xb83fd20300a1b2c4
Ca 1 "H-b83fd20300a1b2c4" # "dgx01 mlx5_0"
[1](b83fd20300a1b2c4) "S-fc6a1c0300e5a100"[3] # lid 57 lmc 0 "leaf01" lid 2 4xNDR
...What to look at: Each switch block lists which device sits on each port, so compare it with the cabling plan: the right node description on the right port number, and 4xNDR on every link. A host on an unexpected port, a missing node, or a link printed as 2xNDR is a cabling or physical-layer finding.
ibhosts; ibswitchesInventory HCAs and switches
Ca : 0xa088c20300000008 ports 1 "dgx08 mlx5_0"
...
Ca : 0xb83fd20300a1b2c4 ports 1 "dgx01 mlx5_0"
Ca : 0x0002c90300ab5678 ports 1 "ufm-02 HCA-1"
Ca : 0x0002c90300ab1234 ports 1 "ufm-01 HCA-1"
Switch : 0xfc6a1c0300e5b100 ports 64 "spine01" enhanced port 0 lid 4 lmc 0
Switch : 0xfc6a1c0300e5a200 ports 64 "leaf02" enhanced port 0 lid 3 lmc 0
Switch : 0xfc6a1c0300e5a100 ports 64 "leaf01" enhanced port 0 lid 2 lmc 0What to look at: Count the lines and compare with the design (for example 8 DGX nodes x 8 compute HCAs plus SM servers, and the planned number of QM9700 leaves and spines). Fewer entries than planned means nodes or switches that are down, uncabled or on an isolated part of the fabric.
iblinkinfoWidth/speed/state of every link
Switch: 0xfc6a1c0300e5a100 leaf01:
2 1[ ] ==( 4X 106.25 Gbps Active/ LinkUp)==> 1 1[ ] "ufm-01 HCA-1" ( )
2 3[ ] ==( 4X 106.25 Gbps Active/ LinkUp)==> 57 1[ ] "dgx01 mlx5_0" ( )
2 4[ ] ==( 4X 106.25 Gbps Active/ LinkUp)==> 59 1[ ] "dgx02 mlx5_0" ( )
2 5[ ] ==( Down/ Polling)==> [ ] "" ( )
2 33[ ] ==( 4X 106.25 Gbps Active/ LinkUp)==> 4 1[ ] "spine01" ( )
...
Switch: 0xfc6a1c0300e5a200 leaf02:
3 3[ ] ==( 4X 106.25 Gbps Active/ LinkUp)==> 58 1[ ] "dgx01 mlx5_1" ( )
3 4[ ] ==( 2X 106.25 Gbps Active/ LinkUp)==> 60 1[ ] "dgx02 mlx5_1" ( Could be 4X 106.25 Gbps)
3 34[ ] ==( 4X 106.25 Gbps Active/ LinkUp)==> 4 4[ ] "spine01" ( )
...What to look at: Every cabled port should show 4X 106.25 Gbps Active/LinkUp. 'Could be 4X' marks a link that trained narrower than both ends support (a degraded cable), Down/Polling means no partner signal, and Active is replaced by Init when the SM has not configured the link.
perfquery 57 1Error and traffic counters, one port
# Port counters: Lid 57 port 1 (CapMask: 0x5A00)
PortSelect:......................1
CounterSelect:...................0x0000
SymbolErrorCounter:..............0
LinkErrorRecoveryCounter:........0
LinkDownedCounter:...............0
PortRcvErrors:...................0
PortRcvRemotePhysicalErrors:.....0
PortRcvSwitchRelayErrors:........0
PortXmitDiscards:................0
PortXmitConstraintErrors:........0
PortRcvConstraintErrors:.........0
CounterSelect2:..................0x00
LocalLinkIntegrityErrors:........0
ExcessiveBufferOverrunErrors:....0
QP1Dropped:......................0
VL15Dropped:.....................0
PortXmitData:....................4294967295
PortRcvData:.....................4294967295
PortXmitPkts:....................1873301922
PortRcvPkts:.....................1869045117
PortXmitWait:....................21844What to look at: SymbolErrorCounter, LinkErrorRecoveryCounter, LinkDownedCounter and PortRcvErrors should stay at 0 under traffic; values that grow between reads mean a physical-layer problem on this link. The 32-bit data counters pegged at 4294967295 are normal on a busy port, so use -x for real byte counts.
perfquery -x 57 1Extended 64-bit port counters
# Port extended counters: Lid 57 port 1 (CapMask: 0x5A00)
PortSelect:......................1
CounterSelect:...................0x0000
PortXmitData:....................1923847561024
PortRcvData:.....................1918003392768
PortXmitPkts:....................7493205688
PortRcvPkts:.....................7476180468
PortUnicastXmitPkts:.............7493205688
PortUnicastRcvPkts:..............7476180468
PortMulticastXmitPkts:...........0
PortMulticastRcvPkts:............0
SymbolErrorCounter:..............0
LinkErrorRecoveryCounter:........0
LinkDownedCounter:...............0
PortRcvErrors:...................0
PortRcvRemotePhysicalErrors:.....0
PortXmitDiscards:................0
PortXmitWait:....................21844What to look at: PortXmitData and PortRcvData are counted in 4-byte words, so multiply by 4 for bytes; they should keep rising during a test. Error counters climbing here (or after a perfquery -R reset) point at the link, while a large, growing PortXmitWait with no errors points at congestion rather than a bad cable.
ibqueryerrorsFabric-wide error counter sweep
Errors for "leaf02"
GUID 0xfc6a1c0300e5a200 port 4: [SymbolErrorCounter == 214] [LinkErrorRecoveryCounter == 3]
GUID 0xfc6a1c0300e5a200 port 34: [SymbolErrorCounter == 1287] [LinkErrorRecoveryCounter == 14] [LinkDownedCounter == 6]
Errors for "spine01"
GUID 0xfc6a1c0300e5b100 port 4: [SymbolErrorCounter == 1192] [LinkErrorRecoveryCounter == 14] [LinkDownedCounter == 6]
## Summary: 74 nodes checked, 2 bad nodes found
## 212 ports checked, 3 ports have errors beyond threshold
##
## Thresholds:
## [SymbolErrorCounter = 10] [LinkErrorRecoveryCounter = 10] [LinkDownedCounter = 10]
## [PortRcvErrors = 10] [PortRcvRemotePhysicalErrors = 100] [PortXmitDiscards = 100]
## ...What to look at: A clean fabric prints only the Summary with 0 bad nodes. Matching errors on both ends of one link (leaf02 port 34 and spine01 port 4 here) with a rising LinkDownedCounter identify a flapping cable, and PortXmitDiscards alone usually means congestion rather than cabling.
ibtracert 57 112Hop-by-hop path between two LIDs
From ca {0xb83fd20300a1b2c4} portnum 1 lid 57-57 "dgx01 mlx5_0"
[1] -> switch port {0xfc6a1c0300e5a100}[3] lid 2-2 "leaf01"
[33] -> switch port {0xfc6a1c0300e5b100}[1] lid 4-4 "spine01"
[7] -> switch port {0xfc6a1c0300e5a400}[33] lid 6-6 "leaf04"
[10] -> ca port {0xa088c20300000008}[1] lid 112-112 "dgx08 mlx5_0"
To ca {0xa088c20300000008} portnum 1 lid 112-112 "dgx08 mlx5_0"What to look at: Read it to see which leaf and spine ports traffic between the two nodes actually crosses, then check those ports with perfquery. 'no route to dlid' or a path that stops part way means missing forwarding entries, usually an SM or routing problem or a down uplink.
ibdiagnetFull fabric health report
Loading IBDIAGNET from: /usr/lib/x86_64-linux-gnu/ibdiagnet2.1.1
-I- Using port 1 as the local port
Discovery
-I- Discovering ... 74 nodes (10 Switches & 64 CA-s) discovered.
-I- Fabric Discover finished successfully
-I- Duplicated GUIDs detection finished successfully
-I- Duplicated Node Description detection finished successfully
Lids Check
-I- Lids Check finished successfully
Subnet Manager
-I- SM Info retrieving finished successfully
-I- Subnet Manager Check finished successfully
Port Counters
-E- lid=0x0003 dev=54002 leaf02/P34 Performance Monitor counter: symbol_error_counter Value: 1287 (threshold=1)
-I- ... 4 more, see /var/tmp/ibdiagnet2/ibdiagnet2.log
-I- Ports counters value Check finished with errors
Speed / Width checks
-W- Link width 2x is below the supported 4x: leaf02 port 4 <-> dgx02 mlx5_1
-I- Speed / Width checks finished with warnings
Summary
-I- Stage Warnings Errors Comment
-I- Discovery 0 0
-I- Lids Check 0 0
-I- Links Check 0 0
-I- Subnet Manager 0 0
-I- Port Counters 0 5
-I- Nodes Information 0 0
-I- Speed / Width checks 1 0
-I- You can find detailed errors/warnings in: /var/tmp/ibdiagnet2/ibdiagnet2.log
-I- Database : <db_csv> /var/tmp/ibdiagnet2/ibdiagnet2.db_csv
-I- LST : /var/tmp/ibdiagnet2/ibdiagnet2.lstWhat to look at: Read the Summary table first: a sign-off run should show 0 errors and 0 warnings in every stage, then open /var/tmp/ibdiagnet2/ibdiagnet2.log for details. Duplicate GUID or LID errors, a missing master SM, port counter errors, or width/speed warnings like the 2x link above each need fixing before acceptance.
ib_write_bw -d mlx5_0 --report_gbits dgx02RDMA write bandwidth between two nodes
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : ON
TX depth : 128
CQ Moderation : 1
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x39 QPN 0x0128 PSN 0x4f1a2b RKey 0x1fff00 VAddr 0x007f3a5c000000
remote address: LID 0x3b QPN 0x0129 PSN 0x7c3d1e RKey 0x1fff01 VAddr 0x007f12a8000000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 392.61 392.37 0.748386
---------------------------------------------------------------------------------------What to look at: Run 'ib_write_bw -d mlx5_0 --report_gbits' on dgx02 first as the server, then this on the client. On a 400G NDR link BW average should be close to line rate (about 390 Gb/s); roughly half of that points at a degraded 2x link or a PCIe/NUMA problem, and a connection error means the remote server is not running or the port is not Active.
14. Scenarios
15. Searchable command reference
| Command | Purpose |
|---|
16. Flashcards (click to flip)
17. Exam traps
- "Identify the master SM" →
sminfo(state MASTER, LID, GUID, priority) +smpquery nodedesc <LID>for the hostname. ibstatis local only. It shows your port state and the SM's LID, not the SM's state, priority, or name.ibnetdiscovermaps topology. Great for cabling, but it doesn't identify which SM is master.ibsimis a simulator for testing software without hardware. Never the answer for diagnosing a real fabric.- Init + LinkUp = no SM. Down/Polling = cable/partner problem. Rate 200 on NDR = degraded link.
- Master election: highest priority (0–15, 15 highest) wins. Tie → lowest GUID.
- LIDs are assigned by the SM. GUIDs are burned in.
Base lid 0(or 65535) = not configured by an SM. - Port error counters →
perfquery(one port) /ibqueryerrors(fabric). Full health report →ibdiagnet. - Link speed/width across the fabric →
iblinkinfo(or ibdiagnet). - Fabric Manager ≠ Subnet Manager. FM = NVLink/NVSwitch inside a node. SM = InfiniBand.
- Active at Rate 400 doesn't prove a good link. Check errors under load with
mlxlink -c: effective (post-FEC) errors must be zero; raw BER is the early warning. - Passive DAC = short in-rack copper; longer runs need active copper, AOCs, or transceivers + fiber. A cable not rated for NDR links at 200.
- Changing a port (disable/enable/reset) →
ibportstate, not smpquery (smpquery only reads).