Transceivers, Cables & Signal Quality

OSFP and QSFP form factors, copper vs optics, connectors and fiber types, PAM4 and the eye diagram, link training, FEC and BER, optical diagnostics (DDM), transceiver firmware, and how to prove a cable is good with mlxlink, ibdiagnet, and UFM. Written for the NCP-AII blueprint objectives on cable signal quality, cabling, and transceiver firmware.

1. The big picture and where this sits in the exam

Every byte NCCL moves between nodes crosses a port, a module or connector, a cable, and another port. At 400 Gb/s per port, the margin for error in that physical path is tiny. A dirty fiber end, a half-latched copper cable, or a module on old firmware still lets the link come up, but it then corrupts or retransmits data, flaps, or trains at a lower speed. The whole job runs at the pace of that one link.

Blueprint objective	What it means here	Sections
4.4 Validate cables by verifying signal quality	Eye opening, raw and effective BER, symbol errors, optical power, under traffic	Eye, FEC/BER, mlxlink, DDM
4.5 Confirm cabling is correct	Right cable type and speed, right ports, rails, splitters	Media, Splitters, Fabric checks, Cable Validation Tool
4.8 Confirm firmware on transceivers	Module firmware versions match the qualified set	Firmware
1.2 / 1.8 Topologies and cabling during install	Choosing media for reach, form factor, cooling	Form factors, Handling

These objectives sit in Domain 4 (Cluster Test and Verification, 33% of the exam), the largest domain. Domain 2 (Physical Layer Management, 5%) is about BlueField and MIG, not cables, despite its name. See the blueprint section of the bring-up guide.

A link is a phone call on a bad line. "Link up" only means the call connected. Signal quality is whether you can actually understand each word. The eye diagram is a picture of how clearly each word comes through, and BER counts how many words you mishear.

2. Lanes, SerDes, and speed generations

A 400G port is not one 400G wire. It is several parallel lanes, each driven by a SerDes (serializer/deserializer) on the NIC or switch ASIC. Port speed = lanes × per-lane rate.

Generation	Port speed	Lanes × rate	Signaling	Typical form factor
InfiniBand EDR	100 Gb/s	4 × 25G	NRZ	QSFP28
InfiniBand HDR	200 Gb/s	4 × 50G	PAM4	QSFP56
HDR100 (split HDR)	100 Gb/s	2 × 50G	PAM4	Half of a QSFP56 via splitter
InfiniBand NDR	400 Gb/s	4 × 100G	PAM4	OSFP (twin-port on switches), or QSFP112 on some adapters
NDR200 (split NDR)	200 Gb/s	2 × 100G	PAM4	Via splitter
InfiniBand XDR	800 Gb/s	4 × 200G	PAM4	OSFP
400GbE / 800GbE (Spectrum-X)	400 / 800 Gb/s	4 × 100G / 8 × 100G (older 400GbE: 8 × 50G)	PAM4	OSFP, QSFP112, QSFP-DD

NRZ (non-return-to-zero) sends 1 bit per symbol using 2 voltage levels. PAM4 (4-level pulse amplitude modulation) sends 2 bits per symbol using 4 levels, so it doubles the data at the same symbol rate (baud). A 100G PAM4 lane runs at roughly 53 GBd; the extra bits beyond 100G are FEC overhead.

Width matters as much as speed. An NDR port that trains at 4X width and 100G per lane gives 400. If one lane fails, the port may come up at a reduced width (for example 2X or 1X) or a lower speed and show a lower Rate. A reduced rate is a physical-layer symptom first.

Q is Quad (4 lanes). O is Octal (8 lanes). QSFP = 4 lanes; OSFP = 8 lanes. An NDR port needs 4 lanes, so one 8-lane OSFP cage can hold two NDR ports.

3. Form factors: QSFP, OSFP, twin-port, flat-top vs finned

Form factor	Electrical lanes	Where you see it	Notes
QSFP28 / QSFP56	4	EDR / HDR adapters and switches, 100/200GbE	Same mechanical shape; the number is the per-lane class
QSFP112	4 × 100G	Some 400G adapters (for example PCIe ConnectX-7 cards)	One 400G port per cage
QSFP-DD	8 (double-density)	Ethernet switches from other vendors	QSFP width, extra row of contacts
OSFP	8	NDR/XDR InfiniBand switches, ConnectX-7 OSFP adapters, DGX H100 compute ports, Spectrum-4 Ethernet	Slightly larger than QSFP; designed for higher power and better cooling

Twin-port OSFP

A twin-port OSFP module or cable uses all 8 lanes of one OSFP cage as two independent 400G ports. A Quantum-2 NDR switch has 32 OSFP cages that provide 64 NDR ports. On a DGX H100, the 8 compute ConnectX-7 ports come out through 4 OSFP cages, two ports per cage (see the DGX H100 specs guide).

Counting cages is not counting ports. "4 OSFP cages" on a DGX H100 means 8 × 400G compute ports. "32 cages" on a Quantum-2 switch means 64 NDR ports. Each twin-port optical module also has two fiber connectors, one per 400G port.

Flat-top vs finned-top

OSFP modules can draw well over 10 W, so cooling decides the shape:

Finned-top modules carry their own heat-sink fins. They go into cages with no heat sink, typically on air-cooled switches.
Flat-top modules have a flat lid. They go into cages that have a riding heat sink pressing on the module, typically on adapters (HCAs) and DGX systems.

A finned-top module physically won't fit a cage with a riding heat sink, and a flat-top module in a cage with no heat sink can overheat. "The module doesn't fit" or "the module runs hot and the link flaps" can be a wrong-variant problem. Order parts from the vendor's compatibility list for each end of the link.

Exact part numbers, power classes, and which variant goes where change between product generations. Use the NVIDIA LinkX documentation and the system's cabling guide as the source of truth rather than memorizing part numbers.

4. Cable and module types: DAC, ACC, AOC, transceivers

Media	What it is	Reach at 400G (rough)	Diagnostics available	Typical use
DAC (passive copper)	Copper twinax with connectors molded on; no signal electronics	About 1–3 m	ID only (EEPROM): type, length, part number. No optical power or temperature	Inside a rack: node to leaf, switch to switch
ACC (active copper)	Copper with signal-conditioning chips in the ends	A few meters more than DAC	Some module diagnostics	Adjacent racks; thinner than DAC
AOC (active optical)	Fiber with optics permanently attached at both ends	Tens of meters	Module diagnostics; no connectors to clean	Within a row
Multimode transceiver (for example SR4 / 2×SR4)	Pluggable module + multimode fiber (MMF, OM3/OM4)	Roughly 30–50 m	Full DDM: temperature, voltage, TX bias, TX/RX power per lane	Rows, leaf to spine in a hall
Single-mode transceiver (for example DR4 / FR4)	Pluggable module + single-mode fiber (SMF)	About 100–500 m (DR4), about 2 km (FR4)	Full DDM	Long runs, spine to core, between halls

Choose by reach first: copper inside the rack, optics beyond it. Then make sure both ends match: same speed class, compatible module types at each end (an SR module talks to an SR module over multimode fiber; a DR module talks to a DR module over single-mode), and the right form factor and top style for each cage.

An older HDR-rated (200G) cable plugged into an NDR port typically links at a lower rate, not 400. If a link reports half its expected rate, check mlxlink -m for the cable's part number and rating before you suspect firmware or NCCL.

5. Fiber, connectors, and polarity

Item	What to know
Multimode fiber (MMF)	Wider core, cheaper short-reach optics (850 nm VCSELs). OM4 is the common grade for 400G SR. Jackets are often aqua or violet.
Single-mode fiber (SMF)	Narrow core, longer reach (1310 nm). Jackets are often yellow.
MPO / MTP	Multi-fiber push-on connector (8, 12, or 16 fibers in a row). Parallel optics (SR4, DR4) use one fiber pair per lane, so a 4-lane module needs 8 active fibers in one MPO.
LC duplex	Two-fiber connector. Used by wavelength-multiplexed modules (for example FR4), which put all lanes on one fiber pair.
UPC vs APC	UPC = flat polished end; APC = 8° angled end (lower back-reflection), common on single-mode parallel optics. APC connectors are commonly green. Never mate APC with UPC: the angle mismatch causes high loss and can damage the end faces.
Polarity	Each lane's transmit fiber must land on the far end's receive fiber. Pre-terminated trunk cables and cassettes use defined polarity types. A polarity mistake gives no light on the receive side (link stays down) even with perfectly clean fiber.

Aqua or violet is multimode and short. Yellow is single-mode and long. A green connector is angled (APC), so don't mate it with a flat one.

6. Splitters and breakouts

A splitter (breakout) cable has one end that carries more lanes than each of the other ends. Common NDR examples:

Twin-port 800G OSFP (switch) → 2 × 400G: one switch cage feeds two 400G adapter ports.
Twin-port 800G OSFP → 4 × 200G (NDR200): one switch cage feeds four 200G ports.
The optical version is two (or more) fibers from a twin-port module, each going to a separate single-port module.

When a split cable is used, the switch port must also be configured for the split mode (in the switch OS or UFM). With the wrong setting, the sub-ports don't come up or come up at the wrong speed. Also confirm that each leg goes to the planned node and rail: legs of a splitter are easy to swap.

7. The eye diagram

An eye diagram overlays thousands of short slices of the received signal, each exactly one or two symbol periods (UI, unit intervals) long, on top of each other. Clean transitions draw a clear opening in the middle that looks like an eye. The receiver samples each symbol in the center of that opening and decides which level it was.

Feature	What it tells you
Eye height	Vertical opening: how far apart the levels stay after noise and loss. Small height = noise or loss is eating the voltage margin.
Eye width	Horizontal opening: how much timing jitter the receiver tolerates. Narrow width = jitter or slow transitions.
Thick, fuzzy lines	Noise and crosstalk
Crossing points spread out	Jitter
Lines that depend on the previous bits	Inter-symbol interference (ISI) from loss in the channel
Closed eye	The receiver can't tell levels apart reliably: bit errors.

PAM4: three eyes stacked

With four levels there are three decision thresholds, so a PAM4 eye has three smaller eyes stacked vertically. With the same maximum swing, each eye is only about a third as tall as an NRZ eye. That is roughly a 9.5 dB signal-to-noise penalty, which is why PAM4 links at HDR, NDR, and XDR always need FEC.

On an NVIDIA adapter or switch you don't plug in an oscilloscope. The SerDes measures the opening internally. mlxlink reports eye-opening and physical-grade figures per lane where the device supports it, and BER and FEC counters on every device. Low or uneven eye figures and a high raw BER on one lane point at the same thing: a marginal cable, connector, or module.

Which eye fields mlxlink shows depends on the device generation and firmware. Some newer PAM4 devices report limited or no eye data, and you rely on BER, FEC, and module diagnostics instead. The method doesn't change: compare lanes, compare with healthy links, test under load.

8. Interactive eye simulator (NRZ vs PAM4)

Move the sliders and watch the eye close. Noise thickens the lines (crosstalk, dirty optics, low received power). Jitter spreads the crossings (clock or timing problems). Loss / ISI makes each symbol depend on the one before it (long or poor cable, a bad connector). This is a teaching model, not a real channel simulation.

Signaling

Preset

Noise:

Jitter:

Loss / ISI:

Try the same noise with NRZ and then PAM4. PAM4 closes much sooner, because each of its eyes has a third of the height to lose. That is the whole reason 400G links need FEC and are more sensitive to dirt and bad cables than 100G links were.

9. What closes the eye

Cause	Effect on the eye	Typical real-world source	Usual fix
Insertion loss	Smaller height, more ISI	Copper too long for the speed, thin or damaged cable, too many patch panels	Shorter cable, optics instead of copper, fewer connections
Contamination	Lower received power, more noise	Dust or oil on a fiber end face	Inspect and clean both ends
Reflections	Ringing, ghost edges	Poor connector mating, APC/UPC mismatch, damaged end face	Correct connectors, reseat, replace damaged jumper
Crosstalk	Thicker lines	Damaged twinax, poor shielding	Replace cable
Jitter	Narrower width	Signal integrity issues, failing module electronics	Replace module, update firmware
Mechanical stress	Intermittent errors, flaps	Copper bent past its bend radius, fiber pinched by a door or tie, cable not latched	Re-route, relieve strain, reseat until latched
Heat	Errors rising with temperature	Blocked airflow, wrong module top style, failing fan	Fix airflow, correct module variant

10. Equalization and link training

High-speed SerDes don't just send and receive. They actively equalize the channel: the transmitter pre-shapes the signal (feed-forward equalization) and the receiver boosts and cleans it (CTLE, DFE). During link training, both ends negotiate speed, width, and FEC, and tune these settings for this specific cable.

That is why reseating or toggling a port can change results: it forces retraining.
A link that retrains often shows up as rising link error recovery or link down counters.
If training can't reach the target speed at full width, the link may settle lower: reduced rate or width.
Training happens per link. A good port with a marginal cable can train fine one day and badly the next.

Don't "fix" a degraded link by forcing a lower speed and calling it done, unless that's a deliberate, documented decision. Lower speed hides the physical fault and costs bandwidth on every job that crosses that link.

11. FEC and BER, and how long to test

FEC (Forward Error Correction, Reed-Solomon on PAM4 links) adds redundant symbols so the receiver can fix bit errors without a retransmit. That leaves two BER numbers. Raw (pre-FEC) BER is the physical channel's quality. Effective (post-FEC) BER counts errors FEC couldn't fix, and must be zero. The full explanation and a rate calculator are in the InfiniBand guide's physical-layer section. This section adds the question that matters during validation: how long do you have to test before "zero errors" means anything?

If you see zero errors over N bits, the most you can claim (at about 95% confidence) is that BER is below roughly 3 ÷ N. Proving a very low BER takes a long test window under full load. A 10-second idle check proves almost nothing.

Interactive: BER test-time calculator

Line rate (Gb/s)

Test duration (seconds)

Errors observed

Target BER to prove (exponent)

Effective (post-FEC) errors: zero, and not increasing. Raw BER: stable, similar on every lane, comparable to healthy links of the same type, and well inside the thresholds that ibdiagnet / UFM flag. Link down and link error recovery counters: not increasing. Rate and width: as designed (for example 400 at 4X). All measured after clearing counters and running real traffic for a meaningful window.

12. mlxlink in depth

mlxlink ships with MFT (NVIDIA Firmware Tools). Run mst start first. Use either the MST device (/dev/mst/mt4129_pciconf0) or the RDMA name (mlx5_0).

sudo mst start; sudo mst status -v           # device names and their mlx5_X / netdev mapping
sudo mlxlink -d mlx5_0                       # summary: state, physical state, speed, width, FEC, recommendation
sudo mlxlink -d mlx5_0 -m                    # module info: type, vendor, part number, serial, length, FW, temperature
sudo mlxlink -d mlx5_0 -c                    # counters: raw BER, effective BER, symbol errors, per-lane errors, link down
sudo mlxlink -d mlx5_0 --pc                  # clear counters before a measurement window
sudo mlxlink -d mlx5_0 -e                    # eye opening / grade per lane (where supported)
sudo mlxlink -d mlx5_0 --cable --ddm         # module digital diagnostics: temperature, voltage, TX bias, TX/RX power per lane
sudo mlxlink -d mlx5_0 --cable --dump        # raw module EEPROM pages (for vendor support)
sudo mlxlink -d mlx5_0 -c -m -e --json       # machine-readable output for scripts (where supported)

`--cable` is an operation mode, and `-m` isn't its short form

A common mix-up, including in some study notes, is that -m is short for --cable. It isn't. In the mlxlink source (MFT and the open-source mstlink), -m is --show_module, a read-only display. --cable has no short form.

--cable is a mode switch: "the next option is an operation on the module, not on the port." The module operations are only accepted after it:

Command	What it does
`mlxlink -d <dev> --cable --ddm`	Live digital diagnostics: temperature, voltage, TX bias, TX/RX optical power per lane
`mlxlink -d <dev> --cable --dump`	Raw EEPROM pages in hex, for vendor support
`mlxlink -d <dev> --cable --read --page <n> --offset <o> [--length <len>]`	Read bytes from one EEPROM page. `--page` and `--offset` are both required; `--length` defaults to 1 byte.
`mlxlink -d <dev> --cable --write <b1,b2,…> --page <n> --offset <o>`	Write bytes to module memory (changes the module; vendor guidance only)
`mlxlink -d <dev> --cable --prbs_select HOST\|MEDIA --prbs_mode EN\|DS …`	Module PRBS test mode (host or media side), with `--show_diagnostic_info` for its counters
`mlxlink -d <dev> --cable --control_parameters [--tx_equalization …]`	Show or set module control parameters (equalization, emphasis, amplitude)

The tool enforces these rules in current releases, not just old ones:

--ddm, --dump, --read, --write (or --page/--offset/--length) without --cable → "--cable" flag should be specified! So mlxlink -m --ddm fails.
One operation per command: --cable --ddm --dump → Please choose one of cable operations!
--read or --write without both --page and --offset → error. --length works with --read only, and is spelled --length, not --len.
--cable combined with -m is allowed. It reads module info through a different register path (PMCR instead of PDDR), but for everyday checks plain -m is what you want.

Look-alike flags that change things

Several mlxlink short flags look like "show" options but configure the link. Running them on a production port can drop it. Verified against the mlxlink source and the MFT manual:

Flag	Actually means	Easy to confuse with
`-m`	`--show_module` (read-only module info)	"--cable" (no short form)
`-c`	`--show_counters` (read BER and errors)	"clear": clearing is `--pc`
`-e`	`--show_eye`	"errors" (those are in `-c`)
`-k <mode>`	`--fec`: sets FEC (AU, NF, RS, LL, …), optionally per `--fec_speed`	Showing FEC: that's `--show_fec`, or the summary's FEC line
`-s <speeds>`	`--speeds`: sets the allowed speeds	"show speed" (that's in the plain summary)
`-a UP\|DN\|TG`	`--port_state`: brings the port up, down, or toggles it (link flap)	"all" (that's `--all`, and only with `-a`)
`-l <mode>`	`--loopback`: sets a loopback mode (NO, RM, PH, EX, LL, TRAN)	`mlxcables -l`, which is the read length
`-p <port>`	Port number (needed on switches)	`mlxcables -p`, which is the EEPROM page
`-b <csv>`	`--ber_collect`: runs a BER collection and writes a CSV	"show BER" (that's `-c`)
`--module_state UP\|DN\|TG`	Powers the module up, down, or toggles it	`-a` (port state)

mlxcables is the older cable tool. It works on cable devices created by mst cable add (for example mt4129_pciconf0_cable_0) and uses different spellings: --DDM in uppercase, -r -p <page> -o <offset> -l <length> for reads, -q to query. NVIDIA keeps it for EEPROM reads and writes but adds no new features; mlxlink --cable is the way forward. Full option list in the MFT guide.

Rule of thumb: -m, -c, -e, --show_*, and --cable --ddm/--dump/--read only read. -a, -k, -s, -l, --module_state, --serdes_tx, --cable --write, and --cable --control_parameters with values change the hardware. --pc resets counters, so note the old values first if they matter.

Reading the summary output

Field names vary a little by version, but a healthy NDR port looks like this (values illustrative):

Operational Info
State                     : Active
Physical state            : LinkUp
Speed                     : IB-NDR
Width                     : 4x
FEC                       : Standard_RS-FEC - (544,514)
Loopback Mode             : No Loopback
Auto Negotiation          : ON

Troubleshooting Info
Status Opcode             : 0
Recommendation            : No issue was observed

Field	Healthy	If not
State / Physical state	Active / LinkUp	`Polling` = no partner signal (cable, module, far end down). `LinkUp` but state `Init` on InfiniBand = Subnet Manager problem, not a cable.
Speed / Width	Expected speed, full width (NDR, 4x)	Lower speed or narrower width = physical fault, wrong cable rating, or split-mode mismatch
FEC	RS-FEC active on PAM4 links	FEC mismatch between ends can stop the link training
Recommendation	No issue observed	Read it: it often names the suspected issue (for example a cable or module problem)
Counters (`-c`)	Effective errors 0; raw BER low and even across lanes	One lane much worse = that lane's fiber, connector, or electronics

Switch ports can be checked too: in-band through MFT (after adding the IB fabric devices with mst ib add) and selecting the port, or from the switch OS and UFM. Fabric-wide, ibdiagnet collects the same physical-layer data for every link in one run.

13. Optical diagnostics (DDM / DOM)

Active modules (optics, AOCs, some active copper) report digital diagnostic monitoring (DDM, also called DOM): live readings from the module itself.

Reading	What it tells you	Red flag
Temperature	Module health and airflow	Near or above the warning threshold; much hotter than neighbors
Supply voltage	Power to the module	Outside thresholds
TX bias current (per lane)	Laser drive	One lane far higher than others (laser aging) or zero (dead lane)
TX power (per lane)	Light this module sends	Low on one lane: this module's laser
RX power (per lane)	Light arriving from the far end	Low on one lane: dirty or damaged fiber, a bad connector, or the far end's laser

The best optical clue is one lane's RX power much lower than the others. Then check the far end's TX power for that lane. Far-end TX normal and local RX low means the light is lost in between, which usually means a dirty connector or a damaged fiber. Far-end TX low means the far-end module is failing.

Modules carry warning and alarm thresholds in their EEPROM; tools compare readings against them. Passive DACs have no DDM at all: no temperature, no power, only identity.

14. Transceiver firmware 4.8

Active modules run their own firmware. Old or mismatched module firmware can cause link flaps, training failures, wrong reporting, or high BER, and these look exactly like a bad cable.

Query from the host: mlxlink -d mlx5_X -m shows the module's vendor, part number, and firmware version for each port.
Query fabric-wide: ibdiagnet collects cable and module information for every link; UFM shows a cable inventory with part numbers and firmware.
Query on switches: the switch OS's transceiver show commands list modules per port.
Compare against the cluster's firmware recipe and the vendor's qualified-module list. All modules of the same part number should run the same validated firmware.
Update with the vendor-supported method. MFT includes support for burning NVIDIA LinkX module firmware through the host adapter; switches and UFM provide their own update paths. Follow the release notes: some updates need the link to retrain or the module to be reseated to take effect.

for d in $(ls /sys/class/infiniband); do
  echo "== $d"; sudo mlxlink -d $d -m | grep -i -E 'vendor part|serial|firmware|identifier|length'
done                                         # per-node module inventory to diff against the recipe

Module firmware is separate from HCA firmware (flint/mlxfwmanager) and from switch firmware. "Firmware is up to date" on the adapter says nothing about the transceivers plugged into it. NIC and switch firmware tools are covered in the networking firmware (MFT) guide.

15. Fabric-wide checks: ibdiagnet, counters, UFM

ibdiagnet                                    # whole fabric: topology, speed/width, error counters, cable info, BER warnings
ibdiagnet --get_cable_info                   # include cable / module details (option names vary by version)
ibqueryerrors                                # ports whose error counters exceed thresholds
perfquery -x <lid> <port>                    # extended port counters for one port
perfquery -R <lid> <port>                    # reset that port's counters (start a window)
iblinkinfo                                   # every link with its speed and width: spot links below the design rate
ibportstate <lid> <port> disable             # take a flaky port out of service while waiting for parts

IB port counter	Meaning
SymbolErrorCounter	Errors detected at the physical/symbol level
LinkErrorRecoveryCounter	The link had to retrain
LinkDownedCounter	The link went down
PortRcvErrors	Bad packets received (for example CRC errors)
PortXmitDiscards	Packets dropped on transmit (often congestion, not cabling)

UFM continuously collects port and cable telemetry across the fabric, tracks BER trends, alerts on links that degrade, and keeps a cable inventory. During bring-up, an ibdiagnet run with a clean report is a common sign-off artifact. In production, UFM's alerts catch links that degrade over time.

PortXmitDiscards and congestion counters rising does not automatically mean a bad cable. Discards usually mean congestion or routing. Symbol errors, recovery, and link-down counters are the physical-layer ones.

15b. UFM Cable Validation Tool (CVT): validating every cable at bring-up

On a cluster with thousands of cables, running mlxlink port by port or diffing ibnetdiscover output by hand doesn't scale. NVIDIA's Cable Validation Tool (part of the UFM family) checks every link against the cabling plan automatically. It flags miswiring and poor-quality links and gives the cabling crew a remediation list. It covers InfiniBand, Ethernet (Cumulus, SONiC), NVOS/XDR switches, and NVLink fabrics.

How it works

Piece	What it is
Collector ("bring-up server")	A Docker container on any machine on the switches' management network. It reads the topology, deploys and monitors agents, collects reports, and serves the web UI at `https://<collector>/cables_validation` (plus CSV export and REST APIs). It can also run embedded in UFM.
Agents	Small containers deployed (over SSH) on every switch, and optionally on servers. They check link state every 10 seconds and report on change, plus a routine report every 10 minutes. On a state change they also collect an amBER file.
Topology file (the source of truth)	A P2P (point-to-point) Excel workbook; the unified format, which covers IB, Ethernet, XDR and NVLink, is recommended. A legacy `.topo` file (InfiniBand), a `.dot` file (Ethernet), or the topology from UFM also work.

Workflow (web UI wizard)

Load topology: the P2P/topo/dot file, plus optional DC floor layout, IP mapping (if there's no DNS), or HCA mapping files.
Credentials for switches and hosts (defaults, per node, or per credential profile in the P2P file).
Pre-check every device: ping, JSON API, SSH, Docker, LLDP, agent status.
Deploy agents to the devices in scope.
Start validation (optionally with a timeout), then work the report until every link reads Correctly-wired with no syndromes.

docker exec -it cables_bringup bringupcli       # optional CLI; the web UI is the recommended way
load_ptp /data/cluster_p2p.xlsx format=unified_topo
check_switch_status                             # ping / JSON API / agent per switch
deploy_agents all
start_validation timeout=2h
show_statistics                                 # counts per syndrome: wrong-neighbor, wrong-port, flapping, BER…

What it reports

Group	Syndromes	Typical fix
Miswiring	Wrong-neighbor (cable goes to a different device than the plan), Wrong-port (right device, wrong port), Extra-cable (connected but not in the plan), Unknown-neighbor (up, but the peer can't be identified), Media unplugged (no cable or transceiver), NIC name mismatch (Ethernet hosts)	Re-patch to the plan, or fix the plan if it's wrong
Layer-1 integrity	Link down, no signal (module present, no light), Flapping-link (more than 2 carrier transitions in 125 s on switches), Underperforming-link (BER), Anomalous-port (lane power or temperature out of range), RX/TX power mismatch (more than 3 dB between the two ends), ErrDisable (link protection after repeated flaps, or the RX-disable issue)	Clean and inspect, reseat, swap the cable, then the module
Not a cable problem	AdminDown, Negotiation fail (speed, FEC, or config mismatch), Unreachable-device (no agent, no ping, or port 8251 blocked), No report	Fix the configuration or the CVT setup; escalate

CVT's BER criteria for a link: effective BER errors = 0 during the first 125 minutes; raw BER ≤ 1e-6; effective BER ≤ 1.5E-254 for measurements of 6 hours or less, and ≤ 1E-15 beyond 6 hours. Built-in tests add a Golden BER test, on-demand amBER collection, and an advanced flapping test.

Prerequisites (the usual reasons it fails)

Docker on the collector and on the switches and hosts being validated.
TCP port 8251 open between the collector and agents. It opens automatically on InfiniBand switches; on Ethernet (Cumulus NVUE) and NVOS/XDR switches add an ACL rule (nv set acl …, then nv config apply).
SSH to every device for agent deployment, and wget/curl on devices to fetch the agent image.
Servers (if CVT manages them): MFT and OFED installed. Ethernet servers also need lldpd; GB200 compute trays need the NVIDIA Container Toolkit and nvidia-smi.
LLDP running on Ethernet devices (the SONiC LLDP container, for example), because neighbor identity comes from it.

Where CVT fits with the other tools

Question	Best tool
Is every cable plugged into the port the plan says, across the whole cluster?	CVT (against the P2P plan); `ibnetdiscover`/`iblinkinfo` or LLDP for manual spot checks
Is this one link clean, and which lane is bad?	`mlxlink -c`, `-e`, `--cable --ddm`
Fabric-wide health snapshot for sign-off (counters, speeds, cable info)	`ibdiagnet`
Ongoing monitoring in production	UFM telemetry and alerts

Order of operations at bring-up: CVT until all links are correctly wired and clean → ibdiagnet sign-off → perftest and NCCL under load (BER again under traffic). Miswiring found by CVT costs a re-patch; miswiring found by a slow NCCL run costs days.

16. Physical handling: cleaning, latching, bend radius, safety

Inspect, then clean, then inspect again. Use a fiber inspection scope on every optical end face before mating. Clean with a proper one-click cleaner or cleaning cassette for the connector type (MPO or LC). Dust is the single most common cause of high-BER optical links.
Keep dust caps on any unconnected module or fiber end.
Latch fully. Push until it clicks; a pull-tab test should not remove it. A half-seated copper cable is a classic intermittent link.
Respect bend radius. Thick 400G copper is stiff. Bending it hard or pulling it tight across a rack damages it. Don't pinch fiber with cable ties, doors, or rack rails.
Strain relief. Heavy cables hanging from a port stress the cage and connector over time.
Laser safety. Never look into a module or fiber end, even if you think it's dark. 850 nm and 1310 nm light is invisible.
Label both ends and record which node port connects to which switch port; it makes rail and cabling validation fast.

I-C-I-L: Inspect, Clean, Inspect, Latch.

17. Troubleshooting flow

Find the suspect link: ibdiagnet, ibqueryerrors, UFM alerts, a link below design rate in iblinkinfo, or one slow rail in nccl-tests.
Is it the right part? mlxlink -m on both ends: correct speed rating, type, length within spec, correct form factor and top style, firmware on the recipe.
Measure under load: clear counters (mlxlink --pc), run traffic across that link (ib_write_bw or nccl-tests), then read mlxlink -c and -e. Note effective errors, raw BER per lane, and which lane is worst.
For optics, read DDM on both ends: compare TX power at one end with RX power at the other, lane by lane.
Cheapest fixes first: reseat both ends until latched; inspect and clean fiber; remove strain or tight bends.
Swap to isolate: move the cable/module to a known-good port, or put a known-good cable into the suspect port. Errors follow the cable or module → replace it. Errors stay with the port → the HCA, switch port, or its firmware.
Confirm and record: clear counters, re-test under load for a meaningful window, confirm rate/width and zero effective errors, update the cable inventory, return the node to service.

The fix for a physical-layer problem is physical (reseat, clean, replace, correct part, firmware). It's never an NCCL environment variable, a Slurm setting, or a lower MTU.

18. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

sudo mst start; sudo mst status -vStart MFT, list devices and netdevs

Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE             MST                           PCI       RDMA            NET                       NUMA
ConnectX7(rev:0)        /dev/mst/mt4129_pciconf0      18:00.0   mlx5_0          net-ibp24s0               0
ConnectX7(rev:0)        /dev/mst/mt4129_pciconf1      29:00.0   mlx5_1          net-ibp41s0               0
ConnectX7(rev:0)        /dev/mst/mt4129_pciconf2      40:00.0   mlx5_2          net-ibp64s0               0
ConnectX7(rev:0)        /dev/mst/mt4129_pciconf3      4f:00.0   mlx5_3          net-ibp79s0               0
...
ConnectX7(rev:0)        /dev/mst/mt4129_pciconf7      dc:00.0   mlx5_7          net-ibp220s0              1

What to look at: This maps each MST device (/dev/mst/mt4129_pciconfN) to its PCI address, mlx5_X name and netdev, so you know which physical port you are about to test. A ConnectX-7 that is missing from the list, or 'MST PCI configuration module' failing to load, means MFT cannot reach that adapter.

sudo mlxlink -d mlx5_0Link summary and recommendation

Operational Info
State                     : Active
Physical state            : LinkUp
Speed                     : IB-NDR
Width                     : 4x
FEC                       : Standard_RS-FEC - (544,514)
Loopback Mode             : No Loopback
Auto Negotiation          : ON

Supported Info
Enabled Link Speed        : 0x00000080 (NDR)
Supported Cable Speed     : 0x000000fe (NDR,HDR,EDR,FDR,FDR10,QDR,SDR)

Troubleshooting Info
Status Opcode             : 0
Group Opcode              : N/A
Recommendation            : No issue was observed

Tool Information
Firmware Version          : 28.39.1002
amBER Version             : 3.2
MFT Version               : mft 4.26.1-3

What to look at: Healthy is Active / LinkUp, IB-NDR at 4x with RS-FEC (544,514), and 'No issue was observed'. Polling means no signal from the partner (cable, module or far end down), LinkUp with State Init on InfiniBand is a Subnet Manager problem rather than a cable, and a narrower width or lower speed is a physical fault or wrong cable rating.

sudo mlxlink -d mlx5_0 -mModule identity, firmware and DDM summary

Module Info
Identifier                         : OSFP
Compliance                         : 400GBASE-SR4
Cable Technology                   : 850 nm VCSEL
Cable Type                         : Optical Module (separated)
OUI                                : Nvidia
Vendor Name                        : NVIDIA
Vendor Part Number                 : MMA4Z00-NS400
Vendor Serial Number               : MT2301X00001
Rev                                : A6
Wavelength [nm]                    : 850
Transfer Distance [m]              : 50
FW Version                         : 47.171.0
Digital Diagnostic Monitoring      : Yes
Power Class                        : 8.5 W max
Temperature [C]                    : 52 [-5..75]
Voltage [mV]                       : 3281 [3135..3465]
Bias Current [mA]                  : 7.412,7.388,7.450,7.401 [5.5..10]
Rx Power Current [dBm]             : 0.912,1.104,-6.873,1.011 [-7.6..4]
Tx Power Current [dBm]             : 1.203,1.287,1.115,1.198 [-5.6..4]
Module State                       : Ready state
Rx LOS [per lane]                  : 0,0,0,0
Tx Fault [per lane]                : 0,0,0,0

What to look at: Compare Vendor Part Number and the module 'FW Version' against the cluster recipe (this is module firmware, separate from the HCA's 28.39.1002). Here lane 3 Rx power (-6.873 dBm) is about 8 dB below the other lanes and close to its low threshold, the classic sign of a dirty or damaged fiber, so check the far end's Tx power for that lane next.

sudo mlxlink -d mlx5_0 -cRaw/effective BER and per-lane errors

Physical Counters and BER Info
Time Since Last Clear [Min]        : 60.2
Effective Physical Errors          : 0
Effective Physical BER             : 15E-255
Raw Physical Errors Per Lane       : 41022,39870,18833906,40115
Raw Physical BER                   : 1E-8
Link Down Counter                  : 0
Link Error Recovery Counter        : 0
Symbol Errors                      : 0
Symbol BER                         : 15E-255

What to look at: Effective (post-FEC) errors must be 0, and they are here, but lane 3 has roughly 450 times more raw errors than the other lanes. That uneven pre-FEC BER is the early warning of a marginal fiber or optic: clean or swap it before effective errors or link-down events start appearing.

sudo mlxlink -d mlx5_0 -eEye opening and grade per lane

EYE Opening Info
Physical Grade                     : 5821, 5790, 1904, 5866
Height Eye Opening [mV]            : 38, 37, 12, 39
Phase Eye Opening [psec]           : 8, 8, 3, 8

What to look at: Eye height is voltage margin and eye width (phase) is timing margin, so all lanes should be similar. One lane with a much smaller grade and eye (lane 3 here) matches its higher raw BER; on some PAM4 ports and MFT versions this section is not supported and reports N/A, in which case rely on -c and DDM.

sudo mlxlink -d mlx5_0 --cable --ddmLive module DDM per lane

Cable DDM Information
Temperature         : 52C
Voltage             : 3.2810V
Channels            : Channel 1 ,Channel 2 ,Channel 3 ,Channel 4
RX Power            : 0.912dBm ,1.104dBm ,-6.873dBm ,1.011dBm
TX Power            : 1.203dBm ,1.287dBm ,1.115dBm ,1.198dBm
TX Bias             : 7.412mA ,7.388mA ,7.450mA ,7.401mA
DDM Flags
Temperature Alarm high        : 0
Temperature Warning high      : 0
...
Channel 3 Flags
RX Power Alarm high           : 0
RX Power Warning high         : 0
RX Power Warning low          : 1
RX Power Alarm low            : 0
...
DDM Thresholds
Thresholds               : Temperature, Voltage, RX Power, TX Power, TX Bias
High alarm threshold     : 75C ,3.465V ,4.000dBm ,4.000dBm ,10.000mA
High warning threshold   : 70C ,3.400V ,3.000dBm ,3.000dBm ,9.500mA
Low warning threshold    : 0C ,3.200V ,-6.600dBm ,-4.600dBm ,6.000mA
Low alarm threshold      : -5C ,3.135V ,-7.600dBm ,-5.600dBm ,5.500mA

What to look at: Compare each lane's RX power with the others and with the thresholds; Channel 3 has tripped 'RX Power Warning low'. If the far end's TX power on that lane is normal, the light is being lost in the fiber or connector, while a low far-end TX power means the far-end laser is failing, and a DAC will show no DDM at all.

sudo mlxlink -d mlx5_0 --cable --dumpRaw module EEPROM pages for support

Page 0x0 (Lower memory)
0x00000000 : 19 50 07 00 00 00 00 00 00 00 00 00 00 00 34 ab
0x00000010 : 80 2b 00 00 00 00 00 00 00 00 00 00 00 00 00 00
...
Page 0x0 (Upper memory)
0x00000080 : 19 00 02 c9 4e 56 49 44 49 41 20 20 20 20 20 20
0x00000090 : 20 20 20 20 00 02 c9 4d 4d 41 34 5a 30 30 2d 4e
...
Page 0x11
0x00000080 : 44 44 44 44 00 00 00 00 00 00 00 00 00 00 00 00
...

What to look at: This is raw hex for vendor support or for decoding a specific field; you can spot ASCII such as 'NVIDIA' and the part number in the upper page. You do not interpret it by eye on the exam: remember it is a --cable operation, so running --dump without --cable fails.

sudo mlxlink -d mlx5_0 -m --ddmWrong syntax: DDM without --cable

-E- "--cable" flag should be specified!

What to look at: --ddm, --dump, --read and --write are module operations that are only accepted after --cable, and -m is --show_module, not a short form of --cable. Only one cable operation is allowed per command, so '--cable --ddm --dump' fails with 'Please choose one of cable operations!'.

sudo mst cable add; sudo mlxcables -qOlder tool: create and query cable devices

-I- Added 8 cable devices ..

Querying Cables ....

Cable #1:
---------
Cable name    : mt4129_pciconf0_cable_0
>> No FW data to show
-------- Cable EEPROM --------
Identifier    : OSFP (19h)
Technology    : 850 nm VCSEL (00h)
Compliance    : 400GBASE-SR4
OUI           : 0x0002c9
Vendor        : NVIDIA
Serial number : MT2301X00001
Part number   : MMA4Z00-NS400
Revision      : A6
Temperature   : 52 C
Length        : 50 m
...

What to look at: mlxcables works on the cable devices that 'mst cable add' creates (mt4129_pciconf0_cable_0) and uses its own spellings, such as --DDM in uppercase and -r -p -o -l for page, offset and length. Use it to confirm part number and serial; a 'Length' of 1-3 m with a copper technology string means a DAC, which has no DDM.

for d in $(ls /sys/class/infiniband); do echo "== $d"; sudo mlxlink -d $d -m | grep -i -E 'vendor part|serial|firmware|identifier|length'; done

Per-node module inventory for firmware audit

== mlx5_0
Firmware Version                   : 28.39.1002
Identifier                         : OSFP
Vendor Part Number                 : MMA4Z00-NS400
Vendor Serial Number               : MT2301X00001
SMF Length                         : N/A
== mlx5_1
Firmware Version                   : 28.39.1002
Identifier                         : OSFP
Vendor Part Number                 : MMA4Z00-NS400
Vendor Serial Number               : MT2301X00002
SMF Length                         : N/A
...

What to look at: Diff part numbers and serials across nodes against the recipe; a different part number on one port stands out immediately. Note that 'firmware' matches the HCA 'Firmware Version' line from Tool Information, not the module's 'FW Version' line, so add 'fw version' to the grep pattern when auditing module firmware.

ibdiagnet --get_cable_infoFabric-wide run including cable data

Loading IBDIAGNET from: /usr/lib/x86_64-linux-gnu/ibdiagnet2.1.1
-I- Using port 1 as the local port
...
Cable Diagnostic (Plugin)
-I- Cable Info retrieving finished successfully
-W- lid=0x0003 dev=54002 leaf02/P4  RX power on lane 3 is below the low warning threshold
-I- Cable Diagnostic finished with warnings

Phy Diagnostic (Plugin)
-W- lid=0x0003 dev=54002 leaf02/P4  Raw BER 1e-08 is higher than the warning threshold
-I- Phy Diagnostic finished with warnings

Summary
-I- Stage                     Warnings   Errors     Comment
-I- Discovery                 0          0
-I- Port Counters             0          0
-I- Speed / Width checks      0          0
-I- Cable Diagnostic (Plugin) 1          0
-I- Phy Diagnostic (Plugin)   1          0

-I- You can find detailed errors/warnings in: /var/tmp/ibdiagnet2/ibdiagnet2.log
-I- Database                  : <db_csv> /var/tmp/ibdiagnet2/ibdiagnet2.db_csv

What to look at: A sign-off run should finish with 0 errors and 0 warnings; cable and BER warnings name the switch LID and port, which you then inspect with mlxlink and DDM. Option names and plugin stage names vary by ibdiagnet version, so check 'ibdiagnet --help' if --get_cable_info is rejected.

ibqueryerrorsPorts with error counters over threshold

Errors for "leaf02"
   GUID 0xfc6a1c0300e5a200 port 4: [SymbolErrorCounter == 214] [LinkErrorRecoveryCounter == 3]

## Summary: 74 nodes checked, 1 bad nodes found
##          212 ports checked, 1 ports have errors beyond threshold
##
## Thresholds:
##   [SymbolErrorCounter = 10] [LinkErrorRecoveryCounter = 10] [LinkDownedCounter = 10]
##   [PortRcvErrors = 10] [PortRcvRemotePhysicalErrors = 100] [PortXmitDiscards = 100]
##   ...

What to look at: SymbolErrorCounter, LinkErrorRecoveryCounter and LinkDownedCounter are the physical-layer counters that implicate a cable or optic. PortXmitDiscards growing on its own usually means congestion or routing, not a bad cable.

perfquery -x 59 1Extended counters for one port

# Port extended counters: Lid 59 port 1 (CapMask: 0x5A00)
PortSelect:......................1
CounterSelect:...................0x0000
PortXmitData:....................1923847561024
PortRcvData:.....................1918003392768
PortXmitPkts:....................7493205688
PortRcvPkts:.....................7476180468
...
SymbolErrorCounter:..............214
LinkErrorRecoveryCounter:........3
LinkDownedCounter:...............0
PortRcvErrors:...................0
PortRcvRemotePhysicalErrors:.....0
PortXmitDiscards:................0
PortXmitWait:....................18321

What to look at: Reset with 'perfquery -R 59 1', run traffic, then read again: physical error counters that grow during the window confirm the link is bad. Counters that stay at 0 under load mean an old count was historical, not an active fault.

iblinkinfoEvery link's speed and width

Switch: 0xfc6a1c0300e5a200 leaf02:
           3    1[  ] ==( 4X      106.25 Gbps Active/  LinkUp)==>      12    1[  ] "ufm-02 HCA-1" ( )
           3    3[  ] ==( 4X      106.25 Gbps Active/  LinkUp)==>      58    1[  ] "dgx01 mlx5_1" ( )
           3    4[  ] ==( 2X      106.25 Gbps Active/  LinkUp)==>      60    1[  ] "dgx02 mlx5_1" ( Could be 4X 106.25 Gbps)
           3    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           3   33[  ] ==( 4X      106.25 Gbps Active/  LinkUp)==>       4    3[  ] "spine01" ( )
...

What to look at: Every link should be 4X 106.25 Gbps Active/LinkUp. 'Could be 4X' means the link trained below what both ends support (lanes lost to a bad cable or dirty optic), and Down/Polling on a cabled port means no light or signal from the far end.

ibportstate 3 4 disableTake a flaky switch port out of service

Initial Switch PortInfo:
# Port info: Lid 3 port 4
LinkState:.......................Active
PhysLinkState:...................LinkUp
Lid:.............................3
SMLid:...........................1
LinkWidthActive:.................2X
LinkSpeedExtActive:..............106.25 Gbps
After PortInfo set:
# Port info: Lid 3 port 4
LinkState:.......................Down
PhysLinkState:...................Disabled
Lid:.............................3
SMLid:...........................1

What to look at: Disable the switch-side port of a known-bad link (leaf02 LID 3 port 4 here) so jobs stop using it while parts are on order. PhysLinkState Disabled confirms it took effect; it stays down until 'ibportstate 3 4 enable', and disabling the port you are connected through cuts off your own access.

19. Scenarios

Try to name the cause before you open each one.

20. Searchable reference

Item	Meaning

21. Flashcards (click to flip)

22. Exam traps

Link up is not link good. Active at the right rate only proves training succeeded. Signal quality is proven by errors under load.
Effective (post-FEC) errors must be zero. Raw (pre-FEC) errors are normal on PAM4 links; watch for high, rising, or one-lane-worse values.
One lane much worse than the others → that lane's fiber, connector, or module electronics.
Reduced rate or width → physical first: cable rating, damaged lane, split-mode mismatch.
Errors follow the cable → replace the cable. Errors stay with the port → port, HCA, or switch.
Optical link with high BER → inspect and clean before replacing anything.
DACs have no DDM. No temperature or optical power from a passive copper cable.
Transceiver firmware ≠ HCA firmware ≠ switch firmware. Check all three for objective 4.8.
OSFP = 8 lanes (two NDR ports with twin-port); QSFP = 4 lanes.
Finned-top for cages without a heat sink (switches); flat-top for cages with a riding heat sink (adapters, DGX).
Never mate APC with UPC.
PAM4 eyes are a third the height of NRZ eyes, so PAM4 needs FEC and is more sensitive to dirt and loss.
A short idle test proves little. Zero errors over N bits only shows BER below about 3/N.
Port in Init with physical LinkUp is a Subnet Manager problem, not a cable problem.
Validating cabling at scale = UFM Cable Validation Tool (CVT) against the P2P plan, not hand-diffing ibnetdiscover. Wrong-neighbor / wrong-port are wiring; AdminDown / negotiation fail / unreachable-device aren't cable problems.
-m is --show_module, not --cable. --ddm, --dump, --read, and --write only work after --cable, one at a time.
-c reads counters; --pc clears them. -k, -s, -a, and -l change the port (FEC, speeds, state, loopback).
mlxcables ≠ mlxlink spelling: --DDM (uppercase); -p/-o/-l = page/offset/length, while mlxlink's -p is the port and -l is loopback.

23. Practice quiz (34 questions)

Score: 0 / 0