Lustre & Third-Party Storage Tuning

How AI clusters use shared storage: Lustre's architecture, LNet over InfiniBand, striping, client tuning, NFS with nconnect and RDMA, partner filesystems (WEKA, VAST, IBM Storage Scale, DDN), GPUDirect Storage, BCM mounts, and benchmarking. Written for NCP-AII objectives 5.5 (optimize storage) and 1.11 (configure third-party storage).

1. The big picture: why storage starves GPUs

Training reads the dataset over and over, and periodically writes huge checkpoints. With hundreds of GPUs reading at once, a storage system that's fine for home directories becomes the bottleneck: GPU utilization drops, and the job time is spent waiting on I/O.

AI clusters solve this with parallel filesystems (Lustre, IBM Storage Scale, WEKA) or high-performance NFS (VAST, and others), connected over the fast storage fabric (InfiniBand or RoCE), plus local NVMe on each node as a cache.

A parallel filesystem is a supermarket with many checkout lanes. One customer with a big cart can split it across several lanes (striping), and many customers can check out at once. A single NFS server is one lane: fine for a corner shop, a queue for a stadium crowd. High-performance NFS products add many lanes behind one front door.

For the exam: know Lustre's parts (MGS, MDS/MDT, OSS/OST, clients) and LNet, how striping spreads a file over OSTs and when to change it, the main client tunables, how to mount NFS for speed (nconnect, RDMA, large rsize/wsize), what GPUDirect Storage does, and how to measure (IOR, mdtest, fio, gdsio).

2. Storage tiers in an AI cluster

Tier	Example	Use
Local NVMe (per node)	DGX `/raid` (RAID 0 of NVMe drives)	Dataset cache, scratch; fastest, not shared, not protected
Shared high-performance	Lustre, WEKA, VAST, Storage Scale, DDN EXAScaler	Datasets, checkpoints, shared outputs
Shared general NFS	Head node or NAS export	Home directories, software, configs
Object storage	S3-compatible	Long-term datasets and model artifacts

More on local drives and caching in the BCM node storage guide, including where each file type lives and which network it crosses and how model weights are loaded across nodes.

3. Lustre architecture

Part	Role
MGS (Management Server)	Holds the filesystem configuration; clients contact it first when mounting. Stores the persistent (`lctl set_param -P`) settings.
MDS (Metadata Server) with MDTs (Metadata Targets)	Namespace: names, directories, permissions, file layouts, locks. Every open, create, stat, and delete hits it.
OSS (Object Storage Server) with OSTs (Object Storage Targets)	File data, stored as objects. A file is one or more objects on one or more OSTs.
Clients	GPU nodes running the Lustre client kernel modules. After the MDS hands out the layout, they talk to OSSs directly and in parallel.
LNet	Lustre's network layer; runs over InfiniBand/RoCE (`o2ib`) or TCP (`tcp`).

"Maps, Menus, Orders": the MGS has the map, the MDS has the menu (what files exist and where), the OSSs fill the orders (data).

Data bandwidth scales with OSTs; metadata rate (creates, stats) scales with MDTs. Lots of small files stresses the MDS, not the OSTs.

4. LNet: Lustre networking over InfiniBand

Every Lustre endpoint has one or more NIDs (network identifiers) of the form <address>@<network>, for example <ib0-ip>@o2ib or <eth-ip>@tcp. o2ib uses RDMA verbs over InfiniBand or RoCE; the address is the IPoIB address used for connection setup.

# Configure LNet dynamically
sudo modprobe lnet
sudo lnetctl lnet configure
sudo lnetctl net add --net o2ib0 --if ib0
sudo lnetctl net show            # configured networks and their NIDs
lctl list_nids                   # this node's NIDs
lctl ping <server-nid>@o2ib0     # LNet-level reachability test

# Multi-Rail: several interfaces on the same LNet network
sudo lnetctl net add --net o2ib0 --if ib0,ib1
sudo lnetctl peer show           # peers and their NIDs

# Persist
sudo lnetctl export > /etc/lnet.conf   # loaded by the lnet service at boot

The older static method sets options lnet networks="o2ib0(ib0)" in a modprobe config file.

Multi-Rail lets LNet use several NICs per node, adding bandwidth and resilience.
lctl ping failing while ping over IPoIB works means an LNet configuration issue (wrong network name, interface not added, module not loaded), not a cabling one.
LNet routers forward between networks (for example clients on o2ib1 reaching servers on o2ib0).

5. Mounting a Lustre client

# The client modules must match the running kernel
modinfo lustre | grep -e ^version -e vermagic
uname -r

sudo mkdir -p /lustre
sudo mount -t lustre <mgs-nid>@o2ib0:/aifs /lustre
# with a failover MGS: <mgs1-nid>@o2ib0:<mgs2-nid>@o2ib0:/aifs

# /etc/fstab
<mgs-nid>@o2ib0:/aifs  /lustre  lustre  defaults,_netdev,flock  0 0

lfs df -h /lustre        # every MDT/OST and its usage
lfs check servers        # are all targets reachable?

The Lustre client is a kernel module. After a kernel update, the old client modules don't load and the mount fails. Rebuild or install the client package for the new kernel (in BCM: inside the software image, for the image's kernel).

The flock option enables coherent file locking across clients, which some applications need. _netdev makes the mount wait for the network.

6. Striping

A Lustre file's layout says which OSTs hold its data and how it's split:

Stripe count: how many OSTs the file is spread over. The common default is 1.
Stripe size: how much data goes to one OST before moving to the next. The default is 1 MiB.
Data is written round-robin: chunk 0 to the first OST, chunk 1 to the second, and so on, wrapping around.

lfs getstripe /lustre/data/big.bin       # see a file's layout
lfs setstripe -c 8 -S 4M /lustre/ckpt     # new files in this directory: 8 OSTs, 4 MiB stripes
lfs setstripe -c -1 /lustre/shared        # stripe over ALL OSTs
lfs setstripe -c 1 /lustre/smallfiles     # one OST per file (many small files)
lfs migrate -c 8 /lustre/data/big.bin     # re-layout an existing file

Workload	Striping	Why
Many small files, each read by one process	Count 1	Each file on one OST; many files spread naturally; less overhead
One huge file read or written by many processes (shared checkpoint)	Wide (many OSTs or `-1`)	Aggregate bandwidth of many OSTs
File-per-process checkpoints	Count 1–4	The many files already spread the load
Very large single-stream file	Moderate count, larger stripe size	More OSTs in parallel, fewer RPCs

Layouts are set when a file is created. Setting a stripe on a directory changes the default for new files in it. Existing files keep their layout until you lfs migrate (or copy) them.

Too much striping hurts small files: each file touches many OSTs, adding locking and RPC overhead. Stripe wide only for large, shared files.

7. Interactive stripe calculator

File size (GiB)

Stripe count (-1 = all OSTs)

Stripe size (MiB)

OSTs in the filesystem

Bandwidth per OST (GB/s)

Client network limit (GB/s)

A teaching model: real throughput also depends on the OSS servers, the network, client RPC settings, and other users.

8. PFL, DoM, and DNE

Progressive File Layout (PFL)

A PFL layout changes as the file grows: small files stay on one OST, big files spread out automatically. It's a good directory default when you don't know file sizes in advance.

# First 64 MiB on 1 OST, up to 1 GiB on 4 OSTs, the rest over all OSTs
lfs setstripe -E 64M -c 1 -E 1G -c 4 -E -1 -c -1 /lustre/projects

Data on MDT (DoM)

The first component of a file can live on the MDT itself. Small files then need no OST round trip at all.

# First 1 MiB on the MDT, the rest on 4 OSTs
lfs setstripe -E 1M -L mdt -E -1 -c 4 /lustre/smallfiles

DNE (Distributed Namespace)

With several MDTs, directories can be placed on, or spread across, different MDTs so metadata load isn't all on MDT0.

lfs mkdir -i 1 /lustre/teamB          # directory on MDT index 1 (remote directory)
lfs mkdir -c 2 /lustre/hugedir        # striped directory across 2 MDTs
lfs getdirstripe /lustre/hugedir

PFL = Progressively Fatter Layout. DoM = Data on the Metadata server. DNE = the namespace is Distributed, so directories Need not all sit on MDT0 (Each can have its own).

9. Client tuning with lctl

lctl get_param osc.*.max_rpcs_in_flight          # concurrent RPCs per OST (default 8)
sudo lctl set_param osc.*.max_rpcs_in_flight=16
sudo lctl set_param osc.*.max_pages_per_rpc=1024 # 4 MiB RPCs with 4 KiB pages (default 256 = 1 MiB)
sudo lctl set_param osc.*.max_dirty_mb=512       # dirty cache per OST before writes block
sudo lctl set_param llite.*.max_read_ahead_mb=1024
sudo lctl set_param llite.*.max_read_ahead_per_file_mb=256

# Persistent for all clients: run on the MGS
sudo lctl set_param -P osc.*.max_rpcs_in_flight=16

# Statistics
lctl get_param llite.*.stats
lctl get_param osc.*.rpc_stats          # RPC size and in-flight histograms

Tunable	Effect	Raise when
`max_rpcs_in_flight`	Parallel requests to each OST	High-bandwidth networks; few clients driving many OSTs
`max_pages_per_rpc`	Size of each bulk RPC	Large sequential I/O (servers must support the size)
`max_dirty_mb`	Write-back cache per OST	Bursty large writes such as checkpoints
`max_read_ahead_mb`	Read-ahead window	Sequential reads of large files

lctl set_param without -P changes only the running client and is lost on remount or reboot. Use -P on the MGS (or a node startup script / BCM image) to make it stick.

10. Lustre health and capacity checks

Command	Shows
`lfs df -h`	Space per MDT and OST; spot a full or unbalanced OST
`lfs df -i`	Inodes per target; small-file workloads can run out of inodes before space
`lfs check servers`	Whether the client can reach every target
`lctl get_param osc..state` / `mdc..state`	Connection state per target (FULL is healthy)
`lfs quota -u <user> /lustre`	User quota and usage
`lctl dl`	Lustre devices on this node
`dmesg \| grep -i lustre`	Evictions, timeouts, reconnects

A full OST makes writes fail with "No space left on device" for files striped to it, even when df shows free space overall. Rebalance with lfs migrate or by adding OSTs; administrators can also stop new allocations on a full OST.

11. High-performance NFS

Plain NFS uses one TCP connection per mount, which caps throughput on fast networks. Modern clients fix that:

Option	Effect
`nconnect=N`	Up to 16 TCP connections per mount (Linux 5.3+); spreads load across connections and CPU cores
`proto=rdma,port=20049`	NFS over RDMA (InfiniBand/RoCE); port 20049 is the standard NFS/RDMA port
`rsize=1048576,wsize=1048576`	1 MiB read/write sizes
`vers=3` / `vers=4.1` / `vers=4.2`	Protocol version; 4.1+ adds sessions and pNFS
`hard`	Retry forever on server trouble (safe for data); `soft` can return I/O errors and corrupt writes
`noatime`	Skip access-time updates
`fsc`	Use FS-Cache (`cachefilesd`) to cache on local disk; see the BCM storage guide

sudo mount -t nfs -o vers=3,nconnect=16,rsize=1048576,wsize=1048576,hard <nfs-server>:/datasets /data
nfsstat -m                         # mounted NFS filesystems and their effective options
cat /proc/self/mountstats          # per-mount operation counts and latencies
nfsiostat 5                        # NFS throughput and latency every 5 s

Check the effective options with nfsstat -m: servers can negotiate smaller rsize/wsize than requested. For NFS over TCP on Ethernet, use jumbo frames (MTU 9000) end to end.

12. Interactive NFS mount builder

Server (placeholder)

Export

Mount point

Version

Transport

nconnect (1–16)

rsize / wsize (KiB)

Local cache

13. Partner filesystems

Product	Access	Client-side things to know
DDN EXAScaler	Lustre-based	Everything in this guide's Lustre sections applies; use DDN's client package and tuning for the kernel
WEKA	Native POSIX client; also NFS, SMB, S3	The client reserves dedicated CPU cores and huge pages for its frontend processes; choose those cores on the NIC's NUMA node and keep them away from training processes
VAST Data	NFS (including NFS over RDMA), multipath	Mount with nconnect / multipath options per VAST's guide; spread mounts across the VAST virtual IPs
IBM Storage Scale (GPFS)	Native client	The pagepool (client cache) size is the main knob (`mmchconfig pagepool=...`); uses RDMA verbs when configured
NetApp, Pure, Dell PowerScale	NFS (often with nconnect / RDMA)	Vendor mount guides; GDS support varies

Common initial configuration steps (objective 1.11): install the vendor client matching the kernel, configure the storage network interfaces (IP, MTU, RDMA), set up the mount (fstab or BCM fsmounts), apply the vendor's client tuning, confirm GDS support if used, and benchmark against the vendor's expected numbers.

The exam won't expect every vendor's CLI. It expects the pattern: vendor client + correct network + mount + vendor tuning + benchmark, applied identically through the image on every node.

14. GPUDirect Storage

GPUDirect Storage (GDS) lets data move by DMA between storage (local NVMe or a network filesystem's RDMA path) and GPU memory, skipping the bounce buffer in CPU memory. This lowers CPU usage and latency and can raise bandwidth.

Applications use the cuFile API (directly or through libraries such as DALI, KvikIO, or frameworks that support it).
Components: the nvidia-fs kernel module (or the in-kernel P2P DMA path on newer stacks), libcufile, and /etc/cufile.json for configuration.
Supported targets include local NVMe (ext4/xfs), and partner filesystems that implement it (several Lustre distributions, WEKA, VAST, Storage Scale, NFS over RDMA). Check the vendor's support statement.
If the path isn't supported, cuFile can fall back to compatibility mode (POSIX reads through CPU memory) when allow_compat_mode is true. It works, but without the GDS benefit.

/usr/local/cuda/gds/tools/gdscheck -p        # platform check: which filesystems/drivers support GDS
lsmod | grep nvidia_fs
/usr/local/cuda/gds/tools/gdsio -f /lustre/gds/test -d 0 -w 4 -s 8G -i 1M -x 0 -I 0
#   -d GPU index  -w worker threads  -s file size  -i I/O size  -x 0 = GPU direct  -I 0 = read (1 = write)

More detail in the BCM storage guide's GDS section, including the step-by-step enable procedure and exam gotchas.

GDS doesn't make a slow filesystem fast. It removes the CPU copy; the storage and network still set the ceiling. ACS and IOMMU settings matter here too (see the server tuning guide).

15. Storage in Base Command Manager

cmsh
% category use dgx-h100
% fsmounts
% add /lustre
% set device <mgs-nid>@o2ib0:/aifs
% set filesystem lustre
% set mountoptions defaults,_netdev,flock
% commit

Put mounts in the category (or node group) so every node of that type gets the same mounts; BCM writes them to each node's fstab.
Install the filesystem client in the software image, built for the image's kernel (cm-chroot-sw-img into the image, install, then update the nodes). A kernel update in the image needs a matching client.
Put LNet configuration, NFS client settings, and lctl tunables in the image or a node startup script, not by hand on individual nodes.
Configure the storage network interfaces (IPoIB or Ethernet, MTU) in BCM's network and interface settings so they come up before the mount.

16. AI I/O patterns: datasets, small files, checkpoints

Pattern	Problem	What helps
Millions of small images or text files	Metadata-bound: every file is an open + stat + read; the MDS becomes the bottleneck	Pack into large shards (WebDataset tar, TFRecord, Parquet); DoM; DNE; stage to local NVMe
Each epoch re-reads the whole dataset	Shared storage serves the same bytes repeatedly	Cache on local `/raid` (copy at job start, or FS-Cache for NFS)
Large checkpoints from all ranks at once	Write bursts saturate OSTs or the network	File-per-rank with modest striping, or wide striping for one shared file; async checkpointing; stagger writes
Random reads of a big file	Read-ahead wasted	Lower read-ahead for that workload; larger stripes don't help random I/O
Data loader can't keep up	CPU-side decode, too few workers, wrong NUMA node	More loader workers on local cores; GPU decode (DALI); GDS

Small files: Pack, Place, Pre-stage. Pack them into shards, place small data on the MDT (DoM), pre-stage to local NVMe.

17. Benchmarking: IOR, mdtest, fio, gdsio

# Drop the client page cache before read tests (on every client)
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

# IOR: bandwidth. File per process (-F), reorder tasks for reads (-C) so each node reads
# what ANOTHER node wrote (defeats client caching), fsync on write close (-e)
mpirun -np 64 --map-by node ior -a POSIX -w -r -F -C -e -t 1m -b 16g -o /lustre/bench/ior

# mdtest: metadata rate (creates, stats, removes per second)
mpirun -np 64 --map-by node mdtest -F -n 10000 -i 3 -d /lustre/bench/mdtest

# fio: single-node bandwidth/IOPS
fio --name=seqread --rw=read --bs=1M --size=16G --numjobs=8 --direct=1 --ioengine=libaio --iodepth=16 --directory=/lustre/bench --group_reporting

# gdsio: GPU-direct read throughput (see section 14)

Tool	Measures	Watch out for
IOR	Aggregate read/write bandwidth across nodes	Caching (use `-C`, drop caches, test size larger than client RAM)
mdtest	Metadata operations per second	Run in a fresh directory; striping of the test directory
fio	Per-node bandwidth, IOPS, latency	`--direct=1` to bypass the page cache
gdsio	GDS throughput into GPU memory	Compare `-x 0` (GDS) with CPU paths; check compat mode

Aggregate IOR bandwidth near the storage vendor's expected value for the client count; every node gets similar per-node fio numbers; mdtest rates match expectations; GDS shows a true GPU-direct path where expected; results recorded as the baseline.

18. Symptoms and fixes

Symptom	Likely cause	Fix
Large file reads cap at one OST's bandwidth	Stripe count 1	`lfs setstripe -c` wider on the directory; `lfs migrate` the file
Training with small files is slow; OSTs are idle	Metadata bound	Pack into shards; DoM; stage locally
"No space left" while `lfs df` shows space overall	One OST full	Migrate files off it; rebalance; add capacity
Mount fails after a kernel update	Client modules built for the old kernel	Install the client for the new kernel in the image
`lctl ping` fails, IPoIB ping works	LNet not configured on that interface/network	`lnetctl net add`; check the network name
Client evictions and reconnects in dmesg	Network flaps, overloaded servers, timeouts	Check the fabric (ibdiagnet), server load, LNet health
NFS stuck near 1–2 GB/s on a 200/400 Gb/s link	Single TCP connection, small rsize/wsize	`nconnect=16`, 1 MiB sizes, or NFS over RDMA
Benchmark reads are impossibly fast	Reading from the client cache	Drop caches, IOR `-C`, larger test size, `--direct=1`
gdsio throughput same as CPU path	Compatibility mode (GDS not active)	`gdscheck -p`; nvidia-fs loaded; filesystem support; cufile.json
Tuning lost after reboot	`lctl set_param` without `-P`	Persist via `-P` on the MGS or the image

19. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

Illustrative output, not captured from a real system. Hostnames, serial numbers, GUIDs, MAC and IP addresses are invented, and columns and wording vary between versions. Learn the shape of each output and the fields called out under it.

sudo lnetctl net showConfigured LNet networks and NIDs

net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
          status: up
    - net type: o2ib
      local NI(s):
        - nid: 192.0.2.101@o2ib
          status: up
          interfaces:
              0: ib0

What to look at: The o2ib network should list this node's NID on the expected IB interface with status: up (o2ib0 is displayed as o2ib). Only a lo entry means LNet was configured without the RDMA network, and status: down means the IB port or IPoIB address is not up.

lctl ping 192.0.2.10@o2ib0LNet-level reachability test

12345-0@lo
12345-192.0.2.10@o2ib

What to look at: A successful ping returns the remote node's NIDs, each prefixed by 12345-. 'failed to ping ... Input/output error' or a timeout means an LNet or fabric problem, so check this before suspecting the Lustre servers or mount options.

sudo lnetctl peer showPeers and their NIDs (Multi-Rail)

peer:
    - primary nid: 192.0.2.21@o2ib
      Multi-Rail: True
      peer ni:
        - nid: 192.0.2.21@o2ib
          state: NA
        - nid: 192.0.2.31@o2ib
          state: NA
    - primary nid: 192.0.2.22@o2ib
      Multi-Rail: True
      peer ni:
        - nid: 192.0.2.22@o2ib
          state: NA
        - nid: 192.0.2.32@o2ib
          state: NA
...

What to look at: With Multi-Rail each OSS should appear once with Multi-Rail: True and all of its NIDs. A peer with only one NID, or Multi-Rail: False, means traffic to that server can only use one interface.

modinfo lustre | grep -e ^version -e vermagic; uname -rClient modules match running kernel

version:        2.15.4
vermagic:       5.15.0-1042-nvidia SMP mod_unload modversions
5.15.0-1042-nvidia

What to look at: The vermagic kernel must equal uname -r. If modinfo reports 'Module lustre not found', or vermagic shows a different kernel after an update, the client was not rebuilt for the new kernel and the mount fails with 'No such device'.

sudo mount -t lustre 192.0.2.10@o2ib0:/aifs /lustre && mount -t lustreMount the Lustre client

192.0.2.10@o2ib:/aifs on /lustre type lustre (rw,checksum,flock,nouser_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt)

What to look at: The mount line confirms the MGS NID, filesystem name and options such as flock. Errors like 'mount.lustre: mount ... failed: Input/output error' usually mean LNet cannot reach the MGS; 'No such device' means the lustre module is not loaded.

lfs df -h /lustreSpace per MDT and OST

UUID                       bytes        Used   Available Use% Mounted on
aifs-MDT0000_UUID           5.6T       49.4G        5.6T   1% /lustre[MDT:0]
aifs-MDT0001_UUID           5.6T       42.7G        5.6T   1% /lustre[MDT:1]
aifs-OST0000_UUID         116.4T       71.3T       39.2T  65% /lustre[OST:0]
aifs-OST0001_UUID         116.4T       70.8T       39.7T  64% /lustre[OST:1]
aifs-OST0002_UUID         116.4T       72.1T       38.4T  65% /lustre[OST:2]
aifs-OST0003_UUID         116.4T       69.9T       40.6T  63% /lustre[OST:3]
aifs-OST0004_UUID         116.4T       71.6T       38.9T  65% /lustre[OST:4]
aifs-OST0005_UUID         116.4T      112.9T        3.5T  97% /lustre[OST:5]
aifs-OST0006_UUID         116.4T       70.4T       40.1T  64% /lustre[OST:6]
aifs-OST0007_UUID         116.4T       71.2T       39.3T  64% /lustre[OST:7]

filesystem_summary:       931.2T      610.2T      279.7T  69% /lustre

What to look at: Check each OST, not just filesystem_summary: here OST0005 is at 97% while the rest are around 65%. Writes striped onto a full OST fail with 'No space left on device' even though the total shows plenty of free space.

lfs df -i /lustreInodes per MDT and OST

UUID                      Inodes       IUsed       IFree IUse% Mounted on
aifs-MDT0000_UUID   1006632960    61203317   945429643    6% /lustre[MDT:0]
aifs-MDT0001_UUID   1006632960    58440112   948192848    6% /lustre[MDT:1]
aifs-OST0000_UUID    122093568    14400213   107693355   12% /lustre[OST:0]
aifs-OST0001_UUID    122093568    14401400   107692168   12% /lustre[OST:1]
aifs-OST0002_UUID    122093568    14402587   107690981   12% /lustre[OST:2]
aifs-OST0003_UUID    122093568    14403774   107689794   12% /lustre[OST:3]
aifs-OST0004_UUID    122093568    14404961   107688607   12% /lustre[OST:4]
aifs-OST0005_UUID    122093568    14406148   107687420   12% /lustre[OST:5]
aifs-OST0006_UUID    122093568    14407335   107686233   12% /lustre[OST:6]
aifs-OST0007_UUID    122093568    14408522   107685046   12% /lustre[OST:7]

filesystem_summary:  2013265920   119643429  1893622491    6% /lustre

What to look at: Small-file workloads can run out of inodes on the MDTs before running out of space. An MDT IUse% near 100% blocks file creation even when lfs df -h shows free capacity.

lfs check serversAre all targets reachable?

aifs-MDT0000-mdc-ff3c8e4a2b1d9000 active.
aifs-MDT0001-mdc-ff3c8e4a2b1d9000 active.
aifs-OST0000-osc-ff3c8e4a2b1d9000 active.
aifs-OST0001-osc-ff3c8e4a2b1d9000 active.
aifs-OST0002-osc-ff3c8e4a2b1d9000 active.
...
aifs-OST0007-osc-ff3c8e4a2b1d9000 active.

What to look at: Every MDC and OSC line should say active. A target reported as inactive or with a timeout error means the client cannot talk to that server, so I/O to files striped on it hangs.

lfs getstripe /lustre/data/big.binShow a file's layout

/lustre/data/big.bin
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 3
        obdidx           objid           objid           group
             3         8812345        0x867739                0

What to look at: This large file sits on one OST (stripe_count 1), so every reader is limited to one OSS. Large shared files should use more stripes (or -c -1); also check the obdidx is not a full OST.

lfs setstripe -c 8 -S 4M /lustre/ckpt && lfs getstripe -d /lustre/ckptSet and verify a directory's default layout

stripe_count:  8 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

What to look at: The default layout applies only to new files created in the directory; existing files keep their old layout until you lfs migrate them. stripe_offset -1 means the MDS picks the starting OST, which is what you want for balance.

lfs getdirstripe /lustre/hugedirStriped directory across MDTs

lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush
mdtidx           FID[seq:oid:ver]
     0           [0x200000bd1:0x5:0x0]
     1           [0x240000bd0:0x3:0x0]

What to look at: lmv_stripe_count 2 confirms the directory's entries are spread over MDT0 and MDT1. A stripe count of 0 or 1 means all metadata for this huge directory still hits a single MDT.

lctl get_param osc.*.max_rpcs_in_flightConcurrent RPCs per OST

osc.aifs-OST0000-osc-ff3c8e4a2b1d9000.max_rpcs_in_flight=8
osc.aifs-OST0001-osc-ff3c8e4a2b1d9000.max_rpcs_in_flight=8
osc.aifs-OST0002-osc-ff3c8e4a2b1d9000.max_rpcs_in_flight=8
...
osc.aifs-OST0007-osc-ff3c8e4a2b1d9000.max_rpcs_in_flight=8

What to look at: 8 is the default; high-bandwidth clients are often tuned to 16 or more. Values that differ between clients point at settings applied with set_param on one node instead of set_param -P on the MGS.

lctl get_param osc.*.stateConnection state per OST

osc.aifs-OST0000-osc-ff3c8e4a2b1d9000.state=
current_state: FULL
state_history:
 - [ 1794273185, CONNECTING ]
 - [ 1794273185, FULL ]
...
osc.aifs-OST0005-osc-ff3c8e4a2b1d9000.state=
current_state: FULL
state_history:
 - [ 1794273185, CONNECTING ]
 - [ 1794273185, FULL ]
 - [ 1794273267, DISCONN ]
 - [ 1794273267, CONNECTING ]
 - [ 1794273307, EVICTED ]
 - [ 1794273307, RECOVER ]
 - [ 1794273307, FULL ]
...

What to look at: FULL is the healthy state. A history with DISCONN and EVICTED entries, as on OST0005 here, shows the client lost and re-established its connection; match the timestamps with dmesg and check the fabric and server load.

lctl get_param osc.*.rpc_statsRPC size and in-flight histograms

osc.aifs-OST0000-osc-ff3c8e4a2b1d9000.rpc_stats=
snapshot_time:         1794280012.118220337 secs.nsecs
read RPCs in flight:  0
write RPCs in flight: 0
pending write pages:  0
pending read pages:   0

                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                      12   0   0   |          3   0   0
2:                       0   0   0   |          0   0   0
...
512:                     4   0   0   |          8   0   0
1024:                48211  99 100   |      51230  99 100

                        read                    write
rpcs in flight        rpcs   % cum % |       rpcs   % cum %
1:                    1204   2   2   |        980   1   1
...
15:                   3301   6  83   |       3120   6  82
16:                   8120  16 100   |       9204  17 100
...

What to look at: Most RPCs should land in the largest pages-per-rpc bucket (1024 pages = 4 MiB after tuning) and in-flight counts should reach the configured maximum under load. Mostly small RPCs point at small, random or unaligned application I/O.

dmesg -T | grep -i lustreEvictions, timeouts, reconnects

[Tue Nov 10 09:13:02 2026] Lustre: Lustre: Build Version: 2.15.4
[Tue Nov 10 09:13:05 2026] Lustre: Mounted aifs-client
[Tue Nov 10 09:14:27 2026] Lustre: aifs-OST0005-osc-ff3c8e4a2b1d9000: Connection to aifs-OST0005 (at 192.0.2.23@o2ib) was lost; in progress operations using this service will wait for recovery to complete
[Tue Nov 10 09:15:07 2026] LustreError: 167-0: aifs-OST0005-osc-ff3c8e4a2b1d9000: This client was evicted by aifs-OST0005; in progress operations using this service will fail.
[Tue Nov 10 09:15:07 2026] Lustre: aifs-OST0005-osc-ff3c8e4a2b1d9000: Connection restored to 192.0.2.23@o2ib (at 192.0.2.23@o2ib)

What to look at: Mount and build messages are normal. 'Connection ... was lost' followed by 'This client was evicted' means in-flight I/O failed and applications may see EIO; repeated evictions point at fabric flaps or an overloaded OSS.

nfsstat -mEffective NFS mount options

/data from 10.20.0.50:/datasets
 Flags: rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nconnect=16,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.20.0.50,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.20.0.50

What to look at: These are the negotiated values, which can be lower than what you asked for. Check rsize/wsize are 1048576, nconnect=16 is present and the mount is hard; a smaller rsize means the server capped it.

/usr/local/cuda/gds/tools/gdscheck -pGPUDirect Storage platform check

GDS release version: 1.9.1.3
nvidia_fs version:  2.19 libcufile version: 2.12
Platform: x86_64
============
ENVIRONMENT:
============
=====================
DRIVER CONFIGURATION:
=====================
NVMe               : Supported
NVMeOF             : Unsupported
SCSI               : Unsupported
ScaleFlux CSD      : Unsupported
NVMesh             : Unsupported
DDN EXAScaler      : Supported
IBM Spectrum Scale : Unsupported
NFS                : Supported
BeeGFS             : Unsupported
WekaFS             : Unsupported
Userspace RDMA     : Unsupported
--Mellanox PeerDirect : Enabled
--rdma library        : Not Loaded (libcufile_rdma.so)
--rdma devices        : Not configured
--rdma_device_status  : Up: 0 Down: 0
=====================
CUFILE CONFIGURATION:
=====================
properties.use_compat_mode : true
properties.force_compat_mode : false
...
=========
GPU INFO:
=========
GPU index 0 NVIDIA H100 80GB HBM3 bar:1 bar size (MiB):131072 supports GDS, IOMMU State: Disabled
...
==============
PLATFORM INFO:
==============
IOMMU: disabled
Platform verification succeeded

What to look at: The filesystem you use (DDN EXAScaler, which is Lustre-based, here) must say Supported and each GPU must say supports GDS. Unsupported for your filesystem, a missing nvidia_fs version, or IOMMU enabled means cuFile silently falls back to compatibility mode through host memory.

/usr/local/cuda/gds/tools/gdsio -f /lustre/gds/test -d 0 -w 4 -s 8G -i 1M -x 0 -I 0GPU-direct read throughput

IoType: READ XferType: GPUD Threads: 4 DataSetSize: 8388608/8388608(KiB) IOSize: 1024(KiB) Throughput: 11.873451 GiB/sec, Avg_Latency: 328.913612 usecs ops: 8192 total_time 0.673745 secs

What to look at: XferType GPUD confirms the GPU-direct path; compare Throughput against -x 2 (CPU to GPU through host memory). If GPUD is no faster, or throughput is far below the client NIC rate, check gdscheck, PCIe locality between the GPU and NIC, and Lustre striping.

mpirun -np 64 --map-by node ior -a POSIX -w -r -F -C -e -t 1m -b 16g -o /lustre/bench/iorIOR aggregate bandwidth

IOR-3.3.0: MPI Coordinated Test of Parallel I/O
Began               : Tue Nov 10 15:02:11 2026
Command line        : ior -a POSIX -w -r -F -C -e -t 1m -b 16g -o /lustre/bench/ior
Machine             : Linux dgx01
TestID              : 0
StartTime           : Tue Nov 10 15:02:11 2026
Path                : /lustre/bench/ior.00000000
FS                  : 931.2 TiB   Used FS: 65.5%   Inodes: 1920.0 Mi   Used Inodes: 5.9%

Options:
api                 : POSIX
test filename       : /lustre/bench/ior
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 8
tasks               : 64
clients per node    : 8
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 16 GiB
aggregate filesize  : 1 TiB

Results:

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
write     78412      78413      0.000812    16777216   1024       0.021300   13.36      0.412000   13.37      0
read      91233      91235      0.000698    16777216   1024       0.008100   11.49      0.010200   11.49      0

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) ...
write       78412.33   78412.33   78412.33       0.00   78412.33   78412.33   78412.33       0.00   13.37223 ...
read        91233.10   91233.10   91233.10       0.00   91233.10   91233.10   91233.10       0.00   11.49311 ...
Finished            : Tue Nov 10 15:02:37 2026

What to look at: Check that task offset is 1 (from -C, so each node reads data another node wrote) and that the aggregate file size is much larger than client memory. Read bandwidth far above what the storage can deliver means caching skewed the result; very low write bandwidth often traces back to stripe count or a full OST.

mpirun -np 64 --map-by node mdtest -F -n 10000 -i 3 -d /lustre/bench/mdtestMetadata operation rates

-- started at 11/10/2026 15:20:03 --

mdtest-3.3.0 was launched with 64 total task(s) on 8 node(s)
Command line used: mdtest '-F' '-n' '10000' '-i' '3' '-d' '/lustre/bench/mdtest'
Path: /lustre/bench
FS: 931.2 TiB   Used FS: 65.5%   Inodes: 1920.0 Mi   Used Inodes: 5.9%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000
64 tasks, 640000 files

SUMMARY rate: (of 3 iterations)
   Operation                     Max            Min           Mean        Std Dev
   ---------                     ---            ---           ----        -------
   File creation             :      61234.112      58120.554      59811.307       1281.442
   File stat                 :     212388.901     204117.338     208922.170       3462.118
   File read                 :      98877.234      95002.771      97115.902       1602.334
   File removal              :      54321.665      51987.120      53098.443        953.221
   Tree creation             :       1532.118       1201.334       1388.901        137.552
   Tree removal              :        412.556        388.901        401.112          9.660
-- finished at 11/10/2026 15:21:44 --

What to look at: Rates are operations per second across all tasks; compare creation and stat rates with your baseline and between iterations. A large Std Dev, or creation rates that collapse as you add clients, points at a single busy MDT, which DNE striped directories can spread.

fio --name=seqread --rw=read --bs=1M --size=16G --numjobs=8 --direct=1 --ioengine=libaio --iodepth=16 --directory=/lustre/bench --group_reporting

Single-node sequential read bandwidth

seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-3.28
Starting 8 processes
Jobs: 8 (f=8): [R(8)][100.0%][r=18.1GiB/s][r=18.5k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=8): err= 0: pid=73122: Tue Nov 10 15:40:12 2026
  read: IOPS=18.4k, BW=18.0GiB/s (19.3GB/s)(128GiB/7121msec)
    slat (usec): min=21, max=2211, avg=48.31, stdev=19.02
    clat (usec): min=402, max=61233, avg=6895.12, stdev=2102.77
     lat (usec): min=431, max=61290, avg=6943.55, stdev=2103.01
    clat percentiles (usec):
     |  1.00th=[ 2737],  5.00th=[ 3884], 10.00th=[ 4555], 20.00th=[ 5342],
     | 30.00th=[ 5866], 40.00th=[ 6325], 50.00th=[ 6783], 60.00th=[ 7177],
     | 70.00th=[ 7701], 80.00th=[ 8455], 90.00th=[ 9372], 95.00th=[10421],
     | 99.00th=[13698], 99.50th=[16057], 99.90th=[28443], 99.95th=[38011],
     | 99.99th=[47449]
   bw (  MiB/s): min=15102, max=20411, per=100.00%, avg=18421.33, stdev=180.22, samples=112
   iops        : min=15102, max=20411, avg=18421.18, stdev=180.20, samples=112
  lat (usec)   : 500=0.01%, 750=0.02%, 1000=0.05%
  lat (msec)   : 2=0.61%, 4=5.21%, 10=87.33%, 20=6.62%, 50=0.14%, 100=0.01%
  cpu          : usr=0.61%, sys=9.83%, ctx=128911, majf=0, minf=4211
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=131072,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=18.0GiB/s (19.3GB/s), 18.0GiB/s-18.0GiB/s (19.3GB/s-19.3GB/s), io=128GiB (137GB), run=7121-7121msec

What to look at: The read line gives IOPS and bandwidth, and err= 0 confirms no I/O errors. Compare BW with the node's storage NIC capacity; a long clat tail at 99.99th, or bandwidth far below the network rate, points at striping, RPC tuning or one slow OST. --direct=1 makes sure you measured storage, not page cache.

cmsh -c 'category use dgx-h100; fsmounts; list'BCM mounts for the DGX category

Device                         Mountpoint (key)     Filesystem
------------------------------ -------------------- ----------
$localnfsserver:/cm/shared     /cm/shared           nfs
$localnfsserver:/home          /home                nfs
192.0.2.10@o2ib0:/aifs         /lustre              lustre

What to look at: After the commit, the /lustre entry with filesystem lustre should appear in the category's fsmounts, and BCM then writes it to /etc/fstab on every node in the category. If it is missing, the commit was not done or the change was made in a different category than the nodes use.

20. Scenarios

21. Searchable reference

Item	Meaning

22. Flashcards (click to flip)

23. Exam traps

MDS/MDT = metadata, OSS/OST = data, MGS = configuration. Small-file slowness points to metadata.
Default stripe: count 1, size 1 MiB. Big shared files need wider striping.
Striping applies to new files; existing files need lfs migrate.
-c -1 = all OSTs.
o2ib is LNet over InfiniBand/RoCE; lctl ping tests LNet, not IP.
lctl set_param -P (on the MGS) persists; without -P it's temporary.
nconnect maxes at 16; NFS over RDMA uses port 20049.
The Lustre client is a kernel module: it must match the kernel in the image.
GDS skips the CPU bounce buffer; compatibility mode means it's not really active.
Defeat caching in benchmarks: drop caches, IOR -C, --direct=1.
Put mounts in the BCM category, clients and tunables in the image.

24. Practice quiz (24 questions)

Score: 0 / 0