DGX & Cluster Storage

Everything storage on a DGX cluster: the storage hierarchy, NVMe and nvme-cli, RAID levels and why /raid is RAID 0, mdadm, local caching (FS-Cache, staging, container caches), shared and parallel filesystems, GPUDirect Storage, tuning and benchmarking with fio, data loading and checkpointing. Part B covers changing node storage layouts the BCM way.

1. The storage hierarchy for AI

A training job reads data that starts far away (shared storage) and must end up in GPU memory. Each tier is faster and smaller than the one below it:

1b. Where each file lives, and which network it crosses

Knowing the tiers isn't enough: questions describe a specific file (a container image, a checkpoint, a home directory) and ask where it belongs or why something is slow. This map covers the usual artifacts on a DGX cluster. Paths are typical examples; sites name their mounts differently.

Block, file, and object: what the "type" column means

Which network, step by step: one training job

2. Disk basics: disks, partitions, filesystems

$ lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT
nvme0n1      1.7T disk
├─nvme0n1p1  512M part vfat    /boot/efi
└─nvme0n1p2  1.7T part linux_raid_member
  └─md0      1.7T raid1 ext4   /
nvme1n1      1.7T disk          (mirror of nvme0n1)
nvme2n1      3.5T disk linux_raid_member
  └─md1     27.9T raid0 ext4   /raid
... nvme3n1 – nvme9n1 (the other 7 data drives, also in md1)

3. NVMe fundamentals

4. nvme-cli: inventory, health, and erase

sudo nvme list                       # all NVMe devices: model, serial, size, firmware
sudo nvme smart-log /dev/nvme2       # health
#   critical_warning      : 0        <- non-zero = act now
#   temperature           : 38 C
#   available_spare       : 100%
#   percentage_used       : 3%       <- endurance consumed (can exceed 100%)
#   data_units_written    : ...
#   media_errors          : 0        <- should stay 0
#   num_err_log_entries   : 0
#   unsafe_shutdowns      : 4
sudo nvme error-log /dev/nvme2
sudo nvme id-ctrl /dev/nvme2 | grep -E 'mn|fr'    # model, firmware revision
sudo nvme fw-log /dev/nvme2

# Erasing (DESTROYS DATA)
sudo nvme format /dev/nvme2n1 --ses=1    # secure erase user data
sudo nvme sanitize /dev/nvme2 -a 2       # block-erase sanitize (drive-wide)

5. RAID levels

6. RAID calculator

7. Software RAID with mdadm

DGX systems use Linux software RAID (md, managed with mdadm) rather than a hardware RAID card. NVMe drives are fast enough that a RAID controller would be a bottleneck, and software RAID is flexible and transparent.

cat /proc/mdstat                     # all md arrays and their state
#   md1 : active raid0 nvme9n1[7] nvme8n1[6] ... nvme2n1[0]
#   md0 : active raid1 nvme1n1p2[1] nvme0n1p2[0]
#         [UU]                       <- both mirror halves Up; [U_] = degraded

sudo mdadm --detail /dev/md0         # state, members, failed/spare devices
sudo mdadm --examine /dev/nvme0n1p2  # RAID metadata on one member

# RAID 1: replace a failed mirror member
sudo mdadm /dev/md0 --fail /dev/nvme1n1p2 --remove /dev/nvme1n1p2
# (swap the drive, partition it like the other one, then)
sudo mdadm /dev/md0 --add /dev/nvme1n1p2      # rebuild starts; watch /proc/mdstat

# RAID 0: there's nothing to rebuild. A lost member means re-creating the array:
sudo mdadm --create /dev/md1 --level=0 --raid-devices=8 /dev/nvme{2..9}n1
sudo mkfs.ext4 /dev/md1   # then mount at /raid and refill the cache

8. How storage is laid out on DGX systems

9. Why /raid is RAID 0 (and why that's fine)

Why it's effective

1. Aggregate bandwidth. Striping spreads each large read across all 8 drives at once, so a big sequential read gets roughly 8× one drive's throughput. That's what keeps a data loader feeding 8 GPUs.
2. Aggregate IOPS. Small random reads (many small image or audio files) are spread across 8 controllers and their queues.
3. All capacity usable. No space spent on mirrors or parity: ~30 TB usable from ~30 TB raw.
4. No write penalty. No parity to compute or read-modify-write, so staging a dataset or writing scratch output is as fast as the drives allow.
5. Simple and cheap in software. md RAID 0 has almost no CPU overhead.

Why the risk is acceptable

• /raid is designed as a cache / scratch tier. Its contents are copies of data whose master lives on shared storage (or is temporary by nature).
• If a drive fails, you lose cached data, not unique data: replace the drive, recreate the array, and the cache refills.
• Training jobs write checkpoints to shared storage, so a failed /raid costs at most a restart, not lost work.

10. Local caching techniques

FS-Cache with cachefilesd (persistent NFS cache)

Linux FS-Cache lets network filesystems (NFS) cache file data on a local disk. The cachefilesd daemon manages that cache. DGX systems use /raid as the cache location, so repeated reads of NFS-hosted datasets come from local NVMe instead of the network.

# /etc/cachefilesd.conf
dir /raid/fscache
tag dgxcache
brun 25%   bcull 15%   bstop 5%     # start culling when free space drops

sudo systemctl enable --now cachefilesd

# Mount NFS with the fsc option to enable caching for that mount
sudo mount -t nfs -o fsc,vers=4.1 nfs01:/datasets /mnt/datasets
cat /proc/fs/fscache/stats            # hit/miss counters

Other caching patterns

11. Shared storage: NFS and parallel filesystems

12. GPUDirect Storage (GDS)

Normally, reading a file into GPU memory goes storage → CPU memory (bounce buffer) → GPU. GPUDirect Storage lets storage DMA directly into GPU memory, skipping the CPU bounce buffer.

Enabling GDS, step by step

This follows NVIDIA's GDS Installation and Troubleshooting Guide and the DGX OS 7 user guide. On a BCM cluster, make these changes in the software image (packages, modprobe files) and the category (kernel parameters), not on single nodes.

1. Check the hardware path. A GDS-capable data-center GPU (A100, H100, H200, B200, L4, L40S and others; see the support matrix), supported storage (local NVMe, NVMe-oF, or a GDS-enabled filesystem such as Lustre/EXAScaler, WekaFS, Storage Scale, BeeGFS, VAST/NFS over RDMA), and a working driver: nvidia-smi. Check PCIe placement with nvidia-smi topo -m. GDS is fastest when the NVMe drive or NIC sits under the same PCIe switch as the GPU.
2. IOMMU off or pass-through (x86). Check with journalctl -k -b | grep -i iommu. NVIDIA recommends intel_iommu=off / amd_iommu=off for the best functionality and performance; iommu=pt is supported on some platforms (for example DGX A100). A fully enabled IOMMU isn't guaranteed to work. This step is skipped on Arm (Grace) platforms.
3. Disable PCIe ACS for peer-to-peer performance: kernel parameter pci=disable_acs_redir=…, then update-grub and reboot. gdscheck -p reports switches that still have ACS enabled.
4. Storage transport drivers, only if needed. NVMe, NVMe-oF and NFS over RDMA need NVIDIA's patched storage drivers: DOCA (doca-ofed, mlnx-nvme-dkms, plus mlnx-nfsrdma-dkms for NFS over RDMA), or the legacy mlnxofedinstall --with-nvmf --with-nfsrdma. Run update-initramfs -u and reboot. Not required on DGX OS 6 or later. Don't mix MLNX_OFED and DOCA storage packages.
5. Install GDS. DGX OS: sudo apt install nvidia-gds. This meta-package pulls in nvidia-fs (a DKMS kernel module), libcufile and gds-tools. On other distros it comes from the CUDA repository; from CUDA 13.4, nvidia-fs is installed separately from the nvidia-fs downloads page. The kernel headers for the running kernel must be present so DKMS can build the module.
6. Load the module. sudo modprobe nvidia_fs, check with lsmod | grep nvidia_fs and cat /proc/driver/nvidia-fs/stats. For RDMA filesystems, lsmod | grep nvidia_fs | grep ib_core shows it's hooked into the RDMA stack. WekaFS and Storage Scale also need nvidia_peermem loaded (persist it with /etc/modules-load.d/).
7. Prepare the filesystem. Only ext4 and XFS are supported block filesystems. ext4 must be mounted with data=ordered written explicitly in the mount options, or the library won't recognize it. GDS supports md RAID 0 only. NVMe devices with block-integrity capability aren't supported. For parallel filesystems, follow the vendor's GDS client settings (for Lustre, LNet multi-rail peer selection).
8. Configure cuFile if the defaults don't fit. /etc/cufile.json is installed by the package. Edit it to list RDMA NIC IPs (rdma_dev_addr_list for WekaFS and NFS over RDMA), set logging, toggle allow_compat_mode, or deny-list mounts. Per-application or container overrides go in a separate file pointed to by CUFILE_ENV_PATH_JSON.
9. Verify, then measure. /usr/local/cuda/gds/tools/gdscheck.py -p (older releases ship a gdscheck binary). It lists each filesystem or device as nvfs/dmabuf/p2pdma (real GDS paths) or compat, and shows IOMMU and ACS state. Then gdsio … -x 0 (GDS) compared with -x 1 (CPU path), and gds_stats -p <pid> -l 3 while the job runs.

journalctl -k -b | grep -i iommu                 # want: IOMMU off (or pt where supported)
sudo apt install nvidia-gds                      # DGX OS; pulls nvidia-fs (DKMS), libcufile, gds-tools
sudo modprobe nvidia_fs && lsmod | grep nvidia_fs
mount | grep data=ordered                        # ext4 must show data=ordered explicitly
/usr/local/cuda/gds/tools/gdscheck.py -p         # NVMe : nvfs  (not just "compat")
/usr/local/cuda/gds/tools/gdsio -f /raid/gds/f -d 0 -w 8 -s 10G -i 1M -x 0 -I 0
cat /proc/driver/nvidia-fs/stats                 # GDS read/write counters climbing = real GDS
tail cufile.log                                  # written in the app's working directory by default

The newer path: PCI P2PDMA (no nvidia-fs for NVMe)

On recent stacks, local NVMe can use the upstream Linux PCI P2PDMA feature instead of nvidia-fs. It needs:

• Kernel 6.2 or later.
• An Ampere-or-newer GPU on the supported list.
• NVIDIA driver options NVreg_RegistryDwords="RMForceStaticBar1=1;RmForceDisableIomapWC=1;" on x86.
• NVMe multipath disabled (nvme_core multipath=N).
• "use_pci_p2pdma": true in cufile.json.

gdscheck -p then shows NVMe : p2pdma. If both paths are available, P2PDMA takes precedence. On x86, P2PDMA doesn't support RAID volumes (only Arm does). The DGX OS 7 GDS install steps are written for systems that do not use P2PDMA.

GDS exam gotchas

13. Filesystems and tuning

14. Benchmarking with fio

# Sequential read bandwidth (large blocks, many jobs, direct I/O to bypass the page cache)
fio --name=seqread --directory=/raid/fiotest --rw=read --bs=1M --size=20G \
    --numjobs=16 --iodepth=32 --ioengine=libaio --direct=1 --group_reporting

# Random 4K read IOPS
fio --name=randread --directory=/raid/fiotest --rw=randread --bs=4k --size=10G \
    --numjobs=16 --iodepth=64 --ioengine=libaio --direct=1 --group_reporting

# Sequential write (staging / checkpoint-like)
fio --name=seqwrite --directory=/raid/fiotest --rw=write --bs=1M --size=20G \
    --numjobs=8 --iodepth=32 --ioengine=libaio --direct=1 --group_reporting

15. Data loading and checkpointing patterns

Feeding the GPUs

• Multiple loader workers per GPU, with prefetching and pinned memory (for example PyTorch num_workers, pin_memory=True).
• Sharded datasets: fewer, larger files read sequentially beat millions of tiny files.
• GPU-side decoding with NVIDIA DALI moves JPEG/video decode and augmentation from CPUs to GPUs, which often removes a CPU bottleneck that looks like a storage problem.
• Watch for the symptom: GPU utilization dips between steps while CPU iowait or loader time rises.

Checkpoints

• Checkpoints must go to shared, protected storage, because a node (and its /raid) can die at any time.
• Size estimate with mixed-precision Adam: roughly ~16 bytes per parameter (weights + gradients/master weights + two optimizer moments). A 70B-parameter model is about 1.1 TB per full checkpoint.
• Patterns: asynchronous checkpointing (write in the background while training continues), sharded checkpoints (every rank writes its part in parallel), writing locally first and then copying to shared storage, and keeping only the last N.

15b. Loading model weights across a cluster

A job can't start until its weights are in GPU memory. On one node that's a file read; across dozens of nodes it's a storage and network event.

The problem: a thundering herd

A 70B-parameter model in BF16 is about 140 GB (2 bytes per parameter). If 32 inference replicas each read the full file from shared storage at the same moment, that's about 4.5 TB requested in one burst. NFS on the in-band network collapses under that. A parallel filesystem handles it far better, but it still competes with every other job's checkpoint traffic. Load time becomes start-up time, and for autoscaled inference it becomes user-visible latency.

What each GPU actually needs

Two loading patterns

• File format matters. safetensors files can be memory-mapped, so a rank reads only the tensors it needs instead of deserialising the whole file. Old pickle-based .bin checkpoints are loaded whole.
• GDS (section 12) can move weights from NVMe or a supported parallel FS straight into GPU memory, skipping the CPU bounce buffer.
• Some loaders read once per node, or once per replica group, and broadcast to the other GPUs over NVLink or the compute fabric, which trades storage reads for fast GPU-to-GPU copies.
• Don't keep weights in $HOME, pull them from the internet in every job, or make every rank read the full file and then throw most of it away.

16. The scenario

17. Quick recap: BCM, the head node, and software images

18. Where BCM keeps the partition layout

BCM stores the partition layout as part of the node's provisioning definition on the head node, not on the node. In BCM this is the disk setup: an XML description of disks, partitions, sizes, filesystems, mount points, and software RAID. It's attached to the category (or to an individual node as an override), and it's used together with the software image whenever the node is provisioned.

<!-- simplified idea of a BCM disk setup -->
<diskSetup>
  <device>
    <blockdev>/dev/nvme0n1</blockdev>
    <partition id="efi">  <size>512M</size> <filesystem>fat</filesystem>  <mountPoint>/boot/efi</mountPoint> </partition>
    <partition id="root"> <size>200G</size> <filesystem>ext4</filesystem> <mountPoint>/</mountPoint> </partition>
    <partition id="var">  <size>max</size>  <filesystem>ext4</filesystem> <mountPoint>/var</mountPoint> </partition>
  </device>
  <!-- plus RAID definitions, e.g. the OS mirror and the /raid stripe -->
</diskSetup>

The idea to remember is that both the OS contents and the disk layout (including RAID) are defined centrally on the head node.

19. What "reprovisioning" means (AUTO vs FULL)

Provisioning means putting the OS onto a node. Reprovisioning means doing it again so the node picks up a changed definition.

20. Protecting /raid and other local data

• BCM's image sync works by copying the software image onto the node. Directories holding node-local data (such as /raid, local scratch, caches) must be excluded from sync, or a sync could delete their contents.
• BCM categories carry exclude lists for this (for example separate lists for normal image updates, sync installs, and full installs).
• The disk setup can define the /raid array so routine (AUTO) boots leave its data alone, while a FULL install rebuilds it as part of the layout.

cmsh
% category use dgx-h100
% show                      # look for exclude-list properties for sync/update/full install
% set excludelistupdate     # opens an editor; e.g. ensure "- /raid/*" style entries exist
% commit

21. Why fixing nodes by hand over SSH backfires

The by-hand approach (SSH in and run parted / resize2fs on each node) feels natural, but in BCM it's wrong for several reasons:

1. It isn't persistent. The head node still has the old layout. The next time BCM reprovisions that node (a new node, a reinstall after a failure, or AUTO noticing the partitions don't match), it rebuilds the disk to the old size and your change is gone.
2. It causes configuration drift. Some nodes end up resized and some don't, so the nodes are no longer identical, which is exactly what BCM exists to prevent.
3. It doesn't scale. Doing it by hand on 32, 128, or 1000 DGX nodes is slow and error-prone.
4. It's risky. Resizing a live root partition (on top of an md mirror) by hand can corrupt the filesystem. BCM's reprovisioning builds the layout cleanly from a known definition.

22. Know your tools: which ones touch storage?

23. Weighing the common approaches

24. What the real procedure looks like

1. Drain the nodes in Slurm so no jobs are running on them:

   scontrol update nodename=dgx[01-04] state=drain reason="repartition OS disk"

2. Back up anything stored only on the nodes' local disks (ideally nothing: /raid is a cache).
3. Edit the disk layout on the head node for the category:

   cmsh
   % category use dgx-h100
   % set disksetup        # opens the disk setup XML in an editor: change the root partition size
   % commit

   (If the OS contents also need to change, update the software image on the head node too.)
4. Tell BCM the nodes need a full install on their next boot:

   % device use dgx01
   % set nextinstallmode full
   % commit

5. Reboot the nodes (BCM power control through the BMC; see Redfish & BMC). They PXE boot, and the node installer repartitions the disks with the new layout and copies the image.
6. Verify on a node (lsblk, df -h, cat /proc/mdstat), then resume the nodes in Slurm.

Any new or replacement DGX added to that category later gets the new layout automatically. That's the payoff of doing it centrally.

25. How to spot the right answers on similar questions

26. "Where should this data live?" chooser

27. Sample outputs

What the commands in this guide print. Expand a command to see a typical result and what to check in it.

NAME          SIZE TYPE  FSTYPE            MOUNTPOINT
nvme0n1       1.7T disk
├─nvme0n1p1   512M part  vfat              /boot/efi
└─nvme0n1p2   1.7T part  linux_raid_member
  └─md0       1.7T raid1 ext4              /
nvme1n1       1.7T disk
├─nvme1n1p1   512M part  vfat
└─nvme1n1p2   1.7T part  linux_raid_member
  └─md0       1.7T raid1 ext4              /
nvme2n1       3.5T disk  linux_raid_member
└─md1        27.9T raid0 ext4              /raid
nvme3n1       3.5T disk  linux_raid_member
└─md1        27.9T raid0 ext4              /raid
...
nvme9n1       3.5T disk  linux_raid_member
└─md1        27.9T raid0 ext4              /raid

Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          S6EXNE0T100001       SAMSUNG MZ1L21T9HCLS-00A07               1           1.92  TB /   1.92  TB    512   B +  0 B   GDC7302Q
/dev/nvme1n1          S6EXNE0T100002       SAMSUNG MZ1L21T9HCLS-00A07               1           1.92  TB /   1.92  TB    512   B +  0 B   GDC7302Q
/dev/nvme2n1          X1A0A0EXAMPL         KIOXIA KCM6DRUL3T84                      1           3.84  TB /   3.84  TB    512   B +  0 B   0107
/dev/nvme3n1          X1A0A0EXAMPM         KIOXIA KCM6DRUL3T84                      1           3.84  TB /   3.84  TB    512   B +  0 B   0107
...
/dev/nvme9n1          X1A0A0EXAMPT         KIOXIA KCM6DRUL3T84                      1           3.84  TB /   3.84  TB    512   B +  0 B   0106

Smart Log for NVME device:nvme2 namespace-id:ffffffff
critical_warning                        : 0
temperature                             : 38 C (311 Kelvin)
available_spare                         : 100%
available_spare_threshold               : 10%
percentage_used                         : 3%
endurance group critical warning summary: 0
data_units_read                         : 412,904,113
data_units_written                      : 98,233,710
host_read_commands                      : 3,301,882,019
host_write_commands                     : 905,771,402
controller_busy_time                    : 1,894
power_cycles                            : 41
power_on_hours                          : 6,512
unsafe_shutdowns                        : 4
media_errors                            : 0
num_err_log_entries                     : 0
Warning Temperature Time                : 0
Critical Composite Temperature Time     : 0

mn        : KIOXIA KCM6DRUL3T84
fr        : 0107
frmw      : 0x17
mntmt     : 273

Checks
------
Verify installed DIMM memory sticks ...................... Healthy
Number of logical CPU cores [224] ........................ Healthy
Verify chassis fan presence .............................. Healthy
...
Check SMART status of NVMe device nvme2n1 ................ Healthy
Check SMART status of NVMe device nvme9n1 ................ Healthy
Verify md0 RAID array status ............................. Healthy
Verify md1 RAID array status ............................. Healthy
Verify GPU 0000:1b:00.0 link speed [32GT/s] .............. Healthy
...

Health Summary
--------------
205 out of 205 checks are healthy
0 out of 205 checks are unhealthy
0 out of 205 checks are unknown
0 out of 205 checks are informational

Overall system status is Healthy

Personalities : [raid1] [raid0] [linear] [multipath] [raid6] [raid5] [raid4] [raid10]
md1 : active raid0 nvme9n1[7] nvme8n1[6] nvme7n1[5] nvme6n1[4] nvme5n1[3] nvme4n1[2] nvme3n1[1] nvme2n1[0]
      30004846592 blocks super 1.2 512k chunks

md0 : active raid1 nvme1n1p2[1] nvme0n1p2[0]
      1874715648 blocks super 1.2 [2/2] [UU]
      bitmap: 3/14 pages [12KB], 65536KB chunk

unused devices: <none>

/dev/md0:
           Version : 1.2
     Creation Time : Mon Oct  5 10:12:44 2026
        Raid Level : raid1
        Array Size : 1874715648 (1787.86 GiB 1919.71 GB)
     Used Dev Size : 1874715648 (1787.86 GiB 1919.71 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue Nov 10 19:05:12 2026
             State : clean, degraded
    Active Devices : 1
   Working Devices : 1
    Failed Devices : 1
     Spare Devices : 0

Consistency Policy : bitmap

              Name : dgx01:0  (local to host dgx01)
              UUID : 3a1f9c2e:5b7d4e10:9c8a7b6d:1e2f3a4b
            Events : 4821

    Number   Major   Minor   RaidDevice State
       0     259        2        0      active sync   /dev/nvme0n1p2
       -       0        0        1      removed

       1     259        5        -      faulty   /dev/nvme1n1p2

mdadm: set /dev/nvme1n1p2 faulty in /dev/md0
mdadm: hot removed /dev/nvme1n1p2 from /dev/md0

mdadm: added /dev/nvme1n1p2

$ cat /proc/mdstat
md0 : active raid1 nvme1n1p2[2] nvme0n1p2[0]
      1874715648 blocks super 1.2 [2/1] [U_]
      [===>.................]  recovery = 17.4% (326201344/1874715648) finish=41.2min speed=626104K/sec
      bitmap: 3/14 pages [12KB], 65536KB chunk

FS-Cache statistics
Cookies: idx=3 dat=12873 spc=0
Objects: alc=12850 nal=0 avl=12850 ded=0
ChkAux : non=0 ok=0 upd=0 obs=0
Pages  : mrk=1532876 unc=0
Acquire: n=12876 nul=0 noc=0 ok=12876 nbf=0 oom=0
Lookups: n=12850 neg=12840 pos=10 crt=12840 tmo=0
Invals : n=0 run=0
Updates: n=0 nul=0 run=0
Relinqs: n=24 nul=0 wcr=0 rtr=0
Retrvls: n=482391 ok=401220 wt=1203 nod=81171 nbf=0 int=0 oom=0
Retrvls: ops=482391 owt=2 abt=0
Stores : n=81171 ok=81171 agn=0 nbf=0 oom=0
Stores : ops=81171 run=162342 pgs=81171 rxd=81171 olm=0
...

Nov 10 08:12:01 dgx01 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1063-nvidia root=/dev/md0 ro intel_iommu=off console=tty0
Nov 10 08:12:01 dgx01 kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1063-nvidia root=/dev/md0 ro intel_iommu=off console=tty0
Nov 10 08:12:01 dgx01 kernel: DMAR: IOMMU disabled

nvidia_fs             266240  0

/dev/md1 on /raid type ext4 (rw,relatime,stripe=256,data=ordered)

 GDS release version: 1.7.2.11
 nvidia_fs version:  2.17 libcufile version: 2.12
 Platform: x86_64
 ============
 ENVIRONMENT:
 ============
 =====================
 DRIVER CONFIGURATION:
 =====================
 NVMe P2PDMA        : Unsupported
 NVMe               : Supported
 NVMeOF             : Unsupported
 SCSI               : Unsupported
 ...
 NFS                : Unsupported
 WekaFS             : Unsupported
 Userspace RDMA     : Unsupported
 --Mellanox PeerDirect : Enabled
 --rdma library        : Not Loaded (libcufile_rdma.so)
 =====================
 CUFILE CONFIGURATION:
 =====================
 properties.use_pci_p2pdma : false
 properties.use_compat_mode : true
 properties.force_compat_mode : false
 ...
 =========
 GPU INFO:
 =========
 GPU index 0 NVIDIA H100 80GB HBM3 bar:1 bar size (MiB):131072 supports GDS, IOMMU State: Disabled
 ...
 GPU index 7 NVIDIA H100 80GB HBM3 bar:1 bar size (MiB):131072 supports GDS, IOMMU State: Disabled
 ==============
 PLATFORM INFO:
 ==============
 IOMMU: disabled
 Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
 Platform verification succeeded

IoType: READ XferType: GPUD Threads: 8 DataSetSize: 10485760/10485760(KiB) IOSize: 1024(KiB) Throughput: 12.851403 GiB/sec, Avg_Latency: 607.312500 usecs ops: 10240 total_time 0.778164 secs

GDS Version: 1.7.2.11
NVFS statistics(ver: 4.0)
NVFS Driver(version: 2.17.5)
Mellanox PeerDirect Supported: True
IO stats: Disabled, peer IO stats: Disabled
Logging level: info

Active Shadow-Buffer (MiB): 0
Active Process: 0
Reads                           : err=0 io_state_err=0
Sparse Reads                    : n=0 io=0 holes=0 pages=0
Writes                          : err=0 io_state_err=0 pg-cache=0 pg-cache-fail=0 pg-cache-eio=0
Mmap                            : n=8 ok=8 err=0 munmap=8
Bar1-map                        : n=8 ok=8 err=0 free=8 callbacks=0 active=0 delay-frees=0
Error                           : cpu-gpu-pages=0 sg-ext=0 dma-map=0 dma-ref=0
Ops                             : Read=10240 Write=0 BatchIO=0

seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [R(16)][100.0%][r=48.6GiB/s][r=49.8k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=16): err= 0: pid=40211: Tue Nov 10 19:40:02 2026
  read: IOPS=49.1k, BW=47.9GiB/s (51.5GB/s)(320GiB/6676msec)
    slat (usec): min=8, max=1210, avg=21.40, stdev=9.85
    clat (usec): min=412, max=48121, avg=10392.11, stdev=2101.66
     lat (usec): min=431, max=48140, avg=10413.62, stdev=2101.70
    clat percentiles (usec):
     |  1.00th=[ 5604],  5.00th=[ 7111], 50.00th=[10421], 95.00th=[13829],
     | 99.00th=[16581], 99.90th=[21890], 99.99th=[37487]
  cpu          : usr=0.61%, sys=6.12%, ctx=318201, majf=0, minf=8457
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
...
Run status group 0 (all jobs):
   READ: bw=47.9GiB/s (51.5GB/s), 47.9GiB/s-47.9GiB/s (51.5GB/s-51.5GB/s), io=320GiB (344GB), run=6676-6676msec

Disk stats (read/write):
    md1: ios=327012/0, merge=0/0, ticks=3391201/0, in_queue=3391201, util=99.71%, aggrUtil=99.60%

# Exclude list for imageupdate (sync from image to running node)
- /cm/shared/*
- /home/*
- /proc/*
- /sys/*
- /dev/*
- /tmp/*
- /raid/*
- /var/log/*
- /var/spool/*
- /var/lib/docker/*
...

(no output on success)

$ sinfo -R
REASON                                   USER      TIMESTAMP           NODELIST
repartition OS disk                      root      2026-11-10T19:45:12 dgx[01-04]

[bcm-head01->device[dgx01]]% set nextinstallmode full
[bcm-head01->device*[dgx01*]]% commit
[bcm-head01->device[dgx01]]% get nextinstallmode
FULL
[bcm-head01->device[dgx01]]% get installmode
AUTO

28. Scenarios

29. Searchable reference

30. Flashcards (click to flip)

31. Exam traps

32. Practice quiz (34 questions)

33. Glossary