How to Enable NVIDIA GPUDirect Storage on Kubernetes

TL;DR

NVIDIA GPUDirect Storage, usually shortened to GDS, can move data between supported storage and GPU memory without routing the payload through a CPU bounce buffer. On Kubernetes, enabling the feature is not only a Helm setting. A working deployment depends on a supported GPU and operating system, the NVIDIA open GPU kernel module, a compatible filesystem or storage client, correct PCIe placement, and, for RDMA-backed network storage, a validated NVIDIA Network Operator configuration.

This tutorial builds the deployment in stages. It separates local NVMe from network-attached storage, installs Network Operator only where the data path requires it, installs GPU Operator with GDS enabled, validates nvidia-fs and cuFile, runs controlled gdsio comparisons, measures CPU use and I/O performance, and provides a rollback path that restores compatibility-mode or conventional CPU-mediated I/O.

Introduction

GPUDirect Storage is often described as a faster path from storage to the GPU. That description is accurate, but incomplete. The feature only produces useful results when the server, PCIe fabric, storage path, kernel modules, container runtime, and Kubernetes device exposure all agree on how peer-to-peer data movement should work.

A Helm installation can report success while the workload silently runs in compatibility mode. The application still completes, but data moves through system memory and the expected CPU, latency, or throughput improvement never appears. That is why the implementation must be treated as an end-to-end data-path validation exercise rather than a single operator feature flag.

This walkthrough uses a conservative production method. It validates the platform first, preserves a CPU-path fallback, enables the minimum required operators, and proves the result with a repeatable A/B test.

What You Will Accomplish

By the end of this tutorial, you will be able to:

• decide whether the target node and storage path are realistic GDS candidates
• distinguish local NVMe deployment requirements from RDMA-backed network storage requirements
• install NVIDIA Network Operator when the storage data path needs NVIDIA networking components
• install NVIDIA GPU Operator with GDS and the open GPU kernel module enabled
• verify the nvidia-fs kernel module, cuFile configuration, GPU topology, and RDMA resource exposure
• run gdsio tests that compare the direct GDS path with the CPU bounce-buffer path
• measure throughput, latency, IOPS, GPU activity, storage behavior, and host CPU consumption
• identify compatibility mode and common filesystem, kernel, topology, and container causes
• disable GDS safely without immediately removing the complete GPU or network stack

What GPUDirect Storage Changes

Traditional GPU data ingestion usually performs two major transfers. Storage first places data into system memory, then the CPU or a CPU-controlled copy moves that data into GPU memory. The path is functional, but it consumes memory bandwidth, CPU cycles, and additional copy operations.

GDS uses cuFile and supported kernel, driver, filesystem, storage, and networking components to establish a direct data path between storage and GPU memory. The CPU still coordinates control operations. It is not removed from the system. The important change is that the bulk payload does not have to bounce through CPU memory.

The reader should notice that GDS has two common physical patterns. Local NVMe can share a PCIe fabric with the GPU. Network storage can use an RDMA-capable NIC and a supported storage protocol or client. Both paths still depend on topology.

This distinction matters operationally. A successful file read does not prove that GDS was used. The same application can succeed through direct mode or compatibility mode. The validation process must therefore prove both functionality and path selection.

Prerequisites

Before starting, prepare the following:

• cluster-admin access for Helm releases, CustomResourceDefinitions, namespace labels, DaemonSets, and node-level operators
• a supported Kubernetes distribution and container runtime
• at least one supported NVIDIA GPU node with shell or out-of-band access for kernel and topology checks
• an approved local NVMe, NFS over RDMA, NVMe over Fabrics, or vendor filesystem test path
• a dedicated PersistentVolumeClaim that maps to the storage path being validated
• kubectl, Helm, lspci, numactl, sysstat, and standard Linux storage tools
• an approved GDS tools image containing gdscheck.py and gdsio that matches the installed CUDA and cuFile stack
• a maintenance window for any driver, kernel, BIOS, IOMMU, ACS, or storage-mount change
• a documented CPU-path fallback and the current Helm values for existing NVIDIA operators

Use a non-production node or a controlled pilot pool for the first implementation. GDS touches kernel modules, device exposure, storage clients, and potentially network drivers, so a shared production node is the wrong place to discover an unsupported combination.

Establish the Supported Baseline

The commands in this article use a July 25, 2026 research baseline. NVIDIA documentation changes frequently, so confirm the current support matrix and release notes before reproducing the versions shown here.

At this baseline, the conservative supported combination used in the walkthrough is:

Layer	Tutorial baseline	What to verify before deployment
Kubernetes integration	NVIDIA GPU Operator 26.3.3	Current operator release, supported Kubernetes distribution, container runtime, and operating system
GDS kernel path	NVIDIA open GPU kernel module	Do not assume that a CUDA-capable driver automatically satisfies current GDS requirements
Network integration	NVIDIA Network Operator 25.7.0 for the conservative GDS path	NVIDIA’s GDS-specific support table and the storage vendor’s tested combination
Local filesystems	XFS or EXT4, with supported mount behavior	Filesystem, mount options, kernel, block device, and cuFile support
Network storage	Supported NFS, NVMe over Fabrics, or vendor filesystem path	NIC, RDMA stack, storage client, mount options, server platform, and vendor certification
Test tooling	gdscheck.py, gdsio, host monitoring tools	Tool version matches the installed CUDA and GDS stack

There is a version nuance worth making explicit. Newer GPU Operator release notes describe general support for Network Operator 26.1.0, while the GDS-specific platform support table still lists Network Operator 25.7.0 for the documented Ubuntu GDS combinations. That does not prove that 26.1.0 is unsuitable. It means the supported GDS combination should be confirmed rather than inferred from general operator interoperability.

Run the Hardware, Kernel, and Storage Preflight

Do not install the operators until the target node passes a preflight review. This stage prevents Kubernetes from hiding a server-design problem behind healthy DaemonSets.

Confirm the GPU, Driver, Kernel, and Operating System

Run these commands on each candidate GPU node:

nvidia-smi --query-gpu=name,pci.bus_id,driver_version --format=csv
uname -r
cat /etc/os-release
nvidia-smi topo -m
lspci -tv

Validate the exact GPU family, operating system, kernel, and container runtime against the current GPU Operator and GDS support matrices. Do not treat a working CUDA workload as proof of GDS support. GDS has additional kernel-module, filesystem, and topology dependencies.

The nvidia-smi topo -m output should be reviewed for the relationship among the GPU, NIC, and CPU NUMA nodes. The lspci -tv tree provides a second view of the PCIe hierarchy and can reveal whether devices sit under a common switch or cross a root complex.

Inspect the Filesystem and Mount Options

Replace /data with the actual test mount:

findmnt -T /data -o SOURCE,FSTYPE,OPTIONS,TARGET
lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINTS,MODEL,TRAN
stat -f -c 'filesystem=%T block_size=%S' /data

For a local block-device path, current NVIDIA guidance identifies XFS and EXT4 as supported baseline filesystems. EXT4 must use ordered data mode. A mount using data=writeback is not an acceptable GDS baseline.

Check the effective EXT4 mode when applicable:

findmnt -T /data -no FSTYPE,OPTIONS

For network storage, record the filesystem client, mount protocol, transport, NIC, interface, and storage target. The label “NFS” is not enough. An NFS over TCP mount and an NFS over RDMA mount represent different data paths and prerequisites.

Review IOMMU and PCIe ACS Carefully

Inspect the current boot and PCIe state:

cat /proc/cmdline
dmesg | grep -Ei 'iommu|dmar|amd-vi'
lspci -vvv | grep -B 2 -A 12 -E 'ACSCtl|Access Control Services'

NVIDIA’s GDS guidance notes that IOMMU and PCIe Access Control Services can affect peer-to-peer behavior and performance. On some non-DGX x86 systems, the documented recommendation is to disable IOMMU for the best GDS functionality and performance. Some validated DGX platforms support passthrough behavior instead.

This is not a casual tuning change. IOMMU and ACS can be part of the platform’s virtualization, isolation, and security model. Do not change either control only because a benchmark is slow. Confirm the server vendor’s supported BIOS configuration, understand the effect on other workloads, schedule a maintenance window, and preserve the original boot and firmware settings for rollback.

Treat PCIe Topology as a Design Requirement

PCIe topology is not a minor optimization. It determines whether the direct path is actually direct at the physical layer.

The best-performing local pattern normally places the GPU and NVMe device beneath the same PCIe switch. For network storage, the preferred pattern places the GPU and RDMA NIC beneath the same switch. Cross-socket or cross-root-complex paths can add hops, consume inter-socket bandwidth, and reduce the benefit of GDS.

Use this review order:

1. Map GPU PCIe addresses with nvidia-smi.
2. Map NIC and NVMe addresses with lspci.
3. Compare the devices in lspci -tv.
4. Compare GPU and NIC relationships in nvidia-smi topo -m.
5. Confirm NUMA placement with numactl --hardware and device-local NUMA files under /sys.
6. Pin the benchmark to the intended node and GPU before drawing conclusions.

Example NUMA checks:

numactl --hardware
cat /sys/bus/pci/devices/0000:65:00.0/numa_node
cat /sys/bus/pci/devices/0000:ca:00.0/numa_node

Replace the PCIe addresses with the GPU, NIC, or NVMe addresses from the target server.

A topology mismatch does not always make GDS nonfunctional. It can make the measured result disappointing. That difference is important. Compatibility testing answers whether the path works. Topology testing answers whether the server is designed to benefit from it.

Decide Between Local and Network Storage

The Network Operator is not a universal prerequisite for every GDS deployment. It is needed when the network-attached data path depends on NVIDIA networking drivers, RDMA device plugins, secondary networking, or related components. A local NVMe path can use GPU Operator and GDS without installing Network Operator.

Storage pattern	Network Operator	Primary dependency	Topology priority	Main operational risk
Local NVMe with XFS or EXT4	Usually no	Supported filesystem, block device, nvidia-fs, and local PCIe path	GPU and NVMe beneath the same switch where possible	Mount options, ACS, IOMMU, or an indirect PCIe route
NFS over RDMA	Usually yes	RDMA NIC, validated OFED or DOCA stack, NFS client and server support	GPU and NIC beneath the same switch where possible	Falling back to TCP or exposing the wrong RDMA devices
NVMe over Fabrics with RDMA	Usually yes	RDMA transport, target and initiator compatibility, multipath design	GPU and NIC locality plus storage-network design	Queue, failover, multipath, or driver mismatch
Vendor parallel filesystem	Depends	Vendor-certified GDS client and exact version combination	Vendor topology and affinity guidance	Assuming generic GDS support applies to every client release

This branch should be decided before installation:

Preserve a Baseline and Rollback Point

Before changing the cluster, collect a conventional I/O baseline and save the current state:

helm list -A
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get daemonsets -A
kubectl get crds | grep -E 'nvidia|mellanox'

Also record:

• current GPU driver and CUDA versions
• current kernel and boot parameters
• current NIC and storage-client versions
• the storage mount definition and mount options
• the existing Node Feature Discovery owner, if any
• current CPU, throughput, latency, and IOPS for a representative workload
• the Helm values for any existing GPU or Network Operator release

Use helm get values -n -a to preserve the complete release values. Store the output in the change record rather than relying on Helm history alone.

Prepare the NVIDIA Helm Repository

Use the NVIDIA NGC Helm repository or an approved internal mirror. Keep the repository address in a shell variable so the article body remains portable and the cluster team can substitute an internal artifact source.

export NVIDIA_HELM_REPOSITORY=''

helm repo add nvidia "$NVIDIA_HELM_REPOSITORY"
helm repo update
helm search repo nvidia/network-operator --versions | head
helm search repo nvidia/gpu-operator --versions | head

Do not install the first chart version returned by a repository search. Compare it with the GDS support matrix, GPU Operator release notes, Network Operator release notes, and the storage vendor’s qualified combination.

Install Network Operator for an RDMA Storage Path

Skip this section for a local NVMe-only deployment unless another platform requirement needs Network Operator.

Create and Label the Namespace

Network drivers and device plugins require privileged host integration. Label the dedicated namespace accordingly:

kubectl create namespace nvidia-network-operator --dry-run=client -o yaml | kubectl apply -f -

kubectl label namespace nvidia-network-operator \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/audit=privileged \
  pod-security.kubernetes.io/warn=privileged \
  --overwrite

Install the Conservative GDS Baseline

helm upgrade --install network-operator nvidia/network-operator \
  --namespace nvidia-network-operator \
  --version v25.7.0 \
  --wait \
  --timeout 20m

This chart can also deploy Node Feature Discovery. Only one component should own NFD. In this walkthrough, Network Operator owns it, and the later GPU Operator installation disables its bundled NFD deployment. In a cluster with an existing NFD installation, add the Network Operator chart value that disables NFD and retain the existing owner.

Check the operator before adding a NIC policy:

kubectl get pods -n nvidia-network-operator -o wide
kubectl get daemonsets -n nvidia-network-operator
kubectl get crds | grep -E 'nicclusterpolicy|mellanox'

Create the NIC and RDMA Device Policy

The exact policy depends on the NIC, operating system, storage protocol, and validated DOCA or OFED release. The example below shows the shape of a policy without pretending that one driver build fits every cluster.

apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  name: nic-cluster-policy
spec:
  ofedDriver:
    image: doca-driver
    repository: nvcr.io/nvidia/mellanox
    version: ""
    upgradePolicy:
      autoUpgrade: false
  rdmaSharedDevicePlugin:
    image: k8s-rdma-shared-dev-plugin
    repository: nvcr.io/nvidia/mellanox
    version: ""
    config: |
      {
        "configList": [
          {
            "resourceName": "rdma_shared_device_a",
            "rdmaHcaMax": 1000,
            "selectors": {
              "ifNames": [""]
            }
          }
        ]
      }

Save the manifest as nic-cluster-policy.yaml, replace all placeholders, then apply it:

kubectl apply -f nic-cluster-policy.yaml
kubectl get nicclusterpolicy
kubectl describe nicclusterpolicy nic-cluster-policy
kubectl get pods -n nvidia-network-operator -o wide

The resourceName becomes part of the Kubernetes extended resource presented to workloads. Use a stable, environment-specific name and document which physical interfaces it selects.

Inspect the selected node’s allocatable resources:

kubectl describe node  | sed -n '/Allocatable:/,/System Info:/p'

Do not continue until the expected RDMA resource is visible and the driver pods are healthy on the intended GPU nodes.

Install GPU Operator with GDS Enabled

Create and Label the Namespace

kubectl create namespace gpu-operator --dry-run=client -o yaml | kubectl apply -f -

kubectl label namespace gpu-operator \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/audit=privileged \
  pod-security.kubernetes.io/warn=privileged \
  --overwrite

Install GPU Operator 26.3.3

The following installation enables GDS, explicitly selects the open GPU kernel module, and prevents a second NFD installation because Network Operator owns NFD in this design:

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --version v26.3.3 \
  --set gds.enabled=true \
  --set driver.kernelModuleType=open \
  --set nfd.enabled=false \
  --wait \
  --timeout 30m

For a local NVMe deployment without Network Operator, remove --set nfd.enabled=false unless NFD is already managed elsewhere.

For a cluster with a preinstalled NVIDIA driver, add --set driver.enabled=false only after confirming that the host driver uses the supported open kernel module and matches the GDS, CUDA, and operator requirements. Disabling operator driver management does not remove the requirement for a correct host driver.

Inspect the ClusterPolicy and pods:

kubectl get clusterpolicy
kubectl describe clusterpolicy cluster-policy
kubectl get pods -n gpu-operator -o wide
kubectl get daemonsets -n gpu-operator

Wait for the operator components on the target node to reach their expected ready state. A healthy control plane does not by itself prove that the GDS data path is active, but unhealthy operator pods must be resolved before workload testing.

Validate the GDS Components on the Node

Run the following checks on the target node after the operator deployment completes.

Verify the NVIDIA Kernel Modules

lsmod | grep -E '^nvidia|nvidia_fs|nvidia_peermem'
modinfo nvidia_fs 2>/dev/null | head -n 20

The key GDS signal is the nvidia_fs module. RDMA deployments may also require nvidia_peermem, depending on the selected network and storage stack.

Check for initialization errors:

dmesg | grep -Ei 'nvidia-fs|nvidia_fs|peermem|rdma|iommu|acs' | tail -n 100

Run the cuFile Platform Check

Inside a validated GDS tools container, or on a host where the matching CUDA GDS tools are installed:

GDSCHECK=$(find /usr/local/cuda* -type f -path '*/gds/tools/gdscheck.py' | head -n 1)

printf 'gdscheck=%s\n' "$GDSCHECK"
"$GDSCHECK" -p

Review the output for:

• driver and nvidia-fs availability
• supported filesystems and block devices
• peer-to-peer DMA status
• PCIe ACS warnings
• the effective cuFile configuration
• compatibility-mode settings
• storage-specific support information

Do not reduce this output to a single pass or fail line. Save the complete result with the test evidence because it helps explain why two servers with the same GPU can behave differently.

Confirm the Physical Topology Again

nvidia-smi topo -m
lspci -tv

The operator deployment does not change the server’s physical topology. Repeating the check at this stage ties the installed data path to the hardware map used for performance interpretation.

Expose Only the Devices the Test Workload Needs

A production GDS container should not receive every host device merely because a privileged proof of concept worked.

Use the Kubernetes device plugins and CDI integration supplied by the operators wherever possible:

• request nvidia.com/gpu for the GPU
• request the specific RDMA extended resource for a network-storage path
• omit the RDMA resource for local NVMe testing
• mount /run/udev read-only only when the tools or storage client require device discovery
• add IPC_LOCK when the RDMA or memory-registration path requires it
• avoid manually mounting all of /dev/infiniband
• avoid privileged: true for production workloads
• keep spec.hostUsers at its default behavior because current GPU Operator runtime hooks do not support pods configured with hostUsers: false

GPU Operator 26.3.3 also contains a relevant device-exposure correction. Earlier behavior could inject broad InfiniBand verb devices when GDS or MOFED feature flags were present. The current release infers the required device exposure from loaded modules. That is another reason to use a current maintenance release instead of copying an older chart version from a previous lab.

Create a Least-Privilege gdsio Test Job

The image used below must contain gdscheck.py, gdsio, and a CUDA or cuFile userspace stack compatible with the node driver. Build and approve that image through the normal container supply-chain process. Do not use an arbitrary public image for a privileged storage test.

apiVersion: batch/v1
kind: Job
metadata:
  name: gdsio-validation
  namespace: gds-test
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app: gdsio-validation
    spec:
      restartPolicy: Never
      nodeSelector:
        kubernetes.io/hostname: ""
      containers:
        - name: gdsio
          image: ""
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
            runAsUser: 0
            capabilities:
              add:
                - IPC_LOCK
          resources:
            limits:
              nvidia.com/gpu: 1
              rdma/rdma_shared_device_a: 1
          command:
            - /bin/bash
            - -lc
          args:
            - |
              set -euo pipefail
              GDSCHECK=$(find /usr/local/cuda* -type f -path '*/gds/tools/gdscheck.py' | head -n 1)
              GDSIO=$(find /usr/local/cuda* -type f -path '*/gds/tools/gdsio' | head -n 1)
              "$GDSCHECK" -p
              "$GDSIO" -f /data/gds-test.bin -d 0 -w 8 -s 8G -i 1M -x 0 -I 1
              "$GDSIO" -f /data/gds-test.bin -d 0 -w 8 -s 8G -i 1M -x 0 -I 0 -T 120
          volumeMounts:
            - name: data
              mountPath: /data
            - name: udev
              mountPath: /run/udev
              readOnly: true
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: gds-data
        - name: udev
          hostPath:
            path: /run/udev
            type: Directory

For local NVMe, remove the rdma/rdma_shared_device_a resource. For network storage, replace it with the exact resource name created by the NIC policy. The PVC must map to the storage path being validated, not a generic default StorageClass.

Create the namespace, apply the job, and watch the logs:

kubectl create namespace gds-test --dry-run=client -o yaml | kubectl apply -f -

kubectl label namespace gds-test \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/audit=privileged \
  pod-security.kubernetes.io/warn=privileged \
  --overwrite

kubectl apply -f gdsio-validation.yaml
kubectl logs -n gds-test job/gdsio-validation -f
kubectl describe job -n gds-test gdsio-validation

The dedicated namespace is labeled for privileged host integration because the validation pod can require a hostPath mount and an added capability. Restrict RBAC to the test team and keep the pod-level privileged setting disabled. A privileged namespace is not a reason to remove the container’s explicit security controls.

If the job remains pending, inspect resource allocation before changing security settings. A missing RDMA resource, GPU resource, PVC binding, or node selector should not be “fixed” by making the container privileged.

Test GDS with gdsio

gdsio is useful because it can exercise multiple paths with the same file, I/O size, thread count, GPU, and duration.

The most important path values for this comparison are:

gdsio path	Meaning	Why it matters
-x 0	Storage to GPU through GDS	Tests the intended direct path
-x 1	Storage to CPU memory	Establishes a conventional host-memory path
-x 2	Storage to CPU memory, then GPU	Represents the CPU bounce-buffer comparison
-x 4	Cached CPU path to GPU	Useful for cache-aware investigation, not a direct GDS comparison

Run gdsio -h in the installed tools image and confirm the switches for that exact version before automation.

Create the Test File

Write the file before performing directory or file reads:

GDSIO=$(find /usr/local/cuda* -type f -path '*/gds/tools/gdsio' | head -n 1)

"$GDSIO" \
  -f /data/gds-test.bin \
  -d 0 \
  -w 8 \
  -s 8G \
  -i 1M \
  -x 0 \
  -I 1

Use a test file that is larger than the useful host page cache and representative of the production data path. An 8 GiB file may be too small for a high-memory server, so adjust the value to the node and storage system.

Compare Direct GDS with the Bounce-Buffer Path

Run the direct-path read:

"$GDSIO" \
  -f /data/gds-test.bin \
  -d 0 \
  -w 8 \
  -s 8G \
  -i 1M \
  -x 0 \
  -I 0 \
  -T 120

Then run the CPU bounce-buffer comparison with every other parameter unchanged:

"$GDSIO" \
  -f /data/gds-test.bin \
  -d 0 \
  -w 8 \
  -s 8G \
  -i 1M \
  -x 2 \
  -I 0 \
  -T 120

For a small-block random-read profile:

"$GDSIO" \
  -f /data/gds-test.bin \
  -d 0 \
  -w 32 \
  -s 8G \
  -i 4K \
  -x 0 \
  -I 2 \
  -T 120

Do not compare results from different files, mounts, GPUs, NUMA placements, thread counts, I/O sizes, or cache states. GDS should be evaluated as a controlled path change, not as two unrelated benchmark runs.

Measure CPU Use, Latency, and Throughput

The primary success criterion should not be “the GDS number is larger.” GDS can be valuable because it reduces CPU consumption, improves latency consistency, or preserves throughput while freeing host resources for data preparation and orchestration.

Capture Process CPU Consumption

Run the test in the background and observe the process:

"$GDSIO" \
  -f /data/gds-test.bin \
  -d 0 \
  -w 8 \
  -s 8G \
  -i 1M \
  -x 0 \
  -I 0 \
  -T 120 &

GDSIO_PID=$!
pidstat -u -r -p "$GDSIO_PID" 1
wait "$GDSIO_PID"

Repeat the same collection for -x 2.

Observe the Host, Storage Device, and GPU

Use separate terminals or a monitoring sidecar:

mpstat -P ALL 1
iostat -xz 1
nvidia-smi dmon -d 1

For network storage, add the cluster’s NIC, RDMA, switch, and storage-system telemetry. A host-only view cannot distinguish a GPU bottleneck from a congested fabric or storage target.

Enable cuFile Statistics for Validation

The cuFile configuration can report operation counts, throughput, and latency statistics. Merge the following setting into the existing test container’s /etc/cufile.json; do not replace the full configuration with this fragment:

{
  "profile": {
    "cufile_stats": 2
  }
}

Use a disposable test image layer or ConfigMap-based configuration so the benchmark does not mutate an operator-managed host file.

Record Results in a Repeatable Format

Test	Path	I/O pattern	I/O size	Threads	Throughput	IOPS	Average latency	Host CPU	GPU or PCIe notes
A	GDS -x 0	Sequential read	1 MiB	8	Record	Record	Record	Record	Record
B	Bounce -x 2	Sequential read	1 MiB	8	Record	Record	Record	Record	Record
C	GDS -x 0	Random read	4 KiB	32	Record	Record	Record	Record	Record
D	Bounce -x 2	Random read	4 KiB	32	Record	Record	Record	Record	Record

A production decision should use several runs, warm-up periods, representative file sizes, and workload-specific acceptance criteria. One peak throughput number is not enough.

Recognize Compatibility Mode

Compatibility mode is an intentional fallback that lets cuFile perform I/O through a CPU-mediated POSIX path when direct GDS is not available. It is valuable for continuity, but it can hide an incomplete deployment.

Use these checks:

GDSCHECK=$(find /usr/local/cuda* -type f -path '*/gds/tools/gdscheck.py' | head -n 1)
"$GDSCHECK" -p

grep -n 'allow_compat_mode' /etc/cufile.json
lsmod | grep nvidia_fs

Indicators that the workload may be using compatibility mode include:

• nvidia-fs is not loaded
• the filesystem or mount is unsupported or denylisted
• cuFile trace output identifies a POSIX I/O mode
• the application succeeds even after the direct GDS module is intentionally removed
• CPU use and memory-copy behavior resemble the -x 2 comparison

To force a temporary compatibility-mode test without changing the cluster-wide operator state:

GDSIO=$(find /usr/local/cuda* -type f -path '*/gds/tools/gdsio' | head -n 1)

CUFILE_FORCE_COMPAT_MODE=true \
  "$GDSIO" \
  -f /data/gds-test.bin \
  -d 0 \
  -w 8 \
  -s 8G \
  -i 1M \
  -x 0 \
  -I 0 \
  -T 120

This is useful for proving that the workload has a functional fallback. It is not evidence that the direct path is working.

Troubleshoot the Most Common Failure Modes

Symptom	Likely area	Diagnostic action	Corrective direction
GPU Operator is healthy, but nvidia_fs is absent	Driver or GDS component mismatch	Inspect GPU Operator pods, ClusterPolicy, driver type, and node kernel logs	Confirm open kernel module support, chart values, kernel compatibility, and GDS driver deployment
gdscheck.py -p reports an unsupported filesystem	Filesystem or client support	Check findmnt, client version, and current GDS support matrix	Use a supported filesystem or certified client and mount combination
EXT4 path falls back or fails	Mount mode	Inspect EXT4 mount options	Use supported ordered data mode, then remount during a maintenance window
Network test pod is pending	Resource exposure	Inspect node allocatable resources and pod events	Correct the RDMA resource name, NIC selector, GPU request, or node placement
Network workload runs but no RDMA benefit appears	Transport fallback	Verify mount transport, RDMA counters, NIC selection, and storage-server support	Correct the RDMA client, server, mount, or fabric configuration
GDS works but throughput is lower than expected	PCIe or NUMA topology	Compare nvidia-smi topo -m, lspci -tv, NUMA placement, and ACS state	Move the workload to a better device pairing or redesign the server placement
High CPU use remains with -x 0	Compatibility mode or surrounding pipeline work	Compare cuFile mode, process CPU, page cache, decompression, and preprocessing	Prove direct mode first, then isolate non-I/O CPU work
Container only works with privileged: true	Device or permission model	Compare injected GPU, nvidia-fs, RDMA, udev, and capability access	Request exact device resources and capabilities instead of broad privilege
Kernel instability or I/O errors appear after an upgrade	Known release interaction	Compare kernel, CUDA, GDS, DOCA, storage-client, and release-note versions	Move to a validated combination or use compatibility mode until corrected

Filesystem, Kernel, and Release Caveats

GDS support is version-sensitive. As of the July 25, 2026 baseline used for this article, current NVIDIA release notes include caveats that should be treated as deployment gates, not footnotes.

Examples include:

• a current WEKA filesystem limitation with CUDA 13.1 and later
• DGX Spark support limited to compatibility mode, with nvidia-fs not loaded
• a Linux 6.17 and later interaction involving nvidia-fs and an NVMe SGL path that can cause I/O failures or kernel hangs, with a documented alternative path
• a DOCA 3.3.x interaction that does not support a particular GDS NVMe and nvidia-fs combination, with an earlier DOCA branch identified as the workaround

These examples will age. The operational lesson is permanent: pin the complete combination, including GPU driver, open kernel module, CUDA, GDS, kernel, Network Operator, DOCA or OFED, storage client, filesystem, and server firmware. Test the combination as a unit before rolling it across a GPU fleet.

Also avoid mixing buffered I/O and O_DIRECT against overlapping regions of the same file. Even when both modes are individually supported, mixing them can create coherency and performance problems that make benchmark results difficult to interpret.

Safely Disable GDS and Roll Back

Rollback should preserve GPU scheduling and network stability wherever possible. Disabling the complete GPU or Network Operator stack is usually too disruptive as a first response.

Stop and Preserve Evidence

kubectl delete job -n gds-test gdsio-validation --ignore-not-found
kubectl get pods -A -o wide
kubectl get events -A --sort-by='.lastTimestamp' | tail -n 200

Save operator logs, gdscheck.py -p output, kernel logs, benchmark results, topology maps, and Helm values before changing the state.

Disable GDS in GPU Operator

helm upgrade gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --version v26.3.3 \
  --reuse-values \
  --set gds.enabled=false \
  --wait \
  --timeout 30m

Confirm that GDS-related pods are removed or reconciled:

kubectl get pods -n gpu-operator | grep -E 'gds|nvidia-fs' || true

On each node, confirm the module state:

lsmod | grep nvidia_fs || true

Do not force-remove an in-use kernel module. Cordon and drain one node at a time, stop GDS consumers, and reboot during a maintenance window if the module remains referenced.

Use Compatibility Mode as a Workload-Level Fallback

Set CUFILE_FORCE_COMPAT_MODE=true for the affected workload when it must continue before the operator-level issue is corrected. This preserves functionality through CPU-mediated I/O and gives the team time to resolve the compatibility problem without pretending that direct GDS remains active.

Remove Network Operator Only When It Is Truly Unused

Do not uninstall Network Operator merely because GDS is disabled. Other RDMA, SR-IOV, secondary network, or storage workloads may depend on it.

When the operator is dedicated to this test and no workload depends on its resources:

kubectl delete nicclusterpolicy nic-cluster-policy --ignore-not-found
helm uninstall network-operator -n nvidia-network-operator

Drain affected nodes and follow the current Network Operator uninstallation procedure before removing host drivers. Restore the original NIC driver, boot, IOMMU, ACS, and mount configuration only through the server and storage vendors’ supported process.

Production Readiness Checklist

Before declaring the GDS path ready for application teams, confirm:

• the GPU, server, operating system, kernel, container runtime, and Kubernetes distribution are supported
• the NVIDIA open GPU kernel module is active
• nvidia-fs is loaded on every intended node
• the filesystem, storage client, and mount options are supported
• the GPU-to-NVMe or GPU-to-NIC PCIe path is documented
• IOMMU and ACS decisions are approved and reversible
• the Network Operator version and component matrix are validated for RDMA storage
• only one NFD deployment owns node discovery
• the test pod requests exact GPU and RDMA resources
• the test succeeds without broad privileged access
• gdscheck.py -p evidence is retained
• direct -x 0 and bounce-buffer -x 2 tests use identical parameters
• CPU, throughput, latency, IOPS, GPU, NIC, and storage telemetry are captured
• compatibility mode is detectable and intentionally tested
• a workload fallback and operator rollback are documented
• known release-note caveats are reviewed at every upgrade

Conclusion

Enabling NVIDIA GPUDirect Storage on Kubernetes is not primarily an operator-installation task. It is a data-path engineering task that Kubernetes helps package and repeat.

The reliable sequence is to validate the server and storage design first, install Network Operator only when the storage path requires RDMA integration, install GPU Operator with GDS and the open kernel module enabled, prove nvidia-fs and cuFile readiness, and compare direct and bounce-buffer paths under identical conditions. The result should be judged through CPU use, latency, throughput, IOPS, and operational stability, not through a single peak benchmark number.

The most important production control is a tested fallback. Compatibility mode and a Helm-level GDS disablement path let the application continue while the platform team resolves a kernel, filesystem, network, or topology issue. That makes GDS an engineered capability rather than a fragile performance experiment.

External References