How to Configure GPUDirect RDMA and Prove Multi-Node GPU Performance with NCCL

TL;DR

A working GPU driver, an RDMA device inside a pod, and a completed NCCL test do not prove that GPUDirect RDMA is working efficiently. Production validation must prove the complete path: GPU topology, GPU-to-NIC affinity, PCIe peer access, IOMMU and ACS behavior, RDMA fabric health, container resource exposure, NCCL transport selection, and repeatable multi-node AllReduce bandwidth.

This runbook uses a layered method. First, map the physical topology. Second, prove the raw RDMA path with host and GPU memory. Third, run nccl-tests with controlled interface selection and evidence-grade logging. Finally, compare algorithm bandwidth, bus bandwidth, scaling behavior, transport selection, and error counters against a topology-specific baseline.

The current research baseline for this runbook is NVIDIA Network Operator 26.4.0 and NCCL 2.30.7. Treat those versions as a verified publication baseline, not a command to upgrade an existing cluster without compatibility testing.

Introduction

Multi-node GPU performance problems are often diagnosed from the top of the stack. An engineer runs all_reduce_perf, sees disappointing bus bandwidth, and begins changing NCCL environment variables.

That is usually the wrong starting point.

NCCL sits above several independent failure domains. A poor AllReduce result can be caused by the GPU topology, a remote NUMA path, PCIe Access Control Services, an incorrect HCA, an unhealthy InfiniBand rail, a lossy RoCE configuration, missing GPU memory registration support, a container runtime limitation, an MPI launch problem, or a silent fallback to sockets. Tuning NCCL before proving those layers can make the problem harder to understand.

NVIDIA’s current troubleshooting structure reflects this reality. It separates GPU topology, GPU-to-NIC communication, ACS, networking, runtime and MPI, performance tuning, logging, and multi-node NVLink into distinct diagnostic areas. NVIDIA AI Cluster Runtime follows the same evidence-oriented model by using NCCL AllReduce to measure aggregate multi-node bus bandwidth as a performance validation signal.

This runbook turns that model into an operational sequence for Kubernetes-based GPU clusters using the NVIDIA Network Operator, although the host-level topology and RDMA tests also apply to bare-metal and scheduler-managed clusters.

Scenario and Definition of Success

The target scenario is a cluster with two or more GPU nodes connected by InfiniBand or RoCE. Each node has one or more NVIDIA GPUs and one or more RDMA-capable NICs or HCAs. Workloads run in containers and use NCCL collectives across nodes.

Typical symptoms include:

• all_reduce_perf completes but bus bandwidth is much lower than expected.
• One node pair performs well while another pair performs poorly.
• Single-node NCCL is healthy but multi-node scaling collapses.
• Performance changes when a different GPU or NIC is selected.
• NCCL logs show Socket transport when RDMA was expected.
• The job hangs during communicator initialization.
• ibv_reg_mr, ibv_create_qp, timeout, retry, or connection errors appear.
• RoCE counters show drops, pause behavior, congestion notifications, or retransmission symptoms.
• Performance is stable with one GPU per node but falls when all GPUs participate.

A successful validation does not require one universal bandwidth number. It requires a defensible evidence chain.

Layer	Required proof	Failure interpretation
GPU topology	Expected GPU peer paths and PCIe or NVLink relationships	Local topology or peer-access issue
GPU-to-NIC path	Intended GPU and HCA share the expected PCIe and NUMA locality	Cross-socket or root-complex penalty
RDMA enablement	Compatible driver path, RDMA devices, and GPU memory registration	GPUDirect path unavailable or incomplete
Fabric	Expected link state, rate, latency, bandwidth, and clean counters	Network or physical fabric issue
Container	GPU, RDMA resource, shared memory, topology, and permissions exposed	Runtime or pod specification issue
NCCL transport	Intended IB or verbs transport selected, without socket fallback	Interface or plugin selection issue
Collective performance	Stable plateau bandwidth, correct results, and expected scaling	End-to-end performance issue
Repeatability	Comparable results across repeated runs and node pairs	Intermittent fault, congestion, or thermal issue

The acceptance threshold should be defined for the exact server, GPU, NIC, rail, PCIe, and software configuration. Do not compare a two-rail eight-GPU node against a generic vendor peak or a result from a different topology.

The End-to-End GPUDirect RDMA Path

The most important mental model is that NCCL performance is constrained by a physical path, not by a single software component.

The path to notice is the relationship between the GPU, the PCIe hierarchy, the CPU socket, and the selected NIC. A fast NIC does not compensate for a poor GPU-to-NIC path. A healthy fabric does not compensate for traffic crossing the inter-socket link before it reaches the HCA. GPUDirect RDMA is valuable because it avoids unnecessary host-memory staging, but it still depends on a topology that permits efficient PCIe peer traffic.

Prerequisites and Safety Boundaries

Complete these checks before changing drivers, BIOS settings, Network Operator policy, or NCCL configuration.

Required access and tooling

You need:

• Administrative access to the Kubernetes cluster.
• Root or equivalent diagnostic access on representative GPU nodes.
• A maintenance window for driver, IOMMU, ACS, or firmware changes.
• A versioned NCCL test image or a reproducible nccl-tests build.
• A working MPI or scheduler launch method for multi-node execution.
• Access to switch telemetry for the InfiniBand or RoCE fabric.
• A known-good node pair, rail, or previous benchmark for comparison when available.
• A rollback copy of Helm values, custom resources, network attachments, and node settings.

Version baseline

For this publication, the verified documentation baseline is:

• NVIDIA Network Operator 26.4.0.
• NCCL 2.30.7.
• GPU Operator 25.3.x or newer for the current Network Operator GPUDirect integration guidance.
• DOCA-OFED 5.5-1.0.3.2 or newer as the published minimum compatibility floor.

A minimum supported version is not automatically the best production version. Use the platform support matrix for the GPU Operator, Network Operator, Kubernetes release, kernel, GPU driver, DOCA-OFED release, and container runtime that you actually operate.

Production safety rules

Do not make these changes as casual troubleshooting experiments:

• Do not disable IOMMU globally without understanding virtualization, device assignment, security, and support implications.
• Do not disable ACS dynamically across every PCIe bridge without a vendor-approved change plan.
• Do not force NCCL_NET_GDR_LEVEL=SYS to make an unsuitable remote path appear acceptable.
• Do not raise InfiniBand timeouts or retry counts to conceal a bad rail.
• Do not persist debug-only NCCL variables in production workload manifests.
• Do not upgrade the GPU driver, DOCA-OFED, Network Operator, GPU Operator, CUDA, NCCL, and MPI stack simultaneously.
• Do not accept a benchmark that completes with correctness errors, transport fallback, or growing fabric counters.

Map PCIe Topology and GPU-to-NIC Affinity

Begin on every node that will participate in the benchmark. The goal is to identify which GPU should use which HCA and whether the path crosses a PCIe host bridge or CPU socket.

Capture the GPU topology

nvidia-smi topo -m
nvidia-smi topo -p2p p
nvidia-smi topo -p2p n

nvidia-smi topo -m shows the relationship between GPUs, CPUs, and visible NICs. Common path labels indicate progressively more expensive paths:

• PIX: one PCIe switch.
• PXB: multiple PCIe bridges without crossing the host bridge.
• PHB: traffic crosses a PCIe host bridge.
• NODE: traffic crosses host bridges within one NUMA node.
• SYS: traffic crosses the CPU interconnect between NUMA nodes or sockets.
• NV#: GPUs communicate through the indicated number of bonded NVLink connections.

The exact best path depends on the server design. In general, a GPU and NIC connected through the same local PCIe switch or root complex are preferable to a path that crosses the CPU interconnect.

The peer matrices confirm whether CUDA reports PCIe and NVLink peer access between GPU pairs. A topology label alone is not proof that peer access is enabled.

Map NICs, HCAs, PCIe devices, and NUMA nodes

lspci -tv
lspci -nn | grep -Ei 'NVIDIA|Mellanox|Ethernet|InfiniBand'
ibdev2netdev
rdma link show
numactl --hardware

cat /sys/class/net/ens1f0/device/numa_node
readlink -f /sys/class/infiniband/mlx5_0/device

Replace ens1f0 and mlx5_0 with the actual network interface and HCA names.

A returned NUMA node of -1 means the kernel is not reporting a useful locality value for that device. Do not assume that means the device is universally local. Confirm the PCIe path with lspci, the server block diagram, and nvidia-smi topo -m.

Build an affinity map

Create a simple node-specific table before running NCCL.

GPU	Preferred HCA and port	NUMA node	Topology path	Alternate path
GPU 0	mlx5_0:1	0	Local PCIe switch	mlx5_1:1 across socket
GPU 1	mlx5_0:1	0	Local root complex	mlx5_1:1 across socket
GPU 4	mlx5_1:1	1	Local PCIe switch	mlx5_0:1 across socket

Use actual data from the node. Do not copy the example values.

This map becomes the basis for MPI rank placement, pod scheduling, NCCL_IB_HCA, and rail validation. It also provides a fast explanation when one GPU or node pair performs differently from another.

Validate IOMMU and ACS Behavior

IOMMU and PCIe Access Control Services are not automatically good or bad. Their effect depends on whether the node is bare metal, virtualized, using passthrough, or relying on platform-specific address translation.

Inspect the current state

cat /proc/cmdline
dmesg | grep -Ei 'IOMMU|DMAR|AMD-Vi'
sudo lspci -vvv | grep ACSCtl

If ACSCtl entries show SrcValid+, ACS might be redirecting peer traffic on that bridge. Inspect the full lspci -vvv output for the GPU, HCA, upstream bridges, and PCIe switches rather than treating one grep result as proof.

Bare-metal guidance

On some bare-metal systems, IOMMU or ACS can redirect PCIe peer traffic through the CPU root complex. The result can be lower bandwidth or a hang even when CUDA peer capability appears valid.

The safe operational sequence is:

1. Capture the current BIOS and kernel configuration.
2. Identify the exact bridges in the GPU-to-HCA path.
3. Confirm the server and platform vendor’s supported GPUDirect settings.
4. Change one setting in a controlled window.
5. Reboot when required.
6. Repeat the raw GPU, RDMA, and NCCL tests.
7. Restore the prior setting if the result does not improve or violates support requirements.

NVIDIA documents an operating-system method that can alter ACS on compatible PCIe switches, but a production runbook should not apply that method broadly. A BIOS or vendor-supported configuration is easier to audit and survives reboot predictably.

Virtual-machine guidance

Virtualized environments often require ACS for isolation and device assignment. Address Translation Services and the hypervisor’s passthrough implementation become part of the supported path.

Do not copy a bare-metal recommendation into a virtual GPU cluster. Validate the hypervisor, guest kernel, IOMMU mode, ATS support, virtual PCIe topology, and GPU-to-NIC passthrough design as one system.

Install NVIDIA Network Operator

NVIDIA Network Operator coordinates the network components required to expose high-performance networking to Kubernetes workloads. It can manage the DOCA-OFED driver, RDMA device plugins, secondary networking components, and related custom resources.

The installation itself does not prove that the RDMA path is correct. Treat it as the mechanism for creating a controlled and observable configuration.

Prepare the namespace and chart source

The following example assumes the NVIDIA NGC Helm repository is already configured on the administration workstation.

kubectl create namespace nvidia-network-operator

kubectl label namespace nvidia-network-operator \
  pod-security.kubernetes.io/enforce=privileged \
  --overwrite

helm repo update

If Node Feature Discovery is already installed by GPU Operator or another platform component, disable the duplicate deployment in the Network Operator Helm values. Running multiple independently managed instances can create ownership and labeling confusion.

Install a pinned operator version

helm install network-operator nvidia/network-operator \
  --namespace nvidia-network-operator \
  --version v26.4.0 \
  --wait

For an existing environment, use the version approved by the platform compatibility matrix. The current command is a reproducible example for the research baseline, not a recommendation to replace a supported release without testing.

Verify the operator before applying network policy

kubectl -n nvidia-network-operator get pods -o wide
kubectl get crds | grep -E 'mellanox|sriov|network'
helm list -n nvidia-network-operator

The operator deployment is only the control plane. A NicClusterPolicy or NicNodePolicy is required to describe the components that should run on the nodes.

Configure RDMA Resources

The correct resource model depends on the required isolation and the physical network design.

Choose the resource pattern

RDMA shared device

Use a shared device when multiple pods may share the same HCA and the environment accepts that isolation model. This is common for initial validation and some trusted training clusters.

SR-IOV virtual function

Use SR-IOV when workloads require stronger device separation, dedicated virtual functions, or explicit tenant-level network policy. This adds PF and VF lifecycle, switch configuration, resource naming, and placement requirements.

Host network

Use host networking only as a controlled diagnostic or when the platform design explicitly supports it. It can simplify the first proof of the host RDMA path, but it is not a substitute for a production pod-network design.

Configure the shared RDMA resource

The following is an illustrative fragment to merge into the validated Network Operator policy for the installed release. It is not a complete production NicClusterPolicy.

spec:
  rdmaSharedDevicePlugin:
    config: |
      {
        "configList": [
          {
            "resourceName": "rdma_shared_device_a",
            "rdmaHcaMax": 63,
            "selectors": {
              "ifNames": ["ens1f0"],
              "linkTypes": []
            }
          }
        ]
      }

Change:

• resourceName to the stable resource name used by your workload platform.
• ifNames to the intended RDMA-capable interfaces on the target nodes.
• The selector strategy if nodes use different predictable interface names.
• The HCA allocation limit according to the tested workload density.

Do not select every RDMA-capable interface merely because it exists. The selector should expose the rails and ports that belong to the approved training network.

Verify reconciliation and allocatable resources

kubectl get nicclusterpolicies.mellanox.com -A
kubectl get nicclusterpolicies.mellanox.com -A -o yaml
kubectl -n nvidia-network-operator get pods -o wide

kubectl describe node  |
  sed -n '/Capacity:/,/Allocated resources:/p'

Confirm that:

• The policy reports Ready.
• The DOCA-OFED and device-plugin pods are healthy on every intended node.
• The expected RDMA resource appears in node capacity and allocatable resources.
• Nodes without the approved HCA are not advertising the resource accidentally.
• Resource names are consistent across the scheduler, pod specification, and validation job.

Choose Between RoCE and InfiniBand Operationally

RoCE and InfiniBand both provide RDMA, but their operational failure modes differ.

Area	InfiniBand	RoCE
Link layer	Native InfiniBand fabric	Ethernet
Addressing	LIDs and GIDs managed by the fabric	IP, VLAN, and GID selection are central
Fabric control	Requires a functioning Subnet Manager	Depends on Ethernet control, routing, and QoS design
Loss behavior	Fabric-native behavior and diagnostics	Usually requires carefully engineered lossless or congestion-managed behavior
Primary health tools	ibstat, ibstatus, sminfo, perfquery, ibdiagnet	ethtool, switch counters, PFC, ECN, CNP, queue, and drop telemetry
Common hidden issue	Missing SM, downshifted link, bad rail, pinned-memory limit	MTU mismatch, PFC or ECN inconsistency, congestion, wrong VLAN or GID
NCCL selection concern	Correct HCA and port	Correct HCA, port, address family, and RoCE network

InfiniBand checks

ibstatus
ibstat
sudo sminfo
sudo perfquery -x 
sudo ibdiagnet

Verify:

• Port state is Active.
• Physical state is LinkUp.
• Link layer is InfiniBand.
• Rate matches the expected port speed.
• A Subnet Manager is reachable.
• Error, recovery, down, and discard counters are not increasing during the test.
• Every rail has a consistent topology and routing state.

RoCE checks

ibstatus
ibstat
ethtool -S ens1f0
rdma statistic

Verify:

• The verbs port reports Ethernet link layer.
• The correct VLAN and IP network are attached to the pod or host.
• MTU is consistent from endpoint through the switching fabric.
• PFC, ECN, CNP, queue-drop, pause, and congestion counters match the intended fabric design.
• Switch ports are not downshifted or accumulating physical errors.
• The selected GID belongs to the intended RoCE network.

Current NCCL releases dynamically select a GID for RoCE. Leave NCCL_IB_GID_INDEX unset unless a specific fabric requirement has been proven. An old copied GID index can force the wrong network after a driver, firmware, or interface-order change.

Meet the Container and Pod Networking Requirements

A host can pass every RDMA test while a pod still fails because the container does not receive the same devices, topology view, memory limits, and network path.

Required pod capabilities

The validation worker should have:

• One explicit GPU resource per rank.
• The intended RDMA resource.
• The required secondary network attachment when the design uses Multus.
• /dev/infiniband devices supplied through the device plugin.
• Sufficient locked memory for RDMA registrations.
• Adequate /dev/shm.
• An accurate /sys topology view.
• CPU and memory placement aligned with the selected GPU and HCA.
• A pinned, known NCCL, CUDA, MPI, and nccl-tests image.
• A network path that allows MPI or launcher bootstrap traffic between workers.

Representative validation pod fragment

apiVersion: v1
kind: Pod
metadata:
  name: nccl-rdma-check
  annotations:
    k8s.v1.cni.cncf.io/networks: rdma-net
spec:
  restartPolicy: Never
  nodeSelector:
    nvidia.com/gpu.present: "true"
  containers:
    - name: test
      image: 
      securityContext:
        capabilities:
          add:
            - IPC_LOCK
      resources:
        limits:
          nvidia.com/gpu: 1
          rdma/rdma_shared_device_a: 1
      volumeMounts:
        - name: dshm
          mountPath: /dev/shm
  volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 2Gi

Change the image, network attachment, GPU count, and RDMA resource name for the environment. The resource prefix can differ when SR-IOV, DRA, or another device-plugin model is used.

This fragment is deliberately incomplete. A multi-node test also needs a launcher, worker discovery, security policy, image-pull configuration, host key or MPI setup, and the cluster’s scheduling controls.

Shared memory and locked memory

NCCL and MPI can fail or behave unpredictably when the container has insufficient shared memory or pinned-memory limits. Validate from inside the actual worker container:

df -h /dev/shm
ulimit -l
ls -l /dev/infiniband
mount | grep ' /sys '

For Docker-based diagnostics, an equivalent test container commonly needs a larger shared-memory allocation and unlimited memlock. In Kubernetes, express the requirement through the pod security and runtime design rather than assuming the host limit automatically reaches the container.

Topology-aware placement

For dedicated performance nodes, Kubernetes CPU Manager and Topology Manager can help align exclusive CPU resources, GPUs, and device-plugin hints. A single-numa-node policy can reject a pod when the requested resources cannot be satisfied from one NUMA node, which is preferable to silently accepting a remote placement in strict performance pools.

This is a node-level platform policy. Test it in a dedicated pool before changing a shared production cluster.

Prove the RDMA Path Before Running NCCL

Do not use NCCL as the first network test. First prove the link, then host-memory RDMA, then GPU-memory RDMA.

Verify devices and link state

Run inside the same execution boundary that will run the benchmark whenever possible.

ibv_devices
ibv_devinfo
ibstat
ibstatus
rdma link show
ibdev2netdev

Confirm that the pod or container sees the intended HCA and port, not merely any RDMA device.

Test host-memory bandwidth

On the server node:

ib_write_bw -d mlx5_0 -a

On the client node:

ib_write_bw -d mlx5_0  -a

Repeat for each intended rail and both directions. Record the command, HCA, port, message size, link rate, and output.

Test GPU-memory bandwidth

When the installed perftest build supports CUDA memory:

ib_write_bw -d mlx5_0 \
  --use_cuda= \
   \
  -a

For a supported DMA-BUF path:

ib_write_bw -d mlx5_0 \
  --use_cuda= \
  --use_cuda_dmabuf \
   \
  -a

Compare:

• Host-memory RDMA.
• GPU-memory RDMA through nvidia-peermem.
• GPU-memory RDMA through DMA-BUF when supported.
• Local GPU and local HCA.
• Local GPU and remote-socket HCA.

If host-memory RDMA is healthy but GPU-memory RDMA is poor or fails, focus on the GPU driver, nvidia-peermem or DMA-BUF support, PCIe topology, ACS, and container device exposure.

Verify GPU memory registration support

sudo modprobe nvidia-peermem
lsmod | grep -E 'nvidia[_-]peermem'

On systems using the open GPU kernel driver and a supported kernel, NCCL can use DMA-BUF without nvidia-peermem. Determine which path is expected for the approved software stack and confirm it in the NCCL logs.

Measure latency and watch counters during the test

ib_write_lat -d mlx5_0
rdma statistic
mlxlink -d 

For RoCE, capture endpoint and switch counters before and after the run. For InfiniBand, capture port counters and rail diagnostics. A bandwidth number without counter deltas can miss retries, congestion, and a degrading physical link.

Select the Correct NCCL Interfaces

NCCL uses different selectors for bootstrap traffic and RDMA data traffic. Confusing them is a common source of silent fallback or inconsistent behavior.

Establish a minimal diagnostic environment

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,ENV
export NCCL_DEBUG_FILE=/evidence/nccl.%h.%p.log

export NCCL_SOCKET_IFNAME='=eno1'
export NCCL_IB_HCA='=mlx5_0:1'

Change:

• eno1 to the IP interface used for NCCL bootstrap and socket fallback.
• mlx5_0:1 to the approved verbs HCA and port.
• The evidence path to persistent storage available in each worker.

The leading equals sign requests an exact match. Without it, prefix matching can include interfaces you did not intend to use.

NCCL_SOCKET_IFNAME does not choose the verbs HCA. It chooses IP interfaces for connection setup and socket transport. NCCL_IB_HCA chooses the verbs device and port.

Use rail policy deliberately

export NCCL_CROSS_NIC=0

Use NCCL_CROSS_NIC=0 only when the fabric is designed as symmetric rails and each ring should remain on the same rail. The default behavior can cross NICs when needed. A blanket value copied from another topology can reduce performance or break connectivity.

Handle RoCE address selection conservatively

For a fabric that requires IPv4 and RoCE v2:

unset NCCL_IB_GID_INDEX
export NCCL_IB_ADDR_FAMILY=AF_INET
export NCCL_IB_ROCE_VERSION_NUM=2

Leave these variables unset when automatic selection already chooses the correct network. Add a constraint only to solve a demonstrated ambiguity.

Use transport overrides as experiments

Force the verbs transport:

export NCCL_NET=IB

Create a socket-only comparison:

export NCCL_NET=Socket

Disable GPU Direct RDMA as a negative control:

export NCCL_NET_GDR_LEVEL=LOC

The negative control is useful because a large performance drop when GDR is disabled supports the conclusion that the direct GPU-to-NIC path was active and beneficial. It is not absolute proof by itself, so confirm the selected transport and registration method in the logs.

Remove NCCL_NET, NCCL_NET_GDR_LEVEL, debug logging, and other diagnostic overrides after the test. NVIDIA warns that debug and workaround variables can cause suboptimal behavior, hangs, or crashes when left in production scripts.

Build and Run nccl-tests

Use a versioned build that matches the NCCL library under test. Record the source commit or image digest.

Build with MPI support

make MPI=1 \
  NAME_SUFFIX=_mpi \
  MPI_HOME=/path/to/mpi \
  CUDA_HOME=/path/to/cuda \
  NCCL_HOME=/path/to/nccl \
  -j

Successful execution creates binaries such as all_reduce_perf_mpi.

Validate the launcher before NCCL

mpirun -np 2 -N 1 --hostfile hosts hostname

Confirm that one process starts on each target node and that output returns cleanly. Then run a small MPI collective or a simple process launch using the same hostfile and network interface. This separates MPI bootstrap failures from NCCL failures.

Run a controlled two-node AllReduce

The following example uses two nodes with eight GPUs each, one MPI process per GPU.

mpirun -np 16 -N 8 \
  --hostfile hosts \
  -x NCCL_DEBUG \
  -x NCCL_DEBUG_SUBSYS \
  -x NCCL_DEBUG_FILE \
  -x NCCL_SOCKET_IFNAME \
  -x NCCL_IB_HCA \
  -x NCCL_CROSS_NIC \
  ./build/all_reduce_perf_mpi \
  -b 8M \
  -e 8G \
  -f 2 \
  -g 1 \
  -w 5 \
  -n 20 \
  -c 1

What to change:

• -np is the total number of MPI processes.
• -N is the number of processes per node for this launcher.
• hosts must contain the intended nodes.
• -g 1 assigns one GPU per process.
• -b, -e, and -f define the message-size sweep.
• -w defines untimed warm-up iterations.
• -n defines measured iterations.
• -c 1 enables correctness checking.

The total NCCL rank count equals processes multiplied by threads multiplied by GPUs per thread. For a one-process-per-GPU design, total MPI processes should equal the total participating GPUs.

Use a progressive test matrix

Do not jump directly to the largest cluster.

Test	Purpose
One GPU	Confirm the container and binary initialize
All GPUs on one node	Prove local PCIe and NVLink behavior
One GPU per node	Isolate inter-node network performance
Two GPUs per node	Introduce local GPU topology and network sharing
All GPUs across two nodes	Validate the complete node design
Same test across additional nodes	Measure scaling and identify bad node pairs
One rail at a time	Identify a degraded HCA, port, cable, or switch path
Socket-only control	Quantify fallback behavior
GDR-disabled control	Quantify the benefit of the direct GPU-memory path

Run each acceptance case at least three times under comparable cluster load. Keep the first run as warm-up evidence, but base acceptance on repeatable steady-state results.

Interpret Algorithm Bandwidth and Bus Bandwidth

nccl-tests reports time, algorithm bandwidth, bus bandwidth, and correctness information.

Algorithm bandwidth

Algorithm bandwidth is the logical data size divided by operation time:

algbw = S / t

This metric answers a workload-oriented question: how quickly did this collective complete for a message of size S?

Bus bandwidth

Bus bandwidth normalizes the communication work performed by a collective so that the result can be compared more meaningfully with hardware communication capacity.

For AllReduce with n ranks:

busbw = algbw * 2 * (n - 1) / n

This is the number that NVIDIA AI Cluster Runtime uses as an aggregate training-performance signal for its NCCL AllReduce validation.

Bus bandwidth is not a guarantee that every NIC, GPU, and rail is saturated equally. It is also not the sum of every advertised physical link rate. Compare it with an effective topology-specific expectation that accounts for:

• Number and speed of NICs per node.
• Number of independent rails.
• PCIe generation and negotiated width.
• GPU-to-NIC affinity.
• GPU count and rank mapping.
• NVLink or NVSwitch topology.
• Collective algorithm and protocol selected by NCCL.
• Message size.
• Container and scheduler placement.
• Fabric overhead and congestion.
• Software versions.

What a healthy result looks like

A defensible healthy result has these characteristics:

• Correctness errors remain zero.
• Bandwidth rises with message size and reaches a stable large-message plateau.
• Repeated runs have low variance under controlled load.
• One-GPU-per-node testing approaches the expected network behavior for the selected rail.
• Full-node testing scales consistently with the validated design.
• NCCL logs show the intended network transport and HCA.
• Logs do not show an unexpected Socket fallback.
• GDR-enabled results outperform the GDR-disabled negative control when the topology should benefit.
• Fabric error and retry counters remain clean or within the approved baseline.
• Node pairs and rails produce comparable results.

Avoid declaring success from one maximum line in one run. A stable plateau and repeatable scaling pattern are stronger evidence than a single peak.

Diagnose Poor AllReduce Performance

Use the following workflow before changing tuning parameters.

Symptom-driven diagnosis

Symptom	Likely domain	Best next proof
Hang during NCCL initialization	Bootstrap interface, unreachable interface, MPI, firewall, bad HCA	Validate MPI host launch, exact socket interface, verbs device, and logs
Socket transport selected	Missing RDMA plugin, HCA selection, plugin load failure	Force NCCL_NET=IB as a diagnostic and inspect NET logs
Host RDMA fast, GPU RDMA slow	GDR registration, topology, ACS, driver path	Compare local and remote GPU/HCA pairs and inspect registration method
One rail slow	Physical link, switch port, routing, congestion	Run ib_write_bw, latency, link, and counter tests per rail
RoCE degrades under load	PFC, ECN, congestion, MTU, queue design	Correlate endpoint and switch counters during the run
One GPU per node fast, all GPUs slow	PCIe oversubscription, rank placement, rail contention	Compare GPU/HCA affinity and per-rail traffic
Single node slow	GPU peer access, NVLink, local PCIe, ACS	Run topology, peer matrix, and local bandwidth tests
Intermittent long tails	Congestion, retries, thermal behavior, noisy neighbor	Capture per-iteration timing, latency, counters, clocks, and cluster load
ibv_reg_mr failure	Pinned-memory limit or registration path	Check ulimit -l, container capability, and driver logs
Performance changes after restart	Module, driver, GID, interface ordering, policy drift	Compare loaded modules, policy, logs, and device mappings before and after

Do not tune around a broken layer

Variables such as NCCL_IB_TIMEOUT, NCCL_IB_RETRY_CNT, channel counts, protocol selection, and algorithm selection can be useful after the system checks are healthy. They should not be the first response to packet loss, an unhealthy rail, a remote-socket HCA, or a missing GPU-memory registration path.

A timeout increase can convert a visible fabric fault into a slower intermittent job. That is not a performance fix.

Control NUMA Placement and Cross-Socket Traffic

NUMA problems become visible when the benchmark expands from one GPU to all GPUs on the node.

Capture CPU and device locality

numactl --hardware
lscpu -e=CPU,NODE,SOCKET,CORE
nvidia-smi topo -m

cat /sys/class/net/ens1f0/device/numa_node
cat /sys/class/net/ens2f0/device/numa_node

Map each MPI rank to:

• One GPU.
• A CPU set on the same NUMA node when possible.
• The HCA nearest that GPU.
• The intended fabric rail.

A random CPU binding can still produce a valid NCCL result, but it increases variance and can introduce remote memory or control-path traffic. Record the launcher binding policy as part of the benchmark.

Recognize cross-socket signatures

Cross-socket penalties often appear as:

• Specific GPUs consistently underperform.
• Performance drops only when GPUs attached to the second CPU participate.
• One HCA performs well for half the GPUs and poorly for the other half.
• Bus bandwidth plateaus below the known node baseline even though raw link tests are healthy.
• Performance changes significantly when NCCL_IB_HCA selects another HCA.
• CPU interconnect utilization rises during the collective.

Do not solve this by increasing the allowed GDR distance without understanding the topology. NCCL_NET_GDR_LEVEL=SYS permits a broader path; it does not turn a remote HCA into a local one.

Kubernetes placement boundary

Topology Manager can align resources only when the device plugins provide useful topology hints and the requested CPU, GPU, and RDMA resources can be satisfied together. For strict GPU training pools, validate the kubelet policy, pod scope, CPU requests, device-plugin topology, and admission behavior as one design.

A pod scheduled successfully is not proof that it received the best GPU-to-NIC path.

Collect Evidence Before Escalation

A vendor escalation should not begin with only an all_reduce_perf screenshot. Package evidence so the support engineer can reconstruct the path and compare failure domains.

Hardware and topology evidence

Collect:

nvidia-smi -q
nvidia-smi topo -m
nvidia-smi topo -p2p p
nvidia-smi topo -p2p n
lspci -tv
sudo lspci -vvv
numactl --hardware
dmidecode

Also record:

• Server vendor and model.
• GPU model and count.
• NIC model, firmware, port count, and link speed.
• PCIe generation and negotiated width.
• BIOS version and relevant IOMMU or ACS settings.
• Cable, switch port, rail, rack, and fabric path.
• Any virtualization or passthrough layer.

Software evidence

Collect:

uname -a
nvidia-smi
nvcc --version
modinfo nvidia
modinfo nvidia-peermem
ofed_info -s
mpirun --version

Record:

• Kubernetes version.
• Container runtime.
• GPU Operator version.
• Network Operator version.
• GPU driver version.
• DOCA-OFED version.
• CUDA version.
• NCCL version.
• MPI implementation and version.
• nccl-tests commit or image digest.
• Kernel and operating-system image.

Kubernetes evidence

helm get values network-operator \
  -n nvidia-network-operator \
  --all

kubectl get nicclusterpolicies.mellanox.com -A -o yaml
kubectl get nicnodepolicies.mellanox.com -A -o yaml
kubectl get network-attachment-definitions.k8s.cni.cncf.io -A -o yaml
kubectl get pods -A -o wide
kubectl describe node 
kubectl get pod  -o yaml

Redact credentials, registry secrets, tokens, and unrelated tenant data before sharing.

Network evidence

Collect before and immediately after the benchmark:

• ibstat and ibstatus.
• rdma link show and rdma statistic.
• ib_write_bw host-memory results.
• ib_write_bw GPU-memory results.
• ib_write_lat results.
• mlxlink output.
• InfiniBand Subnet Manager and port counters.
• RoCE endpoint and switch PFC, ECN, CNP, queue, drop, pause, and physical error counters.
• Fabric topology or rail map.

NCCL evidence

Preserve:

• Exact launch command.
• Hostfile or scheduler allocation.
• Complete exported NCCL environment.
• Per-rank NCCL logs.
• Full nccl-tests output.
• Message-size range, warm-up count, iterations, and correctness settings.
• Rank-to-GPU and GPU-to-HCA mapping.
• Node pair and rail used.
• GDR-enabled, GDR-disabled, verbs-forced, and socket-only comparison runs.
• Date, time, cluster load, and run identifier.

A compact comparison matrix is more useful than a folder of unlabeled logs.

Run ID	Nodes	GPUs per node	Rail	Transport	GDR	Large-message busbw	Errors	Notes
R01	1	8	Local	P2P/NVLink	N/A	Baseline	0	Single-node
R02	2	1	A	IB	On	Measured	0	Network isolation
R03	2	8	A+B	IB	On	Measured	0	Full node
R04	2	8	A+B	IB	Off	Measured	0	Negative control
R05	2	8	N/A	Socket	N/A	Measured	0	Fallback control

Replace the placeholder values with actual measurements.

Define Upgrade and Rollback Boundaries

Performance validation should become a release gate, not a one-time commissioning task.

Capture the pre-change state

Before an operator, driver, firmware, kernel, NCCL, or container-image change:

helm get values network-operator \
  -n nvidia-network-operator \
  --all > network-operator-values.before.yaml

kubectl get nicclusterpolicies.mellanox.com -A -o yaml \
  > nicclusterpolicy.before.yaml

kubectl get network-attachment-definitions.k8s.cni.cncf.io -A -o yaml \
  > network-attachments.before.yaml

Also capture the exact baseline NCCL matrix and fabric counters.

Upgrade in bounded failure domains

Use these controls:

• Pin the Helm chart and test-image versions.
• Validate component compatibility before the change.
• Cordon and drain one node or one approved failure domain at a time.
• Set a conservative maxParallelUpgrades value for DOCA-OFED driver upgrades.
• Confirm PodDisruptionBudgets permit the intended drain behavior.
• Keep automatic driver upgrades disabled until the canary path is accepted.
• Re-run host RDMA, GPU RDMA, single-node NCCL, and two-node NCCL before expanding.
• Stop when the first topology-specific regression appears.

The current Network Operator lifecycle supports upgrades within a major release or to the next major release. Do not assume an arbitrary multi-release jump or downgrade is supported.

Understand what Helm rollback does not restore

A Helm rollback can restore chart-managed Kubernetes objects. It does not necessarily restore:

• The previous loaded kernel module.
• The previous GPU or DOCA-OFED driver state.
• NIC firmware.
• BIOS or IOMMU settings.
• A prior node kernel.
• A changed SR-IOV VF layout.
• Switch configuration.
• GID ordering.
• A drained workload’s in-memory state.

Driver rollback can require pod eviction, node drain, module reload, or reboot. Treat it as a node lifecycle operation, not a simple manifest reversal.

Define rollback criteria before the change

Rollback when any of these occurs:

• The operator policy does not reconcile to Ready.
• RDMA resources disappear from intended nodes.
• GPU-memory RDMA fails after host-memory RDMA remains healthy.
• NCCL selects Socket unexpectedly.
• Correctness errors appear.
• Bus bandwidth regresses beyond the approved tolerance.
• Variance or tail latency exceeds the approved threshold.
• Fabric counters begin increasing during the benchmark.
• The canary node cannot return to the previous tested state.

Document the boundary between a Kubernetes rollback and a node-image rollback. That distinction prevents a chart rollback from being declared successful while the data-plane driver remains changed.

Operational Acceptance Checklist

Use this final checklist before declaring the multi-node GPU fabric ready.

Topology

• Every GPU-to-HCA path is mapped.
• NUMA locality is known.
• PCIe width and speed match the design.
• GPU peer-access matrices are captured.
• ACS and IOMMU behavior is understood and supportable.

Network Operator and pod exposure

• Network Operator is pinned to a validated version.
• NicClusterPolicy or NicNodePolicy is Ready.
• Expected RDMA resources are allocatable.
• The worker requests the correct GPU and RDMA resources.
• Secondary networking is attached when required.
• /dev/infiniband, /dev/shm, memlock, and /sys are correct.

Fabric

• Every intended port is active at the expected rate.
• Host-memory RDMA meets the baseline.
• GPU-memory RDMA meets the baseline.
• Latency is stable.
• Error, retry, drop, and congestion counters are acceptable.
• Every rail has been tested independently.

NCCL

• Bootstrap and verbs interfaces are selected deliberately.
• Logs show the intended transport and HCA.
• No unexpected Socket fallback occurs.
• Correctness errors are zero.
• Large-message bandwidth reaches a stable plateau.
• Repeated runs remain within the approved variance.
• One-GPU-per-node and full-node scaling are both understood.
• GDR-disabled and socket-only controls have been captured.
• Results are compared with the topology-specific baseline.

Lifecycle

• Pre-change manifests and values are saved.
• Baseline evidence is retained.
• Canary scope and stop criteria are defined.
• Driver and node rollback requirements are documented.
• The escalation package can reconstruct hardware, software, topology, fabric, and test conditions.

Conclusion

GPUDirect RDMA is not a feature that can be proven by finding an RDMA device or setting one NCCL variable. It is an end-to-end data path that begins in GPU memory, crosses a specific PCIe topology, reaches a specific HCA, traverses a specific fabric rail, and lands on a remote GPU through another equally specific path.

The most reliable operational method is therefore layered. Prove local GPU communication first. Prove host-memory and GPU-memory RDMA second. Confirm the container sees the same devices and topology as the host. Then run NCCL with evidence-grade logging, controlled interface selection, repeatable message-size sweeps, and negative controls.

Algorithm bandwidth tells you how quickly the collective completed. Bus bandwidth helps normalize the communication work and compare utilization across rank counts. Neither number is meaningful without the hardware topology, transport logs, raw network tests, counter evidence, and software baseline that produced it.

Once the process is repeatable, NCCL AllReduce becomes more than a benchmark. It becomes a release gate for GPU nodes, RDMA fabrics, operator changes, driver upgrades, and cluster expansion.

External References