Ethernet, RoCE, or InfiniBand? Designing the Network Fabric for Enterprise AI

TL;DR

No network fabric is universally superior for enterprise AI.

A well-designed Ethernet network is normally the right foundation for out-of-band management, in-band platform services, north-south inference, registry access, and many storage flows. It is also sufficient for single-node training and inference workloads that do not exchange latency-sensitive data across GPU nodes.

A specialized RDMA fabric becomes important when inter-node communication sits on the critical path of useful GPU work. Frequent all-reduce operations, tensor or expert parallelism across nodes, large synchronous jobs, direct storage-to-GPU paths, and expensive GPU idle time all strengthen the case for RoCEv2 or InfiniBand.

RoCEv2 preserves an Ethernet and IP operating model, but it requires disciplined congestion engineering across hosts, NICs, switches, queues, and telemetry. InfiniBand provides a purpose-built RDMA fabric with its own subnet-management and operational model. Either can perform well when the complete stack is validated. Either can fail badly when treated as a simple switch-speed decision.

The practical answer is to classify traffic first, measure the workload’s communication pattern, decide whether RDMA is required, and only then select the fabric that the organization can operate and support.

Introduction

AI infrastructure discussions often jump directly from GPU count to switch speed. The design meeting quickly turns into a debate over 200, 400, or 800 gigabits per second, followed by a second debate over Ethernet, RoCE, or InfiniBand.

That sequence is backwards.

The network requirement is created by the workload’s data movement, synchronization, and failure behavior. An inference platform serving independent requests has a different network profile from a multi-node training job that pauses every step for collective communication. A storage path feeding large sequential datasets has different congestion behavior from an all-to-all mixture-of-experts exchange. A Kubernetes control plane should not share the same failure assumptions as a GPU compute rail.

The architect’s job is therefore not to select the fastest fabric on a product sheet. It is to decide which traffic needs ordinary resilient IP connectivity, which traffic needs high throughput, and which traffic needs an engineered RDMA path with tightly controlled latency and loss.

This article uses NVIDIA’s current enterprise reference architectures and Network Operator 26.4 documentation as the July 2026 support baseline. It also incorporates IEEE Ethernet and data-center-bridging standards, Dell PowerScale storage guidance, and community reports that expose recurring implementation mistakes. Those community discussions are treated as diagnostic signals, not as substitutes for vendor support matrices.

Scope, Assumptions, and Terminology Guardrails

The title presents Ethernet, RoCE, and InfiniBand as three choices, but that shorthand needs correction.

RoCE is Ethernet. RoCEv2 carries RDMA traffic over UDP/IP on an Ethernet fabric. The real comparison is usually among conventional TCP or UDP Ethernet, RDMA over Ethernet, and native InfiniBand RDMA.

This article focuses on data-center and campus-adjacent AI fabrics inside one operational site or availability zone. It does not recommend stretching a lossless fabric across a wide-area multicloud network. Cross-cloud and cross-region AI workflows should normally use standard routed connectivity, data replication, object transfer, checkpoint movement, or application-level distribution. The tightly controlled compute fabric should end where its failure and congestion domain can still be engineered.

The term lossless fabric also needs restraint. It does not mean that links never experience errors or that packets can never be lost. It means that the architecture is engineered to prevent loss for selected traffic classes under the validated congestion envelope. That outcome depends on endpoint congestion control, switch queues, buffer headroom, topology, cabling, firmware, and operating discipline.

Finally, GPU count is a useful sizing input, but it is not the decision by itself. Eight GPUs inside one NVLink domain can create less scale-out traffic than four GPUs split across four servers. A 64-GPU inference farm running independent replicas may be less network-sensitive than a 16-GPU job using cross-node tensor parallelism.

Start With Traffic Planes, Not Fabric Brands

Enterprise AI platforms carry several traffic classes at the same time. They should be designed separately even when some of them share physical switches.

Traffic plane	Typical flows	Dominant behavior	Design bias
Out-of-band management	BMC, switch management, console, power control	Low bandwidth, high operational importance	Physically separate, simple, redundant, reachable during platform failure
In-band management and platform services	Kubernetes API, image registry, DNS, identity, monitoring, logging, model control plane	Bursty, many small flows, dependent on stable IP services	Resilient Ethernet with clear security and quality-of-service policy
Storage and data	Dataset reads, checkpoint writes, model artifacts, vector data, NFS, object storage, backup	Sustained throughput, bursts, metadata sensitivity, recovery traffic	High-capacity Ethernet or validated storage fabric, often physically or logically isolated
East-west GPU compute	NCCL collectives, gradients, activations, expert exchange, distributed inference state	Synchronous, latency-sensitive, bursty incast, throughput-sensitive	Dedicated or strongly isolated RDMA fabric when communication is on the critical path
North-south inference	API requests, token streams, client responses, gateway traffic	Request-response, fan-in and fan-out, tail-latency sensitive	Resilient routed Ethernet, load balancing, security, and measured oversubscription

The architecture below shows why a single statement such as “the AI network is 400 gigabit Ethernet” is incomplete.

The most expensive design mistake is allowing a workload burst in one plane to destabilize another. A checkpoint storm should not make the Kubernetes API unreachable. A pause-frame problem on an RDMA priority should not freeze ordinary storage or management traffic. A failed fabric manager should not remove the only path to the switches it manages.

Scale-Up and Scale-Out Communication Change the Answer

AI network design becomes clearer when the architect separates scale-up from scale-out communication.

Scale-Up Communication

Scale-up communication occurs inside a tightly coupled compute domain. Depending on the platform, GPUs may communicate through NVLink and NVSwitch, through PCIe peer-to-peer paths, or through a combination of both. The key characteristic is that communication remains inside the server or rack-scale GPU domain.

When a model, its parallelism strategy, and its memory requirements fit inside that domain, the external compute fabric may not be the performance limiter. This is one reason inference-focused designs can often omit a dedicated GPU east-west fabric.

Scale-Out Communication

Scale-out communication crosses server or rack boundaries through network adapters and switches. At that point, topology, NIC-to-GPU locality, congestion behavior, routing, and collective communication efficiency become critical.

Different parallelism patterns stress the fabric differently:

• Data parallel training commonly uses all-reduce operations at step boundaries. The network can determine how long every GPU waits before the next step.
• Tensor parallelism exchanges model-state fragments frequently. Cross-node placement can make latency and bandwidth part of nearly every layer execution.
• Pipeline parallelism moves activations between stages. Imbalanced stages and network delay create pipeline bubbles.
• Mixture-of-experts models can generate heavy all-to-all traffic when tokens are routed to experts on other nodes.
• Independent inference replicas generally need north-south capacity but little GPU-to-GPU scale-out traffic.
• Disaggregated or model-parallel inference can reintroduce sensitive east-west traffic through remote state, prefill-decode separation, or cross-node tensor exchange.

The first design question should therefore be:

Does the workload cross a node boundary during its performance-critical execution path?

When the answer is no, ordinary high-quality Ethernet may be sufficient. When the answer is yes, the next question is how often the workload synchronizes and how much GPU time is lost while it waits.

What Ethernet, RoCE, and InfiniBand Actually Change

Criterion	Conventional Ethernet	RoCEv2 Ethernet	InfiniBand
Primary transport model	TCP, UDP, HTTP, storage and application protocols	RDMA over UDP/IP on Ethernet	Native InfiniBand transport and RDMA
Routability	Standard IP routing	IP-routable, subject to validated RoCE design	Routed within the InfiniBand fabric through its fabric-control model
Congestion model	Drops, queues, TCP control, application retries	ECN, endpoint congestion control, selected PFC use, or a validated lossy-RoCE design	Credit-based link behavior plus InfiniBand congestion and routing controls
Fabric control	Familiar Ethernet switching and routing	Ethernet switching plus DCB, RDMA, NIC, queue, and congestion tuning	Subnet manager, partitions, fabric routing, and InfiniBand management
Kubernetes exposure	Primary CNI and normal interfaces	Multus plus RDMA device plugin, SR-IOV, host-device, or shared device model	Multus plus InfiniBand CNI, SR-IOV or shared device model, PKeys, and subnet management
Operational skill profile	Broad enterprise networking skill base	Advanced Ethernet plus RDMA and host-stack expertise	Dedicated InfiniBand fabric and subnet-management expertise
Strong fit	Management, north-south inference, registries, general storage, independent replicas	Organizations wanting RDMA with an Ethernet and IP operating model	Dedicated large-scale training and HPC environments with mature IB operations
Main trap	Assuming link speed alone solves distributed-compute latency	Treating RoCE as ordinary Ethernet with jumbo frames enabled	Treating the Kubernetes operator as the fabric control plane

Conventional Ethernet

Conventional Ethernet remains the default enterprise fabric because it is routable, observable, widely supported, and compatible with existing security and operations tooling. It can use 25, 100, 200, 400, or faster links without becoming an RDMA fabric.

That distinction matters. A 400-gigabit TCP path may offer excellent aggregate throughput while still creating more CPU involvement, protocol overhead, tail latency, and synchronization variability than an engineered RDMA path. Conversely, a badly configured RoCE path can perform worse than a stable TCP design.

RoCEv2

RoCEv2 preserves Ethernet cabling, switching, and IP addressing while allowing applications to use RDMA semantics. It is attractive when an organization wants one networking technology family across compute, storage, and services, or when the network team already operates advanced data-center bridging and telemetry.

RoCE is not activated by one switch command. It is an end-to-end behavior that includes:

• supported NIC and GPU combinations
• host and container drivers
• traffic-class mapping
• ECN marking and endpoint reaction
• PFC scope when PFC is used
• buffer thresholds and headroom
• MTU consistency
• routing and path symmetry
• application use of RDMA
• operational telemetry under congestion

NVIDIA Spectrum-X is an example of a validated Ethernet platform that coordinates switches, SuperNICs, software, and congestion behavior. It should not be reduced to “generic Ethernet with PFC.”

InfiniBand

InfiniBand is a purpose-built switched fabric with native RDMA semantics and a mature history in high-performance computing. It introduces a separate operational model, including a subnet manager, fabric routing, partitions, link-state and congestion telemetry, and often a dedicated management platform.

That separation can be an advantage. A dedicated compute fabric avoids interaction with ordinary enterprise traffic and can make the performance envelope easier to preserve. It can also add hardware, skills, tooling, and lifecycle boundaries that the organization does not already possess.

InfiniBand is not automatically faster in every enterprise scenario. It is strongest when the workload actually benefits from its fabric characteristics and the operations team can maintain the complete system.

When a Well-Designed Ethernet Network Is Sufficient

Standard Ethernet is normally sufficient when one or more of the following conditions apply:

• Training or inference remains inside one server or one scale-up GPU domain.
• Inference is delivered through independent replicas with no cross-node model parallelism.
• The primary traffic is API, token streaming, image pull, monitoring, logging, registry, or control-plane communication.
• Storage performance targets can be met through NFS over TCP, object storage, block storage, or another validated IP storage path.
• The platform is still a proof of value and representative testing shows that the network is not limiting useful GPU work.
• Virtual-machine mobility, operational simplicity, or multitenant abstraction is more valuable than direct device access.
• The team does not yet have the telemetry, automation, and change-control maturity required to operate an RDMA fabric safely.

This does not mean the Ethernet design can be casual. It still needs redundant leaf switches, sufficient uplink capacity, consistent MTU, resilient routing, storage-path diversity, failure-domain analysis, and useful telemetry.

The practical design principle is to avoid paying for complexity before the workload proves that it needs it, while preserving a credible expansion path. Retrofitting dedicated compute NICs, optics, rails, and switches later can require downtime and re-cabling, so an inference-first platform should still reserve rack space, PCIe capacity, switch ports, and cable routes for future training.

When a Specialized RDMA Fabric Becomes Necessary

A specialized RDMA fabric becomes a strong requirement when inter-node communication consumes a material portion of job time or creates unacceptable variability.

The strongest indicators are:

• The job uses frequent synchronous collectives across nodes.
• Tensor, pipeline, or expert parallelism crosses the server boundary.
• A slow rank delays every other rank.
• GPU utilization drops during communication phases even though compute and memory capacity are available.
• The platform must maintain predictable completion time for expensive long-running jobs.
• The cluster spans multiple racks and ordinary oversubscription creates repeatable step-time penalties.
• GPUDirect RDMA or GPUDirect Storage is part of the supported workload design.
• Checkpoint, data-loader, and compute traffic compete on a shared path.
• The cost of idle accelerators exceeds the cost of the specialized fabric and its operations.

There is no responsible universal threshold such as “InfiniBand is required above 32 GPUs.” GPU count is a proxy for communication scale, not the actual requirement. A better threshold is measurable:

Add or upgrade the RDMA compute fabric when representative multi-node jobs cannot meet throughput, completion-time, or GPU-efficiency objectives after software placement and topology issues have been corrected.

GPUDirect RDMA Matters Because It Changes the Data Path

Without a direct peer path, network data may be staged through system memory and involve additional CPU copies or coordination. GPUDirect RDMA allows a supported NIC or DPU to exchange data directly with GPU memory across PCIe. GPUDirect Storage applies a related direct-path model between storage and GPU memory.

The benefit is not only lower latency. Avoiding bounce buffers can reduce CPU overhead and improve effective bandwidth. That can be valuable when many GPUs exchange large tensors or when storage streams data directly into GPU memory.

GPUDirect RDMA is also a support-stack feature, not merely a hardware capability. The architect must validate:

• GPU, NIC, DPU, and server compatibility
• PCIe root-complex and NUMA placement
• IOMMU and Access Control Services behavior
• GPU driver, CUDA, kernel, and network-driver alignment
• DMA-BUF or legacy peer-memory method
• container runtime and operator versions
• firmware and switch operating-system versions
• Kubernetes resource exposure
• application and communication-library use of the direct path

NVIDIA’s current GPU Operator guidance recommends DMA-BUF where the supported kernel and open GPU kernel module allow it. Legacy peer-memory integration remains relevant for other combinations. The selected method must follow the current platform matrix rather than a copied installation command from an older article.

GPUDirect does not remove the fabric bottleneck. It makes the endpoint path more efficient, which can expose congestion, rail imbalance, or switch oversubscription more clearly.

Kubernetes Needs Both a Network Attachment and an RDMA Resource

Kubernetes adds a control and scheduling layer to the physical fabric. NVIDIA Network Operator can manage networking drivers, device plugins, secondary-network components, and policies. NVIDIA GPU Operator manages the GPU software stack. Multus allows a pod to receive a secondary high-performance interface while the primary CNI continues to carry cluster and service traffic.

A pod does not gain working RDMA merely because a secondary interface appears. It normally needs two things:

• a Multus network attachment that connects the pod to the intended fabric
• an extended resource request that causes the scheduler and device plugin to allocate the RDMA-capable device

The following example is intentionally conceptual. The network name and resource key must match the Network Operator, SR-IOV, or shared-device policy deployed in the target cluster.

apiVersion: v1
kind: Pod
metadata:
  name: distributed-trainer
  annotations:
    k8s.v1.cni.cncf.io/networks: ai-rdma-rail0
spec:
  containers:
    - name: trainer
      image: registry.local/ai/trainer:approved
      securityContext:
        capabilities:
          add:
            - IPC_LOCK
      resources:
        requests:
          nvidia.com/gpu: "1"
          rdma/rail0: "1"
        limits:
          nvidia.com/gpu: "1"
          rdma/rail0: "1"

Successful scheduling is not sufficient validation. Inside the pod, the operator should confirm the secondary interface, RDMA device, GID or LID state, selected rail, GPU visibility, and actual collective performance.

Shared RDMA Device Versus SR-IOV

Exposure model	Strength	Tradeoff	Best fit
RDMA shared-device plugin	Simpler host networking, efficient sharing	Weaker device-level isolation and less deterministic allocation	Trusted clusters, labs, or controlled shared environments
SR-IOV virtual function	Dedicated VF, stronger isolation, predictable resource accounting	Finite VF inventory, more lifecycle complexity, PF/VF and IPAM planning	Multitenant clusters and production workloads needing direct allocation
Host-device attachment	Straightforward direct interface assignment	Device becomes unavailable to other workloads and scheduling is rigid	Specialized nodes with static workload placement
PCI passthrough to a VM	Near-native control in the guest	Device dedication, IOMMU dependence, reduced mobility	Performance-focused virtualized nodes with fixed placement
Paravirtualized vNIC	Operational flexibility and VM mobility	Usually not the direct GPUDirect or native RDMA data path	Management, application, and conventional storage traffic

Current Network Operator documentation supports specific KubeVirt SR-IOV VFIO combinations, requires IOMMU, and states that live migration is not supported for VMs using that passthrough model. Host-side RDMA is also unavailable for a VF passed through with VFIO because the guest owns the device.

For vSphere and other hypervisors, direct GPU-to-NIC peer paths add further topology and configuration requirements. A VM can have both a flexible management vNIC and a passed-through performance NIC, but the platform team must treat the latter as a placement and recovery constraint.

The Support Matrix Is Part of the Architecture

As of the July 2026 research baseline, NVIDIA Network Operator 26.4 documents exact combinations of operating system, Kubernetes version, distribution, adapter, GPU, and feature support. The matrix includes explicit Rancher RKE2 combinations rather than treating every Kubernetes distribution as equivalent. GPUDirect RDMA currently requires a compatible GPU Operator release and validated networking stack.

The same documentation warns against casually mixing component versions because the operator release is tested as a coordinated set. That warning should become a lifecycle rule:

Pin the operator, driver, CNI, device-plugin, firmware, kernel, and GPU software baseline as one tested platform release.

RoCE Requires Congestion Engineering, Not Just Fast Ethernet

RoCEv2 can provide excellent performance, but its reliability depends on a coherent congestion design.

PFC, ECN, and Endpoint Congestion Control

Priority-based Flow Control, standardized through IEEE 802.1Qbb, can pause one Ethernet priority without pausing every traffic class on the link. It is intended to prevent buffer overflow for selected traffic, not to make the entire network lossless.

Explicit Congestion Notification allows a switch to mark traffic before queues overflow. The receiving endpoint can return congestion feedback, and the sender’s congestion-control algorithm reduces its rate. IEEE 802.1Qau defines Ethernet congestion-notification concepts, while IEEE 802.1Qaz covers Enhanced Transmission Selection and Data Center Bridging Exchange behavior used to coordinate traffic classes.

A healthy RoCE control loop looks like this:

The operational danger is overusing PFC. Pause propagation can create head-of-line blocking, congestion spreading, or a pause storm. PFC should normally be limited to the intended RDMA traffic class, protected with watchdog behavior, and observed through transmit, receive, duration, and queue counters.

NVIDIA documentation also describes lossy RoCE designs that rely on ECN and congestion control without traditional PFC behavior. That option should only be adopted as a validated end-to-end architecture. It is not permission to disable PFC in a design that was tested with it.

Buffer and Headroom Configuration

Switch buffer configuration must reflect port speed, cable distance, MTU, pause-response time, and burst behavior. A copied threshold from a different topology can either waste buffers or allow loss before congestion control reacts.

The network team should understand:

• which queue carries RDMA
• where ECN marking starts
• where PFC pause starts
• how much headroom exists per port and priority
• whether dynamic buffers can starve another class
• how incast behaves at leaf and spine layers
• how a failed or slow receiver affects upstream devices

Lossless behavior must be validated under congestion, not inferred from an idle ping test.

MTU Consistency and Jumbo-Frame Failures

Jumbo frames can reduce per-packet overhead, but consistency matters more than selecting a fashionable number. A common design uses a 9000-byte host or workload MTU with a larger switch port MTU that accommodates Ethernet overhead. The exact supported values depend on the platform.

Common failure patterns include:

• physical functions set to jumbo MTU while virtual functions remain at 1500
• switch ports set correctly while a bond, VLAN, bridge, or overlay remains smaller
• the host path is jumbo-capable but the Multus network attachment uses a default MTU
• storage and compute interfaces use different assumptions
• one leaf-to-spine uplink remains at its default
• the VM vNIC, guest interface, and passed-through VF do not agree
• encapsulation overhead reduces the effective tenant MTU

Small probes can succeed while large RDMA messages stall or fragment elsewhere in the stack. Validation must originate inside the actual pod or VM and traverse the same VLAN, VF, rail, switch path, and destination used by the workload.

Rail-Optimized Design Preserves GPU and NIC Locality

A rail is a repeated GPU-to-NIC-to-switch path across compute nodes. Rail optimization connects corresponding NIC or GPU positions through the same leaf or plane so collective traffic can remain local to that rail rather than crossing unnecessarily between switch groups.

Rail optimization starts inside the server. The GPU, NIC, PCIe root complex, CPU socket, and NUMA placement should align. Cabling then preserves that relationship through the fabric. Kubernetes resource names and topology labels should expose the same rail identity so the scheduler does not allocate a GPU on one locality domain and an RDMA device on another.

Multi-plane designs add resilience and aggregate capacity. They should be independent enough that a failed switch, optic, cable group, or software plane does not remove all GPU-to-GPU connectivity. Application libraries such as NCCL can then use or fail over among validated paths.

Oversubscription Must Follow Workload Sensitivity

Oversubscription is not inherently bad. It becomes bad when the workload’s simultaneous demand exceeds the fabric’s available bisection bandwidth often enough to violate the service objective.

Traffic plane	Oversubscription posture	Reason
Out-of-band management	High oversubscription can be acceptable with redundancy	Traffic is light, but reachability must survive failures
In-band management	Moderate oversubscription is normally acceptable	Bursts matter, but sustained line rate is uncommon
North-south inference	Size to measured request, response, and model-loading peaks	Independent requests can often share capacity efficiently
Storage and checkpoint	Use measured read, write, metadata, rebuild, and recovery bursts	Checkpoint synchronization can create severe temporary demand
Synchronous GPU compute	Prefer nonblocking or prove the acceptable ratio through representative tests	Every delayed rank can extend step time across the job
Asynchronous batch or loosely coupled jobs	Some measured oversubscription may be acceptable	Work can tolerate queueing or variable completion time

For a large synchronous training fabric, a 1:1 nonblocking design is a reasonable starting bias. It is not a universal commandment for every AI network. Storage, inference, and management can use different ratios when evidence supports them.

The test is not whether every port can run at line rate independently. The test is whether the simultaneous communication pattern can run without creating unacceptable GPU wait time, tail latency, or recovery risk.

Storage Traffic Needs Its Own Capacity Model

AI storage design is not a secondary detail. GPUs can be idle because the compute fabric is slow, but they can also be idle because datasets, checkpoints, model shards, or metadata arrive too slowly.

PowerScale and similar AI data platforms often connect over high-speed Ethernet even when the GPU compute fabric uses InfiniBand. That is a valid separation of concerns. The storage fabric can use NFS over TCP, NFS over RDMA, multipath client behavior, GPUDirect Storage, or another validated protocol without forcing the compute fabric to use the same transport.

The storage capacity model should include:

• aggregate sequential dataset read rate
• checkpoint size, frequency, and completion objective
• concurrent checkpoint writes from multiple jobs
• small-file and metadata operations
• data-preparation and shuffle traffic
• model and container image distribution
• failure, rebuild, and rebalance traffic
• backup, replication, and retention workflows
• client-to-storage path diversity
• per-client and per-storage-node connection behavior

Dell’s PowerScale guidance highlights an important nuance: multipath and multiple connections can improve aggregate streaming behavior across files and nodes, but they do not make every single-file or lock-sensitive workload scale linearly. The application access pattern still matters.

A converged RoCE fabric can carry compute and storage, but those flows must have explicit traffic classes, capacity protection, failure isolation, and observability. A checkpoint burst should not consume the buffers or queues required by a collective operation.

There is also a current software compatibility detail worth recording in the architecture decision. NVIDIA Network Operator 26.4 documents a conflict in one driver-container configuration between host-side NFS RDMA enablement and host-side NVMe over RDMA. This is exactly the kind of version-specific interaction that disappears from high-level diagrams but can block a production design. Storage protocol, host-driver installation method, and operator configuration must be validated together.

Telemetry Must Explain GPU Wait Time

Traditional interface up or down monitoring is insufficient for AI fabrics. The operations model must connect application symptoms to GPU, PCIe, NIC, switch, and storage evidence.

Minimum Telemetry Set

Collect and correlate:

• link utilization by direction and rail
• ECN marks and congestion notifications
• PFC transmit and receive frames
• pause duration and watchdog events
• queue occupancy, buffer pressure, and drops
• CRC, symbol, FEC, and physical-link errors
• InfiniBand congestion indicators such as transmit wait
• route, path, PKey, GID, and LID state where applicable
• NIC firmware and driver health
• PCIe width, speed, replay, and topology
• NVLink and GPU peer-path health
• NCCL collective latency and bandwidth
• per-rank completion time
• GPU utilization, memory utilization, and stall periods
• storage read, write, metadata, queue, and client-path latency
• pod, VF, and RDMA resource allocation

Dashboards should align timestamps across these layers. A graph showing falling GPU utilization beside rising PFC duration and collective latency is operationally useful. Three separate dashboards owned by three teams are not.

Symptoms Commonly Mistaken for GPU or Application Problems

Symptom	Network or fabric cause to investigate
GPU utilization becomes sawtoothed during multi-node runs	Collective synchronization delay, congestion, rail imbalance, or storage starvation
One rank consistently finishes late	Bad optic, FEC errors, NUMA mismatch, PCIe locality, path asymmetry, or overloaded leaf
NCCL initialization hangs	Interface selection, routing, firewall, PKey, subnet-manager, GID, VF, or device-exposure problem
Small tests pass but large transfers fail	MTU mismatch, buffer threshold, fragmentation, or one smaller segment in the path
Training works on one node but fails on two	RDMA path, peer-memory integration, ACS/IOMMU, CNI, fabric routing, or firewall issue
Checkpoint time increases as cluster use grows	Shared storage uplink, metadata bottleneck, queue contention, or competing rebuild traffic
Pod has a secondary interface but no RDMA device	Missing extended resource request, device-plugin failure, or wrong resource selector
VM sees the VF but cannot use RDMA	Guest driver, VFIO ownership, firmware mode, IOMMU, or unsupported combination
InfiniBand link is physically up but traffic does not flow	Missing or unhealthy subnet manager, partition issue, or fabric-routing state
Inference tail latency spikes during model rollout	Registry, model-cache, storage, or north-south congestion rather than GPU execution

NVIDIA’s NCCL troubleshooting guidance correctly starts with topology, low-level fabric, bandwidth, latency, and NIC diagnostics before changing communication-library tuning. That order prevents teams from hiding a physical or configuration fault behind environment-variable experimentation.

Reference Topology for Inference-Focused Private AI

This topology is appropriate when production inference uses independent replicas or keeps model parallelism inside each node.

Design characteristics

• Use conventional resilient Ethernet for north-south serving, storage, registries, control-plane services, and monitoring.
• Size links from request volume, model-loading behavior, token-streaming traffic, and storage demand rather than GPU count alone.
• Keep out-of-band management separate.
• Reserve a credible path for future compute NICs or RoCE without buying the full training fabric on day one.
• Add dedicated RDMA only when distributed inference, remote GPU state, or direct storage paths prove that east-west communication is material.

This is the lowest-complexity architecture and often the best enterprise starting point.

Reference Topology for Moderate Multi-Node Training

This topology fits a small group of GPU servers running distributed fine-tuning or training, often across two to eight nodes. The node range is a planning example, not a fabric threshold.

Design characteristics

• Use a dedicated or strongly isolated RDMA compute plane.
• Prefer a nonblocking leaf design at this scale unless testing proves an acceptable alternative.
• Keep storage and management on resilient Ethernet unless the validated storage design calls for a dedicated RDMA path.
• Use two paths or planes when job availability justifies the cost.
• Choose RoCE when the organization can operate DCB, ECN, PFC policy, and host-to-switch telemetry consistently.
• Choose InfiniBand when a dedicated fabric, subnet-management model, and established IB operational skill are a better fit.
• Validate single-node, two-node, and full-cluster collective performance before production acceptance.

Reference Topology for Large Distributed Training

Large synchronous training requires a fabric designed as part of the compute system rather than as shared data-center connectivity.

Design characteristics

• Use a rail-optimized, nonblocking leaf-spine or super-spine topology.
• Use multiple independent planes when the platform’s availability and performance objectives justify them.
• Preserve GPU, NIC, PCIe, rail, and switch locality through cabling and scheduling.
• Deploy dedicated fabric management, telemetry, topology validation, and change control.
• Keep storage, service, and out-of-band networks distinct from the compute failure domain.
• Treat optics, cabling, firmware, and topology files as controlled platform configuration.
• Select InfiniBand or a validated RoCE platform from workload evidence, supportability, and operating maturity, not from ideology.

At this scale, the cost of specialized networking should be compared with the cost of thousands of accelerators waiting at synchronization points.

A Practical Fabric Decision Framework

Use the following sequence before selecting a product or bill of materials.

Decision Matrix

Scenario	Default direction	Why	Evidence that can change the answer
Independent inference replicas	Conventional Ethernet	Traffic is mainly north-south and storage-oriented	Cross-node model parallelism, remote state, or measured east-west bottleneck
Single-node training or fine-tuning	Conventional Ethernet plus strong storage	Scale-out collectives are absent	Dataset or storage path requires validated RDMA
Moderate multi-node training	Benchmark first, then RoCEv2 or InfiniBand	Communication may or may not dominate	NCCL tests and real job traces show acceptable TCP performance
Large synchronous training	Dedicated RDMA, rail-optimized, near nonblocking	Collective delay directly consumes GPU time	A validated alternative meets completion-time and resilience objectives
Shared training and inference platform	Separate traffic classes and often separate compute fabric	Mixed traffic creates competing bursts and operating risks	Strict scheduling and measured convergence prove isolation is sufficient
Virtualized multitenant platform	Ethernet for general traffic, selective SR-IOV or passthrough for RDMA	Direct devices reduce mobility and increase lifecycle constraints	A validated virtualized GPUDirect design meets recovery and support needs
Storage-heavy RAG or data engineering	High-capacity Ethernet, optionally storage RDMA	Data movement may dominate without GPU collectives	Direct-to-GPU storage path and measured client throughput justify RDMA
Small team with limited fabric operations	Conventional Ethernet until skills and telemetry mature	Complexity can reduce availability more than it improves speed	Managed or validated solution transfers enough operational burden

Workload Size and GPU Count

Larger jobs increase the probability that the fabric matters, but count should remain a screening question. Record nodes, GPUs per node, rails per node, racks, expected concurrent jobs, and failure reserve. Then map those counts to actual communication.

Model Communication Pattern

The communication pattern is the strongest technical criterion. Measure bytes transferred, collective type, synchronization frequency, rank skew, step time, and the proportion of time spent waiting on communication.

Operational Maturity

RoCE requires mature Ethernet automation, queue and buffer governance, telemetry, and host-fabric coordination. InfiniBand requires subnet-manager, partition, topology, routing, and IB telemetry skills. Conventional Ethernet is simpler, but still needs capacity and failure engineering.

The right fabric is the most advanced design the organization can operate reliably, not the most advanced design it can purchase.

Budget and Reversibility

Budget analysis should include switches, optics, cabling, NICs, DPUs, licenses, fabric management, support, lab capacity, spare parts, training, and operational labor. It should also include the cost of accelerator idle time and the disruption of retrofitting a compute fabric later.

The three community discussions supplied with this assignment reinforce several practical lessons.

The RKE2 discussion shows how operator components can remain not ready even when the cluster itself is healthy. The lesson is not that RKE2 is unsupported. The current platform matrix now lists explicit RKE2 combinations. The lesson is that distribution, operating system, container runtime, Node Feature Discovery, labels, driver state, and operator component readiness must be verified as one stack.

The Network Operator and OpenSM discussion exposes a control-plane misunderstanding. Kubernetes can deploy drivers, CNIs, and device plugins, but an InfiniBand fabric still needs a functioning subnet manager and fabric-management design. The cluster operator does not replace the fabric control plane.

The DGX B200 SR-IOV discussion shows a different boundary. A GPU scheduler such as Run:ai is not inherently required to create SR-IOV resources. The foundational requirements are working physical interfaces, correct firmware and BIOS settings, PF and VF configuration, device-plugin discovery, CNI and IPAM design, and accurate resource requests. A scheduler can add policy and fairness later, but it cannot repair a broken hardware exposure path.

Community posts are useful because they show where experienced teams become confused. They should trigger validation questions, not override current product documentation.

Validate the Fabric Before Procurement and Again Before Production

A defensible decision requires evidence from the intended workload, not only a synthetic bandwidth number.

Establish the Baseline

Start with one node. Validate GPU-to-GPU paths, PCIe and NUMA topology, local storage, framework behavior, and expected single-node throughput. This separates fabric problems from application and compute problems.

Add the Smallest Scale-Out Case

Run the same workload across two nodes. Validate interface selection, RDMA device exposure, GPUDirect use, collective bandwidth, collective latency, MTU, route or partition state, and failure behavior.

Test the Communication Pattern, Not Only a Generic Benchmark

Use collective tests to characterize the fabric, but also run representative training, fine-tuning, inference, and checkpoint workflows. Record:

• useful samples or tokens per second
• step time and communication share
• per-rank skew
• GPU utilization during collectives
• checkpoint completion time
• storage read and metadata latency
• inference throughput and tail latency
• recovery after path or node loss

Create Congestion Deliberately

Run competing jobs, storage reads, checkpoint writes, image pulls, and telemetry export. Confirm that queue policy and fabric isolation behave as designed. A fabric that only works while empty is not production-ready.

Inject Failures

Disable a link, rail, leaf, or storage path within the approved test environment. Confirm application behavior, alternate-path use, telemetry, alerting, and recovery. For InfiniBand, validate subnet-manager high availability and partition behavior. For RoCE, validate congestion response and PFC watchdog behavior.

Define Acceptance Evidence

The production gate should include:

• supported component and version matrix
• topology and cabling record
• MTU validation from workload endpoints
• collective and representative-workload results
• congestion and failure-test evidence
• telemetry dashboard and alert ownership
• rollback and recovery procedure
• known limitations and exceptions
• lifecycle and upgrade sequence
• responsible owner for host, fabric, Kubernetes, storage, and application layers

A green switch dashboard is not enough. The platform is ready when the workload meets its service objective and the operations team can explain why.

Conclusion

Enterprise AI does not need one universal network. It needs a set of traffic planes with different performance, resilience, security, and operational requirements.

Conventional Ethernet is the right default for management, north-south inference, platform services, and many storage paths. It is also the sensible starting point when workloads remain inside one GPU node or when evidence shows that scale-out communication is not limiting useful work.

RoCEv2 and InfiniBand become justified when inter-node communication enters the workload’s critical path. RoCE keeps the design inside an Ethernet and IP ecosystem but demands disciplined congestion engineering. InfiniBand creates a purpose-built RDMA fabric but adds a distinct subnet-management and operating model. Neither choice eliminates the need for topology alignment, version control, telemetry, failure testing, and application-level validation.

The final decision should be traceable: workload communication pattern, GPU and node scale, storage behavior, service objective, operational maturity, support matrix, and total cost. When those inputs are visible, the fabric choice stops being a vendor debate and becomes an engineering decision.

External References