Your AI Factory Is a Data Pipeline: Where PowerScale, PowerFlex, vSAN, Object Storage, and Local NVMe Belong

Introduction

Enterprise AI architecture is often drawn from the compute layer outward. The GPU cluster sits in the middle, Kubernetes or virtual machines surround it, and storage appears as one cylinder at the bottom labeled data. That drawing is convenient, but it hides the design problem that causes many AI platforms to underperform or become expensive to operate.

An AI factory is not one workload. It is a data pipeline that repeatedly lands, transforms, shares, reads, rewrites, versions, serves, observes, protects, and eventually retires data. The raw corpus entering the platform does not behave like a vector database. A model checkpoint does not behave like a container registry database. Temporary shuffle data does not deserve the same protection as an approved production model. Telemetry does not need the same placement as a training dataset.

The storage decision should therefore begin with data classification and access behavior, not with a product name. This article builds a practical placement model for local NVMe, vSAN, PowerFlex block storage, PowerScale NFS, and object storage. It also shows how those services can be combined in a VCF 9.1 AI platform without pretending that one of them should carry every stage of the lifecycle.

TL;DR

There is no universal enterprise AI storage tier. Use PowerScale NFS when many workers or tenants need concurrent POSIX-style access to shared datasets, curated content, distributed checkpoints, or RAG source files. Use PowerFlex block storage or vSAN for low-latency ReadWriteOnce platform state, databases, queues, registry metadata, and stateful Kubernetes services. Use object storage for durable raw data, model artifacts, long-retention telemetry, lifecycle-managed copies, and immutable protection. Use local NVMe for disposable scratch, staging, cache, shuffle, and hot model files that can be rebuilt.

The correct architecture is a placement policy:

• classify the data by mutability, sharing, I/O shape, durability, and lifecycle;
• express the intent through storage classes, access modes, buckets, and retention policies;
• measure throughput, latency, metadata rate, and cache effectiveness under the real workload;
• protect durable data outside the failure domain that serves it;
• treat storage-induced GPU idle time as a platform defect that must be measured and corrected.

The Wrong Question Is Which Storage Platform Is Best for AI

The phrase AI storage collapses several incompatible workload patterns into one purchasing category. It encourages teams to compare products by headline throughput or usable capacity, then place every AI service on the winner. The result may look standardized, but it usually creates one or more of these problems:

• a high-throughput file platform is forced to host transactional databases that need predictable latency;
• a block platform is stretched into a shared-file role that the application does not coordinate safely;
• object storage is mounted through a gateway and treated as if it were a normal POSIX file system;
• durable shared arrays are used for temporary shuffle data that should have remained local;
• local NVMe is treated as durable because it is fast;
• every checkpoint and intermediate copy receives the same retention treatment as a production model;
• GPU capacity is added when the real bottleneck is data loading, metadata, network, or cache refill.

NVIDIA’s DGX SuperPOD reference architecture separates high-performance storage from user storage because the first is optimized for parallel throughput while the second is optimized for NFS access, IOPS, metadata, logs, configuration, and shared administrative use [1]. The useful lesson is not that every enterprise should reproduce that exact design. The lesson is that even a tightly integrated GPU platform assumes different storage services for different data behaviors.

The architecture question is therefore:

Which storage contract best matches this data stage, this access pattern, this recovery objective, and this operating model?

Classify the Data Before Choosing the Storage

A useful classification model asks eight questions before a storage platform is selected.

Classification input	Architecture question	Why it changes placement
Durability	Can the data be rebuilt, or is it a system of record?	Rebuildable data can use local or short-lived capacity. Durable data needs snapshots, replication, backup, or immutability.
Sharing	Does one node write, or do many nodes need concurrent access?	Single-writer state usually fits block-backed RWO. Shared training and curation often require RWX file access or an object-native application.
Mutability	Is the data append-only, repeatedly rewritten, or immutable after approval?	Mutable databases, append-heavy logs, and immutable artifacts have different consistency and protection needs.
I/O shape	Are operations large and sequential, small and random, or metadata-heavy?	Aggregate bandwidth, IOPS, response time, and namespace performance become different bottlenecks.
Locality	Must the first byte arrive in milliseconds, or can data be staged?	Local NVMe and cache can reduce repeated remote reads, but introduce synchronization and eviction requirements.
Recovery	What are the RPO, RTO, and restart-window requirements?	A restart checkpoint may need rapid recovery but short retention. A production model may require long retention and immutability.
Tenancy	Which teams, projects, or namespaces may read, write, or delete it?	Access zones, buckets, namespaces, quotas, encryption keys, and storage policies must align to the tenant boundary.
Lifecycle	When should data move, become immutable, be archived, or be deleted?	Lifecycle policy changes capacity cost, legal exposure, and the number of uncontrolled copies.

This classification should produce a storage service definition, not a direct product choice. Examples include:

• ai-rwx-throughput for shared file datasets;
• ai-rwo-low-latency for transactional platform state;
• ai-object-governed for durable artifacts and raw data;
• ai-local-ephemeral for rebuildable scratch;
• ai-immutable-protection for retained copies outside the primary failure domain.

The platform team can then map those contracts to PowerScale, PowerFlex, vSAN, object storage, or node-local media. That indirection matters because applications should request a capability without embedding a storage product name into every manifest.

The following decision path is intentionally simple. It does not replace testing, but it prevents the most common category errors.

The key point is that the decision starts with survival, sharing, and interface semantics. Brand selection comes later.

The AI Lifecycle Is Not One Storage Workload

The same information may move through several storage contracts during its life. A raw document can arrive in an object bucket, be copied into a shared curation workspace, be transformed into training shards, be staged onto local NVMe, contribute to a checkpoint on shared storage, and eventually produce a governed model artifact in a registry.

A good design makes movement between these stages explicit. It records which copy is authoritative, which copy is a cache, which copy is protected, who owns deletion, and which event promotes data from one lifecycle state to another.

What Each Storage Architecture Is Actually Good At

The five storage choices in this article are not interchangeable. Each one offers a different interface, locality model, failure model, and operational burden.

Storage architecture	Natural access contract	Strongest AI roles	Poor fit or caution	Metrics that matter most
Local NVMe	Node-local filesystem, ephemeral volume, or local PV	Scratch, shuffle, staging, decompressed shards, hot model cache, temporary embeddings, compile cache	Durable records, shared state, anything that must survive node replacement without a copy elsewhere	Local read/write bandwidth, queue depth, fill rate, eviction rate, cache hit ratio, rebuild time
vSAN	vSphere-integrated block-backed VM and Kubernetes storage; RWX through vSAN File Service where enabled	VCF management and workload-domain storage, VM disks, RWO Kubernetes state, compact platform services, registries and databases with validated policies	Treating all HCI capacity as a shared training file system; mixing bursty AI I/O with management workloads without policy and headroom	p95/p99 latency, IOPS, resync impact, policy compliance, congestion, usable capacity, failure-domain health
PowerFlex block	VMFS, block device, filesystem PVC, or raw block through supported integration	High-performance VM datastores, RWO databases, vector-service state, queues, registry metadata, job-specific checkpoint volumes	Shared writable filesystem semantics unless a cluster-aware application and supported multi-attach design are proven	p95/p99 latency, IOPS, bandwidth, rebuild behavior, pool headroom, path health, topology placement
PowerScale NFS	Shared POSIX-style file namespace, commonly RWX through CSI	Shared datasets, curation workspaces, training shards, RAG source content, shared checkpoints, home directories, collaborative file workflows	Latency-sensitive transactional databases without application validation; uncontrolled millions-of-files namespaces	Aggregate GiB/s, metadata ops/s, open/stat latency, file-count behavior, NFS client concurrency, network saturation
Object storage	S3-compatible object API, bucket and key semantics	Raw data, model artifacts, large immutable objects, long-term telemetry, backup copies, lifecycle-managed data, distributed services designed for object storage	Applications requiring rename-heavy POSIX semantics, in-place updates, or low-latency random block access	GET/PUT rate, first-byte latency, aggregate throughput, request errors, object-count distribution, lifecycle backlog

Local NVMe Belongs in the Fast but Disposable Layer

Local NVMe is where data should go when proximity to the worker matters more than persistence. It is ideal for temporary data-loader caches, unpacked training shards, shuffle files, intermediate features, model compilation artifacts, and inference model caches.

The design condition is simple: the platform must be able to delete the node and rebuild the data. That means every local copy needs a durable source, a refill process, capacity controls, and a clear eviction policy.

Local NVMe also changes scheduling. A workload that stages 4 TB onto one worker cannot be moved freely without restaging. Local persistent volumes, device passthrough, or host-local datastores can create node affinity and maintenance constraints. In virtualized Kubernetes, an emptyDir volume may be backed by the worker VM’s virtual disk rather than physical host NVMe. Architects must verify the actual media path before claiming that a workload is using local NVMe.

vSAN Belongs Where VCF Integration and Policy Are the Priority

vSAN is a natural fit for VCF management-domain services and for workload-domain services that benefit from vSphere integration, Storage Policy Based Management, lifecycle alignment, and VM-level operational consistency. It can support Kubernetes RWO volumes through the vSphere storage integration, while vSAN File Service can provide NFS-backed RWX volumes when that service is enabled and supported [6].

The caveat is resource coupling. In an HCI design, storage performance and capacity are linked to the cluster that also runs compute. A large checkpoint burst, resync event, maintenance operation, or noisy tenant can affect other services if policies, network design, and reserves are weak. The decision should therefore be based on measured latency and failure behavior under concurrent cluster operations, not on idle-state benchmark results.

PowerFlex Belongs in the Low-Latency Shared Block Layer

PowerFlex provides shared block storage that can support VMFS and CSI-backed volumes. Dell’s current CSI examples show RWO filesystems for snapshots, expansion, and clones, and also document raw block use cases that require application-aware handling [10]. That makes PowerFlex a strong fit for PostgreSQL, registry metadata, queues, vector-service state, control-plane databases, and other services that want low-latency persistent block storage.

For VCF designs, support must be treated precisely. Broadcom documents supported PowerFlex configurations with ESXi 8.x and 9.x, while Dell publishes a VCF 9.0 implementation guide for PowerFlex-based principal storage [7], [8]. A VCF 9.1 design must still validate the exact PowerFlex release, SDC or NVMe/TCP path, server configuration, driver, firmware, lifecycle sequence, and storage-model support. A nearby 9.0 guide is useful engineering evidence, but it is not a substitute for a 9.1 support matrix.

PowerScale Belongs in the Shared Data Layer

PowerScale NFS is most valuable when many workers, pods, VMs, or teams need concurrent access to the same file-oriented data. Dell’s PowerScale CSI documentation shows RWX provisioning, snapshots, expansion, cloning, topology, and controller high availability [9]. A Dell EKS Anywhere implementation article also demonstrates the practical RWX pattern, but it should be read as an implementation example rather than as a current support statement [24].

PowerScale is therefore a strong candidate for curated corpora, shared training shards, RAG source content, distributed checkpoint directories, team workspaces, and file-based model repositories. The design still depends on the workload. A directory containing 20 large Parquet shards creates a different load from a directory containing 200 million tiny JSON files. Both may occupy the same capacity, but they stress the namespace, clients, and network very differently.

Object Storage Belongs in the Durable API-Native Layer

Object storage is the default candidate for data that is large, durable, globally addressable by key, lifecycle-managed, or retained independently of the compute platform. Raw datasets, immutable training snapshots, model artifacts, long-term logs, backup copies, and evidence packages often fit this model.

The tradeoff is semantic. Object storage is not a block device and is not a normal POSIX filesystem. Applications that support S3-style APIs can scale well with it. Applications that depend on atomic rename, file locking, memory-mapped random access, or repeated in-place updates may not. A filesystem gateway can bridge interfaces, but it does not erase the semantic and performance differences.

Object storage also makes lifecycle and immutability first-class controls. S3 lifecycle policies can transition, archive, or delete objects, while Object Lock demonstrates retention-period and legal-hold semantics [17], [18]. An enterprise object platform may implement equivalent concepts differently, so the exact control must be verified rather than assumed from the API label alone.

ReadWriteOnce, ReadWriteMany, and CSI Are Architecture Contracts

Kubernetes access modes are not decorative YAML fields. They express how a volume is expected to be mounted, but they do not automatically provide application-level coordination or security. Kubernetes defines RWO as read-write by one node, RWX as read-write by many nodes, and RWOP as read-write by one pod. The documentation also warns that access modes are primarily used for matching and mounting; they do not by themselves enforce write protection after the volume is mounted [2].

That leads to four practical rules:

• use RWO or RWOP for databases, queues, registry metadata, and services with one active writer;
• use RWX for shared datasets, collaborative curation, distributed file-based checkpoints, and multi-worker readers or writers;
• use object APIs when the application understands buckets and keys, rather than forcing object storage through a PVC;
• use ephemeral volumes for scratch that should disappear with the pod, while remembering that the actual backing media must be verified [3].

CSI integration turns these contracts into provisioned storage. PowerScale naturally maps to shared RWX file services. PowerFlex naturally maps to RWO block-backed services, with raw multi-attach reserved for cluster-aware applications and validated configurations. vSAN commonly supports RWO through the vSphere integration, with RWX provided through vSAN File Service when enabled. Object storage usually uses a native SDK, gateway, or supported Container Object Storage Interface implementation rather than a conventional CSI filesystem volume.

The following example uses capability-oriented storage class names. The platform team should map these names to the approved PowerScale, PowerFlex, or vSAN implementation.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: curated-dataset
  namespace: ai-team-a
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ai-rwx-throughput
  resources:
    requests:
      storage: 20Ti
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vector-metadata
  namespace: ai-team-a
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ai-rwo-low-latency
  resources:
    requests:
      storage: 2Ti
---
apiVersion: v1
kind: Pod
metadata:
  name: training-worker
  namespace: ai-team-a
spec:
  containers:
    - name: trainer
      image: registry.example.internal/ai/trainer:approved
      volumeMounts:
        - name: dataset
          mountPath: /data
          readOnly: true
        - name: scratch
          mountPath: /scratch
  volumes:
    - name: dataset
      persistentVolumeClaim:
        claimName: curated-dataset
    - name: scratch
      emptyDir:
        sizeLimit: 2Ti

Change the capacity, namespace, image, and storage class names to match the platform’s approved contracts. Successful execution means the PVCs bind, the pod schedules in the intended topology, the shared dataset is readable from multiple workers, scratch lands on the intended node media, and checksum tests survive pod replacement where persistence is expected.

Common failures include an RWX request against a block-only class, an RWO multi-attach error after rescheduling, a topology rule that leaves a PVC pending, a reclaim policy that deletes data unexpectedly, or an emptyDir volume that consumes the worker’s root disk instead of the intended NVMe tier.

Measure the Workload, Not the Product Sheet

Storage sizing for AI should begin with the service envelope. Capacity alone is not sufficient. The architecture team needs a measurable requirement for bandwidth, IOPS, latency, metadata, concurrency, protection overhead, and recovery time.

Training Read Throughput

A first-order throughput estimate is:

Required shared read throughput =
workers x bytes consumed per second per worker
x (1 - verified local cache hit ratio)
x headroom factor

Illustrative example:

32 workers x 180 MB/s x (1 - 0.20 cache hit) x 1.20 headroom
= 5,529.6 MB/s
= approximately 5.5 GB/s sustained shared read throughput

The calculation is only useful if the bytes-per-second figure is measured from the real loader, file format, augmentation pipeline, and model. Compressed files may reduce storage traffic but increase CPU pressure. Small records may shift the bottleneck from bandwidth to metadata. Distributed training may add synchronization stalls that look like storage delays.

Checkpoint Write Throughput

Checkpoint requirements should be expressed as a completion window:

Minimum checkpoint throughput = checkpoint size / allowed write window

A 2.4 TB checkpoint that must complete in five minutes requires approximately 8 GB/s of sustained application write throughput before protocol, replication, snapshot, and contention overhead are added. If the checkpoint blocks training, the business value of shortening that window may be higher than the value of increasing peak benchmark throughput elsewhere.

Metadata and Small-File Performance

A small-file corpus needs explicit metadata targets:

• opens, stats, creates, renames, and deletes per second;
• p95 and p99 namespace-operation latency;
• directory width and depth;
• file-count growth rate;
• concurrent client count;
• cache-warm and cache-cold behavior.

A platform can deliver excellent large-block throughput and still starve GPUs while workers wait to discover and open millions of tiny files. Sharding records into larger containers such as Parquet, WebDataset tar files, or another application-appropriate format can reduce metadata pressure, but it changes update granularity and recovery behavior. That is an application and data-engineering decision, not a storage tuning trick.

Block Latency and IOPS

Databases, vector metadata, queues, and registry services need latency measured at the application-relevant block size, read/write ratio, queue depth, and concurrency. Report p95 and p99 response time, not only average latency. Test during snapshots, rebuilds, maintenance, and neighboring workload bursts.

Object Performance

Object workloads need GET and PUT request rates, first-byte latency, aggregate throughput, object-size distribution, error rate, retry behavior, multipart-upload behavior, and listing performance. A benchmark using 1 GB objects does not validate a RAG ingestion pipeline built from 50 KB documents.

GPU Data Starvation

Low GPU utilization is not proof of a storage bottleneck. It may be caused by CPU preprocessing, insufficient data-loader workers, Python serialization, network loss, synchronization barriers, or memory pressure. The commissioning test must correlate:

• GPU active time and memory utilization;
• batch preparation and queue-empty time;
• file or object read latency;
• storage bandwidth and IOPS;
• network throughput, drops, and retransmits;
• CPU utilization and run queue;
• cache hit ratio and refill time;
• training-step variance across workers.

A useful initial acceptance target is to keep storage-attributable wait below 5 percent of training-step time at the planned concurrency, then refine that threshold from workload economics. It is not a universal standard. The important requirement is that the team can measure the wait, reproduce it, and distinguish it from CPU, network, or framework stalls.

Data-Stage-to-Storage Decision Matrix

The matrix below is a starting placement model. The preferred storage is not automatically the only copy. Many stages require a primary service, a local acceleration layer, and a separate protection target.

AI data stage	Dominant access pattern	Access contract	Preferred primary placement	Secondary or acceleration placement	Protection and lifecycle guidance
Data ingest	Sequential writes, bursts, append, many producers	Object API or RWX landing share	Object storage for API-native ingest; PowerScale NFS for file and appliance feeds	Local NVMe buffer only when backpressure and replay are engineered	Validate checksums, record source and classification, replicate or protect before deleting the source copy
Raw datasets	Large durable objects, mostly read, low mutation	Object API; sometimes shared read-only file	Object storage as system of record; PowerScale when POSIX consumers dominate	PowerScale working copy or local staged subset	Version, retain, and apply legal or policy holds where required; use lifecycle tiers for cold data
Data curation	Mixed reads/writes, rename, metadata, collaborative access	RWX file plus RWO workflow state	PowerScale NFS for shared workspace; PowerFlex or vSAN for pipeline databases	Local NVMe for temporary transforms	Snapshot active work, publish immutable curated versions, delete abandoned intermediates by policy
Training datasets	Parallel reads, large sequential shards, repeated epochs	RWX file or object-native reader	PowerScale NFS for POSIX and shared-file loaders; object storage for engines designed for object APIs	Local NVMe cache on GPU workers	Keep an authoritative version, measure cache hit ratio, retain data lineage and manifest hashes
Checkpoints	Large burst writes, periodic, restart-sensitive	RWX for distributed shared checkpoint; RWO for job-specific writer	PowerScale for multi-worker shared checkpoints; PowerFlex or vSAN for single-writer job volumes	Local NVMe temporary checkpoint before asynchronous durable copy, only if loss is acceptable	Retain enough generations to meet restart and rollback objectives; copy selected checkpoints to object storage
Temporary scratch	High-churn read/write, shuffle, spill, decompression	Ephemeral or node-local	Local NVMe	Remote scratch only when jobs must survive node loss	No backup; enforce quotas, low-watermark eviction, secure deletion where required, and rebuild procedures
Model artifacts	Large immutable binaries, versioned promotion	Object API or read-mostly file	Object storage; PowerScale when consumers require files	Local NVMe inference cache	Sign, hash, scan, version, replicate, and make approved releases immutable
Model registry	Transactional metadata plus large artifact references	RWO database plus object/file artifact store	PowerFlex or vSAN for registry database; object storage or PowerScale for artifacts	Database cache on service nodes	Back up metadata and artifacts consistently; preserve lineage, aliases, approvals, and promotion records
Container registry	Small transactional metadata plus large immutable layer blobs	RWO database/cache plus object or shared blob store	PowerFlex or vSAN for database and cache; object storage or PowerScale for blobs	Node-local image cache	Scan and sign images, replicate critical projects, protect metadata, and enforce retention for stale layers
Vector databases	Random I/O, WAL, metadata, index segments, memory-mapped files, engine-specific object use	Usually RWO block; some distributed engines use object storage	PowerFlex or vSAN for low-latency state; follow the engine’s architecture for object-backed segments	Local NVMe for hot indexes or cache	Test crash consistency, snapshot compatibility, rebuild time, quorum behavior, and index restoration
RAG source content	File and object ingest, metadata, reprocessing, retention	RWX file or object API	PowerScale for enterprise file corpora; object storage for governed document copies	Local parsing and embedding scratch on NVMe	Preserve source ACL mapping, provenance, classification, freshness, deletion, and re-index events
Production inference	Read-mostly model load, cache, small service state	Object/file source plus local cache; RWO for service state	Object storage or PowerScale for approved models; PowerFlex or vSAN for service databases	Local NVMe model cache and runtime cache	Pre-warm critical models, verify cold-start time, protect configuration and audit state, rebuild caches automatically
Logs and telemetry	Append-heavy, small writes, index and compaction, high retention growth	RWO hot store, stream, or object archive	PowerFlex or vSAN for hot observability services; object storage for cold logs and evidence	Local buffering with bounded queues	Define retention by signal class, protect audit evidence, prevent telemetry from consuming workload capacity
Backup and retention	Sequential copy, immutable versions, infrequent restore	Object API or separate replicated file target	Immutable object storage and/or a separate PowerScale target	None on the primary failure domain	Separate credentials and failure domains, test restore, validate immutability, and record RPO/RTO evidence

The matrix deliberately separates registry metadata from registry blobs, vector metadata from vector segments, and model records from model binaries. MLflow documents the same architectural distinction: large run artifacts live in an artifact store, while parameters, metrics, and tags live in a backend database [19]. Its model registry adds lineage, versioning, aliases, and governance metadata rather than turning the artifact store into a database [20].

Vector databases reinforce the same lesson. Milvus, for example, separates metadata, object persistence for large index and log files, and write-ahead-log services [23]. The storage layout of a vector platform must follow the selected engine’s consistency and recovery model, not the generic label vector database.

Small Files and Large Files Change the Answer

AI storage conversations often quote a single throughput number. That number is nearly meaningless without an object-size and operation distribution.

Large-File Training

Large, sequential training shards favor aggregate bandwidth, parallel readers, efficient prefetch, and a network that can sustain the storage path. PowerScale or an object-native data loader may fit well. Local NVMe caching can reduce repeated remote reads across epochs.

The design questions are:

• how many workers read concurrently;
• whether every worker reads the same shards or unique shards;
• whether data is compressed;
• how the loader prefetches;
• whether cache refill competes with training traffic;
• how quickly a failed node can restage its working set.

Small-File Corpora

Millions of small files shift pressure to directory traversal, stat calls, opens, closes, permissions, and client-side metadata caches. The storage system, NFS clients, network, and application runtime all participate in the bottleneck.

The corrective actions may include:

• reorganizing directories;
• sharding tiny records into larger application containers;
• precomputing manifests;
• increasing loader concurrency carefully;
• caching metadata or staged shards locally;
• separating curation namespaces from production training namespaces;
• testing at the expected file count, not on an empty filesystem.

Mixed Workloads

A shared AI platform rarely has one pattern at a time. A nightly checkpoint burst may coincide with RAG ingestion, container image pulls, observability compaction, and another team’s training job. Commissioning must run those workloads together. Isolated component benchmarks establish a ceiling; mixed-load tests establish whether the service is operable.

Data Protection Is Part of Placement

Protection is not a feature added after primary storage selection. It is one of the placement criteria.

Snapshots Are Not Backups

A snapshot can improve recovery speed, but a snapshot controlled by the same system, credentials, and failure domain is not an independent backup. CSI snapshots are valuable for application workflows and rapid recovery, and Dell documents snapshot support for both PowerScale and PowerFlex [9], [10]. They still need an external protection design.

Replication Is Not Immutability

Replication can reproduce corruption, accidental deletion, or malicious change. PowerScale SyncIQ can replicate directory-level data, snapshots, ACLs, and metadata to another cluster [14]. That supports availability and recovery, but the replica needs retention and access controls that prevent the same failure from destroying both copies.

PowerScale SmartLock and object-locking mechanisms can create retention controls for protected data [16], [18]. The operational requirement is to verify which identities can shorten retention, delete versions, alter clocks, change replication, or disable protection. Immutability that shares unrestricted administrative credentials with the production platform is weaker than it appears.

Assign Protection by Data Class

Data class	Example	Typical protection intent
Authoritative raw data	Original documents, images, records	Durable versioned copy, retention policy, replication, legal hold where applicable
Rebuildable derived data	Tokenized shards, temporary embeddings, transformed features	Recreate from source; short retention; protect only when rebuild cost exceeds copy cost
Restart data	Training checkpoints	Multiple recent generations, rapid restore, selective long-term retention
Production artifact	Approved model, tokenizer, configuration, evaluation evidence	Immutable version, signatures and hashes, replicated copy, controlled promotion and rollback
Transactional platform state	Registry DB, vector metadata, workflow DB	Application-consistent backup, tested point-in-time recovery, low RPO where required
Operational evidence	Audit logs, access records, model deployment events	Retention based on security and compliance policy, restricted deletion, searchable restore
Disposable working data	Local cache, scratch, shuffle	No backup; bounded capacity; automatic rebuild and cleanup

Object lifecycle policies can transition or delete data according to age and state [17]. That is useful for raw data, artifacts, logs, and backups, but lifecycle rules should be linked to data ownership and legal policy. A storage administrator should not invent retention periods in isolation.

Multitenancy, Capacity Efficiency, and Governance

A multi-tenant AI platform needs more than Kubernetes namespaces. Storage boundaries must exist at the file, block, object, identity, network, quota, and encryption layers.

Multitenancy Controls

For PowerScale, use separate access zones and unique base directories where isolation is required. Dell’s OneFS guidance recommends unique base directories and separate zones for different clients or users [15]. For object storage, use tenant or project buckets, scoped credentials, bucket policies, encryption keys, and lifecycle rules. For PowerFlex and vSAN, use distinct storage classes, policies, pools or datastores where appropriate, namespace quotas, and workload topology rules.

Dell CSM Authorization can place a policy proxy between CSI drivers and compatible storage systems to apply RBAC and usage rules [13]. That can strengthen storage tenancy, but it does not replace application authorization, Kubernetes RBAC, network policy, or the backend platform’s own access model.

Storage classes should be treated as governed service offerings. A useful catalog might include:

• bronze shared file, lower-cost capacity and standard recovery;
• gold shared file, higher throughput and faster recovery;
• low-latency block, RWO with defined p99 latency and snapshot policy;
• immutable object, versioned with retention controls;
• local ephemeral, no durability and strict capacity limits.

Each class should publish capacity limits, performance envelope, access mode, reclaim policy, snapshot support, replication behavior, chargeback unit, and support owner.

Capacity Efficiency

AI data multiplies quickly. A raw corpus can create curated copies, tokenized copies, training shards, checkpoints, artifacts, embeddings, indexes, and backups. Capacity planning must count the lifecycle copies, not only the source dataset.

Required usable capacity =
authoritative data
+ active derived copies
+ checkpoint generations
+ model and registry artifacts
+ hot telemetry
+ protection copies
+ operational reserve
- verified reclaimable or deduplicated capacity

Do not assume a compression or deduplication ratio before measuring the real data. Precompressed media, encrypted artifacts, quantized weights, and frequently changing checkpoints may produce limited savings. Snapshot efficiency also depends on the change pattern and retention count.

Object lifecycle and PowerScale data-management features can reduce the cost of cold data, but capacity efficiency is not only a media-tier question. It also requires deletion ownership, dataset manifests, expiration events, abandoned-project cleanup, and visibility into who created each copy.

Governance and Lifecycle

Every durable AI data object should have enough metadata to answer:

• who owns it;
• which source produced it;
• what classification applies;
• which consent, residency, or contractual rules apply;
• which model, experiment, or index consumed it;
• whether it is authoritative, derived, cached, or protected;
• when it expires;
• who can approve deletion or legal hold;
• which copy is used for recovery.

RAG adds another governance problem. Deleting a source document may require deletion from the source store, parsing workspace, chunk store, embedding index, cache, evaluation set, and backup policy. Data lifecycle is incomplete until the downstream deletion and re-index workflow is tested.

A Reference VCF 9.1 AI Platform with PowerFlex and PowerScale

The reference pattern below uses VCF 9.1 as the operating foundation, PowerFlex for shared block services, PowerScale for shared file data, object storage for durable artifacts and protection, and local NVMe for acceleration.

It is a design pattern, not a blanket support declaration. Broadcom documents NFS and Fibre Channel storage models for VCF 9.1 [4], [5], and supports defined PowerFlex configurations with ESXi 9.x [7]. Dell’s detailed VCF implementation paper currently referenced here is for VCF 9.0 [8]. Before deployment, validate the exact VCF 9.1 build, server bill of materials, PowerFlex version and protocol, CSI/CSM versions, VKS version, firmware, drivers, lifecycle sequence, and support ownership.

Management and Workload-Domain Storage

Use vSAN or another explicitly supported principal-storage model for the management domain according to the organization’s VCF design standard. Keep management services insulated from AI data-path bursts through domain separation, storage policy, capacity reserve, and network design.

For the AI workload domain, PowerFlex can provide VMFS datastores and low-latency persistent block services where the exact configuration is supported. If the desired PowerFlex principal-storage topology is not approved for the selected VCF 9.1 lifecycle, use a supported principal-storage design and attach PowerFlex as supplemental storage rather than forcing the architecture.

A Broadcom community discussion about moving VCF environments away from vVols illustrates why principal-storage choices have long-lived migration consequences [25]. Community posts are not product support statements, but the operational concern is valid: changing a workload domain’s primary storage model can be far more disruptive than adding a supplemental data service.

PowerScale Shared Data Plane

Connect PowerScale through a dedicated high-bandwidth data network sized for aggregate worker demand, failure operation, and concurrent platform traffic. Present capability-oriented storage classes through the supported PowerScale CSI driver. Separate tenants with access zones, directories, export policy, identity, network paths, and quotas.

Recommended uses include:

• curated and versioned training datasets;
• shared RAG source repositories;
• distributed checkpoint directories;
• team workspaces and controlled exchange areas;
• file-based model artifacts where consumers require POSIX access.

Do not place every namespace in one unrestricted RWX tree. The namespace design must preserve ownership, blast radius, quota, and deletion boundaries.

PowerFlex Stateful Service Plane

Use PowerFlex-backed RWO volumes for services that need predictable block behavior:

• PostgreSQL or another registry backend;
• Harbor database and cache where the deployment model supports externalized services;
• workflow databases and queues;
• vector database metadata, WAL, or active index state when required by the selected engine;
• observability databases;
• job-specific checkpoint volumes with one active writer.

Use raw block or multi-attach only when the application is designed to coordinate writers, the CSI driver and platform support the mode, and crash-consistency tests prove recovery. A multi-attach-capable volume does not make an ordinary filesystem cluster-safe.

Object Artifact and Protection Plane

Use object storage for:

• raw authoritative datasets;
• immutable curated releases;
• model binaries and evaluation evidence;
• container layers when the registry architecture supports external object storage;
• long-retention logs and audit evidence;
• backup copies and retained checkpoints;
• vector segments for engines explicitly designed for object-backed persistence.

Harbor separates persistent data and database configuration, and can use external storage rather than only a local data directory [21]. Broadcom’s VCF Private AI Foundation 9.1 documentation also includes a local platform Harbor registry workflow for storing models used by Private AI Services [22]. The architecture should therefore separate registry metadata availability from blob capacity and protection.

Local NVMe Acceleration Plane

Use local NVMe for data that can be recreated automatically:

• staged training shards;
• decompressed records;
• shuffle and spill;
• compiled kernels;
• inference model cache;
• temporary vector-index build files;
• bounded log buffers during remote-service interruption.

Publish an explicit local-cache contract: maximum size, expected hit ratio, pre-warm method, refill bandwidth, eviction order, node-affinity behavior, and rebuild SLO. Local media is an acceleration service, not an unprotected system of record.

Observability and Commissioning

Dell CSM Observability can expose capacity, bandwidth, IOPS, response-time, and topology metrics for supported PowerScale and PowerFlex CSI resources through OpenTelemetry, Prometheus, and Grafana [12]. Use that data with VCF, Kubernetes, network, application, and GPU telemetry to build one evidence chain.

CSI functionality should also be commissioned, not assumed. Dell Cert-CSI includes volume I/O, scaling, snapshot, and multi-attach tests [11]. Add platform-specific tests for:

• PVC creation and deletion under concurrency;
• RWO rescheduling after node failure;
• RWX access from multiple worker nodes;
• snapshot and restore with checksum validation;
• volume expansion;
• topology-aware scheduling;
• CSI controller failure and leader transition;
• storage-array path failure;
• network-link loss;
• mixed training, checkpoint, registry, and telemetry load;
• recovery from a protected copy outside the primary system.

Operational Ownership and Failure Boundaries

A multi-storage AI platform only works when ownership is explicit.

Capability	Accountable owner	Required evidence
Storage service catalog	Platform architecture	Published access modes, SLOs, policies, limits, and support matrix
PowerScale data namespaces	Storage and data platform teams	Access zones, exports, quotas, identity mapping, throughput and metadata tests
PowerFlex and vSAN stateful services	Storage and VCF teams	Datastore and CSI health, latency SLOs, topology, snapshots, recovery tests
Local NVMe cache	AI platform and workload teams	Rebuild method, eviction controls, cache metrics, node-loss test
Object lifecycle and immutability	Data governance and cyber-recovery teams	Retention policy, legal-hold process, credential separation, restore evidence
Kubernetes storage classes	Kubernetes platform team	Driver versions, reclaim policy, binding mode, access mode, expansion and snapshot behavior
Dataset and model lineage	Data science and MLOps teams	Source manifests, hashes, experiment linkage, approval and promotion records
Capacity forecasting	FinOps, platform, and storage teams	Growth by data class, copy count, reserve, protection overhead, forecast variance
Incident recovery	Service owner and operations	Runbooks, RPO/RTO results, dependency map, escalation path

Failure domains must be visible. A PowerScale snapshot on the same cluster does not protect against cluster-wide administrative compromise. A PowerFlex snapshot does not protect an application if the registry database and artifact store are restored to inconsistent points. An object replica using the same privileged identity may not survive credential compromise. A local cache is expected to disappear and should not trigger a data-loss incident when it does.

A Practical Placement Workflow

Use this workflow for every new AI data service.

Define the Data Product

Name the authoritative data, owner, consumers, classification, expected size, growth, retention, and deletion event. Avoid starting with a PVC size or an array name.

Capture the Access Pattern

Measure object size, file count, read/write ratio, concurrency, sharing, append behavior, rename behavior, transaction pattern, and locality requirement.

Define the Service Objective

Set measurable requirements for aggregate throughput, p95/p99 latency, metadata operations, checkpoint window, cold-start time, RPO, RTO, and availability.

Choose the Interface Contract

Select RWO, RWOP, RWX, object API, or ephemeral node-local storage. Confirm that the application and driver support the contract.

Choose Primary and Secondary Placement

Select the durable primary, acceleration tier, and protection target separately. Do not call three uncontrolled copies a tiering strategy.

Map Tenancy and Governance

Define namespace, access zone, bucket, directory, quota, identity, encryption key, network path, lineage, and retention policy.

Prove the Design Under Failure and Concurrency

Run real data sizes, file counts, worker counts, snapshots, rebuilds, maintenance, network faults, node loss, and restore tests. Record evidence.

Monitor and Revisit

Measure service-level indicators continuously. Revisit placement when workload shape, model size, concurrency, retention, platform version, or cost changes.

Common Design Mistakes

One Giant RWX Namespace

A single shared tree is easy to create and difficult to govern. It expands blast radius, makes quotas ambiguous, mixes metadata workloads, and complicates deletion. Use explicit tenant and data-product boundaries.

Registry Databases on General Shared File Storage Without Validation

A registry has transactional metadata and large blobs. Put the database on validated low-latency persistent storage and place blobs on a supported file or object backend. Back them up consistently.

Local NVMe Treated as Durable

Fast local media can hide an incomplete protection design. Delete a worker during acceptance testing. If the team cannot restore service automatically, the local tier contains state it was not supposed to own.

Object Storage Forced Through POSIX Semantics

A gateway can be useful, but it does not make rename, locking, consistency, and small-file behavior identical to a native filesystem. Test the exact application path.

Only Large Sequential Benchmarks

AI platforms need large-stream tests, small-file metadata tests, block-latency tests, object-request tests, mixed-load tests, and recovery tests. One bandwidth chart cannot validate the pipeline.

Snapshots Presented as the Backup Strategy

Snapshots improve local recovery. Independent backup, immutability, credential separation, and restore testing provide resilience.

Storage Classes Named After Products

A class named powerscale-fast or powerflex-gold couples application manifests to today’s implementation. A class named ai-rwx-throughput or ai-rwo-low-latency describes the contract and permits controlled evolution.

GPU Expansion Before Data-Path Diagnosis

Adding GPUs to a starved pipeline increases the cost of waiting. Correlate GPU, application, CPU, network, cache, and storage telemetry before purchasing more accelerators.

Conclusion

An enterprise AI factory should not be designed around one universal storage platform. It should be designed around data stages and measurable access patterns.

PowerScale NFS belongs in the shared data plane where many workers and teams need concurrent file access. PowerFlex and vSAN belong in the persistent state plane where databases, queues, registries, VM disks, and RWO Kubernetes services need controlled latency and lifecycle integration. Object storage belongs in the durable artifact, raw-data, lifecycle, and protection plane. Local NVMe belongs next to the worker as a disposable acceleration layer.

The differentiator is not owning all five technologies. It is giving each data class an explicit contract: authoritative or rebuildable, shared or single-writer, sequential or random, mutable or immutable, hot or retained, local or recoverable. CSI storage classes, object buckets, access zones, policies, quotas, telemetry, and recovery tests should express that contract operationally.

The best storage architecture is the one that keeps GPUs supplied, preserves the right data, deletes the wrong copies, survives failure, and remains supportable through the VCF lifecycle. That answer will not be one array. It will be a governed data pipeline.

External References