Green Hardware Does Not Mean a Ready AI Platform: Commissioning VCF Private AI Services

TL;DR

A rack of healthy GPU servers is not a production-ready VCF Private AI platform. Production acceptance should prove the complete service chain: supported hardware, aligned ESXi and NVIDIA software, a healthy VCF 9.1 workload domain, stable NSX and shared infrastructure services, a ready Supervisor and VKS layer, functioning tenant controls, trusted Harbor and model supply paths, deployable Private AI Services, a working NVIDIA NIM endpoint, persistent model behavior, complete telemetry, enforceable quotas, and tested recovery.

The commissioning decision should distinguish three states:

• Installation complete: Components exist and their installers finished.
• Platform available: Administrators can create tenant boundaries and provision supported AI resources.
• Service ready: An authorized tenant can consume a governed model endpoint, with monitoring, quotas, recovery, and operational ownership proven end to end.

Do not approve production tenants until the service-ready gate has evidence, an owner, and a remediation plan for every failed control.

Introduction

AI infrastructure projects often reach an awkward point where every dashboard looks green, the GPUs appear in inventory, and the delivery team is under pressure to declare success. The servers are powered, ESXi is installed, the workload domain exists, and perhaps a test VM can run nvidia-smi. That is meaningful progress, but it is not the same as a commissioned AI service.

VCF Private AI Services sits above a long dependency chain. The service depends on physical hardware, firmware, drivers, vGPU software, VCF workload-domain health, NSX, DNS, NTP, certificates, identity, Supervisor services, Kubernetes, registries, model repositories, storage, scheduling, telemetry, and tenant policy. A defect in any layer can produce a platform that looks installed but fails when the first production tenant requests a model endpoint.

Broadcom’s VCF 9.1 and VMware Private AI Foundation with NVIDIA documentation describes the platform as an integrated path for AI workloads across Deep Learning VMs, VKS clusters, Private AI Services, Harbor-backed model storage, and NVIDIA acceleration [1]-[7]. NVIDIA’s current vGPU, GPU Operator, DCGM, and NIM guidance adds another set of compatibility and runtime dependencies that must be validated as a single bill of materials rather than as independent products [10]-[14].

This article provides an end-to-end commissioning framework for architecture teams, platform engineering teams, VCF administrators, network and storage teams, MLOps engineers, and operational acceptance authorities. The objective is simple: prove the complete path from physical GPU hardware to a governed, observable, recoverable model endpoint.

Commissioning Starts by Defining the Acceptance States

The most important design decision is not which test command to run. It is what the organization means by ready.

Readiness state	What it proves	What it does not prove	Minimum decision authority
Installation complete	Software and infrastructure components were deployed without an unresolved installer failure	Tenant usability, service behavior, governance, scale, recovery, or supportability	Build lead
Platform available	Core administrators can configure organizations, projects, namespaces, infrastructure mappings, registries, and supported AI consumption paths	A production tenant can deploy and operate a model service under realistic policy and failure conditions	Platform owner and architecture lead
Service ready	An authorized tenant can deploy or consume the approved model endpoint, telemetry is complete, quotas work, dependencies recover, and evidence is accepted	Future capacity, every model, every failure scenario, or every application integration	Service owner, operations, security, architecture, and business acceptance authority

This separation prevents a common delivery failure: treating the completion of an installation workflow as proof that the delivered service is safe to advertise to tenants.

A production commissioning plan should also declare its test boundary. At minimum, record the exact VCF 9.1 build, ESXi build, vCenter and NSX versions, server model, BIOS and firmware bundle, GPU model, NVIDIA vGPU branch, guest driver branch, Deep Learning VM image version, VKS and Kubernetes versions, GPU Operator version, Private AI Services version, NIM or model-runtime version, Harbor version, and storage classes used. Version-sensitive claims must be checked against the current Broadcom Compatibility Guide, Broadcom release notes, and NVIDIA support matrices before the test window [2], [3], [10], [11], [21].

The Complete Commissioning Path

The platform should be tested as a chain, not as a collection of product consoles. The diagram below shows the dependency order and the evidence handoff between layers.

The architecture team should notice two things. First, every upper layer inherits the failure modes of the layers below it. Second, the final acceptance test must start from the tenant boundary. An administrator proving that a component exists is not equivalent to a tenant proving that the service works with least-privilege access.

Validate the Physical GPU Platform Before VCF Commissioning

Server Firmware and BIOS Must Match the Certified Configuration

Begin with the exact server configuration, not merely the server model. The Broadcom Compatibility Guide records certified server, device, BIOS, firmware, driver, and accelerator combinations. An approved chassis with an unapproved firmware bundle is not the same tested configuration [21].

Validate and capture evidence for:

• Server model, motherboard revision, CPU generation, memory population, and NUMA topology.
• BMC, BIOS, CPLD, power-supply, riser, PCIe switch, GPU, NIC, HBA, RAID, NVMe, and drive firmware.
• BIOS settings for virtualization, IOMMU, Above 4G Decoding, Resizable BAR when required, SR-IOV where used, power policy, and PCIe generation.
• GPU slot placement, PCIe lane width, link speed, auxiliary power, thermal status, and vendor health counters.
• NIC port mapping, speed, MTU capability, RDMA or advanced passthrough settings where designed, transceiver support, and switch-side configuration.
• Storage media identity, endurance class, firmware, controller mode, queue depth, path redundancy, and vSAN or external-array certification.

Do not accept screenshots showing only that the GPU is present. Capture inventory exports and compare every component to the approved bill of materials. Run sustained GPU, network, memory, and storage burn-in tests long enough to reveal thermal throttling, PCIe instability, packet errors, path failover defects, or media faults.

Green Health LEDs Are Necessary but Insufficient

A server can pass a basic hardware health check while still failing under AI load. AI commissioning should include concurrent stress because GPU workloads can expose power, cooling, NUMA, PCIe, and storage problems that ordinary VM workloads do not.

A useful test combines:

• Sustained GPU memory allocation and compute load.
• Host CPU and memory pressure representative of preprocessing or inference support services.
• East-west and north-south network throughput.
• Repeated model-cache reads and writes.
• Storage latency measurement during concurrent GPU activity.
• BMC telemetry capture for temperature, fan speed, power draw, corrected errors, and throttling events.

The pass condition is not simply “the test completed.” The architecture team should define acceptable error counts, thermal headroom, clock stability, packet loss, storage latency, and recovery behavior before testing begins.

Align ESXi, NVIDIA vGPU Manager, Guest Drivers, and Licensing

NVIDIA vGPU is a software compatibility chain. The ESXi version, vGPU Manager branch, GPU model, vGPU profile, guest operating system, guest driver, and NVIDIA licensing method must be mutually supported. NVIDIA explicitly documents compatible and incompatible vGPU Manager and guest driver combinations, and an incompatible guest driver can prevent the vGPU from initializing [10], [11].

Commissioning should verify:

• The GPU and selected vGPU profile are supported on the exact ESXi build.
• The installed NVIDIA vGPU Manager is the approved release, not simply the newest available package.
• The guest driver branch used by Deep Learning VMs and Kubernetes worker nodes is compatible with the host vGPU Manager.
• Shared Direct, passthrough, MIG-backed vGPU, or another allocation mode matches the approved architecture.
• NVIDIA License System connectivity, entitlements, lease behavior, and failure alerts are functioning.
• Host maintenance, evacuation, vMotion, reboot, and lifecycle workflows have documented support boundaries for the selected GPU mode.
• A vGPU-enabled test VM starts on every intended host and exposes the correct profile, memory, driver, and license state.

Use an explicit compatibility record rather than relying on installer defaults. The evidence should include host VIB or component versions, guest driver versions, the selected profile, nvidia-smi output, licensing state, and links to the governing support matrices in the acceptance repository.

The following commands are useful evidence collectors. Object names and permissions will vary by environment.

# ESXi host: confirm installed NVIDIA software components.
esxcli software vib list | grep -i nvidia

# Guest or GPU-enabled worker node: confirm device, driver, memory, and license state.
nvidia-smi

# Kubernetes: confirm GPU labels and allocatable resources.
kubectl get nodes -L nvidia.com/gpu.present
kubectl get node  -o jsonpath="{.status.allocatable.nvidia\.com/gpu}"

# Confirm NVIDIA and telemetry workloads are running.
kubectl get pods -A | grep -E 'nvidia|gpu-operator|dcgm'

Successful execution should show the approved software branch, the expected GPU or vGPU profile, and the planned allocatable resource count. A zero value, missing label, unlicensed state, or node-to-node difference is a commissioning failure until explained and remediated.

Prove the VCF 9.1 Workload Domain Is Operationally Healthy

The GPU workload domain must be healthy as a VCF domain before it is accepted as an AI platform. VCF Operations diagnostics is designed to surface known issues and the operational state of private cloud infrastructure, while VCF Operations also provides monitoring, capacity, alerting, and integration capabilities [3], [19].

Validate the following domain-level conditions:

• VCF inventory is consistent across the installer or fleet view, vCenter, NSX, storage, and VCF Operations.
• vCenter, NSX Managers, NSX Edge nodes, vSAN or external storage, and management appliances have no unresolved critical health alarms.
• The cluster uses the approved vSphere Lifecycle Manager image and reports compliance.
• Host profiles, distributed switching, storage policies, HA, DRS, admission control, and failure domains match the design.
• Capacity headroom exists for Supervisor control-plane services, VKS nodes, Private AI Services components, model endpoints, telemetry, maintenance, and a host failure.
• Backup, restore, certificate, password, and lifecycle ownership are assigned.
• VCF Logs or the selected log platform receives the infrastructure and service logs required for incident response.

Do not suppress alarms to make a commissioning report green. Every suppressed alert needs a documented risk decision, expiry date, owner, and support position.

Validate NSX, DNS, NTP, Certificates, and Identity as One Dependency Set

Private AI deployments depend heavily on infrastructure services that are often tested independently. Commissioning should test them through the AI request path.

NSX Networking Must Support Every Control and Data Flow

Validate management, workload, registry, storage, model-download, telemetry, and tenant application paths. This includes:

• Supervisor and VKS control-plane communication.
• VKS worker-node networks and pod or service connectivity.
• Harbor access from administrative tools, Deep Learning VMs, VKS clusters, Private AI Services components, and model runtimes.
• Access to approved external sources such as NGC or other model repositories when the environment is connected.
• Internal mirror access when the environment is disconnected.
• Load-balancer address allocation and routing for model endpoints.
• East-west policy between model runtime, vector or data services, application components, telemetry, and management tools.
• North-south ingress, API consumers, and administrative access.
• MTU consistency, route symmetry, firewall ownership, and name-resolution behavior.

A simple ping is not enough. Test the actual protocols, TLS handshakes, image layers, model artifacts, health probes, and API request sizes that the production service will use.

DNS, NTP, and Certificates Are Commissioning Controls

The Broadcom community thread supplied with this assignment describes a VCF 9.1 deployment that completed only after an NTP problem was corrected [18]. That thread is a community signal, not authoritative support guidance, but it illustrates the larger point: time synchronization is a platform dependency, not a housekeeping item.

Validate:

• Forward and reverse DNS for every required FQDN and virtual IP.
• Consistent DNS answers from management, workload, Kubernetes, and tenant networks.
• NTP source reachability, stratum, drift, leap status, and synchronization across ESXi, vCenter, NSX, appliances, Kubernetes nodes, Harbor, and identity systems.
• Certificate chains, subject alternative names, trust stores, expiry dates, revocation behavior, and rotation ownership.
• Mutual trust between automation, Supervisor, registry, model runtime, telemetry, and application clients.
• Identity federation, group mapping, service accounts, robot accounts, token lifetime, secret rotation, and break-glass access.

Inject controlled defects. Use an invalid DNS record in a test namespace, block an NTP source, present an untrusted registry certificate, and expire a non-production token. The platform should fail visibly, generate actionable telemetry, and recover through a documented procedure.

Commission Supervisor and VKS Before Installing AI Services

vSphere Supervisor provides the integrated control plane used to expose infrastructure through Kubernetes APIs, while VKS provides the Kubernetes consumption path for containerized AI workloads [9], [17]. A Supervisor that reports healthy but lacks usable VM classes, storage classes, namespace permissions, or routable workload networks is not ready for AI.

Validate:

• Supervisor control-plane health and supported lifecycle state.
• Correct workload networks, address pools, routing, DNS, and load-balancer integration.
• Approved GPU-enabled VM classes and non-GPU support classes.
• Storage classes for control-plane nodes, worker nodes, persistent volumes, registries, and model caches.
• Namespace creation, quotas, permissions, content access, and policy mapping.
• VKS cluster provisioning, upgrade, scale-out, node replacement, and deletion.
• Kubernetes API access from approved administrative and automation locations.
• Backup and restore boundaries for namespaces, clusters, persistent volumes, and service configuration.

The first acceptance cluster should be created using the same self-service or automation path that production tenants will use. A one-off cluster built by an administrator with elevated credentials can conceal missing policies and permissions.

Validate NVIDIA GPU Operator and Device Discovery

The NVIDIA GPU Operator automates deployment of the components required to expose NVIDIA GPUs to Kubernetes, including drivers where appropriate, the container toolkit, the Kubernetes device plugin, GPU feature discovery, validation, and DCGM-based telemetry [12]. Broadcom’s Private AI documentation also identifies the GPU Operator as a dependency for GPU-accelerated VKS clusters [4], [17].

Commissioning should verify more than pod status:

• The GPU Operator version is approved for the VKS and Kubernetes versions in use.
• Driver deployment mode is deliberate. Do not allow both host-managed and operator-managed drivers to compete.
• GPU Feature Discovery labels match the physical or virtual GPU presented to each node.
• The device plugin advertises the correct resource type and quantity.
• Validator workloads complete on every GPU node.
• DCGM Exporter can read the devices and emit metrics.
• Node replacement or scale-out produces the same validated state without manual repair.
• MIG, time-slicing, vGPU, or full-GPU settings match the approved tenant isolation model.
• A pod requesting the approved GPU resource is scheduled, starts, executes a CUDA workload, and releases the resource cleanly.

Capture node labels, allocatable resources, operator custom resources, pod logs, validator results, and the exact Helm or operator values used. Treat drift between GPU nodes as a failed control.

Establish Organizations, Projects, Namespaces, and Tenant Permissions

VCF Automation organizations and projects provide the administrative and logical boundaries used to group users and infrastructure. vSphere Namespaces add resource quotas and user permissions at the Supervisor layer [8], [9]. Private AI Services in VCF 9.1 can be activated and managed within organization namespaces, which makes the mapping between these layers part of the production security model [22].

Define and test the hierarchy explicitly:

This is an operating-model mapping, not a claim that each layer is automatically created or enforced by the layer above it. The commissioning team must verify the actual integrations, assignments, and policy boundaries used in its VCF design.

The commissioning test should prove:

• Provider administrators can create and govern organizations without becoming routine tenant operators.
• Organization administrators can manage only their assigned organization scope.
• Project members see only the infrastructure, catalog content, secrets, endpoints, and actions they are entitled to use.
• Namespace permissions align with the organization and project model.
• Development and production are separated by policy, credentials, network controls, quotas, and promotion workflow.
• Service accounts use minimum required privileges and have rotation and revocation procedures.
• An unauthorized tenant cannot list, modify, invoke, or infer the existence of another tenant’s models or endpoints.
• Audit logs record administrative and tenant actions with useful identity context.

A role matrix should be part of the acceptance evidence. Test both allowed and denied actions. Security is not proven by a successful administrator workflow alone.

Validate Harbor, Image Pulls, and Model Downloads

Broadcom documents Harbor as the local private OCI registry used for Private AI Services components and as a central model store that can distribute validated models across Deep Learning VMs, Kubernetes clusters, and Private AI Services workloads [5]-[7]. That makes Harbor part of the runtime supply chain, not just an installation convenience.

Commission Harbor across four dimensions.

Availability and Trust

Validate high availability where required, TLS trust, DNS, load balancing, storage capacity, backup, restore, replication, garbage collection, retention, and upgrade ownership. Confirm that failure of one registry component does not leave production endpoints dependent on an undocumented manual recovery.

Authorization and Governance

Test project isolation, robot accounts, token expiry, least-privilege pull and push permissions, artifact immutability, scanning policy, signature or provenance policy where implemented, and audit logging. A model-store administrator should not automatically become a tenant endpoint administrator.

Image-Pull Behavior

Perform cold pulls from every production consumption path. A cached image on one worker node does not prove registry availability. Test pulls from a newly created VKS node, a newly deployed Deep Learning VM, and the Private AI Services runtime path.

Model-Download Behavior

Test the approved connected or disconnected workflow. In connected environments, validate egress controls, NGC credentials, gated-model credentials, download duration, retry behavior, and checksum or provenance validation. In disconnected environments, validate artifact mirroring, local availability, version pinning, and the process for promoting new artifacts into the trusted registry. Private AI Services 2.1 includes an artifact mirroring capability intended to support disconnected operations, but the local operational workflow still needs commissioning evidence [22].

Prove Private AI Services Deployment Dependencies

Private AI Services should not be installed until its prerequisite services have passed their own gates. The deployment plan should identify at least:

• Healthy VCF 9.1 workload-domain resources.
• A functioning Supervisor and namespace.
• Required VM classes and storage classes.
• NSX networking and load-balancer capacity.
• DNS, NTP, certificates, and identity integration.
• Local Harbor availability and credentials.
• Private AI Services packages and mirrored artifacts where required.
• GPU Operator components and supported GPU resources.
• Capacity for service control components, model runtimes, telemetry, and maintenance.
• VCF Automation organization and project mappings.

After installation, validate the service APIs, user interface, CLI where used, internal controllers, health probes, logs, metrics, and dependency references. Confirm that a service restart, namespace reconciliation, or node replacement returns the system to the declared desired state.

The Private AI Services installer finishing is only the installation-complete gate. Platform availability requires tenant-scoped activation and successful service use. Service readiness requires a governed model endpoint and operational evidence.

Exercise Both Deep Learning VM and Kubernetes Consumption Paths

VMware Private AI Foundation with NVIDIA supports multiple AI consumption models. Broadcom documents Deep Learning VMs for GPU-enabled virtual-machine workflows and VKS clusters for GPU-accelerated Kubernetes workloads [1], [4], [17]. These paths serve different operating models and should be commissioned separately.

Deep Learning VM Path

Validate:

• Deployment through the approved catalog, automation, or kubectl workflow.
• Correct VM class, vGPU or passthrough assignment, storage policy, network, DNS, and identity.
• Guest driver, CUDA, framework, NGC, and license compatibility.
• Persistent data disks and model-cache placement.
• JupyterLab or other approved access path, with authentication and network controls.
• DCGM metrics and log forwarding.
• Reboot, host maintenance, restore, and image lifecycle behavior.
• Removal of temporary credentials, test data, and elevated access before tenant handoff.

Kubernetes Path

Validate:

• VKS cluster creation from the production workflow.
• GPU worker-node provisioning and consistent device discovery.
• GPU Operator health, resource advertisement, scheduling, and telemetry.
• Registry secrets, NGC or model-repository credentials, and certificate trust.
• Persistent-volume provisioning and model-cache behavior.
• NetworkPolicy, ingress, service exposure, secrets, quotas, and audit logs.
• Node drain, worker replacement, pod rescheduling, and cluster upgrade behavior.

Do not treat success on one path as evidence for the other. A Deep Learning VM can succeed while Kubernetes image pulls, device discovery, storage classes, or quotas remain broken.

Deploy and Validate a Governed NVIDIA NIM Endpoint

NVIDIA NIM packages optimized inference runtimes as containerized microservices with standard APIs. On Kubernetes, the NVIDIA NIM Operator can manage deployment, scaling, health probes, GPU scheduling, and persistent model caching through resources such as NIMService and NIMCache [14]. Within VCF Private AI Services, the Model Runtime path can expose governed model endpoints from approved model content stored in Harbor [5], [7].

The architecture team should choose and document the production service boundary rather than mixing the two models accidentally:

• Private AI Services Model Runtime: Best when the platform team wants a higher-level governed model service abstracted from the underlying Kubernetes implementation.
• Direct VKS plus NIM Operator: Best when the application or platform team needs Kubernetes-native control over NIM custom resources, scaling, cache placement, and release workflows.
• Deep Learning VM plus NIM container: Useful for controlled experimentation, validation, or specialized VM-based operations, but it should not be mistaken for the governed shared-service path unless that is the approved design.

The production commissioning test should deploy one representative NIM endpoint using the exact tenant workflow. Validate:

• The tenant can select only approved model artifacts and GPU classes.
• Image and model pulls work from a cold state.
• The endpoint reaches ready state within the defined startup objective.
• Health, readiness, and liveness probes behave correctly.
• The endpoint exposes the expected API and model identifier.
• Authentication and authorization are enforced.
• A representative inference request returns a valid response.
• Concurrency, latency, throughput, GPU memory use, and error rates are measured against acceptance thresholds.
• Scaling behavior does not violate quota or create uncontrolled GPU consumption.
• Logs and metrics identify the tenant, namespace, endpoint, model, runtime, and GPU resource without exposing sensitive prompt content.

A minimal endpoint validation can use environment variables so the article does not hard-code a site-specific address or secret.

# Confirm that the endpoint is reachable and exposes the expected model metadata.
curl -fsS -H "Authorization: Bearer $NIM_TOKEN" \
  "$NIM_ENDPOINT/v1/models"

# Submit a representative completion request stored in an approved test payload.
curl -fsS -H "Authorization: Bearer $NIM_TOKEN" \
  -H "Content-Type: application/json" \
  --data @commissioning-request.json \
  "$NIM_ENDPOINT/v1/completions"

The pass condition should include status code, schema validation, expected model identity, response quality checks, latency, telemetry correlation, and audit evidence. A single successful curl response is not a complete service acceptance test.

Validate Persistent Storage and Model-Cache Behavior

Model endpoints often appear healthy after a warm start because images and model artifacts are already cached. Commissioning must prove the cold path and the recovery path.

NVIDIA’s NIM Operator documentation describes persistent model caching through NIMCache so artifacts can be populated once and reused across pod restarts and scaling events [14]. The broader NIM runtime also supports configurable cache locations. Regardless of implementation, the architecture team must know where model artifacts live, who owns the storage, how access is controlled, and what happens when a pod moves.

Test at least these scenarios:

• Initial model download to an empty cache.
• Pod restart on the same node.
• Pod reschedule to another node.
• Worker-node replacement.
• Cache volume remount after storage-path interruption.
• Registry or upstream repository unavailable while a valid cache exists.
• Cache unavailable while the registry remains available.
• Model version replacement, rollback, and concurrent old/new endpoint operation.
• Storage-capacity threshold, cleanup, and alert behavior.
• Backup and restore of required configuration and persistent metadata.

Record download duration, cache population time, endpoint readiness time, storage throughput, capacity consumed, failure messages, and recovery steps. The service-level objective should distinguish warm restart, cold restart, and complete artifact rehydration.

Prove DCGM and Infrastructure Telemetry End to End

DCGM Exporter exposes NVIDIA GPU metrics for Prometheus-compatible monitoring and can run as a daemonset on Kubernetes GPU nodes. The NVIDIA GPU Operator deploys DCGM Exporter by default unless it is disabled [12], [13]. Broadcom also documents Private AI Services metrics and DCGM-based monitoring for Deep Learning VMs [15], [16].

A production-ready monitoring design should correlate four views:

• Physical and hypervisor: Server health, PCIe errors, GPU health, ESXi state, vGPU profile inventory, host contention, network, and storage.
• Kubernetes or VM runtime: Node, pod, VM, container, PVC, image-pull, restart, and scheduling signals.
• GPU: Utilization, memory, temperature, power, clocks, ECC or XID errors, throttling, and per-workload attribution where supported.
• Model service: Endpoint readiness, request count, latency, tokens or workload-specific throughput, queue depth, error rate, model identity, and scaling activity.

The commissioning test should deliberately create a measurable workload and prove that the same transaction can be traced from the tenant request to the endpoint, runtime, GPU, host, network, and storage. Alert routing should reach the correct on-call team, and the runbook should explain which team owns each layer.

Avoid exposing sensitive prompts, retrieved documents, model inputs, or secrets in logs and metric labels. Observability must improve diagnosis without becoming a data-loss path.

Validate Scheduler Behavior, Quotas, and Capacity Protection

A platform is not multi-tenant merely because multiple tenants can log in. Commissioning must prove that one tenant cannot consume the GPU pool, starve platform services, or bypass the intended placement model.

VCF Automation supports resource quota policies for users, projects, and organizations, while vSphere Namespaces support resource quotas and permissions [9], [20]. Kubernetes adds requests, limits, priorities, taints, tolerations, affinity, and quota controls. The final design should coordinate these layers rather than assuming one quota mechanism controls everything.

Test:

• Organization, project, and namespace quotas at their intended scopes.
• Maximum GPU, vCPU, memory, storage, endpoint, and deployment counts.
• Scheduler placement across supported GPU nodes and profiles.
• Behavior when the requested GPU class is unavailable.
• Behavior when a tenant reaches quota.
• Priority and preemption policy, including protection for platform services.
• Development versus production reservations and burst rules.
• Scale-out under load and scale-in after demand falls.
• Capacity headroom during host maintenance or failure.
• Denial messages, events, metrics, and support escalation paths.

A useful negative test submits a workload that intentionally exceeds the tenant quota. The workload should remain unscheduled or be rejected with an actionable reason, while other tenants and platform services remain healthy.

# Inspect namespace quotas and current consumption.
kubectl get resourcequota -A
kubectl describe resourcequota -n 

# Inspect pending workloads and scheduler events.
kubectl get pods -A --field-selector=status.phase=Pending
kubectl describe pod  -n 

The pass condition is controlled refusal, not eventual success through an undocumented administrator override.

Use Failure Injection to Prove Recovery

Failure testing should be controlled, reversible, approved, and performed before production tenants arrive. The goal is not chaos for its own sake. It is to prove that the service fails predictably and that operational teams can restore it with evidence.

Recommended commissioning scenarios include:

Failure injected	Expected platform behavior	Required evidence
One GPU worker node drained or powered off	Workload reschedules or remains unavailable according to the declared availability design	Scheduler events, endpoint state, recovery time, capacity impact
DCGM Exporter stopped	Monitoring alert fires without breaking inference	Alert, missing-series detection, runbook execution
Harbor temporarily unavailable	Existing warm endpoints continue if designed; new cold pulls fail clearly	Endpoint tests, pull errors, alert, recovery proof
Model-cache volume unavailable	Endpoint fails or falls back according to design, without corrupting artifacts	Pod events, storage alarms, cache recovery results
DNS record incorrect	Dependency fails visibly and recovers after correction	DNS test, service events, recovery time
NTP source blocked	Drift alert appears before token, certificate, or cluster behavior becomes unsafe	Time metrics, alert, remediation log
Registry certificate untrusted	Pull is rejected rather than silently bypassing trust	TLS error, audit event, corrected trust test
Tenant quota exhausted	Additional workload is denied while existing services remain stable	Quota status, scheduler event, tenant-facing message
NIM pod or runtime process terminated	Controller recreates workload and endpoint returns within objective	Restart event, readiness timeline, request success
One NSX Edge or network path fails	Traffic follows the approved availability behavior	Routing, load-balancer, flow, and endpoint evidence
Credential revoked	Access fails immediately or within documented token lifetime	Audit log, API response, token-expiry evidence

Every test needs a declared success criterion, maximum recovery time, rollback point, safety owner, and data-protection check. Failed tests are not reasons to hide evidence. They are inputs to remediation and retest.

Commissioning Scorecard

Use the following scorecard as the operational acceptance record. Replace the sample evidence descriptions with links or identifiers from the organization’s evidence repository.

Commissioning control	Pass / Fail	Evidence required	Accountable owner	Remediation if failed
Server model and component certification	PASS / FAIL	Broadcom Compatibility Guide record, OEM BOM, serial inventory	Infrastructure architect	Replace unsupported component or move to certified configuration
BIOS and firmware baseline	PASS / FAIL	Exported firmware report, approved baseline comparison	Hardware operations	Apply approved OEM bundle and rerun burn-in
GPU health and PCIe placement	PASS / FAIL	BMC health, PCIe width and speed, sustained GPU test	Hardware operations	Correct slot, riser, power, cooling, or replace GPU
NIC and network firmware	PASS / FAIL	NIC inventory, firmware, switch-port and error counters	Network engineering	Align firmware, optics, MTU, switch configuration, or replace device
Storage certification and performance	PASS / FAIL	HCL check, path test, latency and throughput results	Storage engineering	Correct firmware, policy, pathing, capacity, or media
ESXi lifecycle compliance	PASS / FAIL	vLCM image and compliance report	VCF platform team	Remediate host image and rerun cluster compliance
NVIDIA vGPU Manager alignment	PASS / FAIL	Host component version and NVIDIA support-matrix record	VCF platform team	Install approved branch and reboot or remediate hosts
Guest driver and licensing	PASS / FAIL	nvidia-smi, guest driver, profile, license status	AI platform team	Align guest image, driver branch, profile, or license service
VCF 9.1 workload-domain health	PASS / FAIL	VCF inventory, VCF Operations diagnostics, active-alert review	VCF service owner	Resolve alarms, drift, capacity, lifecycle, or integration defects
NSX routing and load balancing	PASS / FAIL	Flow tests, route tables, load-balancer checks, failover result	Network engineering	Correct segments, routes, firewall, MTU, pool, or edge capacity
DNS readiness	PASS / FAIL	Forward and reverse lookup matrix from each network zone	Core infrastructure	Correct records, forwarding, search domains, or resolver reachability
NTP readiness	PASS / FAIL	Synchronization status, drift metrics, failure-injection result	Core infrastructure	Correct sources, firewall, hierarchy, or host configuration
Certificate trust and rotation	PASS / FAIL	Chain validation, SAN review, expiry report, rotation test	PKI and security	Replace certificate, update trust store, or assign rotation workflow
Identity and role mapping	PASS / FAIL	Group-to-role matrix, allowed and denied action tests	IAM and security	Correct federation, groups, service accounts, or policy
Supervisor readiness	PASS / FAIL	Control-plane health, network, VM class, storage class evidence	VCF platform team	Correct Supervisor configuration or lifecycle state
VKS lifecycle readiness	PASS / FAIL	Create, scale, upgrade, replace-node, and delete tests	Kubernetes platform team	Correct cluster class, image, networking, storage, or automation
GPU Operator health	PASS / FAIL	Operator resources, validator logs, node labels, allocatable GPUs	Kubernetes platform team	Pin supported version, correct driver mode, or repair nodes
Organization and project governance	PASS / FAIL	Tenant hierarchy, project mappings, administrator scope tests	Cloud governance	Correct organization, project, policy, or role design
Namespace permissions and quotas	PASS / FAIL	ResourceQuota output, allowed and denied tenant tests	Kubernetes and security	Correct quota, permission, policy, or project mapping
Harbor availability and trust	PASS / FAIL	Health, TLS, project policy, backup and restore, failover test	Registry service owner	Repair HA, storage, certificate, access, or backup design
Image-pull validation	PASS / FAIL	Cold-pull logs from new DLVM, VKS node, and runtime path	AI platform team	Correct credentials, trust, routing, mirrors, or artifact path
Model-download and provenance	PASS / FAIL	Download logs, checksum or signature evidence, promotion record	MLOps and security	Re-mirror, validate, pin version, or block untrusted model
Private AI Services deployment	PASS / FAIL	Service health, controller state, logs, namespace activation	AI platform team	Repair prerequisite, package, configuration, capacity, or service
Deep Learning VM consumption path	PASS / FAIL	Tenant deployment, GPU, storage, telemetry, reboot and recovery	AI platform team	Correct image, VM class, network, driver, storage, or policy
VKS AI workload path	PASS / FAIL	Tenant cluster, CUDA workload, scheduling, persistence, recovery	Kubernetes platform team	Correct operator, resource, storage, network, or policy
NVIDIA NIM endpoint	PASS / FAIL	Endpoint deployment, API tests, model identity, latency, auth	MLOps service owner	Correct model, image, cache, GPU, endpoint, or access configuration
Persistent storage and cache	PASS / FAIL	Cold start, restart, reschedule, capacity, restore results	Storage and MLOps	Correct storage class, access mode, capacity, affinity, or cache design
DCGM and infrastructure telemetry	PASS / FAIL	Dashboards, metrics, alerts, trace correlation, retention	Observability owner	Repair exporter, scrape, integration, labeling, or alert routing
Scheduler and quota protection	PASS / FAIL	Placement, saturation, denial, priority, and maintenance tests	Platform architecture	Correct requests, quotas, priorities, reservations, or capacity
Failure recovery	PASS / FAIL	Approved injection records, recovery time, data-integrity checks	Service operations	Remediate design or runbook and repeat the failed scenario
Operational documentation and support	PASS / FAIL	Runbooks, RACI, escalation, vendor support data, known issues	Service owner	Complete ownership, support, and lifecycle documentation
Final tenant acceptance	PASS / FAIL	Least-privilege tenant deploys and invokes governed endpoint	Acceptance authority	Keep service closed, remediate failed controls, and retest

A failed mandatory control should keep the service out of production. Conditional acceptance should be reserved for defects with documented compensating controls, a named risk owner, an expiry date, and an approved remediation window.

Evidence Required for Operational Acceptance

The commissioning package should be reproducible by someone who was not part of the build. At minimum, retain:

• Approved architecture, scope, assumptions, and acceptance criteria.
• Component and version matrix with source-verification date.
• Hardware inventory and Compatibility Guide evidence.
• Firmware, BIOS, driver, vGPU Manager, guest driver, and license records.
• VCF 9.1 workload-domain health, lifecycle, capacity, and alert exports.
• NSX topology, routing, firewall, load-balancer, and failover results.
• DNS, NTP, certificate, identity, and service-account validation.
• Supervisor, VM class, storage class, namespace, and VKS evidence.
• GPU Operator configuration, node labels, allocatable resources, and validator logs.
• Organization, project, namespace, role, quota, and policy mappings.
• Harbor health, access, artifact, backup, restore, and audit evidence.
• Cold image-pull and model-download records.
• Private AI Services installation and tenant activation evidence.
• Deep Learning VM and VKS consumption-path results.
• NIM endpoint deployment, API, performance, authentication, and scaling results.
• Persistent storage and model-cache test results.
• DCGM, VCF Operations, logs, alerts, and trace-correlation evidence.
• Failure-injection procedures, results, recovery times, and data-integrity checks.
• Known issues, accepted risks, compensating controls, and remediation backlog.
• RACI, on-call ownership, escalation path, maintenance windows, and support entitlement details.
• Final signatures from architecture, platform, network, storage, security, operations, MLOps, and service ownership.

Evidence should be time-stamped, tied to the exact environment and versions, and stored in a controlled repository. Screenshots without context are weak evidence. Prefer exported configuration, command output, test records, logs, metrics, and signed acceptance decisions.

The Production Readiness Decision

The final go or no-go meeting should answer one question: can a production tenant safely obtain and operate the promised AI service through the intended governance model?

Approve the service only when:

• The exact component stack is supported and documented.
• The VCF 9.1 domain and shared services are healthy.
• Tenant roles and quotas work for both allowed and denied actions.
• Harbor and model supply paths work from a cold state.
• Private AI Services and at least one production-representative consumption path are proven.
• A governed NVIDIA NIM endpoint meets functional and operational thresholds.
• Storage and cache recovery behavior are understood.
• Telemetry reaches the correct owners and supports diagnosis.
• Failure tests meet recovery objectives.
• Operations accepts the runbooks, evidence, support boundaries, and remaining risks.

If only the installers are complete, report installation complete. If administrators can create resources but tenants cannot yet consume a governed endpoint, report platform available. Use service ready only when the tenant-to-model path has been proven and accepted.

Conclusion

A green GPU server is an input to a private AI platform, not the finished service. VCF Private AI Services depends on an integrated chain of hardware, firmware, ESXi, NVIDIA software, VCF workload-domain services, NSX, shared infrastructure, Supervisor, VKS, tenant governance, registries, model artifacts, storage, scheduling, telemetry, and operations.

The architecture team’s job is to turn that chain into evidence. That means testing the supported bill of materials, validating every dependency through the production path, exercising both positive and negative tenant actions, deploying a representative NVIDIA NIM endpoint, measuring its behavior, and proving recovery when components fail.

The most useful commissioning outcome is not a polished green dashboard. It is a defensible acceptance record that clearly states what is installed, what is available, what is ready, who owns each layer, and what must happen when the platform stops behaving as designed.

External References