Close Menu
    Subscribe
    Can NVIDIA NIM Really Operate Disconnected? An Enterprise Guide to Air-Gapped Private AI
    Guides & Tutorials

    Can NVIDIA NIM Really Operate Disconnected? An Enterprise Guide to Air-Gapped Private AI

    gvfx00@gmail.comBy No Comments27 Mins Read


    Table of Contents

    Introduction

    NVIDIA NIM can operate without Internet access, but that statement is easy to oversimplify.

    The container does not become air-gap ready merely because an administrator pulled it once. A production NIM service depends on a complete software and artifact chain: the OCI image, model weights, model profiles, runtime manifests, GPU drivers, container runtime integration, Kubernetes operators, storage classes, certificates, secrets, and a repeatable update process. If any one of those dependencies still points to a public registry or model repository, the deployment is disconnected only until the next cold start, node replacement, scaling event, or security patch.

    NVIDIA’s current NIM for Large Language Models documentation describes a two-phase air-gap workflow: prepare the required assets on a connected system, then transfer and run them locally without outbound access or public-source API keys [1], [2]. As of the July 2026 documentation baseline, the current NIM LLM release is 2.0.8, and the 2.0 architecture differs materially from older 1.x examples that still appear in forum threads and internal runbooks [3], [4].

    The practical question is therefore not whether the NIM process can start with the network disconnected. The real question is whether the enterprise has built a closed, supportable, and testable software supply chain around it.

    TL;DR

    Yes, NVIDIA NIM can run in a restrictive or fully air-gapped environment, provided every runtime dependency is available internally and has been validated under blocked-egress conditions.

    A dependable design separates four artifact planes: container images, model artifacts and profiles, Kubernetes and GPU platform dependencies, and entitlement plus evidence records. Mirror NIM and operator images into an internal registry, preload the exact model profiles or local model store, preserve model-free runtime manifests when applicable, establish private-CA trust, and remove public credentials from the disconnected runtime. Then prove cold start, pod restart, node restart, scale-out, and rollback while outbound traffic is denied.

    The most common failure is not the GPU. It is an incomplete import: a missing profile, an untrusted certificate, a registry pull failure, a stale 1.x configuration, or a hidden call to NGC, Hugging Face, an object-download endpoint, a Helm repository, or an operating-system package repository.

    Define the Operating Model Before Designing the Mirror

    Air-gapped is often used as a broad label for several materially different network models. The controls, evidence, and operational burden change depending on which model the organization actually requires.

    Operating model Production Internet path Artifact import method Runtime source of truth Operational implication
    Connected Direct or enterprise-proxied access Pull on demand Public and internal sources Simplest lifecycle, largest external dependency surface
    Restricted egress Approved proxy and allowlist Automated through controlled egress Public sources reached through policy controls Requires proxy, DNS, firewall, and private-CA engineering
    Dark site No direct production egress Periodic import from a connected staging enclave Internal registry, model store, and package repositories Requires a formal promotion and evidence process
    Fully air-gapped No network path to public or connected networks Reviewed physical, one-way, or out-of-band transfer Only locally controlled services and media Highest assurance and highest lifecycle burden

    These definitions should be written into the architecture decision record. A dark site with a scheduled import gateway is not the same as a fully air-gapped facility. A restrictive firewall that still allows NGC through a proxy is not disconnected. Neither model is inherently better, but treating them as interchangeable produces weak controls and misleading test results.

    The design should also state whether the service must survive only loss of public connectivity or loss of an internal dependency. A NIM pod can be independent of NGC while still depending on Harbor, a shared model volume, DNS, the Kubernetes API, and an internal certificate authority. Air-gapped does not mean dependency-free.

    Treat Disconnected NIM as a Four-Plane Supply Chain

    The most useful mental model is to separate the assets that teams frequently mix together.

    Plane 1: Container images. This includes the NIM image and, for Kubernetes, the NIM Operator, GPU Operator, driver, container toolkit, device plugin, GPU Feature Discovery, DCGM, validation, and other required images. An internal OCI registry such as Harbor belongs here.

    Plane 2: Model artifacts. This includes model weights, tokenizer files, configuration files, profile metadata, checksums, and model-free runtime manifests. These artifacts belong in a NIM cache, model store, persistent volume, or supported internal object store. They are not automatically embedded in the NIM image.

    Plane 3: Platform dependencies. This includes Helm charts, custom resource definitions, admission webhooks, storage provisioners, ingress components, certificate tooling, monitoring agents, operating-system packages, and Kubernetes base images. The GPU Operator documentation explicitly notes that disconnected installations may require both a local image registry and a local operating-system package repository unless precompiled driver containers remove that package-download dependency [10].

    Plane 4: Entitlement and evidence. This includes accepted governing terms, product entitlement, approved model terms, NVIDIA AI Enterprise licensing records where applicable, image digests, model checksums, vulnerability reports, import approvals, and proof that the resulting service works without public access.

    A deployment is not ready until all four planes are complete.

    Choose the NIM Packaging Model Before Downloading Anything

    The import workflow changes depending on whether the team is using a model-specific NIM or a model-free NIM.

    Model-Specific NIM

    A model-specific NIM image includes a manifest that describes the supported model and profiles, but it does not include the model weights. During connected preparation, the administrator uses the container to discover and download the required profile into the NIM cache. The cache can be transferred directly, or it can be converted into a properly formatted model store with create-model-store [2], [5].

    This model is a strong fit when the organization wants a curated, certified deployment with a known profile and GPU combination. The critical design decision is which profile hashes to import. Downloading every profile simplifies fallback but consumes more transfer time and storage. Downloading only one profile reduces footprint but can leave the service unable to start after a GPU topology change.

    Model-Free NIM

    A model-free NIM can serve a supported model from NGC, Hugging Face, S3-compatible storage, Google Cloud Storage, ModelScope, or local storage [5], [6]. For a strict air gap, the cleanest pattern is usually a local model path or an internally hosted model repository.

    When a remote model URI is used during the connected phase, current NIM 2.0 behavior can generate and persist a runtime manifest in NIM_CACHE_PATH. If the same persistent cache is mounted later, NIM can reuse that manifest during an air-gapped restart without making authentication or outbound calls [2], [6]. Losing that cache may force manifest regeneration and break restart in the disconnected zone.

    Do Not Carry 1.x Runbooks Forward Unchanged

    NIM LLM 2.0 uses a vLLM-based architecture, replaces nim-run with nim-serve, changes configuration behavior, and replaces NIM_MODEL_NAME with NIM_MODEL_PATH for relevant workflows [4]. Older forum posts are still useful as operational signals, but they should not be treated as current deployment instructions.

    Pin the exact supported image and profile combination. Do not use a mutable latest tag in a disconnected production process. A reproducible import must identify the image by version and digest, the profile by hash, the model artifacts by checksum, and the target GPU topology by validated support data.

    Build a Controlled Import Zone

    The connected preparation system should be an enterprise service, not an engineer’s laptop that happens to have enough disk space.

    It needs controlled outbound access, NGC credentials, access to any gated model source, sufficient storage for images and model artifacts, malware and vulnerability scanning, signature verification, and a transfer mechanism approved for the target security zone. It should also match the production CPU architecture and be able to exercise the same NIM packaging workflow.

    Authenticate and Accept Terms Before the Change Window

    NVIDIA requires NGC authentication to pull NIM images, using the special $oauthtoken username and an NGC API key. Governing terms for a specific NIM must also be accepted before the first download [1]. These actions belong in the connected preparation phase, not in the production rollout window.

    A simple preparation sequence looks like this:

    export NIM_IMAGE="nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.8"

printf '%s' "$NGC_API_KEY" | \
  docker login nvcr.io --username '$oauthtoken' --password-stdin

docker pull "$NIM_IMAGE"
docker inspect --format="{{index .RepoDigests 0}}" "$NIM_IMAGE"

    The first command identifies the exact image tag. The login uses an API key without writing it into command history. The final command captures the immutable repository digest that should appear in the import manifest.

    Create an Artifact Manifest

    Every import batch should carry a machine-readable inventory. This is the bridge between security review, platform operations, and rollback.

    release_id: nim-llama31-8b-2026-07
source_baseline:
  nim_llm_version: 2.0.8
  documentation_review_date: 2026-07-23
container:
  source_image: nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.8
  source_digest: sha256:REPLACE_WITH_CAPTURED_DIGEST
  internal_image: harbor.ai.corp/nim/llama-3.1-8b-instruct@sha256:REPLACE
model:
  deployment_mode: model-specific
  profile_hash: REPLACE_WITH_PROFILE_HASH
  artifact_checksum_manifest: checksums.sha256
platform:
  gpu_operator_bundle: REPLACE_WITH_APPROVED_VERSION
  nim_operator_bundle: REPLACE_WITH_APPROVED_VERSION
  driver_branch: REPLACE_WITH_SUPPORTED_BRANCH
controls:
  terms_accepted: true
  vulnerability_scan_id: SCAN-REPLACE
  import_approval_id: CHG-REPLACE
  rollback_release_id: nim-llama31-8b-previous

    The values must be changed for the selected NIM, GPU, Kubernetes distribution, and internal registry. Successful execution means the manifest can be reconciled against the internal registry, model store, change record, and live workload.

    Mirror Container Images into Harbor or an Equivalent Registry

    Harbor is a reasonable fit because it provides projects, access controls, retention, vulnerability scanning, and content-trust capabilities. The same architecture principles apply to another enterprise OCI registry.

    A production registry design should include:

    • a quarantine project for newly imported artifacts
    • a promoted production project that accepts only approved digests
    • separate robot or service accounts for import, scanning, promotion, and cluster pull
    • immutable tags or, preferably, digest-pinned deployment manifests
    • TLS with a certificate trusted by every node and container runtime
    • high availability, backup, restore testing, and capacity monitoring
    • offline updates for the vulnerability scanner database
    • retention that preserves the active release, the rollback release, and evidence-linked historical versions
    • audit logs that identify who imported and promoted each artifact

    Harbor’s current production guidance requires HTTPS and a resolvable FQDN, and its scanner can be operated with a manually supplied offline vulnerability database [14]. Content trust can add signature verification, but signature enforcement should be tested with the deployment toolchain before it becomes a production gate [15].

    Mirror the Whole Dependency Graph

    For Docker-only NIM, the primary OCI artifact may be one NIM image. For Kubernetes, the graph is larger:

    • NIM service image
    • NIM Operator images
    • model puller images
    • GPU Operator images and subcomponents
    • driver container, when operator-managed
    • NVIDIA Container Toolkit components
    • device plugin and GPU Feature Discovery
    • DCGM and DCGM Exporter, when used
    • validator images
    • ingress, sidecar, policy, CSI, and observability images referenced by the deployment
    • Helm charts and custom resource definitions

    A mirror that contains only the visible NIM service image is incomplete. The next cluster rebuild or operator upgrade will expose the missing dependencies.

    Replace Public Pull Secrets with Internal Registry Credentials

    NIM Operator examples use an NGC registry secret and a separate NGC API key secret for model downloads [9]. In a fully pre-staged disconnected runtime, those public credentials should not be required. Use an internal registry pull secret for images and local credentials only for an internal model store if the storage service requires them.

    Keeping NGC or Hugging Face tokens in the disconnected namespace creates unnecessary secret-management risk and can hide an architectural defect. If the service starts only when those tokens are present, it is probably still trying to reach an external source.

    Preload Model Artifacts and Runtime Profiles

    Container-image mirroring and model-artifact staging are different workflows. NIM uses a manifest-driven model download process, and model assets are stored under NIM_CACHE_PATH, which defaults to /opt/nim/.cache and must be writable during download and cache construction [5].

    Discover the Profiles on the Target GPU Class

    Run profile discovery on hardware that represents the intended deployment. The selected profile encodes details such as parallelism, precision, and runtime compatibility.

    export LOCAL_NIM_CACHE="/staging/nim-cache"
export PROFILE_HASH="REPLACE_WITH_APPROVED_PROFILE"

mkdir -p "$LOCAL_NIM_CACHE"

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  "$NIM_IMAGE" list-model-profiles

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  "$NIM_IMAGE" download-to-cache -p "$PROFILE_HASH"

    The first container lists available profiles. The second downloads the approved profile. Success means the cache contains all referenced files and the profile can be started on the intended GPU topology before transfer.

    For stronger portability, create a model store after the cache is populated:

    export MODEL_STORE="/staging/model-store"
mkdir -p "$MODEL_STORE"

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$MODEL_STORE:/model-store" \
  "$NIM_IMAGE" create-model-store \
    -p "$PROFILE_HASH" \
    -m /model-store

    A model store produces an explicit local model directory. A transferred cache preserves the profile-oriented cache structure. Both can work, but the team should standardize one pattern and document the restart behavior.

    Preserve Runtime Manifests for Model-Free NIM

    If a model-free NIM first starts from a remote URI, retain the persistent cache containing the generated runtime manifest. Do not copy only the obvious weight files and assume the service will reconstruct the rest later. The restart test should prove that the manifest is found locally and that no authentication call occurs.

    After a pre-staged workflow has been validated, NIM_DISABLE_MODEL_DOWNLOAD=true can provide an additional guard against startup downloads in the scenarios where that setting fits the deployment. It is not a substitute for a complete cache or model store [18].

    A local-path model-free deployment avoids that dependency:

    docker run --rm --gpus all \
  --name nim-airgap \
  --shm-size=16GB \
  -p 8000:8000 \
  -e NIM_MODEL_PATH=/model-repo \
  -e NIM_SERVED_MODEL_NAME=enterprise-llm \
  -v /srv/models/enterprise-llm:/model-repo:ro \
  harbor.ai.corp/nim/model-free-nim@sha256:REPLACE

    The model directory is mounted read-only, and the container image is pinned by digest. If the selected release requires a writable cache for generated runtime data, provide a separate writable volume rather than making the governed model store mutable.

    Separate the Image Registry from the Model Cache

    The registry and model cache solve different problems and usually have different performance, retention, and security characteristics.

    Design area OCI image registry Model artifact cache or store
    Primary content Container layers, manifests, signatures Weights, tokenizer, profiles, runtime manifests
    Typical access Layer pull during scheduling or node replacement Large sequential reads during NIM startup
    Change unit Image version or digest Model version and profile
    Retention Active, rollback, security hold Active profiles, rollback models, import staging
    Integrity Image digest and signature File checksums, model manifest, provenance
    Performance risk Slow image pull and node startup Long cold start and GPU starvation during load
    Kubernetes pattern Registry pull secret PVC, ReadWriteMany storage, local model store, or internal object storage

    NVIDIA recommends local model caching for NIM Operator deployments and network-backed storage for production so multiple pods can share the cache [8]. That does not mean any network filesystem is sufficient. A model cache is part of the serving path during startup and scale-out.

    Size for Versions, Profiles, and Import Headroom

    Use a capacity model rather than a single model-size estimate:

    Required model capacity =
  Sum(selected profile artifact sizes)
  x retained model versions
  x required physical copies
  + import staging space
  + checksum and extraction workspace
  + rollback reserve
  + operational headroom

    Measure the actual profile on disk after download. Quantization, tensor parallelism, tokenizer files, engine artifacts, and model-store formatting can make the required footprint differ from the headline parameter count.

    Measure the Startup Path

    Track at least:

    • mount and attach time
    • sustained read throughput during model load
    • metadata latency and file-open rate
    • concurrent startup behavior for multiple replicas
    • cache hit and miss behavior
    • checksum-verification duration
    • time from pod scheduling to readiness
    • storage saturation during node recovery

    A fast steady-state inference service can still have unacceptable recovery time if model loading from shared storage takes too long.

    Close DNS, Proxy, Firewall, and Certificate Dependencies

    Disconnected design fails most often at trust and name resolution boundaries.

    DNS

    The production zone needs resolvable internal names for the OCI registry, model store, Kubernetes API, storage endpoints, monitoring systems, identity services, and internal time sources. Do not depend on public DNS responses to resolve internal mirrors.

    Test DNS after the public resolvers and default route are removed. A cached answer is not proof of a sustainable design.

    Proxy and Firewall

    Restricted-egress preparation systems may need HTTP_PROXY, HTTPS_PROXY, and a carefully scoped NO_PROXY. The no-proxy list should include internal registry, Kubernetes, storage, and service domains to prevent internal traffic from being sent through an outbound inspection layer.

    Do not build a firewall rule set from one successful API call. Model downloads can involve authentication, catalog, API, and object-delivery endpoints. The connected staging test should capture actual destination names and flows for the exact NIM and model source, then compare them with the approved vendor documentation and enterprise proxy policy.

    In a dark site or fully air-gapped runtime, public egress should be denied and treated as a control, not as a transient outage.

    Private Certificate Authorities

    A TLS-inspecting proxy or internal registry signed by a private CA requires trust at multiple layers:

    • the connected staging host
    • Docker or containerd
    • Kubernetes worker nodes
    • the NIM container during connected model download
    • the NIM Operator model puller
    • Helm and GitOps tooling
    • internal object-store clients

    NVIDIA’s current guidance requires a combined CA bundle when a proxy re-signs public HTTPS traffic. Replacing the default trust store with only the corporate CA can break trust for public endpoints during the connected phase [2], [5].

    The two NVIDIA forum threads supplied with this assignment show why the visible error can be misleading. An apparent config.json download problem was associated with InvalidCertificate(UnknownIssuer) on an object-download path, and another on-premises case also surfaced certificate trust failures [16], [17]. The lesson is to trace the underlying TLS and network exception, not stop at the filename in the warning.

    Make Kubernetes Offline Before Calling NIM Offline

    A Kubernetes deployment adds dependencies that a single Docker host does not have. NVIDIA NIM Operator supports air-gapped patterns, but the operator still depends on preloaded or internally mirrored images and model assets [7].

    The disconnected bill of materials should include:

    • the Kubernetes distribution’s own disconnected-install assets
    • NIM Operator chart, images, CRDs, and admission components
    • NIM service and model-puller images
    • GPU Operator chart and all component images
    • supported driver strategy
    • local operating-system packages when the driver container needs them
    • storage provisioner and CSI images
    • ingress, service mesh, policy, and certificate-management images
    • monitoring and logging images
    • private registry pull secrets and CA distribution
    • namespace, quota, network policy, and security-context manifests

    The GPU Operator normally pulls both container images and operating-system packages. NVIDIA documents two viable offline patterns: mirror the OS packages into a local repository, or use a supported precompiled driver container that removes that download requirement [10].

    Driver and CUDA Responsibilities

    The NIM container carries the user-space libraries expected by that image. The host or Kubernetes nodes still need a compatible NVIDIA driver and container runtime integration. A full host CUDA toolkit is not automatically required merely to run a containerized NIM, but the driver, NVIDIA Container Toolkit, device exposure, and image compatibility must match the current support matrix.

    GPU Operator can manage much of this stack, but it also expands the mirrored dependency graph. Pin the operator, driver branch, node operating system, kernel, container runtime, and NIM version as one tested platform baseline.

    Image Pull and Model Pull Are Separate Secrets

    In a connected NIM Operator deployment, an NGC registry secret pulls images and an API-key secret enables model downloads [9]. In a disconnected deployment:

    • image pulls should authenticate to the internal registry
    • model reads should use the internal PVC, model store, or object service
    • public-source credentials should be absent unless an approved internal proxy deliberately uses them
    • secrets should be encrypted at rest and restricted to the namespaces that require them

    A pod that reaches ImagePullBackOff has not reached NIM yet. A pod that starts the image but fails to find a profile has passed the registry layer and failed in the model layer. Keep those diagnostic domains separate.

    Validate Licensing and Entitlement Before Import

    Offline operation does not remove licensing, support, or governing-term obligations.

    Some NIM microservices are freely available, while others require NVIDIA AI Enterprise. Production branches, long-term support branches, and enterprise support are tied to NVIDIA AI Enterprise offerings [11]. NVIDIA AI Enterprise is licensed per GPU for covered software, with specific inclusions and activation conditions for selected hardware [12].

    The import workflow should therefore record:

    • the NIM and model governing terms accepted
    • the account or organization that accepted them
    • the applicable NVIDIA AI Enterprise entitlement or free-use basis
    • the covered GPU inventory and support contract
    • expiration or renewal dates
    • any separate vGPU or NVIDIA License System requirement in the underlying platform
    • the release and security-update rights associated with the entitlement

    Do not assume every NIM performs a live license-server check at container startup. Also do not assume that successful offline startup proves the organization is licensed for production use. Entitlement validation is an administrative control that must be completed before artifact promotion.

    Prove Cold Start, Restart, and Recovery

    A disconnected-readiness test must recreate the events that invalidate warm-cache assumptions.

    Test from a Clean Node

    Use a node that does not already contain the NIM image layers. Confirm that it pulls from the internal registry, mounts the internal model storage, selects the approved profile, and becomes ready while public egress is blocked.

    This proves that the environment can recover after node replacement. Starting a pod on a node with cached image layers proves less.

    Test a True Model Cold Start

    Use a new NIM service instance that points to the approved internal model source. Capture:

    • selected profile hash
    • model path or cache path
    • image digest
    • storage mount
    • time to readiness
    • log evidence that no remote fetch was attempted
    • firewall or proxy logs showing zero public connection attempts

    Test Restarts After Outages

    At minimum, test:

    Failure event Expected result Evidence
    NIM pod deletion Replacement pod becomes ready from internal sources Pod events, image digest, readiness time
    GPU node reboot Driver and device plugin recover, NIM reloads GPU Operator state, device discovery, NIM health
    Loss of public DNS and default route Service restarts normally Network policy and firewall logs
    Internal registry outage after image is cached Existing pod remains up; reschedule behavior matches design Node cache evidence and documented limitation
    Model storage interruption Startup fails clearly, then recovers after storage restoration Storage events and NIM logs
    Certificate rotation Registry and model-store clients trust the new chain Pull test, mount test, TLS verification
    Upgrade rollback Previous image and model cache restore service Change record, rollback timing, health checks

    A fully air-gapped service is still vulnerable to simultaneous loss of an internal registry and a clean-node replacement. Document that dependency instead of claiming the service is self-contained.

    Collect a Restart Evidence Pack

    The acceptance record should contain:

    • deployment manifest and configuration
    • image digest and registry project
    • model profile hash or local model path
    • model checksum manifest
    • GPU and driver inventory
    • operator and chart versions
    • certificate-chain identifier
    • blocked-egress policy evidence
    • startup and readiness timestamps
    • health and inference test output
    • packet, proxy, or firewall evidence showing no unexpected outbound calls
    • rollback result
    • responsible approver

    A useful service check can remain simple:

    curl --fail --silent localhost:8000/v1/health/live
curl --fail --silent localhost:8000/v1/health/ready
curl --fail --silent localhost:8000/v1/models | jq .

    Success means all three calls complete locally, the expected model identifier is returned, and the service logs show that the approved local profile or model path was loaded.

    Design Upgrades as New Imports, Not In-Place Mutations

    Disconnected systems need a deliberate release train.

    The current NIM Operator upgrade model requires a new NIMCache for a newer model or profile version, followed by an update to the NIMService or NIMPipeline. Cached model profile versions are not updated in place, and multiple replicas are required to avoid availability impact during a rolling change [13].

    A practical release process is:

    The organization should define separate response targets for critical vulnerabilities, routine patches, model updates, certificate changes, and platform changes. A disconnected site that cannot import a critical fix within its risk deadline is not operationally ready, even if the first installation succeeded.

    The vulnerability process must also update the scanner’s offline database. Otherwise, a registry may report that an image is clean only because the local scanner has stale intelligence.

    Assign Operational Ownership

    Disconnected AI introduces an import function that often does not exist in a connected platform team.

    Capability Accountable owner Responsible team Required evidence
    NGC entitlement and terms AI platform owner Vendor management or AI platform Accepted terms, entitlement record
    Connected artifact retrieval Software supply-chain owner Import team Source, time, digest, checksum
    Vulnerability and malware review Security Security engineering Scan report and exception record
    Registry promotion Platform engineering Registry administrators Approved digest and audit log
    Model-store promotion AI platform owner MLOps or model operations Profile, checksum, provenance
    GPU stack compatibility Infrastructure owner Kubernetes and GPU platform team Driver, operator, GPU validation
    Disconnected deployment Service owner Platform operations Manifest, health, restart evidence
    Patch and rollback decision Change authority Joint service team Risk assessment and rollback plan

    The key control is separation of duties. The person who downloads an artifact should not be the only person who approves it for production. The application team should not own private-CA distribution. The registry team should not decide whether a model’s terms permit the intended use.

    Disconnected-Readiness Checklist

    Operating Model and Scope

    • The environment is classified as connected, restricted-egress, dark-site, or fully air-gapped.
    • Allowed transfer channels and security review steps are documented.
    • Internal dependencies that may remain reachable are explicitly listed.
    • Recovery expectations for loss of registry, model storage, DNS, and Kubernetes control plane are defined.

    NIM and Model Selection

    • The deployment uses an exact NIM version and immutable image digest.
    • The model-specific or model-free modality is documented.
    • The target GPU, profile hash, precision, and parallelism are validated against the support matrix.
    • Older 1.x commands and environment variables have been removed or intentionally translated.
    • Model governing terms and organizational use are approved.

    Container and Platform Artifacts

    • NIM images are mirrored into the internal registry.
    • NIM Operator and GPU Operator charts, CRDs, and images are mirrored when used.
    • Driver, container toolkit, device plugin, GPU Feature Discovery, DCGM, and validator dependencies are included.
    • Required CSI, ingress, certificate, policy, and observability images are included.
    • Operating-system packages or a supported precompiled driver strategy are available offline.
    • All deployment manifests point to internal image locations or pinned digests.

    Model Artifacts

    • Required model profiles were discovered on representative hardware.
    • The approved profile was downloaded and started successfully before transfer.
    • A complete cache or model store was transferred.
    • Model-free runtime manifests are preserved when remote-URI preparation was used.
    • Model and artifact checksums are recorded and revalidated after transfer.
    • The production model path or PVC is durable and backed up according to policy.

    Registry, Storage, and Trust

    • Harbor or the equivalent registry uses production TLS and a resolvable internal name.
    • Every node and container runtime trusts the registry CA.
    • Import, scan, promote, and pull identities are separated.
    • Immutable tags or digest-pinned deployments are enforced.
    • The vulnerability database has an offline update procedure.
    • Model storage has adequate capacity, throughput, metadata performance, and rollback space.
    • Registry and model-store backup plus restore have been tested.

    Network and Secrets

    • Internal DNS, NTP, Kubernetes, registry, storage, and monitoring paths are validated.
    • Proxy variables and NO_PROXY are correct in the connected preparation zone.
    • A combined public and private CA bundle is used where TLS inspection occurs.
    • Public NGC and Hugging Face credentials are absent from the air-gapped runtime.
    • Internal registry and model-store secrets are scoped and encrypted.
    • Public egress is denied during the final acceptance test.

    Licensing, Lifecycle, and Operations

    • NVIDIA AI Enterprise entitlement is validated where required.
    • Per-GPU coverage and separate vGPU licensing dependencies are recorded when applicable.
    • The import owner, security approver, registry owner, model owner, and service owner are named.
    • Patch, vulnerability, certificate, model, and operator update procedures are documented.
    • The previous approved image and model release are retained for rollback.
    • Import evidence is stored with the change record.

    Validation

    • A clean node can pull the image from the internal registry.
    • A true model cold start succeeds from internal storage.
    • Pod deletion and GPU node reboot tests pass.
    • Scale-out does not trigger a public download.
    • Firewall or packet evidence shows no unexpected outbound traffic.
    • Health, model-list, and inference tests pass after restart.
    • Rollback to the prior release has been timed and validated.

    Troubleshooting Decision Tree

    Use the first observable failure to identify the layer. Do not jump directly to GPU diagnostics when the image or model was never delivered.

    Common Failure Patterns

    config.json or generation_config.json Fetch Failures

    A filename-oriented warning does not identify the root cause. Check the nested exception and destination category.

    Likely causes include:

    • missing or invalid source credentials during connected preparation
    • governing terms not accepted
    • blocked authentication, catalog, or object-download endpoint
    • TLS inspection with an untrusted private CA
    • incomplete transferred cache
    • missing model-free runtime manifest
    • a stale NIM 1.x workflow running against a 2.0 image

    The two community threads demonstrate that UnknownIssuer can surface beneath a config.json download symptom [16], [17]. Validate the full certificate chain from inside the same container and through the same proxy path used by the download command.

    Missing Model Profile

    If NIM reports that no compatible profile is available:

    • verify the image version and model match
    • run profile discovery on representative hardware
    • confirm the profile hash exists in the transferred cache
    • confirm the GPU count and topology match the profile
    • verify the expected cache path is mounted
    • check whether automatic selection chose a different profile during preparation
    • confirm the support matrix did not remove the profile for the selected GPU

    Do not solve a missing profile by enabling public egress in production. Repair the import.

    TLS Errors

    For UnknownIssuer, hostname mismatch, or certificate-chain errors:

    • inspect the registry or proxy certificate chain
    • verify the private root and intermediates are included
    • use a combined trust bundle during connected public downloads
    • distribute the internal CA to containerd or Docker and Kubernetes nodes
    • verify the NIM container and model puller receive the bundle
    • confirm system time is correct
    • avoid disabling certificate verification as a production workaround

    Registry Failures

    For ImagePullBackOff, unauthorized pulls, or manifest errors:

    • resolve the internal registry name from the node
    • test TLS trust from the node runtime, not only an administrator workstation
    • verify the repository and digest exist
    • validate pull-secret namespace and service-account attachment
    • confirm the imported artifact includes the required architecture
    • inspect Harbor project policy, retention, and signature enforcement
    • confirm the registry is not returning a stale or deleted manifest

    Blocked Outbound Calls

    A blocked outbound call during final testing is useful evidence. Identify what requested it.

    • NIM service process: likely missing model, profile, or runtime manifest
    • model puller: cache resource still configured for NGC or Hugging Face
    • GPU Operator driver container: missing local OS packages or wrong precompiled driver path
    • Helm or GitOps controller: chart or dependency still references a public repository
    • vulnerability scanner: offline database not imported
    • application sidecar: telemetry, identity, or policy component has an external default

    The correct fix is to internalize or remove the dependency, not create an undocumented firewall exception.

    Conclusion

    NVIDIA NIM can operate disconnected, but only when the enterprise treats disconnection as a software supply-chain and operational-readiness problem.

    The OCI image is one dependency. The model profile, model weights, runtime manifest, GPU stack, Kubernetes operators, storage, certificate chain, license evidence, and update process are equally important. A mirror that handles only containers creates a deployment that appears successful during a warm start and fails during the event that matters most: a cold restart after an outage.

    The strongest design uses a controlled import zone, immutable artifacts, separate image and model stores, explicit ownership, and a repeatable validation pack. Production acceptance should require a clean-node pull, a true model cold start, pod and node recovery, blocked-egress evidence, and a tested rollback.

    That is the standard that turns “the container started once” into a defensible air-gapped private AI service.

    External References

    Related posts:

    After You Migrate: Cleanup, Governance, and Preventing Unmanaged Disks from Coming Back

    What is An AI Strategy and Why Every Business Needs One

    3 Strategic Mistakes Leaders Can Easily Avoid When Thinking About AI Integration

    Share.

    Related Posts

    Add A Comment
    Leave A Reply