Why Your GPU Is Idle: A Layer by Layer Troubleshooting Guide for Enterprise Inference

Introduction

An enterprise inference service can look busy while its GPU remains nearly idle. The application may be accepting requests, retrieving documents, validating permissions, tokenizing prompts, waiting on storage, retrying dependencies, or building responses. None of those activities prove that enough executable work is reaching the accelerator.

This is why GPU troubleshooting often goes wrong. Teams begin with the last dashboard in the chain, see low utilization, and immediately change batch sizes, add model instances, or blame the hardware. They optimize the GPU before proving that the GPU is the bottleneck.

A more reliable method starts with the expected demand and follows the complete request path. At every layer, the operator asks three questions: did work arrive, how long did it wait, and how long did the layer spend servicing it? Only after those answers are known should the team decide whether low GPU utilization is normal, whether work is being blocked upstream, or whether the GPU is executing inefficiently.

TL;DR

Low GPU utilization is not automatically a fault. It may be the correct result for a latency-first service, a lightly loaded endpoint, a small model, reserved failover capacity, or an application whose time is dominated by retrieval, tokenization, storage, networking, or external tool calls.

Use a deterministic sequence:

• Confirm that meaningful inference demand is expected.
• Verify that requests reach the inference service and that GPU-backed pods are scheduled correctly.
• Separate cold-start and model-loading time from steady-state execution.
• Measure CPU, host memory, storage, network, and tokenization before tuning the accelerator.
• Correlate request queueing, concurrency, batching, model instances, GPU memory, SM activity, Tensor Core activity, and DRAM activity.
• Inspect PCIe, NVLink, clocks, power, temperature, XID events, and reliability counters only after the software path is understood.
• Optimize for the service objective, not for a dashboard target of 100 percent GPU utilization.
• Protect p95 and p99 latency with headroom, bounded queues, admission control, and representative load testing.

The First Question Is Whether the GPU Should Be Busy

A GPU should be busy only when useful work is available, admitted, scheduled, prepared, and ready to execute. An idle GPU is therefore not enough evidence to declare an incident.

Normal idle state

Low utilization can be healthy when:

• The service is designed for low latency and traffic arrives intermittently.
• The request rate is below the tested saturation point.
• The endpoint maintains spare capacity for p95 and p99 protection.
• A small model finishes each request quickly and spends most of its time waiting for the next one.
• The workload is CPU-heavy, retrieval-heavy, or network-heavy by design.
• Multiple replicas are deployed for availability, but traffic is not high enough to keep all replicas active.
• A multi-GPU model uses only the GPUs required by its selected runtime profile.
• A canary, disaster-recovery, or maintenance reserve is intentionally idle.
• Autoscaling has not yet removed excess capacity because the stabilization window has not elapsed.

In these cases, raising utilization may make the system worse. Larger batches and deeper queues can improve throughput while increasing time to first token, end-to-end latency, and tail latency.

Unexpected low utilization

Low utilization becomes suspicious when all of the following are true:

• Representative demand is reaching the service.
• The request queue or client backlog is growing.
• The service is missing throughput, latency, or cost objectives.
• The model is loaded and healthy.
• The GPU is assigned to the workload.
• The measurement window is long enough to represent steady-state behavior.
• The observed GPU activity remains materially below the known-good baseline.

The difference matters. “The GPU is at 20 percent” is not a diagnosis. “The GPU is at 20 percent while 150 requests are pending, p99 latency is rising, and the known-good baseline is 70 percent under the same workload” is actionable evidence.

Decide What the Service Is Optimizing

GPU utilization has no useful meaning without a service objective.

Latency-first inference

A latency-first endpoint prioritizes fast response and predictable tail behavior. Typical examples include interactive assistants, fraud decisions, real-time recommendations, and online vision services.

The design usually favors:

• Small or moderate batches.
• Strict queue-delay limits.
• Spare GPU and CPU headroom.
• Admission control before saturation.
• More replicas instead of deeper queues.
• Isolation from batch jobs.
• Conservative concurrency limits.
• Strong p95 and p99 monitoring.

A latency-first service may be correctly engineered at 30 to 60 percent average GPU activity if that headroom prevents tail latency from collapsing during bursts.

Throughput-first inference

A throughput-first service prioritizes completed work per unit of time. Typical examples include offline embeddings, bulk classification, document processing, and overnight scoring.

The design usually favors:

• Larger batches.
• Dynamic or in-flight batching.
• Higher concurrency.
• Longer queue windows when the completion objective allows them.
• Multiple execution streams or model instances when measured to help.
• Efficient data staging.
• Sustained GPU activity.
• Requests per second, samples per second, or tokens per second as primary outcomes.

Here, low utilization under a persistent backlog is more likely to indicate an upstream feed problem, inefficient execution, or incorrect model placement.

Cost-first inference

A cost-first service optimizes useful work per GPU-hour while preserving minimum latency and reliability objectives. It may accept moderate queueing, schedule work into windows, use smaller accelerators, or consolidate workloads through carefully governed sharing.

The important metric is not utilization alone. It is the cost per successful request, token, document, image, or business outcome.

Follow One Request Through the Complete Path

The most useful mental model is a dependency chain. The GPU can execute only after every earlier layer has completed enough work to feed it.

At each transition, record four values:

1. Arrival rate: how much work entered the layer.
2. Queue time: how long work waited before service.
3. Service time: how long the layer spent processing it.
4. Completion rate: how much useful work left the layer.

This turns troubleshooting into evidence collection. If the client sends 500 requests per second but the inference server receives 80, the GPU is not the first problem. If the server receives 500 and 450 remain pending while SM activity is low, the investigation moves to batching, runtime execution, host-device transfer, synchronization, or scheduling. If the queue grows while SM activity, clocks, and power are already high, the GPU may simply be at capacity.

Establish Prerequisites and Safety Boundaries

Do not tune a production inference service without a reproducible baseline and a rollback path.

Capture:

• GPU model, memory size, firmware, driver, and operating mode.
• Bare-metal, passthrough, vGPU, MIG, or time-sliced allocation.
• Kubernetes, GPU Operator, device plugin, container toolkit, and runtime versions.
• Model name, model version, precision, quantization, runtime profile, and engine build.
• Triton, NIM, TensorRT, TensorRT-LLM, vLLM, or other serving-runtime version.
• Number of replicas, model instances, GPUs per instance, and selected parallelism.
• CPU, host memory, storage class, network path, and NUMA placement.
• Request rate, concurrency, batch size, prompt length, output length, and input distribution.
• p50, p95, p99, throughput, error rate, queue time, time to first token, and inter-token latency where applicable.
• The exact configuration change under test.
• The previous known-good configuration.

Avoid invasive profiling on a live shared GPU. Kernel profilers, stress diagnostics, clock changes, GPU resets, and extended DCGM diagnostics can alter performance or disrupt workloads. Use a canary node, maintenance window, or drained GPU for deep analysis.

Build a Baseline Before Diagnosing

A single screenshot is not a baseline. GPU activity is bursty, sampling intervals differ, and short kernels can disappear between dashboard samples.

Build two baselines.

Single-request latency floor

Send one request at a time after the model is warm. This shows the minimum practical latency for the current model, input shape, output length, runtime, and hardware.

Record:

• Client-observed end-to-end latency.
• Server request time.
• Queue time.
• Compute-input time.
• Compute time.
• Compute-output time.
• Time to first token and inter-token latency for streaming language models.
• CPU and GPU activity during the request.

If single-request latency is already too high, batching will not solve the root cause. Investigate model choice, precision, kernel execution, data movement, and runtime configuration first.

Saturation curve

Increase concurrency or request rate in controlled steps. At each step, record throughput, latency percentiles, queue time, errors, and accelerator metrics.

The expected curve has three regions:

The operating point should normally remain before or near the knee, depending on the latency objective. Past the knee, throughput flattens while queueing and tail latency rise rapidly.

Use representative demand

A useful baseline includes:

• Warm-start and cold-start tests.
• Realistic input sizes and distributions.
• Realistic prompt and output-token lengths.
• Steady-state traffic.
• Bursts.
• A long enough soak test to expose memory growth, thermal behavior, and retries.
• Failure scenarios such as a replica restart or dependency slowdown.
• The same network path and authentication flow used in production.

Synthetic batch-one requests with fixed tiny inputs may produce a clean chart and a misleading conclusion.

Metric Thresholds Are Investigation Triggers, Not Universal Limits

The following values are practical starting points for investigation. They are not vendor support limits, universal service-level objectives, or proof of a fault. GPU architecture, model type, runtime, sampling window, precision, and workload shape can change the correct interpretation.

Signal	Starting investigation trigger	Likely interpretation	Critical caveat
Offered load	Below roughly 10 to 20 percent of tested capacity, with no queue	Low GPU activity is probably normal	Capacity must come from a representative benchmark, not a theoretical GPU rating
Queue time	Sustained above 10 to 20 percent of the p95 latency budget	Queueing is consuming too much of the service objective	Some throughput-first jobs can tolerate far more queueing
Pending requests	Nonzero and growing for several normal service times	Active instances cannot drain the admitted work	Confirm the metric is at the model that actually executes, not only an outer gateway
p99-to-p50 ratio	Greater than roughly 2 to 3 times, or materially worse than baseline	Tail amplification, retries, contention, or queue bursts	Long-output generative workloads naturally have broad distributions
CPU utilization	Sustained above roughly 80 to 90 percent on preprocessing cores	CPU feed bottleneck, thread contention, or tokenization pressure	Aggregate node CPU can hide one saturated core or one throttled container
CPU throttling	More than roughly 5 to 10 percent of periods under load	CPU limits may be starving preprocessing or runtime threads	Metric definitions vary by monitoring stack
Host memory headroom	Less than roughly 10 to 15 percent, rising faults, reclaim, or swap	Host memory pressure may delay input preparation and model access	File cache is useful memory; do not treat all used RAM as pressure
Storage wait	Sustained I/O wait above roughly 10 percent, or latency above known-good baseline	Model, tokenizer, dataset, or cache reads are limiting progress	I/O wait alone can be ambiguous in containers and virtual machines
GPU SM activity	Below roughly 30 percent while requests are persistently pending	GPU is underfed, launch-bound, synchronized, or poorly batched	Small latency-first workloads may be healthy below this value
GPU SM activity	Roughly 30 to 70 percent	Ambiguous, inspect Tensor activity, DRAM activity, queueing, clocks, and kernel gaps	Averages can hide alternating bursts and idle gaps
GPU SM activity	Above roughly 70 percent with a growing queue and stable clocks	Compute capacity is likely the current limit	High activity does not prove efficient kernels or good business throughput
GPU memory used	Above roughly 90 percent with variable KV cache or batch growth	OOM risk, fragmentation, or limited batching headroom	High memory use can be healthy and does not mean high compute activity
GPU memory used	Below roughly 30 percent	Small model, one instance, or conservative cache sizing	Low memory allocation does not prove wasted compute capacity
DRAM activity	Sustained above roughly 60 to 70 percent while SM or Tensor activity is lower	Memory-bandwidth-bound workload is plausible	Practical maximums vary and may remain below 100 percent
PCIe or NVLink errors	Counters increase during a controlled test	Interconnect, topology, or hardware issue requires review	Link speed may downshift at idle and recover under load
Thermal state	Thermal clock event or clock decline correlated with temperature near the device limit	Cooling or airflow may be reducing frequency	Device limits differ; temperature alone is not enough
Power state	Active power-limit event with clocks below known-good load baseline	Power cap may constrain performance	Reaching a power limit can be normal for a fully utilized GPU
Reliability	Any new XID, uncorrectable ECC event, row-remap concern, or repeated NVLink fault	Driver, platform, or hardware investigation is required	Response depends on the event class; not every XID requires replacement

The strongest diagnostic signal is not a single threshold. It is a correlated pattern across demand, queueing, service time, GPU execution, and hardware state.

Verify That Work Is Reaching the Inference Boundary

The application can appear busy without submitting enough inference work.

Common upstream causes include:

• Authentication and authorization delays.
• Retrieval-augmented generation queries.
• Vector-database latency.
• Document parsing.
• Prompt construction.
• External tool calls.
• Policy checks.
• Rate limiting.
• Client-side serialization.
• Retries with backoff.
• Sequential client behavior.
• Load-balancer connection limits.
• Downstream response validation.

Compare the client request count with the inference server request count over the same time window. If the counts differ, find the layer where work disappears or waits.

For a generative AI application, also separate model time from orchestration time. A multi-step agent may spend seconds calling search, databases, APIs, and approval services while the GPU executes for only a fraction of the transaction.

Evidence to collect

• Requests accepted at the application boundary.
• Requests forwarded to the model endpoint.
• Requests accepted by the serving runtime.
• Request failures and retries at each layer.
• Gateway, service-mesh, and load-balancer queueing.
• Client concurrency and connection reuse.
• Per-stage traces for retrieval, prompt assembly, model call, and postprocessing.

If the inference server has no pending work and the accelerator is idle, do not tune the batcher yet. Increase representative offered load or accept that the current demand does not require more GPU time.

Confirm GPU Allocation and Kubernetes Scheduling

A pod can be Pending because it cannot acquire a GPU. A pod can also be Running while the model does not execute on the expected device.

Start with cluster state:

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp
kubectl describe pod 
kubectl describe node 
kubectl get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.product

Check for:

• Insufficient nvidia.com/gpu resources.
• A GPU request that does not match the available MIG profile.
• Node selectors that exclude the actual GPU nodes.
• Affinity or anti-affinity rules that cannot be satisfied.
• Taints without matching tolerations.
• Namespace quotas.
• Pod priority and preemption.
• Topology constraints.
• Image-pull, init-container, secret, or PVC failures.
• Device-plugin registration failures.
• A node labeled as GPU-capable while the driver stack is unhealthy.
• A runtime class or container-runtime configuration mismatch.

A clear resource request removes ambiguity:

resources:
  requests:
    cpu: "4"
    memory: "16Gi"
    nvidia.com/gpu: 1
  limits:
    cpu: "8"
    memory: "24Gi"
    nvidia.com/gpu: 1

GPU extended-resource requests and limits must resolve to the same whole-number allocation. CPU and memory requests should also reflect the preprocessing and serving work. A pod with one GPU and too little CPU can be perfectly scheduled and still starve the accelerator.

Confirm device visibility inside the container

Validate:

• The expected GPU UUID or device is visible.
• The serving runtime selected the GPU backend.
• The model profile maps to the expected number of GPUs.
• CUDA initialization succeeded.
• The process is attached to the expected GPU.
• MIG or vGPU allocation matches the intended service class.
• The container did not fall back to CPU execution.

“All GPUs are visible” is not a placement policy. NIM, Triton backends, TensorRT-LLM, and other runtimes select model profiles and parallelism based on configuration and compatibility. A container may see four GPUs while the selected profile intentionally uses one or two.

Sharing can obscure attribution

Time slicing can increase the number of workloads sharing a GPU, but it can complicate attribution. A pod-level dashboard may not cleanly explain which time-sliced workload consumed each interval. When troubleshooting, isolate the service on a canary GPU or correlate process-level and application-level signals.

Confirm the Model Is Ready, Warm, and Resident

A Running container is not the same as a ready inference service.

Cold-start work can include:

• Pulling a large container image.
• Downloading model artifacts.
• Reading model weights from shared storage.
• Verifying licenses or entitlements.
• Selecting a runtime profile.
• Building or loading an optimized engine.
• Allocating GPU memory.
• Creating KV cache.
• Compiling kernels.
• Capturing CUDA graphs.
• Loading tokenizer files.
• Running model warmup.
• Starting every configured model instance.

During this period, the application may retry readiness checks or queue requests while GPU activity remains intermittent.

Separate three states

Container ready: the process is alive and its health endpoint responds.

Model ready: the requested model version is loaded and can accept inference.

Steady-state ready: caches are populated, warmup is complete, and latency has stabilized.

Measure all three. Do not compare cold-start measurements with a warm production baseline.

Storage and cache implications

Persistent local or high-performance cache can reduce restart time and avoid repeated model downloads. However, a cache does not eliminate the need to measure:

• Cache hit rate.
• Read latency.
• Model artifact size.
• Tokenizer and configuration reads.
• Concurrent replica starts.
• Storage saturation during rolling updates.
• Available capacity for multiple model versions.

Model loading can be a storage problem, a network problem, a CPU decompression problem, or an engine-build problem. The GPU may simply be waiting for the platform to finish preparing the model.

Inspect CPU Preprocessing and Tokenization

Inference pipelines often place significant work on the CPU before a GPU kernel can launch.

Examples include:

• JSON and protocol parsing.
• Image decoding and resizing.
• Audio decoding and feature extraction.
• Text normalization.
• Tokenization.
• Padding and shape construction.
• Retrieval result assembly.
• Safety preprocessing.
• Memory copies.
• Python backend logic.
• Dynamic-shape bookkeeping.

Common symptoms

• High CPU utilization with low GPU activity.
• One saturated CPU core while node-level CPU looks moderate.
• CPU throttling caused by restrictive limits.
• Long compute-input time.
• Sawtooth GPU activity with large gaps between bursts.
• Better performance when concurrency increases only until CPU saturates.
• GPU utilization improves when preprocessing is bypassed in a synthetic test.
• Tokenization latency rises sharply with long prompts.

Corrective actions

• Increase CPU requests and limits based on measured demand.
• Remove unnecessary CPU limits where policy permits and throttling is proven.
• Use optimized native tokenizers and vectorized libraries.
• Increase preprocessing workers carefully.
• Avoid unbounded worker counts that create context switching.
• Pin CPU and memory close to the GPU NUMA node where topology matters.
• Cache immutable tokenizer and preprocessing assets.
• Move suitable preprocessing to GPU only after measuring the transfer and implementation cost.
• Separate preprocessing and inference services when they need independent scaling.
• Profile Python callbacks and serialization before blaming CUDA.

Tokenization can dominate short-generation workloads. For long prompts, time to first token includes more than GPU generation. It can include network transport, queueing, tokenization, prefill, and framework overhead.

Inspect Host Memory Pressure

GPU memory and host memory are different constraints.

A model may fit comfortably in HBM while the host experiences:

• Memory reclaim.
• Page faults.
• Swap activity.
• Container OOM termination.
• Eviction pressure.
• Pinned-memory pool exhaustion.
• Large temporary buffers.
• Duplicate model copies.
• Cache churn during rolling updates.

Host memory pressure delays input preparation and host-to-device transfer. It can also create irregular latency that appears as a GPU scheduling problem.

Check memory working set, resident set, page faults, reclaim, swap, OOM events, pinned-memory usage, and NUMA locality. Preserve enough headroom for burst batches, tokenizer buffers, runtime metadata, and rolling replacement pods.

Inspect Storage Throughput and Latency

Storage can starve inference in two different phases.

Startup path

Model weights, engines, tokenizer files, and runtime profiles must be read before the service becomes ready. Slow shared storage can extend recovery, scaling, and deployment times even if steady-state inference later runs from GPU memory.

Request path

Some services read request data, images, audio, documents, embeddings, or feature files during each request. A shared file system may provide high sequential throughput but poor metadata or small-file behavior.

Measure:

• Read latency at p50, p95, and p99.
• Aggregate throughput.
• Small-file and metadata performance.
• Concurrent model-load behavior.
• Cache-hit behavior.
• Storage queue depth.
• I/O wait.
• Network path to remote storage.
• Performance during rolling restarts.

A useful isolation test places the model and representative input data on local NVMe. If GPU activity and latency improve materially, the bottleneck is upstream of the accelerator.

Inspect Network and Input-Pipeline Limitations

Network problems do not always produce packet loss. They can appear as insufficient request arrival rate, serialization delay, connection churn, or cross-zone latency.

Check:

• Client-to-gateway latency.
• Gateway-to-runtime latency.
• HTTP versus gRPC overhead for the payload shape.
• TLS negotiation and connection reuse.
• Maximum connections and streams.
• Service-mesh sidecar resource limits.
• Load-balancer algorithms.
• Uneven distribution across replicas.
• Cross-node and cross-zone placement.
• MTU and retransmissions.
• Bandwidth for images, audio, tensors, or long prompts.
• Backpressure and client timeouts.
• Synchronous clients that send the next request only after the previous one completes.

For batch-oriented NIM and Triton workloads, asynchronous independent requests often allow the server to form efficient batches. A client that sends one request, waits, and then sends the next can make a powerful GPU look idle.

Inspect Queue Delay, Request Concurrency, and Batching

Queueing is the hinge between available demand and executable GPU work.

Read the queue before tuning

A growing pending-request count means the server has accepted work that has not begun backend execution. Correlate it with queue time, compute time, and GPU activity.

• No queue and low GPU activity: demand or client concurrency is probably too low, or the latency-first design is intentionally under-saturated.
• Growing queue and low GPU activity: work is blocked by preprocessing, batching policy, synchronization, runtime overhead, host-device transfer, or inefficient kernels.
• Growing queue and high GPU activity: capacity is likely exhausted.
• High queue time with low compute time: scheduling and batch formation dominate.
• Low queue time with high compute time: model execution dominates.

Client batch, dynamic batch, and in-flight batch are different

Client batch: one request contains multiple samples.

Dynamic batching: the server combines separate compatible requests into one execution batch.

In-flight batching: a generative runtime continuously admits and schedules sequence work as requests progress through prefill and decode.

Do not treat these as interchangeable. A server may prefer individual asynchronous requests so that it can batch them according to current demand and sequence state.

Start with a controlled configuration

The following Triton configuration is illustrative, not a production recommendation:

max_batch_size: 8

dynamic_batching {
  max_queue_delay_microseconds: 2000
}

instance_group [
  {
    kind: KIND_GPU
    count: 1
  }
]

The example permits batches up to eight and allows a short queue window to assemble them. The correct queue delay may be zero for a strict latency service, or much larger for offline work. Sweep it against the latency budget rather than copying the example.

A practical starting method is:

1. Test one model instance with no dynamic batching.
2. Enable dynamic batching with zero or minimal queue delay.
3. Sweep maximum queue delay up to a small fraction of the p95 budget.
4. Sweep concurrency through and beyond the expected operating range.
5. Compare achieved batch size, throughput, p95, p99, and GPU activity.
6. Add another model instance only if the first instance leaves execution gaps that the second can fill.
7. Re-test because additional instances consume memory and can reduce the batching opportunity available to each instance.

Preferred batch sizes should be used only when the engine has a measured advantage at specific batch sizes. They can otherwise hold requests unnecessarily.

Find the concurrency knee

For NIM and other generative runtimes, sweep concurrency from one to above the configured batch or sequence capacity. The useful operating point is usually the highest throughput that still satisfies the latency objective.

When concurrency exceeds the runtime’s useful capacity, throughput often plateaus while time to first token and end-to-end latency rise. More queued work does not create more GPU capacity.

Determine Whether GPU Execution Is Efficient

Once demand, scheduling, model readiness, and the input pipeline are proven, inspect GPU execution.

GPU utilization is not GPU efficiency

A generic GPU utilization percentage often indicates whether kernels were active during a sampling interval. It does not reveal:

• Whether the kernels used Tensor Cores.
• Whether the SMs had enough active work.
• Whether the workload was memory-bandwidth bound.
• Whether launches were separated by CPU gaps.
• Whether synchronization serialized execution.
• Whether data transfers overlapped compute.
• Whether the kernels performed useful work efficiently.
• Whether the service met its latency or throughput objective.

Use a richer metric set:

• GPU utilization.
• SM active.
• SM occupancy.
• Tensor pipe activity.
• DRAM active.
• GPU memory used and free.
• Power.
• Graphics and memory clocks.
• Temperature.
• PCIe transmit and receive throughput.
• NVLink bandwidth.
• XID and reliability events.

High SM occupancy is not automatically better. Occupancy is one enabling condition, not a business outcome. A kernel can have high occupancy and poor algorithmic efficiency, or lower occupancy and excellent latency.

Look for launch-bound behavior

A launch-bound workload has a CPU or framework that cannot enqueue GPU work quickly enough.

Symptoms include:

• Short kernels separated by idle gaps.
• CPU thread saturation.
• Enqueue time close to or greater than GPU compute time.
• Frequent framework callbacks.
• Many small operations.
• Dynamic-shape overhead.
• Improved performance with CUDA graphs.
• Improved performance when batch size increases modestly.

Potential actions include kernel fusion, CUDA graphs, larger batches, a more optimized backend, reduced Python overhead, or a different engine profile.

Look for synchronization-bound behavior

Synchronization can serialize a pipeline that appears concurrent.

Sources include:

• Blocking host waits.
• Device-wide synchronization.
• Synchronous memory copies.
• Framework barriers.
• Sequential ensemble stages.
• Pipeline-parallel bubbles.
• A client waiting for every response before issuing the next request.

Use a timeline profiler on a canary environment to identify CPU gaps, blocking calls, transfers, and kernel overlap. Do not begin with kernel-level tuning if the timeline shows the GPU waiting on the application.

Look for memory-bound behavior

If DRAM activity is high while SM or Tensor activity is moderate, the workload may be limited by memory bandwidth, cache behavior, or data movement.

Possible actions include:

• Quantization.
• Tensor layout changes.
• Operator fusion.
• Reducing unnecessary copies.
• Reusing buffers.
• Increasing arithmetic intensity.
• Choosing a runtime engine optimized for the model.
• Adjusting batch size.
• Improving KV-cache policy.

The best action depends on the model and runtime. Do not infer “memory-bound” from high GPU memory allocation. Allocation and bandwidth activity are different measurements.

Validate precision and Tensor Core use

A GPU can execute a model correctly while missing expected acceleration because of:

• Unsupported or mixed precision.
• Tensor dimensions that do not align well.
• Fallback operators.
• Dynamic shapes that prevent the best engine.
• An engine built for the wrong hardware.
• Quantization that introduces dequantization overhead.
• Framework execution instead of an optimized engine.
• Small batches or shapes that do not create enough work.

Compare the deployed service with an isolated runtime benchmark on the same model, shape, precision, and GPU. If the isolated engine performs well but the service does not, the bottleneck is in orchestration, scheduling, transfer, or integration. If both are slow, inspect the engine and kernels.

Understand Multiple Model Instances

Multiple model instances can improve utilization by allowing one instance to execute while another waits on host-device transfer or other work. They can also make performance worse.

Benefits may include:

• Better overlap.
• Higher concurrency.
• Improved throughput for models with execution gaps.
• Independent queues and execution contexts.

Costs may include:

• Additional GPU memory.
• Smaller dynamic batches per instance.
• More contention.
• Higher p95 and p99 latency.
• More CPU and host-memory pressure.
• Harder troubleshooting.

Test instance counts rather than assuming “more instances equals more utilization.” A common useful sequence is one, two, and four instances, stopping when throughput flattens or tail latency worsens.

For many deployments, dynamic batching with one instance can already fill the GPU. Adding instances then divides the request stream and consumes memory without increasing useful work.

Validate Tensor Parallelism, Pipeline Parallelism, and Replicas

Multi-GPU inference introduces a different class of idle behavior.

Tensor parallelism

Tensor parallelism splits model tensors and operations across GPUs. It may be required to fit the model or reduce per-request latency, but every step can require communication.

Low per-GPU utilization may result from:

• Collective communication.
• Uneven shard work.
• Small batch or sequence dimensions.
• PCIe instead of higher-bandwidth GPU interconnect.
• Cross-NUMA placement.
• Synchronization between ranks.
• A model that is too small for the selected parallel degree.

More GPUs do not guarantee proportionally more throughput.

Pipeline parallelism

Pipeline parallelism places different model stages on different GPUs. Idle “bubbles” occur when a stage waits for another stage.

Bubbles can be reduced with sufficient microbatches or concurrency, but larger microbatch counts can affect memory and latency.

Data parallel replicas

Data parallelism places independent model copies on GPUs and distributes requests among them. If one GPU is busy and others are idle, inspect the load balancer, connection affinity, request routing, health state, and replica readiness.

Multiple models on one GPU

Hosting several models on one GPU can improve aggregate use when their demand patterns complement each other. It can also create memory pressure, unpredictable latency, and eviction or loading delays.

Treat model colocation as a service policy. Define memory reservations, priority, maximum concurrency, admission control, and latency objectives for every model.

Inspect PCIe, NVLink, and NUMA Placement

Host-device transfer and GPU-to-GPU communication can limit utilization.

PCIe checks

Inspect:

• Current and maximum link generation.
• Current and maximum link width.
• Host-to-device and device-to-host throughput.
• PCIe replay or error counters.
• NUMA affinity.
• Pinned-memory behavior.
• Whether transfers overlap compute.
• Whether a virtualized path changes expected bandwidth.

A PCIe link may downshift while idle. Judge link state under a representative load, not only at rest.

Large transfers, unpinned host memory, remote NUMA memory, and synchronous copies can create low SM activity even when the application is moving substantial data.

NVLink checks

For tensor or pipeline parallel workloads, inspect:

• Link topology.
• Link health.
• Transmit and receive bandwidth.
• NVLink error counters.
• Rank placement.
• Collective communication time.
• Whether traffic falls back to PCIe.

A communication-heavy model can make all GPUs look partially busy while delivering poor scaling. Compare one-GPU and multi-GPU throughput, latency, and communication overhead.

Useful topology and operating-state commands include:

nvidia-smi topo -m
nvidia-smi -q
nvidia-smi dmon -s pcu -f gpu-dmon.csv -c 300

Interpret current PCIe state under load. Record the device UUID or PCI bus ID so measurements are attached to the correct GPU.

Inspect Thermal, Power, and Clock Behavior

When the software path is healthy but performance is below baseline, inspect the GPU operating state.

Thermal throttling

Look for:

• Temperature rising until clocks fall.
• Thermal clock-event reasons.
• Fan or cooling anomalies.
• Rack inlet temperature changes.
• Blocked airflow.
• Multiple adjacent accelerators heating together.
• Performance that degrades during a long soak test.

Do not use a universal temperature threshold. GPUs have device-specific limits, and the active thermal event is stronger evidence than temperature alone.

Power limits

A GPU can reach its configured power limit during healthy full-load operation. It becomes suspicious when:

• Clocks are lower than the known-good baseline.
• Throughput is lower under the same workload.
• A power-limit event is active.
• The configured limit differs from the approved baseline.
• Multiple GPUs share a constrained platform power envelope.

Changing power limits or application clocks is a controlled infrastructure change. Do not use it as an ad hoc production fix.

Clock behavior

Collect graphics, SM, and memory clocks during the benchmark. Low clocks at idle are normal. The question is whether clocks rise and remain appropriate under steady load.

Correlate clocks with:

• GPU activity.
• Temperature.
• Power.
• Throttle reasons.
• Application throughput.
• Kernel timeline.

Inspect Reliability and Hardware Faults

Hardware and driver faults are less common than upstream bottlenecks, but they require decisive handling.

Watch for:

• New XID events.
• Correctable and uncorrectable ECC errors.
• Pending page retirements.
• Row-remapping concerns.
• PCIe replay growth.
• NVLink errors.
• GPU “fallen off the bus” events.
• Repeated driver resets.
• A device that disappears from the node.
• DCGM diagnostic failures.

Use a staged response:

1. Preserve logs, timestamps, GPU UUID, node identity, workload, and recent changes.
2. Stop scheduling new work to the affected node.
3. Drain or evacuate workloads when the event class warrants it.
4. Run readiness checks.
5. Run deeper diagnostics only after the GPU is removed from production service.
6. Compare firmware, driver, BIOS, PCIe, and platform support baselines.
7. Open a support case with the collected evidence.
8. Return the node only after a controlled validation test passes.

Do not run extended stress diagnostics against a GPU serving production requests.

Use the Troubleshooting Decision Tree

The decision tree below preserves the order of evidence. It prevents the team from jumping directly to kernel tuning.

The loop ends only when the change improves the service outcome. A higher GPU percentage with worse p99 latency is a regression.

Protect p95 and p99 While Increasing Throughput

The easiest way to raise GPU utilization is to add queueing. It is also one of the easiest ways to damage user experience.

Use these controls:

• Set explicit request timeouts.
• Bound the queue.
• Reject or shed excess load before the service enters collapse.
• Reserve headroom for bursts.
• Separate interactive and batch workloads.
• Cap prompt and output lengths where appropriate.
• Use priority and admission policy.
• Scale on queue delay or pending work, not only GPU utilization.
• Warm new replicas before routing traffic.
• Prevent rolling updates from removing too much capacity.
• Test failure scenarios at peak load.
• Monitor p95 and p99 by model, input class, tenant, and endpoint.

For streaming language models, track at least:

• Time to first token.
• Inter-token latency.
• End-to-end latency.
• Input tokens.
• Output tokens.
• Request concurrency.
• Tokens per second.
• Queue time.
• Cancellation and timeout rate.

A service can maintain acceptable average latency while its longest prompts, largest images, or one tenant experience severe tail latency. Percentiles need workload context.

Run a Reproducible Load-Test Sequence

Use the following sequence for every material tuning change.

Warmup

Load the model, populate caches, complete warmup, and wait for metrics to stabilize. Record cold-start time separately.

Latency floor

Run concurrency one with representative inputs. Establish the minimum practical latency.

Concurrency sweep

Increase concurrency in controlled steps. Continue beyond the expected operating point until throughput plateaus or the latency limit is exceeded.

Request-rate sweep

Use a controlled arrival rate, including a realistic distribution when the production service is bursty. This separates offered load from closed-loop client behavior.

Batch and instance sweep

Test:

• Batch size.
• Dynamic queue delay.
• Model instance count.
• Replica count.
• Precision and engine profile.
• Prompt and output lengths.
• Tensor or pipeline parallel degree where applicable.

Change one variable at a time.

Burst and soak

Apply short bursts above normal demand, then run a long steady test. Watch for:

• Queue recovery.
• Tail-latency growth.
• Memory growth.
• Thermal drift.
• Clock changes.
• Retry storms.
• Replica imbalance.
• Storage degradation.
• Reliability counters.

Failure test

Restart a replica, remove a GPU node, delay a dependency, or reduce available capacity in a controlled environment. Verify that admission control, autoscaling, and routing protect the service objective.

Record the complete test context

A benchmark result without configuration is not reusable evidence. Store the hardware, software, model, inputs, runtime flags, concurrency, rate, batch policy, duration, and telemetry interval with every result.

Use a Layer-by-Layer Evidence Matrix

Layer	Primary evidence	Healthy interpretation	Typical failure pattern
Demand	Request rate, concurrency, backlog	Offered load matches the scenario	Application busy but few model calls
Gateway	Accepted and forwarded requests, latency, errors	Counts reconcile with the runtime	Auth, rate limit, connection, or retry delay
Kubernetes	Pod state, events, allocated extended resources	Correct pod on correct GPU node	Pending pod, mismatched MIG resource, affinity conflict
Model readiness	Load time, readiness, warmup, cache	Model resident before test	Download, storage, engine build, warmup delay
CPU	Per-core use, throttling, preprocessing time	CPU feeds runtime without long gaps	Tokenizer or decode saturation
Host memory	Working set, faults, reclaim, swap, OOM	Stable headroom	Reclaim, pinned-memory pressure, rolling-update spike
Storage	Read latency, throughput, IOPS, metadata	Model and inputs arrive on time	Slow cache, shared-storage contention
Network	Arrival rate, latency, retransmits, connections	Requests distributed evenly	Sequential client, mesh limits, cross-zone delay
Queue	Pending count, queue duration	Bounded and aligned to SLO	Growing backlog or excessive batch wait
Batching	Achieved batch, executions, inferences	Larger useful batches without tail regression	Batch one under load or oversized batches
Runtime	Compute-input, infer, output, enqueue timeline	Continuous useful execution	Launch gaps, fallback ops, synchronization
GPU compute	SM, occupancy, Tensor activity	Activity matches model and baseline	Underfed, launch-bound, inefficient kernels
GPU memory	Used/free, allocation failures, cache	Enough headroom for workload variation	OOM, fragmentation, restricted batching
Memory bandwidth	DRAM activity	Appropriate for model behavior	Bandwidth-bound execution
Interconnect	PCIe/NVLink traffic and errors	Expected topology and stable links	Remote NUMA, low link state under load, errors
Hardware state	Clocks, power, temperature, throttle reasons	Stable under representative load	Thermal or power throttling
Reliability	XID, ECC, row remap, link faults	No new relevant events	Node drain and hardware escalation required
Service outcome	Throughput, p95, p99, errors, cost	Objective met with safe headroom	Higher utilization but worse service

Validate Every Change and Preserve Rollback

After each change, compare against the same baseline and workload.

A change passes only when it produces the intended outcome without unacceptable side effects.

Validate:

• Throughput.
• p50, p95, and p99 latency.
• Time to first token and inter-token latency.
• Queue time and pending requests.
• Error, cancellation, and timeout rates.
• Achieved batch size.
• CPU and host-memory pressure.
• GPU SM, Tensor, DRAM, memory, power, clocks, and temperature.
• Replica distribution.
• Cold-start and recovery behavior.
• Cost per unit of useful work.

Rollback when:

• p95 or p99 exceeds the approved budget.
• Throughput does not improve materially.
• OOM, eviction, or instability appears.
• Queue recovery becomes slower.
• Error or cancellation rate increases.
• Thermal or power behavior moves outside the platform baseline.
• One tenant or workload class is harmed.
• The change makes restart or failover behavior worse.

Keep the previous model configuration, deployment manifest, engine, runtime profile, and autoscaling policy ready for immediate restoration. Use canary GPU nodes and controlled traffic percentages for changes that affect batching, instance count, precision, parallelism, or engine builds.

Final Operator Checklist

Before declaring the GPU underutilized, confirm:

• The expected demand is real and reaches the model endpoint.
• The service objective is explicitly latency-first, throughput-first, or cost-first.
• Client and server request counts reconcile.
• The pod is scheduled and owns the intended GPU resource.
• The runtime uses the expected GPU and model profile.
• The model is warm and resident.
• CPU preprocessing and tokenization are not saturated or throttled.
• Host memory has stable headroom.
• Storage and network meet the tested baseline.
• Queue time and pending work are understood.
• Concurrency is high enough to expose capacity.
• Dynamic batching and model instances were measured, not guessed.
• GPU memory allocation is not being confused with SM activity.
• SM, Tensor, DRAM, and kernel timeline signals support the diagnosis.
• Tensor, pipeline, or data parallel placement matches the intended design.
• PCIe, NVLink, NUMA, clocks, power, and thermal state are normal under load.
• No new XID, ECC, or link fault indicates a hardware problem.
• The proposed fix improves the service objective and preserves p95 and p99.
• A rollback path is documented.

Conclusion

An expensive GPU can remain idle while an AI application looks busy because the accelerator is only the final execution layer in a much longer system. Requests can wait in the client, gateway, scheduler, model loader, CPU pipeline, storage path, network path, tokenizer, batcher, runtime, or interconnect before useful work reaches an SM.

The correct response is not to chase a universal utilization percentage. First prove that demand is expected. Then prove that it reaches the serving runtime. Follow queue time and service time through every layer. Only after upstream constraints are eliminated should the team tune batching, concurrency, model instances, kernels, parallelism, and hardware behavior.

A healthy latency-first service may keep deliberate headroom. A throughput-first service with a growing queue and low SM activity has a real feed or execution problem. A service with a growing queue, high SM activity, stable clocks, and stable hardware may simply need more capacity or a more efficient model.

The practical goal is not a busier GPU. It is a service that delivers the required latency, throughput, reliability, and cost with evidence, safe headroom, and a repeatable operating model.

External References