The practical controls that reduce context, model calls, loops, and unit cost while preserving task quality.
GenAI prototypes rarely fail a cost review because of one expensive prompt.
They fail when that prompt becomes a workflow.
A production agent may add system instructions, conversation history, memory, retrieved documents, tool definitions, tool results, reasoning, retries, validation calls, and a final response. What appears to be one user request can become several model invocations before the task is complete.
The visible prompt is only the beginning.
The more useful unit of analysis is the entire agent run.
That distinction matters because token optimization is not simply an exercise in shortening prompt text. Tool schemas, files, message formatting, conversation state, retrieval results, and internal reasoning can all contribute to consumption.
The objective is not:
Use the fewest possible tokens.
The objective is:
Use the fewest tokens required to complete the task successfully, safely, and at the expected quality level.
Start With Cost per Successful Task
Many teams measure average tokens per API call.
That metric is useful, but incomplete.
A smaller model call may look inexpensive while causing more retries. An aggressive output limit may reduce one response but trigger a second call. A weak retrieval configuration may save context tokens while increasing hallucinations, escalations, and human rework.
This changes the optimization conversation.
You are no longer asking which individual call is cheapest. You are asking which architecture produces an acceptable outcome with the least total consumption.
That means tracking:
- Input tokens
- Cached-input tokens
- Output tokens
- Reasoning tokens
- Model and service tier
- Agent turns
- Tool calls and tool retries
- Retrieval volume
- Validation calls
- Latency
- Final task status
- Human overrides or rejections
Without run-level telemetry, token reduction becomes guesswork.
Build a Token Control Plane
Token management should not depend on every developer remembering a list of prompt-engineering tips.
It should be built into the execution path.
The important point in the following design is that cost controls exist before, during, and after model execution.
What should stand out is that the model is only one part of the control path.
Admission policies decide how much the task is allowed to consume. The context builder controls what enters the model. The router determines which model receives the work. The agent loop limits repeated execution. The output contract controls generation. Tracing shows whether the run delivered value.
This is more than a cost dashboard.
It is an enforcement model.
Separate Token Reduction From Rate Reduction
Not every optimization reduces actual token volume.
That distinction is important when reporting savings.
This prevents a common reporting mistake.
Prompt caching and batch processing may lower spend substantially, but they do not necessarily make the workload itself leaner. Context reduction, selective retrieval, and stopping rules address the underlying consumption.
A mature program uses both.
Instrument the Whole Run
Begin with traces, not prompt rewrites.
For each production task, capture a common usage envelope:
{
"task_type": "incident_triage",
"model_route": [
"economical",
"reasoning_escalation"
],
"input_tokens": 14200,
"cached_input_tokens": 9000,
"generated_tokens": 1850,
"agent_turns": 4,
"tool_calls": 5,
"tool_retries": 1,
"retrieved_chunks": 6,
"task_status": "accepted"
}
The exact fields will vary by provider, but the operating model should remain consistent.
Measure the median and high-percentile cost for each task type. An average can hide a small number of runaway agent executions that consume a disproportionate share of the budget.
Look for patterns such as:
- Large prompts repeatedly sent without cache reuse
- Tool responses copied into every later turn
- Agents revisiting the same plan
- Duplicate retrieval results
- Expensive models handling simple extraction tasks
- Validation calls that rarely change the answer
- Failed runs that consume nearly as much as successful ones
Also protect the telemetry itself.
Agent traces may contain prompts, tool arguments, tool results, customer information, system details, and sensitive business data. Redaction, access control, encryption, and retention policy must be part of the observability design.
Put a Budget at the Run Boundary
Every agent run should begin with a budget.
That budget can include:
- Maximum cumulative input tokens
- Maximum generated tokens
- Maximum reasoning tokens
- Maximum agent turns
- Maximum tool calls
- Maximum retries per tool
- Maximum retrieved context
- Maximum elapsed time
- Escalation conditions
- Human-review conditions
Use different budget profiles for different workloads.
A customer-support summarization task should not receive the same reasoning allowance as a production change review. A high-consequence security recommendation may justify more context, additional validation, and a stronger model.
Budgets should be based on task class, consequence, and service-level expectations.
They should not be based on developer preference.
Route Before You Reason
Do not send every request directly to the most capable model.
Use deterministic software where the outcome is governed by clear rules.
Use smaller models for:
- Classification
- Extraction
- Formatting
- Routing
- Tagging
- Simple summarization
- Structured transformation
Reserve more expensive reasoning models for:
- Ambiguous requests
- Novel planning
- Multi-source synthesis
- Difficult technical analysis
- Failed validation
- High-consequence decisions
The caveat is important.
Routing requires evaluations.
The cheapest model is not cheaper when its failures create retries, escalations, incorrect actions, or human rework.
Treat Context as a Working Set
Long context windows make it technically possible to send large histories.
They do not make it operationally sensible.
A production agent usually needs:
- Stable instructions
- Current task state
- A compact summary of relevant history
- A small number of recent turns
- Retrieved evidence for the current question
- Only the tool results required for the next decision
It rarely needs:
- The entire raw conversation
- Every prior tool response
- Duplicated documents
- Stale plans
- Repeated system instructions
- Full application logs
- Unfiltered search results
- Data from completed workflow stages
Use structured state outside the prompt.
Store decisions, identifiers, approvals, workflow status, and durable memory in an application database or state service. Inject only what the current step requires.
A useful context model looks like this:
Memory should be selective.
It should not become an ever-growing transcript.
Make Retrieval Earn Its Tokens
Retrieval-augmented generation can improve grounding, but poorly tuned retrieval can flood the model with loosely related text.
Control retrieval through:
- Metadata filters
- Tenant and authorization filters
- Relevance thresholds
- A small initial
top_k - Reranking
- Duplicate removal
- Question-aware chunk selection
- Bounded chunk sizes
- Bounded tool-result sizes
Retrieve the evidence needed to answer the question.
Do not retrieve the maximum amount the context window can hold.
Every retrieved chunk should justify its place in the prompt.
If a chunk does not improve grounding, answer quality, or decision confidence, it is consuming budget without creating value.
Design for Caching, but Report It Correctly
Provider prompt caching is valuable when requests share long, stable prefixes.
Typical cache candidates include:
- System instructions
- Policy definitions
- Tool schemas
- Long reference documents
- Repeated few-shot examples
- Stable product configuration
- Stable tenant configuration
Place stable content first and dynamic user content later.
This increases the probability that repeated prompt prefixes can be reused.
But prompt caching usually does not shrink the logical context received by the model.
It reduces recomputation, latency, or the rate charged for cached input.
Report cached-input savings separately from true token-volume reduction.
Application-level caching is different because it can prevent the model call entirely.
Useful candidates include:
- Repeated public questions
- Stable document summaries
- Retrieval results
- Tool responses from slow systems
- Deterministic classifications
- Previously validated structured outputs
Application caches need:
- Permission-aware keys
- Tenant boundaries
- Data-version identifiers
- Expiration policies
- Explicit invalidation
- Auditability
- A bypass path for sensitive or changing data
A cached response that crosses tenant boundaries or survives an important data change is not an optimization.
It is an incident.
Bound Reasoning and Output
Do not use maximum reasoning effort for every task.
Match reasoning depth to task complexity, and define an output contract before generation begins.
Examples include:
- Return five fields, not an unrestricted essay
- Produce a recommendation, evidence, and risk
- Return only changed configuration objects
- Explain only when confidence falls below a threshold
- Generate a summary first and expand on request
- Return machine-readable JSON for downstream automation
- Stop after the acceptance criteria have been satisfied
Output contracts reduce unnecessary generation while making responses easier to validate.
Consider the difference:
Weak instruction: Analyze this incident and tell me everything you think. Bounded instruction: Return: - probable cause - supporting evidence - confidence score - next diagnostic action - escalation condition Limit the response to the evidence required for the next operator decision.
The second version is not merely shorter.
It is operationally clearer.
Token limits still require testing. A limit that is too high wastes consumption. A limit that is too low can produce incomplete results and trigger another expensive attempt.
The correct value is the smallest budget that reliably completes the evaluated task.
Put Stopping Rules Inside the Agent Loop
Agentic systems multiply token spend because they make repeated decisions.
An unbounded loop can repeatedly call the same tool, reconsider the same plan, retrieve similar evidence, or ask another agent to review work that already passed validation.
Useful stopping controls include:
- Maximum model turns
- Maximum tool calls
- Maximum cumulative token spend
- One retry for known transient failures
- Duplicate tool-call detection
- Repeated-argument detection
- No-state-change detection
- Cycle detection
- Final-answer validation
- Escalation to a person
- Deterministic fallback
Multi-agent fan-out deserves even stronger scrutiny.
Planner, researcher, critic, reviewer, and formatter agents may create an impressive demonstration, but every additional role adds instructions, context, handoffs, and another opportunity for retries.
Use another agent only when evaluation shows that specialization improves the final outcome enough to justify the additional cost and operational complexity.
Move Non-Interactive Work to Batch
Some work does not need an immediate response.
Common examples include:
- Offline classification
- Document enrichment
- Evaluation runs
- Repository embeddings
- Scheduled report generation
- Back-office summarization
- Large-scale data extraction
- Dataset labeling
- Compliance analysis that can complete later
Move these jobs to batch or lower-cost asynchronous execution paths.
Batch processing reduces unit price rather than token volume, and the tradeoff is delayed completion.
Interactive and offline workloads should not share the same execution path by default.
Do not use an interactive reasoning model simply because it is the easiest API to call.
Turn the Controls Into Policy
The following vendor-neutral example shows how these recommendations can become an enforceable agent profile rather than a collection of informal guidelines.
The values are illustrative. They should be calibrated through evaluations using representative production tasks.
version: 1
profiles:
interactive_standard:
budgets:
max_total_input_tokens: 18000
max_total_generated_tokens: 2500
max_agent_turns: 5
max_tool_calls: 6
max_retries_per_tool: 1
context:
history_strategy: summary_plus_recent
recent_messages: 6
retrieval_top_k: 4
max_tool_result_tokens: 800
routing:
default_tier: economical
escalate_on:
- high_consequence
- low_confidence
- failed_validation
stop_conditions:
duplicate_tool_call: true
repeated_arguments: true
no_state_change_turns: 2
cache:
prompt_prefix: enabled
application_result_cache: permission_aware
telemetry:
optimization_unit: cost_per_successful_task
record:
- input_tokens
- cached_input_tokens
- generated_tokens
- reasoning_tokens
- agent_turns
- tool_calls
- retries
- task_status
The policy should be versioned alongside prompts, tool schemas, retrieval settings, and model-routing rules.
Successful execution means every run can show:
- Which profile was applied
- Which budget was consumed
- Why escalation occurred
- Which stop rule ended the run
- Whether the outcome passed evaluation
What can go wrong is equally important.
Tight retrieval limits can remove required evidence. Low output caps can cause incomplete responses. Hard turn limits can interrupt legitimate workflows. Aggressive routing can send complex work to an incapable model.
Every cost policy needs a quality gate and a controlled escalation path.
Use a Practical Optimization Sequence
Do not begin by rewriting every prompt.
Start with the controls that expose and prevent the largest sources of waste.
Establish run-level traces
Identify the task types, prompts, tools, and agent paths responsible for the most total spend.
Add hard safety limits
Cap turns, tool calls, retries, generated output, and total run consumption before optimizing individual prompts.
Reduce repeated context
Compact history, trim tool results, remove duplicated instructions, and tune retrieval.
Introduce model routing
Keep the economical path as the default and make escalation evidence-based.
Implement caching
Use provider prompt caching for stable prefixes and application caching where validated results can safely be reused.
Move offline work to batch
Separate interactive requirements from work that can tolerate delayed completion.
Validate against accepted outcomes
Compare success rate, latency, cost, human overrides, and failure modes before and after each change.
This sequence matters.
It prevents a team from celebrating lower token counts while quietly degrading the system.
Final Thought
High GenAI spend is often treated as a pricing problem.
In agentic systems, it is usually an architecture and control problem.
The agent carries too much context. It retrieves too much evidence. It uses an expensive model too early. It calls too many tools. It retries without changing state. It generates more explanation than the workflow needs. It delegates work to additional agents without proving that the handoff creates value.
The answer is not to make every prompt shorter.
The answer is to build a governed execution path where context, model choice, reasoning depth, tool access, retries, caching, and stopping conditions are explicit.
The cheapest agent is not the one that uses the fewest tokens in one call.
It is the one that completes the right task, at the required quality, with the least total consumption and the smallest operational blast radius.
External References
OpenAI API Documentation: Token Counting
https://developers.openai.com/api/docs/guides/token-counting
OpenAI API Documentation: Prompt Caching
https://developers.openai.com/api/docs/guides/prompt-caching
OpenAI API Documentation: Compaction
https://developers.openai.com/api/docs/guides/compaction
OpenAI API Documentation: Reasoning Models
https://developers.openai.com/api/docs/guides/reasoning
OpenAI API Documentation: Batch Processing
https://developers.openai.com/api/docs/guides/batch
OpenAI Agents SDK: Running Agents
https://openai.github.io/openai-agents-python/ref/run/
Microsoft Foundry: Agent Tracing and Observability
https://learn.microsoft.com/en-us/azure/foundry/observability/concepts/trace-agent-concept
Amazon Bedrock: Intelligent Prompt Routing
https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html
