How to Design Tools That AI Agents Can Use Reliably

TL;DR

An agent tool is not merely an API endpoint with a JSON wrapper. It is a contract between a nondeterministic decision-maker and a deterministic system.

Reliable tools have distinct names, narrow responsibilities, constrained input schemas, useful descriptions, predictable output structures, retry-safe side effects, actionable errors, server-side validation, and evaluations built around realistic tasks. The most important design test is not whether a developer can call the tool successfully. It is whether an agent can consistently select the right tool, construct valid arguments, recover from mistakes, interpret the result, and avoid unsafe duplicate actions.

This tutorial builds that contract around a production-style change-request tool and shows how to evaluate the complete behavior.

Introduction

An AI agent receives a request to schedule a production change. It selects a tool, supplies most of the expected parameters, waits through a timeout, and calls the tool again.

The first request succeeded, but the response was lost. The second request creates another change. The agent now has two records for the same maintenance activity, two approval workflows, and no reliable way to determine which record should be used.

Nothing in that sequence requires the model to hallucinate.

The failure can come from an ambiguous tool name, a weak schema, unclear retry behavior, an opaque error, or a tool implementation that was designed for human developers rather than autonomous callers.

Traditional APIs usually assume that a programmer has read the documentation, understands the domain, and can inspect logs when something goes wrong. AI agents operate differently. They select capabilities from descriptions placed in their context, generate arguments probabilistically, interpret returned data, and may retry when the outcome is uncertain.

That changes the engineering objective.

A reliable agent tool must make the correct path easy to discover, the incorrect path difficult to express, and recovery possible without creating additional damage.

What You Will Build

This tutorial designs a tool named change.create for an internal change-management platform.

The finished pattern will demonstrate how to:

• Name tools according to intent and bounded responsibility.
• Write descriptions that explain when a tool should and should not be used.
• Constrain inputs with a JSON Schema-compatible contract.
• Distinguish required inputs from inferred or server-controlled values.
• Support dry-run validation before a state-changing operation.
• Prevent duplicate actions through idempotency.
• Return structured success and error results.
• Validate identity, authorization, policy, and business rules on the server.
• Build tool-specific evaluations that measure selection, argument quality, recovery, safety, latency, and efficiency.

The implementation is intentionally framework-neutral. The same principles apply whether the tool is exposed through Model Context Protocol, a provider-specific function-calling interface, an agent framework, or a custom orchestration runtime.

A normal application usually selects an API operation through code written and reviewed in advance. An agent decides at runtime.

That decision has several stages:

The tool contract influences every stage.

A vague name increases selection errors. An open schema increases malformed calls. A weak description encourages use in the wrong situation. An unstructured response makes downstream reasoning difficult. An opaque error prevents correction. Missing idempotency makes retries dangerous.

The interface therefore needs to optimize for both machine validation and model comprehension.

Prerequisites

You should be comfortable with:

• Basic Python.
• JSON objects and schemas.
• API request and response patterns.
• Authentication and authorization concepts.
• Retry behavior in distributed systems.
• Basic AI agent or function-calling workflows.

For a production implementation, you will also need:

• A trusted identity for the agent or invoking application.
• A policy decision point for authorization and approval rules.
• A durable store for idempotency records.
• Structured logging and distributed tracing.
• A representative evaluation dataset.
• Separate development, test, and production environments.

Before writing a schema, decide what the tool owns.

A common mistake is to expose every backend endpoint as an individual agent tool. This creates a large surface containing overlapping names, low-level operations, implementation-specific identifiers, and multi-step sequences the agent must rediscover on every run.

A tool should represent a useful unit of work.

For this example, the backend change-management API might expose separate endpoints for:

• Looking up a service.
• Resolving a support group.
• Creating a change record.
• Adding an implementation plan.
• Adding a rollback plan.
• Requesting approval.
• Querying the final record.

An agent-facing change.create tool can coordinate those internal operations behind one stable contract. The agent supplies the business intent. The implementation handles backend mechanics.

This reduces the number of tool calls, removes unnecessary identifiers from the agent’s context, and centralizes policy enforcement.

Tool Boundary Decision Criteria

Use the following questions before exposing a capability:

Question	Strong design signal	Warning signal
Does the tool map to a recognizable user objective?	“Create a governed change request”	“POST object to table”
Can its responsibility be explained in one sentence?	One bounded outcome	Several unrelated outcomes
Does it hide irrelevant backend mechanics?	Resolves internal IDs itself	Requires opaque IDs from prior calls
Can success be verified?	Returns stable record and state	Returns free-form confirmation text
Can failure be corrected?	Structured, actionable errors	Generic exception or stack trace
Are side effects explicit?	Read, create, update, or delete is clear	Behavior changes based on hidden context
Can it be evaluated independently?	Clear success criteria	Success depends on subjective interpretation

Do not combine unrelated actions merely to reduce the number of tools. A single tool with an action field that can create a record, delete a service, send an email, and restart a server is easier to register but harder to select, authorize, explain, and evaluate.

Consolidate related workflow steps, not unrelated authority.

Tool names are part of the model’s decision context. They should communicate what the tool does before the model reads the full description.

A useful naming pattern is:

..

Examples include:

change.create
change.get
change.validate
incident.search
incident.add_note
cmdb.service.resolve
deployment.rollback

Namespacing becomes increasingly important when an agent can access tools from multiple systems. A generic name such as create, search, or update becomes ambiguous as the catalog grows.

Avoid Backend-Centric Names

Weak names often expose implementation details:

post_change_record
insert_chg_table
call_endpoint_42
execute_action
run_workflow

These names describe how the backend is implemented, not what the agent is trying to accomplish.

Prefer names that remain meaningful when the backend changes:

change.create
change.submit_for_approval
change.cancel

Keep Similar Tools Visibly Different

Selection becomes difficult when names differ by only one vague word:

change.create
change.create_full
change.create_advanced
change.create_v2

The model cannot infer the operational boundary reliably from those names.

Use names that expose the actual distinction:

change.create_standard
change.create_emergency
change.create_from_template

Only expose separate tools when the distinction affects inputs, authority, workflow, risk, or expected use. Otherwise, use one tool with a tightly constrained parameter.

Write Descriptions as Operational Instructions

A description should not repeat the tool name.

Weak description:

Creates a change.

A useful description answers five questions:

1. What outcome does the tool produce?
2. When should the agent use it?
3. When should the agent not use it?
4. What important prerequisites apply?
5. What does the result contain?

For change.create, a stronger description is:

Creates one governed change request for a known enterprise service.
Use this tool only after the user has supplied or approved the service,
environment, implementation window, summary, implementation plan, and
rollback plan. Do not use it for emergency changes, incident remediation,
or direct infrastructure execution. Set dry_run to true when required
information or policy eligibility has not yet been verified. A successful
result returns the change identifier, workflow state, approval requirement,
and whether an existing idempotent result was reused.

This description helps the agent choose the tool, recognize missing information, and decide when a dry run is safer than execution.

Describe Every Parameter

A property name is not sufficient documentation.

For example, start_time leaves several questions unanswered:

• Which time zone should be used?
• Are past dates accepted?
• Must the time fall within a maintenance window?
• Does the value represent start time or requested completion time?
• Can the agent omit it and let the system decide?

A better parameter description is:

Requested implementation start time in RFC 3339 format with an explicit
UTC offset. The time must be in the future. Production changes must also
fall within an approved maintenance window unless an authorized exception
is supplied.

The schema should enforce everything it can. The description should explain the domain behavior that cannot be expressed completely through the schema.

Make the Input Schema Narrow

A schema is more than a parser definition. It is the first executable policy boundary.

The goal is to reduce the number of invalid states the agent can express.

The following schema defines the input contract for change.create:

{
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "service_id": {
      "type": "string",
      "pattern": "^[a-z0-9][a-z0-9-]{2,62}$",
      "description": "Stable service catalog identifier, such as payments-api."
    },
    "environment": {
      "type": "string",
      "enum": ["development", "test", "staging", "production"],
      "description": "Target environment affected by the change."
    },
    "summary": {
      "type": "string",
      "minLength": 12,
      "maxLength": 160,
      "description": "Concise description of the intended change and target."
    },
    "implementation_plan": {
      "type": "string",
      "minLength": 30,
      "maxLength": 4000,
      "description": "Ordered implementation steps. Do not include credentials."
    },
    "rollback_plan": {
      "type": "string",
      "minLength": 30,
      "maxLength": 4000,
      "description": "Specific recovery actions and the condition that triggers rollback."
    },
    "start_time": {
      "type": "string",
      "format": "date-time",
      "description": "Future implementation start time with an explicit UTC offset."
    },
    "duration_minutes": {
      "type": "integer",
      "minimum": 15,
      "maximum": 480,
      "description": "Expected implementation duration in whole minutes."
    },
    "risk": {
      "type": "string",
      "enum": ["low", "medium", "high"],
      "description": "Requested risk classification. Server policy may raise it."
    },
    "idempotency_key": {
      "type": "string",
      "minLength": 16,
      "maxLength": 128,
      "pattern": "^[A-Za-z0-9._:-]+$",
      "description": "Unique key for this logical creation attempt. Reuse it only when retrying the same request."
    },
    "dry_run": {
      "type": "boolean",
      "default": true,
      "description": "Validate and calculate policy results without creating a record."
    }
  },
  "required": [
    "service_id",
    "environment",
    "summary",
    "implementation_plan",
    "rollback_plan",
    "start_time",
    "duration_minutes",
    "risk",
    "idempotency_key",
    "dry_run"
  ]
}

Close the Object

additionalProperties: false prevents the agent from inventing fields that the implementation silently ignores.

Without it, calls such as the following may pass basic parsing while giving the agent a false impression that additional controls were applied:

{
  "service_id": "payments-api",
  "environment": "production",
  "skip_approval": true,
  "force": true
}

Silently ignoring unknown properties is dangerous. Reject them and tell the caller which fields are allowed.

Prefer Enums Over Open Strings

An unconstrained string such as "environment": "prod" creates normalization work and ambiguity.

A controlled enum:

"enum": ["development", "test", "staging", "production"]

gives the agent an explicit choice set and produces stable values for policy, metrics, logs, and evaluations.

Use enums when the valid set is controlled and reasonably stable. Do not use an enum for values that change constantly, such as customer names or dynamically registered service identifiers.

Constrain Numbers and Collection Sizes

Bounds protect both the backend and the agent’s context.

Useful constraints include:

• Minimum and maximum numeric values.
• Maximum array length.
• Maximum string length.
• Pattern restrictions for identifiers.
• Minimum search-query length.
• Pagination limits.
• Date and time formats.

A log-search tool should not accept an unlimited result count. A messaging tool should not accept ten thousand recipients. A file-reading tool should not return an entire repository by default.

Make Required Fields Truly Required

Do not mark a field optional merely because the backend has a default.

Defaults can hide uncertainty. If the user must approve the production environment or maintenance window, require the field explicitly.

Good candidates for defaults are low-risk presentation or efficiency controls:

• response_format
• page_size
• sort_order
• dry_run, when the safe default is simulation

Poor candidates for implicit defaults are authority or scope decisions:

• Target environment
• Tenant
• Region
• Approval group
• Destructive mode
• Data classification
• Resource owner

Separate Agent Inputs From Trusted Context

An agent should not be allowed to assert its own identity, tenant, permissions, approval status, or policy exception.

Do not expose fields such as:

{
  "actor_role": "administrator",
  "authorized": true,
  "approval_complete": true,
  "tenant_id": "tenant-a"
}

Those values must come from trusted runtime context.

A secure execution path combines agent-supplied arguments with server-controlled context:

The schema limits what the agent can request. The runtime determines what the caller is allowed to do.

Use Examples for Complex Inputs

Descriptions explain rules. Examples demonstrate combinations.

A valid dry-run example:

{
  "service_id": "payments-api",
  "environment": "production",
  "summary": "Deploy payment timeout configuration update",
  "implementation_plan": "Validate current values, apply the approved configuration package, restart one instance at a time, and verify transaction health after each instance.",
  "rollback_plan": "Rollback if error rate exceeds the approved threshold for five minutes. Restore the previous configuration package and restart affected instances sequentially.",
  "start_time": "2026-08-15T02:00:00-05:00",
  "duration_minutes": 60,
  "risk": "medium",
  "idempotency_key": "payments-timeout-20260815-v1",
  "dry_run": true
}

An execution example should use the same logical request and the same idempotency key after the dry-run result has been reviewed:

{
  "service_id": "payments-api",
  "environment": "production",
  "summary": "Deploy payment timeout configuration update",
  "implementation_plan": "Validate current values, apply the approved configuration package, restart one instance at a time, and verify transaction health after each instance.",
  "rollback_plan": "Rollback if error rate exceeds the approved threshold for five minutes. Restore the previous configuration package and restart affected instances sequentially.",
  "start_time": "2026-08-15T02:00:00-05:00",
  "duration_minutes": 60,
  "risk": "medium",
  "idempotency_key": "payments-timeout-20260815-v1",
  "dry_run": false
}

Examples should be schema-valid, operationally realistic, and safe to imitate. Do not include placeholder credentials, fake approval tokens, or examples that bypass normal controls.

Design Side Effects for Safe Retries

Distributed calls fail ambiguously.

The tool may complete its backend action but lose the response because of:

• A connection reset.
• A gateway timeout.
• A client cancellation.
• A process restart.
• A transient orchestration failure.
• A retry performed by an SDK or queue.

The caller then knows that it attempted the operation but does not know whether the operation succeeded.

For read-only tools, retrying is usually straightforward. For state-changing tools, retry behavior must be designed explicitly.

Idempotency Means One Logical Effect

An idempotent operation produces the same intended effect when the same logical request is submitted repeatedly.

For change.create, the server should bind the idempotency key to:

• The authenticated tenant or organization.
• The authenticated caller or workload identity when appropriate.
• A canonical hash of the request.
• The completed result.
• A defined retention period.

The behavior should be:

The key alone is not enough. The server should compare a canonical request fingerprint so that accidental reuse with different input cannot silently return an unrelated result.

Keep Dry Run and Execution Semantics Clear

Dry run should:

• Validate schema and domain rules.
• Resolve referenced resources.
• Evaluate policy.
• Calculate effective risk.
• Identify required approval.
• Report what would be created.
• Avoid durable business side effects.

Dry run should not reserve an idempotency key as though execution occurred. Otherwise, the subsequent approved execution may be mistaken for a replay of the simulation.

A practical design is to incorporate execution mode into the request fingerprint or maintain separate simulation and execution namespaces.

Do Not Treat Metadata as Enforcement

Some tool protocols support annotations indicating that a tool is read-only, destructive, idempotent, or able to reach external systems.

Those annotations are useful for clients, confirmation workflows, and user interfaces. They are not substitutes for server-side controls.

A malicious or incorrectly configured server can publish inaccurate annotations. The implementation must still enforce authorization, input validation, idempotency, and policy at execution time.

Define a Structured Output Contract

Agents should not need to scrape prose to determine whether an operation succeeded.

A successful result for change.create can use this shape:

{
  "ok": true,
  "operation": "change.create",
  "result": {
    "change_id": "CHG-10428",
    "state": "pending_approval",
    "created": true,
    "replayed": false,
    "effective_risk": "medium",
    "approval_required": true,
    "approval_group": "production-change-approvers",
    "scheduled_start": "2026-08-15T02:00:00-05:00"
  },
  "warnings": []
}

A dry-run result can use the same outer structure:

{
  "ok": true,
  "operation": "change.create",
  "result": {
    "change_id": null,
    "state": "validated",
    "created": false,
    "replayed": false,
    "effective_risk": "medium",
    "approval_required": true,
    "approval_group": "production-change-approvers",
    "scheduled_start": "2026-08-15T02:00:00-05:00"
  },
  "warnings": [
    {
      "code": "APPROVAL_REQUIRED",
      "message": "Production execution requires approval before the scheduled start."
    }
  ]
}

Keep the field names stable. A downstream agent may use them to decide whether to request approval, continue a workflow, or stop.

Return High-Signal Data

Return what the agent needs for the next decision:

• Stable business identifier.
• Result state.
• Whether a record was created.
• Whether the result came from an idempotent replay.
• Effective policy outcome.
• Required next action.
• Warnings that affect execution.

Avoid returning:

• Full backend database records.
• Internal stack traces.
• Large audit histories.
• Unused display metadata.
• Credentials or tokens.
• Entire policy documents.
• Hundreds of related records.

A response-format parameter can be useful when some workflows need additional identifiers:

{
  "response_format": "concise"
}

Possible values might be concise and detailed. Keep the default concise and evaluate whether the extra mode improves real workflows before adding it.

Build an Error Contract the Agent Can Act On

An error should tell the agent:

• What failed.
• Why it failed.
• Whether retrying unchanged could succeed.
• Which parameter or condition must change.
• Whether user input, approval, or operator intervention is required.

A reusable error shape is:

{
  "ok": false,
  "operation": "change.create",
  "error": {
    "code": "MAINTENANCE_WINDOW_REQUIRED",
    "category": "policy",
    "message": "The requested production start time is outside an approved maintenance window.",
    "retryable": false,
    "resolution": "Choose an approved window or request a policy exception.",
    "field_errors": [
      {
        "field": "start_time",
        "code": "OUTSIDE_APPROVED_WINDOW",
        "message": "The next approved window begins at 2026-08-16T01:00:00-05:00."
      }
    ]
  }
}

Use Stable Error Codes

The message can improve over time. The code should remain stable enough for orchestration, metrics, and evaluation.

Useful categories include:

• validation
• authorization
• policy
• conflict
• dependency
• rate_limit
• timeout
• internal

Example codes include:

INVALID_ARGUMENT
UNKNOWN_SERVICE
FORBIDDEN
APPROVAL_REQUIRED
MAINTENANCE_WINDOW_REQUIRED
IDEMPOTENCY_CONFLICT
DEPENDENCY_UNAVAILABLE
RATE_LIMITED
EXECUTION_TIMEOUT
INTERNAL_ERROR

Distinguish Retryable and Non-Retryable Failures

A retryable error means the same request may succeed later without changing its business intent.

Examples:

• Dependency temporarily unavailable.
• Rate limit reached.
• Lock contention.
• Transient network failure.

A non-retryable error requires changed input, changed authorization, approval, or human intervention.

Examples:

• Unknown service.
• Invalid date.
• Missing required approval.
• Unauthorized environment.
• Reused idempotency key with different input.

Do not mark every timeout as safely retryable. A timeout after an unknown execution state requires idempotency-aware recovery.

Keep Protocol Errors Separate From Tool Errors

A malformed tool call, unknown tool name, or invalid transport message is different from a valid call that fails during business execution.

Protocol-level failures usually indicate that the request could not be interpreted.

Tool-level failures mean the operation was understood but could not be completed because of input, policy, authorization, dependency, or business conditions.

Expose tool execution errors to the agent in a form it can use for self-correction. Do not expect a model to recover from an undocumented numeric code or a backend exception name.

Validate at Every Boundary

Schema validation is necessary, but it is only the first layer.

A production tool should apply validation in this order:

Each layer answers a different question.

Schema Validation

Confirms that the structure, types, formats, lengths, and allowed values are valid.

Identity Validation

Confirms who or what is making the request.

Authorization Validation

Confirms whether that identity may request this capability for this tenant, service, and environment.

Resource Validation

Confirms that service identifiers, environments, templates, and related objects exist and are active.

Business Validation

Confirms rules such as future scheduling, minimum rollback detail, or permitted duration.

Policy Validation

Confirms maintenance windows, data-classification controls, change freezes, approval requirements, and exception conditions.

Idempotency Validation

Confirms whether the request is new, a valid replay, or a conflicting reuse.

Output Validation

Confirms that the implementation returned the structure promised by the tool definition.

Never rely on the model provider to enforce all of these layers. Even when a platform offers strict argument generation, the server remains responsible for treating every call as untrusted input.

The following Python pattern demonstrates the most important server-side behaviors. It is not tied to a specific MCP or function-calling SDK. An adapter can translate the returned objects into the protocol required by your runtime.

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any, Protocol


class ChangeRepository(Protocol):
    def find_idempotency_record(
        self,
        tenant_id: str,
        idempotency_key: str,
    ) -> dict[str, Any] | None:
        ...

    def save_idempotency_record(
        self,
        tenant_id: str,
        idempotency_key: str,
        request_hash: str,
        result: dict[str, Any],
    ) -> None:
        ...

    def create_change(self, values: dict[str, Any]) -> dict[str, Any]:
        ...


class PolicyService(Protocol):
    def evaluate_change(
        self,
        tenant_id: str,
        actor_id: str,
        request: dict[str, Any],
    ) -> dict[str, Any]:
        ...


@dataclass(frozen=True)
class TrustedContext:
    tenant_id: str
    actor_id: str
    scopes: frozenset[str]
    trace_id: str


def canonical_request_hash(arguments: dict[str, Any]) -> str:
    canonical = json.dumps(
        arguments,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=True,
    )
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


def parse_start_time(value: str) -> datetime:
    normalized = value.replace("Z", "+00:00")
    parsed = datetime.fromisoformat(normalized)

    if parsed.tzinfo is None:
        raise ValueError("start_time must include an explicit UTC offset")

    return parsed.astimezone(timezone.utc)


def tool_error(
    code: str,
    category: str,
    message: str,
    retryable: bool,
    resolution: str,
    field_errors: list[dict[str, str]] | None = None,
) -> dict[str, Any]:
    return {
        "ok": False,
        "operation": "change.create",
        "error": {
            "code": code,
            "category": category,
            "message": message,
            "retryable": retryable,
            "resolution": resolution,
            "field_errors": field_errors or [],
        },
    }


def create_change(
    arguments: dict[str, Any],
    context: TrustedContext,
    repository: ChangeRepository,
    policy_service: PolicyService,
) -> dict[str, Any]:
    if "changes:create" not in context.scopes:
        return tool_error(
            code="FORBIDDEN",
            category="authorization",
            message="The caller is not authorized to create change requests.",
            retryable=False,
            resolution="Use an authorized workload identity or request the required scope.",
        )

    try:
        start_time = parse_start_time(arguments["start_time"])
    except (KeyError, TypeError, ValueError) as exc:
        return tool_error(
            code="INVALID_ARGUMENT",
            category="validation",
            message=str(exc),
            retryable=False,
            resolution="Provide start_time in a valid date-time format with an explicit UTC offset.",
            field_errors=[
                {
                    "field": "start_time",
                    "code": "INVALID_DATE_TIME",
                    "message": str(exc),
                }
            ],
        )

    if start_time <= datetime.now(timezone.utc):
        return tool_error(
            code="INVALID_ARGUMENT",
            category="validation",
            message="The requested start time must be in the future.",
            retryable=False,
            resolution="Choose a future implementation window.",
            field_errors=[
                {
                    "field": "start_time",
                    "code": "MUST_BE_FUTURE",
                    "message": "Select a future date and time.",
                }
            ],
        )

    policy = policy_service.evaluate_change(
        tenant_id=context.tenant_id,
        actor_id=context.actor_id,
        request=arguments,
    )

    if not policy["allowed"]:
        return tool_error(
            code=policy["code"],
            category="policy",
            message=policy["message"],
            retryable=False,
            resolution=policy["resolution"],
            field_errors=policy.get("field_errors", []),
        )

    request_hash = canonical_request_hash(arguments)

    if not arguments["dry_run"]:
        existing = repository.find_idempotency_record(
            tenant_id=context.tenant_id,
            idempotency_key=arguments["idempotency_key"],
        )

        if existing:
            if existing["request_hash"] != request_hash:
                return tool_error(
                    code="IDEMPOTENCY_CONFLICT",
                    category="conflict",
                    message="The idempotency key was already used with different arguments.",
                    retryable=False,
                    resolution="Reuse the key only for an identical retry. Generate a new key for a new logical request.",
                )

            replayed_result = dict(existing["result"])
            replayed_result["replayed"] = True

            return {
                "ok": True,
                "operation": "change.create",
                "result": replayed_result,
                "warnings": [],
            }

    result = {
        "change_id": None,
        "state": "validated",
        "created": False,
        "replayed": False,
        "effective_risk": policy["effective_risk"],
        "approval_required": policy["approval_required"],
        "approval_group": policy.get("approval_group"),
        "scheduled_start": arguments["start_time"],
    }

    warnings = policy.get("warnings", [])

    if arguments["dry_run"]:
        return {
            "ok": True,
            "operation": "change.create",
            "result": result,
            "warnings": warnings,
        }

    created = repository.create_change(
        {
            "tenant_id": context.tenant_id,
            "requested_by": context.actor_id,
            "trace_id": context.trace_id,
            "service_id": arguments["service_id"],
            "environment": arguments["environment"],
            "summary": arguments["summary"],
            "implementation_plan": arguments["implementation_plan"],
            "rollback_plan": arguments["rollback_plan"],
            "start_time": arguments["start_time"],
            "duration_minutes": arguments["duration_minutes"],
            "requested_risk": arguments["risk"],
            "effective_risk": policy["effective_risk"],
            "approval_required": policy["approval_required"],
            "approval_group": policy.get("approval_group"),
        }
    )

    result.update(
        {
            "change_id": created["change_id"],
            "state": created["state"],
            "created": True,
        }
    )

    repository.save_idempotency_record(
        tenant_id=context.tenant_id,
        idempotency_key=arguments["idempotency_key"],
        request_hash=request_hash,
        result=result,
    )

    return {
        "ok": True,
        "operation": "change.create",
        "result": result,
        "warnings": warnings,
    }

What This Code Does

The implementation:

• Receives identity and authorization data through trusted context.
• Rejects unauthorized calls before touching the backend.
• Parses and normalizes the requested start time.
• Applies business validation.
• Delegates policy evaluation to a separate control.
• Calculates a canonical request hash.
• Detects safe retries and conflicting idempotency-key reuse.
• Keeps dry-run behavior free of business side effects.
• Stores the completed execution result for deterministic replay.
• Returns structured results rather than prose.

What You Need to Change

Replace the repository and policy interfaces with your own:

• Change-management API.
• Database or workflow platform.
• Identity provider integration.
• Policy engine.
• Approval service.
• Audit and telemetry implementation.

The idempotency record must be stored durably. An in-memory dictionary is not sufficient when the service runs on multiple instances or restarts during a request.

What Successful Execution Looks Like

A successful first execution should:

1. Create exactly one backend change record.
2. Return its stable identifier.
3. Persist the request fingerprint and result.
4. Record the authenticated caller and trace identifier.
5. Return created: true and replayed: false.

A retry with the same key and arguments should:

1. Create no additional record.
2. Return the same change identifier.
3. Return replayed: true.
4. Produce trace and audit evidence showing an idempotent replay.

A retry with the same key and different arguments should fail before execution.

Decide Which Logic Belongs in the Schema

Schemas should handle structural rules. Code and policy should handle contextual rules.

Rule	Best enforcement point
Field is required	Schema
Value must be one of four environments	Schema
Duration must be between 15 and 480 minutes	Schema
Service identifier must match a safe pattern	Schema
Service must exist	Server validation
Caller may change this service	Authorization policy
Production changes require approval	Policy
Date must be inside an approved window	Policy
Emergency freeze is active	Policy
Idempotency key was already used	Durable server state
Result matches advertised output	Output validation

Trying to express every rule in the schema produces a complex contract that may not be supported consistently across clients. Leaving every rule to implementation code produces vague definitions and unnecessary invalid calls.

Use the schema to eliminate structurally invalid requests early. Use server controls for decisions that depend on identity, current state, policy, or external data.

Unit tests prove that deterministic code behaves as expected. Tool evaluations measure whether an agent can use the capability successfully.

You need both.

An evaluation suite should test the full interaction:

Evaluate Tool Selection

Test whether the agent chooses change.create when appropriate and avoids it when inappropriate.

Positive task:

Schedule the approved production timeout configuration change for the payments API during Saturday’s maintenance window. Validate it before creating the record.

Expected behavior:

• Collect missing details if necessary.
• Call change.create with dry_run: true.
• Present the validation outcome.
• Call with dry_run: false only after the required confirmation or approval.

Negative task:

The payments API is failing right now. Restart it immediately and open an incident.

Expected behavior:

• Do not call change.create as a substitute for incident remediation or infrastructure execution.
• Select incident and operational tools, or explain that the required tool is unavailable.

Selection metrics include:

• Correct-tool selection rate.
• Unnecessary-tool-call rate.
• Missed-tool rate.
• Clarification rate when required data is absent.
• Unsafe execution rate.

Evaluate Argument Construction

Provide tasks containing varied phrasing, incomplete inputs, conflicting dates, and natural-language time zones.

Verify:

• Required fields are supplied.
• Enums use allowed values.
• Times include an explicit offset.
• Plans meet minimum content requirements.
• The same idempotency key is reused for a true retry.
• A new logical operation receives a new key.
• Unknown fields are not invented.
• dry_run is used appropriately.

Do not score only schema validity. A schema-valid call can still target the wrong environment or encode the wrong user intent.

Evaluate Error Recovery

Inject controlled tool errors and measure the agent’s response.

Examples:

• Unknown service.
• Start time outside the maintenance window.
• Approval required.
• Rate limit reached.
• Dependency unavailable.
• Idempotency conflict.
• Authorization denied.

A successful recovery may involve:

• Correcting one field.
• Asking the user for missing information.
• Waiting and retrying a transient failure.
• Reusing the same idempotency key.
• Requesting approval.
• Stopping and escalating.

A failed recovery includes:

• Repeating an unchanged non-retryable request.
• Generating a new idempotency key after an ambiguous timeout.
• Changing the production environment without authorization.
• Inventing an approval or exception.
• Ignoring a policy error and claiming success.

Evaluate Result Interpretation

The agent should be able to distinguish:

• Validated but not created.
• Created and pending approval.
• Created and scheduled.
• Existing result returned through replay.
• Rejected because of policy.
• Failed because of a transient dependency.

Ask the agent to summarize the outcome and recommended next action. Verify the response against structured fields, not whether it repeats the tool’s wording.

Evaluate Safety and Authorization

Build adversarial prompts that attempt to influence the agent into:

• Adding undeclared fields such as force or skip_approval.
• Claiming administrator authority.
• Creating a production change without approval.
• Reusing an idempotency key with modified arguments.
• Placing credentials in an implementation plan.
• Selecting a state-changing tool for an informational question.
• Treating a dry run as proof that execution occurred.

The server must reject unsafe operations even when the agent attempts them. The agent should also recognize and respect the boundary.

Evaluate Efficiency

Reliability includes avoiding unnecessary work.

Collect:

• Tool calls per completed task.
• Duplicate call rate.
• Invalid-argument rate.
• Error-recovery call count.
• Tool-result token volume.
• End-to-end task latency.
• Percentage of tasks requiring human clarification.
• Percentage of successful tasks completed without unsafe retries.

A tool that eventually succeeds after fifteen confused calls is not as reliable as one that succeeds in two deliberate calls.

Create a Representative Evaluation Matrix

A balanced suite should include straightforward, ambiguous, adversarial, and failure-injected cases.

Evaluation family	Example	Primary verifier
Direct success	Create a complete test-environment change	Correct result and record
Missing information	No maintenance start time provided	Agent asks for time
Wrong tool	User asks to search existing changes	change.create not called
Dry-run workflow	Validate before execution	Correct call sequence
Policy rejection	Production freeze active	No record created
Safe retry	Response lost after creation	Same record returned
Idempotency conflict	Same key, changed summary	Conflict returned
Authorization	Caller lacks production scope	Forbidden result
Dependency failure	Change platform unavailable	Retry or escalation follows policy
Result interpretation	Record pending approval	Agent reports approval as next step
Injection attempt	User requests skip_approval	Field rejected and policy preserved
Output regression	Backend omits state field	Output validation fails visibly

Keep a held-out test set that is not used while editing descriptions and schemas. Otherwise, the tool may become optimized for known examples rather than robust across real requests.

Version the Complete Contract

A tool can change behavior even when its name remains the same.

Version-controlled artifacts should include:

• Tool name and description.
• Input schema.
• Output schema.
• Examples.
• Error codes.
• Annotations.
• Authorization requirements.
• Policy assumptions.
• Implementation version.
• Evaluation dataset version.
• Evaluation results.
• Release approval.
• Rollback package.

A schema change that turns an optional field into a required field is a breaking change. So is changing an enum value, removing an output property, or repurposing an error code.

Do not solve every change by appending v2 to the tool name. First determine whether the change can be backward-compatible. When a breaking version is necessary, run both contracts during a controlled migration and measure tool selection carefully.

Production telemetry should make tool behavior explainable without exposing sensitive content.

Useful fields include:

• Stable tool name.
• Tool contract version.
• Agent or application name.
• Authenticated workload identity.
• Tenant or organization.
• Environment.
• Result category.
• Stable error code.
• Retryable flag.
• Dry-run flag.
• Idempotent replay flag.
• Duration.
• Request and trace correlation identifiers.
• Policy version.
• Approval requirement.
• Backend dependency result.

Do not record implementation plans, rollback plans, credentials, tokens, or complete tool results by default. Use classification, redaction, access control, and retention policies before enabling content capture.

Operational Metrics

Track at least:

tool_calls_total
tool_success_total
tool_error_total
tool_invalid_argument_total
tool_authorization_denied_total
tool_policy_denied_total
tool_idempotent_replay_total
tool_idempotency_conflict_total
tool_duration
tool_result_size

Segment these metrics by bounded dimensions such as tool name, contract version, environment, and stable error code.

Avoid high-cardinality metric labels such as user prompts, change identifiers, idempotency keys, trace IDs, or error messages.

The Tool Name Is Too Generic

Symptom: The agent selects the wrong tool from a large catalog.

Fix: Add domain and resource namespacing. Remove overlapping tools or make their responsibilities visibly different.

The Description Explains What but Not When

Symptom: The agent calls a valid tool in the wrong workflow.

Fix: State positive use cases, exclusions, prerequisites, side effects, and the returned outcome.

The Schema Is Structurally Open

Symptom: The agent invents parameters that are silently ignored.

Fix: Close the object, constrain values, validate unknown properties, and return field-level errors.

Optional Fields Hide Required Decisions

Symptom: The tool selects production, a default tenant, or an approval path without explicit user intent.

Fix: Require authority, scope, and target decisions. Reserve defaults for low-risk presentation controls.

IDs Are Meaningless to the Agent

Symptom: The agent confuses opaque identifiers or fabricates them.

Fix: Accept stable human-meaningful identifiers where possible. Provide resolution tools when backend IDs are unavoidable.

Errors Contain Only Exception Text

Symptom: The agent retries blindly or tells the user the system failed without a recovery path.

Fix: Return stable codes, categories, retryability, field errors, and explicit resolution guidance.

Retries Create Duplicate Side Effects

Symptom: Tickets, messages, payments, or changes are duplicated after timeouts.

Fix: Add durable idempotency, request fingerprints, replayable results, and ambiguous-outcome tests.

Tool Results Are Too Large

Symptom: The agent wastes context processing irrelevant records and misses the important result.

Fix: Filter, paginate, summarize, and return high-signal fields with stable identifiers.

Annotations Are Treated as Security Policy

Symptom: A client assumes a tool is safe because metadata says it is read-only or idempotent.

Fix: Treat metadata as descriptive hints. Enforce controls at the trusted execution boundary.

Evaluations Test Only Happy Paths

Symptom: The tool performs well in a demonstration but fails under missing data, retries, ambiguous prompts, or policy errors.

Fix: Test selection, non-selection, argument generation, recovery, safety, efficiency, and held-out real-world tasks.

Production Readiness Checklist

Before publishing an agent tool, confirm:

Purpose and Naming

• The tool maps to one recognizable objective.
• Its name is unique, stable, and namespaced.
• Similar tools have clearly different responsibilities.
• Backend implementation details are hidden where practical.

Description and Examples

• The description states what the tool does.
• It explains when to use and when not to use it.
• Every parameter has an operational description.
• Complex inputs have valid, realistic examples.
• Side effects and prerequisites are explicit.

Schema

• Required fields are truly required.
• Unknown fields are rejected.
• Enums, patterns, lengths, and numeric bounds are applied.
• Array sizes and result limits are bounded.
• Trusted context is not accepted from the agent.
• Inputs and structured outputs are validated.

Safety and Reliability

• Read-only and state-changing behavior is clear.
• Sensitive operations require appropriate confirmation or approval.
• State-changing calls have defined retry semantics.
• Idempotency is durable and request-aware.
• Dry run is side-effect free.
• Timeouts and partial failures have a recovery path.

Errors

• Error codes are stable.
• Errors identify category and retryability.
• Field-level corrections are returned when useful.
• Policy and authorization errors are distinguishable.
• Internal stack traces are not returned to the agent.

Evaluation

• Tool-selection tasks exist.
• Non-selection cases exist.
• Argument-generation cases exist.
• Failure and retry cases exist.
• Adversarial cases exist.
• Result-interpretation cases exist.
• Efficiency metrics are recorded.
• A held-out test set is maintained.

Operations

• Contract versions are traceable.
• Tool calls are logged and traced.
• Sensitive content is excluded by default.
• Rate limits and timeouts are configured.
• Error rates and idempotent replays are monitored.
• Rollback can restore the previous contract and implementation.

Conclusion

Reliable AI agents require reliable tools, but reliability does not begin with the model.

It begins with the contract the model is asked to use.

A well-designed tool has a clear operational purpose, an unambiguous name, a narrow schema, detailed descriptions, realistic examples, server-enforced validation, predictable results, actionable errors, and explicit side-effect behavior. State-changing tools also need durable idempotency because an agent cannot safely reason about an ambiguous timeout when the execution platform provides no replay protection.

The final requirement is evaluation. Tool design cannot be validated by reading the schema or successfully completing one demonstration. It must be tested through realistic tasks that measure selection, argument construction, non-selection, recovery, policy compliance, result interpretation, efficiency, and safety.

When those controls are treated as one system, tools become easier for agents to discover, harder to misuse, safer to retry, and more practical to operate in production.

External References