How to Execute, Verify, and Roll Back Agent Actions: Enterprise Agent Control Plane Series, Part 4

TL;DR

An agent action is not complete when the MCP server returns a successful response. It is complete only when the runtime proves that the intended post-condition exists in the authoritative system and that no unapproved side effect occurred.

The execution runtime should:

• revalidate identity, policy, approval, target state, and tool version immediately before execution
• acquire a bounded execution lease
• generate and enforce a trusted idempotency key
• isolate code, processes, files, credentials, and network access
• classify timeouts and unknown outcomes before retrying
• verify post-conditions through an authoritative read path
• record evidence for the action and every recovery step
• distinguish retry, rollback, compensation, forward recovery, containment, and manual intervention
• stop at explicit points of no return unless the action has stronger approval and recovery planning

The governing principle is:

The runtime does not trust that an action succeeded. It proves what changed, decides whether the outcome is acceptable, and contains the workflow when certainty is not available.

Introduction

Part 1 of this series placed MCP beneath an enterprise agent control plane. Part 2 defined the trusted controller that owns identity, workflow state, policy, approvals, budgets, retries, and termination. Part 3 governed the MCP gateway and server layer through admission control, dynamic tool exposure, schema enforcement, credential separation, routing, and domain authorization.

Part 4 follows an authorized request beyond the MCP server boundary.

This is the point where an agent stops being an information system and starts becoming an operational actor. A ticket is created. A service is restarted. A firewall rule changes. A customer receives a message. A deployment begins. A refund is issued. Data is deleted.

The model may have selected the right tool. The controller may have authorized the request. The gateway may have routed it correctly. The MCP server may have validated the input and returned a valid result.

None of those conditions alone prove that the business outcome occurred safely.

Distributed systems fail between request and response. A timeout can hide a completed side effect. A retry can duplicate a change. A server can report success before an asynchronous job finishes. A downstream API can acknowledge a request that later fails. A cancellation can stop the client from waiting without stopping the external action. A rollback can overwrite a legitimate concurrent change. A compensation step can fail after the original action succeeded.

The execution runtime must own that uncertainty.

Where Part 4 Fits in the Series

The five-part series separates related concerns so that one protocol or service is not expected to perform every control function.

Part	Control boundary	Primary responsibility
Part 1	Architecture boundary	Separate agent reasoning, controller authority, MCP access, runtime execution, and system-of-record state
Part 2	Trusted controller	Own identity, durable state, policy, approvals, budgets, retries, cancellation, and termination
Part 3	MCP gateway and servers	Govern server admission, tool exposure, schemas, credentials, routing, and domain authorization
Part 4	Execution and recovery runtime	Enforce preconditions, idempotency, isolation, verification, evidence, compensation, and rollback boundaries
Part 5	MCP and A2A architecture	Separate agent-to-tool access from agent-to-agent delegation and collaboration

Part 4 is where authorization becomes controlled side effect execution.

Scope and Assumptions

This article assumes that:

• the trusted controller has already authorized an action
• any required human approval is valid and scoped to the exact action
• the MCP gateway has resolved an approved server and tool contract
• the MCP server still performs domain authorization and business validation
• the enterprise system remains authoritative for the resource state
• the runtime can persist execution and recovery evidence
• the workflow may span synchronous and asynchronous operations

The article does not assume that every tool supports native rollback or idempotency.

When the downstream capability lacks those properties, the runtime must compensate through stronger preconditions, narrower authority, external deduplication, reconciliation, staged execution, or manual recovery.

MCP gives clients and servers structured ways to invoke tools, return results, report errors, track progress, cancel requests, and, in the stable November 25, 2025 specification, represent some long-running work as experimental Tasks.

These protocol outcomes are useful, but they describe the interaction with the MCP server. The enterprise workflow needs a separate success definition.

Consider a tool that requests a rolling restart of a production service.

A successful MCP response might mean:

• the request was valid
• the server accepted the operation
• the downstream API returned an operation identifier
• the server created an asynchronous job

The business success condition might require:

• only the approved service was targeted
• no more than one instance was unavailable at a time
• the deployment version remained unchanged
• service health returned within the approved window
• error rate did not exceed the threshold
• no duplicate restart job was created
• the system reached a stable final state

Those conditions cannot be inferred from a generic success flag.

The controller should define the expected post-condition before execution, and the runtime should test it afterward.

The Execution Runtime at a Glance

The runtime should treat every consequential tool invocation as a stateful operation with preconditions, an execution envelope, a result classification, verification, and a recovery decision.

The runtime should not allow the model to skip any of these stages by claiming that an action is safe, reversible, already approved, or complete.

The Trusted Execution Envelope

The controller should hand the runtime a versioned execution envelope. The envelope binds the authorized intent to the exact technical action.

A practical envelope contains:

• run ID
• action ID
• authenticated requester
• agent workload identity
• tenant and organization
• target environment
• target resource
• approved operation
• normalized arguments or argument hash
• MCP server identity and version
• MCP tool identity and contract hash
• policy version
• approval identifier and expiration
• authority budget
• idempotency key
• preconditions
• expected post-conditions
• timeout and deadline
• retry policy
• cancellation behavior
• compensation reference
• evidence requirements

The runtime must reject the envelope when material values no longer match the current system.

Examples include:

• the approval expired
• the tool schema changed
• the server version changed
• the target resource version changed
• the environment entered a freeze window
• the user or workload identity was revoked
• the policy version is no longer compatible
• the authority budget was consumed by another branch
• the requested operation no longer matches the original plan

An approval is not permission to execute any later variation of the action.

Planning and approval can occur seconds or hours before execution. The target system can change during that time.

The runtime should perform a fresh authoritative read immediately before a consequential write.

Useful preconditions include:

• target exists
• target belongs to the expected tenant
• current resource version matches the reviewed version
• current state allows the operation
• no conflicting job is active
• maintenance or change window is open
• dependent systems are healthy
• capacity is available
• backup or restore point exists when required
• rollback artifact is still valid
• policy and approval are still current

Use optimistic concurrency fields such as record versions, entity tags, generation numbers, or revision IDs where the downstream system supports them.

A precondition failure should normally return the workflow to planning or approval. The runtime should not silently update the action to fit the new state.

That would turn a previously approved action into a different unapproved action.

Idempotency Is a Contract, Not a Retry Flag

Retries are unavoidable in distributed systems. Duplicate side effects are not.

An idempotent operation allows the same authorized intent to be submitted more than once without creating additional mutation. The implementation typically relies on a client-provided idempotency token or an equivalent deduplication record.

The runtime should generate the key, not the model.

A useful idempotency identity binds to:

• tenant
• workflow type
• run ID
• action ID
• target resource
• normalized operation
• normalized argument hash
• approval version

The server should persist enough information to distinguish:

• the first accepted request
• a legitimate retry of the same request
• a conflicting request that reused the key with different arguments
• an expired key outside the allowed replay window

Do not use timestamps as idempotency keys

Timestamps do not reliably identify business intent. Clock skew, parallel requests, and retries from different workers can produce collisions or false uniqueness.

Do not generate a new key for every retry

A new key converts a retry into a new side effect request.

Do not assume an idempotent label proves behavior

MCP tool annotations are advisory unless the server is trusted, and even a trusted server may implement the behavior incorrectly. Idempotency needs tests against the real downstream effect.

Test the unknown-outcome path

The most important test is not two immediate identical calls.

It is:

1. submit the action
2. interrupt the response path after the downstream system receives it
3. retry using the same idempotency key
4. prove that only one side effect exists
5. prove that the original result can be recovered

Use Execution Leases to Control Concurrency

A workflow can be delivered more than once by a queue, resumed by a second worker, or retried after a timeout while the original operation is still running.

The runtime should acquire a lease before performing consequential work.

A lease should identify:

• action ID
• owning worker
• acquisition time
• expiration time
• heartbeat or renewal time
• target resource
• operation class

Leases prevent two runtime workers from advancing the same action concurrently.

They do not automatically prevent two unrelated workflows from changing the same resource. For that case, use a resource-level lock, optimistic concurrency check, domain conflict rule, or downstream transaction mechanism.

A lease expiration should not immediately trigger another execution. The runtime must first reconcile whether the previous worker produced a side effect.

Isolate the Execution Environment

The execution runtime should receive only the access needed for one bounded action.

For code, shell, file, browser, data-processing, or deployment tools, use controls such as:

• ephemeral sandbox or job
• non-root execution
• read-only base file system
• explicit writable working directory
• restricted mounted paths
• no inherited developer credentials
• no container-engine socket
• outbound network deny by default
• approved destination allowlist
• CPU, memory, process, and storage limits
• maximum output size
• execution deadline
• secret injection immediately before use
• secret removal after completion
• artifact scanning before promotion
• cleanup after termination

Isolation should be selected by consequence, not by convenience.

A read-only service status query may need only a narrow API identity. A generated script that changes infrastructure should run inside a disposable sandbox with no ambient authority and should invoke a separately governed deployment interface.

The model should not receive a general shell simply because the desired outcome is difficult to express through a narrow tool.

Separate Prepare, Commit, and Verify When Possible

High-impact tools are easier to govern when execution is divided into phases.

Prepare

The server validates inputs, reads current state, calculates the proposed change, checks dependencies, and returns a change plan or preview.

Commit

The runtime submits the exact approved plan using the same target version, plan hash, tool version, and approval context.

Verify

The runtime queries the authoritative system and tests the defined post-conditions.

This pattern reduces the gap between what the reviewer saw and what the tool executed.

The prepare response should contain a stable plan identifier or hash. The commit operation should reject the request if the underlying resource state or plan inputs changed.

Not every platform supports a native two-phase operation. A domain-specific MCP server can still implement an application-level equivalent by separating simulation, proposal, execution, and status tools.

Timeouts Limit Waiting, Not Necessarily Work

A timeout answers one question:

How long will this caller wait?

It does not automatically answer:

Did the external operation stop?

The stable MCP lifecycle specification recommends configurable request timeouts and cancellation after a request exceeds its timeout. MCP cancellation is intentionally limited. A receiver may ignore a cancellation when the request is unknown, already complete, or cannot be canceled.

This distinction matters operationally.

When a timeout occurs, the runtime should classify the action as an unknown outcome until it can determine whether the side effect started or completed.

Do not immediately retry a timed-out write.

Use one of these reconciliation methods:

• query the operation ID
• query by idempotency key
• read the target resource version
• search an authoritative audit log
• check the downstream job queue
• query the domain-specific status tool
• ask an operator when the system provides no reliable status path

A progress notification can show that work is still active, but progress should never extend execution indefinitely. Enforce an absolute deadline even when progress continues.

MCP Tasks Help Track Work, but the Runtime Still Owns Outcome

The stable November 25, 2025 MCP specification describes Tasks as experimental durable state machines for deferred request execution. The protocol includes task status, result retrieval, progress, authorization-context requirements, expiration, and cancellation behavior.

Those capabilities help a runtime track long-running MCP operations.

They do not replace the runtime’s business execution ledger.

An MCP Task can report completed while the enterprise post-condition remains false. A Task can report canceled while the underlying work continues. The stable specification explicitly keeps a canceled task in the canceled state even if execution later completes or fails.

That means the controller and runtime still need to record:

• what business action the task represented
• which target and approval it was bound to
• whether the action produced a side effect
• whether the post-condition was verified
• whether compensation is required
• whether the task record expired before evidence was retained

Current MCP work is also moving the experimental Tasks design toward an extension model. Enterprises should isolate their business workflow state from protocol-version changes and treat MCP task identifiers as one execution reference within a broader run.

Classify the Outcome Before Choosing Recovery

The runtime should not reduce every failure to success or error.

Use a richer result model.

Result class	Meaning	Default runtime response
Not started	Validation or authorization failed before any side effect	Record denial or failure, do not compensate
Accepted	Downstream system accepted asynchronous work	Track operation, do not mark workflow complete
Completed and verified	Expected post-condition exists and no prohibited change was found	Commit success and advance workflow
Completed but unverified	Server reports success, but authoritative verification is unavailable	Pause, retry verification, or escalate
Partially completed	Some approved steps occurred, others did not	Contain and evaluate compensation or forward recovery
Failed before side effect	No mutation occurred	Retry only when the failure is transient and policy allows
Failed after side effect	Mutation occurred but later processing failed	Verify current state and choose recovery
Unknown outcome	Response path failed and side-effect state is uncertain	Reconcile before retry
Compensated	Recovery action restored an acceptable business state	Close as recovered, not as original success
Compensation failed	Recovery action did not reach a safe state	Escalate with evidence and block further automation

This classification should come from trusted runtime and domain logic, not the model’s interpretation of a message.

Verify Against the Authoritative System

Verification is a separate operation with a separate question:

Does the system of record now satisfy the approved post-condition?

The verification path should be independent from the write path where practical.

Examples include:

• use a read tool after a write tool
• query the enterprise API directly through a read-only identity
• inspect the asynchronous operation record
• compare the current resource revision with the expected revision
• validate health through the monitoring system rather than the deployment system
• confirm an external message through the communication platform’s delivery state
• verify a financial operation through the ledger rather than the submission API

The verification identity should normally be read-only and should not be able to repair the result silently.

Define expected, tolerated, and prohibited outcomes

Verification should compare more than one value.

For a service restart:

• expected: all instances healthy and operation complete
• tolerated: one instance still warming within the approved period
• prohibited: version changed, capacity reduced, or unrelated service restarted

For a firewall change:

• expected: exact rule exists in the approved policy section
• tolerated: policy compilation still pending for a bounded period
• prohibited: broader source, destination, port, or scope than approved

For a customer message:

• expected: one message created for the intended recipient using approved content
• tolerated: delivery pending
• prohibited: duplicate message, wrong recipient, or unapproved attachment

Verification can also fail

A verification API may be unavailable, stale, or eventually consistent.

The runtime needs a verification policy containing:

• maximum verification duration
• polling interval
• expected consistency delay
• independent data sources
• required confidence level
• escalation threshold
• behavior when evidence conflicts

Do not convert lack of evidence into evidence of success.

Practical Post-Condition Examples

Action	Precondition	Authoritative post-condition	Recovery option
Create service ticket	No existing ticket for the idempotency key	One ticket exists with expected fields and owner	Close duplicate or correct bounded fields
Restart service	No deployment or restart already active	Approved instances are healthy and version unchanged	Restore capacity, restart failed instance, or escalate
Change firewall rule	Current policy revision matches reviewed revision	Exact rule exists and effective policy matches approved scope	Remove exact rule or apply compensating policy revision
Send external email	Recipient, content hash, and approval are current	One message record exists for the intended recipient	Send correction or recall when supported
Issue refund	Transaction is eligible and not already refunded	Ledger contains one refund for approved amount	Corrective debit or finance-led compensation
Delete record	Retention, legal hold, and backup checks pass	Record is absent and deletion evidence exists	Restore from protected copy when policy permits
Deploy application	Artifact digest and target revision match approval	Approved digest is running and service objectives pass	Redeploy prior approved artifact or forward-fix

The recovery option is not automatically safe. It requires its own policy, authority, idempotency, verification, and evidence.

Rollback, Compensation, and Forward Recovery Are Different

Teams often use rollback as a generic word for every recovery action. That hides important differences.

Retry

Repeat the same forward operation because the failure is classified as transient and the action is idempotent.

Native rollback

Use a platform mechanism to restore a prior version or transaction state. Examples include redeploying a previous artifact, restoring a configuration revision, or rolling back a database transaction that has not committed.

Compensation

Perform a new business action that offsets an earlier successful action. Cancel a reservation, issue a correcting transaction, remove a created access grant, or send a corrective message.

Compensation does not necessarily restore the exact original state. Concurrent activity or legal and business rules may make that impossible or undesirable.

Forward recovery

Move the system from the failed intermediate state to a valid target state without undoing completed work. This may be safer than reversal when the original action is partially complete.

Containment

Stop further damage while preserving the current state for investigation. Disable the agent, revoke the credential, block the route, isolate the workload, or pause the workflow.

Manual recovery

Transfer control to a qualified operator with the evidence needed to complete or reverse the work safely.

The controller should store the chosen recovery type explicitly. A compensated workflow is not the same as a workflow that never failed.

Choose Recovery Through a Deterministic Decision Path

The model may recommend a recovery path. Trusted policy and domain logic must select and authorize it.

Define Points of No Return

Some operations cannot be safely or meaningfully undone.

Examples include:

• releasing sensitive information to an external party
• sending a legally binding communication
• executing a market transaction
• deleting data after the recovery window closes
• rotating a credential that immediately invalidates active systems
• triggering a physical process
• publishing a public statement

The workflow should identify these points before execution.

Before a point of no return, require stronger evidence that:

• all reversible validation steps have completed
• the target and payload are exact
• authorization and approval remain valid
• dependencies are healthy
• the expected outcome is measurable
• the incident and escalation owner is known
• compensation limitations are visible to the approver

Place irreversible steps as late as possible in the workflow.

A rollback plan that says “undo the action” is not a recovery plan when the action is irreversible.

Compensation Is Another Production Workflow

A compensating transaction can fail, be retried, conflict with concurrent changes, or create its own side effects.

Treat compensation as a first-class workflow with:

• its own action ID
• authorization and approval policy
• idempotency key
• preconditions
• bounded credential
• execution lease
• timeout
• result classification
• post-condition verification
• evidence
• escalation path

The runtime must retain the information needed to compensate each completed step. That may include:

• previous resource version
• created record identifier
• original value
• reservation identifier
• financial transaction identifier
• prior artifact digest
• approved removal rule
• downstream operation ID

Do not attempt to reconstruct recovery information from the model transcript after the failure.

Cancellation Must Reconcile External State

A cancellation request should stop new scheduling and signal in-flight work, but the runtime must not mark the business action canceled until it understands the external state.

A safe cancellation sequence is:

1. record who requested cancellation and why
2. stop creating new model and tool operations
3. send protocol or runtime cancellation signals
4. revoke the execution lease when appropriate
5. prevent additional side effects
6. query the downstream operation state
7. verify the authoritative resource
8. compensate or contain partial work when required
9. record the final disposition

A user closing a window or a client abandoning a request is not proof that the action stopped.

This is especially important for asynchronous jobs and task-based execution.

Build an Execution Ledger

The runtime should preserve an append-only execution ledger that can reconstruct the action and its recovery.

Useful records include:

• run and action identifiers
• parent workflow state
• requester and workload identity
• target and environment
• server, tool, and contract versions
• policy and approval versions
• normalized argument hash
• idempotency key
• lease history
• precondition snapshot
• start and end times
• MCP request and task identifiers
• downstream operation identifier
• progress events
• timeout and cancellation events
• structured result classification
• verification queries and findings
• compensation actions
• final post-condition
• operator interventions
• terminal reason code

The ledger should not contain raw secrets, tokens, or unrestricted sensitive content.

Evidence payloads may need a restricted forensic store separate from ordinary operational telemetry.

Trace the Complete Action Path

OpenTelemetry provides a vendor-neutral foundation for traces, metrics, and logs. Current GenAI conventions can describe agent, model, and tool operations, but the enterprise still needs a runtime telemetry contract for policy, execution, verification, and recovery.

A useful trace contains spans or events for:

• controller authorization
• runtime preflight
• lease acquisition
• precondition read
• MCP tool execution
• downstream operation
• progress or task polling
• verification
• retry decision
• compensation
• final workflow transition

Record low-cardinality attributes such as:

• agent name and version
• workflow type
• tool name
• server version
• environment
• result class
• recovery type
• error category
• verification outcome

Keep prompts, tool arguments, tool results, customer data, and secrets disabled by default. Capture them only through explicit policy in a protected evidence store.

The goal is not to log everything.

The goal is to preserve enough evidence to prove what was authorized, what executed, what changed, and how the final state was determined.

Example Runtime Policy

The following YAML is an application-level policy example, not an MCP protocol object. It shows how the controller and runtime can define execution, verification, and recovery requirements around a consequential tool.

agent_execution_policy:
  version: 5

  action:
    type: service.restart.execute
    risk_tier: high
    environment: production

  preflight:
    revalidate_identity: true
    revalidate_policy: true
    revalidate_approval: true
    require_current_resource_version: true
    require_no_conflicting_operation: true
    acquire_execution_lease: true
    reserve_authority_budget: true

  execution:
    sandbox_profile: api_only
    credential_source: workload_identity
    require_idempotency_key: true
    maximum_attempts: 2
    request_timeout_seconds: 30
    absolute_deadline_seconds: 300
    maximum_concurrent_actions: 1
    retry_unknown_outcome: false

  progress:
    accept_protocol_progress: true
    progress_may_extend_request_timeout: true
    progress_may_extend_absolute_deadline: false

  verification:
    required: true
    authoritative_read_tool: service.health.read
    maximum_duration_seconds: 180
    poll_interval_seconds: 10
    expected:
      operation_state: completed
      healthy_instances: all
      application_version: unchanged
    prohibited:
      duplicate_operation: true
      unrelated_service_change: true

  recovery:
    transient_failure: retry_same_idempotency_key
    unknown_outcome: reconcile
    partial_completion: contain_then_evaluate
    verification_failure: manual_or_compensate
    compensation_workflow: service.restart.recover.v2
    irreversible_step: false

  evidence:
    record_precondition_snapshot: true
    record_tool_contract_hash: true
    record_downstream_operation_id: true
    record_verification_result: true
    record_recovery_actions: true
    capture_raw_credentials: false
    capture_sensitive_content: false

  emergency:
    action_kill_switch: true
    tool_kill_switch: true
    credential_revocation: true
    operator_handoff: true

The values must be changed for the actual tool, platform, service objectives, and recovery capability.

Successful enforcement means a timeout cannot trigger a blind duplicate restart, a stale approval cannot execute, a changed application version fails verification, and operators receive enough evidence to determine the final state.

Example Deterministic Execution Function

The following Python-like example shows the runtime order. It is intentionally framework-neutral.

from dataclasses import dataclass
from enum import Enum
from typing import Any


class Outcome(str, Enum):
    VERIFIED = "verified"
    NOT_STARTED = "not_started"
    UNKNOWN = "unknown"
    PARTIAL = "partial"
    FAILED = "failed"


@dataclass(frozen=True)
class ExecutionEnvelope:
    run_id: str
    action_id: str
    tool_name: str
    arguments: dict[str, Any]
    idempotency_key: str
    approval_id: str
    expected_postcondition: dict[str, Any]


def execute_action(envelope: ExecutionEnvelope) -> Outcome:
    runtime_policy.revalidate(envelope)

    with execution_leases.acquire(envelope.action_id) as lease:
        current = authoritative_reader.read_target(envelope)
        preconditions.assert_valid(envelope, current)

        prior = idempotency_store.lookup(envelope.idempotency_key)
        if prior is not None:
            return reconcile_existing(envelope, prior)

        idempotency_store.reserve(
            key=envelope.idempotency_key,
            action_id=envelope.action_id,
            argument_hash=hash_arguments(envelope.arguments),
        )

        evidence.record_preflight(envelope, current, lease)

        try:
            result = mcp_executor.call_tool(
                name=envelope.tool_name,
                arguments=envelope.arguments,
                idempotency_key=envelope.idempotency_key,
            )
        except RequestTimeout as exc:
            evidence.record_timeout(envelope, exc)
            return reconcile_unknown_outcome(envelope)
        except TransientFailure as exc:
            if retry_policy.permits(envelope, exc):
                return retry_same_intent(envelope)
            evidence.record_failure(envelope, exc)
            return Outcome.FAILED

        classified = result_classifier.classify(result)
        evidence.record_execution_result(envelope, result, classified)

        if classified.requires_reconciliation:
            return reconcile_unknown_outcome(envelope)

        verification = verifier.check(
            envelope=envelope,
            observed=authoritative_reader.read_target(envelope),
        )
        evidence.record_verification(envelope, verification)

        if verification.passed:
            idempotency_store.complete(
                envelope.idempotency_key,
                verification,
            )
            return Outcome.VERIFIED

        recovery = recovery_policy.choose(envelope, verification)
        recovery_runtime.execute(recovery)
        return Outcome.PARTIAL

A production implementation must also handle cancellation, lease expiry, asynchronous task polling, approval revocation, policy changes, circuit breakers, secure credential injection, telemetry, and compensation verification.

The important property is that the model does not decide whether the operation started, whether a timeout is retryable, whether the post-condition passed, or whether compensation is authorized.

Common Execution and Recovery Failure Modes

Failure mode	What happens	Corrective control
Success equals HTTP or MCP success	Workflow advances before the business outcome exists	Define and verify authoritative post-conditions
Blind retry after timeout	Duplicate side effects occur	Classify as unknown outcome and reconcile first
New idempotency key on retry	Every retry becomes a new action	Reuse one trusted key for the same authorized intent
Model-generated idempotency key	Duplicate detection depends on probabilistic output	Generate keys in trusted runtime code
No precondition recheck	Approved action executes against stale state	Read authoritative state immediately before commit
Cancellation equals stopped	Client exits while external work continues	Reconcile downstream state before terminal cancellation
Compensation treated as simple undo	Recovery overwrites concurrent legitimate changes	Use domain-specific compensating actions and verification
Rollback plan stored only in prompt	Required recovery data is missing during failure	Persist recovery metadata in the execution ledger
Verification uses the write response	Server validates its own claim of success	Use an independent authoritative read path
Progress extends forever	Misbehaving work consumes resources indefinitely	Enforce an absolute deadline
Shared long-lived credential	One action has broad reusable authority	Inject short-lived action-scoped credentials
Sandbox has ambient access	Generated code reaches files, networks, or sockets outside scope	Use explicit mounts, egress deny, and ephemeral execution
Recovery has no idempotency	Compensation repeats and creates new damage	Treat recovery as a first-class idempotent workflow
Irreversible action occurs early	Later validation failure cannot be recovered	Place points of no return after reversible checks
Evidence records only final status	Operators cannot reconstruct partial work	Store preflight, execution, verification, and recovery events

A Practical Implementation Sequence

Select one bounded action

Begin with one action that has clear preconditions, a measurable post-condition, and a realistic recovery path.

Good starting actions include creating a draft, opening a ticket, adding a bounded label, or restarting one noncritical service instance.

Do not begin with arbitrary shell execution or broad production changes.

Define the action contract

Document:

• target type
• allowed arguments
• expected side effects
• preconditions
• post-conditions
• prohibited outcomes
• idempotency scope
• timeout
• retry classification
• cancellation behavior
• recovery options
• point of no return
• evidence requirements

Add an execution ledger

Persist one action record before invoking the tool. Store state transitions and evidence as the operation advances.

Implement preflight reads

Query the authoritative system, compare versions, check conflicts, and reserve the authority budget.

Add idempotency and leases

Prove that duplicate queue delivery, worker restart, and response loss do not create duplicate side effects.

Isolate the runtime

Remove ambient credentials and network access. Inject only the short-lived identity required for the action.

Add result classification

Distinguish accepted, completed, partial, failed-before-side-effect, failed-after-side-effect, and unknown outcomes.

Implement independent verification

Create a read path that does not depend on the model or the write response.

Implement one recovery workflow

Test a native rollback, compensating action, forward fix, or manual handoff using the same control discipline as the original action.

Test cancellation and timeouts

Interrupt execution at every boundary:

• before downstream receipt
• after downstream receipt but before response
• during asynchronous processing
• after side effect but before verification
• during compensation

Add observability and alerts

Alert on unknown outcomes, verification failures, compensation failures, lease expiry, duplicate suppression, and irreversible actions requiring manual review.

Expand authority gradually

Move through read, simulate, propose, approved execution, and bounded autonomy only after evidence shows that the runtime controls failure correctly.

Runtime Validation Tests

A production readiness test suite should include:

• duplicate delivery with the same idempotency key
• duplicate delivery with conflicting arguments
• timeout before downstream receipt
• timeout after downstream receipt
• lost response after successful mutation
• stale resource version
• expired approval
• revoked workload identity
• changed tool contract
• concurrent action against the same target
• task record expiration
• ignored cancellation
• progress notification flood
• partial multi-step completion
• verification API outage
• eventual consistency delay
• compensation retry
• compensation failure
• operator kill switch
• forensic reconstruction from evidence

A workflow that passes only the happy path is not production-ready.

Operating Metrics for Agent Execution

Measure the runtime as a production service.

Useful metrics include:

• authorized actions started
• precondition failures
• duplicate actions suppressed
• idempotency conflicts
• execution lease conflicts
• timeouts by action type
• unknown-outcome rate
• verification latency
• verification failure rate
• retries by classified cause
• cancellation-to-containment latency
• partial completion rate
• compensation attempts
• compensation success rate
• manual recovery rate
• irreversible action count
• sandbox policy violations
• post-condition regressions
• cost per verified action

The most important denominator is often verified business outcomes, not tool calls.

A low tool error rate can hide a high verification failure rate.

Operational Ownership

Capability	Primary owner	Evidence required
Action contract	Business service owner and application owner	Allowed outcomes, prohibited outcomes, recovery requirements
Runtime engine	AI platform or workflow platform team	Availability, state recovery, leases, limits, release history
Idempotency	Tool owner and runtime owner	Key scope, retention, conflict tests, duplicate suppression results
Sandbox and execution identity	Platform security and identity teams	Runtime profile, scopes, egress policy, credential lifetime
Preconditions and verification	Enterprise system owner	Authoritative read path, consistency behavior, success criteria
Compensation workflow	Business and application owners	Recovery steps, authority, tests, manual fallback
Telemetry and evidence	SRE, security operations, and compliance	Trace contract, retention, redaction, access controls
Incident response	Agent service owner and operations	Kill switches, escalation path, recovery runbook
Final business outcome	Named agent service owner	Verified result, residual risk, acceptance criteria

The team that owns the MCP server may not own the complete business recovery. That ownership must be assigned before production deployment.

Execution, Verification, and Recovery Checklist

Before execution

• Identity, policy, approval, and budget are revalidated.
• Server, tool, schema, and contract versions match approval.
• Current resource state is read from the authoritative system.
• Preconditions and concurrency rules pass.
• An execution lease is acquired.
• The trusted runtime generates the idempotency key.
• The short-lived credential and sandbox are prepared.
• The post-condition and recovery policy are already defined.

During execution

• The request uses the approved target and normalized arguments.
• Timeouts and an absolute deadline are enforced.
• Progress is correlated but cannot extend work indefinitely.
• Output size and resource use are bounded.
• Protocol, tool, and downstream operation identifiers are recorded.
• Cancellation stops new work and signals in-flight work.

After execution

• The result is classified beyond success or failure.
• Unknown outcomes are reconciled before retry.
• The authoritative system is queried independently.
• Expected, tolerated, and prohibited outcomes are evaluated.
• Evidence is retained without exposing raw credentials.
• The workflow advances only after verified success.

During recovery

• Retry, rollback, compensation, forward recovery, containment, and manual recovery are distinguished.
• Recovery has its own authority and idempotency key.
• Compensation progress is persisted.
• Compensation is independently verified.
• Irreversible steps trigger stronger review.
• Failed recovery blocks further autonomous action and escalates.

Conclusion

The execution runtime is the last control boundary before an authorized agent proposal becomes a real enterprise side effect.

Its job is not merely to send the MCP request. It must revalidate the action, acquire exclusive execution authority, enforce idempotency, isolate the runtime, limit resources, classify uncertain outcomes, verify the authoritative post-condition, preserve evidence, and choose a safe recovery path when the outcome is wrong or unknown.

This is why protocol success cannot be the business success condition.

A tool response may prove that a server accepted or completed a request. It does not prove that the correct resource changed, that the change stayed inside the approved boundary, that an asynchronous job finished, that no duplicate action occurred, or that the service reached a healthy state.

The practical operating model is:

Execute inside a bounded runtime. Verify against the system of record. Retry only when the intent is idempotent. Compensate through a separately governed workflow. Escalate when the final state cannot be proven.

Part 5 will complete the series by placing MCP and A2A into one enterprise architecture. It will show where agent-to-tool execution ends, where agent-to-agent delegation begins, and how identity, task state, artifacts, policy, and observability should cross that boundary without creating a second uncontrolled control plane.

External References