Durable Execution for Agents That Outlive a Web Request

Why ordinary request handlers fail

Agents wait on APIs, humans, queues, and rate limits for minutes or days. Keeping an HTTP process alive during that time couples business progress to one machine and one network connection. Deployments, crashes, and client disconnects then erase in-memory plans or trigger blind retries. Durable execution persists each completed transition and schedules future work independently. The user can leave, return later, and observe the same logical run.

A workflow history records commands and their results: model call requested, tool activity scheduled, timer set, approval received, and state committed. On recovery, deterministic workflow code replays that history to reconstruct state without repeating external effects. Non-deterministic operations such as current time, random choices, and network calls must be represented as recorded activities, which is the boundary a durable runtime enforces for you. Otherwise replay follows a different branch and the runtime cannot tell which world is authoritative.

Durability does not mean storing an enormous prompt after every token. Persist semantic checkpoints around meaningful operations, with references to model inputs and results in an object store when necessary. Decide retention from audit, privacy, and debugging requirements, which the NIST generative AI profile treats as a governance obligation rather than an implementation detail. Encrypt sensitive payloads, isolate tenants, and support deletion across history and blobs. Operational convenience is not a license to retain private model context forever.

Exactly once is built from at-least-once pieces

Most queues and activity systems deliver work at least once. A worker may complete an external call and crash before acknowledging it, causing the same activity to run again. Consequential tools therefore need idempotency keys stable across retries. The external system should return the original result for a repeated key. When it lacks native idempotency, store an operation ledger and reconcile ambiguous outcomes before another attempt.

Database updates can use transactions and uniqueness constraints, but distributed side effects cannot share one atomic commit. The outbox pattern records intended events in the same transaction as local state, then publishes them asynchronously. Inbound events use deduplication keys, and a standard envelope such as CloudEvents gives that key a defined place to live rather than a per-integration convention. Sagas compensate completed steps when a later step fails, though compensation is a business action, not time reversal: refunding a payment differs from never charging it and may require communication.

Activities need explicit retry policies by error class. Timeouts and temporary rate limits may retry with backoff and jitter. Authentication failure, schema rejection, or denied approval should not. Set heartbeat intervals for long work so the runtime can distinguish slow progress from a dead worker. A cancellation request should propagate into activities that can safely stop, while non-cancellable writes finish and reconcile before the workflow reports its terminal state.

Humans are asynchronous workflow participants

Approval is not a blocking modal inside one browser session. Model it as a persisted state with a request identifier, exact proposed action, expiration, and eligible approvers. Send notifications as activities, then wait for an authenticated signal. If the proposal changes, invalidate the old approval. A vague approval for “the plan” should never authorize later model-generated parameters the reviewer did not see, which is also the path by which an injected instruction reaches a real action.

Resumption context must explain what changed while the workflow waited. Prices, permissions, documents, and tool availability may be stale. Revalidate preconditions after approval and before execution, especially for high-impact actions. If material facts changed, return to approval with a clear diff. This prevents a valid human decision from being applied to a different real-world situation hours later.

Users also need pause, cancel, and amend operations. Define which checkpoints accept amendments and how they affect already completed work. Cancellation may launch compensation rather than stopping instantly. The product should show completed, pending, compensating, and failed states honestly, which is an architecture decision about a component that can be wrong rather than a copy decision. Hiding workflow complexity behind a perpetual spinner makes long-running automation feel broken and encourages users to submit duplicates.

Version workflows without breaking replay

Changing workflow code while old runs exist can break deterministic replay. A new branch, reordered command, or changed loop may no longer match recorded history. Use explicit version markers so old executions follow their original path while new runs use updated logic. Activity implementations can often change more freely because their results are recorded, but their input and output schemas still require backward-compatible evolution.

Test replay against captured histories from production, including failures, approvals, and cancellations. Those histories are a regression suite in the same sense that an evaluation harness that can block a release is one: the cases come from real failures and the gate is allowed to say no. Unit tests starting from an empty run miss compatibility problems that only appear halfway through a year-long workflow. Provide migrations only when necessary and prefer allowing old versions to drain. If a security fix requires immediate change, design a controlled reset or continue-as-new operation that carries validated state into a fresh history.

Observability should join workflow run, model request, activity attempt, external idempotency key, and user-facing task. Track replay failures, retry counts, queue delay, approval age, compensation rate, and stuck-state detectors. Operators need a safe way to inspect and signal runs without editing history directly. Where that visibility is missing, recovering a stalled system starts with diagnosis rather than repair, because nobody can say yet which runs completed. Durable systems make failure recoverable, but only if the team can understand the persisted state and act on it.

Research foundation and scope

Durable execution records workflow progress so computation can resume after process or infrastructure failure. Temporal's programming model distinguishes replayable workflow logic from external activities, while CloudEvents standardizes event envelopes [TEMPORAL] [CLOUDEVENTS]. These sources establish the mechanism, but they do not remove the need to measure the local implementation. A paper's result belongs to its checkpoint, dataset, hardware, traffic, and experimental protocol; a standard describes a contract or risk practice, not a guarantee that a particular product follows it. The useful engineering move is to convert each cited idea into a hypothesis that can be falsified with representative inputs and observable intermediate state.

Exactly-once business effects do not emerge from a queue guarantee. Retries, acknowledgements, database commits, provider calls, human delays, and workflow-code changes cross different failure domains. Write these conditions into the design review before selecting a library or provider. Specify the authoritative source of truth, the authenticated principal, the versions that influence behavior, the maximum consequence of an error, and what the system will do when required evidence is missing. This boundary statement makes later optimization honest: teams can compare speed, cost, or convenience only after candidates satisfy the same correctness, authorization, privacy, and recovery contract.

Read [TEMPORAL], [CLOUDEVENTS], [NIST-GAI] together rather than treating one source as a recipe. Temporal durable execution documentation supplies one part of the foundation; CloudEvents specification supplies a second perspective; Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile helps connect the mechanism to evaluation, governance, or operations. The citations are deliberately primary papers, standards, or official documentation. Product-specific claims still require local evidence, and any dated behavior must be rechecked against the cited version before an article or architecture decision is refreshed.

A production implementation blueprint

Persist intent before execution, use deterministic workflow state, isolate nondeterministic activities, assign stable idempotency keys, store effect receipts, and model timers, signals, approvals, compensation, cancellation, and deadlines explicitly. Begin with one thin vertical path that preserves all decision evidence. Give each run a stable identifier; capture normalized input, authenticated scope, relevant dependency versions, selected policy, timing, result status, and any external receipt. Keep large or sensitive payloads in a separately governed store and pass bounded references through the orchestration layer. This produces a debuggable system without making raw customer content the default telemetry substrate.

Make state transitions explicit and make every external boundary return a typed result. Distinguish validation failure, permission denial, missing evidence, rate limit, timeout, dependency outage, user cancellation, and internal defect. The distinction matters because each permits a different response: clarify, retry, fall back, defer, compensate, or stop. A generic exception handler that asks the model to try again turns uncertainty into repeated cost and can duplicate irreversible effects. Retries require an idempotency strategy and a stored receipt, not optimism.

Roll out the implementation as a versioned policy. Record which cohort receives it, what baseline it replaces, which metrics can stop the rollout, and how to reverse stateful artifacts such as indexes, caches, adapters, or workflow histories. Review privacy and retention alongside performance. If raw content is needed temporarily for diagnosis, state the purpose, access, sampling, encryption, and deletion window. Production readiness means the team can explain, contain, and reverse behavior under load—not merely demonstrate the happy path once.

Failure lab and evaluation plan

Terminate workers between every pair of state transitions and replay histories against unchanged code. Then test activity retries, duplicate events, late human approval, provider outage, and deployment of a compatible workflow version. Keep the arrival pattern, data distribution, authorization, and dependency versions close to production. Use a paired baseline where possible and retain per-case results so averages cannot hide a severe slice. Offline evaluation should localize the failing stage; shadow evaluation should expose integration and traffic effects; a canary should confirm user-facing outcomes with a small blast radius. Each stage needs an exit criterion rather than a vague request to 'look good.'

Exercise crash after external commit but before acknowledgement, duplicated webhook, reordered tool result, cancellation during a retry, expired credentials, poison messages, schema migration, and an unavailable compensation action. Add adversarial boundary cases and partial failures, not only malformed inputs. Terminate a worker after an external system may have committed, revoke access after a cache was populated, change one dependency version, and delay one branch until its data is stale. Confirm that cancellation releases resources and that a fallback preserves semantics. The test passes only when the external outcome, audit evidence, and user-visible state agree; a fluent final message is not proof of completion.

Evaluation data needs lineage equal to production data. Record why each case exists, who reviewed the expected behavior, which source or policy supports it, and when it must be refreshed. Separate generated cases from production incidents and prevent test cases from leaking into training. When a metric changes, inspect changed examples before accepting an aggregate. A statistically small regression can still block release if it crosses an authorization, privacy, safety, or irreversible-action boundary.

Operating metrics and decision record

Use end-to-end completion, stuck workflow age, retry and compensation counts, duplicate-effect incidents, replay failures, approval wait, cancellation latency, history growth, and recovery time after worker loss. Define every metric with a numerator, denominator, unit, time window, exclusions, and responsible owner. Segment by workload and consequence so a dominant easy class cannot hide a rare costly failure. Pair service signals with product outcomes: latency without completion encourages fast useless answers, while answer quality without queue and cost data can produce an uneconomic service. Review percentiles and distributions rather than relying on a single average.

Design for at-least-once delivery and prove business-level idempotency with receipts; reserve compensation for effects that cannot be made idempotent. Put that choice in a short architecture decision record containing context, alternatives, evidence, assumptions, selected policy, rollout, reversal trigger, and unresolved risk. Link the evaluation snapshot and source versions. This document prevents a benchmark from becoming folklore and tells the next engineer which evidence must be repeated when the model, provider, traffic, data, regulation, or product promise changes.

Monitor leading and lagging signals after release. Leading signals reveal pressure—queue age, cache allocation, disagreement, validation failure, policy denial, or retry growth—before users report harm. Lagging signals show whether the job was actually completed and trusted. Alert on a service objective tied to user consequence, and make the run identifier available to support without exposing private content. Every serious miss should become a reproducible regression case and, when relevant, a new threat or data-quality control.

Primary sources and further reading

1. Temporal durable execution documentationTemporal Technologies · official documentation
2. CloudEvents specificationCloud Native Computing Foundation · standard
3. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence ProfileNational Institute of Standards and Technology · standard

Frequently asked questions

Methodology

This article is written from production agent work rather than from a benchmark, so it makes no throughput or reliability claims that a reader cannot reproduce. The mechanisms are stated against the primary specifications cited below: the workflow and activity split as Temporal documents it, the event envelope as the CloudEvents specification defines it, and the governance and retention obligations from the NIST generative AI profile.

Every failure claim is written as a drill you can run yourself against your own system, which is the standard the evaluation harness article applies to release gates: terminate a worker between two transitions and replay the history against unchanged code, deliver the same webhook twice, let an approval expire, and revoke a credential while a retry is in flight. Where 2026 practice is genuinely unsettled, such as how much model context belongs in a persisted history, the article states the trade rather than naming a winner.

Put a durable boundary around your own agent

If your agent already survives the happy path and loses runs on a deploy, the fix is usually one boundary decision rather than a framework migration. The work is naming the transitions that have to be persisted, writing the idempotency contract for each consequential tool, and deciding what a human is actually approving.

That is a technical review, not a rebuild: Vishal reads the run history with your team and writes the decision down so your engineers can implement it. Where the loop itself is the problem rather than its durability, Agents That Finish is the longer argument, and a build engagement is the route when the agent has to be designed from the failure modes up.

Book a discovery call