Incident Response for Systems That Can Be Wrong Fluently

Define incidents in product and policy language

HTTP availability alone cannot detect an AI incident when the system responds quickly with unsafe, unsupported, or operationally damaging behavior. The production boundary is larger than the feature that makes the technique attractive: callers arrive with different authorization, freshness, latency, and quality requirements, while the underlying model and data keep changing. A design is therefore incomplete until it states which inputs are part of identity, what may be reused, what must be recomputed, and which failure should be visible to the caller. Treating those choices as an explicit contract prevents an optimization from quietly becoming a correctness or governance bug.

Incident classes combine technical symptoms with outcome thresholds for groundedness, policy breach, tool misuse, cost, latency, and user harm. This mechanism couples layers that teams often measure separately. A change in routing alters traffic shape; a change in traffic shape alters cache behavior and queue time; a change in retained context alters cost, privacy exposure, and model behavior. Preserve the intermediate decisions needed for replay: normalized inputs, policy version, selected route, relevant hashes, model revision, timing, and outcome. The final text alone cannot explain why the system behaved as it did, and it cannot support a safe rollback when the frontier model changes.

Attach owners and severity rules to each class, instrument leading signals, and allow frontline teams to escalate outcome failures without proving a model defect. Use time from harmful behavior to classified incident as the primary operating signal, but review it beside tail latency, unit cost, policy violations, and an outcome metric owned by the product. Roll out behind a versioned decision with a comparison cohort and an explicit reversal trigger. Representative evaluation must include adversarial boundaries, stale state, partial dependency failure, and the busiest traffic window. That evidence converts a plausible architecture into an operable system and gives the team a durable baseline for the next model, workload, or regulation.

Containment uses capability switches

A single global off switch forces teams to choose between continued harm and total product outage. The production boundary is larger than the feature that makes the technique attractive: callers arrive with different authorization, freshness, latency, and quality requirements, while the underlying model and data keep changing. A design is therefore incomplete until it states which inputs are part of identity, what may be reused, what must be recomputed, and which failure should be visible to the caller. Treating those choices as an explicit contract prevents an optimization from quietly becoming a correctness or governance bug.

Layered controls can disable one model, route, retrieval source, memory feature, tool, tenant, prompt version, or autonomous action while preserving safe read-only behavior. This mechanism couples layers that teams often measure separately. A change in routing alters traffic shape; a change in traffic shape alters cache behavior and queue time; a change in retained context alters cost, privacy exposure, and model behavior. Preserve the intermediate decisions needed for replay: normalized inputs, policy version, selected route, relevant hashes, model revision, timing, and outcome. The final text alone cannot explain why the system behaved as it did, and it cannot support a safe rollback when the frontier model changes.

Prebuild and exercise kill switches, require no deployment for emergency policy changes, and define degraded modes that communicate limitations honestly. Use time to contain one capability without full service shutdown as the primary operating signal, but review it beside tail latency, unit cost, policy violations, and an outcome metric owned by the product. Roll out behind a versioned decision with a comparison cohort and an explicit reversal trigger. Representative evaluation must include adversarial boundaries, stale state, partial dependency failure, and the busiest traffic window. That evidence converts a plausible architecture into an operable system and gives the team a durable baseline for the next model, workload, or regulation.

Reconstruction needs more than prompts and outputs

The same prompt can produce different behavior when retrieval candidates, policy, tools, temperature, provider version, or external state differ. The production boundary is larger than the feature that makes the technique attractive: callers arrive with different authorization, freshness, latency, and quality requirements, while the underlying model and data keep changing. A design is therefore incomplete until it states which inputs are part of identity, what may be reused, what must be recomputed, and which failure should be visible to the caller. Treating those choices as an explicit contract prevents an optimization from quietly becoming a correctness or governance bug.

A replay envelope records normalized input, authorization, retrieval ranks, model and prompt versions, tool receipts, policy decisions, random seed where supported, and stream events. This mechanism couples layers that teams often measure separately. A change in routing alters traffic shape; a change in traffic shape alters cache behavior and queue time; a change in retained context alters cost, privacy exposure, and model behavior. Preserve the intermediate decisions needed for replay: normalized inputs, policy version, selected route, relevant hashes, model revision, timing, and outcome. The final text alone cannot explain why the system behaved as it did, and it cannot support a safe rollback when the frontier model changes.

Store privacy-aware evidence at every boundary, preserve immutable incident snapshots, and maintain a replay environment that can substitute unavailable external tools. Use percentage of severe incidents reproducible to the decision boundary as the primary operating signal, but review it beside tail latency, unit cost, policy violations, and an outcome metric owned by the product. Roll out behind a versioned decision with a comparison cohort and an explicit reversal trigger. Representative evaluation must include adversarial boundaries, stale state, partial dependency failure, and the busiest traffic window. That evidence converts a plausible architecture into an operable system and gives the team a durable baseline for the next model, workload, or regulation.

Recovery proves the system changed

A prompt edit is not a completed remediation until the failure is encoded, the fix is evaluated, and production signals confirm the expected movement. The production boundary is larger than the feature that makes the technique attractive: callers arrive with different authorization, freshness, latency, and quality requirements, while the underlying model and data keep changing. A design is therefore incomplete until it states which inputs are part of identity, what may be reused, what must be recomputed, and which failure should be visible to the caller. Treating those choices as an explicit contract prevents an optimization from quietly becoming a correctness or governance bug.

The incident becomes a regression case, expands into neighboring adversarial cases, and gates release through offline, shadow, canary, and monitored rollout stages. This mechanism couples layers that teams often measure separately. A change in routing alters traffic shape; a change in traffic shape alters cache behavior and queue time; a change in retained context alters cost, privacy exposure, and model behavior. Preserve the intermediate decisions needed for replay: normalized inputs, policy version, selected route, relevant hashes, model revision, timing, and outcome. The final text alone cannot explain why the system behaved as it did, and it cannot support a safe rollback when the frontier model changes.

Assign an architecture owner, document displaced risk, compare pre- and post-fix slices, and retire temporary containment only after evidence reaches the agreed threshold. Use recurrence rate for the incident class across later releases as the primary operating signal, but review it beside tail latency, unit cost, policy violations, and an outcome metric owned by the product. Roll out behind a versioned decision with a comparison cohort and an explicit reversal trigger. Representative evaluation must include adversarial boundaries, stale state, partial dependency failure, and the busiest traffic window. That evidence converts a plausible architecture into an operable system and gives the team a durable baseline for the next model, workload, or regulation.

Research foundation and scope

NIST SP 800-61 Rev. 3 integrates incident response with cybersecurity risk management, while the NIST generative-AI profile and OWASP LLM guidance identify AI-specific risk classes [NIST-IR] [NIST-GAI] [OWASP-LLM]. These sources establish the mechanism, but they do not remove the need to measure the local implementation. A paper's result belongs to its checkpoint, dataset, hardware, traffic, and experimental protocol; a standard describes a contract or risk practice, not a guarantee that a particular product follows it. The useful engineering move is to convert each cited idea into a hypothesis that can be falsified with representative inputs and observable intermediate state.

An AI incident may return HTTP 200 while producing unsupported advice, leaking data, misusing a tool, amplifying poisoned retrieval, or causing runaway cost. Availability monitoring alone cannot identify it. Write these conditions into the design review before selecting a library or provider. Specify the authoritative source of truth, the authenticated principal, the versions that influence behavior, the maximum consequence of an error, and what the system will do when required evidence is missing. This boundary statement makes later optimization honest: teams can compare speed, cost, or convenience only after candidates satisfy the same correctness, authorization, privacy, and recovery contract.

Read [NIST-IR], [NIST-GAI], [OWASP-LLM] together rather than treating one source as a recipe. SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management supplies one part of the foundation; Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile supplies a second perspective; OWASP Top 10 for LLM Applications 2025 helps connect the mechanism to evaluation, governance, or operations. The citations are deliberately primary papers, standards, or official documentation. Product-specific claims still require local evidence, and any dated behavior must be rechecked against the cited version before an article or architecture decision is refreshed.

A production implementation blueprint

Define outcome-based severity, preserve privacy-aware replay evidence, assign owners, prebuild capability switches, isolate models and indexes, stop dangerous tools, retain receipts, and communicate degraded behavior honestly. Begin with one thin vertical path that preserves all decision evidence. Give each run a stable identifier; capture normalized input, authenticated scope, relevant dependency versions, selected policy, timing, result status, and any external receipt. Keep large or sensitive payloads in a separately governed store and pass bounded references through the orchestration layer. This produces a debuggable system without making raw customer content the default telemetry substrate.

Make state transitions explicit and make every external boundary return a typed result. Distinguish validation failure, permission denial, missing evidence, rate limit, timeout, dependency outage, user cancellation, and internal defect. The distinction matters because each permits a different response: clarify, retry, fall back, defer, compensate, or stop. A generic exception handler that asks the model to try again turns uncertainty into repeated cost and can duplicate irreversible effects. Retries require an idempotency strategy and a stored receipt, not optimism.

Roll out the implementation as a versioned policy. Record which cohort receives it, what baseline it replaces, which metrics can stop the rollout, and how to reverse stateful artifacts such as indexes, caches, adapters, or workflow histories. Review privacy and retention alongside performance. If raw content is needed temporarily for diagnosis, state the purpose, access, sampling, encryption, and deletion window. Production readiness means the team can explain, contain, and reverse behavior under load—not merely demonstrate the happy path once.

Failure lab and evaluation plan

Tabletop and chaos-test model regression, indirect injection, contaminated corpus, provider outage, runaway loop, authorization escape, cost spike, sensitive trace, and false-positive safety block. Exercise narrow containment. Keep the arrival pattern, data distribution, authorization, and dependency versions close to production. Use a paired baseline where possible and retain per-case results so averages cannot hide a severe slice. Offline evaluation should localize the failing stage; shadow evaluation should expose integration and traffic effects; a canary should confirm user-facing outcomes with a small blast radius. Each stage needs an exit criterion rather than a vague request to 'look good.'

Include missing model version, incomplete retrieval trace, mutable prompt records, delayed third-party logs, rollback that leaves caches warm, incident tooling sharing the same failed dependency, and recurrence after a superficial prompt patch. Add adversarial boundary cases and partial failures, not only malformed inputs. Terminate a worker after an external system may have committed, revoke access after a cache was populated, change one dependency version, and delay one branch until its data is stale. Confirm that cancellation releases resources and that a fallback preserves semantics. The test passes only when the external outcome, audit evidence, and user-visible state agree; a fluent final message is not proof of completion.

Evaluation data needs lineage equal to production data. Record why each case exists, who reviewed the expected behavior, which source or policy supports it, and when it must be refreshed. Separate generated cases from production incidents and prevent test cases from leaking into training. When a metric changes, inspect changed examples before accepting an aggregate. A statistically small regression can still block release if it crosses an authorization, privacy, safety, or irreversible-action boundary.

Operating metrics and decision record

Use time to detect, classify, contain, and recover; capability-specific containment, replay coverage, unauthorized effects, affected users, evidence completeness, regression-test creation, recurrence, and temporary-control retirement. Define every metric with a numerator, denominator, unit, time window, exclusions, and responsible owner. Segment by workload and consequence so a dominant easy class cannot hide a rare costly failure. Pair service signals with product outcomes: latency without completion encourages fast useless answers, while answer quality without queue and cost data can produce an uneconomic service. Review percentiles and distributions rather than relying on a single average.

Recovery is complete only after the failure is reproducible, the architecture or policy changed, regression evidence passes, canary outcomes confirm improvement, and temporary containment is safely removed. Put that choice in a short architecture decision record containing context, alternatives, evidence, assumptions, selected policy, rollout, reversal trigger, and unresolved risk. Link the evaluation snapshot and source versions. This document prevents a benchmark from becoming folklore and tells the next engineer which evidence must be repeated when the model, provider, traffic, data, regulation, or product promise changes.

Monitor leading and lagging signals after release. Leading signals reveal pressure—queue age, cache allocation, disagreement, validation failure, policy denial, or retry growth—before users report harm. Lagging signals show whether the job was actually completed and trusted. Alert on a service objective tied to user consequence, and make the run identifier available to support without exposing private content. Every serious miss should become a reproducible regression case and, when relevant, a new threat or data-quality control.

Primary sources and further reading

1. SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk ManagementNational Institute of Standards and Technology · standard
2. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence ProfileNational Institute of Standards and Technology · standard
3. OWASP Top 10 for LLM Applications 2025OWASP Foundation · standard