Prompt Injection Defense in Depth

Understand the confused-deputy problem

Direct prompt injection comes from the user; indirect injection arrives inside content the system retrieves or opens, and it is the first entry in the OWASP LLM top ten for good reason. A web page can instruct an agent to reveal data, alter a transaction, or contact an attacker. The model sees both legitimate instructions and hostile text as tokens, and instruction hierarchy is a behavioral tendency rather than isolation. Quoting, XML tags, and reminders help performance but cannot establish a security boundary.

The core risk is a confused deputy: an attacker influences a model that holds permissions the attacker lacks. The impact depends on reachable tools and data. A read-only summarizer may produce a bad summary; a browsing agent connected to email and payments can leak or act. Reduce authority before improving detection. If a task needs one document and one draft, do not grant the run an entire mailbox and send permission. On a long-running agent the same rule extends over time, because an approval granted hours ago is authority the run is still carrying.

Define attack goals beyond system-prompt disclosure. Exfiltration can encode secrets in URLs, tool arguments, generated images, or innocuous-looking text. Integrity attacks alter decisions, availability attacks create loops, and persistence attacks write malicious content into memory. Threat scenarios should follow data through tools and outputs, which is how MITRE ATLAS catalogues them, because blocking one phrase such as “ignore previous instructions” barely changes a determined attacker.

Separate content authority from tool authority

Every context item needs provenance and authority metadata. System policy, authenticated user intent, retrieved evidence, and web content should not be concatenated as peers. The orchestrator can render clear boundaries and tell the model how each source may be used, but enforcement happens outside generation. Text from a document may support an answer; it cannot approve a tool or change destination constraints.

Use capability-scoped tools and bind them to the initiating user, tenant, task, and expiration. Validate destinations, amounts, resource identifiers, and data flow against policy. For outbound requests, block arbitrary URLs or apply domain allowlists and egress proxies. A model should not be able to place private context inside a query parameter to an untrusted host. Network policy is a stronger exfiltration control than asking the model not to leak.

Require confirmation for actions whose full parameters were influenced by untrusted content. The confirmation view shows the exact effect and highlights external destinations. Do not ask users to approve raw JSON they cannot assess, because an approval surface nobody can read is an architecture problem rather than a UI problem. Some actions should remain forbidden even with confirmation when the model cannot establish necessary provenance or when data would cross a restricted boundary.

Detect, minimize, and contain

Injection classifiers and pattern rules can prioritize risk, but adaptive attacks, multilingual text, encoding, and benign security discussions guarantee errors. Use detection to filter, quarantine, lower trust, or require review rather than as the only gate. Test classifiers on obfuscated and context-dependent attacks alongside benign technical content, and calibrate them the way you would calibrate any other model judge. Record which source triggered the signal so the system can exclude that item instead of discarding the entire task.

Minimize context before the model sees it. Extract relevant fields from structured sources, render pages as bounded text, remove hidden content, and avoid following links automatically. Use a separate low-privilege model or deterministic parser to inspect untrusted documents before they enter a capable agent. Summarization is not sanitization because an injected instruction can survive or influence the summary, but isolation limits what the inspection stage can do.

Taint tracking can mark values derived from untrusted content and restrict where they flow. A proposed email address copied from a web page, for example, should not become an approved destination without validation. Full semantic taint tracking is difficult, yet even field-level provenance for URLs, recipients, commands, and identifiers catches practical attacks. Preserve provenance through model-generated plans rather than losing it in a prose summary.

Red-team complete attack chains

Tests should place payloads in HTML, PDFs, image text, issue comments, tool results, filenames, calendar descriptions, and memory. Evaluate whether the agent reads secrets, calls tools, changes plans, persists instructions, or sends data externally. Include attacks that split instructions across several sources or wait until a later turn, which is exactly where a run that survives a restart widens the window. Component tests miss these delayed chains.

Use canary secrets and fake high-impact tools in a controlled environment. Measure unauthorized action rate, attempted exfiltration, detection, user-confirmation quality, and recovery. A model refusing in its final text does not count as safe if it already issued a dangerous call. Capture the full structured trace and convert every successful chain into a regression test covering the responsible boundary.

Assume some injections will succeed at the model layer and ask whether architecture still prevents harm. This mindset produces narrow permissions, explicit data-flow policies, isolated browsing, and verified actions. Improve prompts and classifiers as additional friction, but never make business security depend on perfect instruction following. The defensible goal is bounded impact under model compromise.

Research foundation and scope

OWASP identifies prompt injection as a leading LLM-application risk, and research on indirect injection demonstrates how untrusted external content can influence an application-integrated model [OWASP-LLM] [INDIRECT-PI]. The architectural problem resembles a confused deputy. These sources establish the mechanism, but they do not remove the need to measure the local implementation. A paper's result belongs to its checkpoint, dataset, hardware, traffic, and experimental protocol; a standard describes a contract or risk practice, not a guarantee that a particular product follows it. The useful engineering move is to convert each cited idea into a hypothesis that can be falsified with representative inputs and observable intermediate state.

Instructions and data share the model's token channel, so prompt wording cannot create a hard privilege boundary. Retrieval, fine-tuning, or a second model may reduce particular attacks but cannot authorize actions. Write these conditions into the design review before selecting a library or provider. Specify the authoritative source of truth, the authenticated principal, the versions that influence behavior, the maximum consequence of an error, and what the system will do when required evidence is missing. This boundary statement makes later optimization honest: teams can compare speed, cost, or convenience only after candidates satisfy the same correctness, authorization, privacy, and recovery contract.

Read [OWASP-LLM], [INDIRECT-PI], [NIST-800-154] together rather than treating one source as a recipe. OWASP Top 10 for LLM Applications 2025 supplies one part of the foundation; More than you've asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models supplies a second perspective; Guide to Data-Centric System Threat Modeling helps connect the mechanism to evaluation, governance, or operations. The citations are deliberately primary papers, standards, or official documentation. Product-specific claims still require local evidence, and any dated behavior must be rechecked against the cited version before an article or architecture decision is refreshed.

A production implementation blueprint

Label untrusted content, minimize it, separate read and write capabilities, validate tool calls against authenticated policy, require confirmation for consequential effects, constrain egress, encode rendered output, and log decisions without leaking secrets. Begin with one thin vertical path that preserves all decision evidence. Give each run a stable identifier; capture normalized input, authenticated scope, relevant dependency versions, selected policy, timing, result status, and any external receipt. Keep large or sensitive payloads in a separately governed store and pass bounded references through the orchestration layer. This produces a debuggable system without making raw customer content the default telemetry substrate.

Make state transitions explicit and make every external boundary return a typed result. Distinguish validation failure, permission denial, missing evidence, rate limit, timeout, dependency outage, user cancellation, and internal defect. The distinction matters because each permits a different response: clarify, retry, fall back, defer, compensate, or stop. A generic exception handler that asks the model to try again turns uncertainty into repeated cost and can duplicate irreversible effects. Retries require an idempotency strategy and a stored receipt, not optimism.

Roll out the implementation as a versioned policy. Record which cohort receives it, what baseline it replaces, which metrics can stop the rollout, and how to reverse stateful artifacts such as indexes, caches, adapters, or workflow histories. Review privacy and retention alongside performance. If raw content is needed temporarily for diagnosis, state the purpose, access, sampling, encryption, and deletion window. Production readiness means the team can explain, contain, and reverse behavior under load—not merely demonstrate the happy path once.

Failure lab and evaluation plan

Red-team direct and indirect attacks in web pages, documents, images, tool results, memory, and multi-turn conversations. Test encoded, translated, fragmented, and authority-impersonating variants through the full tool chain. Keep the arrival pattern, data distribution, authorization, and dependency versions close to production. Use a paired baseline where possible and retain per-case results so averages cannot hide a severe slice. Offline evaluation should localize the failing stage; shadow evaluation should expose integration and traffic effects; a canary should confirm user-facing outcomes with a small blast radius. Each stage needs an exit criterion rather than a vague request to 'look good.'

Force a detector miss, a poisoned high-ranked document, a malicious tool description, data exfiltration through a URL, cross-tenant retrieval, and approval-text spoofing. Verify that privileges remain unavailable despite model compliance. Add adversarial boundary cases and partial failures, not only malformed inputs. Terminate a worker after an external system may have committed, revoke access after a cache was populated, change one dependency version, and delay one branch until its data is stale. Confirm that cancellation releases resources and that a fallback preserves semantics. The test passes only when the external outcome, audit evidence, and user-visible state agree; a fluent final message is not proof of completion.

Evaluation data needs lineage equal to production data. Record why each case exists, who reviewed the expected behavior, which source or policy supports it, and when it must be refreshed. Separate generated cases from production incidents and prevent test cases from leaking into training. When a metric changes, inspect changed examples before accepting an aggregate. A statistically small regression can still block release if it crosses an authorization, privacy, safety, or irreversible-action boundary.

Operating metrics and decision record

Use attack-chain success, unauthorized tool proposal and execution, sensitive-data egress, detector recall by channel, false blocks, approval bypass, time to containment, and residual capability after each control fails. Define every metric with a numerator, denominator, unit, time window, exclusions, and responsible owner. Segment by workload and consequence so a dominant easy class cannot hide a rare costly failure. Pair service signals with product outcomes: latency without completion encourages fast useless answers, while answer quality without queue and cost data can produce an uneconomic service. Review percentiles and distributions rather than relying on a single average.

Assume some injected instructions reach the model; design so that following them still cannot cross authorization, data-access, egress, or confirmation boundaries. Put that choice in a short architecture decision record containing context, alternatives, evidence, assumptions, selected policy, rollout, reversal trigger, and unresolved risk. Link the evaluation snapshot and source versions. This document prevents a benchmark from becoming folklore and tells the next engineer which evidence must be repeated when the model, provider, traffic, data, regulation, or product promise changes.

Monitor leading and lagging signals after release. Leading signals reveal pressure—queue age, cache allocation, disagreement, validation failure, policy denial, or retry growth—before users report harm. Lagging signals show whether the job was actually completed and trusted. Alert on a service objective tied to user consequence, and make the run identifier available to support without exposing private content. Every serious miss should become a reproducible regression case and, when relevant, a new threat or data-quality control.

Primary sources and further reading

1. OWASP Top 10 for LLM Applications 2025OWASP Foundation · standard
2. More than you've asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language ModelsOriginal research paper · paper
3. Guide to Data-Centric System Threat ModelingNational Institute of Standards and Technology · standard

Frequently asked questions

Methodology

The risk framing here follows the OWASP Top 10 for LLM Applications, which ranks prompt injection first, and the indirect injection research that demonstrated how retrieved content reaches an application-integrated model. The threat-path method, working from valuable data outward rather than from the model inward, is NIST SP 800-154.

The article deliberately publishes no detection rates. Classifier numbers do not transfer across corpora, languages, or encodings, and quoting one would imply a guarantee that the next obfuscation removes. Every control named here is one that can be enforced outside the model and verified by an operator: an authorization decision, an egress rule, a confirmation surface, or a provenance field. Controls that only work when the model cooperates are described as friction, not as defense.

Find out whether your agent's blast radius is actually bounded

The useful question is not whether your model can be tricked. It is what an attacker gets when it is: which tools the run can reach, whose permissions they carry, where an outbound request can go, and which of those actions a human is really confirming.

Vishal runs that as an independent review of the trust boundary with your engineers, working from the tool surface and the egress path rather than from the prompt. The Agent Trust Boundary is the long-form version of the same argument, and where the containment has to be designed into a system that is still being built, that is a build engagement.

Book a discovery call