Model Context Protocol in Production

MCP standardizes a boundary, not your business logic

The Model Context Protocol gives clients a common way to discover and use tools, resources, and prompts. It reduces bespoke integration code between AI hosts and capability providers. The server still owns domain semantics, authorization, validation, idempotency, and data quality. Publishing a broad internal API through MCP without redesign simply gives models a standardized way to reach an unsafe interface.

Choose primitives deliberately. Resources expose addressable context, tools perform computations or actions, and prompts offer reusable interaction templates. A customer record lookup may be a tool because it requires parameters and authorization, while a stable handbook section may be a resource. Keep side effects out of resources and make write tools unmistakable in name, description, and response. Clients need to reason about capability risk before invocation.

Schemas are the executable product surface. Use narrow arguments, bounded results, stable identifiers, and structured error codes. Describe side effects, prerequisites, pagination, and whether retry is safe. Avoid returning huge opaque text blobs when the client needs three fields. Version incompatible behavior through capability names or negotiated server versions instead of silently changing what an existing tool does.

Authenticate the principal at every call

An MCP connection does not automatically establish what the user may do. Bind requests to an authenticated principal and validate audience, tenant, scopes, and token expiry. Servers should not trust identity claims embedded in model arguments. For delegated access, preserve which user initiated the action and which host is acting on their behalf. Service credentials alone erase accountability and often grant excessive authority.

Apply least privilege per server and per tool. A research session should receive read capabilities, while a publishing tool can require a different connection and explicit consent. Short-lived tokens reduce exposure and make revocation meaningful. Secrets remain in the host or server credential layer and never enter model context. Tool results should be filtered according to the caller, even when the model asks for broader fields.

Consequential tools need scoped confirmation and idempotency. The server can return a proposed change or approval requirement before committing. Confirmation must bind exact parameters and expire. Idempotency keys survive client retries and reconnects. Server-side audit records capture principal, capability, arguments hash, policy decision, result, and external receipt without logging unnecessary sensitive content.

Design for unreliable transports and clients

Local and remote transports have different failure and trust characteristics. Remote connections need TLS, origin and redirect controls, deadlines, request limits, and clear reconnect behavior. A client disconnect can occur after a server commits a write but before the result arrives. Stable operation IDs and result lookup let the client reconcile rather than repeat. Tool contracts should document these ambiguous states.

Capability discovery can change during a session. Clients should handle server version changes, removed tools, and updated schemas without assuming a permanent catalog. Servers can provide capability metadata and health, while clients pin a snapshot for an active workflow when reproducibility matters. A model should not continue a cached plan after a required tool disappears; the orchestrator revalidates before every call.

Bound result size and support pagination or resource handles. Streaming is useful for progress and large reads, but each event needs sequence and completion semantics. Cancellation should propagate to cancellable work; committed writes reconcile normally. Apply backpressure so a fast server cannot overwhelm a client’s context or event queue. Protocol interoperability does not eliminate distributed-systems engineering.

Operate an ecosystem, not a demo server

Trace host request, model decision, MCP call, server work, downstream API, and external receipt with shared correlation identifiers. Measure discovery failures, schema errors, authorization denials, tool latency, retries, result sizes, and client cancellations. Separate model-proposed invalid calls from server defects. This tells capability owners whether to improve descriptions, validation, performance, or client integration.

Test servers with ordinary contract tests and adversarial clients. Fuzz arguments, omit required authorization, replay idempotency keys, disconnect mid-write, inject instructions into resource content, and request excessive pages. Maintain conformance fixtures for supported protocol versions. A tool that works in one preferred host may still rely on undocumented behavior and fail elsewhere.

Publish ownership, service objectives, change policy, and deprecation windows for internal servers. Keep a registry with risk tier and data classification so hosts can decide which capabilities are eligible. MCP makes capability distribution easier; governance prevents that convenience from becoming uncontrolled tool sprawl. The mature outcome is reusable, observable, least-privilege infrastructure that many AI products can share.

Research foundation and scope

MCP standardizes JSON-RPC-based interaction among hosts, clients, and servers, with negotiated capabilities for resources, prompts, tools, and related features [MCP-SPEC]. Its authorization specification describes protected-resource and authorization-server discovery [MCP-AUTH]. These sources establish the mechanism, but they do not remove the need to measure the local implementation. A paper's result belongs to its checkpoint, dataset, hardware, traffic, and experimental protocol; a standard describes a contract or risk practice, not a guarantee that a particular product follows it. The useful engineering move is to convert each cited idea into a hypothesis that can be falsified with representative inputs and observable intermediate state.

Protocol conformance does not authorize a business action, make server output trustworthy, or guarantee transport reliability. The host still owns user consent, principal propagation, policy, isolation, and rendering safety. Write these conditions into the design review before selecting a library or provider. Specify the authoritative source of truth, the authenticated principal, the versions that influence behavior, the maximum consequence of an error, and what the system will do when required evidence is missing. This boundary statement makes later optimization honest: teams can compare speed, cost, or convenience only after candidates satisfy the same correctness, authorization, privacy, and recovery contract.

Read [MCP-SPEC], [MCP-AUTH], [OWASP-LLM] together rather than treating one source as a recipe. Model Context Protocol specification, 2025-11-25 supplies one part of the foundation; Model Context Protocol authorization specification supplies a second perspective; OWASP Top 10 for LLM Applications 2025 helps connect the mechanism to evaluation, governance, or operations. The citations are deliberately primary papers, standards, or official documentation. Product-specific claims still require local evidence, and any dated behavior must be rechecked against the cited version before an article or architecture decision is refreshed.

A production implementation blueprint

Pin a supported specification revision, validate initialization and capability negotiation, authenticate every protected request, expose narrow tools, bind calls to the current principal, limit output, and preserve request and effect identifiers. Begin with one thin vertical path that preserves all decision evidence. Give each run a stable identifier; capture normalized input, authenticated scope, relevant dependency versions, selected policy, timing, result status, and any external receipt. Keep large or sensitive payloads in a separately governed store and pass bounded references through the orchestration layer. This produces a debuggable system without making raw customer content the default telemetry substrate.

Make state transitions explicit and make every external boundary return a typed result. Distinguish validation failure, permission denial, missing evidence, rate limit, timeout, dependency outage, user cancellation, and internal defect. The distinction matters because each permits a different response: clarify, retry, fall back, defer, compensate, or stop. A generic exception handler that asks the model to try again turns uncertainty into repeated cost and can duplicate irreversible effects. Retries require an idempotency strategy and a stored receipt, not optimism.

Roll out the implementation as a versioned policy. Record which cohort receives it, what baseline it replaces, which metrics can stop the rollout, and how to reverse stateful artifacts such as indexes, caches, adapters, or workflow histories. Review privacy and retention alongside performance. If raw content is needed temporarily for diagnosis, state the purpose, access, sampling, encryption, and deletion window. Production readiness means the team can explain, contain, and reverse behavior under load—not merely demonstrate the happy path once.

Failure lab and evaluation plan

Run protocol conformance, cancellation, timeout, reconnect, pagination, capability downgrade, and auth-expiry tests. Treat every resource and tool response as untrusted data and fuzz it through the host renderer and model context. Keep the arrival pattern, data distribution, authorization, and dependency versions close to production. Use a paired baseline where possible and retain per-case results so averages cannot hide a severe slice. Offline evaluation should localize the failing stage; shadow evaluation should expose integration and traffic effects; a canary should confirm user-facing outcomes with a small blast radius. Each stage needs an exit criterion rather than a vague request to 'look good.'

Include confused-deputy authorization, token audience mismatch, malicious tool description, oversized result, server restart, duplicate call after transport loss, incompatible revision, prompt injection in a resource, and consent bypass. Add adversarial boundary cases and partial failures, not only malformed inputs. Terminate a worker after an external system may have committed, revoke access after a cache was populated, change one dependency version, and delay one branch until its data is stale. Confirm that cancellation releases resources and that a fallback preserves semantics. The test passes only when the external outcome, audit evidence, and user-visible state agree; a fluent final message is not proof of completion.

Evaluation data needs lineage equal to production data. Record why each case exists, who reviewed the expected behavior, which source or policy supports it, and when it must be refreshed. Separate generated cases from production incidents and prevent test cases from leaking into training. When a metric changes, inspect changed examples before accepting an aggregate. A statistically small regression can still block release if it crosses an authorization, privacy, safety, or irreversible-action boundary.

Operating metrics and decision record

Track negotiated capabilities, auth failures, tool validation, latency and cancellation, duplicate effects, server-version distribution, unsafe rendering blocks, consent decisions, and diagnosable protocol errors. Define every metric with a numerator, denominator, unit, time window, exclusions, and responsible owner. Segment by workload and consequence so a dominant easy class cannot hide a rare costly failure. Pair service signals with product outcomes: latency without completion encourages fast useless answers, while answer quality without queue and cost data can produce an uneconomic service. Review percentiles and distributions rather than relying on a single average.

Use MCP as an integration boundary while keeping business services independently authorized and idempotent; never expose a generic backdoor merely because the protocol can describe it. Put that choice in a short architecture decision record containing context, alternatives, evidence, assumptions, selected policy, rollout, reversal trigger, and unresolved risk. Link the evaluation snapshot and source versions. This document prevents a benchmark from becoming folklore and tells the next engineer which evidence must be repeated when the model, provider, traffic, data, regulation, or product promise changes.

Monitor leading and lagging signals after release. Leading signals reveal pressure—queue age, cache allocation, disagreement, validation failure, policy denial, or retry growth—before users report harm. Lagging signals show whether the job was actually completed and trusted. Alert on a service objective tied to user consequence, and make the run identifier available to support without exposing private content. Every serious miss should become a reproducible regression case and, when relevant, a new threat or data-quality control.

Primary sources and further reading

1. Model Context Protocol specification, 2025-11-25Model Context Protocol · standard
2. Model Context Protocol authorization specificationModel Context Protocol · standard
3. OWASP Top 10 for LLM Applications 2025OWASP Foundation · standard