Transformer Inference, From HTTP Request to the Next Token

Prefill and decode are different workloads

An inference request begins with prefill: tokenization produces input IDs, the model embeds them, and every transformer layer processes the full prompt in parallel. Matrix multiplications are large enough to keep GPU tensor cores busy, so prefill is usually compute-bound. A 16,000-token prompt is not merely a longer version of a 200-token prompt; it creates a much larger attention workload and delays every other request sharing the device. This is why time to first token should be measured independently from the rate of later tokens.

Decode starts after prefill and produces one token per sequence per step. The model still reads billions of weights, but each request contributes only one new query vector. Without batching, those weight reads perform very little arithmetic and the GPU becomes memory-bandwidth-bound. Continuous batching combines decode steps from unrelated requests, amortizing the same weight access across many sequences. The scheduler therefore controls utilization as much as the model kernel does, especially when prompt lengths and requested output lengths vary widely.

Treating prefill and decode as one queue creates head-of-line blocking. A large document summarization prompt can stall chat requests whose users expect a first token within a few hundred milliseconds. Production systems increasingly separate or disaggregate the phases, reserving capacity for latency-sensitive prefill while decode workers consume transferred KV state. Disaggregation adds network traffic and operational complexity, but it gives each phase a scheduler, service objective, and scaling rule aligned with its real resource profile.

The scheduler is the hidden model architecture

Static batching waits for a group to finish together, wasting capacity whenever one sequence generates more tokens than its neighbors. Continuous batching removes completed sequences after every decode iteration and immediately admits new work. The policy must balance batch width against latency: admitting a long prefill may improve throughput while making every active decoder wait. Useful controls include separate token budgets for prefill and decode, maximum chunk sizes, request priorities, and aging so low-priority work eventually advances.

Chunked prefill divides a large prompt into bounded token blocks and interleaves those blocks with decode iterations. The technique prevents a single request from monopolizing a device, but each chunk adds scheduling and kernel-launch overhead. The right chunk size depends on model width, GPU generation, expected concurrency, and latency targets. Benchmark with the actual prompt distribution; an average prompt length hides the tail that drives queueing. Record scheduled tokens per iteration, not only requests per second, because tokens are the meaningful unit of work.

Admission control belongs before the GPU, where the server can reject or defer work based on estimated token cost. A request declares prompt length immediately, but its generated length is uncertain, so reserve conservatively and revise the reservation during decoding. If a customer requests 32,000 output tokens, do not let that theoretical maximum strand capacity forever. Combine tenant quotas, queue deadlines, and cancellation propagation. A disconnected client should release its sequence, KV pages, and scheduler slot within one iteration rather than becoming invisible zombie load.

Kernels, quantization, and memory movement

Model serving speed is often limited by bytes moved rather than floating-point operations. Fused kernels reduce round trips through high-bandwidth memory by combining operations such as normalization, projection, bias, and activation. FlashAttention avoids materializing the full attention score matrix, processing tiles in on-chip memory instead. These optimizations change memory behavior without changing model semantics. Their payoff depends on sequence length and batch shape, which is why a benchmark on one fixed prompt rarely predicts production performance.

Weight quantization reduces both memory footprint and bandwidth demand. An eight-bit or four-bit representation may let a model fit on fewer GPUs and increase decode speed, but dequantization kernels, group size, calibration, and outlier handling determine the real result. Evaluate task quality and token probabilities, not only perplexity on a convenient corpus. KV-cache quantization is a separate decision: it saves capacity that grows with concurrent sequence length, yet errors can accumulate across long generation and hurt retrieval-heavy or code workloads first.

Tensor parallelism splits layer operations across GPUs and requires collectives on nearly every layer. Pipeline parallelism reduces that communication frequency but introduces bubbles and more complicated batching. On a fast intra-node fabric, tensor parallelism may be straightforward; across ordinary Ethernet, communication can erase the gain from extra accelerators. Start with a placement model that accounts for weights, KV cache, temporary activations, and allocator fragmentation. A model that technically fits can still fail under realistic concurrency when cache growth consumes the remaining memory.

A benchmark that predicts production

A useful load test samples real prompt lengths, output lengths, arrival patterns, tenants, and cancellation rates. Open-loop testing preserves the requested arrival rate even when the server slows, exposing queue collapse that closed-loop clients conceal. Report time to first token, inter-token latency, end-to-end latency, successful tokens per second, and the fraction rejected by admission control. Percentiles must be segmented by workload class because a blended P95 can make both interactive chat and offline summarization look healthier than either truly is.

Warm the server, but also test cold model load, graph compilation, autoscaling, and cache eviction. Production incidents happen during transitions: a deployment replaces replicas, a traffic burst forces scale-out, or a popular long context fills memory. Inject failures into one worker and verify that gateways retry only requests that are safe to replay. Mid-generation retries require deterministic sampling state or explicit user-visible continuation; silently starting over can duplicate tool calls or stream contradictory text.

Capacity planning should produce a safe operating envelope rather than one maximum number. Map arrival rate and input-token rate against first-token and decode service objectives, then identify where queueing becomes nonlinear. Leave headroom for prompt-length variance, hardware degradation, and background maintenance. Track the same envelope after every model, runtime, driver, or quantization change. The serving stack is a coupled system, so a model advertised as faster may reduce usable throughput if it emits longer answers or consumes more cache per token.

Research foundation and scope

Transformer serving alternates a parallel prompt-prefill phase with a sequential decode phase. FlashAttention demonstrates why input/output movement across the GPU memory hierarchy matters, while PagedAttention shows that scheduler-visible KV allocation determines feasible batch width [FLASH] [VLLM]. These sources establish the mechanism, but they do not remove the need to measure the local implementation. A paper's result belongs to its checkpoint, dataset, hardware, traffic, and experimental protocol; a standard describes a contract or risk practice, not a guarantee that a particular product follows it. The useful engineering move is to convert each cited idea into a hypothesis that can be falsified with representative inputs and observable intermediate state.

A benchmark result is valid only for its model, numerical format, device topology, prompt-length distribution, output-length distribution, sampling policy, and concurrency. Single-request tokens per second cannot predict queueing under mixed traffic. Write these conditions into the design review before selecting a library or provider. Specify the authoritative source of truth, the authenticated principal, the versions that influence behavior, the maximum consequence of an error, and what the system will do when required evidence is missing. This boundary statement makes later optimization honest: teams can compare speed, cost, or convenience only after candidates satisfy the same correctness, authorization, privacy, and recovery contract.

Read [FLASH], [VLLM], [SPEC] together rather than treating one source as a recipe. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness supplies one part of the foundation; Efficient Memory Management for Large Language Model Serving with PagedAttention supplies a second perspective; Fast Inference from Transformers via Speculative Decoding helps connect the mechanism to evaluation, governance, or operations. The citations are deliberately primary papers, standards, or official documentation. Product-specific claims still require local evidence, and any dated behavior must be rechecked against the cited version before an article or architecture decision is refreshed.

A production implementation blueprint

Instrument request admission, tokenization, queue wait, prefill, every decode iteration, streaming, cancellation, and cache release as separate spans. Schedule by token budgets and protect interactive prefill from unbounded long-context work. Begin with one thin vertical path that preserves all decision evidence. Give each run a stable identifier; capture normalized input, authenticated scope, relevant dependency versions, selected policy, timing, result status, and any external receipt. Keep large or sensitive payloads in a separately governed store and pass bounded references through the orchestration layer. This produces a debuggable system without making raw customer content the default telemetry substrate.

Make state transitions explicit and make every external boundary return a typed result. Distinguish validation failure, permission denial, missing evidence, rate limit, timeout, dependency outage, user cancellation, and internal defect. The distinction matters because each permits a different response: clarify, retry, fall back, defer, compensate, or stop. A generic exception handler that asks the model to try again turns uncertainty into repeated cost and can duplicate irreversible effects. Retries require an idempotency strategy and a stored receipt, not optimism.

Roll out the implementation as a versioned policy. Record which cohort receives it, what baseline it replaces, which metrics can stop the rollout, and how to reverse stateful artifacts such as indexes, caches, adapters, or workflow histories. Review privacy and retention alongside performance. If raw content is needed temporarily for diagnosis, state the purpose, access, sampling, encryption, and deletion window. Production readiness means the team can explain, contain, and reverse behavior under load—not merely demonstrate the happy path once.

Failure lab and evaluation plan

Replay an open-loop arrival trace containing short chat, long documents, cancellations, and bursts. Compare continuous batching and chunked prefill configurations at equal quality while recording time to first token and inter-token latency by prompt-length bucket. Keep the arrival pattern, data distribution, authorization, and dependency versions close to production. Use a paired baseline where possible and retain per-case results so averages cannot hide a severe slice. Offline evaluation should localize the failing stage; shadow evaluation should expose integration and traffic effects; a canary should confirm user-facing outcomes with a small blast radius. Each stage needs an exit criterion rather than a vague request to 'look good.'

Include cold model load, graph compilation, cache exhaustion, one failed worker, disconnected clients, a slow tokenizer, and communication degradation between tensor-parallel devices. Verify that retries do not duplicate a stream or a tool side effect. Add adversarial boundary cases and partial failures, not only malformed inputs. Terminate a worker after an external system may have committed, revoke access after a cache was populated, change one dependency version, and delay one branch until its data is stale. Confirm that cancellation releases resources and that a fallback preserves semantics. The test passes only when the external outcome, audit evidence, and user-visible state agree; a fluent final message is not proof of completion.

Evaluation data needs lineage equal to production data. Record why each case exists, who reviewed the expected behavior, which source or policy supports it, and when it must be refreshed. Separate generated cases from production incidents and prevent test cases from leaking into training. When a metric changes, inspect changed examples before accepting an aggregate. A statistically small regression can still block release if it crosses an authorization, privacy, safety, or irreversible-action boundary.

Operating metrics and decision record

Use time to first token, inter-token latency, successful output tokens per second, queued tokens, cache pages in use, rejection rate, and cost per completed response. Publish percentiles by workload class rather than one blended number. Define every metric with a numerator, denominator, unit, time window, exclusions, and responsible owner. Segment by workload and consequence so a dominant easy class cannot hide a rare costly failure. Pair service signals with product outcomes: latency without completion encourages fast useless answers, while answer quality without queue and cost data can produce an uneconomic service. Review percentiles and distributions rather than relying on a single average.

Choose the serving configuration from a safe operating envelope where latency objectives hold under the expected token arrival rate, then retain headroom for deployments, traffic variance, and long-context tails. Put that choice in a short architecture decision record containing context, alternatives, evidence, assumptions, selected policy, rollout, reversal trigger, and unresolved risk. Link the evaluation snapshot and source versions. This document prevents a benchmark from becoming folklore and tells the next engineer which evidence must be repeated when the model, provider, traffic, data, regulation, or product promise changes.

Monitor leading and lagging signals after release. Leading signals reveal pressure—queue age, cache allocation, disagreement, validation failure, policy denial, or retry growth—before users report harm. Lagging signals show whether the job was actually completed and trusted. Alert on a service objective tied to user consequence, and make the run identifier available to support without exposing private content. Every serious miss should become a reproducible regression case and, when relevant, a new threat or data-quality control.

Primary sources and further reading

1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessOriginal research paper · paper
2. Efficient Memory Management for Large Language Model Serving with PagedAttentionOriginal research paper · paper
3. Fast Inference from Transformers via Speculative DecodingOriginal research paper · paper