Category archive
Serving a model is a scheduling problem with a budget attached.
Latency, throughput and cost in a language model system are set by the parts around the weights: the scheduler, the cache, the gateway, and the memory budget every request competes for. These articles measure those parts and treat the model itself as the one input that is already fixed.
Articles in Inference Systems
Inference SystemsTransformer Inference, From HTTP Request to the Next Token
A systems-level tour of prefill, decode, batching, memory movement, and the scheduling decisions that determine real-world LLM latency.
Inference SystemsKV Cache Engineering: Pages, Prefix Reuse, and Memory Pressure
How to size, allocate, reuse, compress, and evict the state that makes autoregressive decoding practical.
Inference SystemsSemantic Caching Without Serving the Wrong Answer Faster
A production design for similarity keys, freshness, authorization, invalidation, and the economics that determine when semantic reuse is safe.
Inference SystemsSpeculative Decoding Without Hand-Waving
Draft models, acceptance math, tree proposals, and the operational details that decide whether speculation makes serving faster.
Inference SystemsThe LLM Gateway Is a Policy Engine, Not a Proxy
Designing the routing, budgets, identity, resilience, and evidence layer between products and a changing model portfolio.
Inference SystemsFine-Tuning with LoRA: Data, Adapters, and Production Operations
When fine-tuning is justified, how low-rank adapters work, and what it takes to evaluate, serve, and update them safely.
Inference SystemsLLM Cost Engineering: From Token Prices to Unit Economics
Modeling full request cost, eliminating waste, forecasting margins, and optimizing without disguising quality regressions.
Apply the category
Put this against a real system.
If a decision in Inference Systems is in front of you right now, the fastest version of this is a call: bring the architecture, the failure you are seeing, and the constraint you cannot move.
Discuss the system