Blog
Inference SystemsKV Cache Engineering: Pages, Prefix Reuse, and Memory Pressure
How to size, allocate, reuse, compress, and evict the state that makes autoregressive decoding practical.
Inference SystemsSemantic Caching Without Serving the Wrong Answer Faster
A production design for similarity keys, freshness, authorization, invalidation, and the economics that determine when semantic reuse is safe.
Inference SystemsSpeculative Decoding Without Hand-Waving
Draft models, acceptance math, tree proposals, and the operational details that decide whether speculation makes serving faster.
Evaluation and SecuritySynthetic Evaluation Data That Finds Real Failures
How to generate, filter, diversify, and maintain synthetic cases without turning an evaluation suite into a mirror of its generator.
Inference SystemsThe LLM Gateway Is a Policy Engine, Not a Proxy
Designing the routing, budgets, identity, resilience, and evidence layer between products and a changing model portfolio.
Evaluation and SecurityPrivacy-Preserving AI Is a Dataflow Architecture
Purpose limitation, minimization, isolation, retention, redaction, and verifiable deletion across retrieval, models, tools, traces, and feedback loops.
Product ArchitectureIncident Response for Systems That Can Be Wrong Fluently
A response playbook for quality regressions, prompt attacks, retrieval contamination, runaway agents, cost spikes, and provider failures.
RetrievalProduction RAG Is an Evidence Pipeline, Not a Vector Search
Designing retrieval-augmented generation around ingestion quality, query planning, evidence assembly, and verifiable answers.
RetrievalHybrid Search: Making Lexical and Vector Retrieval Cooperate
A practical design for candidate generation, score normalization, rank fusion, metadata filters, and retrieval evaluation.
RetrievalRerankers: The Quality Layer Between Search and Generation
How cross-encoders, late interaction, listwise ranking, and calibration turn noisy candidates into usable evidence.
AgentsEngineering the Agent Tool Loop
A concrete architecture for tool selection, state transitions, permissions, budgets, and recovery in production agents.
AgentsDurable Execution for Agents That Outlive a Web Request
Persisting checkpoints, replaying deterministically, handling human approval, and surviving crashes without duplicating side effects.