Making inference fast and cheap

Inference limits come in three kinds: GPU compute during prefill, memory bandwidth during decode, and memory capacity for weights and KV ca…

Part 1 of 5

Basics

What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.

1.1Definition

The one-paragraph summary, then where its responsibility starts and stops.

Inference limits come in three kinds: GPU compute during prefill, memory bandwidth during decode, and memory capacity for weights and KV cache. Each acceleration technique relaxes one: smaller numbers, fewer tokens, better batching, or guessing ahead.

Levers you control
  • Route easy requests to smaller models
  • Send fewer tokens: trim context, cache prefixes
  • Quantise weights and KV cache
  • Batch continuously and page the KV cache
  • Decode speculatively with a draft model
  • Distil a big model into a small task model
What doesn’t help
  • Adding GPUs to a decode-bound workload without more batching
  • Raising max tokens ‘just in case’
  • Optimising before measuring TTFT, TPOT and cost per request
  • Quantising without re-running your evaluation set

1.2Key concepts

Ideas this deep dive relies on, in one sentence each.

01KV cache

Stored attention keys and values per token during generation; it limits concurrent requests per GPU.

02Continuous batching

Serving technique that adds and removes requests from a running batch every step.

03PagedAttention

Managing the KV cache in blocks to cut memory waste.

04Quantisation

Storing weights or activations at lower precision to save memory and speed up inference.

05LoRA

Fine-tuning by training small adapter matrices instead of all weights.

1.3A real optimisation story

A concrete example first, so the mechanics have something to attach to.

A support summariser on a self-hosted 70B model costs too much and p95 latency is 9 seconds. The team measures first: average prompt 6,000 tokens, output 250, GPU utilisation 35% because each replica processes one request at a time.

Step one moves to vLLM with continuous batching and paged KV cache: throughput per GPU roughly triples and p95 drops, because the GPU now decodes many requests per step. Step two trims the prompt: summarising ticket history once and caching it cuts input to 2,500 tokens and halves TTFT.

Step three quantises to FP8 weights and KV cache after the evaluation set shows less than 0.5% quality change, freeing memory for more concurrent requests. Step four routes short tickets to an 8B model distilled on the 70B model’s best summaries; 60% of traffic moves there.

Result: cost per summary falls by a large multiple and p95 latency drops below 3 seconds. The same evaluation set and latency dashboard gated every step.

Part 2 of 5

Question

Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.

2.1Target questions

Keep them in mind while you read the detail.

  1. Your LLM feature is too slow. Where do you start?
  2. Why does continuous batching help so much?
  3. Does speculative decoding change outputs?
  4. How do you cut cost by 10× without hurting quality?

Model answers are in part 4, after the detail that supports them.

2.2Interview rounds

System design rounds where this component is part of a strong answer.

Part 3 of 5

Detailed answer

How it actually works: the components, the states it moves through, a full run, and the code.

3.1Where each technique acts

The parts and how they connect.

BEFORE THE MODELINSIDE THE SERVERModel routingsmall vs largeContext trimmingsummaries ·retrievalPrompt cachingreuse prefixesDistillationtask-specific smallmodelSemantic cachesimilar question →answerBatch APIsoffline, cheaperCont. batchingmany decodes perstepPaged KV cacheno wasted memoryFlashAttentionIO-aware kernelsQuantisationFP8 · INT8 · INT4Speculativedraft + verifyParallelismtensor · pipeline
ServiceData storeAI component
Boxes are components; arrows show calls and data flow.
Routing and distillation
The cheapest token is one processed by a smaller model. Classify requests and send routine ones to a small model, often distilled from the large one on your task.
Context trimming and caching
Input tokens dominate cost for retrieval-heavy apps. Summarise history, retrieve only relevant chunks, and keep stable prefixes identical so provider prompt caching or engine prefix caching applies.
Continuous batching
Instead of waiting for a batch to finish, the server adds and removes requests every decode step, keeping the GPU full.
Paged KV cache
vLLM’s PagedAttention stores KV cache in fixed-size blocks, like virtual memory, so long and short requests share memory without fragmentation.
FlashAttention
Exact attention computed in tiles that fit in fast on-chip memory, reducing slow memory reads; standard in modern engines.
Quantisation
Store weights (and optionally KV cache) in 8 or 4 bits. FP8 on recent GPUs keeps quality close to FP16; INT4 methods such as AWQ and GPTQ shrink further with more risk.
Speculative decoding
A small draft model proposes several tokens; the large model checks them in one forward pass and keeps the accepted prefix. Output quality is unchanged; speed rises when drafts are often right.

3.2Techniques in one line each

NameMeaning
Prefix cachingReuse computed KV for a repeated prompt start, such as a long system prompt.
Semantic cachingReturn a stored answer for a sufficiently similar past question; only for stable, non-personal answers.
Batch APIsAsynchronous bulk processing at a discount from major providers, for work that can wait hours.
Tensor parallelismSplit each layer across GPUs for big models; needs fast interconnect such as NVLink.
Disaggregated servingRun prefill and decode on separate GPU pools; useful at very large scale.

3.3Speculative decoding step

One run, step by step.

SchedulerDraft model (1B)Target model (70B)KV cacheClientpropose next 5 tokenst1 t2 t3 t4 t5verify all 5 in one passread cached K,Vaccept t1–t3, correct t4stream 4 tokensOne expensive pass produced 4 tokens instead of 1
Solid arrows are calls; dashed arrows are responses.
  1. Speed-up depends on the acceptance rate, which is high for predictable text such as code and templated answers.
  2. Because the target model verifies every token, the output matches what it would have produced on its own.

3.4Code

Working code for the core of it.

Serving with the main levers on (vLLM)

serving-main-levers.shShellCopy
1vllm serve meta-llama/Llama-3.1-70B-Instruct \2  --tensor-parallel-size 4 \3  --quantization fp8 --kv-cache-dtype fp8 \4  --enable-prefix-caching \5  --max-num-seqs 256 --gpu-memory-utilization 0.92
5 linesUTF-8Shell

Routing by difficulty (TypeScript)

routing-by-difficulty.tsTypeScriptCopy
1const route = (req: Req) =>2  req.tokens < 800 && classifier.isRoutine(req.intent) ? 'small-8b'3  : req.needsTools || req.tokens > 6000 ? 'large-70b'4  : 'medium';5 6const res = await llm.generate({ model: route(req), messages, maxTokens: 300 });   // cap output too
6 linesUTF-8TypeScript

Keep the cached prefix stable

keep-cached-prefix.tsTypeScriptCopy
1// Same bytes every call → prefix cache hits. Put variable parts at the end.2const messages = [3  { role: 'system', content: SYSTEM_PROMPT_V12 },     // never interpolate dates or names here4  { role: 'system', content: TOOL_GUIDE_V4 },5  ...recentTurns,6  { role: 'user', content: question },7];
7 linesUTF-8TypeScript

3.5Tested code, explained

A working implementation, then what each part does and why.

Static versus continuous batching

One long request shouldn't hold up everyone else. Run both functions on the same jobs, and the difference is plain.

continuous-batching.tsTypeScriptCopy
1export interface Job {2  arrival: number;3  tokens: number;4}5 6const byArrival = (jobs: Job[]) => jobs.map((_, i) => i).sort((a, b) => jobs[a].arrival - jobs[b].arrival);7 8/** Static batching: the whole batch waits for its longest request. Returns each job's finish step. */9export function staticBatching(jobs: Job[], size: number): number[] {10  const done = new Array<number>(jobs.length);11  const queue = byArrival(jobs);12  let t = 0;13  let next = 0;14  while (next < queue.length) {15    t = Math.max(t, jobs[queue[next]].arrival);16    const batch: number[] = [];17    while (batch.length < size && next < queue.length && jobs[queue[next]].arrival <= t) batch.push(queue[next++]);18    t += Math.max(...batch.map((i) => jobs[i].tokens));19    for (const i of batch) done[i] = t;20  }21  return done;22}23 24/** Continuous batching: finished jobs leave after any step, and waiting jobs join at once. */25export function continuousBatching(jobs: Job[], size: number): number[] {26  const done = new Array<number>(jobs.length);27  const queue = byArrival(jobs);28  const left = jobs.map((j) => j.tokens);29  const running = new Set<number>();30  let t = 0;31  let next = 0;32  while (running.size > 0 || next < queue.length) {33    while (running.size < size && next < queue.length && jobs[queue[next]].arrival <= t) running.add(queue[next++]);34    if (running.size === 0) {35      t = jobs[queue[next]].arrival;36      continue;37    }38    t++; // one step: one token for everyone in the batch39    for (const i of [...running]) {40      if (--left[i] === 0) {41        done[i] = t;42        running.delete(i);43      }44    }45  }46  return done;47}
47 linesUTF-8TypeScript
t += Math.max(...)
Static batching: everyone waits for the longest request in the batch.
running.delete(i)
Continuous batching: a finished request leaves after any step.
running.add(queue[next++])
A waiting request joins at the very next step, which is where the throughput gain comes from.

Tested by npm run test:modules.

Accepting a speculative token

A small model guesses tokens, and the big model checks them. This rule keeps the output distribution exactly the big model's.

speculative-accept.tsTypeScriptCopy
1function sample(dist: number[], rand: () => number): number {2  let r = rand();3  for (let i = 0; i < dist.length; i++) {4    r -= dist[i];5    if (r <= 0) return i;6  }7  return dist.length - 1;8}9 10/**11 * One speculative step. The draft proposes x ~ q; the target accepts it with min(1, p(x)/q(x));12 * otherwise it samples from the leftover max(0, p - q). The output follows p exactly.13 */14export function speculativeStep(p: number[], q: number[], rand: () => number = Math.random) {15  const x = sample(q, rand);16  if (rand() < Math.min(1, p[x] / q[x])) return { token: x, accepted: true };17  const residual = p.map((pi, i) => Math.max(0, pi - q[i]));18  const mass = residual.reduce((a, b) => a + b, 0);19  return { token: sample(residual.map((r) => r / mass), rand), accepted: false };20}21 22/** Expected acceptance rate: the overlap of the two distributions. */23export const acceptRate = (p: number[], q: number[]) => p.reduce((sum, pi, i) => sum + Math.min(pi, q[i]), 0);
23 linesUTF-8TypeScript
Math.min(1, p[x] / q[x])
Accept the draft's token in proportion to how much the target agrees with it.
Math.max(0, pi - q[i])
On rejection, sample from what the draft under-weighted. This makes the maths come out exact.
acceptRate
The overlap of the two distributions, which predicts your speed-up.

Tested by npm run test:modules.

Reusing a shared prompt prefix

Many requests start with the same system prompt, so its KV entries can be reused.

prefix-cache.tsTypeScriptCopy
1/** vLLM-style prefix cache: each full block's key chains its parent's hash, its tokens and a salt. */2export class PrefixCache {3  private blocks = new Map<string, true>(); // Map order doubles as LRU order4  private blockTokens: number;5  private capacity: number;6 7  constructor(blockTokens = 16, capacity = 1024) {8    this.blockTokens = blockTokens;9    this.capacity = capacity;10  }11 12  private keys(tokens: number[], salt: string): string[] {13    // At least one token must still run through the model, so never cache the very last one.14    const whole = Math.floor((tokens.length - 1) / this.blockTokens);15    const out: string[] = [];16    let parent = salt;17    for (let b = 0; b < whole; b++) {18      parent = `${parent}|${tokens.slice(b * this.blockTokens, (b + 1) * this.blockTokens).join(",")}`;19      out.push(parent);20    }21    return out;22  }23 24  /** Number of prompt tokens the engine can skip. */25  lookup(tokens: number[], salt = ""): number {26    let hits = 0;27    for (const key of this.keys(tokens, salt)) {28      if (!this.blocks.has(key)) break;29      this.blocks.delete(key); // move to most recently used30      this.blocks.set(key, true);31      hits++;32    }33    return hits * this.blockTokens;34  }35 36  insert(tokens: number[], salt = ""): void {37    for (const key of this.keys(tokens, salt)) {38      this.blocks.delete(key);39      this.blocks.set(key, true);40      if (this.blocks.size > this.capacity) this.blocks.delete(this.blocks.keys().next().value as string);41    }42  }43}
43 linesUTF-8TypeScript
parent = `${parent}|...`
Each block's key includes its parent's, so a match means the whole prefix matches.
salt
Keeps tenants apart, because shared prefixes can otherwise leak through timing.
if (!this.blocks.has(key)) break
Stops at the first miss. Later blocks are useless without the earlier ones.
tokens.length - 1
At least one token must run through the model to produce the next one.

Tested by npm run test:modules.

3.6Where to apply each lever

Hosted APIs

  • Model choice and routing are your biggest levers.
  • Prompt caching and batch APIs reduce cost for repeated prefixes and offline work.
  • Cap max output tokens and ask for concise structures.

Self-hosted GPUs

  • Continuous batching, paged KV cache and prefix caching through vLLM, SGLang or TensorRT-LLM.
  • FP8 quantisation on recent GPUs; tensor parallelism sized by memory.
  • Autoscale on queue depth and KV utilisation.

On device

  • Small models, 4-bit quantisation and hardware delegates (GPU, Neural Engine, NPU).
  • Short prompts and short outputs; cache results locally.
Part 4 of 5

Q&A

The questions from part 2, now with model answers. Try each one out loud before opening it.

4.1Interview questions

Your LLM feature is too slow. Where do you start?
Measure TTFT versus TPOT. Slow first token means long prompts or queueing: trim context, cache prefixes, add capacity. Slow streaming means decode limits: batching, quantisation, speculative decoding or a smaller model.
Why does continuous batching help so much?
Decode is memory-bandwidth bound; reading the weights once to produce tokens for many requests amortises that cost.
Does speculative decoding change outputs?
No. The target model verifies every proposed token, so the result matches what it would have generated alone; only speed changes.
How do you cut cost by 10× without hurting quality?
Route most traffic to a small or distilled model, trim and cache context, cap outputs, batch offline work, and prove quality on an evaluation set at every step.
Part 5 of 5

Extra depth

Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.

5.1Engines and tools

Show detailsHide details
NeedOptionsWhen to pick
Serving enginesvLLM · SGLang · TensorRT-LLM · Hugging Face TGI · llama.cppvLLM or SGLang for general GPU serving; TensorRT-LLM for maximum NVIDIA performance; llama.cpp for CPU and edge.
QuantisationFP8 · AWQ · GPTQ · GGUF (llama.cpp)FP8 first on supported GPUs; INT4 when memory is the hard limit.
KernelsFlashAttention · fused kernels in enginesUsually provided by the engine; check versions match your GPU.
Distillation and fine-tuningLoRA on a small model using large-model outputsWhen a narrow task is high volume and quality can be measured.

5.2Measure before and after

Show detailsHide details
MetricWhy it matters
TTFT and TPOT, p50 and p95User-visible speed.
Tokens per second per GPUEfficiency of batching and kernels.
Cost per request and per successful taskThe business number; includes retries.
Cache hit rates (prefix, semantic)Show whether prompt structure is cache-friendly.
Quality score on the eval setThe guardrail for every optimisation.

5.3Trade-offs to watch

Show detailsHide details
CaseHow it is handled
Quantisation hurts a niche skillEvaluate per task, not on generic benchmarks; keep a higher-precision fallback.
Semantic cache serves a stale or personal answerCache only public, stable answers with TTLs; include version in the key.
Bigger batches raise latencyCap batch size or tokens per step to protect p95.
Router misroutes hard requestsEscalate when the small model’s confidence or validation fails.