Basics
What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.
1.1Definition
The one-paragraph summary, then where its responsibility starts and stops.
Inference limits come in three kinds: GPU compute during prefill, memory bandwidth during decode, and memory capacity for weights and KV cache. Each acceleration technique relaxes one: smaller numbers, fewer tokens, better batching, or guessing ahead.
- Route easy requests to smaller models
- Send fewer tokens: trim context, cache prefixes
- Quantise weights and KV cache
- Batch continuously and page the KV cache
- Decode speculatively with a draft model
- Distil a big model into a small task model
- Adding GPUs to a decode-bound workload without more batching
- Raising max tokens ‘just in case’
- Optimising before measuring TTFT, TPOT and cost per request
- Quantising without re-running your evaluation set
1.2Key concepts
Ideas this deep dive relies on, in one sentence each.
Stored attention keys and values per token during generation; it limits concurrent requests per GPU.
Serving technique that adds and removes requests from a running batch every step.
Managing the KV cache in blocks to cut memory waste.
Storing weights or activations at lower precision to save memory and speed up inference.
Fine-tuning by training small adapter matrices instead of all weights.
1.3A real optimisation story
A concrete example first, so the mechanics have something to attach to.
A support summariser on a self-hosted 70B model costs too much and p95 latency is 9 seconds. The team measures first: average prompt 6,000 tokens, output 250, GPU utilisation 35% because each replica processes one request at a time.
Step one moves to vLLM with continuous batching and paged KV cache: throughput per GPU roughly triples and p95 drops, because the GPU now decodes many requests per step. Step two trims the prompt: summarising ticket history once and caching it cuts input to 2,500 tokens and halves TTFT.
Step three quantises to FP8 weights and KV cache after the evaluation set shows less than 0.5% quality change, freeing memory for more concurrent requests. Step four routes short tickets to an 8B model distilled on the 70B model’s best summaries; 60% of traffic moves there.
Result: cost per summary falls by a large multiple and p95 latency drops below 3 seconds. The same evaluation set and latency dashboard gated every step.
Question
Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.
2.1Target questions
Keep them in mind while you read the detail.
- Your LLM feature is too slow. Where do you start?
- Why does continuous batching help so much?
- Does speculative decoding change outputs?
- How do you cut cost by 10× without hurting quality?
Model answers are in part 4, after the detail that supports them.
2.2Interview rounds
System design rounds where this component is part of a strong answer.
Detailed answer
How it actually works: the components, the states it moves through, a full run, and the code.
3.1Where each technique acts
The parts and how they connect.
- Routing and distillation
- The cheapest token is one processed by a smaller model. Classify requests and send routine ones to a small model, often distilled from the large one on your task.
- Context trimming and caching
- Input tokens dominate cost for retrieval-heavy apps. Summarise history, retrieve only relevant chunks, and keep stable prefixes identical so provider prompt caching or engine prefix caching applies.
- Continuous batching
- Instead of waiting for a batch to finish, the server adds and removes requests every decode step, keeping the GPU full.
- Paged KV cache
- vLLM’s PagedAttention stores KV cache in fixed-size blocks, like virtual memory, so long and short requests share memory without fragmentation.
- FlashAttention
- Exact attention computed in tiles that fit in fast on-chip memory, reducing slow memory reads; standard in modern engines.
- Quantisation
- Store weights (and optionally KV cache) in 8 or 4 bits. FP8 on recent GPUs keeps quality close to FP16; INT4 methods such as AWQ and GPTQ shrink further with more risk.
- Speculative decoding
- A small draft model proposes several tokens; the large model checks them in one forward pass and keeps the accepted prefix. Output quality is unchanged; speed rises when drafts are often right.
3.2Techniques in one line each
| Name | Meaning |
|---|---|
| Prefix caching | Reuse computed KV for a repeated prompt start, such as a long system prompt. |
| Semantic caching | Return a stored answer for a sufficiently similar past question; only for stable, non-personal answers. |
| Batch APIs | Asynchronous bulk processing at a discount from major providers, for work that can wait hours. |
| Tensor parallelism | Split each layer across GPUs for big models; needs fast interconnect such as NVLink. |
| Disaggregated serving | Run prefill and decode on separate GPU pools; useful at very large scale. |
3.3Speculative decoding step
One run, step by step.
- Speed-up depends on the acceptance rate, which is high for predictable text such as code and templated answers.
- Because the target model verifies every token, the output matches what it would have produced on its own.
3.4Code
Working code for the core of it.
Serving with the main levers on (vLLM)
1vllm serve meta-llama/Llama-3.1-70B-Instruct \2 --tensor-parallel-size 4 \3 --quantization fp8 --kv-cache-dtype fp8 \4 --enable-prefix-caching \5 --max-num-seqs 256 --gpu-memory-utilization 0.92Routing by difficulty (TypeScript)
1const route = (req: Req) =>2 req.tokens < 800 && classifier.isRoutine(req.intent) ? 'small-8b'3 : req.needsTools || req.tokens > 6000 ? 'large-70b'4 : 'medium';5 6const res = await llm.generate({ model: route(req), messages, maxTokens: 300 }); // cap output tooKeep the cached prefix stable
1// Same bytes every call → prefix cache hits. Put variable parts at the end.2const messages = [3 { role: 'system', content: SYSTEM_PROMPT_V12 }, // never interpolate dates or names here4 { role: 'system', content: TOOL_GUIDE_V4 },5 ...recentTurns,6 { role: 'user', content: question },7];3.5Tested code, explained
A working implementation, then what each part does and why.
Static versus continuous batching
One long request shouldn't hold up everyone else. Run both functions on the same jobs, and the difference is plain.
1export interface Job {2 arrival: number;3 tokens: number;4}5 6const byArrival = (jobs: Job[]) => jobs.map((_, i) => i).sort((a, b) => jobs[a].arrival - jobs[b].arrival);7 8/** Static batching: the whole batch waits for its longest request. Returns each job's finish step. */9export function staticBatching(jobs: Job[], size: number): number[] {10 const done = new Array<number>(jobs.length);11 const queue = byArrival(jobs);12 let t = 0;13 let next = 0;14 while (next < queue.length) {15 t = Math.max(t, jobs[queue[next]].arrival);16 const batch: number[] = [];17 while (batch.length < size && next < queue.length && jobs[queue[next]].arrival <= t) batch.push(queue[next++]);18 t += Math.max(...batch.map((i) => jobs[i].tokens));19 for (const i of batch) done[i] = t;20 }21 return done;22}23 24/** Continuous batching: finished jobs leave after any step, and waiting jobs join at once. */25export function continuousBatching(jobs: Job[], size: number): number[] {26 const done = new Array<number>(jobs.length);27 const queue = byArrival(jobs);28 const left = jobs.map((j) => j.tokens);29 const running = new Set<number>();30 let t = 0;31 let next = 0;32 while (running.size > 0 || next < queue.length) {33 while (running.size < size && next < queue.length && jobs[queue[next]].arrival <= t) running.add(queue[next++]);34 if (running.size === 0) {35 t = jobs[queue[next]].arrival;36 continue;37 }38 t++; // one step: one token for everyone in the batch39 for (const i of [...running]) {40 if (--left[i] === 0) {41 done[i] = t;42 running.delete(i);43 }44 }45 }46 return done;47}t += Math.max(...)- Static batching: everyone waits for the longest request in the batch.
running.delete(i)- Continuous batching: a finished request leaves after any step.
running.add(queue[next++])- A waiting request joins at the very next step, which is where the throughput gain comes from.
Tested by npm run test:modules.
Accepting a speculative token
A small model guesses tokens, and the big model checks them. This rule keeps the output distribution exactly the big model's.
1function sample(dist: number[], rand: () => number): number {2 let r = rand();3 for (let i = 0; i < dist.length; i++) {4 r -= dist[i];5 if (r <= 0) return i;6 }7 return dist.length - 1;8}9 10/**11 * One speculative step. The draft proposes x ~ q; the target accepts it with min(1, p(x)/q(x));12 * otherwise it samples from the leftover max(0, p - q). The output follows p exactly.13 */14export function speculativeStep(p: number[], q: number[], rand: () => number = Math.random) {15 const x = sample(q, rand);16 if (rand() < Math.min(1, p[x] / q[x])) return { token: x, accepted: true };17 const residual = p.map((pi, i) => Math.max(0, pi - q[i]));18 const mass = residual.reduce((a, b) => a + b, 0);19 return { token: sample(residual.map((r) => r / mass), rand), accepted: false };20}21 22/** Expected acceptance rate: the overlap of the two distributions. */23export const acceptRate = (p: number[], q: number[]) => p.reduce((sum, pi, i) => sum + Math.min(pi, q[i]), 0);Math.min(1, p[x] / q[x])- Accept the draft's token in proportion to how much the target agrees with it.
Math.max(0, pi - q[i])- On rejection, sample from what the draft under-weighted. This makes the maths come out exact.
acceptRate- The overlap of the two distributions, which predicts your speed-up.
Tested by npm run test:modules.
Reusing a shared prompt prefix
Many requests start with the same system prompt, so its KV entries can be reused.
1/** vLLM-style prefix cache: each full block's key chains its parent's hash, its tokens and a salt. */2export class PrefixCache {3 private blocks = new Map<string, true>(); // Map order doubles as LRU order4 private blockTokens: number;5 private capacity: number;6 7 constructor(blockTokens = 16, capacity = 1024) {8 this.blockTokens = blockTokens;9 this.capacity = capacity;10 }11 12 private keys(tokens: number[], salt: string): string[] {13 // At least one token must still run through the model, so never cache the very last one.14 const whole = Math.floor((tokens.length - 1) / this.blockTokens);15 const out: string[] = [];16 let parent = salt;17 for (let b = 0; b < whole; b++) {18 parent = `${parent}|${tokens.slice(b * this.blockTokens, (b + 1) * this.blockTokens).join(",")}`;19 out.push(parent);20 }21 return out;22 }23 24 /** Number of prompt tokens the engine can skip. */25 lookup(tokens: number[], salt = ""): number {26 let hits = 0;27 for (const key of this.keys(tokens, salt)) {28 if (!this.blocks.has(key)) break;29 this.blocks.delete(key); // move to most recently used30 this.blocks.set(key, true);31 hits++;32 }33 return hits * this.blockTokens;34 }35 36 insert(tokens: number[], salt = ""): void {37 for (const key of this.keys(tokens, salt)) {38 this.blocks.delete(key);39 this.blocks.set(key, true);40 if (this.blocks.size > this.capacity) this.blocks.delete(this.blocks.keys().next().value as string);41 }42 }43}parent = `${parent}|...`- Each block's key includes its parent's, so a match means the whole prefix matches.
salt- Keeps tenants apart, because shared prefixes can otherwise leak through timing.
if (!this.blocks.has(key)) break- Stops at the first miss. Later blocks are useless without the earlier ones.
tokens.length - 1- At least one token must run through the model to produce the next one.
Tested by npm run test:modules.
3.6Where to apply each lever
Hosted APIs
- Model choice and routing are your biggest levers.
- Prompt caching and batch APIs reduce cost for repeated prefixes and offline work.
- Cap max output tokens and ask for concise structures.
Self-hosted GPUs
- Continuous batching, paged KV cache and prefix caching through vLLM, SGLang or TensorRT-LLM.
- FP8 quantisation on recent GPUs; tensor parallelism sized by memory.
- Autoscale on queue depth and KV utilisation.
On device
- Small models, 4-bit quantisation and hardware delegates (GPU, Neural Engine, NPU).
- Short prompts and short outputs; cache results locally.
Q&A
The questions from part 2, now with model answers. Try each one out loud before opening it.
4.1Interview questions
Your LLM feature is too slow. Where do you start?
Why does continuous batching help so much?
Does speculative decoding change outputs?
How do you cut cost by 10× without hurting quality?
Extra depth
Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.
5.1Engines and tools
Show detailsHide details
| Need | Options | When to pick |
|---|---|---|
| Serving engines | vLLM · SGLang · TensorRT-LLM · Hugging Face TGI · llama.cpp | vLLM or SGLang for general GPU serving; TensorRT-LLM for maximum NVIDIA performance; llama.cpp for CPU and edge. |
| Quantisation | FP8 · AWQ · GPTQ · GGUF (llama.cpp) | FP8 first on supported GPUs; INT4 when memory is the hard limit. |
| Kernels | FlashAttention · fused kernels in engines | Usually provided by the engine; check versions match your GPU. |
| Distillation and fine-tuning | LoRA on a small model using large-model outputs | When a narrow task is high volume and quality can be measured. |
5.2Measure before and after
Show detailsHide details
| Metric | Why it matters |
|---|---|
| TTFT and TPOT, p50 and p95 | User-visible speed. |
| Tokens per second per GPU | Efficiency of batching and kernels. |
| Cost per request and per successful task | The business number; includes retries. |
| Cache hit rates (prefix, semantic) | Show whether prompt structure is cache-friendly. |
| Quality score on the eval set | The guardrail for every optimisation. |
5.3Trade-offs to watch
Show detailsHide details
| Case | How it is handled |
|---|---|
| Quantisation hurts a niche skill | Evaluate per task, not on generic benchmarks; keep a higher-precision fallback. |
| Semantic cache serves a stale or personal answer | Cache only public, stable answers with TTLs; include version in the key. |
| Bigger batches raise latency | Cap batch size or tokens per step to protect p95. |
| Router misroutes hard requests | Escalate when the small model’s confidence or validation fails. |