Basics
Start with the product and the ideas it depends on. If these are clear, the design that follows will feel obvious rather than memorised.
1.1Product in plain words
Who uses it, what a normal day looks like, and why it is harder than it seems.
Running an open-source language model on the company’s own GPUs instead of paying an AI provider per request, for cost and data-residency reasons. It serves the assistant and internal tools at about 200 requests per second at peak.
Who uses it
- Product teams: call one internal API and expect fast, reliable streaming.
- The platform team: operates GPUs, models and capacity.
- Finance: wants cost per request lower than hosted APIs.
A day in the life
- An assistant request arrives at the internal gateway.
- The router sends it to the replica with the most free memory.
- The GPU reads the prompt and starts streaming words back within a second.
- Many other requests share the same GPUs at the same moment through batching.
- When traffic rises at lunchtime, a warm spare replica joins before users notice.
Why it is hard
70 billion parameters do not fit on one GPU at full precision.
Each active conversation holds a cache in GPU memory.
Loading a model onto GPUs takes minutes, so scaling must be proactive.
Idle GPUs burn money; overloaded ones hurt users.
Like a busy restaurant kitchen where each chef cooks many orders at once, and the limit is not the chefs but counter space for orders in progress.
1.2Key concepts
The building blocks this design uses. Each one appears again in the answer.
Prefill reads the whole prompt at once and is limited by compute; decode generates one token at a time and is limited by memory bandwidth. They scale differently.
While generating, the model keeps attention keys and values for every token so far. Its size, not the weights, usually limits how many requests fit on a GPU.
Requests join and leave a running batch token by token, so the GPU stays busy instead of waiting for the slowest request.
A 70B model does not fit on one GPU, so each layer's matrices are split across several GPUs that work in lockstep.
Running weights in FP8 or INT4 halves or quarters memory and speeds up decode, at a small, measurable quality cost.
Time to first token (how fast the answer starts) and tokens per second (how fast it streams) matter more than average request time.
Learn them properly in the deep dives:
Question
Now read the question closely. Agree on what you are building, ask what is unclear, and work out the numbers before you draw anything.
2.1Brief and scope
The question exactly as the interviewer asks it, and what they want to learn from your answer.
For cost and data-residency reasons we will self-host a 70-billion-parameter open model for our assistant and internal tools. Expected load is 200 requests per second at peak, with about 2,000 input and 300 output tokens each. Design the serving platform on GPUs.
What it tests: GPU memory arithmetic, engine concepts (continuous batching, KV cache, parallelism), latency metrics, slow-start autoscaling and cost.
The answer in one sentence. An inference engine with continuous batching and paged KV cache, replicas across several GPUs each, routing by free memory, and warm-pool autoscaling.
How to spend the 45 minutes
- Clarify~5 minAsk questions, agree on scope and write down assumptions.
- Estimate~5 minRough numbers for users, data and traffic, and what they imply.
- High-level design~11 minComponents, data flow and the key API on the board.
- Deep dive~16 minThe two or three hardest parts, with real mechanisms.
- Trade-offs and wrap-up~9 minDecisions, failure modes, and what you would do next.
2.2Clarify the problem
The first five minutes. Each answer removes a guess from your design.
| You ask | Interviewer answers |
|---|---|
| Latency targets? | Time to first token under 1 second at p95; at least 30 tokens per second per stream. |
| Which GPUs are available? | 80 GB H100-class GPUs, 8 per node. |
| Is some traffic batch or offline? | Yes, nightly document processing can use spare capacity. |
| Quality tolerance for quantisation? | Acceptable if evaluation scores drop less than 1%. |
2.3Estimate the scale
Say the arithmetic out loud. What matters is the order of magnitude and what it forces you to do.
| Metric | Estimate | How |
|---|---|---|
| Weights at FP16 | ≈ 140 GB | 70B × 2 bytes; needs at least 2 GPUs |
| Weights at FP8 | ≈ 70 GB | fits on 1 GPU but leaves little room for KV cache |
| KV cache per token | ≈ 0.3 MB at FP16 | 2 × 80 layers × 8 KV heads × 128 dims × 2 bytes |
| KV cache per request | ≈ 0.7 GB | 2,300 tokens × 0.3 MB |
| Output tokens per second | ≈ 60k tok/s at peak | 200 req/s × 300 tokens |
| Input tokens per second | ≈ 400k tok/s | prefill is compute-heavy |
Detailed answer
The design itself, from the big picture down to the code: components first, then one request end to end, then the parts interviewers push hardest on.
3.1Architecture
A gateway routes requests to model replicas. Each replica is an inference engine spanning a few GPUs with tensor parallelism; an autoscaler keeps warm capacity ahead of demand.
- Inference engine
- vLLM (or TensorRT-LLM, SGLang) provides continuous batching, which adds new requests to the running batch every step, and PagedAttention, which stores the KV cache in pages so memory is not wasted on padding.
- Tensor parallelism
- Each replica splits the model across 4 GPUs with tensor parallelism over NVLink. That leaves most memory for KV cache, which is what limits how many requests a replica serves at once.
- Router
- Sends requests to the replica with the most free KV cache, and keeps conversations with the same long system prompt together to benefit from prefix caching.
- Autoscaler
- Scales on queue depth and KV cache utilisation, not CPU. Loading 70 GB of weights takes minutes, so it keeps a warm pool and scales ahead of known peaks.
- Small model pool
- Routine requests go to an 8B model at a fraction of the cost; the gateway decides by task type.
- Batch traffic
- Nightly jobs use spare capacity at low priority and are pre-empted when interactive traffic rises.
3.2Key flow, step by step
A request through the platform.
- Prefill (reading the prompt) is compute-bound and sets time to first token; decode (writing tokens) is memory-bandwidth-bound and sets tokens per second.
- Continuous batching keeps GPUs busy without making short requests wait for long ones.
- The resource that actually runs out drives scaling: KV cache memory.
3.3Front end
The users of this platform are other engineers and their apps. Its front end is the client SDK that apps use to stream from it reliably, and the operations console the platform team uses to see and control the fleet.
Screen map
- Client SDK
- Wraps streaming with timeouts, cancellation, retry on 429 and 503 with Retry-After, and returns usage with every response.
- Playground
- Lets teams try prompts against each model and see tokens, latency and cost before writing code.
- Fleet dashboard
- Live charts of time to first token, time per output token, KV cache use and queue depth per model and replica, with p95 lines.
- Rollout control
- Moves a new engine or model version from canary to full fleet only when evaluation and latency gates pass.
Client layers
| Layer | Technology | Responsibility |
|---|---|---|
| Client SDK | TypeScript and Kotlin libraries | Streaming, retries, cancellation, usage |
| Console UI | React + charting library | Fleet, usage, rollouts |
| Live data | WebSocket or server-sent events from metrics API | Update charts every few seconds |
| Auth | SSO with roles | Only platform engineers can roll out models |
Every state the user can see
| State | What the user sees | How it works |
|---|---|---|
| Healthy | Green status, p95 under target | All SLOs met |
| Degraded | Amber with the breaching metric | TTFT or error rate above SLO |
| Scaling | ‘Adding 2 replicas’ progress | Autoscaler action in progress |
| Canary | Split view: canary vs stable | New version on a fraction of traffic |
| Rate limited (client) | Retry after N seconds | 429 from the gateway |
Code
Client SDK: stream with cancel and polite retry (TypeScript)
1export async function* streamChat(req: ChatRequest, opts: { signal?: AbortSignal; maxRetries?: number } = {}) {2 for (let attempt = 0; ; attempt++) {3 const res = await fetch(GATEWAY + '/v1/chat', { method: 'POST', body: JSON.stringify({ ...req, stream: true }), signal: opts.signal });4 if ((res.status === 429 || res.status === 503) && attempt < (opts.maxRetries ?? 2)) {5 const wait = Number(res.headers.get('Retry-After') ?? 1) * 1000;6 await sleep(wait * (0.5 + Math.random())); // jittered, server-paced7 continue;8 }9 if (!res.ok) throw new LlmError(res.status, await res.text());10 yield* parseSse(res.body!); // tokens, then a usage event11 return;12 }13}Live fleet chart subscription (React)
1function useFleetMetrics(model: string) {2 const [points, setPoints] = useState<MetricPoint[]>([]);3 useEffect(() => {4 const es = new EventSource('/ops/metrics?model=' + encodeURIComponent(model));5 es.onmessage = e => setPoints(p => [...p.slice(-299), JSON.parse(e.data)]); // keep last 300 points6 return () => es.close();7 }, [model]);8 return points;9}3.4Data model and APIs
The tables and contracts that make the design concrete.
Capacity estimate
1Per replica: 4 × 80 GB = 320 GB2 weights FP8 ≈ 70 GB3 activations/overhead ≈ 30 GB4 KV cache budget ≈ 220 GB (FP8 KV ≈ 0.15 MB/token)5 → ≈ 1.4M cached tokens ≈ 600 concurrent requests of 2,300 tokens6Measured decode throughput per replica (benchmark!) ≈ 4–6k tok/s7Peak need 60k tok/s → 10–15 replicas → 40–60 GPUs, plus warm poolEngine launch
1vllm serve meta-llama/Llama-3.1-70B-Instruct \2 --tensor-parallel-size 4 --quantization fp8 --kv-cache-dtype fp8 \3 --max-model-len 8192 --enable-prefix-caching --gpu-memory-utilization 0.923.5Tested code, explained
The core of this design as working code, with what each part does and why.
How much memory one token costs
This arithmetic sets how many requests a GPU can serve at once. Weights are fixed, and the KV cache grows with every token.
1/** Bytes of KV cache one token costs: a key and a value, per layer, per KV head. */2export function kvBytesPerToken(m: { layers: number; kvHeads: number; headDim: number; bytesPerValue: number }): number {3 return 2 * m.layers * m.kvHeads * m.headDim * m.bytesPerValue;4}5 6/** Requests that fit in the memory left after loading the weights. */7export function concurrentRequests(freeGiB: number, tokensPerRequest: number, bytesPerToken: number): number {8 return Math.floor((freeGiB * 1024 ** 3) / (tokensPerRequest * bytesPerToken));9}2 * m.layers * ...- A key and a value, for each layer, each head and each dimension.
freeGiB- What's left after loading the weights. Divide by the cost of a whole request, and you have your concurrency limit.
Tested by npm run test:modules.
Static versus continuous batching
One long request shouldn't hold up everyone else. Run both functions on the same jobs, and the difference is plain.
1export interface Job {2 arrival: number;3 tokens: number;4}5 6const byArrival = (jobs: Job[]) => jobs.map((_, i) => i).sort((a, b) => jobs[a].arrival - jobs[b].arrival);7 8/** Static batching: the whole batch waits for its longest request. Returns each job's finish step. */9export function staticBatching(jobs: Job[], size: number): number[] {10 const done = new Array<number>(jobs.length);11 const queue = byArrival(jobs);12 let t = 0;13 let next = 0;14 while (next < queue.length) {15 t = Math.max(t, jobs[queue[next]].arrival);16 const batch: number[] = [];17 while (batch.length < size && next < queue.length && jobs[queue[next]].arrival <= t) batch.push(queue[next++]);18 t += Math.max(...batch.map((i) => jobs[i].tokens));19 for (const i of batch) done[i] = t;20 }21 return done;22}23 24/** Continuous batching: finished jobs leave after any step, and waiting jobs join at once. */25export function continuousBatching(jobs: Job[], size: number): number[] {26 const done = new Array<number>(jobs.length);27 const queue = byArrival(jobs);28 const left = jobs.map((j) => j.tokens);29 const running = new Set<number>();30 let t = 0;31 let next = 0;32 while (running.size > 0 || next < queue.length) {33 while (running.size < size && next < queue.length && jobs[queue[next]].arrival <= t) running.add(queue[next++]);34 if (running.size === 0) {35 t = jobs[queue[next]].arrival;36 continue;37 }38 t++; // one step: one token for everyone in the batch39 for (const i of [...running]) {40 if (--left[i] === 0) {41 done[i] = t;42 running.delete(i);43 }44 }45 }46 return done;47}t += Math.max(...)- Static batching: everyone waits for the longest request in the batch.
running.delete(i)- Continuous batching: a finished request leaves after any step.
running.add(queue[next++])- A waiting request joins at the very next step, which is where the throughput gain comes from.
Tested by npm run test:modules.
Reusing a shared prompt prefix
Many requests start with the same system prompt, so its KV entries can be reused.
1/** vLLM-style prefix cache: each full block's key chains its parent's hash, its tokens and a salt. */2export class PrefixCache {3 private blocks = new Map<string, true>(); // Map order doubles as LRU order4 private blockTokens: number;5 private capacity: number;6 7 constructor(blockTokens = 16, capacity = 1024) {8 this.blockTokens = blockTokens;9 this.capacity = capacity;10 }11 12 private keys(tokens: number[], salt: string): string[] {13 // At least one token must still run through the model, so never cache the very last one.14 const whole = Math.floor((tokens.length - 1) / this.blockTokens);15 const out: string[] = [];16 let parent = salt;17 for (let b = 0; b < whole; b++) {18 parent = `${parent}|${tokens.slice(b * this.blockTokens, (b + 1) * this.blockTokens).join(",")}`;19 out.push(parent);20 }21 return out;22 }23 24 /** Number of prompt tokens the engine can skip. */25 lookup(tokens: number[], salt = ""): number {26 let hits = 0;27 for (const key of this.keys(tokens, salt)) {28 if (!this.blocks.has(key)) break;29 this.blocks.delete(key); // move to most recently used30 this.blocks.set(key, true);31 hits++;32 }33 return hits * this.blockTokens;34 }35 36 insert(tokens: number[], salt = ""): void {37 for (const key of this.keys(tokens, salt)) {38 this.blocks.delete(key);39 this.blocks.set(key, true);40 if (this.blocks.size > this.capacity) this.blocks.delete(this.blocks.keys().next().value as string);41 }42 }43}parent = `${parent}|...`- Each block's key includes its parent's, so a match means the whole prefix matches.
salt- Keeps tenants apart, because shared prefixes can otherwise leak through timing.
if (!this.blocks.has(key)) break- Stops at the first miss. Later blocks are useless without the earlier ones.
tokens.length - 1- At least one token must run through the model to produce the next one.
Tested by npm run test:modules.
Serving with Ray Serve
This scales replicas with load and restarts unhealthy ones.
1from ray import serve2from starlette.requests import Request3from vllm import LLM, SamplingParams4 5 6@serve.deployment(7 ray_actor_options={"num_gpus": 1},8 autoscaling_config={"min_replicas": 2, "max_replicas": 20, "target_ongoing_requests": 16},9 max_ongoing_requests=32,10 health_check_period_s=10,11)12class Generator:13 def __init__(self, model: str):14 self.llm = LLM(model=model)15 16 def check_health(self) -> None:17 self.llm.generate(["ping"], SamplingParams(max_tokens=1))18 19 async def __call__(self, request: Request) -> dict:20 body = await request.json()21 output = self.llm.generate([body["prompt"]], SamplingParams(max_tokens=body.get("max_tokens", 256)))22 return {"text": output[0].outputs[0].text}23 24 25app = Generator.bind(model="meta-llama/Llama-3.1-8B-Instruct")target_ongoing_requests- Ray adds replicas when each one holds more than this.
max_ongoing_requests- Back-pressure: beyond this, requests queue rather than run out of GPU memory.
check_health- Generates one token, which proves the GPU works, and not merely the process.
Syntax-checked. The GPU libraries it needs don't run in our tests.
3.6Hard parts, in depth
Where interviewers spend the second half of the round.
The numbers that matter
Time to first token (TTFT) and time per output token (TPOT) are the user-facing latencies. Throughput in tokens per second per GPU is the cost metric. Track all three per model and replica; averages hide the tail.
Benchmark with your real prompt lengths. A 2,000-token prompt changes the balance between prefill and decode completely compared with chat benchmarks.
Getting more from each GPU
FP8 weights and KV cache roughly halve memory with small quality loss on current GPUs. Prefix caching avoids recomputing a long shared system prompt. Speculative decoding uses a small draft model to propose tokens the large model verifies in one pass, raising tokens per second.
Separate prefill and decode onto different replicas only at large scale; it adds complexity.
Cold starts and reliability
Keep weights on local NVMe or a fast shared cache so a new replica loads in minutes, not tens of minutes. Health-check with a real generation, not a TCP ping. Drain replicas gracefully: stop new requests, finish streams, then stop.
Q&A
The questions that come after the design, with model answers. Answer each one out loud before opening it.
4.1Follow-up questions
What the interviewer asks once the main design is on the board.
When is self-hosting worth it?
How do you serve many fine-tuned variants?
How do you charge teams internally?
4.2Front-end follow-ups
Questions aimed at the client: rendering, state and devices.
What belongs in the client SDK rather than each app?
What would you show on the main ops screen?
4.3Incidents, probes and traps
Realistic production incidents and pointed probes, grouped by topic.
The product 1
ProbeWhen is self-hosting a bad idea?
Requirements 1
TrapWhich latency numbers do you ask for?
Scale 1
ProbeWalk me through the KV cache maths.
Architecture 2
IncidentLatency spikes every afternoon; GPUs show 95% utilisation and the queue grows.
ProbeWhy route by free KV cache rather than round robin?
The key flow 1
IncidentA few very long prompts make everyone slow.
Front end 2
IncidentClient apps retry aggressively during overload and make it worse.
ProbeWhat does the ops console show first?
Data and APIs 1
TrapWhy store the KV cache in FP8?
The hard parts 2
IncidentA new engine version doubled benchmark throughput but production quality dropped.
ProbeHow do you serve 30 fine-tuned variants?
Trade-offs 1
ProbeTensor parallel 8 or 4?
Failures 2
IncidentA node failed during peak and 300 streams dropped.
IncidentScaling up takes 25 minutes because of cold starts.
Extra depth
What turns a good answer into a strong one: the decisions you rejected, how the system fails, and the signals of seniority.
5.1Trade-offs and decisions
Every choice has a cost. Naming it is what interviewers listen for.
Show detailsHide details
| Decision | Chosen | Rejected | Why |
|---|---|---|---|
| Precision | FP8 | FP16 | Fits more KV cache per GPU; verify the quality drop on your evaluation set. |
| Parallelism | TP=4 per replica | TP=2 or TP=8 | Balances KV headroom and per-replica throughput; benchmark to confirm. |
| Scaling signal | Queue depth + KV use | CPU or GPU utilisation | GPUs look busy even when they could take more; KV memory is the true limit. |
| Model mix | Router to 8B and 70B | 70B for everything | Most requests don’t need the large model. |
5.2Failure modes
What breaks in production, and how the design keeps working when it does.
Show detailsHide details
| What breaks | How the design copes |
|---|---|
| GPU node failure | Replicas spread across nodes; router removes unhealthy replicas within seconds; streams retry on another replica. |
| Traffic spike beyond warm pool | Queue with a short timeout, shed lowest-priority traffic, route overflow to a hosted provider. |
| Long prompts exhaust KV cache | Per-tenant context limits and pre-emption of batch jobs. |
| Quality regression after an engine upgrade | Canary replica with evaluation suite before fleet rollout. |
5.3Performance and accessibility
The client-side details that separate a demo from a product.
Performance
- Clients open the stream immediately and render the first token as soon as it arrives.
- Dashboards downsample to a few hundred points per chart.
- The SDK reuses connections (HTTP/2) to avoid handshake latency per request.
Accessibility and localisation
- Charts have data tables and text summaries for screen readers.
- Status uses icons and words, not colour alone.
5.4Senior versus principal
The same design, answered at two levels. Use the rubric to score yourself.
A senior answer deploys a model on GPUs behind an API.
A principal answer does the memory arithmetic, explains prefill versus decode, continuous batching and KV cache limits, picks parallelism and precision with evidence, scales on the right signal with warm pools, and frames when self-hosting pays off.
Score yourself
5.5Whiteboard
Draw the design here, in this order. It stays in this browser.
- Gateway → router → replicas
- One replica drawn as 4 GPUs with weights and KV cache
- Autoscaler fed by KV use and queue depth
- Small-model pool beside the large one
- Batch queue using spare capacity