Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
vLLM runs one loop: schedule, allocate, execute, free. Each pass generates one token for every running request. A scheduler picks the requests, a block manager gives them KV cache memory in small fixed blocks, and a worker runs one batched forward pass. Memory, not compute, usually caps concurrency, so vLLM spends its cleverness there.
PagedAttention treats the KV cache like virtual memory, so one GPU holds 2–4× more concurrent requests.
1.2Key concepts
One idea per card, in plain words.
Stores attention keys and values for every token so far, per layer. It grows with each token and usually fills GPU memory before compute runs out.
Older engines reserve max_tokens up front for each request. The vLLM paper measured only 20–38% of that memory holding real token state.
Splits the KV cache into fixed blocks of 16 tokens. A per-sequence block table maps logical blocks to any free physical block, like an OS page table.
Adds and removes requests between token steps, so a batch never waits for its slowest member.
Lets parallel samples of one prompt share blocks. A sample copies a block only when it writes into a shared one.
Frees memory when blocks run out: the engine stops the newest request, then recomputes or swaps its cache back later.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Your inference fleet returns 429s at 40% GPU utilisation, and nvidia-smi shows memory full. Explain what limits throughput, and how an engine like vLLM changes it.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Does 'memory full' mean out of memory? | No. vLLM claims its pool up front (90% by default), so nvidia-smi always reads full. Check the engine's KV cache usage metric instead. |
| What load shape? | 200 requests per second at peak, with about 2,000 input and 300 output tokens each. |
| Can the model change? | No. Only the serving layer may change. |
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Queue the request
The API server tokenises the prompt and adds it to a waiting queue. No GPU memory moves yet.
- Schedule every step
Before each forward pass the scheduler walks the running list, gives a request a new block if its last one filled, then admits waiting requests while blocks remain. The default policy serves first come, first served.
- Preempt on shortage
No free block? The scheduler evicts the most recently admitted request, then recomputes it later or swaps its blocks to CPU memory. Frequent preemption signals overload, so alert on it.
- Execute one batched step
The worker flattens all running requests into one batch and passes the block tables alongside. The attention kernel reads keys and values block by block, so physical blocks need not sit together.
- Stream and free
Each new token streams back. A finished request returns its blocks at once, and the next step admits someone new.
3.2Size the KV cache
Per token: 2 × layers × KV heads × head dimension × bytes per value. The example uses FP16 and 2,300 tokens per request.
| Model | Layers | KV heads | Head dim | Per token | Per request |
|---|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 128 | 128 KiB | 288 MiB |
| Llama 3.1 70B | 80 | 8 | 128 | 320 KiB | 719 MiB |
3.3Code
Working code for the core of it.
How much memory one token costs
This arithmetic sets how many requests a GPU can serve at once. Weights are fixed, and the KV cache grows with every token.
1/** Bytes of KV cache one token costs: a key and a value, per layer, per KV head. */2export function kvBytesPerToken(m: { layers: number; kvHeads: number; headDim: number; bytesPerValue: number }): number {3 return 2 * m.layers * m.kvHeads * m.headDim * m.bytesPerValue;4}5 6/** Requests that fit in the memory left after loading the weights. */7export function concurrentRequests(freeGiB: number, tokensPerRequest: number, bytesPerToken: number): number {8 return Math.floor((freeGiB * 1024 ** 3) / (tokensPerRequest * bytesPerToken));9}2 * m.layers * ...- A key and a value, for each layer, each head and each dimension.
freeGiB- What's left after loading the weights. Divide by the cost of a whole request, and you have your concurrency limit.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why does PagedAttention raise throughput but not per-token speed?
Why fixed blocks of 16 tokens?
Recompute or swap after preemption?
What do you monitor?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Max model length
- Set it as low as the product allows. Long contexts dominate memory, and vLLM reports the maximum concurrency at full length when it starts.
- Prefill stalls
- A long prompt slows the next token for everyone in the batch. Chunked prefill splits prompts across steps.
- Counterpoint
- Paged blocks add an indirection to every attention read. When request lengths barely vary and memory has room, simpler engines match vLLM. Measure on your workload before switching.