vLLM High-Throughput Engine Architecture

PagedAttention treats the KV cache like virtual memory, so one GPU holds 2–4× more concurrent requests.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

vLLM runs one loop: schedule, allocate, execute, free. Each pass generates one token for every running request. A scheduler picks the requests, a block manager gives them KV cache memory in small fixed blocks, and a worker runs one batched forward pass. Memory, not compute, usually caps concurrency, so vLLM spends its cleverness there.

Why it matters

PagedAttention treats the KV cache like virtual memory, so one GPU holds 2–4× more concurrent requests.

1.2Key concepts

One idea per card, in plain words.

01KV cache

Stores attention keys and values for every token so far, per layer. It grows with each token and usually fills GPU memory before compute runs out.

02Internal fragmentation

Older engines reserve max_tokens up front for each request. The vLLM paper measured only 20–38% of that memory holding real token state.

03PagedAttention

Splits the KV cache into fixed blocks of 16 tokens. A per-sequence block table maps logical blocks to any free physical block, like an OS page table.

04Continuous batching

Adds and removes requests between token steps, so a batch never waits for its slowest member.

05Copy-on-write

Lets parallel samples of one prompt share blocks. A sample copies a block only when it writes into a shared one.

06Preemption

Frees memory when blocks run out: the engine stops the newest request, then recomputes or swaps its cache back later.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Your inference fleet returns 429s at 40% GPU utilisation, and nvidia-smi shows memory full. Explain what limits throughput, and how an engine like vLLM changes it.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Does 'memory full' mean out of memory?No. vLLM claims its pool up front (90% by default), so nvidia-smi always reads full. Check the engine's KV cache usage metric instead.
What load shape?200 requests per second at peak, with about 2,000 input and 300 output tokens each.
Can the model change?No. Only the serving layer may change.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Queue the request

    The API server tokenises the prompt and adds it to a waiting queue. No GPU memory moves yet.

  2. Schedule every step

    Before each forward pass the scheduler walks the running list, gives a request a new block if its last one filled, then admits waiting requests while blocks remain. The default policy serves first come, first served.

  3. Preempt on shortage

    No free block? The scheduler evicts the most recently admitted request, then recomputes it later or swaps its blocks to CPU memory. Frequent preemption signals overload, so alert on it.

  4. Execute one batched step

    The worker flattens all running requests into one batch and passes the block tables alongside. The attention kernel reads keys and values block by block, so physical blocks need not sit together.

  5. Stream and free

    Each new token streams back. A finished request returns its blocks at once, and the next step admits someone new.

3.2Size the KV cache

Per token: 2 × layers × KV heads × head dimension × bytes per value. The example uses FP16 and 2,300 tokens per request.

ModelLayersKV headsHead dimPer tokenPer request
Llama 3.1 8B328128128 KiB288 MiB
Llama 3.1 70B808128320 KiB719 MiB

3.3Code

Working code for the core of it.

How much memory one token costs

This arithmetic sets how many requests a GPU can serve at once. Weights are fixed, and the KV cache grows with every token.

kv-size.tsTypeScriptCopy
1/** Bytes of KV cache one token costs: a key and a value, per layer, per KV head. */2export function kvBytesPerToken(m: { layers: number; kvHeads: number; headDim: number; bytesPerValue: number }): number {3  return 2 * m.layers * m.kvHeads * m.headDim * m.bytesPerValue;4}5 6/** Requests that fit in the memory left after loading the weights. */7export function concurrentRequests(freeGiB: number, tokensPerRequest: number, bytesPerToken: number): number {8  return Math.floor((freeGiB * 1024 ** 3) / (tokensPerRequest * bytesPerToken));9}
9 linesUTF-8TypeScript
2 * m.layers * ...
A key and a value, for each layer, each head and each dimension.
freeGiB
What's left after loading the weights. Divide by the cost of a whole request, and you have your concurrency limit.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why does PagedAttention raise throughput but not per-token speed?
It packs more requests into the same memory. Each token still takes one forward step, but the batch grows. The paper reports 2–4× the throughput of TGI and Orca-style engines on the same hardware.
Why fixed blocks of 16 tokens?
Small blocks waste little memory but cost more table lookups; large blocks do the reverse. Sixteen balances the two, and waste stays under one block per sequence.
Recompute or swap after preemption?
Recompute suits short sequences because prefill runs fast. Swap suits long ones, at the cost of CPU memory and transfer time.
What do you monitor?
KV cache usage, waiting requests, preemptions, time to first token and tokens per second. GPU utilisation alone hides most problems.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Max model length
Set it as low as the product allows. Long contexts dominate memory, and vLLM reports the maximum concurrency at full length when it starts.
Prefill stalls
A long prompt slows the next token for everyone in the batch. Chunked prefill splits prompts across steps.
Counterpoint
Paged blocks add an indirection to every attention read. When request lengths barely vary and memory has room, simpler engines match vLLM. Measure on your workload before switching.