Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
Plain data parallelism keeps a full copy of everything on every GPU, which wastes memory. ZeRO removes the redundancy in three stages: first optimiser state, then gradients, then the weights themselves. Each stage saves more memory and communicates more.
Shards optimiser state, gradients and weights across GPUs, trading extra communication for models that otherwise cannot fit.
1.2Key concepts
One idea per card, in plain words.
Shards optimiser state, which is the largest part under Adam.
Also shards gradients, using reduce-scatter instead of all-reduce.
Also shards weights. Each layer gathers its weights just before it runs.
Rebuilds one full checkpoint from the shards when you need it.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Train a 70B model on 8 GPUs with 80 GB each, using mixed-precision Adam. Which ZeRO stage fits? Show the memory, and say what you pay in communication.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Activations and gradient checkpointing? | Enable checkpointing. The table counts weights, gradients and optimiser state only. |
| Can you offload to CPU? | ZeRO-Offload can, but training slows sharply. |
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Count the bytes
With FP16 weights and gradients, and Adam's FP32 master copy plus two moments: 2 + 2 + 12 = 16 bytes per parameter.
- Shard the optimiser
Each GPU updates only its share, then all-gathers the new weights.
- Shard the gradients
Reduce-scatter gives each GPU the summed gradients for its share.
- Shard the weights
Each block all-gathers its parameters for the forward pass and frees them afterwards.
- Save the shards
Each rank writes its shard. A separate step consolidates them for inference.
3.270B model on 8 GPUs
Rough figures: weights, gradients and optimiser state per GPU, in decimal GB.
| Stage | Per GPU | Extra communication |
|---|---|---|
| No ZeRO | 1,120 GB | All-reduce gradients |
| ZeRO-1 | 385 GB | All-gather updated weights |
| ZeRO-2 | 262 GB | Reduce-scatter gradients |
| ZeRO-3 | 140 GB | All-gather weights for every block |
3.3Code
Working code for the core of it.
What each ZeRO stage saves
Each stage shards one more thing across GPUs.
1export type ZeroStage = 0 | 1 | 2 | 3;2 3/**4 * Rough GPU memory per device, in decimal GB, for mixed-precision Adam:5 * 2 bytes of weights, 2 of gradients, 12 of optimiser state (FP32 master copy plus two moments).6 */7export function zeroGbPerGpu(params: number, gpus: number, stage: ZeroStage): number {8 const weights = stage >= 3 ? 2 / gpus : 2;9 const grads = stage >= 2 ? 2 / gpus : 2;10 const optim = stage >= 1 ? 12 / gpus : 12;11 return (params * (weights + grads + optim)) / 1e9;12}stage >= 1- Optimiser state goes first, because it's the largest part at 12 bytes per weight.
stage >= 3- Weights go last, and that's what lets a model larger than one GPU fit at all.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why does ZeRO-1 save so much?
When is ZeRO-3 worth it?
How does it relate to FSDP?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Checkpoint size
- Sharded checkpoints load only onto the same layout unless consolidated or resharded.
- Counterpoint
- ZeRO reduces memory, not training time. Sometimes larger GPUs and ZeRO-1 finish first.