DeepSpeed ZeRO-3 Memory Partitioning

Shards optimiser state, gradients and weights across GPUs, trading extra communication for models that otherwise cannot fit.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

Plain data parallelism keeps a full copy of everything on every GPU, which wastes memory. ZeRO removes the redundancy in three stages: first optimiser state, then gradients, then the weights themselves. Each stage saves more memory and communicates more.

Why it matters

Shards optimiser state, gradients and weights across GPUs, trading extra communication for models that otherwise cannot fit.

1.2Key concepts

One idea per card, in plain words.

01ZeRO-1

Shards optimiser state, which is the largest part under Adam.

02ZeRO-2

Also shards gradients, using reduce-scatter instead of all-reduce.

03ZeRO-3

Also shards weights. Each layer gathers its weights just before it runs.

04Consolidation

Rebuilds one full checkpoint from the shards when you need it.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Train a 70B model on 8 GPUs with 80 GB each, using mixed-precision Adam. Which ZeRO stage fits? Show the memory, and say what you pay in communication.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Activations and gradient checkpointing?Enable checkpointing. The table counts weights, gradients and optimiser state only.
Can you offload to CPU?ZeRO-Offload can, but training slows sharply.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Count the bytes

    With FP16 weights and gradients, and Adam's FP32 master copy plus two moments: 2 + 2 + 12 = 16 bytes per parameter.

  2. Shard the optimiser

    Each GPU updates only its share, then all-gathers the new weights.

  3. Shard the gradients

    Reduce-scatter gives each GPU the summed gradients for its share.

  4. Shard the weights

    Each block all-gathers its parameters for the forward pass and frees them afterwards.

  5. Save the shards

    Each rank writes its shard. A separate step consolidates them for inference.

3.270B model on 8 GPUs

Rough figures: weights, gradients and optimiser state per GPU, in decimal GB.

StagePer GPUExtra communication
No ZeRO1,120 GBAll-reduce gradients
ZeRO-1385 GBAll-gather updated weights
ZeRO-2262 GBReduce-scatter gradients
ZeRO-3140 GBAll-gather weights for every block

3.3Code

Working code for the core of it.

What each ZeRO stage saves

Each stage shards one more thing across GPUs.

zero-memory.tsTypeScriptCopy
1export type ZeroStage = 0 | 1 | 2 | 3;2 3/**4 * Rough GPU memory per device, in decimal GB, for mixed-precision Adam:5 * 2 bytes of weights, 2 of gradients, 12 of optimiser state (FP32 master copy plus two moments).6 */7export function zeroGbPerGpu(params: number, gpus: number, stage: ZeroStage): number {8  const weights = stage >= 3 ? 2 / gpus : 2;9  const grads = stage >= 2 ? 2 / gpus : 2;10  const optim = stage >= 1 ? 12 / gpus : 12;11  return (params * (weights + grads + optim)) / 1e9;12}
12 linesUTF-8TypeScript
stage >= 1
Optimiser state goes first, because it's the largest part at 12 bytes per weight.
stage >= 3
Weights go last, and that's what lets a model larger than one GPU fit at all.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why does ZeRO-1 save so much?
Adam's state outweighs weights and gradients combined, so sharding it alone frees most memory.
When is ZeRO-3 worth it?
When weights themselves don't fit. Below that, ZeRO-2 runs faster.
How does it relate to FSDP?
FSDP is PyTorch's native take on the same idea, roughly ZeRO-3.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Checkpoint size
Sharded checkpoints load only onto the same layout unless consolidated or resharded.
Counterpoint
ZeRO reduces memory, not training time. Sometimes larger GPUs and ZeRO-1 finish first.