Async Zero-Redundancy Optimizer Checkpointing

Saves sharded state in the background, so checkpoints stop stalling GPUs.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

Blocking saves stop every GPU for minutes. Async checkpointing copies state to CPU memory quickly, lets training continue, and writes in the background. A commit marker, written last, separates complete saves from broken ones.

Why it matters

Saves sharded state in the background, so checkpoints stop stalling GPUs.

1.2Key concepts

One idea per card, in plain words.

01Snapshot

Copies state before returning, so training can't change it mid-write.

02Sharded save

Each rank writes its own shard, so no rank needs the whole model in memory.

03Commit marker

Written after all shards. No marker means incomplete.

04Backpressure

A new save waits if the last still runs, so saves never pile up.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Saving a sharded 70B model stalls training for four minutes every hour. Build a checkpointer that blocks only for the copy, survives a crash mid-write, and always loads the newest complete save.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Enough CPU memory for a copy?Yes, but plan for it: one shard per rank.
Shared storage?Yes, with slow, unreliable writes.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Copy the state

    Take a snapshot synchronously. This is the only part that blocks training.

  2. Write in the background

    Queue the write, so training continues at once.

  3. Serialise saves

    Chain writes, so at most one runs at a time.

  4. Commit last

    Write the marker after the shards. In multi-rank runs, rank 0 writes it after all ranks confirm.

  5. Load the newest committed

    On restart, ignore any directory without a marker.

  6. Clean up

    Keep the last few checkpoints, and delete older ones only after a newer commit.

3.2Code

Working code for the core of it.

Saving without stopping training

Writing a large checkpoint can take minutes. Training should only pause for the copy.

async-checkpoint.tsTypeScriptCopy
1export interface Storage {2  write(path: string, data: string): Promise<void>;3  list(): Promise<string[]>;4}5 6/** Copy the state fast, write it in the background, and write the commit marker last. */7export class AsyncCheckpointer<S> {8  private storage: Storage;9  private inFlight: Promise<void> = Promise.resolve();10 11  constructor(storage: Storage) {12    this.storage = storage;13  }14 15  /** Blocks training only for the copy. A second save waits for the first to finish writing. */16  save(step: number, rank: number, state: S): Promise<void> {17    const snapshot = JSON.stringify(state);18    this.inFlight = this.inFlight.then(async () => {19      await this.storage.write(`ckpt-${step}/rank-${rank}.json`, snapshot);20      await this.storage.write(`ckpt-${step}/COMMIT`, "ok"); // absent means incomplete21    });22    return this.inFlight;23  }24 25  /** The newest checkpoint with a commit marker. Half-written ones are ignored. */26  async latest(): Promise<number | null> {27    const files = await this.storage.list();28    const steps = files.filter((f) => f.endsWith("/COMMIT")).map((f) => Number(f.match(/ckpt-(\d+)/)?.[1]));29    return steps.length ? Math.max(...steps) : null;30  }31}
31 linesUTF-8TypeScript
JSON.stringify(state)
The copy. It stands for moving tensors to host memory, and this is the only part that blocks.
this.inFlight.then(...)
Writes one at a time, so a slow disk can't pile up saves.
COMMIT
Written last. A checkpoint without it is half-written and gets ignored on restart.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why not write straight from GPU memory?
Training would have to wait, or the tensors would change mid-write.
What if a save fails?
Log it and alert. Training continues, but your recovery point ages, so alert on checkpoint age.
How do you test it?
Inject failures between writes, then assert that the loader picks the previous checkpoint.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Resharding
Restarting with a different GPU count needs a format that records tensor layout. torch.distributed.checkpoint stores this.
Counterpoint
Small models save in seconds. Async adds failure modes that a short blocking save doesn't have.