Offline outbox with idempotent sync

Writes persist first, sync later, survive kills, and are never applied twice.

Part 1 of 5

Basics

The concepts this component is built on. With these in mind, every line of the solution has a reason.

1.1Key concepts

One idea per card, in the order the solution uses them.

01Offline-first writes

The app saves a change locally and returns at once. Sending it to the server happens later, in the background.

02Outbox queue

A durable, ordered list of changes waiting to be sent. It survives app kills because it lives in the database, not in memory.

03Idempotency key

A unique id per change. The server remembers the ids it has applied, so a retried send is answered with 'duplicate' instead of being applied twice.

04Retryable vs permanent errors

Timeouts and 5xx responses may succeed later; validation errors never will. Retrying the second kind only blocks the queue.

Part 2 of 5

Question

What exactly you are asked to build, what must be true when you are done, and what to ask before you start.

2.1Prompt

The question as the interviewer asks it, and the requirements to check your solution against.

Interviewer

Field technicians edit jobs with no signal. Implement the client side of the outbox: changes must never be lost, must reach the server in order, and must not be applied twice when we retry.

Requirements

  • Enqueue persists before returning
  • Drain in order; stop when offline
  • Retry with backoff, honouring server hints
  • Park permanently failing ops for the user
  • Only one drain at a time

2.2Clarify first

Questions worth asking, and the answers this interviewer gives.

You askInterviewer answers
Does the server support idempotency keys?Yes: it stores processed op ids and returns 'duplicate' for repeats.
What if one op is permanently invalid?Park it so the queue keeps moving, and show it to the user.
Who triggers drain?A unique WorkManager job, app start and connectivity changes.
Part 3 of 5

Detailed answer

The design decisions, the complete Kotlin solution, and a walk through it one block at a time.

3.1Design decisions

Settle these before writing code; each one shows up in the solution.

Op id = idempotency key

Generated on the device before the first send, so a retry after a lost response is harmless.

Persist on every transition

Enqueue, success and retry all save, so a kill at any line loses nothing.

Typed results

A sealed interface of Applied, Duplicate, Retryable and Rejected makes the policy explicit, and when is checked to be exhaustive.

Two locks, two jobs

One Mutex guards the queue; a second, taken with tryLock, keeps drain single-flight. The network call holds neither.

3.2Solution

Complete, compiling Kotlin with coroutines, built with warnings as errors and covered by tests/mobile/MobileTests.kt. The Java and TypeScript tabs do the same job with each language's own tools: Java compiled and run in our tests, TypeScript run in our tests.

KotlinJavaTypeScript
Outbox.ktKotlinCopy
1import java.io.IOException2import java.util.UUID3import kotlin.time.Duration4import kotlinx.coroutines.sync.Mutex5import kotlinx.coroutines.sync.withLock6 7/**8 * One pending local change. [id] is generated on the device and doubles as the9 * idempotency key, so a retried push is applied by the server at most once.10 */11data class PendingOp(12  val entity: String,13  val payload: String,14  val id: String = UUID.randomUUID().toString(),15  val attempts: Int = 0,16)17 18sealed interface PushResult {19  data object Applied : PushResult20  data object Duplicate : PushResult                   // server already applied this id: success21  data class Retryable(val after: Duration?) : PushResult   // 429 / 5xx22  data class Rejected(val reason: String) : PushResult      // 4xx validation: never retry23}24 25fun interface SyncTransport {26  suspend fun push(op: PendingOp): PushResult          // throws IOException when offline27}28 29interface OutboxStore {30  suspend fun load(): List<PendingOp>31  suspend fun save(ops: List<PendingOp>)32}33 34/**35 * Durable, ordered queue of local writes. The UI enqueues and moves on; a sync job36 * drains it whenever the OS gives us time. Safe to kill at any line.37 */38class Outbox(39  private val store: OutboxStore,40  private val transport: SyncTransport,41  private val maxAttempts: Int = 8,42) {43  sealed interface DrainOutcome {44    data object Drained : DrainOutcome45    data object Offline : DrainOutcome46    data class Backoff(val delay: Duration) : DrainOutcome47    data object AlreadyRunning : DrainOutcome48  }49 50  private val state = Mutex()                          // guards queue, parked and loaded51  private val draining = Mutex()                       // single flight52  private val queue = ArrayDeque<PendingOp>()53  private val parked = mutableListOf<PendingOp>()      // needs the user's attention54  private var loaded = false55 56  suspend fun pendingCount(): Int = state.withLock { loadIfNeeded(); queue.size }57  suspend fun parkedOps(): List<PendingOp> = state.withLock { parked.toList() }58 59  /** Persist first, then return: if the app dies a millisecond later, the intent survives. */60  suspend fun enqueue(op: PendingOp) = state.withLock {61    loadIfNeeded()62    queue.addLast(op)63    store.save(queue.toList())64  }65 66  suspend fun drain(): DrainOutcome {67    if (!draining.tryLock()) return DrainOutcome.AlreadyRunning   // a drain is already running68    try {69      while (true) {70        val op = state.withLock { loadIfNeeded(); queue.firstOrNull() } ?: return DrainOutcome.Drained71        val result = try {72          transport.push(op)73        } catch (e: IOException) {74          return DrainOutcome.Offline                  // keep everything; try again later75        }76        state.withLock {77          when (result) {78            PushResult.Applied, PushResult.Duplicate -> queue.removeFirst()79            is PushResult.Rejected -> parked += queue.removeFirst()80            is PushResult.Retryable -> {81              val next = op.copy(attempts = op.attempts + 1)82              queue.removeFirst()83              if (next.attempts >= maxAttempts) {84                parked += next85              } else {86                queue.addFirst(next)87                store.save(queue.toList())88                return DrainOutcome.Backoff(result.after ?: Backoff.delay(next.attempts))89              }90            }91          }92          store.save(queue.toList())                   // checkpoint after every op93        }94      }95    } finally {96      draining.unlock()97    }98  }99 100  private suspend fun loadIfNeeded() {101    if (loaded) return102    val saved = store.load()103    queue.addAll(0, saved)104    loaded = true105  }106}
106 linesUTF-8Kotlin
Outbox.javaJavaCopy
1import java.io.IOException;2import java.time.Duration;3import java.util.ArrayDeque;4import java.util.ArrayList;5import java.util.List;6import java.util.UUID;7import java.util.concurrent.locks.ReentrantLock;8 9/**10 * Durable, ordered queue of local writes. The UI enqueues and moves on; a sync job11 * drains it whenever the OS gives us time. Safe to kill at any line.12 */13public final class Outbox {14  /**15   * One pending local change. [id] is generated on the device and doubles as the16   * idempotency key, so a retried push is applied by the server at most once.17   */18  public record PendingOp(String entity, String payload, String id, int attempts) {19    public PendingOp(String entity, String payload) { this(entity, payload, UUID.randomUUID().toString(), 0); }20 21    PendingOp retried() { return new PendingOp(entity, payload, id, attempts + 1); }22  }23 24  public sealed interface PushResult permits Applied, Duplicate, Retryable, Rejected {}25  public record Applied() implements PushResult {}26  public record Duplicate() implements PushResult {}                          // server already applied this id: success27  public record Retryable(Duration after) implements PushResult {}            // 429 / 5xx28  public record Rejected(String reason) implements PushResult {}              // 4xx validation: never retry29 30  public interface SyncTransport {31    PushResult push(PendingOp op) throws IOException;                         // throws IOException when offline32  }33 34  public interface OutboxStore {35    List<PendingOp> load() throws IOException;36    void save(List<PendingOp> ops) throws IOException;37  }38 39  public sealed interface DrainOutcome permits Drained, Offline, RetryLater, AlreadyRunning {}40  public record Drained() implements DrainOutcome {}41  public record Offline() implements DrainOutcome {}42  public record RetryLater(Duration delay) implements DrainOutcome {}43  public record AlreadyRunning() implements DrainOutcome {}44 45  private final OutboxStore store;46  private final SyncTransport transport;47  private final int maxAttempts;48  private final Object state = new Object();                                  // guards queue, parked and loaded49  private final ReentrantLock draining = new ReentrantLock();                 // single flight50  private final ArrayDeque<PendingOp> queue = new ArrayDeque<>();51  private final List<PendingOp> parked = new ArrayList<>();                   // needs the user's attention52  private boolean loaded;53 54  public Outbox(OutboxStore store, SyncTransport transport, int maxAttempts) {55    this.store = store;56    this.transport = transport;57    this.maxAttempts = maxAttempts;58  }59 60  public int pendingCount() throws IOException {61    synchronized (state) { loadIfNeeded(); return queue.size(); }62  }63 64  public List<PendingOp> parkedOps() {65    synchronized (state) { return List.copyOf(parked); }66  }67 68  /** Persist first, then return: if the app dies a millisecond later, the intent survives. */69  public void enqueue(PendingOp op) throws IOException {70    synchronized (state) {71      loadIfNeeded();72      queue.addLast(op);73      store.save(new ArrayList<>(queue));74    }75  }76 77  public DrainOutcome drain() throws IOException {78    if (!draining.tryLock()) return new AlreadyRunning();                     // a drain is already running79    try {80      while (true) {81        PendingOp op;82        synchronized (state) { loadIfNeeded(); op = queue.peekFirst(); }83        if (op == null) return new Drained();84        PushResult result;85        try {86          result = transport.push(op);87        } catch (IOException e) {88          return new Offline();                                               // keep everything; try again later89        }90        synchronized (state) {91          if (result instanceof Applied || result instanceof Duplicate) {92            queue.removeFirst();93          } else if (result instanceof Rejected) {94            parked.add(queue.removeFirst());95          } else if (result instanceof Retryable retry) {96            PendingOp next = op.retried();97            queue.removeFirst();98            if (next.attempts() >= maxAttempts) {99              parked.add(next);100            } else {101              queue.addFirst(next);102              store.save(new ArrayList<>(queue));103              return new RetryLater(retry.after() != null ? retry.after() : Backoff.delay(next.attempts()));104            }105          }106          store.save(new ArrayList<>(queue));                                 // checkpoint after every op107        }108      }109    } finally {110      draining.unlock();111    }112  }113 114  private void loadIfNeeded() throws IOException {115    if (loaded) return;116    List<PendingOp> saved = store.load();117    for (int i = saved.size() - 1; i >= 0; i--) queue.addFirst(saved.get(i));118    loaded = true;119  }120}
120 linesUTF-8Java
Outbox.tsTypeScriptCopy
1import { Backoff } from "./Backoff.ts";2 3/**4 * One pending local change. [id] is generated on the device and doubles as the5 * idempotency key, so a retried push is applied by the server at most once.6 */7export interface PendingOp {8  entity: string;9  payload: string;10  id: string;11  attempts: number;12}13 14export const newOp = (entity: string, payload: string): PendingOp => ({ entity, payload, id: crypto.randomUUID(), attempts: 0 });15 16export type PushResult =17  | { kind: "applied" }18  | { kind: "duplicate" }                              // server already applied this id: success19  | { kind: "retryable"; afterMs?: number }            // 429 / 5xx20  | { kind: "rejected"; reason: string };              // 4xx validation: never retry21 22/** Thrown by the transport when the device has no network. */23export class OfflineError extends Error {}24 25export interface SyncTransport {26  push(op: PendingOp): Promise<PushResult>;27}28 29export interface OutboxStore {30  load(): Promise<PendingOp[]>;31  save(ops: PendingOp[]): Promise<void>;32}33 34export type DrainOutcome = { kind: "drained" } | { kind: "offline" } | { kind: "backoff"; delayMs: number } | { kind: "alreadyRunning" };35 36/** Runs one async section at a time, so a save cannot interleave with another change. */37class Mutex {38  private tail: Promise<void> = Promise.resolve();39  async run<T>(section: () => Promise<T>): Promise<T> {40    const before = this.tail;41    let release!: () => void;42    this.tail = new Promise<void>((resolve) => (release = resolve));43    await before;44    try {45      return await section();46    } finally {47      release();48    }49  }50}51 52/**53 * Durable, ordered queue of local writes. The UI enqueues and moves on; a sync job54 * drains it whenever the OS gives us time. Safe to kill at any line.55 */56export class Outbox {57  private store: OutboxStore;58  private transport: SyncTransport;59  private maxAttempts: number;60  private state = new Mutex();                         // guards queue, parked and loaded61  private draining = false;                            // single flight62  private queue: PendingOp[] = [];63  private parked: PendingOp[] = [];                    // needs the user's attention64  private loaded = false;65 66  constructor(store: OutboxStore, transport: SyncTransport, maxAttempts = 8) {67    this.store = store;68    this.transport = transport;69    this.maxAttempts = maxAttempts;70  }71 72  pendingCount(): Promise<number> {73    return this.state.run(async () => {74      await this.loadIfNeeded();75      return this.queue.length;76    });77  }78 79  parkedOps(): Promise<PendingOp[]> {80    return this.state.run(async () => [...this.parked]);81  }82 83  /** Persist first, then return: if the app dies a millisecond later, the intent survives. */84  enqueue(op: PendingOp): Promise<void> {85    return this.state.run(async () => {86      await this.loadIfNeeded();87      this.queue.push(op);88      await this.store.save([...this.queue]);89    });90  }91 92  async drain(): Promise<DrainOutcome> {93    if (this.draining) return { kind: "alreadyRunning" };   // a drain is already running94    this.draining = true;95    try {96      for (;;) {97        const op = await this.state.run(async () => {98          await this.loadIfNeeded();99          return this.queue[0];100        });101        if (!op) return { kind: "drained" };102        let result: PushResult;103        try {104          result = await this.transport.push(op);105        } catch (e) {106          if (e instanceof OfflineError) return { kind: "offline" };   // keep everything; try again later107          throw e;108        }109        const stop = await this.state.run(async (): Promise<DrainOutcome | undefined> => {110          if (result.kind === "applied" || result.kind === "duplicate") {111            this.queue.shift();112          } else if (result.kind === "rejected") {113            this.parked.push(this.queue.shift()!);114          } else {115            const next = { ...op, attempts: op.attempts + 1 };116            this.queue.shift();117            if (next.attempts >= this.maxAttempts) {118              this.parked.push(next);119            } else {120              this.queue.unshift(next);121              await this.store.save([...this.queue]);122              return { kind: "backoff", delayMs: result.afterMs ?? Backoff.delay(next.attempts) };123            }124          }125          await this.store.save([...this.queue]);      // checkpoint after every op126          return undefined;127        });128        if (stop) return stop;129      }130    } finally {131      this.draining = false;132    }133  }134 135  private async loadIfNeeded(): Promise<void> {136    if (this.loaded) return;137    const saved = await this.store.load();138    this.queue.unshift(...saved);139    this.loaded = true;140  }141}
141 linesUTF-8TypeScript

3.3Line-by-line walkthrough

The same code in the order you would explain it in the room. The line numbers follow the Kotlin file; the Java and TypeScript tabs above have the same steps in the same order.

1

The op carries its own key

A UUID created on the device. The server dedupes by it, so retries are safe.

Outbox.ktKotlinCopy
11data class PendingOp(12  val entity: String,13  val payload: String,14  val id: String = UUID.randomUUID().toString(),15  val attempts: Int = 0,16)
lines 11–16UTF-8Kotlin
2

Every server answer has a policy

Duplicate counts as success; 429/5xx is retryable; 4xx validation is rejected for good.

Outbox.ktKotlinCopy
18sealed interface PushResult {19  data object Applied : PushResult20  data object Duplicate : PushResult                   // server already applied this id: success21  data class Retryable(val after: Duration?) : PushResult   // 429 / 5xx22  data class Rejected(val reason: String) : PushResult      // 4xx validation: never retry23}
lines 18–23UTF-8Kotlin
3

Persist before returning

The UI gets control back only after the intent is on disk.

Outbox.ktKotlinCopy
60  suspend fun enqueue(op: PendingOp) = state.withLock {61    loadIfNeeded()62    queue.addLast(op)63    store.save(queue.toList())64  }
lines 60–64UTF-8Kotlin
4

Single flight

tryLock fails if a drain is already running, so a second trigger returns at once instead of sending twice.

Outbox.ktKotlinCopy
67    if (!draining.tryLock()) return DrainOutcome.AlreadyRunning   // a drain is already running68    try {
lines 67–68UTF-8Kotlin
5

Push in order; offline stops

Peek under the lock, push without it. An IOException means no network: keep everything and wait for the next trigger.

Outbox.ktKotlinCopy
70        val op = state.withLock { loadIfNeeded(); queue.firstOrNull() } ?: return DrainOutcome.Drained71        val result = try {72          transport.push(op)73        } catch (e: IOException) {74          return DrainOutcome.Offline                  // keep everything; try again later75        }
lines 70–75UTF-8Kotlin
6

Success or permanent failure

Remove applied and duplicate ops; park rejected ones so the queue keeps moving.

Outbox.ktKotlinCopy
77          when (result) {78            PushResult.Applied, PushResult.Duplicate -> queue.removeFirst()79            is PushResult.Rejected -> parked += queue.removeFirst()
lines 77–79UTF-8Kotlin
7

Retry with backoff

Count the attempt, persist it, and return the server's Retry-After or a jittered backoff. Too many attempts → park.

Outbox.ktKotlinCopy
80            is PushResult.Retryable -> {81              val next = op.copy(attempts = op.attempts + 1)82              queue.removeFirst()83              if (next.attempts >= maxAttempts) {84                parked += next85              } else {86                queue.addFirst(next)87                store.save(queue.toList())88                return DrainOutcome.Backoff(result.after ?: Backoff.delay(next.attempts))89              }90            }
lines 80–90UTF-8Kotlin
8

Checkpoint after each op

A kill now replays at most one op, and the server ignores the duplicate.

Outbox.ktKotlinCopy
92          store.save(queue.toList())                   // checkpoint after every op
lines 92–92UTF-8Kotlin

Complexity: O(1) per op plus one store write; the queue is FIFO

Part 4 of 5

Q&A

The questions interviewers ask after the code works. Answer out loud, then open.

4.1Follow-up questions

How would you batch?
Send up to N ops per request; the server returns a result per op id and the client applies the same policy to each.
What about ordering across entities?
One queue keeps global order. If entities are independent you can run one queue per entity type for parallelism.
How does the UI show pending state?
Each local row has a sync-status column updated in the same Room transaction as the outbox.
Part 5 of 5

Extra depth

The cases that break naive solutions, how the same design looks on iOS, and what to practise next.

5.1Edge cases

Say these before the interviewer does.

Show detailsHide details
  • Killed between the server applying an op and us saving: the retry returns 'duplicate'
  • An op that fails validation forever
  • Two triggers arrive at once
  • The queue grows large offline: batch pushes and compact edits to the same record

5.2On iOS

The same design in Swift and Apple's frameworks.

Show detailsHide details
ApproachSwift / iOS
StorageA GRDB or SwiftData table, written in the same transaction as the entity change.
SchedulingBGProcessingTask with requiresNetworkConnectivity, plus a drain on launch and on NWPathMonitor changes.
ConcurrencyAn actor with a draining flag; actors are re-entrant at await, so the flag is still needed.