Basics
What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.
1.1Definition
The one-paragraph summary, then where its responsibility starts and stops.
Sync engines move local changes to the server and server changes into the local database. No UI, and no business logic beyond merging. The OS will start, stop and kill it at any line, so every line must survive that.
- Drain the outbox in order, in batches, with op_ids
- Pull changes after the stored cursor, page by page
- Apply each page and save the cursor in one transaction
- Run conflict rules and mark rows that need the user
- Respect a time budget and report a result to the scheduler
- Emit metrics for every run
- Decide when to run (the background-work layer does)
- Move file bytes (the transfer manager does)
- Render anything or hold UI state
- Keep state in memory between runs
- Trust the device clock for ordering
1.2Key concepts
Ideas this deep dive relies on, in one sentence each.
Table of pending local writes, committed in the same transaction as the data, drained by the sync engine.
The last server sequence number a client has seen; pulls ask for changes after it.
A marker row that records a deletion so other devices can delete too.
Write-ahead log mode in SQLite: readers do not block the single writer.
Android library for deferrable, guaranteed background work.
React Native native module using JSI with a typed spec.
1.3Real-life run
A concrete example first, so the mechanics have something to attach to.
Priya, a field technician, edits twelve inspection jobs in a basement with no signal. Every save writes the job row and an outbox entry in one SQLite transaction, so the app feels instant. Meanwhile two supervisors change instructions on two of those same jobs from the office.
At 4:12 pm she walks outside. The phone regains a network, WorkManager’s network constraint is met, and the scheduler calls run(reason = network_back, budget = 8 minutes). The coordinator takes the single-flight lock and reads the cursor c_88411 from disk.
The push stage reads 37 outbox operations, sends them in two batches of up to 25, and gets back 36 applied and 1 conflict: the supervisor changed the same ‘valve 3 status’ field. One transaction deletes acknowledged operations and marks their rows synced; it marks a conflicting row conflict and attaches the server’s value.
The pull stage asks for changes after c_88411, receives 140 changes in one page, applies them in a single transaction and stores c_88551. The run takes 2.1 seconds, emits a trace, and returns success. Priya sees eleven green ticks and one amber ‘needs review’ badge with both values side by side.
Question
Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.
2.1Target questions
Keep them in mind while you read the detail.
- Why is the engine single-flight?
- Why push before pull?
- How do you test a sync engine?
- What happens if the OS cancels the run halfway?
Model answers are in part 4, after the detail that supports them.
2.2Interview rounds
System design rounds where this component is part of a strong answer.
Detailed answer
How it actually works: the components, the states it moves through, a full run, and the code.
3.1Anatomy
The parts and how they connect.
- Coordinator
- Entry point. Takes a single-flight lock so two triggers never run two syncs, computes a deadline from the budget, runs push then pull, and maps exceptions to a result the scheduler understands: success, retry after N seconds, needs login, or disabled.
- Push stage
- Reads the next batch of outbox rows that are ready (not waiting for a retry time), serialises them with op_ids and base versions, and handles each per-operation result inside one transaction.
- Pull stage
- Loops over pages from the stored cursor until the server says has_more is false or the deadline arrives. A 410 Gone response switches it to the rebuild path.
- Merge policy
- Pure functions, one per entity type, that decide what to do when a server change meets a row with pending local edits. Pure functions make this the easiest part to unit test.
- Applier
- Writes a page of changes in one transaction: inserts, updates, deletes from tombstones, and the new cursor. Observers fire once per page, not once per row.
- HTTP client and auth
- A shared client with timeouts, gzip, HTTP/2 and an authenticator that refreshes tokens once and retries. If refresh fails, the run ends with needs login and sync pauses.
3.2States
| State | Meaning |
|---|---|
| Idle | Nothing running. Waiting for a run request. |
| Pushing | Draining the outbox in batches. |
| Pulling | Applying pages after the cursor. |
| Backoff | Last run failed with a retryable error; next attempt not before a jittered time. |
| Needs login | Token refresh failed. Sync pauses until the user signs in. |
| Rebuilding | Cursor too old (410). Push outbox, keep unsynced rows, import snapshot. |
| Disabled | Remote kill switch on. Runs return immediately. |
3.3Run loop
One run, step by step.
- Every arrow that changes local state ends in a transaction, so the run can be killed between any two arrows without losing or duplicating data.
- Push always runs before pull, so the server’s answer already reflects this device’s edits.
- The result tells the scheduler what to do next, which keeps retry policy in one place.
3.4Code
Working code for the core of it.
Engine core (Kotlin, coroutines)
1class SyncEngine(2 private val db: AppDatabase, private val api: SyncApi, private val merge: MergePolicy,3 private val cfg: SyncConfig, private val clock: Clock, private val telemetry: Telemetry,4) {5 private val lock = Mutex()6 7 suspend fun run(reason: Reason, budget: Duration): RunResult {8 if (cfg.killSwitch) return RunResult.Disabled9 if (!lock.tryLock()) return RunResult.AlreadyRunning // single flight10 val span = telemetry.start("sync.run", reason)11 return try {12 val deadline = clock.now() + budget13 pushAll(deadline)14 pullAll(deadline)15 RunResult.Success(morePending = db.outbox().readyCount() > 0)16 } catch (e: AuthExpiredException) { RunResult.NeedsLogin }17 catch (e: IOException) { RunResult.Retry(backoffMs(db.meta().failedRuns())) }18 finally { span.end(); lock.unlock() }19 }20 21 private suspend fun pushAll(deadline: Instant) {22 while (clock.now() < deadline) {23 val batch = db.outbox().nextReady(limit = cfg.pushBatch, now = clock.now())24 if (batch.isEmpty()) return25 val res = api.push(PushRequest(cfg.deviceId, batch.map { it.toWire() }))26 db.withTransaction {27 res.results.forEach { r ->28 when (r.status) {29 Status.APPLIED, Status.DUPLICATE -> { db.outbox().delete(r.opId); db.entities().markSynced(r.entityId, r.version) }30 Status.CONFLICT -> db.entities().markConflict(r.entityId, r.serverFields)31 Status.REJECTED -> db.outbox().fail(r.opId, r.reason) // visible to the user, never retried32 }33 }34 }35 }36 }37 38 private suspend fun pullAll(deadline: Instant) {39 var cursor = db.cursors().get(SCOPE_DEFAULT)40 do {41 val page = api.pull(cursor, limit = cfg.pullPage)42 if (page.gone) { rebuild(); return } // 410: older than tombstone TTL43 db.withTransaction {44 page.changes.forEach { applyOne(it) }45 db.cursors().set(SCOPE_DEFAULT, page.nextCursor)46 }47 cursor = page.nextCursor48 } while (page.hasMore && clock.now() < deadline)49 }50}Applying one change with pending local edits
1private suspend fun applyOne(ch: Change) {2 val local = db.entities().find(ch.id)3 when {4 ch.deleted && local?.hasPendingOps == true -> db.entities().markConflict(ch.id, deletedOnServer = true)5 ch.deleted -> db.entities().delete(ch.id)6 local == null -> db.entities().insert(ch.toEntity())7 !local.hasPendingOps -> db.entities().replace(ch.toEntity())8 else -> db.entities().replace(merge.resolve(local, ch)) // keep fields the user changed locally9 }10}Retry timing with full jitter
1fun backoffMs(attempt: Int, baseMs: Long = 2_000, capMs: Long = 60 * 60_000): Long {2 val ceiling = min(capMs, baseMs shl min(attempt, 20))3 return Random.nextLong(0, ceiling + 1) // spreads a million devices across the window4}Staying inside the OS time budget (Kotlin)
1// The OS gives background work a fixed window. Stop at a safe point instead of being killed mid-write.2suspend fun SyncEngine.runWithin(reason: Reason, budget: Duration): RunResult =3 withTimeoutOrNull(budget - 2.seconds) { // keep 2 s to commit and report4 run(reason, budget)5 } ?: RunResult.Retry(backoffMs = 0) // out of time: resume in the next window6 7private suspend fun SyncEngine.pullPage(cursor: String): String? {8 currentCoroutineContext().ensureActive() // cancellation point between pages9 val page = api.pull(cursor, limit = cfg.pullPage)10 withContext(NonCancellable) { // never stop half way through a page11 db.withTransaction {12 page.changes.forEach { applyOne(it) }13 db.cursors().set(SCOPE_DEFAULT, page.nextCursor)14 }15 }16 return page.nextCursor.takeIf { page.hasMore }17}3.5Tested code, explained
A working implementation, then what each part does and why.
The contract between phone and server
Write this on the board early. These few types decide how the whole system behaves when things fail.
1sealed interface OpResult {2 val opId: String3 data class Applied(override val opId: String, val seq: Long) : OpResult4 data class Conflict(override val opId: String, val theirs: Theirs) : OpResult5 data class Rejected(override val opId: String, val reason: String) : OpResult6}7 8/** One line of the server's change log. */9data class Change(val seq: Long, val jobId: String, val field: String, val value: String, val version: Int)10data class PullPage(val changes: List<Change>, val next: Long, val hasMore: Boolean)11 12/** 429 or 5xx, or no network at all. Keep the ops and try again later. */13class TryLater(val afterMs: Long? = null) : IOException("try later")14 15/** 410: this device's cursor is older than the log keeps. */16class CursorTooOld(val snapshotUrl: String) : IOException("cursor too old")17 18interface SyncApi {19 suspend fun push(ops: List<Op>): List<OpResult>20 suspend fun pull(cursor: Long, limit: Int): PullPage21}OpResult- Three answers per op: applied, conflict or rejected. One batch can contain all three.
val seq: Long- The line number in the server's log. The cursor is simply the last number the phone has read.
TryLater- 429, 5xx or no network. A separate type means retry code can't confuse this with a bad request.
CursorTooOld- The server's 410. The phone must rebuild from a snapshot, because some deletes may be gone from the log.
Compiled and tested by npm run test:mobile.
One sync run
This is the 2.1 seconds at 4:12 pm. Read it top to bottom: lock, push until empty, pull until caught up, report.
1class SyncEngine(2 private val db: LocalDb,3 private val api: SyncApi,4 private val batchSize: Int = 25,5 private val pageSize: Int = 500,6) {7 private val lock = Mutex()8 9 suspend fun run(hasTime: () -> Boolean = { true }): RunOutcome {10 if (!lock.tryLock()) return RunOutcome.AlreadyRunning // single flight11 var applied = 012 var conflicts = 013 var pulled = 014 try {15 while (db.outbox.isNotEmpty() && hasTime()) {16 val results = api.push(db.outbox.take(batchSize)) // may throw: ops stay queued17 applyResults(results)18 applied += results.count { it is OpResult.Applied }19 conflicts += results.count { it is OpResult.Conflict }20 }21 var more = true22 while (more && hasTime()) {23 val page = api.pull(db.cursor, pageSize)24 applyPage(page)25 pulled += page.changes.size26 more = page.hasMore27 }28 return RunOutcome.Finished(applied, conflicts, pulled, morePending = db.outbox.isNotEmpty() || more)29 } catch (e: TryLater) {30 return RunOutcome.RetryLater(e.afterMs)31 } catch (e: CursorTooOld) {32 return RunOutcome.RebuildNeeded(e.snapshotUrl)33 } finally {34 lock.unlock()35 }36 }37 // applyResults and applyPage follow38}lock.tryLock()- Single flight. If a run is already active, this one returns at once, because that run will see any new ops.
db.outbox.take(batchSize)- Takes 25 at a time, and reads them fresh each loop, so edits made during the run go out too.
api.push(...)- If this throws, nothing was removed locally. The ops wait for the next run.
while (more && hasTime())- Keeps pulling pages until the server has no more, or the budget ends.
finally { lock.unlock() }- Runs even after a failure, so one bad run can't block sync forever.
Compiled and tested by npm run test:mobile.
Applying the server's answers
The push came back. One transaction records every outcome, so the outbox and the job states always agree.
1/** One transaction: finished ops leave the outbox together with their job's new state. */2private fun applyResults(results: List<OpResult>) = db.transaction {3 for (r in results) {4 val op = outbox.first { it.opId == r.opId }5 outbox = outbox - op6 val job = jobs.getValue(op.jobId)7 val updated = when (r) {8 // A later success on this job must not clear a conflict or a failure.9 is OpResult.Applied ->10 if (job.syncState == SyncState.PENDING && outbox.none { it.jobId == op.jobId }) {11 job.copy(syncState = SyncState.SYNCED)12 } else job13 is OpResult.Conflict -> job.copy(syncState = SyncState.CONFLICT, theirs = r.theirs)14 is OpResult.Rejected -> job.copy(syncState = SyncState.FAILED, error = r.reason)15 }16 jobs = jobs + (op.jobId to updated)17 }18}outbox = outbox - op- A finished op leaves the queue. Applied, conflict and rejected ops all stop being retried.
job.syncState == SyncState.PENDING- Priya's story found this bug: a later success on the same job used to clear its amber badge. Only a pending job may turn synced.
outbox.none { it.jobId == op.jobId }- A job only shows a tick once all its ops are done.
OpResult.Conflict- Stores the other person's value next to hers, ready for the conflict screen.
Compiled and tested by npm run test:mobile.
Applying a pulled page
Changes and the new cursor commit together. That one choice removes a whole family of sync bugs.
1/** One transaction: the data and the cursor can never disagree. */2private fun applyPage(page: PullPage) = db.transaction {3 for (c in page.changes) {4 val job = jobs[c.jobId] ?: Job(c.jobId)5 val unsent = outbox.any { it.jobId == c.jobId && it.field == c.field }6 val inConflict = job.theirs?.field == c.field7 jobs = jobs + (c.jobId to job.copy(8 // Never overwrite something the user changed and the server has not settled.9 answers = if (unsent || inConflict) job.answers else job.answers + (c.field to c.value),10 version = maxOf(job.version, c.version),11 ))12 }13 cursor = page.next14}unsent || inConflict- Never overwrite something the user changed and the server hasn't settled. This is what keeps leaking on Priya's screen.
maxOf(job.version, c.version)- The version still moves forward, so her eventual choice is based on version 5.
cursor = page.next- Saved in the same transaction. If the phone dies, both the changes and the bookmark roll back, and the pull simply repeats.
Compiled and tested by npm run test:mobile.
When to try again
After an outage, every phone wants to retry. This spreads them out.
1/** Full jitter: anywhere from zero up to the exponential cap, so phones spread out after an outage. */2fun retryDelayMs(attempt: Int, random: Random, baseMs: Long = 1_000, capMs: Long = 300_000): Long {3 val exponential = baseMs shl attempt.coerceIn(0, 20)4 return random.nextLong(minOf(exponential, capMs) + 1)5}baseMs shl attempt- Doubles the wait each attempt: 1, 2, 4, 8 seconds and so on.
minOf(exponential, capMs)- Stops growing at five minutes, so a phone never waits for hours.
random.nextLong(...)- Full jitter picks anywhere from zero up to that limit. Without it, phones that failed together retry together.
Compiled and tested by npm run test:mobile.
3.6Platforms
Android
- Engine in a Kotlin module with no Android UI dependencies, so it is unit-testable on the JVM.
- Room DAOs with withTransaction for every state change; WAL is on by default.
- OkHttp with an Authenticator for token refresh, gzip and HTTP/2; Retrofit or Ktor on top.
- A CoroutineWorker calls run() with a budget of a few minutes and maps the result to Result.success() or Result.retry().
iOS
- A Swift package with the engine as an actor, so state is never touched from two tasks at once.
- GRDB DatabasePool in WAL mode; write transactions for every state change.
- A default URLSession for API calls (they are short); background sessions are for big files only.
- BGAppRefreshTask gives about 30 seconds, so the budget is around 25; BGProcessingTask allows longer runs while charging.
React Native
- The engine stays native, and a Turbo Module exposes requestSync(reason) and a status observable.
- JavaScript reads the same SQLite database through WatermelonDB or op-sqlite and re-renders on change.
- No sync logic in JavaScript, because the JS runtime may not exist when the OS wakes the app.
Shared core (KMP)
- Engine in commonMain with SQLDelight for the database and Ktor for HTTP.
- expect/actual for the scheduler hooks, crypto and clocks.
- One test suite runs the protocol against a fake server for both platforms.
Q&A
The questions from part 2, now with model answers. Try each one out loud before opening it.
4.1Interview questions
Why is the engine single-flight?
Why push before pull?
How do you test a sync engine?
What happens if the OS cancels the run halfway?
Extra depth
Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.
5.1Technology choices
Show detailsHide details
| Need | Options | When to pick |
|---|---|---|
| Local database | Room · GRDB · SQLDelight · WatermelonDB · op-sqlite | Room or GRDB natively; SQLDelight for KMP; WatermelonDB for RN with its sync protocol. |
| Wire format | JSON + gzip · Protobuf (Wire, swift-protobuf) | Start with JSON; switch hot endpoints to Protobuf when payloads or parse time hurt on low-end phones. |
| Transport | HTTPS request/response · WebSocket hint channel · gRPC | HTTPS for push and pull; a socket only to say ‘cursor moved’ while the app is open. |
| Off-the-shelf sync | PowerSync · ElectricSQL · Replicache / Zero · Couchbase Lite · Firestore offline | Buy when sync is not your differentiator; build when you need custom merge rules, residency or evidence guarantees. |
| Testing | Fake server · property-based tests · fault injection | Generate random sequences of edits, kills and network failures and assert convergence. |
5.2How real products do it
Show detailsHide details
Summaries of what these teams and projects have described publicly. Check their own write-ups for detail.
| Who | What they describe |
|---|---|
| Dropbox | Publicly described rewriting its desktop sync engine, Nucleus, in Rust, with heavy use of deterministic and randomised testing to find rare sync bugs. |
| Linear | Has described a client that keeps a local copy of workspace data, applies changes optimistically, and receives ordered changes from the server tracked by an increasing sync ID. |
| Figma | Has described a server-authoritative multiplayer model where clients send property-level changes and the server resolves conflicts per property. |
| Replicache (Rocicorp) | Documents a push/pull protocol where each client mutation carries an ID and the server records the last processed mutation ID per client, which is the op_id pattern. |
5.3Metrics to emit
Show detailsHide details
| Metric | Why it matters |
|---|---|
| sync.run.duration (p50, p95) | Slow runs hit OS budgets and get cancelled. |
| sync.freshness (time since last success) | The single best indicator users are seeing stale data. |
| outbox.depth and outbox.oldest_age | Growing values mean pushes are failing silently. |
| push.result by status | Spikes in rejected or conflict usually mean a bug, not users. |
| pull.pages per run, 410 count | Too many pages means devices sync too rarely; 410s mean TTL is too short. |
| run.result by reason and OS | Shows which triggers actually deliver fresh data. |
5.4Edge cases
Show detailsHide details
| Case | How it is handled |
|---|---|
| Killed mid-transaction | SQLite rolls back; the batch is pushed again and deduped by op_id. |
| Server applied, response lost | Retry returns DUPLICATE with the original result; row marked synced. |
| Pulled change for a row with pending edits | Merge policy runs; unresolved fields become a conflict for the user. |
| Delete arrives for a row edited offline | Marked conflict instead of deleting the user’s work. |
| Clock on the phone is wrong | Ordering comes from server sequence numbers; local timestamps are display-only. |
| Schema of a pulled change is newer than the app | Unknown fields are kept as raw JSON and preserved on push; never dropped. |