The sync engine

Sync engines move local changes to the server and server changes into the local database. No UI, and no business logic beyond merging. The …

Part 1 of 5

Basics

What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.

1.1Definition

The one-paragraph summary, then where its responsibility starts and stops.

Sync engines move local changes to the server and server changes into the local database. No UI, and no business logic beyond merging. The OS will start, stop and kill it at any line, so every line must survive that.

Its job
  • Drain the outbox in order, in batches, with op_ids
  • Pull changes after the stored cursor, page by page
  • Apply each page and save the cursor in one transaction
  • Run conflict rules and mark rows that need the user
  • Respect a time budget and report a result to the scheduler
  • Emit metrics for every run
Not its job
  • Decide when to run (the background-work layer does)
  • Move file bytes (the transfer manager does)
  • Render anything or hold UI state
  • Keep state in memory between runs
  • Trust the device clock for ordering

1.2Key concepts

Ideas this deep dive relies on, in one sentence each.

01Outbox

Table of pending local writes, committed in the same transaction as the data, drained by the sync engine.

02Cursor

The last server sequence number a client has seen; pulls ask for changes after it.

03Tombstone

A marker row that records a deletion so other devices can delete too.

04WAL

Write-ahead log mode in SQLite: readers do not block the single writer.

05WorkManager

Android library for deferrable, guaranteed background work.

06Turbo Module

React Native native module using JSI with a typed spec.

1.3Real-life run

A concrete example first, so the mechanics have something to attach to.

Priya, a field technician, edits twelve inspection jobs in a basement with no signal. Every save writes the job row and an outbox entry in one SQLite transaction, so the app feels instant. Meanwhile two supervisors change instructions on two of those same jobs from the office.

At 4:12 pm she walks outside. The phone regains a network, WorkManager’s network constraint is met, and the scheduler calls run(reason = network_back, budget = 8 minutes). The coordinator takes the single-flight lock and reads the cursor c_88411 from disk.

The push stage reads 37 outbox operations, sends them in two batches of up to 25, and gets back 36 applied and 1 conflict: the supervisor changed the same ‘valve 3 status’ field. One transaction deletes acknowledged operations and marks their rows synced; it marks a conflicting row conflict and attaches the server’s value.

The pull stage asks for changes after c_88411, receives 140 changes in one page, applies them in a single transaction and stores c_88551. The run takes 2.1 seconds, emits a trace, and returns success. Priya sees eleven green ticks and one amber ‘needs review’ badge with both values side by side.

Part 2 of 5

Question

Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.

2.1Target questions

Keep them in mind while you read the detail.

  1. Why is the engine single-flight?
  2. Why push before pull?
  3. How do you test a sync engine?
  4. What happens if the OS cancels the run halfway?

Model answers are in part 4, after the detail that supports them.

2.2Interview rounds

System design rounds where this component is part of a strong answer.

Part 3 of 5

Detailed answer

How it actually works: the components, the states it moves through, a full run, and the code.

3.1Anatomy

The parts and how they connect.

SYNC ENGINE (NATIVE MODULE)STORAGERun requestreason · budgetAuth providertoken refreshHTTP clientOkHttp · URLSessionCoordinatorsingle-flight lockPush stageoutbox batchesPull stagecursor pagesRemote configbatch sizes · killswitchMerge policyLWW · field · 409Applierbatched txn +cursorTelemetryspans · countersOutbox tablepending opsSQLiteentities · cursorlimits12ackPOST /pushGET /pulltokenconflictspagesresolved1 txnevents
ClientServiceData store
Boxes are components; arrows show calls and data flow.
Coordinator
Entry point. Takes a single-flight lock so two triggers never run two syncs, computes a deadline from the budget, runs push then pull, and maps exceptions to a result the scheduler understands: success, retry after N seconds, needs login, or disabled.
Push stage
Reads the next batch of outbox rows that are ready (not waiting for a retry time), serialises them with op_ids and base versions, and handles each per-operation result inside one transaction.
Pull stage
Loops over pages from the stored cursor until the server says has_more is false or the deadline arrives. A 410 Gone response switches it to the rebuild path.
Merge policy
Pure functions, one per entity type, that decide what to do when a server change meets a row with pending local edits. Pure functions make this the easiest part to unit test.
Applier
Writes a page of changes in one transaction: inserts, updates, deletes from tombstones, and the new cursor. Observers fire once per page, not once per row.
HTTP client and auth
A shared client with timeouts, gzip, HTTP/2 and an authenticator that refreshes tokens once and retries. If refresh fails, the run ends with needs login and sync pauses.

3.2States

StateMeaning
IdleNothing running. Waiting for a run request.
PushingDraining the outbox in batches.
PullingApplying pages after the cursor.
BackoffLast run failed with a retryable error; next attempt not before a jittered time.
Needs loginToken refresh failed. Sync pauses until the user signs in.
RebuildingCursor too old (410). Push outbox, keep unsynced rows, import snapshot.
DisabledRemote kill switch on. Runs return immediately.

3.3Run loop

One run, step by step.

SchedulerCoordinatorPush stageServerPull + applyrun(reason, budget)lock, load cursorpush()read 25 ready opsPOST /push batch24 applied · 1 conflictack, mark conflict (1 txn)pull(cursor)GET /pull?cursorpage + next cursorapply page + cursor (1 txn)Loop until has_more = false or the budget endsSuccess / Retry(after) / NeedsLogin
Solid arrows are calls; dashed arrows are responses.
  1. Every arrow that changes local state ends in a transaction, so the run can be killed between any two arrows without losing or duplicating data.
  2. Push always runs before pull, so the server’s answer already reflects this device’s edits.
  3. The result tells the scheduler what to do next, which keeps retry policy in one place.

3.4Code

Working code for the core of it.

Engine core (Kotlin, coroutines)

engine-core-kotlin.ktKotlinCopy
1class SyncEngine(2  private val db: AppDatabase, private val api: SyncApi, private val merge: MergePolicy,3  private val cfg: SyncConfig, private val clock: Clock, private val telemetry: Telemetry,4) {5  private val lock = Mutex()6 7  suspend fun run(reason: Reason, budget: Duration): RunResult {8    if (cfg.killSwitch) return RunResult.Disabled9    if (!lock.tryLock()) return RunResult.AlreadyRunning          // single flight10    val span = telemetry.start("sync.run", reason)11    return try {12      val deadline = clock.now() + budget13      pushAll(deadline)14      pullAll(deadline)15      RunResult.Success(morePending = db.outbox().readyCount() > 0)16    } catch (e: AuthExpiredException) { RunResult.NeedsLogin }17      catch (e: IOException) { RunResult.Retry(backoffMs(db.meta().failedRuns())) }18      finally { span.end(); lock.unlock() }19  }20 21  private suspend fun pushAll(deadline: Instant) {22    while (clock.now() < deadline) {23      val batch = db.outbox().nextReady(limit = cfg.pushBatch, now = clock.now())24      if (batch.isEmpty()) return25      val res = api.push(PushRequest(cfg.deviceId, batch.map { it.toWire() }))26      db.withTransaction {27        res.results.forEach { r ->28          when (r.status) {29            Status.APPLIED, Status.DUPLICATE -> { db.outbox().delete(r.opId); db.entities().markSynced(r.entityId, r.version) }30            Status.CONFLICT -> db.entities().markConflict(r.entityId, r.serverFields)31            Status.REJECTED -> db.outbox().fail(r.opId, r.reason)       // visible to the user, never retried32          }33        }34      }35    }36  }37 38  private suspend fun pullAll(deadline: Instant) {39    var cursor = db.cursors().get(SCOPE_DEFAULT)40    do {41      val page = api.pull(cursor, limit = cfg.pullPage)42      if (page.gone) { rebuild(); return }                          // 410: older than tombstone TTL43      db.withTransaction {44        page.changes.forEach { applyOne(it) }45        db.cursors().set(SCOPE_DEFAULT, page.nextCursor)46      }47      cursor = page.nextCursor48    } while (page.hasMore && clock.now() < deadline)49  }50}
50 linesUTF-8Kotlin

Applying one change with pending local edits

applying-one-change.ktKotlinCopy
1private suspend fun applyOne(ch: Change) {2  val local = db.entities().find(ch.id)3  when {4    ch.deleted && local?.hasPendingOps == true -> db.entities().markConflict(ch.id, deletedOnServer = true)5    ch.deleted -> db.entities().delete(ch.id)6    local == null -> db.entities().insert(ch.toEntity())7    !local.hasPendingOps -> db.entities().replace(ch.toEntity())8    else -> db.entities().replace(merge.resolve(local, ch))   // keep fields the user changed locally9  }10}
10 linesUTF-8Kotlin

Retry timing with full jitter

retry-timing-full.ktKotlinCopy
1fun backoffMs(attempt: Int, baseMs: Long = 2_000, capMs: Long = 60 * 60_000): Long {2  val ceiling = min(capMs, baseMs shl min(attempt, 20))3  return Random.nextLong(0, ceiling + 1)        // spreads a million devices across the window4}
4 linesUTF-8Kotlin

Staying inside the OS time budget (Kotlin)

staying-inside-os.ktKotlinCopy
1// The OS gives background work a fixed window. Stop at a safe point instead of being killed mid-write.2suspend fun SyncEngine.runWithin(reason: Reason, budget: Duration): RunResult =3  withTimeoutOrNull(budget - 2.seconds) {             // keep 2 s to commit and report4    run(reason, budget)5  } ?: RunResult.Retry(backoffMs = 0)                  // out of time: resume in the next window6 7private suspend fun SyncEngine.pullPage(cursor: String): String? {8  currentCoroutineContext().ensureActive()             // cancellation point between pages9  val page = api.pull(cursor, limit = cfg.pullPage)10  withContext(NonCancellable) {                        // never stop half way through a page11    db.withTransaction {12      page.changes.forEach { applyOne(it) }13      db.cursors().set(SCOPE_DEFAULT, page.nextCursor)14    }15  }16  return page.nextCursor.takeIf { page.hasMore }17}
17 linesUTF-8Kotlin

3.5Tested code, explained

A working implementation, then what each part does and why.

The contract between phone and server

Write this on the board early. These few types decide how the whole system behaves when things fail.

SyncEngine.ktKotlinCopy
1sealed interface OpResult {2  val opId: String3  data class Applied(override val opId: String, val seq: Long) : OpResult4  data class Conflict(override val opId: String, val theirs: Theirs) : OpResult5  data class Rejected(override val opId: String, val reason: String) : OpResult6}7 8/** One line of the server's change log. */9data class Change(val seq: Long, val jobId: String, val field: String, val value: String, val version: Int)10data class PullPage(val changes: List<Change>, val next: Long, val hasMore: Boolean)11 12/** 429 or 5xx, or no network at all. Keep the ops and try again later. */13class TryLater(val afterMs: Long? = null) : IOException("try later")14 15/** 410: this device's cursor is older than the log keeps. */16class CursorTooOld(val snapshotUrl: String) : IOException("cursor too old")17 18interface SyncApi {19  suspend fun push(ops: List<Op>): List<OpResult>20  suspend fun pull(cursor: Long, limit: Int): PullPage21}
21 linesUTF-8Kotlin
OpResult
Three answers per op: applied, conflict or rejected. One batch can contain all three.
val seq: Long
The line number in the server's log. The cursor is simply the last number the phone has read.
TryLater
429, 5xx or no network. A separate type means retry code can't confuse this with a bad request.
CursorTooOld
The server's 410. The phone must rebuild from a snapshot, because some deletes may be gone from the log.

Compiled and tested by npm run test:mobile.

One sync run

This is the 2.1 seconds at 4:12 pm. Read it top to bottom: lock, push until empty, pull until caught up, report.

SyncEngine.ktKotlinCopy
1class SyncEngine(2  private val db: LocalDb,3  private val api: SyncApi,4  private val batchSize: Int = 25,5  private val pageSize: Int = 500,6) {7  private val lock = Mutex()8 9  suspend fun run(hasTime: () -> Boolean = { true }): RunOutcome {10    if (!lock.tryLock()) return RunOutcome.AlreadyRunning     // single flight11    var applied = 012    var conflicts = 013    var pulled = 014    try {15      while (db.outbox.isNotEmpty() && hasTime()) {16        val results = api.push(db.outbox.take(batchSize))     // may throw: ops stay queued17        applyResults(results)18        applied += results.count { it is OpResult.Applied }19        conflicts += results.count { it is OpResult.Conflict }20      }21      var more = true22      while (more && hasTime()) {23        val page = api.pull(db.cursor, pageSize)24        applyPage(page)25        pulled += page.changes.size26        more = page.hasMore27      }28      return RunOutcome.Finished(applied, conflicts, pulled, morePending = db.outbox.isNotEmpty() || more)29    } catch (e: TryLater) {30      return RunOutcome.RetryLater(e.afterMs)31    } catch (e: CursorTooOld) {32      return RunOutcome.RebuildNeeded(e.snapshotUrl)33    } finally {34      lock.unlock()35    }36  }37  // applyResults and applyPage follow38}
38 linesUTF-8Kotlin
lock.tryLock()
Single flight. If a run is already active, this one returns at once, because that run will see any new ops.
db.outbox.take(batchSize)
Takes 25 at a time, and reads them fresh each loop, so edits made during the run go out too.
api.push(...)
If this throws, nothing was removed locally. The ops wait for the next run.
while (more && hasTime())
Keeps pulling pages until the server has no more, or the budget ends.
finally { lock.unlock() }
Runs even after a failure, so one bad run can't block sync forever.

Compiled and tested by npm run test:mobile.

Applying the server's answers

The push came back. One transaction records every outcome, so the outbox and the job states always agree.

SyncEngine.ktKotlinCopy
1/** One transaction: finished ops leave the outbox together with their job's new state. */2private fun applyResults(results: List<OpResult>) = db.transaction {3  for (r in results) {4    val op = outbox.first { it.opId == r.opId }5    outbox = outbox - op6    val job = jobs.getValue(op.jobId)7    val updated = when (r) {8      // A later success on this job must not clear a conflict or a failure.9      is OpResult.Applied ->10        if (job.syncState == SyncState.PENDING && outbox.none { it.jobId == op.jobId }) {11          job.copy(syncState = SyncState.SYNCED)12        } else job13      is OpResult.Conflict -> job.copy(syncState = SyncState.CONFLICT, theirs = r.theirs)14      is OpResult.Rejected -> job.copy(syncState = SyncState.FAILED, error = r.reason)15    }16    jobs = jobs + (op.jobId to updated)17  }18}
18 linesUTF-8Kotlin
outbox = outbox - op
A finished op leaves the queue. Applied, conflict and rejected ops all stop being retried.
job.syncState == SyncState.PENDING
Priya's story found this bug: a later success on the same job used to clear its amber badge. Only a pending job may turn synced.
outbox.none { it.jobId == op.jobId }
A job only shows a tick once all its ops are done.
OpResult.Conflict
Stores the other person's value next to hers, ready for the conflict screen.

Compiled and tested by npm run test:mobile.

Applying a pulled page

Changes and the new cursor commit together. That one choice removes a whole family of sync bugs.

SyncEngine.ktKotlinCopy
1/** One transaction: the data and the cursor can never disagree. */2private fun applyPage(page: PullPage) = db.transaction {3  for (c in page.changes) {4    val job = jobs[c.jobId] ?: Job(c.jobId)5    val unsent = outbox.any { it.jobId == c.jobId && it.field == c.field }6    val inConflict = job.theirs?.field == c.field7    jobs = jobs + (c.jobId to job.copy(8      // Never overwrite something the user changed and the server has not settled.9      answers = if (unsent || inConflict) job.answers else job.answers + (c.field to c.value),10      version = maxOf(job.version, c.version),11    ))12  }13  cursor = page.next14}
14 linesUTF-8Kotlin
unsent || inConflict
Never overwrite something the user changed and the server hasn't settled. This is what keeps leaking on Priya's screen.
maxOf(job.version, c.version)
The version still moves forward, so her eventual choice is based on version 5.
cursor = page.next
Saved in the same transaction. If the phone dies, both the changes and the bookmark roll back, and the pull simply repeats.

Compiled and tested by npm run test:mobile.

When to try again

After an outage, every phone wants to retry. This spreads them out.

SyncEngine.ktKotlinCopy
1/** Full jitter: anywhere from zero up to the exponential cap, so phones spread out after an outage. */2fun retryDelayMs(attempt: Int, random: Random, baseMs: Long = 1_000, capMs: Long = 300_000): Long {3  val exponential = baseMs shl attempt.coerceIn(0, 20)4  return random.nextLong(minOf(exponential, capMs) + 1)5}
5 linesUTF-8Kotlin
baseMs shl attempt
Doubles the wait each attempt: 1, 2, 4, 8 seconds and so on.
minOf(exponential, capMs)
Stops growing at five minutes, so a phone never waits for hours.
random.nextLong(...)
Full jitter picks anywhere from zero up to that limit. Without it, phones that failed together retry together.

Compiled and tested by npm run test:mobile.

3.6Platforms

Android

  • Engine in a Kotlin module with no Android UI dependencies, so it is unit-testable on the JVM.
  • Room DAOs with withTransaction for every state change; WAL is on by default.
  • OkHttp with an Authenticator for token refresh, gzip and HTTP/2; Retrofit or Ktor on top.
  • A CoroutineWorker calls run() with a budget of a few minutes and maps the result to Result.success() or Result.retry().

iOS

  • A Swift package with the engine as an actor, so state is never touched from two tasks at once.
  • GRDB DatabasePool in WAL mode; write transactions for every state change.
  • A default URLSession for API calls (they are short); background sessions are for big files only.
  • BGAppRefreshTask gives about 30 seconds, so the budget is around 25; BGProcessingTask allows longer runs while charging.

React Native

  • The engine stays native, and a Turbo Module exposes requestSync(reason) and a status observable.
  • JavaScript reads the same SQLite database through WatermelonDB or op-sqlite and re-renders on change.
  • No sync logic in JavaScript, because the JS runtime may not exist when the OS wakes the app.

Shared core (KMP)

  • Engine in commonMain with SQLDelight for the database and Ktor for HTTP.
  • expect/actual for the scheduler hooks, crypto and clocks.
  • One test suite runs the protocol against a fake server for both platforms.
Part 4 of 5

Q&A

The questions from part 2, now with model answers. Try each one out loud before opening it.

4.1Interview questions

Why is the engine single-flight?
Two concurrent runs could push the same outbox rows twice and apply pages out of order. A lock plus idempotent server writes keeps one logical sync at a time.
Why push before pull?
So the server’s view, which the pull returns, already includes this device’s edits, and the merge only has to handle edits made by others.
How do you test a sync engine?
Pure merge functions with unit tests, the engine against a fake server, and randomised fault injection: random edits, kills between transactions, dropped responses, then assert all replicas converge.
What happens if the OS cancels the run halfway?
Nothing is lost. Every step committed atomically, the cursor only moves with its page, and the next run continues from disk.
Part 5 of 5

Extra depth

Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.

5.1Technology choices

Show detailsHide details
NeedOptionsWhen to pick
Local databaseRoom · GRDB · SQLDelight · WatermelonDB · op-sqliteRoom or GRDB natively; SQLDelight for KMP; WatermelonDB for RN with its sync protocol.
Wire formatJSON + gzip · Protobuf (Wire, swift-protobuf)Start with JSON; switch hot endpoints to Protobuf when payloads or parse time hurt on low-end phones.
TransportHTTPS request/response · WebSocket hint channel · gRPCHTTPS for push and pull; a socket only to say ‘cursor moved’ while the app is open.
Off-the-shelf syncPowerSync · ElectricSQL · Replicache / Zero · Couchbase Lite · Firestore offlineBuy when sync is not your differentiator; build when you need custom merge rules, residency or evidence guarantees.
TestingFake server · property-based tests · fault injectionGenerate random sequences of edits, kills and network failures and assert convergence.

5.2How real products do it

Show detailsHide details

Summaries of what these teams and projects have described publicly. Check their own write-ups for detail.

WhoWhat they describe
DropboxPublicly described rewriting its desktop sync engine, Nucleus, in Rust, with heavy use of deterministic and randomised testing to find rare sync bugs.
LinearHas described a client that keeps a local copy of workspace data, applies changes optimistically, and receives ordered changes from the server tracked by an increasing sync ID.
FigmaHas described a server-authoritative multiplayer model where clients send property-level changes and the server resolves conflicts per property.
Replicache (Rocicorp)Documents a push/pull protocol where each client mutation carries an ID and the server records the last processed mutation ID per client, which is the op_id pattern.

5.3Metrics to emit

Show detailsHide details
MetricWhy it matters
sync.run.duration (p50, p95)Slow runs hit OS budgets and get cancelled.
sync.freshness (time since last success)The single best indicator users are seeing stale data.
outbox.depth and outbox.oldest_ageGrowing values mean pushes are failing silently.
push.result by statusSpikes in rejected or conflict usually mean a bug, not users.
pull.pages per run, 410 countToo many pages means devices sync too rarely; 410s mean TTL is too short.
run.result by reason and OSShows which triggers actually deliver fresh data.

5.4Edge cases

Show detailsHide details
CaseHow it is handled
Killed mid-transactionSQLite rolls back; the batch is pushed again and deduped by op_id.
Server applied, response lostRetry returns DUPLICATE with the original result; row marked synced.
Pulled change for a row with pending editsMerge policy runs; unresolved fields become a conflict for the user.
Delete arrives for a row edited offlineMarked conflict instead of deleting the user’s work.
Clock on the phone is wrongOrdering comes from server sequence numbers; local timestamps are display-only.
Schema of a pulled change is newer than the appUnknown fields are kept as raw JSON and preserved on push; never dropped.