Mobile Observability: Crashes, ANRs, Traces and Rollout Gates

You cannot roll back a phone, so telemetry has to catch a bad release while only 1% of users have it.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

A server fix reaches everyone in minutes. A mobile fix takes days of store review and user updates, so detection must happen early. Phones report crashes, hangs, slow screens and sync health. A staged rollout compares the new version with the old one and stops itself when the new one looks worse.

Why it matters

You cannot roll back a phone, so telemetry has to catch a bad release while only 1% of users have it.

1.2Key concepts

One idea per card, in plain words.

01Crash-free sessions

Share of sessions with no crash. The headline number for release health.

02ANR

Application Not Responding: the main thread blocked for 5 seconds. Android penalises apps with high rates in Play.

03Client trace

Timed spans for one user action, linked to server spans through a shared trace ID.

04Staged rollout

Ships to 1%, then 10%, then everyone, with a check at each step.

05Sampling

Sends a fraction of detailed data. Errors stay at 100%; routine timing does not.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

A release doubled crashes on one manufacturer's phones, and you learned from store reviews five days later. Design the telemetry and gates that would have caught it within an hour.

2.2Clarify first

Each answer removes a guess.

You askAnswer
How many users on the first stage?1%: about 20,000 sessions in the first hours, enough to compare.
Can telemetry use cellular data?Yes for crashes, in small batches. Detailed traces wait for Wi-Fi.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Pick the release signals

    Crash-free sessions, ANR rate, startup time, sync success, and one business metric such as checkout completion.

  2. Buffer on the device

    Write events to disk and upload in batches with the same outbox approach as user data. Crashes that happen offline must still arrive.

  3. Slice by version and device

    Compare the new version with the current one on the same device classes. A global average hid the manufacturer problem.

  4. Link client to server

    Send a trace ID with each request, so a slow screen leads to the slow backend call.

  5. Gate the rollout

    Halt automatically if a signal crosses its threshold with enough sessions to be meaningful.

  6. Keep a kill switch

    A remote flag can turn the feature off immediately. A halted rollout does not help users who already updated.

3.2Code

A working implementation, then what each part does and why.

Deciding whether a rollout continues

This would have stopped version 8.2 at 1%. It compares like with like, and waits for enough data.

ReleaseGate.ktKotlinCopy
1data class Health(val sessions: Int, val crashFree: Double, val outboxP90: Int)2 3enum class Rollout { CONTINUE, WAIT, HALT }4 5/** Compare new with old on the same devices over the same hours. */6fun judge(new: Health, old: Health, minSessions: Int = 20_000): Rollout {7  if (new.sessions < minSessions) return Rollout.WAIT    // too little data to judge8  val moreCrashes = new.crashFree < old.crashFree - 0.0029  val syncStalled = new.outboxP90 > old.outboxP90 * 2    // a stall shows no crash at all10  return if (moreCrashes || syncStalled) Rollout.HALT else Rollout.CONTINUE11}12 13private val STAGES = listOf(1, 10, 50, 100)14 15fun nextStage(percent: Int): Int = STAGES.firstOrNull { it > percent } ?: 100
15 linesUTF-8Kotlin
new.sessions < minSessions
Too little data proves nothing. Wait, rather than react to noise.
old.crashFree - 0.002
A small allowance, so normal variation doesn't halt every release.
new.outboxP90 > old.outboxP90 * 2
Catches a sync stall, which shows up in no crash report. Domain health needs its own signal.

Compiled and tested by npm run test:mobile.

Telemetry that survives being offline

It's an outbox for measurements, so a crash in a basement still gets reported.

ReleaseGate.ktKotlinCopy
1data class Event(val id: String, val name: String, val critical: Boolean)2 3/** An outbox for measurements. It keeps crashes when full, and drops routine events first. */4class TelemetryBuffer(private val maxEvents: Int = 500) {5  private val queue = ArrayDeque<Event>()6  val size: Int get() = queue.size7 8  fun record(e: Event) {9    if (queue.size >= maxEvents) {10      val victim = queue.firstOrNull { !it.critical } ?: queue.first()11      queue.remove(victim)12    }13    queue.addLast(e)14  }15 16  /** Removes a batch only after [upload] succeeds, so a failed upload loses nothing. */17  suspend fun flush(batch: Int = 50, upload: suspend (List<Event>) -> Boolean): Int {18    val events = queue.take(batch)19    if (events.isEmpty() || !upload(events)) return 020    repeat(events.size) { queue.removeFirst() }21    return events.size22  }23}24 25/** Server side. A resend must not inflate the crash rate. */26class EventIntake {27  private val seen = HashSet<String>()28  fun accept(batch: List<Event>): List<Event> = batch.filter { seen.add(it.id) }29}
29 linesUTF-8Kotlin
queue.firstOrNull { !it.critical }
When full, routine events go first. Crash reports are the last to be dropped.
if (events.isEmpty() || !upload(events))
Events leave the queue only after a successful upload.
class EventIntake
The server removes repeats, so a resend can't inflate the crash rate.

Compiled and tested by npm run test:mobile.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why not just wait for Play Console data?
It lags and lacks your business signals. Use it as a cross-check.
How do you avoid false alarms at 1%?
Require a minimum session count and compare against the old version over the same window.
What about user privacy?
Send no content or personal data by default, hash identifiers, and document every field.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Telemetry cost
Every event uses battery and data. Budget events per session, and sample routine ones.
Symbolication
Upload mapping files and dSYMs in CI, or crash reports stay unreadable.
Counterpoint
More dashboards do not mean more detection. Five signals with alerts beat fifty without them.