Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
A server fix reaches everyone in minutes. A mobile fix takes days of store review and user updates, so detection must happen early. Phones report crashes, hangs, slow screens and sync health. A staged rollout compares the new version with the old one and stops itself when the new one looks worse.
You cannot roll back a phone, so telemetry has to catch a bad release while only 1% of users have it.
1.2Key concepts
One idea per card, in plain words.
Share of sessions with no crash. The headline number for release health.
Application Not Responding: the main thread blocked for 5 seconds. Android penalises apps with high rates in Play.
Timed spans for one user action, linked to server spans through a shared trace ID.
Ships to 1%, then 10%, then everyone, with a check at each step.
Sends a fraction of detailed data. Errors stay at 100%; routine timing does not.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
A release doubled crashes on one manufacturer's phones, and you learned from store reviews five days later. Design the telemetry and gates that would have caught it within an hour.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| How many users on the first stage? | 1%: about 20,000 sessions in the first hours, enough to compare. |
| Can telemetry use cellular data? | Yes for crashes, in small batches. Detailed traces wait for Wi-Fi. |
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Pick the release signals
Crash-free sessions, ANR rate, startup time, sync success, and one business metric such as checkout completion.
- Buffer on the device
Write events to disk and upload in batches with the same outbox approach as user data. Crashes that happen offline must still arrive.
- Slice by version and device
Compare the new version with the current one on the same device classes. A global average hid the manufacturer problem.
- Link client to server
Send a trace ID with each request, so a slow screen leads to the slow backend call.
- Gate the rollout
Halt automatically if a signal crosses its threshold with enough sessions to be meaningful.
- Keep a kill switch
A remote flag can turn the feature off immediately. A halted rollout does not help users who already updated.
3.2Code
A working implementation, then what each part does and why.
Deciding whether a rollout continues
This would have stopped version 8.2 at 1%. It compares like with like, and waits for enough data.
1data class Health(val sessions: Int, val crashFree: Double, val outboxP90: Int)2 3enum class Rollout { CONTINUE, WAIT, HALT }4 5/** Compare new with old on the same devices over the same hours. */6fun judge(new: Health, old: Health, minSessions: Int = 20_000): Rollout {7 if (new.sessions < minSessions) return Rollout.WAIT // too little data to judge8 val moreCrashes = new.crashFree < old.crashFree - 0.0029 val syncStalled = new.outboxP90 > old.outboxP90 * 2 // a stall shows no crash at all10 return if (moreCrashes || syncStalled) Rollout.HALT else Rollout.CONTINUE11}12 13private val STAGES = listOf(1, 10, 50, 100)14 15fun nextStage(percent: Int): Int = STAGES.firstOrNull { it > percent } ?: 100new.sessions < minSessions- Too little data proves nothing. Wait, rather than react to noise.
old.crashFree - 0.002- A small allowance, so normal variation doesn't halt every release.
new.outboxP90 > old.outboxP90 * 2- Catches a sync stall, which shows up in no crash report. Domain health needs its own signal.
Compiled and tested by npm run test:mobile.
Telemetry that survives being offline
It's an outbox for measurements, so a crash in a basement still gets reported.
1data class Event(val id: String, val name: String, val critical: Boolean)2 3/** An outbox for measurements. It keeps crashes when full, and drops routine events first. */4class TelemetryBuffer(private val maxEvents: Int = 500) {5 private val queue = ArrayDeque<Event>()6 val size: Int get() = queue.size7 8 fun record(e: Event) {9 if (queue.size >= maxEvents) {10 val victim = queue.firstOrNull { !it.critical } ?: queue.first()11 queue.remove(victim)12 }13 queue.addLast(e)14 }15 16 /** Removes a batch only after [upload] succeeds, so a failed upload loses nothing. */17 suspend fun flush(batch: Int = 50, upload: suspend (List<Event>) -> Boolean): Int {18 val events = queue.take(batch)19 if (events.isEmpty() || !upload(events)) return 020 repeat(events.size) { queue.removeFirst() }21 return events.size22 }23}24 25/** Server side. A resend must not inflate the crash rate. */26class EventIntake {27 private val seen = HashSet<String>()28 fun accept(batch: List<Event>): List<Event> = batch.filter { seen.add(it.id) }29}queue.firstOrNull { !it.critical }- When full, routine events go first. Crash reports are the last to be dropped.
if (events.isEmpty() || !upload(events))- Events leave the queue only after a successful upload.
class EventIntake- The server removes repeats, so a resend can't inflate the crash rate.
Compiled and tested by npm run test:mobile.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why not just wait for Play Console data?
How do you avoid false alarms at 1%?
What about user privacy?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Telemetry cost
- Every event uses battery and data. Budget events per session, and sample routine ones.
- Symbolication
- Upload mapping files and dSYMs in CI, or crash reports stay unreadable.
- Counterpoint
- More dashboards do not mean more detection. Five signals with alerts beat fifty without them.