Basics
Start with the product and the ideas it depends on. If these are clear, the design that follows will feel obvious rather than memorised.
1.1Product in plain words
Who uses it, what a normal day looks like, and why it is harder than it seems.
Deciding where three AI features run: smart reply suggestions in chat, short summaries of long service descriptions, and recognising products in photos. Some phones can run AI models themselves; others need the cloud.
Who uses it
- Users: want instant, private features that work offline.
- Product: wants the features on every phone, not just new ones.
- Finance: wants cloud AI costs to stay low.
A day in the life
- On a new phone, tapping Summarise returns a summary in about a second, with no network call.
- On an older phone, the same button calls the cloud and takes a little longer.
- Offline on an older phone, the button shows the first two sentences instead of failing.
- Taking a photo of a product identifies it on the phone; the photo never leaves the device.
- A weekly dashboard shows which phones run which path, and rules change remotely if a model is slow on some devices.
Why it is hard
The same model can take 300 ms on one phone and 5 seconds on another.
Even small language models are hundreds of megabytes.
Photos and messages are personal; sending them to a server needs a reason and consent.
Users shouldn’t notice which path served them.
Like a shop that answers simple questions at the counter and phones head office only for the hard ones, and still gives you a leaflet if the phone line is down.
1.2Key concepts
The building blocks this design uses. Each one appears again in the answer.
A small model that runs on the phone's NPU or GPU. It is private, free per request and works offline, but it is less capable and varies by device.
A large model on servers: the most capable and consistent, but every call costs money, needs a network and sends data off the device.
Storing a model's weights with fewer bits, for example 4-bit, so it fits in phone memory and runs faster, for a small loss in quality.
A router decides per request: on device when the phone can, cloud as the fallback, and a simple non-AI answer when both fail.
Models are hundreds of megabytes, so they download after install, on Wi-Fi, and update separately from the app.
Android phones differ hugely in chips and memory. Capability checks plus remote rules decide which devices get which path.
Learn them properly in the deep dives:
Question
Now read the question closely. Agree on what you are building, ask what is unclear, and work out the numbers before you draw anything.
2.1Brief and scope
The question exactly as the interviewer asks it, and what they want to learn from your answer.
We want three AI features in our app: smart replies in chat, summarising long service descriptions, and identifying products in photos. Leadership wants low cost and strong privacy. Decide what runs on the device and what runs in the cloud, and design how they work together.
What it tests: On-device runtimes, model size and quantisation, device fragmentation, and routing that degrades gracefully.
The answer in one sentence. An on-device AI router that checks capability, consent and remote rules, then runs a system model, a bundled model or the cloud, with a non-AI fallback.
How to spend the 40 minutes
- Clarify~4 minAsk questions, agree on scope and write down assumptions.
- Estimate~4 minRough numbers for users, data and traffic, and what they imply.
- High-level design~10 minComponents, data flow and the key API on the board.
- Deep dive~14 minThe two or three hardest parts, with real mechanisms.
- Trade-offs and wrap-up~8 minDecisions, failure modes, and what you would do next.
2.2Clarify the problem
The first five minutes. Each answer removes a guess from your design.
| You ask | Interviewer answers |
|---|---|
| Which devices must be supported? | Android from 4 GB RAM and iOS from iPhone 12. |
| Must features work offline? | Smart replies and photo recognition should; summaries may need a network. |
| Can photos leave the device? | Only with consent; default is on-device. |
| App size limits? | The base download should not grow by more than 20 MB. |
2.3Estimate the scale
Say the arithmetic out loud. What matters is the order of magnitude and what it forces you to do.
| Metric | Estimate | How |
|---|---|---|
| Small text model on device | ≈ 1–3B params, 4-bit | about 0.5–1.5 GB on disk |
| Image classifier | ≈ 5–20 MB | fits in the app or downloads on demand |
| Device share able to run a local LLM | varies widely | measure with telemetry, not assumptions |
| Cloud cost avoided | proportional to local share | every on-device request is free at the margin |
Detailed answer
The design itself, from the big picture down to the code: components first, then one request end to end, then the parts interviewers push hardest on.
3.1Architecture
A capability check on the device decides, per feature and per device, whether to run locally, call the cloud, or fall back to a simpler non-AI behaviour.
- AI router
- A small library every feature calls. It checks device capability (OS version, RAM, chip, whether a system model is present), user consent and remote rules, then picks local, cloud or fallback.
- System models
- iOS 26 exposes an on-device language model through Apple’s Foundation Models framework, and supported Android devices expose Gemini Nano through ML Kit’s GenAI APIs. Using them costs no app size.
- Bundled models
- Task-specific models such as an image classifier run through Core ML on iOS and LiteRT (formerly TensorFlow Lite) on Android, using the GPU or neural engine where present.
- On-demand packs
- Larger models download after install, on Wi-Fi, with versioning and integrity checks, so the base app stays small.
- Telemetry by device class
- Latency, battery impact, crash rate and quality signals per device class decide routing rules, which change remotely without a release.
3.2Key flow, step by step
Summarising a service description on two different phones.
- Features never call a model directly; they call the router with a task and constraints.
- Each path produces the same output shape, so UI code is identical.
- Offline with no local model, the feature degrades to a non-AI behaviour instead of failing.
3.3Front end
Here the front end owns the decision. The router is client code, and the UI must make three different execution paths feel like one feature, including downloads, consent and fallbacks.
Screen map
- Smart replies
- Three chips above the keyboard, generated on device after each received message, with no spinner; if they are not ready in 300 ms, they simply don’t appear.
- Summarise
- A button that expands to the summary; the label never mentions where it ran, but internal builds show the source and latency.
- Model download
- Shown once when a feature needs a pack: size, Wi-Fi only by default, progress, cancel. The feature works in its fallback form meanwhile.
- Consent sheet
- Asked only when the on-device recogniser fails and cloud recognition could help; the answer is remembered and changeable in settings.
Client layers
| Layer | Technology | Responsibility |
|---|---|---|
| UI | Compose / SwiftUI | Feature screens identical across paths |
| AI router | Kotlin / Swift library | Capability check, consent, remote rules, timeouts |
| System models | Apple Foundation Models (iOS 26) · Gemini Nano via ML Kit GenAI | On-device text generation where available |
| Bundled models | Core ML · LiteRT | Image classification and small tasks |
| Model packs | Downloader + checksum | On-demand models on Wi-Fi |
| Telemetry | Per device class | Latency, success, battery impact |
Every state the user can see
| State | What the user sees | How it works |
|---|---|---|
| Ready | Feature works; no indicator | A path is available |
| Preparing | ‘Getting smart replies ready’ once | Model pack downloading |
| Local | Instant result | On-device model answered |
| Cloud | Same result, slightly slower | Router chose cloud |
| Fallback | Simpler non-AI version | Offline and no local model |
| Consent needed | Sheet with clear choice | Photo would leave the device |
Code
One interface over each platform’s on-device model (Kotlin Multiplatform)
1// Shared code sees one interface. Android backs it with Gemini Nano (AICore) and iOS with2// Apple's Foundation Models framework, each in its own platform source set.3enum class Availability { READY, DOWNLOADING, UNSUPPORTED }4 5interface OnDeviceModel {6 suspend fun availability(): Availability7 suspend fun generate(instructions: String, input: String): String8}9 10class Summariser(private val local: OnDeviceModel, private val cloudOrFallback: suspend (String) -> AiResult<String>) {11 suspend fun summarise(text: String): AiResult<String> {12 if (local.availability() != Availability.READY) return cloudOrFallback(text) // older device or model not ready13 return try {14 val out = local.generate("Summarise service descriptions in 60 words or fewer.", text)15 AiResult.Ok(out, Source.SYSTEM_MODEL)16 } catch (e: CancellationException) {17 throw e // never swallow cancellation18 } catch (e: Exception) {19 cloudOrFallback(text)20 }21 }22}Router with a time budget and fallback (Kotlin)
1suspend fun summarise(text: String): AiResult<String> {2 val rule = remoteRules.summarise3 if (capabilities.hasSystemModel && capabilities.ramGb >= rule.minRamGbLocal) {4 withTimeoutOrNull(rule.localTimeoutMs) { local.summarise(text) }?.let { return AiResult.Ok(it, Source.SYSTEM_MODEL) }5 }6 if (network.isOnline) return runCatching { cloud.summarise(text) }7 .map { AiResult.Ok(it, Source.CLOUD) }8 .getOrElse { fallback(text) }9 return fallback(text) // first two sentences, never an error10}11 12private fun fallback(text: String) = AiResult.Fallback(text.split(". ").take(2).joinToString(". ") + ".", "offline")3.4Data model and APIs
The tables and contracts that make the design concrete.
Router interface (Kotlin)
1interface AiRouter {2 suspend fun summarise(text: String, maxWords: Int): AiResult<String>3}4sealed interface AiResult<out T> {5 data class Ok<T>(val value: T, val source: Source, val ms: Long) : AiResult<T>6 data class Fallback<T>(val value: T, val reason: String) : AiResult<T>7}8enum class Source { SYSTEM_MODEL, BUNDLED_MODEL, CLOUD }Routing rule (remote config)
1{ "summarise": {2 "prefer": ["SYSTEM_MODEL", "CLOUD"],3 "min_ram_gb_local": 6,4 "cloud_requires_consent": false,5 "disable_local_on": ["SM-A0*"] },6 "photo_identify": { "prefer": ["BUNDLED_MODEL", "CLOUD"], "cloud_requires_consent": true } }3.5Tested code, explained
The core of this design as working code, with what each part does and why.
What 4-bit loading actually saves
The weights shrink, and the constants needed to decode them cost a little extra.
1/** Bits per weight spent on quantisation constants: one 32-bit constant per block of 64 weights. */2export function constantBits(doubleQuant: boolean): number {3 // Double quantisation stores 8-bit constants, plus one 32-bit constant per 256 of those.4 return doubleQuant ? 8 / 64 + 32 / (64 * 256) : 32 / 64;5}6 7export function baseModelGiB(params: number, weightBits: number, doubleQuant: boolean): number {8 return (params * (weightBits + constantBits(doubleQuant))) / 8 / 1024 ** 3;9}32 / 64- One 32-bit constant per 64 weights.
8 / 64 + 32 / (64 * 256)- Double quantisation also compresses those constants.
Tested by npm run test:modules.
How much memory one token costs
This arithmetic sets how many requests a GPU can serve at once. Weights are fixed, and the KV cache grows with every token.
1/** Bytes of KV cache one token costs: a key and a value, per layer, per KV head. */2export function kvBytesPerToken(m: { layers: number; kvHeads: number; headDim: number; bytesPerValue: number }): number {3 return 2 * m.layers * m.kvHeads * m.headDim * m.bytesPerValue;4}5 6/** Requests that fit in the memory left after loading the weights. */7export function concurrentRequests(freeGiB: number, tokensPerRequest: number, bytesPerToken: number): number {8 return Math.floor((freeGiB * 1024 ** 3) / (tokensPerRequest * bytesPerToken));9}2 * m.layers * ...- A key and a value, for each layer, each head and each dimension.
freeGiB- What's left after loading the weights. Divide by the cost of a whole request, and you have your concurrency limit.
Tested by npm run test:modules.
3.6Hard parts, in depth
Where interviewers spend the second half of the round.
Making models fit
Quantisation stores weights in 8 or 4 bits instead of 16 or 32, cutting size and memory several-fold with a small quality loss you must measure. Distillation trains a small model to imitate a big one for a single task.
For narrow tasks such as smart replies, a small fine-tuned model often beats a general model at a fraction of the size.
Fragmentation is the real problem
The same model can run in 300 ms on one phone and 5 seconds on another, or crash on low memory. Ship with conservative rules, collect per-device telemetry, and widen local routing as data proves it is safe.
Privacy as an architecture choice
Photos identified on device never leave it. When the cloud is needed, send the minimum: a cropped region rather than a full photo, and only with consent. Say this explicitly; privacy is a product feature, not a footnote.
Q&A
The questions that come after the design, with model answers. Answer each one out loud before opening it.
4.1Follow-up questions
What the interviewer asks once the main design is on the board.
How do you update a bundled model without an app release?
How do you evaluate on-device quality?
When would you not use on-device AI?
4.2Front-end follow-ups
Questions aimed at the client: rendering, state and devices.
How do you keep the experience identical across paths?
How do you decide which devices run locally?
4.3Incidents, probes and traps
Realistic production incidents and pointed probes, grouped by topic.
The product 1
ProbeWhy not run everything in the cloud?
Requirements 1
TrapWhat do candidates forget to ask?
Scale 1
ProbeHow do you estimate how many devices can run a local model?
Architecture 2
IncidentAfter enabling on-device summaries, crashes rose on 4 GB phones.
ProbeHow do you update a bundled model without an app release?
The key flow 1
IncidentUsers complain summaries differ wildly between phones.
Front end 2
IncidentUsers on mobile data say the app downloaded a gigabyte.
ProbeShould the UI tell users where AI ran?
Data and APIs 1
TrapWhy route through an interface instead of calling models directly?
The hard parts 2
IncidentAfter quantisation, quality dropped badly in Hindi.
ProbeHow do you control battery impact?
Trade-offs 1
ProbeWhy bundle the image classifier but download the language model?
Failures 2
IncidentA corrupted model file causes repeated crashes on start-up.
IncidentThe iOS system model is unavailable because the user turned Apple Intelligence off.
Extra depth
What turns a good answer into a strong one: the decisions you rejected, how the system fails, and the signals of seniority.
5.1Trade-offs and decisions
Every choice has a cost. Naming it is what interviewers listen for.
Show detailsHide details
| Decision | Chosen | Rejected | Why |
|---|---|---|---|
| Smart replies | On device | Cloud | Latency matters, works offline, private, and the task is small. |
| Summaries | System model, else cloud | Cloud only | Free when available; cloud covers older phones. |
| Photo recognition | Bundled classifier, cloud fallback | Cloud only | Private by default and offline; cloud handles rare products. |
| Model delivery | On-demand packs | Bundle everything | Keeps the install small; most users never need every model. |
5.2Failure modes
What breaks in production, and how the design keeps working when it does.
Show detailsHide details
| What breaks | How the design copes |
|---|---|
| Local model is slow on some phones | Telemetry flags the device class; remote rule routes it to cloud. |
| Model download corrupt or partial | Hash check, resume, keep the previous version until the new one verifies. |
| Cloud and local give different answers | Same prompt templates and output limits; evaluate both paths on the same test set. |
| Battery drain complaints | Run heavy inference only on user action, never in the background. |
5.3Performance and accessibility
The client-side details that separate a demo from a product.
Performance
- Run inference only on user action or when a chat message arrives, never in the background.
- Warm the model when the screen opens, not at app start.
- Cap local generation length; long outputs drain battery.
Accessibility and localisation
- AI-generated text is labelled as such where it matters.
- Smart reply chips are reachable and readable by screen readers.
- Users can turn on-device AI features off completely.
5.4Senior versus principal
The same design, answered at two levels. Use the rubric to score yourself.
A senior answer picks a runtime and runs a model on the phone.
A principal answer designs a router with capability checks, remote rules and fallbacks, explains quantisation and fragmentation, makes privacy a design input, and decides per feature with clear criteria.
Score yourself
5.5Whiteboard
Draw the design here, in this order. It stays in this browser.
- Feature → AI router decision box
- Three paths: system model, bundled model, cloud
- Model downloader and CDN
- Remote config feeding routing rules
- Telemetry by device class