Basics
What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.
1.1Definition
The one-paragraph summary, then where its responsibility starts and stops.
LLM features behave like ordinary software with one unreliable, slow, expensive dependency. The patterns from payments or search apply: a gateway, contracts, retries, fallbacks, caching, budgets and observability.
- One internal gateway in front of all model providers
- Versioned prompt templates with owners
- Structured outputs validated against a schema
- Streaming to the client with cancel
- Fallback models and graceful degradation
- Per-feature budgets and per-user limits
- Calling providers directly from mobile apps with embedded keys
- Free-text parsing of model output with regex
- Unbounded retries on a slow provider
- Logging full prompts with personal data
- Shipping prompt changes without an evaluation run
1.2Adding AI to an existing app
A concrete example first, so the mechanics have something to attach to.
A product team wants to add three features in a quarter: listing description rewrite, review summaries and a support assistant. Instead of each team calling providers directly, the platform team ships an LLM gateway in two weeks.
The gateway authenticates internal callers, looks up the prompt template version for the feature, routes to the configured model, enforces JSON schemas on output, retries once on transient errors, falls back to a second provider, and records tokens and cost per feature.
Review summaries run nightly through the provider’s batch API at a lower price. Description rewrites stream to the mobile app over server-sent events. The support assistant uses tools through the same gateway.
When one provider has an outage, traffic moves to the fallback automatically. When cost spikes, the dashboard shows which feature and prompt version caused it.
Question
Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.
2.1Target questions
Keep them in mind while you read the detail.
- Why put a gateway in front of LLM providers?
- How do you get reliable structured data from a model?
- How do you add an LLM to a mobile app safely?
Model answers are in part 4, after the detail that supports them.
2.2Interview rounds
System design rounds where this component is part of a strong answer.
Detailed answer
How it actually works: the components, the states it moves through, a full run, and the code.
3.1An LLM gateway
The parts and how they connect.
- Gateway
- One service owns provider keys, routing, retries, fallbacks, quotas and metering. Teams call a stable internal API instead of provider SDKs.
- Prompt registry
- Templates stored with versions and owners; features reference a version, and rollbacks are configuration.
- Structured output
- Ask for JSON that matches a schema, using the provider’s schema or tool features where available, then validate. Invalid output triggers one repair attempt, then a fallback.
- Resilience
- Timeouts sized to the feature, one retry with jitter for transient errors, circuit breakers per provider, and a fallback model or non-AI path.
- Metering
- Tokens, latency and cost per feature, prompt version and model; daily budgets with alerts.
3.2Patterns at a glance
| Name | Meaning |
|---|---|
| Streaming | Server-sent events from gateway to client; render in small batches; support cancel. |
| Structured outputs | JSON schema in the request, validation on the response. |
| Fallback | Second provider or smaller model; then a non-AI experience. |
| Batch processing | Offline jobs through batch APIs at lower cost. |
| Prompt caching | Stable prefixes reused across calls for lower cost and latency. |
3.3A resilient LLM call
One run, step by step.
- Retries stop at a limit, and fallback runs automatically.
- The caller receives the same schema whichever provider answered.
3.4Code
Working code for the core of it.
Gateway request contract
1POST /v1/llm/generate2{ "feature": "review_summary", "template_version": "v7",3 "input": { "reviews": ["..."], "language": "en" },4 "output_schema": { "type": "object", "required": ["summary", "pros", "cons"],5 "properties": { "summary": { "type": "string", "maxLength": 400 },6 "pros": { "type": "array", "items": { "type": "string" } },7 "cons": { "type": "array", "items": { "type": "string" } } } },8 "stream": false, "user_id": "u-91" }Validate and repair once (TypeScript)
1const out = await callModel(req);2let parsed = safeJson(out.text);3if (!schema.validate(parsed)) {4 const fix = await callModel({ ...req, messages: [...req.messages,5 { role: 'user', content: 'Return valid JSON matching the schema. Errors: ' + schema.errors() }] });6 parsed = safeJson(fix.text);7 if (!schema.validate(parsed)) return fallback(req.feature); // non-AI path or cached result8}9return parsed;3.5Tested code, explained
A working implementation, then what each part does and why.
Assembling a prompt
A prompt is a program with a structure. Keep it in code, so you can test and version it.
1export interface Example {2 input: string;3 output: string;4}5 6/** Assembles a few-shot prompt: rules first, examples next, the real input last. */7export function buildPrompt(opts: { task: string; examples?: Example[]; schema?: object; reason?: boolean; input: string }): string {8 const parts = [opts.task];9 if (opts.reason) parts.push("Work through the problem step by step, then give the final answer.");10 if (opts.schema) parts.push(`Reply with JSON matching this schema:\n${JSON.stringify(opts.schema)}`);11 for (const e of opts.examples ?? []) parts.push(`Input: ${e.input}\nOutput: ${e.output}`);12 parts.push(`Input: ${opts.input}\nOutput:`);13 return parts.join("\n\n");14}parts = [opts.task]- Rules first.
opts.schema- States the output format, which later gets checked by the guard.
Input: ... Output:- The real input goes last, in the same shape as the examples.
Tested by npm run test:modules.
Fitting context into a budget
Retrieval usually returns more than fits. This keeps the best sentences, in the order they were written.
1export interface Sentence {2 text: string;3 score: number;4}5 6/** Keeps the highest-scoring sentences that fit the budget, then restores reading order. */7export function fitContext(sentences: Sentence[], budget: number, count: (s: string) => number): string[] {8 const ranked = sentences.map((s, i) => ({ ...s, i })).sort((a, b) => b.score - a.score);9 const kept: { text: string; i: number }[] = [];10 let used = 0;11 for (const s of ranked) {12 const cost = count(s.text);13 if (used + cost > budget) continue;14 used += cost;15 kept.push(s);16 }17 return kept.sort((a, b) => a.i - b.i).map((s) => s.text);18}sort((a, b) => b.score - a.score)- Picks by relevance.
continue- Skips one that doesn't fit and tries smaller ones, instead of stopping.
sort((a, b) => a.i - b.i)- Restores reading order, because shuffled sentences confuse a model.
Tested by npm run test:modules.
Never trust model output
Treat a model's reply as untrusted input. Validate it, and re-ask with the exact errors.
1export type Field = { type: "string" | "number" | "boolean"; required?: boolean; oneOf?: string[]; min?: number; max?: number };2export type Schema = Record<string, Field>;3 4export function validate(value: unknown, schema: Schema): string[] {5 if (typeof value !== "object" || value === null) return ["expected a JSON object"];6 const obj = value as Record<string, unknown>;7 const errors: string[] = [];8 for (const [key, f] of Object.entries(schema)) {9 const v = obj[key];10 if (v === undefined) {11 if (f.required) errors.push(`${key} is required`);12 continue;13 }14 if (typeof v !== f.type) errors.push(`${key} must be a ${f.type}`);15 else if (f.oneOf && !f.oneOf.includes(v as string)) errors.push(`${key} must be one of ${f.oneOf.join(", ")}`);16 else if (typeof v === "number" && ((f.min !== undefined && v < f.min) || (f.max !== undefined && v > f.max))) errors.push(`${key} is out of range`);17 }18 return errors;19}20 21/** Validate, then re-ask with the exact errors. Never pass unvalidated output downstream. */22export async function guardedCall<T>(prompt: string, llm: (p: string) => Promise<string>, schema: Schema, maxRetries = 2) {23 let ask = prompt;24 let errors: string[] = [];25 for (let attempt = 1; attempt <= maxRetries + 1; attempt++) {26 const raw = await llm(ask);27 try {28 const parsed = JSON.parse(raw) as T;29 errors = validate(parsed, schema);30 if (errors.length === 0) return { ok: true as const, value: parsed, attempts: attempt };31 } catch {32 errors = ["reply was not valid JSON"];33 }34 ask = `${prompt}\n\nYour last reply failed validation:\n- ${errors.join("\n- ")}\nReply with corrected JSON only.`;35 }36 return { ok: false as const, errors, attempts: maxRetries + 1 };37}validate(parsed, schema)- Checks type, range and allowed values, including for tool arguments.
ask = `${prompt}...`- Sends the exact errors back, which fixes most replies in one retry.
ok: false- After the retry limit, return a failure. Never pass bad output downstream.
Tested by npm run test:modules.
3.6Client-side concerns
Mobile apps
- Never embed provider keys; call your gateway with the user’s session.
- Stream with SSE, render in batches, allow cancel and retry.
- Cache results in SQLite; show something useful offline.
Web
- Same gateway; stream with fetch and ReadableStream.
- Disable submit while streaming to avoid duplicate calls.
Privacy
- Redact personal data before logging; set provider data-retention options.
- Region-pin traffic where residency rules apply.
Q&A
The questions from part 2, now with model answers. Try each one out loud before opening it.
4.1Interview questions
Why put a gateway in front of LLM providers?
How do you get reliable structured data from a model?
How do you add an LLM to a mobile app safely?
Extra depth
Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.
5.1Building blocks
Show detailsHide details
| Need | Options | When to pick |
|---|---|---|
| Gateways | In-house service · open-source LLM proxies · cloud AI gateways | In-house when you need custom policy; a proxy to start quickly. |
| Schemas | JSON Schema · Zod · Pydantic | Define once, use for prompts, validation and types. |
| Observability | OpenTelemetry · LLM tracing tools (e.g. Langfuse, LangSmith) | Trace prompts, outputs, tokens and cost per feature. |
| Cost controls | Batch APIs · prompt caching · routing · output caps | Combine; each alone is partial. |
5.2Operational metrics
Show detailsHide details
| Metric | Why it matters |
|---|---|
| Error rate and fallback rate per provider | Provider health and resilience. |
| Schema validation failure rate | Prompt or model regressions. |
| Cost per feature per day | Budget control. |
| p95 latency per feature | User experience. |
5.3Production failures
Show detailsHide details
| Case | How it is handled |
|---|---|
| Provider outage | Circuit breaker, fallback provider, then a non-AI path. |
| Model update changes behaviour | Pin model versions; re-run evaluations before switching. |
| Runaway cost from a loop | Per-feature budgets and per-request token caps. |
| Personal data in logs | Redaction at the gateway; restricted access. |