Basics
Start with the product and the ideas it depends on. If these are clear, the design that follows will feel obvious rather than memorised.
1.1Product in plain words
Who uses it, what a normal day looks like, and why it is harder than it seems.
A shared platform so every team building AI features can test them before release, guard them in production, and see when they go wrong, much like unit tests, CI and monitoring do for normal code.
Who uses it
- Feature teams: change prompts weekly and need to know if they made things worse.
- Reviewers: label samples of AI output for quality.
- Leadership: needs confidence that AI features won’t embarrass the company.
A day in the life
- A product manager edits the support assistant’s prompt.
- CI runs 600 test conversations and scores them automatically.
- Refund accuracy drops; the change is blocked with the failing examples shown.
- After a fix, the change ships to 5% of users and is monitored.
- A strange answer reported by a user becomes a new test case the same day.
Why it is hard
‘Good answer’ is harder to test than ‘returns 200’.
A provider update can change behaviour without any code change.
You need many examples and real traffic to see them.
A platform nobody adopts protects nothing.
Like a restaurant’s tasting routine: every recipe change is tasted against a checklist before it reaches customers, inspectors sample dishes during service, and every complaint becomes a new item on the checklist.
1.2Key concepts
The building blocks this design uses. Each one appears again in the answer.
A fixed collection of realistic inputs with the properties a good answer must have. Every prompt or model change is scored against it before release.
Exact checks (format, citations, banned words) are cheap and reliable; an LLM judge scores fuzzier qualities; people review samples to keep the judge honest.
Runtime checks on input and output that catch personal data, unsafe topics or prompt injection, and block or rewrite before the user sees anything.
A change ships only if its eval scores stay at or above the baseline, like a test suite for prompts.
In production, track thumbs-down, escalations, refusals, cost and latency per feature, and sample conversations for review.
One service for evals, guardrails and monitoring that every team plugs into, instead of each team building its own.
Learn them properly in the deep dives:
Question
Now read the question closely. Agree on what you are building, ask what is unclear, and work out the numbers before you draw anything.
2.1Brief and scope
The question exactly as the interviewer asks it, and what they want to learn from your answer.
Three teams are shipping LLM features: a support assistant, ticket summarisation, and marketing copy generation. Each has had embarrassing failures in production. As principal engineer, design the shared platform for evaluating, guarding and monitoring LLM features.
What it tests: Prompts and models treated as code: tests, release gates, observability, and a platform other teams adopt.
The answer in one sentence. Versioned prompts, layered automatic scoring calibrated against humans, CI gates and canaries, guardrails on every call, and a loop from incidents to test cases.
How to spend the 40 minutes
- Clarify~4 minAsk questions, agree on scope and write down assumptions.
- Estimate~4 minRough numbers for users, data and traffic, and what they imply.
- High-level design~10 minComponents, data flow and the key API on the board.
- Deep dive~14 minThe two or three hardest parts, with real mechanisms.
- Trade-offs and wrap-up~8 minDecisions, failure modes, and what you would do next.
2.2Clarify the problem
The first five minutes. Each answer removes a guess from your design.
| You ask | Interviewer answers |
|---|---|
| What failures happened? | Made-up refund amounts, a summary that leaked a phone number, and off-brand copy. |
| Who changes prompts? | Engineers and product managers; changes go out weekly. |
| Which models? | Two hosted providers and one self-hosted model. |
| Human review capacity? | About 500 reviewed samples per week across teams. |
2.3Estimate the scale
Say the arithmetic out loud. What matters is the order of magnitude and what it forces you to do.
| Metric | Estimate | How |
|---|---|---|
| LLM calls per day | ≈ 3M | three features |
| Evaluation set per feature | 300–1,000 cases | grown from real failures |
| Online sampling for review | ≈ 0.02% | 500 human reviews per week |
| Release gate run | < 15 minutes | in CI on every prompt change |
Detailed answer
The design itself, from the big picture down to the code: components first, then one request end to end, then the parts interviewers push hardest on.
3.1Architecture
Prompts, models and settings are versioned artifacts. Every change runs an evaluation suite in CI; every production call passes guardrails and is traced; a sample flows to human review, and failures become new test cases.
- Prompt registry
- A registry stores prompts, model names, temperatures and tool lists as versioned artifacts with owners. Features reference a version, so a rollback is a config change.
- Eval sets
- Hand-written and production-derived cases with expectations: exact fields that must match, facts that must appear, things that must never appear. Every incident adds cases.
- Scorers
- Cheap deterministic checks first (JSON schema, regex for phone numbers, numbers match the source), then an LLM judge with a rubric for qualities like tone and helpfulness.
- Judge calibration
- Trust the LLM judge only as far as it agrees with human labels on a shared sample, and track agreement like any other metric.
- Guardrails
- Run on every production call: input checks for injection and disallowed requests, output checks for personal data, policy, schema and grounding. A failed check returns a safe fallback.
- Tracing and monitors
- Each call records prompt version, inputs, outputs, latency, tokens and cost. Monitors watch guardrail trigger rates, feedback, and output drift by version.
3.2Key flow, step by step
A product manager edits the support assistant’s prompt.
- Treat a prompt change like a code change: it cannot ship if a gate fails.
- Production rollout is gradual, with the same metrics compared between versions.
- Any failure found later becomes a permanent test case.
3.3Front end
The platform succeeds or fails on its tools. Reviewers need a fast labelling console, prompt authors need a diff-and-test editor, and teams need dashboards they actually open.
Screen map
- Prompt editor
- Templates with variables, a side-by-side diff against the live version, and a button to run the evaluation suite on the draft.
- Compare
- Case-by-case results between two versions, sorted by biggest regression, with the input, both outputs and every scorer’s verdict.
- Label view
- One output at a time with the rubric beside it, keyboard shortcuts for each score, and the source context with grounded spans highlighted.
- Trace viewer
- A production call end to end: prompt version, retrieved context, tool calls, guardrail results, tokens, cost and latency.
Client layers
| Layer | Technology | Responsibility |
|---|---|---|
| UI | React + TypeScript | Editor, review, dashboards |
| Server state | TanStack Query | Runs, cases, labels with cache and optimistic updates |
| Editor | Monaco or CodeMirror | Prompt templates with syntax and diff |
| Charts | A charting library | Quality and cost over time by version |
| Keyboard layer | Global shortcuts | Fast labelling without the mouse |
Every state the user can see
| State | What the user sees | How it works |
|---|---|---|
| Running | Progress ‘412 / 600 cases’ | Eval job streaming results |
| Gate failed | Red banner with top regressions | Threshold not met |
| Gate passed | Green with summary | All thresholds met |
| Needs review | Count badge on the queue | Sampled outputs awaiting labels |
| Judge disagreement | Flag on cases where judge and humans differ | Calibration check |
Code
Fast labelling with keyboard shortcuts (React)
1export function LabelView({ item, onLabel }: { item: ReviewItem; onLabel: (l: Label) => void }) {2 useEffect(() => {3 const onKey = (e: KeyboardEvent) => {4 if (e.target instanceof HTMLTextAreaElement) return; // don't hijack typing5 const map: Record<string, Label> = { '1': 'good', '2': 'minor_issue', '3': 'wrong', '4': 'unsafe' };6 if (map[e.key]) { e.preventDefault(); onLabel(map[e.key]); }7 };8 window.addEventListener('keydown', onKey);9 return () => window.removeEventListener('keydown', onKey);10 }, [item.id, onLabel]);11 12 return (13 <div className="label-view">14 <Context text={item.context} highlights={item.groundedSpans} />15 <Output text={item.output} />16 <Rubric hints={['1 good', '2 minor issue', '3 wrong', '4 unsafe']} />17 </div>18 );19}Optimistic label save and prefetch next (TanStack Query)
1const label = useMutation({2 mutationFn: (l: Label) => api.saveLabel(item.id, l),3 onMutate: () => queryClient.prefetchQuery({ queryKey: ['review', nextId], queryFn: () => api.item(nextId) }),4 onSuccess: () => router.push('/review/' + nextId), // next item is already loaded5});3.4Data model and APIs
The tables and contracts that make the design concrete.
Eval case
1id: refund-uae-annual-072feature: support-assistant3input: "Can I get my money back? I paid yearly in Dubai last month."4context_docs: [doc-311#4]5expect:6 must_include_fact: "pro-rated refund within 30 days"7 must_not_include: ["100% refund", "phone number"]8 tool_calls: [{ name: lookup_plan, args: { region: AE } }]9judge_rubric: [grounded, polite, no_promises_outside_policy]Guardrail chain (TypeScript)
1const checks = [schemaValid, noPII, numbersGroundedIn(context), policyClassifier];2for (const check of checks) {3 const r = await check(output, ctx);4 if (!r.ok) { trace.flag(check.name, r.reason); return safeFallback(feature); }5}6return output;3.5Tested code, explained
The core of this design as working code, with what each part does and why.
Scoring retrieval and answers
Together these separate a retrieval failure from a generation failure.
1const mean = (xs: number[]) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : 0);2const cosine = (a: number[], b: number[]) => a.reduce((s, x, i) => s + x * b[i], 0) / (Math.hypot(...a) * Math.hypot(...b));3 4/** Faithfulness: share of the answer's claims that the retrieved context supports. */5export const faithfulness = (supported: boolean[]) => mean(supported.map(Number));6 7/** Context recall: share of the reference answer's claims that the context can attribute. */8export const contextRecall = (attributed: boolean[]) => mean(attributed.map(Number));9 10/** Context precision: mean precision@k over the ranks that hold a relevant chunk. */11export function contextPrecision(relevantAtRank: boolean[]): number {12 let hits = 0;13 const precisions: number[] = [];14 relevantAtRank.forEach((relevant, i) => {15 if (relevant) precisions.push(++hits / (i + 1));16 });17 return mean(precisions);18}19 20/** Answer relevance: questions regenerated from the answer should resemble the real question. */21export const answerRelevance = (question: number[], generated: number[][]) => mean(generated.map((g) => cosine(question, g)));faithfulness- Did the answer stick to the context?
contextRecall- Did retrieval find what was needed?
contextPrecision- Were the useful chunks ranked high?
answerRelevance- Does the answer address the question?
Tested by npm run test:modules.
Checking an LLM judge
A judge you haven't measured is an opinion. These two functions make it a measurement.
1/** Cohen's kappa: agreement between two raters, beyond what chance would produce. */2export function cohenKappa(a: string[], b: string[]): number {3 const n = a.length;4 const labels = new Set([...a, ...b]);5 const observed = a.filter((x, i) => x === b[i]).length / n;6 let chance = 0;7 for (const l of labels) chance += (a.filter((x) => x === l).length / n) * (b.filter((x) => x === l).length / n);8 return chance === 1 ? 1 : (observed - chance) / (1 - chance);9}10 11export type Verdict = "first" | "second" | "tie";12 13/** Removes position bias: ask twice with the answers swapped; a verdict counts only if both agree. */14export async function debiasedVerdict(judge: (x: string, y: string) => Promise<Verdict>, a: string, b: string): Promise<"a" | "b" | "tie"> {15 const [one, two] = await Promise.all([judge(a, b), judge(b, a)]);16 if (one === "first" && two === "second") return "a";17 if (one === "second" && two === "first") return "b";18 return "tie";19}cohenKappa- Agreement with human labels, beyond what chance would give.
Promise.all([judge(a, b), judge(b, a)])- Judges favour one position, so ask both ways and count only agreement.
Tested by npm run test:modules.
Red-teaming as a test suite
Attacks found once should fail in CI forever after.
1export interface Probe {2 category: string;3 prompt: string;4}5 6/** Runs each probe and its variants against the target; a judge scores every reply. */7export async function runSuite(8 probes: Probe[],9 mutate: (prompt: string) => string[],10 target: (prompt: string) => Promise<string>,11 judge: (category: string, reply: string) => Promise<boolean>12) {13 const perCategory = new Map<string, { attempts: number; successes: number }>();14 const failures: { category: string; prompt: string }[] = [];15 for (const probe of probes) {16 const stats = perCategory.get(probe.category) ?? { attempts: 0, successes: 0 };17 perCategory.set(probe.category, stats);18 for (const prompt of [probe.prompt, ...mutate(probe.prompt)]) {19 stats.attempts++;20 if (await judge(probe.category, await target(prompt))) {21 stats.successes++;22 failures.push({ category: probe.category, prompt }); // keep it: tomorrow's regression test23 }24 }25 }26 const attackSuccessRate = new Map([...perCategory].map(([c, s]) => [c, s.successes / s.attempts]));27 return { attackSuccessRate, failures };28}29 30/** The CI gate: no category may get worse than its baseline. */31export function passesGate(rates: Map<string, number>, baseline: Record<string, number>, margin = 0): boolean {32 return [...rates].every(([category, rate]) => rate <= (baseline[category] ?? 0) + margin);33}mutate(probe.prompt)- Each attack runs in many variations, since models that block one phrasing often miss another.
failures.push(...)- Each success becomes tomorrow's regression test.
passesGate- The build fails if any category gets worse than its baseline.
Tested by npm run test:modules.
3.6Hard parts, in depth
Where interviewers spend the second half of the round.
Make the cheap checks do most of the work
Deterministic checks are fast, free and never disagree with themselves. Rules catch most of the incidents listed, such as invented refund amounts and leaked phone numbers: numbers must appear in the source, and phone patterns get redacted.
Use LLM judges for qualities rules can’t capture, and only after measuring their agreement with humans.
Online signals
Explicit thumbs are sparse. Implicit signals such as edits to generated text, copy rates, escalations to a human and repeated questions are richer. Compare them between prompt versions in an A/B test before full rollout.
Making it a platform
Teams adopt a platform when it saves them work. Provide an SDK that adds tracing and guardrails with one wrapper, templates for eval sets, and a dashboard that shows each team its own quality and cost. Make the gate mandatory only after it is easy.
Q&A
The questions that come after the design, with model answers. Answer each one out loud before opening it.
4.1Follow-up questions
What the interviewer asks once the main design is on the board.
How do you evaluate an agent that takes multiple steps?
How do you switch model providers safely?
What’s your definition of ‘done’ for an LLM feature?
4.2Front-end follow-ups
Questions aimed at the client: rendering, state and devices.
How do you make reviewers fast and consistent?
What does a prompt author see when a gate fails?
4.3Incidents, probes and traps
Realistic production incidents and pointed probes, grouped by topic.
The product 1
ProbeWhy not rely on user thumbs-up and thumbs-down?
Requirements 1
TrapWhat do you ask first?
Scale 1
ProbeAre 500 human reviews a week enough?
Architecture 2
IncidentA provider silently updated a model alias; summaries got longer and broke a UI.
ProbeWhy run rule-based checks before an LLM judge?
The key flow 1
IncidentA prompt change passed CI and still failed in production.
Front end 2
IncidentReviewers disagree 40% of the time.
ProbeWhat makes a review tool fast?
Data and APIs 1
TrapWhy store the prompt version on every trace?
The hard parts 2
IncidentJudge scores improved while user complaints rose.
ProbeHow do you evaluate an agent?
Trade-offs 1
ProbeWhy one shared platform rather than each team building its own?
Failures 2
IncidentTraces containing customer data went to a third-party tracing tool.
IncidentAfter a policy update, guardrails blocked 15% of legitimate answers.
Extra depth
What turns a good answer into a strong one: the decisions you rejected, how the system fails, and the signals of seniority.
5.1Trade-offs and decisions
Every choice has a cost. Naming it is what interviewers listen for.
Show detailsHide details
| Decision | Chosen | Rejected | Why |
|---|---|---|---|
| Scoring | Rules first, LLM judge second | LLM judge for everything | Cheaper, deterministic, and judges drift. |
| Release process | CI gate + canary | Manual spot checks | Repeatable and catches regressions before users do. |
| Guardrail placement | Shared service and SDK | Each team builds its own | Consistency, shared improvements, central policy updates. |
| Human review | Targeted sample | Random sample only | Reviews focus on flagged and low-confidence outputs where they matter most. |
5.2Failure modes
What breaks in production, and how the design keeps working when it does.
Show detailsHide details
| What breaks | How the design copes |
|---|---|
| Judge model changes behaviour | Pin judge versions; re-run calibration when upgrading. |
| Eval set no longer reflects users | Refresh monthly from production traces, stratified by intent. |
| Guardrails too strict, blocking good answers | Track false-positive rate from reviews; tune thresholds per feature. |
| Logs contain personal data | Redact at trace time; restricted access and retention limits. |
5.3Performance and accessibility
The client-side details that separate a demo from a product.
Performance
- Prefetch the next review item so labelling never waits.
- Virtualise long case lists in eval runs.
- Stream eval results as they finish instead of waiting for the whole suite.
Accessibility and localisation
- All review actions have keyboard shortcuts and visible focus.
- Diffs use symbols as well as colour for added and removed text.
5.4Senior versus principal
The same design, answered at two levels. Use the rubric to score yourself.
A senior answer adds a moderation API and logs outputs.
A principal answer versions prompts as code, builds layered scoring with calibrated judges, gates releases in CI, monitors online signals by version, closes the loop from incidents to test cases, and designs the platform so teams want to use it.
Score yourself
5.5Whiteboard
Draw the design here, in this order. It stays in this browser.
- Prompt registry → CI eval gate → deploy
- Scorers: rules then judge, calibrated by humans
- Production call through guardrails with fallback
- Tracing → monitors → review sample
- Loop from review back into eval sets