Shipping LLM features safely: evals, guardrails, monitoring

Three teams are shipping LLM features: a support assistant, ticket summarisation, and marketing copy generation. Each has had embarrassing …

Ready40:00
ClarifyEstimateHigh-level designDeep diveTrade-offs and wrap-up
Start the clockReset Practice mode
Part 1 of 5

Basics

Start with the product and the ideas it depends on. If these are clear, the design that follows will feel obvious rather than memorised.

1.1Product in plain words

Who uses it, what a normal day looks like, and why it is harder than it seems.

A shared platform so every team building AI features can test them before release, guard them in production, and see when they go wrong, much like unit tests, CI and monitoring do for normal code.

Who uses it

  • Feature teams: change prompts weekly and need to know if they made things worse.
  • Reviewers: label samples of AI output for quality.
  • Leadership: needs confidence that AI features won’t embarrass the company.

A day in the life

  1. A product manager edits the support assistant’s prompt.
  2. CI runs 600 test conversations and scores them automatically.
  3. Refund accuracy drops; the change is blocked with the failing examples shown.
  4. After a fix, the change ships to 5% of users and is monitored.
  5. A strange answer reported by a user becomes a new test case the same day.

Why it is hard

Quality is fuzzy

‘Good answer’ is harder to test than ‘returns 200’.

Models change

A provider update can change behaviour without any code change.

Failures are rare and varied

You need many examples and real traffic to see them.

Teams must want it

A platform nobody adopts protects nothing.

Think of it like this

Like a restaurant’s tasting routine: every recipe change is tasted against a checklist before it reaches customers, inspectors sample dishes during service, and every complaint becomes a new item on the checklist.

1.2Key concepts

The building blocks this design uses. Each one appears again in the answer.

01Evaluation set

A fixed collection of realistic inputs with the properties a good answer must have. Every prompt or model change is scored against it before release.

02Scoring methods

Exact checks (format, citations, banned words) are cheap and reliable; an LLM judge scores fuzzier qualities; people review samples to keep the judge honest.

03Guardrails

Runtime checks on input and output that catch personal data, unsafe topics or prompt injection, and block or rewrite before the user sees anything.

04Release gate

A change ships only if its eval scores stay at or above the baseline, like a test suite for prompts.

05Online monitoring

In production, track thumbs-down, escalations, refusals, cost and latency per feature, and sample conversations for review.

06Shared platform

One service for evals, guardrails and monitoring that every team plugs into, instead of each team building its own.

Learn them properly in the deep dives:

Part 2 of 5

Question

Now read the question closely. Agree on what you are building, ask what is unclear, and work out the numbers before you draw anything.

2.1Brief and scope

The question exactly as the interviewer asks it, and what they want to learn from your answer.

Interviewer

Three teams are shipping LLM features: a support assistant, ticket summarisation, and marketing copy generation. Each has had embarrassing failures in production. As principal engineer, design the shared platform for evaluating, guarding and monitoring LLM features.

What it tests: Prompts and models treated as code: tests, release gates, observability, and a platform other teams adopt.

The answer in one sentence. Versioned prompts, layered automatic scoring calibrated against humans, CI gates and canaries, guardrails on every call, and a loop from incidents to test cases.

How to spend the 40 minutes

  1. Clarify~4 minAsk questions, agree on scope and write down assumptions.
  2. Estimate~4 minRough numbers for users, data and traffic, and what they imply.
  3. High-level design~10 minComponents, data flow and the key API on the board.
  4. Deep dive~14 minThe two or three hardest parts, with real mechanisms.
  5. Trade-offs and wrap-up~8 minDecisions, failure modes, and what you would do next.

2.2Clarify the problem

The first five minutes. Each answer removes a guess from your design.

Your turn: Before drawing anything, list the questions you would ask the interviewer.Reveal model answer
You askInterviewer answers
What failures happened?Made-up refund amounts, a summary that leaked a phone number, and off-brand copy.
Who changes prompts?Engineers and product managers; changes go out weekly.
Which models?Two hosted providers and one self-hosted model.
Human review capacity?About 500 reviewed samples per week across teams.

2.3Estimate the scale

Say the arithmetic out loud. What matters is the order of magnitude and what it forces you to do.

Your turn: Write down the numbers that will drive the design.Reveal model answer
MetricEstimateHow
LLM calls per day≈ 3Mthree features
Evaluation set per feature300–1,000 casesgrown from real failures
Online sampling for review≈ 0.02%500 human reviews per week
Release gate run< 15 minutesin CI on every prompt change
Part 3 of 5

Detailed answer

The design itself, from the big picture down to the code: components first, then one request end to end, then the parts interviewers push hardest on.

3.1Architecture

Prompts, models and settings are versioned artifacts. Every change runs an evaluation suite in CI; every production call passes guardrails and is traced; a sample flows to human review, and failures become new test cases.

Your turn: Sketch the components and arrows, then compare.Reveal model answer
BUILD TIMERUN TIMEPrompt registryversioned templatesEval runnerCI gateEval setscases +expectationsScorersrules · LLM judgeHuman reviewlabels · rubricsJudge calibrationagreement withhumansFeature serviceuses prompt v12GuardrailsPII · policy ·schemaTracingprompt · output ·costMonitorsdrift · rates ·alertsExperimentsprompt A/BUser feedbackthumbs · editstrustdeploysamplenew cases
ClientServiceData storeAI component
Reference design. Boxes show components; arrows show who calls whom.
Prompt registry
A registry stores prompts, model names, temperatures and tool lists as versioned artifacts with owners. Features reference a version, so a rollback is a config change.
Eval sets
Hand-written and production-derived cases with expectations: exact fields that must match, facts that must appear, things that must never appear. Every incident adds cases.
Scorers
Cheap deterministic checks first (JSON schema, regex for phone numbers, numbers match the source), then an LLM judge with a rubric for qualities like tone and helpfulness.
Judge calibration
Trust the LLM judge only as far as it agrees with human labels on a shared sample, and track agreement like any other metric.
Guardrails
Run on every production call: input checks for injection and disallowed requests, output checks for personal data, policy, schema and grounding. A failed check returns a safe fallback.
Tracing and monitors
Each call records prompt version, inputs, outputs, latency, tokens and cost. Monitors watch guardrail trigger rates, feedback, and output drift by version.

3.2Key flow, step by step

A product manager edits the support assistant’s prompt.

Your turn: Narrate one request end to end.Reveal model answer
PMRegistryCI evalProductionMonitoringprompt v13 (draft)run suite: 600 casesrules + judge scoringrefund accuracy 99.5% → 97.1% ✗fix, v14run suiteall gates passcanary 5%traces, guardrail ratespromote to 100%
Solid arrows are requests; dashed arrows are responses.
  1. Treat a prompt change like a code change: it cannot ship if a gate fails.
  2. Production rollout is gradual, with the same metrics compared between versions.
  3. Any failure found later becomes a permanent test case.

3.3Front end

The platform succeeds or fails on its tools. Reviewers need a fast labelling console, prompt authors need a diff-and-test editor, and teams need dashboards they actually open.

Your turn: Sketch the screens, the client layers and every UI state the user can see.Reveal model answer

Screen map

Prompt editordiff · variablesEval runscores · failuresComparev12 vs v13Promotecanary · rolloutReview queuesampled outputsLabel viewrubric · shortcutsQuality dashboardby feature ·versionTrace viewerprompt → outputrun suitegates passdrill down
Client
Screens and how users move between them.
Prompt editor
Templates with variables, a side-by-side diff against the live version, and a button to run the evaluation suite on the draft.
Compare
Case-by-case results between two versions, sorted by biggest regression, with the input, both outputs and every scorer’s verdict.
Label view
One output at a time with the rubric beside it, keyboard shortcuts for each score, and the source context with grounded spans highlighted.
Trace viewer
A production call end to end: prompt version, retrieved context, tool calls, guardrail results, tokens, cost and latency.

Client layers

LayerTechnologyResponsibility
UIReact + TypeScriptEditor, review, dashboards
Server stateTanStack QueryRuns, cases, labels with cache and optimistic updates
EditorMonaco or CodeMirrorPrompt templates with syntax and diff
ChartsA charting libraryQuality and cost over time by version
Keyboard layerGlobal shortcutsFast labelling without the mouse

Every state the user can see

StateWhat the user seesHow it works
RunningProgress ‘412 / 600 cases’Eval job streaming results
Gate failedRed banner with top regressionsThreshold not met
Gate passedGreen with summaryAll thresholds met
Needs reviewCount badge on the queueSampled outputs awaiting labels
Judge disagreementFlag on cases where judge and humans differCalibration check

Code

Fast labelling with keyboard shortcuts (React)

fast-labelling-keyboard.tsxTypeScript JSXCopy
1export function LabelView({ item, onLabel }: { item: ReviewItem; onLabel: (l: Label) => void }) {2  useEffect(() => {3    const onKey = (e: KeyboardEvent) => {4      if (e.target instanceof HTMLTextAreaElement) return;            // don't hijack typing5      const map: Record<string, Label> = { '1': 'good', '2': 'minor_issue', '3': 'wrong', '4': 'unsafe' };6      if (map[e.key]) { e.preventDefault(); onLabel(map[e.key]); }7    };8    window.addEventListener('keydown', onKey);9    return () => window.removeEventListener('keydown', onKey);10  }, [item.id, onLabel]);11 12  return (13    <div className="label-view">14      <Context text={item.context} highlights={item.groundedSpans} />15      <Output text={item.output} />16      <Rubric hints={['1 good', '2 minor issue', '3 wrong', '4 unsafe']} />17    </div>18  );19}
19 linesUTF-8TypeScript JSX

Optimistic label save and prefetch next (TanStack Query)

optimistic-label-save.tsxTypeScript JSXCopy
1const label = useMutation({2  mutationFn: (l: Label) => api.saveLabel(item.id, l),3  onMutate: () => queryClient.prefetchQuery({ queryKey: ['review', nextId], queryFn: () => api.item(nextId) }),4  onSuccess: () => router.push('/review/' + nextId),                    // next item is already loaded5});
5 linesUTF-8TypeScript JSX

3.4Data model and APIs

The tables and contracts that make the design concrete.

Your turn: Write the schema or contract you would put on the board.Reveal model answer

Eval case

eval-case.yamlYAMLCopy
1id: refund-uae-annual-072feature: support-assistant3input: "Can I get my money back? I paid yearly in Dubai last month."4context_docs: [doc-311#4]5expect:6  must_include_fact: "pro-rated refund within 30 days"7  must_not_include: ["100% refund", "phone number"]8  tool_calls: [{ name: lookup_plan, args: { region: AE } }]9judge_rubric: [grounded, polite, no_promises_outside_policy]
9 linesUTF-8YAML

Guardrail chain (TypeScript)

guardrail-chain-typescript.tsTypeScriptCopy
1const checks = [schemaValid, noPII, numbersGroundedIn(context), policyClassifier];2for (const check of checks) {3  const r = await check(output, ctx);4  if (!r.ok) { trace.flag(check.name, r.reason); return safeFallback(feature); }5}6return output;
6 linesUTF-8TypeScript

3.5Tested code, explained

The core of this design as working code, with what each part does and why.

Scoring retrieval and answers

Together these separate a retrieval failure from a generation failure.

ragas-metrics.tsTypeScriptCopy
1const mean = (xs: number[]) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : 0);2const cosine = (a: number[], b: number[]) => a.reduce((s, x, i) => s + x * b[i], 0) / (Math.hypot(...a) * Math.hypot(...b));3 4/** Faithfulness: share of the answer's claims that the retrieved context supports. */5export const faithfulness = (supported: boolean[]) => mean(supported.map(Number));6 7/** Context recall: share of the reference answer's claims that the context can attribute. */8export const contextRecall = (attributed: boolean[]) => mean(attributed.map(Number));9 10/** Context precision: mean precision@k over the ranks that hold a relevant chunk. */11export function contextPrecision(relevantAtRank: boolean[]): number {12  let hits = 0;13  const precisions: number[] = [];14  relevantAtRank.forEach((relevant, i) => {15    if (relevant) precisions.push(++hits / (i + 1));16  });17  return mean(precisions);18}19 20/** Answer relevance: questions regenerated from the answer should resemble the real question. */21export const answerRelevance = (question: number[], generated: number[][]) => mean(generated.map((g) => cosine(question, g)));
21 linesUTF-8TypeScript
faithfulness
Did the answer stick to the context?
contextRecall
Did retrieval find what was needed?
contextPrecision
Were the useful chunks ranked high?
answerRelevance
Does the answer address the question?

Tested by npm run test:modules.

Checking an LLM judge

A judge you haven't measured is an opinion. These two functions make it a measurement.

judge-calibration.tsTypeScriptCopy
1/** Cohen's kappa: agreement between two raters, beyond what chance would produce. */2export function cohenKappa(a: string[], b: string[]): number {3  const n = a.length;4  const labels = new Set([...a, ...b]);5  const observed = a.filter((x, i) => x === b[i]).length / n;6  let chance = 0;7  for (const l of labels) chance += (a.filter((x) => x === l).length / n) * (b.filter((x) => x === l).length / n);8  return chance === 1 ? 1 : (observed - chance) / (1 - chance);9}10 11export type Verdict = "first" | "second" | "tie";12 13/** Removes position bias: ask twice with the answers swapped; a verdict counts only if both agree. */14export async function debiasedVerdict(judge: (x: string, y: string) => Promise<Verdict>, a: string, b: string): Promise<"a" | "b" | "tie"> {15  const [one, two] = await Promise.all([judge(a, b), judge(b, a)]);16  if (one === "first" && two === "second") return "a";17  if (one === "second" && two === "first") return "b";18  return "tie";19}
19 linesUTF-8TypeScript
cohenKappa
Agreement with human labels, beyond what chance would give.
Promise.all([judge(a, b), judge(b, a)])
Judges favour one position, so ask both ways and count only agreement.

Tested by npm run test:modules.

Red-teaming as a test suite

Attacks found once should fail in CI forever after.

red-team-loop.tsTypeScriptCopy
1export interface Probe {2  category: string;3  prompt: string;4}5 6/** Runs each probe and its variants against the target; a judge scores every reply. */7export async function runSuite(8  probes: Probe[],9  mutate: (prompt: string) => string[],10  target: (prompt: string) => Promise<string>,11  judge: (category: string, reply: string) => Promise<boolean>12) {13  const perCategory = new Map<string, { attempts: number; successes: number }>();14  const failures: { category: string; prompt: string }[] = [];15  for (const probe of probes) {16    const stats = perCategory.get(probe.category) ?? { attempts: 0, successes: 0 };17    perCategory.set(probe.category, stats);18    for (const prompt of [probe.prompt, ...mutate(probe.prompt)]) {19      stats.attempts++;20      if (await judge(probe.category, await target(prompt))) {21        stats.successes++;22        failures.push({ category: probe.category, prompt }); // keep it: tomorrow's regression test23      }24    }25  }26  const attackSuccessRate = new Map([...perCategory].map(([c, s]) => [c, s.successes / s.attempts]));27  return { attackSuccessRate, failures };28}29 30/** The CI gate: no category may get worse than its baseline. */31export function passesGate(rates: Map<string, number>, baseline: Record<string, number>, margin = 0): boolean {32  return [...rates].every(([category, rate]) => rate <= (baseline[category] ?? 0) + margin);33}
33 linesUTF-8TypeScript
mutate(probe.prompt)
Each attack runs in many variations, since models that block one phrasing often miss another.
failures.push(...)
Each success becomes tomorrow's regression test.
passesGate
The build fails if any category gets worse than its baseline.

Tested by npm run test:modules.

3.6Hard parts, in depth

Where interviewers spend the second half of the round.

Your turn: Pick two areas the interviewer will push on and explain them.Reveal model answer

Make the cheap checks do most of the work

Deterministic checks are fast, free and never disagree with themselves. Rules catch most of the incidents listed, such as invented refund amounts and leaked phone numbers: numbers must appear in the source, and phone patterns get redacted.

Use LLM judges for qualities rules can’t capture, and only after measuring their agreement with humans.

Online signals

Explicit thumbs are sparse. Implicit signals such as edits to generated text, copy rates, escalations to a human and repeated questions are richer. Compare them between prompt versions in an A/B test before full rollout.

Making it a platform

Teams adopt a platform when it saves them work. Provide an SDK that adds tracing and guardrails with one wrapper, templates for eval sets, and a dashboard that shows each team its own quality and cost. Make the gate mandatory only after it is easy.

Part 4 of 5

Q&A

The questions that come after the design, with model answers. Answer each one out loud before opening it.

4.1Follow-up questions

What the interviewer asks once the main design is on the board.

How do you evaluate an agent that takes multiple steps?
Score the trajectory: correct tool calls, correct arguments, and final state, not just the final message.
How do you switch model providers safely?
Run the full eval suite on the candidate, compare cost and latency, then canary with the same monitors.
What’s your definition of ‘done’ for an LLM feature?
Owned eval set with agreed thresholds, guardrails on, tracing on, rollback tested, and a cost budget with alerts.

4.2Front-end follow-ups

Questions aimed at the client: rendering, state and devices.

How do you make reviewers fast and consistent?
A single focused view with the rubric visible, keyboard shortcuts, highlighted grounding, prefetching, and periodic agreement checks between reviewers.
What does a prompt author see when a gate fails?
The top regressed cases with inputs, both versions’ outputs and which scorer failed, so the fix is obvious.

4.3Incidents, probes and traps

Realistic production incidents and pointed probes, grouped by topic.

The product 1

ProbeWhy not rely on user thumbs-up and thumbs-down?
They are sparse, biased and arrive after users were already harmed. Offline evaluation catches regressions before release.
I can answer this

Requirements 1

TrapWhat do you ask first?
What failures happened and who changes prompts, how often. Those decide which checks matter and where the gate goes.
I can answer this

Scale 1

ProbeAre 500 human reviews a week enough?
Yes, if they are targeted: flagged outputs, low-confidence cases and stratified samples, plus regular calibration of the LLM judge against humans.
I can answer this

Architecture 2

IncidentA provider silently updated a model alias; summaries got longer and broke a UI.
Pin model versions, monitor output length and format, and run the evaluation suite on any provider change.
I can answer this
ProbeWhy run rule-based checks before an LLM judge?
They are cheap, fast and deterministic, and they catch many real incidents such as invented numbers and leaked phone numbers.
I can answer this

The key flow 1

IncidentA prompt change passed CI and still failed in production.
The evaluation set did not represent real traffic. Refresh it from production traces regularly and gate the canary on online metrics too.
I can answer this

Front end 2

IncidentReviewers disagree 40% of the time.
The rubric is unclear. Add worked examples, run calibration sessions and track agreement between reviewers.
I can answer this
ProbeWhat makes a review tool fast?
One focused view, keyboard shortcuts, highlighted context and prefetching the next item.
I can answer this

Data and APIs 1

TrapWhy store the prompt version on every trace?
To attribute quality and cost to versions and roll back precisely.
I can answer this

The hard parts 2

IncidentJudge scores improved while user complaints rose.
The judge drifted or was gamed. Re-calibrate against human labels and weigh online metrics more heavily.
I can answer this
ProbeHow do you evaluate an agent?
Score the trajectory: correct tools, correct arguments and the correct final state, not only the final message.
I can answer this

Trade-offs 1

ProbeWhy one shared platform rather than each team building its own?
Consistent guardrails, shared improvements, central policy updates and one place to see quality and cost.
I can answer this

Failures 2

IncidentTraces containing customer data went to a third-party tracing tool.
Redact at the source, sign proper data agreements and restrict access and retention.
I can answer this
IncidentAfter a policy update, guardrails blocked 15% of legitimate answers.
Too many false positives. Run new guardrails in shadow mode first, tune thresholds per feature and track false-positive rates from reviews.
I can answer this
Part 5 of 5

Extra depth

What turns a good answer into a strong one: the decisions you rejected, how the system fails, and the signals of seniority.

5.1Trade-offs and decisions

Every choice has a cost. Naming it is what interviewers listen for.

Your turn: State each decision, the alternative you rejected and why.Reveal model answer
Show detailsHide details
DecisionChosenRejectedWhy
ScoringRules first, LLM judge secondLLM judge for everythingCheaper, deterministic, and judges drift.
Release processCI gate + canaryManual spot checksRepeatable and catches regressions before users do.
Guardrail placementShared service and SDKEach team builds its ownConsistency, shared improvements, central policy updates.
Human reviewTargeted sampleRandom sample onlyReviews focus on flagged and low-confidence outputs where they matter most.

5.2Failure modes

What breaks in production, and how the design keeps working when it does.

Your turn: List what breaks and how the system degrades.Reveal model answer
Show detailsHide details
What breaksHow the design copes
Judge model changes behaviourPin judge versions; re-run calibration when upgrading.
Eval set no longer reflects usersRefresh monthly from production traces, stratified by intent.
Guardrails too strict, blocking good answersTrack false-positive rate from reviews; tune thresholds per feature.
Logs contain personal dataRedact at trace time; restricted access and retention limits.

5.3Performance and accessibility

The client-side details that separate a demo from a product.

Performance

  • Prefetch the next review item so labelling never waits.
  • Virtualise long case lists in eval runs.
  • Stream eval results as they finish instead of waiting for the whole suite.

Accessibility and localisation

  • All review actions have keyboard shortcuts and visible focus.
  • Diffs use symbols as well as colour for added and removed text.

5.4Senior versus principal

The same design, answered at two levels. Use the rubric to score yourself.

Senior answer

A senior answer adds a moderation API and logs outputs.

Principal answer

A principal answer versions prompts as code, builds layered scoring with calibrated judges, gates releases in CI, monitors online signals by version, closes the loop from incidents to test cases, and designs the platform so teams want to use it.

Score yourself

Requirements Asked clarifying questions and stated assumptions. Numbers Estimated scale and used it to make a decision. Architecture Clear components with responsibilities and data flow. Depth Went deep on the hardest part with real mechanisms. Trade-offs Named alternatives and why they lost. Operations Covered failures, monitoring and rollout. Leadership Explained how the team would build and evolve it.

5.5Whiteboard

Draw the design here, in this order. It stays in this browser.

  1. Prompt registry → CI eval gate → deploy
  2. Scorers: rules then judge, calibrated by humans
  3. Production call through guardrails with fallback
  4. Tracing → monitors → review sample
  5. Loop from review back into eval sets
PenBoxArrowTextEraserUndoClear
Tip: draw boxes first, then arrows, then label them with Text.