Normal code either passes a test or it doesn't. A model's output varies, and 'good' is often a judgment call. Evals are how you get a number anyway: a set of examples, a way to grade them, and a score you can compare from one version to the next.
Tian Pan argues that the question to ask candidates is how they'd design an eval before they've touched a prompt, since it shows whether they understand the failure modes. Whether or not you face that exact round, it's a strong way to prepare.
Think of evals as tests plus statistics. Two things catch people out. A small set gives a wide range of uncertainty. And an average can improve while one group of users gets much worse.
- Golden set: Examples with known good outcomes, kept under version control.
- Slice: A group of cases, such as billing questions or non-English users. Check each one.
- LLM judge: A model used to grade outputs. Check it against human labels before you trust it.
- Cohen's kappa: Agreement after removing what chance would produce. 0 is luck, and 1 is perfect.
- Regression: Something that worked before and doesn't now.
Steps:
- List failure modes first. Include those the spec missed: attacks, other languages, ambiguous cases, and errors with uneven costs.
- Build the set from real traffic. Refresh it regularly. Add every production failure.
- Write a rubric two people would agree on. If two graders disagree, the rubric isn't ready.
- Choose graders. Code checks where possible, a judge where necessary, and humans to calibrate.
- Validate any LLM judge. Compare it with human labels using kappa, not just raw agreement.
- Gate releases on slices. Block on any slice that drops, even if the average rises.
How many cases do you need?: Fewer than you'd fear to start, and more than you'd like to decide. 18 of 20 passing sounds like 90%, but the interval reaches down to about 70%. Use small sets to find problems, and larger ones before you make a close call between versions.
Code, Running an eval. A loop over labelled cases, counting passes per slice.
export async function runEval<I, O>(cases: EvalCase<I>[], system: (input: I) => Promise<O>, grade: Grader<I, O>): Promise<EvalReport> {
const report: EvalReport = { total: 0, passed: 0, byTag: {}, failures: [] };
for (const c of cases) {
const tag = (report.byTag[c.tag] ??= { total: 0, passed: 0 });
report.total++;
tag.total++;
let ok = false;
try {
ok = grade(c.input, await system(c.input));
} catch {
ok = false; // a crash is a failed case, not a broken run
}
if (ok) {
report.passed++;
tag.passed++;
} else {
report.failures.push(c.id);
}
}
return report;
}
report.byTag[c.tag]: Results per slice are the point. One overall number hides too much.ok = false; // a crash: One bad case shouldn't end the run. It counts as a failure and you move on.report.failures.push(c.id): Keep the ids. Reading failures is where the real learning happens.
Code, The release gate. Decides whether a new version may ship, and it checks each slice.
/** Checks every slice, because a better average can hide a slice that got much worse. */
export function releaseGate(base: EvalReport, next: EvalReport, maxDrop: number): { ship: boolean; blockers: string[] } {
const rate = (r: { passed: number; total: number }) => (r.total ? r.passed / r.total : 0);
const blockers: string[] = [];
if (rate(next) < rate(base) - maxDrop) blockers.push("overall");
for (const [tag, b] of Object.entries(base.byTag)) {
const n = next.byTag[tag];
if (!n) blockers.push(`${tag}: cases missing`);
else if (rate(n) < rate(b) - maxDrop) blockers.push(tag);
}
return { ship: blockers.length === 0, blockers };
}
rate(next) < rate(base) - maxDrop: The overall check, with a little room for noise.for (const [tag, b] of Object.entries(base.byTag)): A slice check. In our test, the average rises from 90% to 93% while billing falls from 90% to 50%, and this blocks it.cases missing: Deleting the hard cases must not count as a pass.
Code, How sure can you be?. A small eval set gives a wide range. This says how wide.
/** The honest range for a pass rate. 18 of 20 is 90%, but the data can't rule out about 70%. */
export function wilsonInterval(passed: number, total: number, z = 1.96): [number, number] {
if (total === 0) return [0, 1];
const p = passed / total;
const d = 1 + (z * z) / total;
const mid = p + (z * z) / (2 * total);
const spread = z * Math.sqrt((p * (1 - p)) / total + (z * z) / (4 * total * total));
return [(mid - spread) / d, (mid + spread) / d];
}
z = 1.96: The value for a 95% interval.(z * z) / (2 * total): Pulls the estimate toward the middle, which keeps small samples honest.
Code, Checking an LLM judge. Raw agreement flatters a judge when most answers are good. Kappa takes out the agreement you'd get by luck.
/** Agreement after removing what chance alone would give. Check this before an LLM judge gates a release. */
export function cohenKappa(human: boolean[], judge: boolean[]): number {
if (human.length !== judge.length || human.length === 0) throw new Error("need two equal, non-empty label lists");
const n = human.length;
const agree = human.filter((h, i) => h === judge[i]).length / n;
const humanYes = human.filter(Boolean).length / n;
const judgeYes = judge.filter(Boolean).length / n;
const chance = humanYes * judgeYes + (1 - humanYes) * (1 - judgeYes);
return chance === 1 ? 1 : (agree - chance) / (1 - chance);
}
const chance: How often two graders would agree just from how often each says yes.(agree - chance) / (1 - chance): 0 means no better than luck, and 1 means perfect.Failure, A stale golden set: It stops matching what users ask, and the score becomes meaningless.
Failure, An unvalidated judge: It may pass everything, and you'd never know.
Failure, Only an average: In our test, the average rises from 90% to 93% while billing falls from 90% to 50%.
Failure, Mixing up model evals and system evals: A good retrieval score isn't a good product. Measure each layer.
- 95% pass rate, and users complain. Name three possible causes.
- The set no longer matches real traffic, the grader is too lenient, or the failures are concentrated in a slice that the average hides.
- Why use kappa rather than raw agreement?
- If 90% of answers are good, a judge that always says 'good' agrees 90% of the time and has learned nothing. Kappa removes the agreement expected by chance.
- You must decide on a new model this week. What do you run?
- The golden set on both versions, compared per slice, with cost and latency. Also a small live test if you can, and read the disagreements yourself.
- How do production failures feed back?
- Each becomes a labelled case in the set, so that it can't quietly return.
Sources:
- Stop testing implementation, start probing eval design (Individual): https://tianpan.co/blog/2026-04-27-ai-engineer-interview-eval-design
- Interviewing for AI eval engineering roles (Individual): https://www.techinterview.org/post/3233475313/interviewing-ai-eval-engineering-roles-2026/
- What applied-AI engineer interviews test (Individual): https://www.techinterview.org/post/3233476824/ai-engineer-interview-rag-agents-evals/