The evals round

How do you prove an AI feature works, and that the next change didn't break it?

Normal code either passes a test or it doesn't. A model's output varies, and 'good' is often a judgment call. Evals are how you get a number anyway: a set of examples, a way to grade them, and a score you can compare from one version to the next.

Tian Pan argues that the question to ask candidates is how they'd design an eval before they've touched a prompt, since it shows whether they understand the failure modes. Whether or not you face that exact round, it's a strong way to prepare.

Think of evals as tests plus statistics. Two things catch people out. A small set gives a wide range of uncertainty. And an average can improve while one group of users gets much worse.

  • Golden set: Examples with known good outcomes, kept under version control.
  • Slice: A group of cases, such as billing questions or non-English users. Check each one.
  • LLM judge: A model used to grade outputs. Check it against human labels before you trust it.
  • Cohen's kappa: Agreement after removing what chance would produce. 0 is luck, and 1 is perfect.
  • Regression: Something that worked before and doesn't now.

Steps:

  1. List failure modes first. Include those the spec missed: attacks, other languages, ambiguous cases, and errors with uneven costs.
  2. Build the set from real traffic. Refresh it regularly. Add every production failure.
  3. Write a rubric two people would agree on. If two graders disagree, the rubric isn't ready.
  4. Choose graders. Code checks where possible, a judge where necessary, and humans to calibrate.
  5. Validate any LLM judge. Compare it with human labels using kappa, not just raw agreement.
  6. Gate releases on slices. Block on any slice that drops, even if the average rises.

How many cases do you need?: Fewer than you'd fear to start, and more than you'd like to decide. 18 of 20 passing sounds like 90%, but the interval reaches down to about 70%. Use small sets to find problems, and larger ones before you make a close call between versions.

Code, Running an eval. A loop over labelled cases, counting passes per slice.

export async function runEval<I, O>(cases: EvalCase<I>[], system: (input: I) => Promise<O>, grade: Grader<I, O>): Promise<EvalReport> {
  const report: EvalReport = { total: 0, passed: 0, byTag: {}, failures: [] };
  for (const c of cases) {
    const tag = (report.byTag[c.tag] ??= { total: 0, passed: 0 });
    report.total++;
    tag.total++;
    let ok = false;
    try {
      ok = grade(c.input, await system(c.input));
    } catch {
      ok = false; // a crash is a failed case, not a broken run
    }
    if (ok) {
      report.passed++;
      tag.passed++;
    } else {
      report.failures.push(c.id);
    }
  }
  return report;
}
  • report.byTag[c.tag]: Results per slice are the point. One overall number hides too much.
  • ok = false; // a crash: One bad case shouldn't end the run. It counts as a failure and you move on.
  • report.failures.push(c.id): Keep the ids. Reading failures is where the real learning happens.

Code, The release gate. Decides whether a new version may ship, and it checks each slice.

/** Checks every slice, because a better average can hide a slice that got much worse. */
export function releaseGate(base: EvalReport, next: EvalReport, maxDrop: number): { ship: boolean; blockers: string[] } {
  const rate = (r: { passed: number; total: number }) => (r.total ? r.passed / r.total : 0);
  const blockers: string[] = [];
  if (rate(next) < rate(base) - maxDrop) blockers.push("overall");
  for (const [tag, b] of Object.entries(base.byTag)) {
    const n = next.byTag[tag];
    if (!n) blockers.push(`${tag}: cases missing`);
    else if (rate(n) < rate(b) - maxDrop) blockers.push(tag);
  }
  return { ship: blockers.length === 0, blockers };
}
  • rate(next) < rate(base) - maxDrop: The overall check, with a little room for noise.
  • for (const [tag, b] of Object.entries(base.byTag)): A slice check. In our test, the average rises from 90% to 93% while billing falls from 90% to 50%, and this blocks it.
  • cases missing: Deleting the hard cases must not count as a pass.

Code, How sure can you be?. A small eval set gives a wide range. This says how wide.

/** The honest range for a pass rate. 18 of 20 is 90%, but the data can't rule out about 70%. */
export function wilsonInterval(passed: number, total: number, z = 1.96): [number, number] {
  if (total === 0) return [0, 1];
  const p = passed / total;
  const d = 1 + (z * z) / total;
  const mid = p + (z * z) / (2 * total);
  const spread = z * Math.sqrt((p * (1 - p)) / total + (z * z) / (4 * total * total));
  return [(mid - spread) / d, (mid + spread) / d];
}
  • z = 1.96: The value for a 95% interval.
  • (z * z) / (2 * total): Pulls the estimate toward the middle, which keeps small samples honest.

Code, Checking an LLM judge. Raw agreement flatters a judge when most answers are good. Kappa takes out the agreement you'd get by luck.

/** Agreement after removing what chance alone would give. Check this before an LLM judge gates a release. */
export function cohenKappa(human: boolean[], judge: boolean[]): number {
  if (human.length !== judge.length || human.length === 0) throw new Error("need two equal, non-empty label lists");
  const n = human.length;
  const agree = human.filter((h, i) => h === judge[i]).length / n;
  const humanYes = human.filter(Boolean).length / n;
  const judgeYes = judge.filter(Boolean).length / n;
  const chance = humanYes * judgeYes + (1 - humanYes) * (1 - judgeYes);
  return chance === 1 ? 1 : (agree - chance) / (1 - chance);
}
  • const chance: How often two graders would agree just from how often each says yes.

  • (agree - chance) / (1 - chance): 0 means no better than luck, and 1 means perfect.

  • Failure, A stale golden set: It stops matching what users ask, and the score becomes meaningless.

  • Failure, An unvalidated judge: It may pass everything, and you'd never know.

  • Failure, Only an average: In our test, the average rises from 90% to 93% while billing falls from 90% to 50%.

  • Failure, Mixing up model evals and system evals: A good retrieval score isn't a good product. Measure each layer.

95% pass rate, and users complain. Name three possible causes.
The set no longer matches real traffic, the grader is too lenient, or the failures are concentrated in a slice that the average hides.
Why use kappa rather than raw agreement?
If 90% of answers are good, a judge that always says 'good' agrees 90% of the time and has learned nothing. Kappa removes the agreement expected by chance.
You must decide on a new model this week. What do you run?
The golden set on both versions, compared per slice, with cost and latency. Also a small live test if you can, and read the disagreements yourself.
How do production failures feed back?
Each becomes a labelled case in the set, so that it can't quietly return.

Sources: