Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
People can't review every prompt change, so an LLM scores answers instead. It works well at scale, with known flaws: judges favour the first answer, longer answers and answers like their own. Good design removes those biases and checks the judge against people.
LLM judges scale evaluation, but only once you correct position, verbosity and self-enhancement bias.
1.2Key concepts
One idea per card, in plain words.
Asks which of two answers wins. Fewer scale problems than absolute scores.
Judges favour one position. Ask twice with the answers swapped.
Judges favour longer answers even when they say no more.
A judge favours its own model's answers. Use a different model to judge.
Measures agreement beyond chance, between a judge and human labels.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Three teams ship prompt changes weekly and argue about quality. Design an automated judge they can trust as a release signal.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Who writes the rubric? | Each team, for its own feature, reviewed against real failures. |
| What decides a release? | Judge scores gate on a regression; people review borderline changes. |
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Write a rubric
Name the criteria, with examples of good and bad answers. Vague rubrics give noisy scores.
- Ask for reasoning first
Have the judge reason before its verdict, then parse a fixed output format.
- Swap positions
Run each pair both ways. Count a win only when both runs agree.
- Calibrate with people
Label a few hundred pairs by hand, and compute agreement with the judge.
- Watch for drift
Re-run the calibration set whenever the judge model or rubric changes.
3.2Code
Working code for the core of it.
Checking an LLM judge
A judge you haven't measured is an opinion. These two functions make it a measurement.
1/** Cohen's kappa: agreement between two raters, beyond what chance would produce. */2export function cohenKappa(a: string[], b: string[]): number {3 const n = a.length;4 const labels = new Set([...a, ...b]);5 const observed = a.filter((x, i) => x === b[i]).length / n;6 let chance = 0;7 for (const l of labels) chance += (a.filter((x) => x === l).length / n) * (b.filter((x) => x === l).length / n);8 return chance === 1 ? 1 : (observed - chance) / (1 - chance);9}10 11export type Verdict = "first" | "second" | "tie";12 13/** Removes position bias: ask twice with the answers swapped; a verdict counts only if both agree. */14export async function debiasedVerdict(judge: (x: string, y: string) => Promise<Verdict>, a: string, b: string): Promise<"a" | "b" | "tie"> {15 const [one, two] = await Promise.all([judge(a, b), judge(b, a)]);16 if (one === "first" && two === "second") return "a";17 if (one === "second" && two === "first") return "b";18 return "tie";19}cohenKappa- Agreement with human labels, beyond what chance would give.
Promise.all([judge(a, b), judge(b, a)])- Judges favour one position, so ask both ways and count only agreement.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
How good can a judge get?
How do you handle verbosity bias?
Which checks should run before a judge?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Judge cost
- Two judge calls per pair add up. Sample in production, and run in full on the regression set.
- Counterpoint
- A judge measures what its rubric names and misses what it doesn't. Keep reading real conversations.