Eval basics

Reported as a major difference between candidates who've shipped AI and those who've only demoed it.

Reported as a major difference between candidates who've shipped AI and those who've only demoed it.

An eval is a set of examples, a way to grade them, and a score you compare between versions.

Build the set from real traffic, tag cases by slice, and add every production failure.

Small sets have wide uncertainty, averages hide slices, and an LLM judge needs checking against human labels.

What are the three parts of an eval?
Examples, a way to grade them, and a score you can compare.
Why check slices as well as the average?
An average can rise while one group gets much worse.
What does 18 of 20 really tell you?
90% observed, with a 95% interval from about 70% to about 97%.
What does Cohen's kappa measure?
Agreement beyond what chance would produce. 0 is luck, and 1 is perfect.
What's a sign your rubric isn't ready?
Two people grade the same output differently.
95% pass, users are unhappy. What's your first question?
Where did the eval cases come from, and do they still match real traffic?
What happens to a production failure?
It becomes a labelled case in the set, so it can't quietly return.