Your coach gives scores. Are they any good? Score 20 of your mocks yourself, then compare. You're looking at two things: how often it's within a point of you, and whether it's systematically too kind.
Once you have those numbers, you have something few candidates have: a real AI project, with a real failure, a real measurement, and a real fix. 'Tell me about something you built with AI' becomes your strongest answer.
- Bias: A consistent lean in one direction. Here, scoring you too high.
- Calibration: Comparing a judge with human labels and adjusting it.
- STAR: Situation, task, action, result. A common shape for behavioural answers.
Steps:
- Save every session. Transcripts are your data.
- Label 20 by hand. Before you look at the coach's score.
- Compare. Within-one agreement, and bias.
- Change one thing and rerun. That's a real iteration, with a measurement.
- Work out the cost per mock. Tokens times prices. Note how much caching saves.
- Write your story. One failure, one measurement, one fix. Only true numbers.
The story, in four lines: Why: 'I wanted honest practice for an AI interview.' What: 'A mock interviewer. Code owns the phases, and the model writes the words.' What went wrong: 'It scored too kindly. I measured that against my own labels.' What changed: 'One rule moved the bias from this to that, and here's what I'd do next.'
Expect these follow-ups: Why not a bigger model? How would it handle ten thousand users? How would you stop abuse? What if the labels are wrong, given that they're all yours? Having real answers to these is what the project gets you. Your own labels are a real weakness, so say so before they do.
Use your own numbers: Meera's figures are made up for this example. Only ever quote numbers you actually measured. A smaller true number will survive follow-up questions, and an invented one won't.
Code, Does it score like a person?. Score some of your mocks by hand, then compare.
export function agreement(rows: LabelledScore[]) {
if (rows.length === 0) throw new Error("no labelled scores yet");
const share = (test: (r: LabelledScore) => boolean) => rows.filter(test).length / rows.length;
return {
exact: share((r) => r.human === r.model),
withinOne: share((r) => Math.abs(r.human - r.model) <= 1),
/** above 0 means the coach flatters you */
bias: rows.reduce((a, r) => a + (r.model - r.human), 0) / rows.length,
};
}
Math.abs(r.human - r.model) <= 1: People don't agree exactly either, so one point apart is a fair test.bias: The average difference, with its sign. Positive means it's too kind.
Code, When to trust your coach. The same idea as a release gate, for a project of your own.
/** A coach that flatters is the failure that hurts you most, so bias gets its own limit. */
export function coachIsTrustworthy(rows: LabelledScore[]): { ok: boolean; reasons: string[] } {
const a = agreement(rows);
const reasons: string[] = [];
if (rows.length < 20) reasons.push(`only ${rows.length} labelled scores; label at least 20`);
if (a.withinOne < 0.8) reasons.push(`within-one agreement is ${Math.round(a.withinOne * 100)}%, below 80%`);
if (a.bias > 0.5) reasons.push(`scores run ${a.bias.toFixed(1)} points too kind`);
return { ok: reasons.length === 0, reasons };
}
rows.length < 20: Too few labels can't tell you much.a.bias > 0.5: A flattering coach sends you to the interview overconfident, so it gets its own limit.Failure, Looking at the coach's score before labelling: You'll anchor on it. Label blind.
Failure, Only one person's labels: It's a real limit. Say so, and ask a friend to label some.
Failure, Overselling a side project: State honestly who used it: probably just you. That's fine.
- Why does bias get its own limit?
- A coach that flatters is the failure that hurts most: you'd walk in overconfident. Agreement alone could hide that if every score is one point too high.
- What makes this a good interview story?
- It has a real failure, a measurement, a change, and a result, all of which you can defend with detail, since you did them.
- An interviewer asks how it would scale. What's a good first answer?
- Say what would break first, such as cost per session and API rate limits, then how you'd deal with it: caching, queues and limits per user. And say you haven't had to do it yet.