Build your coach: scoring you can trust

How do you get reliable structured output from a model?

At the end, the coach scores you. This is where bad output does real harm: a kind, invented score sends you into an interview overconfident.

Three layers protect you. Ask for a clear format. Check everything in code. And require evidence: each score must quote your own words, and the code confirms that those words really appear in the transcript. A model can't quote something you never said and get away with it.

When the check fails, send the reason back and try once more. If it still fails, show no score. Admitting there's no result beats inventing one.

  • Structured output: Model output in a fixed format, such as JSON. APIs can often enforce the shape, but you still check the meaning.
  • Validation: Code that checks the output before anyone uses it.
  • Grounding: Tying a claim to a source you can check.

Steps:

  1. Define the shape. Four competencies, each with a score and evidence.
  2. Parse it safely. Bad JSON is normal. Handle it.
  3. Check the meaning. Score range, and evidence that's really in the transcript.
  4. Retry once with the reason. Then stop.
  5. Keep the raw output. You'll need it for debugging and evals.

Why check evidence?: Schema enforcement guarantees the shape, and says nothing about truth. The evidence check is a small example of the most useful habit in AI engineering: find something you can verify in code.

Code, Checking the score. Never trust a model's structured output until you've checked it.

/** Returns what's wrong in words, so the same text can go back to the model on a retry. */
export function parseScorecard(raw: string, transcript: string): { card: Scorecard } | { error: string } {
  let data: unknown;
  try {
    data = JSON.parse(raw);
  } catch {
    return { error: "Reply with JSON only." };
  }
  const obj = data as Record<string, ScoreLine | undefined>;
  for (const c of COMPETENCIES) {
    const line = obj?.[c];
    if (!line || !Number.isInteger(line.score) || line.score < 1 || line.score > 5) return { error: `${c}.score must be a whole number from 1 to 5.` };
    if (typeof line.evidence !== "string" || line.evidence.length < 10) return { error: `${c}.evidence must quote the transcript.` };
    if (!transcript.includes(line.evidence)) return { error: `${c}.evidence isn't in the transcript. Copy the candidate's exact words.` };
  }
  return { card: obj as unknown as Scorecard };
}
  • JSON.parse(raw): The first check: is it JSON at all?
  • Number.isInteger(line.score): Keeps scores to whole numbers from 1 to 5.
  • transcript.includes(line.evidence): The most useful line here. Evidence must be real words from the transcript, so a model can't invent a reason for a score.

Code, One retry, then nothing. Give the model one chance to fix it, and tell it what was wrong.

export type Complete = (prompt: string) => Promise<string>;

/** One retry with the reason. After that, show no score: a made-up score is worse than none. */
export async function scoreInterview(complete: Complete, prompt: string, transcript: string): Promise<Scorecard | null> {
  let ask = prompt;
  for (let attempt = 0; attempt < 2; attempt++) {
    const result = parseScorecard(await complete(ask), transcript);
    if ("card" in result) return result.card;
    ask = `${prompt}\n\nYour last reply was rejected: ${result.error}`;
  }
  return null;
}
  • attempt < 2: A strict limit. Endless retries cost money and rarely help.

  • Your last reply was rejected: Sending the reason back gives the retry something to fix.

  • return null: Showing no score is better than showing a made-up one.

  • Failure, Trusting valid JSON: Correct shape, invented content.

  • Failure, Endless retries: They cost money and rarely help. One, then stop.

  • Failure, A default score on failure: A made-up number looks just like a real one.

What does schema enforcement not protect you from?
Wrong or invented content. The shape can be perfect, and the facts false.
Why send the error back on a retry?
So the model knows what to fix. Without the reason, it's likely to make the same mistake.
What should the coach show if scoring fails twice?
No score, and an honest message. A fake number is worse than none, since you'd act on it.