Some AI teams role-play a customer conversation, because their engineers actually talk to customers. One write-up lists weakness here as a common reason for rejection. Even without this round, the skill helps in every behavioural interview.
A complaint is a symptom, and the words are often vague. Your first job is to get a concrete example. Your second is to work out which layer failed. Your third is to say honestly what happens next.
- Reproduction: One real example you can run again. Without one, you're guessing.
- Trace: The record of what the system did for one request.
- Severity: How much harm, to how many people. It decides what happens first.
Steps:
- Listen, and ask for an example. 'Can you show me one from this week?' Vague complaints become specific.
- Ask what they expected. The gap between expected and actual is the bug.
- Read the trace, in order. Tool errors, then retrieval, then citations, then the answer.
- Say what you found, in their words. No jargon. Say what happened and what it means for them.
- Commit to a next step. A workaround now, a date for the fix, or an honest 'I don't know yet, and I'll know by Friday.'
- Count it. Is this one customer, or many? Check across traces before you prioritise.
Prompt, Practise with a difficult customer:
Role-play a frustrated customer of an AI support assistant. You say the bot 'feels dumb'. You're busy, not technical, and vague at first. Only give a concrete example if I ask for one. Don't help me. After 8 exchanges, step out of character and tell me: whether I got a real example, whether I used jargon, whether I promised something I couldn't deliver, and what I should have asked.
Code, From complaint to cause. Walks the stages in order, and reports the first one that failed.
/** The order matters: an early failure explains the later ones, so report the first one you find. */
export function diagnose(t: Trace, latencyLimitMs: number): Finding {
if (t.toolErrors.length > 0) return "tool-failed";
if (!t.neededDocs.some((d) => t.retrievedDocs.includes(d))) return "retrieval-missed";
if (t.citedDocs.some((d) => !t.retrievedDocs.includes(d))) return "made-up-citation";
if (!t.answerUsedNeededDoc) return "context-ignored";
if (t.latencyMs > latencyLimitMs) return "too-slow";
return "no-fault-found";
}
t.toolErrors.length > 0: Check the plumbing first. A failed tool explains everything after it.!t.neededDocs.some: The right text never arrived, so no prompt change will help.t.citedDocs.some: It cites something it never retrieved, which means it made up the citation.!t.answerUsedNeededDoc: The text was there and the model ignored it. Now the prompt is worth changing.
Code, From one complaint to a priority. Run the same check across many traces to see what to fix first.
/** One complaint is a story. Counting findings across traces tells you what to fix first. */
export function rankFindings(traces: Trace[], latencyLimitMs: number): [Finding, number][] {
const counts = new Map<Finding, number>();
for (const t of traces) {
const f = diagnose(t, latencyLimitMs);
counts.set(f, (counts.get(f) ?? 0) + 1);
}
return [...counts].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]));
}
counts.set(f, (counts.get(f) ?? 0) + 1): One customer's story becomes a measurement across all of them.Failure, Defending the product: The customer doesn't care why it's hard. Listen first.
Failure, Jargon: Saying 'our recall dropped' helps nobody. Say 'it couldn't find your new document.'
Failure, Promising a date you can't keep: Say what you know now, and when you'll know more.
- Why check tool errors first in a trace?
- An early failure explains the later ones. If the search tool failed, the bad answer is a symptom.
- The customer's example works fine when you try it. What now?
- Ask what they saw, and look at their actual trace. Differences in documents, permissions, or timing are common causes, and it may be a real intermittent failure.
- What makes a good next step?
- It's specific, it's something you control, and it has a date. 'I'll look into it' isn't one.
Sources:
- What applied-AI engineer interviews test (Individual): https://www.techinterview.org/post/3233476824/ai-engineer-interview-rag-agents-evals/
- Stop testing implementation, start probing eval design (Individual): https://tianpan.co/blog/2026-04-27-ai-engineer-interview-eval-design