The customer conversation round

A customer says your bot feels dumb. What do you do in the next ten minutes?

Some AI teams role-play a customer conversation, because their engineers actually talk to customers. One write-up lists weakness here as a common reason for rejection. Even without this round, the skill helps in every behavioural interview.

A complaint is a symptom, and the words are often vague. Your first job is to get a concrete example. Your second is to work out which layer failed. Your third is to say honestly what happens next.

  • Reproduction: One real example you can run again. Without one, you're guessing.
  • Trace: The record of what the system did for one request.
  • Severity: How much harm, to how many people. It decides what happens first.

Steps:

  1. Listen, and ask for an example. 'Can you show me one from this week?' Vague complaints become specific.
  2. Ask what they expected. The gap between expected and actual is the bug.
  3. Read the trace, in order. Tool errors, then retrieval, then citations, then the answer.
  4. Say what you found, in their words. No jargon. Say what happened and what it means for them.
  5. Commit to a next step. A workaround now, a date for the fix, or an honest 'I don't know yet, and I'll know by Friday.'
  6. Count it. Is this one customer, or many? Check across traces before you prioritise.

Prompt, Practise with a difficult customer:

Role-play a frustrated customer of an AI support assistant. You say the bot 'feels dumb'. You're busy, not technical, and vague at first. Only give a concrete example if I ask for one. Don't help me. After 8 exchanges, step out of character and tell me: whether I got a real example, whether I used jargon, whether I promised something I couldn't deliver, and what I should have asked.

Code, From complaint to cause. Walks the stages in order, and reports the first one that failed.

/** The order matters: an early failure explains the later ones, so report the first one you find. */
export function diagnose(t: Trace, latencyLimitMs: number): Finding {
  if (t.toolErrors.length > 0) return "tool-failed";
  if (!t.neededDocs.some((d) => t.retrievedDocs.includes(d))) return "retrieval-missed";
  if (t.citedDocs.some((d) => !t.retrievedDocs.includes(d))) return "made-up-citation";
  if (!t.answerUsedNeededDoc) return "context-ignored";
  if (t.latencyMs > latencyLimitMs) return "too-slow";
  return "no-fault-found";
}
  • t.toolErrors.length > 0: Check the plumbing first. A failed tool explains everything after it.
  • !t.neededDocs.some: The right text never arrived, so no prompt change will help.
  • t.citedDocs.some: It cites something it never retrieved, which means it made up the citation.
  • !t.answerUsedNeededDoc: The text was there and the model ignored it. Now the prompt is worth changing.

Code, From one complaint to a priority. Run the same check across many traces to see what to fix first.

/** One complaint is a story. Counting findings across traces tells you what to fix first. */
export function rankFindings(traces: Trace[], latencyLimitMs: number): [Finding, number][] {
  const counts = new Map<Finding, number>();
  for (const t of traces) {
    const f = diagnose(t, latencyLimitMs);
    counts.set(f, (counts.get(f) ?? 0) + 1);
  }
  return [...counts].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]));
}
  • counts.set(f, (counts.get(f) ?? 0) + 1): One customer's story becomes a measurement across all of them.

  • Failure, Defending the product: The customer doesn't care why it's hard. Listen first.

  • Failure, Jargon: Saying 'our recall dropped' helps nobody. Say 'it couldn't find your new document.'

  • Failure, Promising a date you can't keep: Say what you know now, and when you'll know more.

Why check tool errors first in a trace?
An early failure explains the later ones. If the search tool failed, the bad answer is a symptom.
The customer's example works fine when you try it. What now?
Ask what they saw, and look at their actual trace. Differences in documents, permissions, or timing are common causes, and it may be a real intermittent failure.
What makes a good next step?
It's specific, it's something you control, and it has a date. 'I'll look into it' isn't one.

Sources: