The AI-off test

How do you know the skill is yours, and that you're ready?

Practising with AI creates a risk: you feel fluent because you recognise good answers, though you can't produce them. It's called the illusion of knowing, and it shows up at the worst moment.

The fix is simple, and it matches what one candidate described in a write-up about their Meta preparation: spend the last stretch without AI to check that the skills transferred.

Then use a clear rule to decide you're ready, because nerves won't tell you. Three AI-off mocks, a decent average, no very weak skill, and a small gap between your scores with and without AI.

  • Transfer: Being able to do something in a new setting, without the support you learned it with.
  • Illusion of knowing: Mistaking familiarity for ability. Rereading and watching solutions both cause it.
  • AI gap: Your average score with AI, minus your average without it.

Steps:

  1. Turn AI off for the final quarter. Close it fully, except during rounds that allow it, which you'll still rehearse with it.
  2. Do at least three full mocks. With a person if you can, as a human reacts differently.
  3. Run the gate. Or work it out by hand.
  4. Fix what it finds. Go back to the weakest skill, and don't redo the whole course.
  5. Rest the day before. Cards only.

If the round allows AI: You still want both skills. Rehearse the AI-allowed round with the assistant too, in the same conditions you'll have on the day, and keep a stretch of AI-off practice for the fundamentals. Reports say the assistant in the room may be less helpful than the one you're used to.

Code, Are you ready?. Scores with AI can flatter you. This compares them with your scores without it.

export function readiness(aiOn: MockScore[], aiOff: MockScore[]): { ready: boolean; reasons: string[] } {
  const reasons: string[] = [];
  if (aiOff.length < 3) return { ready: false, reasons: [`${3 - aiOff.length} more AI-off mock(s) needed`] };
  const recent = aiOff.slice(-3);
  const offAvg = mean(recent.flatMap((m) => FIELDS.map((f) => m[f])));
  if (offAvg < 3.5) reasons.push(`AI-off average is ${offAvg.toFixed(1)}, below 3.5`);
  for (const f of FIELDS) {
    const v = mean(recent.map((m) => m[f]));
    if (v < 3) reasons.push(`${f} is ${v.toFixed(1)}, below 3`);
  }
  if (aiOn.length > 0) {
    const gap = mean(aiOn.slice(-3).flatMap((m) => FIELDS.map((f) => m[f]))) - offAvg;
    if (gap > 0.5) reasons.push(`AI-on scores beat AI-off by ${gap.toFixed(1)}, so the AI is carrying you`);
  }
  return { ready: reasons.length === 0, reasons };
}
  • aiOff.length < 3: One good day proves little. It needs three AI-off mocks.

  • v < 3: One very weak skill can sink a round even when the average looks fine.

  • gap > 0.5: If you score much higher with AI, the AI is doing the work.

  • Failure, Only mocking with AI: You'll have no idea how you do without it until the day you find out.

  • Failure, Only mocking with AI as the interviewer: People interrupt, react and change direction. Do at least one with a person.

  • Failure, Waiting until you feel ready: That feeling may never come, or it may be wrong. Use the numbers.

What does a big AI gap tell you?
The tool is doing work you can't yet do yourself. Find which skill has the biggest gap and practise it without help.
Why three mocks rather than one?
A single good day could be luck. Three in a row is much better evidence.

Sources: