Evaluating Multimodal LLMs on Visual Reasoning

Many vision benchmark items can be answered without the image. Control for that before trusting a score.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

A multimodal model can score well by guessing from the question text and never really looking. A sound eval tests vision directly: run the same questions with a blank image, and count only what the image makes possible.

Why it matters

Many vision benchmark items can be answered without the image. Control for that before trusting a score.

1.2Key concepts

One idea per card, in plain words.

01Visual reasoning

Answering by interpreting what the image shows: reading a chart, counting, following a diagram.

02Image-blind items

Questions the model answers correctly without seeing the image.

03Visual lift

Accuracy with the image minus accuracy with a blank image.

04Visual hallucination

Describing objects that aren't there. POPE tests this directly.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Choose a vision model for reading invoices and charts. Public benchmarks disagree. Design an evaluation you can trust.

2.2Clarify first

Each answer removes a guess.

You askAnswer
What inputs?Phone photos of invoices and exported charts.
What does failure cost?A wrong total costs money, so precision matters more than coverage.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Build from real inputs

    Use your own images, including blurry, rotated and low-resolution ones.

  2. Add a blank control

    Run each question with a blank image, and flag items that still pass.

  3. Report visual lift

    Exclude image-blind items from the true visual score.

  4. Test for invention

    Ask about things that aren't in the image. A good model says so.

  5. Slice the results

    Report by image type and resolution. An average hides what breaks.

3.2Code

Working code for the core of it.

Does the model really use the image?

Many vision questions can be answered from the text alone, which inflates scores.

image-dependence.tsTypeScriptCopy
1export interface VisionResult {2  id: string;3  withImage: boolean;4  blankImage: boolean;5}6 7/**8 * Control run: score the same questions with a blank image. Items answered correctly without9 * the image test recall, not vision, so they should not count towards a visual score.10 */11export function visionReport(results: VisionResult[]) {12  const rate = (f: (r: VisionResult) => boolean) => results.filter(f).length / Math.max(results.length, 1);13  const accuracy = rate((r) => r.withImage);14  const blankAccuracy = rate((r) => r.blankImage);15  const visual = results.filter((r) => !r.blankImage);16  return {17    accuracy,18    blankAccuracy,19    visualLift: accuracy - blankAccuracy,20    trueVisualAccuracy: visual.filter((r) => r.withImage).length / Math.max(visual.length, 1),21    imageBlindItems: results.filter((r) => r.blankImage).map((r) => r.id),22  };23}
23 linesUTF-8TypeScript
blankImage
The control: the same question with a blank image.
visualLift
The real value the image adds.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why not trust a public leaderboard?
Benchmarks may not resemble your images, and the items may leak into training data. Use them to shortlist, then test yourself.
What about resolution?
Models often downscale images. Small print vanishes, so test at the resolution you will send.
How do you score free-text answers?
Exact match for numbers, a calibrated judge for descriptions.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Multiple images
Reasoning across images often fails even when single-image scores look strong. Test it separately.
Counterpoint
For printed documents, classic OCR plus a text model can beat a vision model on accuracy and cost.