Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
A multimodal model can score well by guessing from the question text and never really looking. A sound eval tests vision directly: run the same questions with a blank image, and count only what the image makes possible.
Many vision benchmark items can be answered without the image. Control for that before trusting a score.
1.2Key concepts
One idea per card, in plain words.
Answering by interpreting what the image shows: reading a chart, counting, following a diagram.
Questions the model answers correctly without seeing the image.
Accuracy with the image minus accuracy with a blank image.
Describing objects that aren't there. POPE tests this directly.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Choose a vision model for reading invoices and charts. Public benchmarks disagree. Design an evaluation you can trust.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| What inputs? | Phone photos of invoices and exported charts. |
| What does failure cost? | A wrong total costs money, so precision matters more than coverage. |
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Build from real inputs
Use your own images, including blurry, rotated and low-resolution ones.
- Add a blank control
Run each question with a blank image, and flag items that still pass.
- Report visual lift
Exclude image-blind items from the true visual score.
- Test for invention
Ask about things that aren't in the image. A good model says so.
- Slice the results
Report by image type and resolution. An average hides what breaks.
3.2Code
Working code for the core of it.
Does the model really use the image?
Many vision questions can be answered from the text alone, which inflates scores.
1export interface VisionResult {2 id: string;3 withImage: boolean;4 blankImage: boolean;5}6 7/**8 * Control run: score the same questions with a blank image. Items answered correctly without9 * the image test recall, not vision, so they should not count towards a visual score.10 */11export function visionReport(results: VisionResult[]) {12 const rate = (f: (r: VisionResult) => boolean) => results.filter(f).length / Math.max(results.length, 1);13 const accuracy = rate((r) => r.withImage);14 const blankAccuracy = rate((r) => r.blankImage);15 const visual = results.filter((r) => !r.blankImage);16 return {17 accuracy,18 blankAccuracy,19 visualLift: accuracy - blankAccuracy,20 trueVisualAccuracy: visual.filter((r) => r.withImage).length / Math.max(visual.length, 1),21 imageBlindItems: results.filter((r) => r.blankImage).map((r) => r.id),22 };23}blankImage- The control: the same question with a blank image.
visualLift- The real value the image adds.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why not trust a public leaderboard?
What about resolution?
How do you score free-text answers?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Multiple images
- Reasoning across images often fails even when single-image scores look strong. Test it separately.
- Counterpoint
- For printed documents, classic OCR plus a text model can beat a vision model on accuracy and cost.