Vision Language Models are Biased

TL;DR

A counterfactual-image benchmark for state-of-the-art VLMs (o3, Gemini 2.5 Pro, Claude 3.7, GPT-4o, etc.) showing that they answer visual counting and identification questions from memorized priors about popular subjects rather than from the image in front of them — a 3-stripe Adidas logo modified to 4 stripes is still counted as 3, a 5-legged elephant is still counted as 4. Across seven domains (animals, logos, chess, board games, optical illusions, patterned grids, flags) and ~$2 of API calls, models score an average of 17.05% on counting in counterfactual images. Inserting the subject name as text into the image further degrades accuracy; instructing the model to “rely exclusively on image details” or “double-check” the answer recovers only ~+2 points. The website ships one-click copy-paste of images and prompts, making it cheap for anyone to reproduce.

Key claims

State-of-the-art VLMs average 17.05% accuracy on counting tasks across 7 counterfactual domains, compared to near-100% on the original (unmodified) subjects [§1, §4].
Adding a text label of the canonical subject (e.g. typing “Adidas” onto an Adidas-with-4-stripes image) decreases counting accuracy further — the language signal reinforces the memorized prior against the visual evidence [§4, §5].
Instruction-level mitigations are weak: prompts asking the model to “double-check” or “rely exclusively on what is in the image” lift counting accuracy by only ~+2 points on average — the bias is not a surface obedience problem [§5].
The seven counterfactual domains span animals (extra legs/eyes), logos (extra stripes/lines), chess pieces (extra rook), board-game grids (off-by-one rows), optical illusions, patterned grids, and national flags [§3, project page].
The benchmark is built from procedurally-edited images of well-known subjects where a single visual attribute (count, identity) has been minimally perturbed; ground truth is unambiguous and machine-checkable [§3].
The framework is automated and extensible: a fixed prompt template plus a generator that produces new counterfactuals on demand, making the benchmark hard to game by training-set contamination of a fixed image set [§3, project page].
The pattern reproduces across closed (o3, Gemini 2.5 Pro, Claude 3.7) and open VLMs at similar magnitudes, suggesting the failure mode is architectural / training-distributional rather than a property of any one model family [§4].

Method

The benchmark is constructed by taking images of popular subjects whose canonical visual attributes are widely known (the 3 stripes of Adidas, the 4 legs of a quadruped, the 8×8 chessboard) and applying a single targeted visual edit — adding one stripe, one leg, one extra row — without altering any other aspect of the image. For each (subject, edit) pair the authors generate a counting or identification question with a single objective answer. Each VLM is queried zero-shot through its standard API with one-click reproducible prompts published on the project site.

The “text-overlay” variant adds the subject’s name as printed text inside the image (e.g. “Adidas” on the modified logo) to test whether explicit linguistic cuing of the canonical identity strengthens or weakens the prior bias. Prompt-mitigation experiments prepend instructions like “Rely exclusively on what is in the image; the image may differ from what you expect” before the question. Total API spend across the reported experiments is roughly $2, demonstrated as a deliberate property of the benchmark: it is cheap enough that any practitioner can reproduce it on a new model in a matter of minutes.

Results

Average counting accuracy across all 7 domains and tested VLMs: 17.05% on counterfactual images, vs ~100% on the matched canonical (unmodified) images [§4].
Adidas-style 4-stripe logo: most VLMs answer “3” — locked to the canonical count regardless of the visible stripe count [§4, project page].
Extra-limb animal images (e.g. 5-legged elephant): models report the canonical limb count for the species [§4, project page].
Text-overlay augmentation decreases counting accuracy further across all 7 domains, with the largest drops on subjects whose name is most strongly associated with a numeric attribute (logos, board games) [§4, §5].
“Double-check” and “rely on image only” prompt mitigations: average improvement +2.0 points in counting accuracy — insufficient to recover the lost performance [§5].
The benchmark’s discriminative range across models is wide on identification but compressed on counting — even the strongest tested VLM stays below 30% counting accuracy [Project page leaderboard].

Why it’s interesting

This is the cleanest filed evidence for the “VLMs answer from the LLM prior, not from the image” failure mode at the behavioral level. The wiki’s mechanistic side of this story is Hidden in plain sight: VLMs overlook their visual representations — Fu et al. showed via intermediate-layer probing that the correct visual features are present at every LLM layer in a LLaVA-style stack, and the failure is at the final-layer readout where the LLM’s text prior wins; total-variation distance between sighted and blind VLM answer distributions on vision-centric VQA is small. Vo et al. provide the high-level benchmark that operationalizes that finding: a counterfactual where the visual evidence and the LLM prior point in opposite directions, with an unambiguous ground truth that lets you score “did the model read the image?” at scale.

It complements Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs from the failure-mode end: Izadi et al. propose a training-free input-side fix (horizontal-line scaffold + sequential-scan prompt) for the binding-problem version of the same disconnect, and report large lifts on object-counting in cluttered synthetic scenes. The Vo et al. data suggest that fix would not generalize to the popular-subject-prior regime — the failure here is not parallel-feature-binding but language-prior dominance, and an input scaffold that does not interpose between subject identity and the LLM token stream may not help. A clean A/B between the two would isolate which failure mode dominates on which benchmark.

For Luma’s research team specifically, the benchmark is unusually cheap to adopt: $2 of API spend, one-click prompts, automatic counterfactual generation. The text-overlay-degrades-further result is the operationally useful one — it implies that any in-house VLM pipeline that surfaces subject names (captions, tags, metadata) as text inputs to a downstream VLM step is likely amplifying memorized-prior bias, not anchoring the model to the visual evidence.