Skip to content
Bud's Wiki

VLM Perception Failures

TL;DR

Section titled “TL;DR”

A failure-mode cluster: state-of-the-art VLMs systematically answer vision-centric questions from their LLM prior rather than from the image. The filed evidence spans a behavioral benchmark exposing memorized-subject bias on counterfactual images (Vision Language Models are Biased), a mechanistic probe showing that the correct visual features are present at every layer of a LLaVA-style VLM but are overridden at the final-layer language readout (Hidden in plain sight: VLMs overlook their visual representations), and a training-free input-side fix that overlays horizontal scan lines to force serial visual attention and partially recovers performance (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs). All three converge on the same locus: the LLM half of a VLM is where vision-centric reasoning breaks; the visual encoder usually has the answer in its tokens already.

Key claims

Section titled “Key claims”
  • VLMs answer vision-centric questions from a language prior rather than from the image: o3, Gemini 2.5 Pro, Claude 3.7, and GPT-4o all count 4-stripe Adidas logos as having 3 stripes and 5-legged elephants as having 4, averaging 17.05% counting accuracy on counterfactual images of popular subjects (Vision Language Models are Biased §1, §4).
  • The failure is at the readout, not the representation: probing every LLM layer of a LLaVA-style VLM with the original visual readout (linear probe, DPT head, cosine similarity, Gram matrix) recovers vision-encoder-level accuracy through the projector and most of the LLM, with sharp falloff only at the final language-generation layer (Hidden in plain sight: VLMs overlook their visual representations §4.1, Fig. 5).
  • The output distribution of a sighted VLM matches that of the same VLM run with a blank image — measured by total variation between answer histograms on vision-centric multiple-choice tasks — direct evidence that the model is ignoring the image and answering from the LLM prior (Hidden in plain sight: VLMs overlook their visual representations §3.4, Fig. 4, Table 2).
  • Wrapping the same frozen vision encoder in a VLM inverts the encoder ranking: DINOv2 wins five of six pure-vision evaluations but is disproportionately penalized inside a LLaVA wrapper, so CLIP/SigLIP appear stronger by VLM benchmarks despite being visually weaker — calling into question the methodological premise of using VLM scores to compare encoders (Hidden in plain sight: VLMs overlook their visual representations §3.3, Fig. 2, Table 1).
  • The fix is LLM-side, not encoder-side: at matched 16.7M-parameter LoRA, applying the LoRA to the LLM beats applying it to the projector or the ViT on five of six vision-centric tasks, and is the only intervention that drops the TV-to-blind distance to near zero (Hidden in plain sight: VLMs overlook their visual representations §4.3, Fig. 7, Table 2).
  • A training-free input-side alternative also works for the binding-problem version of the failure: overlaying 5 numbered horizontal lines on the image and prepending “Scan the image sequentially based on horizontal lines exists in the image” lifts GPT-4o 2D visual-search harmonic mean from 0.48 → 0.73 and 2D counting from 12.0% → 38.83% (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs §4.2-4.3, Tables 1-2).
  • Pure-text Chain-of-Thought fails to recover vision-centric performance and can degrade it — only the visual scaffold + sequential-scan prompt jointly help, indicating that language-side reasoning over already-entangled visual features inherits the misbindings (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs §4.6, Fig. 3).
  • Instruction-level prompt mitigations are weak on the memorized-prior failure mode: “double-check” or “rely exclusively on image details” prompts lift counting accuracy by only ~+2 points on Vo et al.’s benchmark (Vision Language Models are Biased §5). The bias is not a surface obedience problem.
  • Adding canonical-subject text into the image makes things worse, not better: inserting “Adidas” as printed text on a counterfactual Adidas logo further reduces counting accuracy (Vision Language Models are Biased §4, §5). Linguistic cues to identity amplify the prior; they don’t anchor the model to the pixels.
  • The failure pattern reproduces on production VLMs with custom-trained encoders (InternVL, Phi-3-V, Qwen-VL), open-weight VLMs (LLaMA-4-Scout, Qwen2.5-VL-7B), and closed frontier models (o3, Gemini 2.5 Pro, Claude 3.5/3.7) — it is architecturally and training-distributionally general, not specific to LLaVA-style stacks (Hidden in plain sight: VLMs overlook their visual representations §3.2, Fig. 3; Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs §4; Vision Language Models are Biased §4).
  • The inverse failure also appears on long-video benchmarks: the LLM head can be unnecessary, not just misleading. On VSI-Super-Recall (10-min to 4-hour videos), a frozen SigLIP2-So400m-512 keeping only the top-4 frames most similar to the query object and aggregating cosine similarities to auxiliary-object prompts — no LLM at all — achieves 95-98% accuracy across all five test splits, beating the full Cambrian-S MLLM pipeline by ~55 percentage points (Solving Spatial Supersensing Without Spatial Supersensing §2, Table 1, Fig. 1). The result reproduces across CLIP-L/14, SigLIP2-B/16, and both SigLIP2-So400m sizes, so it is not an artifact of the strongest contrastive backbone.
  • A third failure mode: frontier VLMs collapse on pixel-precise 3D-grounded output formats. GPT-5 scores 9.0 3D-IoU on object pose, 0% trajectory success rate, infinite distance error on trajectories, and infinite NCE on grasps when asked for ordered 2D keypoint lists with millimeter-level precision; Gemini Robotics-ER 1.5 is the only frontier model that beats chance on these geometric subtasks and still trails the 54.2 / 67.3% / 112.4 / 1.1 human baseline (BOP-Ask: Object-Interaction Reasoning for Vision-Language Models Table). Fine-tuning a 2B NVILA on the BOP-Ask training data lifts pose 3D-IoU from 6.5 → 77.4 and trajectory SR from 0 → 50.8%, beating both the human pose baseline and the proprietary Gemini Robotics-ER 1.5 model — so the capability is trainable from 6D-pose-grounded synthetic data, but does not come for free from frontier scale (BOP-Ask: Object-Interaction Reasoning for Vision-Language Models Table).
  • Counterpoint — the deficiency closes substantially with capability-axis data curation. SenseNova-SI post-trains Qwen3-VL / InternVL3 / Bagel on 8M samples organized under a taxonomy of spatial capabilities, reporting 68.8% VSI-Bench, 43.3% MMSI, 85.7% MindCube, 54.7% ViewSpatial, 47.7% SITE, 63.9% BLINK, 55.5% 3DSR, 72.0% EmbSpatial — well above the prior-art numbers the rest of this page documents as failures, while MMBench-En general capability holds at 84.9% (Scaling Spatial Intelligence with Multimodal Foundation Models). Conflict: Cambrian-S (Cambrian-S: Towards Spatial Supersensing in Video) framed VSI-Bench-class gains as the easy regime and argued data scaling alone runs out on longer-horizon VSI-Super; SenseNova-SI’s 68.8 VSI-Bench is in the same regime as Cambrian-S’s headline but its taxonomy-driven 8M curation is a stronger data-side bet than Cambrian-S’s 590K VSI-590K. The two are not direct head-to-head (no VSI-Super number in the SenseNova-SI abstract), but they instantiate the open dispute on this page: is the spatial failure a data problem (SenseNova-SI’s claim) or a mechanism problem (Cambrian-S’s claim).

Recent contributions

Section titled “Recent contributions”
  • [2026-07-15] Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models: FoveateR is the active counterpart to input-side scaffolding (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs): rather than pre-decorating every input, the VLM itself decides which regions need pixel-precise evidence and emits a non-linguistic foveation action to acquire them on demand mid-reasoning. Doesn’t attack the “answer from LLM prior” bias directly, but reduces the token cost of making high-resolution evidence available exactly where the reasoning trace requests it — a plausible route to bypassing the readout-side failure documented in Hidden in plain sight: VLMs overlook their visual representations by keeping the coarse view small and only spending high-res tokens where needed.
  • [2026-07-14] How Claude Performs on Robotics Tasks: Anthropic’s Claude-Plays-Robotics runs an image↔text-description substitution ablation on the Go2 navigation suite: replace the egocentric RGB with a Gemini-3.1 verbal description of the same scene. Older Claude models (Opus 4, 4.1) improve under text substitution — direct evidence they cannot extract enough precise spatial detail from pixels alone. Opus 4.6/4.7 and Gemini regress under text substitution, evidence newer models pull more spatial information from raw images. Also documents a concrete real-hardware failure mode with a physical Unitree Go2: Opus 4.6 correctly identifies a trash can in the egocentric view but reasons that because it is “to the left of the crosshair” it is safe to proceed — walks into it and drags it several meters. Crosshair overlays help alignment in cleaner scenes but distract from obstacles, and a depth heatmap left the model “clearly confused” in a corridor turn with plants and a water cooler.
  • [2026-06-30] GeneralVLA-2: Geometry-Aware Reconstruction and Governed Memory for Robot Planning: GeoFuse-MV3D (GeneralVLA-2) names monocular SAM3D-style reconstruction hallucinating pose and unseen-side geometry as the v1 GeneralVLA failure mode when calibrated multi-view RGB-D is actually available; fixes it via input-view mask verification, soft visual-hull support, axis-wise refinement, and geometry-only fusion (preserving appearance) — reports CD −2.20% / LPIPS −2.02% / PSNR +2.36% / SSIM +1.03% over MV-SAM3D on GSO-30.
  • [2026-06-29] Do multimodal models imagine electric sheep?: Ramakrishnan et al. invert the page’s failure-mode framing: under open-loop behavior cloning on spatial puzzles, a Qwen3.5 9B VLM’s hidden activations at action boundaries encode enough visual information that a separately-trained MAE-style decoder reconstructs intermediate post-action geometry — emergent mental imagery without any visual supervision past the first frame. Direct positive counterpart to Hidden in plain sight: VLMs overlook their visual representations: there the visual features are present at every layer but ignored at the readout; here they additionally encode predicted future state, and a small auxiliary loss is enough to make the LLM use them (3.4-point avg gain at zero inference cost; 83% → 89.5% with visual tokens in CoT). Suggests the “LLM-head ignores image” failure mode has a training-recipe-side fix that doesn’t require LoRA on closed models or input-side scaffolding.
  • [2026-06-29] ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation: ABACUS attacks the counting failure mode from the training-side, RL post-training angle complementary to the input-side scaffold of Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs: a 3B unified foundation model is adapted with density-aware adaptive zooming, a boundary-aware GRPO count policy (targeting crop-edge double-counting that input-side scaffolds can’t address), and cycle-consistent GRPO where the model’s understanding branch self-critiques generations; claimed SOTA across seven counting benchmarks without benchmark-specific training.
  • [2026-06-25] ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning: ZeroDex targets the single-view VLM depth-drift failure mode with a sensor-free geometric fix: reference-view ray voting walks along the semantic camera ray from the reference view and votes for candidates that are geometrically consistent with neighboring calibrated views, combined with triangulation of view-wise groundings — claimed to beat single-view RGB-D grounding pipelines without requiring depth sensors.
  • [2026-06-23] Seed2.1: A Next-Generation Agent for Real-World Productivity: Seed2.1 reports large leads on the perception-stress benchmarks: BabyVision 73.7 vs Gemini 3.1 Pro 54.4 (+19), WorldVQA 53.0 vs 44.3 (+9), ERQA spatial reasoning 72.0 vs 70.8, ZEROBench (w. tool) 22.0 vs 13.0 (GPT-5.5) / 12.0 (Gemini) — suggesting that the gap between frontier VLMs on these benchmarks and the open-perception-failure literature is now narrowing from the closed-source side.
  • [2026-06-21] OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains: OmniVideo-100K’s motivation explicitly names two failure modes of prior AV-QA data: (a) decoupled per-modality captioning severs sound–source associations and produces inconsistent entity descriptions across segments, and (b) coupling long-text comprehension with QA synthesis in one step yields only localized questions — the data-side counterpart to the audio-blindness diagnosis in Do Audio-Visual Large Language Models Really See and Hear?.
  • [2026-06-20] Spotlight: Identifying and Localizing Video Generation Errors Using VLMs: Spotlight documents two systematic VLM perception failures on AI-generated video: zero-shot Qwen3-VL frequently predicts the entire clip as the error segment (high recall, low precision — failure of temporal localization), and Gemini 2.5 Pro / Qwen3 / windowed Qwen3 all miss a primary error in a simple scene where a car is invisible for most of the clip (Fig. 7b); humans outperform the best VLM by 2× on combined Similarity+Precision and the gap widens at higher severity, opposite to what one would expect if VLMs simply needed more obvious cues.
  • [2026-06-19] Planning with the Views (ViewSuite): ViewSuite adds a fourth failure mode — prospective spatial planning. 13 frontier VLMs ace single-turn view-action tracking (~70%) but collapse on multi-turn Interactive View Planning (Gemini 3.1 Pro 21.3%, every open-weight model <5%), and ≥90% of frontier “successes” follow a visual encounter with the target rather than genuine pose inference. Mechanistically distinct from prior-dominance, head-redundancy, and output-format-collapse. A view-graph-distillation recipe lifts Qwen2.5-VL-7B to 47.8% IVP, beating every proprietary model evaluated, and transfers +9.9 points to MindCube under identical post-training.
  • [2026-06-11] EmbodiedEvalKit — A Unified Evaluation Framework for Embodied AI Benchmarking: EmbodiedEvalKit packages the benchmark roster this concept page documents — BLINK, EmbSpatial, BOP-Ask-adjacent pose/grasp datasets, VSI-Bench, ERQA, SAT, RoboSpatial, COSMOS, OpenEQA — into a single CLI with 14+ supported model families (GPT-5, Gemini 2.5, Gemini Robotics, Qwen2.5/3-VL, InternVL, Molmo, RoboBrain2, VeBrain, MiMo-Embodied, Pelican-VL, Magma, Embodied-R1/R1.5). One eval_<benchmark>.py invocation now produces apples-to-apples numbers across all listed failure modes (memorized-prior, output-format-collapse, head-redundancy) for any of these models without per-model coordinate-system glue.
  • [2026-06-05] EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks: EgoTL-Bench documents long-horizon egocentric failure modes (memory-conditioned planning, distance estimation, room-connectivity drift in world-model rollouts) that frontier VLMs and open-source baselines fail to handle reliably even on minute-scale household tasks.
  • [2026-05-25] Scaling Spatial Intelligence with Multimodal Foundation Models: SenseTime / SLab’s SenseNova-SI. Post-trains Qwen3-VL, InternVL3, and Bagel on SenseNova-SI-8M, 8M samples curated under a rigorous taxonomy of spatial capabilities. Reports 68.8% VSI-Bench, 43.3% MMSI, 85.7% MindCube, 54.7% ViewSpatial, 47.7% SITE, 63.9% BLINK, 55.5% 3DSR, 72.0% EmbSpatial alongside 84.9% MMBench-En. First filed paper that takes the positive side of the spatial-cognition debate this page documents: the gap is framed as taxonomy-driven data curation, not as a representational or readout bug. Sits in direct empirical tension with the data-scaling-runs-out thesis of Cambrian-S: Towards Spatial Supersensing in Video on VSI-Bench-class benchmarks; no VSI-Super eval is in the abstract.
  • [2026-05-25] BOP-Ask: Object-Interaction Reasoning for Vision-Language Models: Bhat et al. (NVIDIA / NYU) build BOP-Ask, a 150k-image / 33.8M-QA dataset for object-interaction reasoning derived from BOP 6D pose datasets (HOPE, HANDAL, YCB-V, LineMOD). Six skills: 3D pose, parallel-jaw grasp (5-keypoint), inter-object RRT trajectory, decluttering order, binary spatial, binary relative-depth. Frontier VLMs (GPT-5, Gemini Robotics-ER 1.5, Molmo-72B, Qwen-VL 2.5, NVILA, RoboRefer, Intern-VL) score far below human on every geometric task; GPT-5 hits infinity on grasp NCE and trajectory distance error. A 2B NVILA fine-tuned on BOP-Ask beats Gemini Robotics-ER 1.5 on every metric and beats the human pose baseline. Adds the “frontier VLMs fail at pixel-precise 3D-grounded output” failure mode, distinct from the existing two on this page.
  • [2026-05-24] Solving Spatial Supersensing Without Spatial Supersensing: Udandarao et al. (Tübingen / Bethge / Prabhu) introduce NoSense, a streaming bag-of-words baseline that uses only SigLIP2 and near-perfectly solves the VSI-Super-Recall long-video benchmark (95-98% across 10-min to 4-hour splits), beating Cambrian-S by ~55 points. Also introduce VSC-Repeat, a self-concatenation perturbation that collapses Cambrian-S’s VSI-Super-Counting accuracy from 42% to 0% by violating a benchmark-construction assumption (rooms never repeat). The NoSense result is the inverse of the “LLM overrides image” pattern already on this page: here the LLM is shown to be unnecessary on a benchmark marketed as requiring spatial cognition + long-horizon reasoning. Same author cluster as Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims (Ameya Prabhu); same methodology pattern of “the simplest baseline that exploits obvious regularities” as a benchmark-side ablation.
  • [2026-05-24] Vision Language Models are Biased: Vo et al. introduce a counterfactual benchmark of popular subjects (Adidas logo, quadruped animals, chess boards, board-game grids, optical illusions, patterned grids, national flags) where a single visual attribute has been minimally perturbed. State-of-the-art VLMs average 17.05% counting accuracy; text-overlay augmentation degrades further; prompt mitigations recover only +2 points. ~$2 of API spend per full run; counterfactual generator is automated and published. This is the behavioral counterpart to the mechanistic story in Hidden in plain sight: VLMs overlook their visual representations.
  • [2026-05-24] Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs: Izadi et al. propose a training-free input-side intervention — overlay 5 numbered horizontal lines + prepend a sequential-scanning instruction — and report large gains on visual search and counting across GPT-4o, Claude 3.5 Sonnet, LLaMA-4 Scout, and Qwen2.5-VL-7B. Pure-text CoT fails to recover the gain. Framed as the visual analogue of Chain-of-Thought, attacking the binding-problem variant of the LLM-prior-dominance failure.
  • [2026-05-24] Hidden in plain sight: VLMs overlook their visual representations: Fu et al. provide the mechanistic diagnosis. Compare LLaVA-v1.5-style VLMs wrapping four frozen ViT-L/14 backbones (DINOv2, ViT-IN1k, CLIP, SigLIP). Wrapping causes a universal drop from near-ceiling visual performance to near-chance VLM performance and inverts the encoder ranking. Intermediate-layer probes show the visual features are intact at every layer; LLM-LoRA at matched parameter count beats projector- and ViT-LoRA on five of six tasks and undoes the language-prior bias measured by TV distance to blind baselines.

Open questions

Section titled “Open questions”
  • The three filed papers converge on “LLM half is the bottleneck” but split on the right intervention: input-side scaffolding (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs), LLM-LoRA at moderate scale (Hidden in plain sight: VLMs overlook their visual representations), or a behavioral benchmark with no proposed fix (Vision Language Models are Biased). A head-to-head A/B on the same benchmark (e.g. counterfactual Adidas logos) of (a) input scaffold, (b) LLM-LoRA, (c) chain-of-thought, (d) “look again” prompting would isolate which class of fix targets which failure sub-mode.
  • The binding-problem failure (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs) and the memorized-prior failure (Vision Language Models are Biased) may have different mechanistic substrates. The horizontal-line scaffold attacks parallel feature pooling; it is unclear whether it helps when the failure is canonical-subject prior dominance rather than cross-object misbinding. Vo et al.’s benchmark is the natural testbed.
  • Fu et al.’s LLM-LoRA result is a training-time fix; Vo et al.’s benchmark is presented for frontier closed models (o3, Gemini 2.5 Pro) that cannot be LoRA-tuned by users. The closed-model audience needs either prompt-side or scaffold-side fixes; the open-model audience can apply LLM-LoRA. No filed work yet bridges the two.
  • The Vo et al. “text overlay makes it worse” result suggests there is a per-token competitive interaction between visual-token-derived identity and text-token-derived identity inside the LLM, and that the text token wins on counter-canonical images. No mechanistic paper has yet localized that interaction.
  • Does the same disconnect generalize to video VLMs and to unified models that internalize the visual encoder? Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation argues removing the encoder helps fine-grained understanding; NEO-unify: Building Native Multimodal Unified Models End to End (NEO-unify) makes a similar bet via Mixture-of-Transformer. Solving Spatial Supersensing Without Spatial Supersensing is a partial data point: on long-video retrieval-style tasks, a fully encoder-only pipeline (no language model whatsoever) beats the encoder+LLM stack — suggesting at least one corner of the video-VLM design space where the language head is net-negative.
  • Vo et al.’s benchmark is constructed from popular subjects whose canonical attributes are widely known. For an open-weight VLM trained on a different data distribution, the same images may not trigger the same priors. The benchmark may need to be re-rolled per training distribution to remain diagnostic.
  • For Luma’s pipeline specifically: any system that emits subject-name captions and then re-ingests them into a downstream VLM step (e.g. for relabeling, filtering, or evaluation) is at risk of amplifying the memorized-prior bias documented here. No filed paper measures the cascade effect.
  • New: how to reconcile “the LLM head ignores the image” with “the LLM head was unnecessary”? Both findings target the same architectural component but reach opposite conclusions about its contribution. Plausibly task-dependent — vision-centric MCQ rewards LLM-prior dominance; long-video object-recall rewards retrieval — but no filed paper varies the task type continuously to map out where on the spectrum the language head crosses from “helpful” to “redundant” to “actively harmful”.
  • New: is the BOP-Ask output-format-collapse failure the same mechanism as the binding-problem failure, or a separate one? BOP-Ask: Object-Interaction Reasoning for Vision-Language Models does not perform the layerwise-probe experiment that would localize whether GPT-5’s grasp/pose collapse is at the LLM readout (matching the Fu et al. pattern) or at the visual representation (a genuinely different bug). The clean diagnostic would be running NVILA-2B’s pre-finetuning visual features through the BOP-Ask grasp head trained as a linear probe — if it recovers performance, the failure is at the LLM readout, matching the existing concept-page story.
  • New: data vs mechanism. Scaling Spatial Intelligence with Multimodal Foundation Models and Cambrian-S: Towards Spatial Supersensing in Video make opposite bets — taxonomy-driven 8M curation vs predictive-world-model inference machinery — about what unlocks spatial reasoning. The clean comparison would be the SenseNova-SI 8M curriculum applied to a Cambrian-S-style base (or vice versa), and joint evaluation on both VSI-Bench and VSI-Super; no filed paper does this.

Papers

Section titled “Papers”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2025-11

Section titled “2025-11”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”