Hidden in plain sight: VLMs overlook their visual representations

TL;DR

UC Berkeley study comparing four ViT backbones (DINOv2, ViT-IN1k, CLIP, SigLIP) to LLaVA-style VLMs built on the same frozen encoders across six vision-centric tasks (depth order, semantic / functional / low-level correspondence, art style, 3D object awareness from MOCHI). Wrapping a vision encoder in a VLM causes a universal collapse from near-ceiling to near-chance performance — e.g. DINOv2 low-level matching 0.819 → 0.170 — and the VLM ranking does not preserve the vision-only ranking (DINOv2 wins on vision evals but loses disproportionately as a VLM backbone). Probing intermediate VLM layers shows the visual features remain intact through the projector and most of the LLM; the failure is at the use step, not the representation step. Prompt-tuning recovers very little; the only ablation that meaningfully closes the gap is LoRA-tuning the LLM itself, which also reduces the VLM’s reliance on its language-prior “blind” answer distribution.

Key claims

Switching from a direct visual readout (linear probe / DPT head / cosine similarity / Gram matrix) to a VQA wrapper over the same frozen encoder produces a universal drop in vision-centric performance, often to chance — e.g. DINOv2 low-level matching 0.819 → 0.170, depth order 0.887 → 0.653, art style 0.675 → 0.530 [§3.2, Table 1].
VLM-induced performance changes the rank ordering of vision backbones: DINOv2 leads on every spatial vision-centric task as a probe but is disproportionately penalized inside a VLM, so CLIP/SigLIP appear stronger when judged only by VLM benchmarks — calling into question prior conclusions (from Cambrian / Prismatic) that vision-language pretraining wins for VLMs [§3.3, Fig. 2].
The drop is not driven by representation degradation: probing the same vision tokens after the projector and at every LLM layer continues to solve the task at vision-encoder-level accuracy, with sharp falloff only in the final LLM layer [§4.1, Fig. 5].
VLM output distributions on multiple-choice vision-centric tasks closely match the same VLM’s “blind” output distribution when run with a blank image, measured by total variation between answer histograms — direct evidence the model is ignoring the image and answering from its LLM prior [§3.4, Fig. 4, Table 2].
Prefix prompt-tuning over 1,000 VQA examples produces only marginal gains and saturates by 5–10 trainable prefix vectors; prompt sensitivity is not the bottleneck [§4.2, Fig. 6].
Equal-parameter LoRA (16.7M params, matched to a full-projector finetune) applied to the LLM beats the same LoRA applied to the projector or the ViT on every task except 3D Object Awareness — the LLM is where the capacity to use the vision features lives [§4.3, Fig. 7].
LLM-LoRA also produces the lowest TV distance between predicted-answer and ground-truth-answer distributions across tasks, undoing the language-prior bias that the projector- and ViT-LoRA variants leave intact [§4.3.1, Table 2].
The pattern reproduces on production VLMs with custom-trained encoders — InternVL, Phi-3-V, Qwen-VL — with one caveat: when the encoder itself cannot do the task, the VLM cannot rescue performance [§3.2, Fig. 3].
Attention-shift analysis after LLM-LoRA shows the model learns to attend to multiple-choice labels, reference points, and other task-relevant text regions; projector- and ViT-LoRA do not consistently shift attention this way [§4.3].

Method

The testbed pairs four ViT-L/14 backbones — DINOv2 (self-supervised), ViT-IN1k (ImageNet-supervised), CLIP (vision-language contrastive), SigLIP (vision-language sigmoid) — with Vicuña-v1.5 via LLaVA-v1.5-style projector training (Karamcheti et al. Prismatic checkpoints), so the only thing varying between the “vision” and “VLM” condition is the wrapping, not the encoder weights. Vision evaluations are standard zero-shot probes: DPT head for depth, patch-feature cosine similarity for correspondence, CLS-embedding pairwise distance for 3D Object Awareness, Gram-matrix MSE for art style. VLM evaluations are multiple-choice VQA in the same data points.

For the intermediate-layer probe, the same visual-readout strategies are applied to the projector output and to the residual stream at each LLM layer, treating the LLM as a sequence of representational checkpoints. Prefix prompt-tuning follows Lester et al. (2021) — prepend N learnable embeddings to the prompt, freeze everything else, train next-token cross-entropy on 1,000 examples per task. Component-LoRA ablations apply rank-controlled LoRA to either the ViT, the projector, or the LLM at matched parameter count (16.7M, equal to full projector finetuning) and train on 5,000 examples per task. Blind baselines re-run all VQA with the image replaced by a blank canvas.

Results

Vision vs VLM headline (DINOv2, then averaged across four encoders) [Table 1]: Semantic Correspondence 0.536 → 0.225 (vision → VLM); Low-level Matching 0.819 → 0.170; Depth Order 0.887 → 0.653; Object Affordance 0.411 → 0.155; Art Style 0.675 → 0.530; 3D Object Awareness 0.598 → 0.315.
Rank-order inversion: DINOv2 wins five of six vision evaluations. After wrapping, CLIP/SigLIP are at parity with or ahead of DINOv2 on several VLM scores — most pronounced on Low-level Matching, where DINOv2-VLM drops to 0.170 while CLIP/SigLIP land at 0.181 / 0.246 despite their visual baselines being 0.532 / 0.632 vs DINOv2’s 0.819 [Table 1].
Probing through the VLM: vision features at every projector and LLM layer remain near vision-encoder-level on the same readout; the only sharp decline is at the last LLM layer, attributed to the final-layer shift toward generating natural language tokens [Fig. 5].
Blind vs sighted: TV distance between sighted and blind VLM answer distributions is small across tasks — Original 0.337 (Sem Corr) / 0.494 (Low-level) / 0.521 (Obj Aff) / 0.259 (Art Style) / 0.386 (3D-Aware) [Table 2, “Original” row], roughly tracking the Blind row.
Prefix tuning [Fig. 6]: marginal gains, no saturation point above chance; 1 vs 5 vs 10 prefix vectors flat.
Component LoRA at matched 16.7M params [Fig. 7]: LLM-LoRA > Projector-LoRA > ViT-LoRA on five of six tasks. Exception: 3D Object Awareness, where all three components fail to recover; the authors attribute this to MOCHI’s reliance on abstract object-level features and a ShapeNet-only fine-tune set.
TV distance after LoRA [Table 2]: LLM-FT drops TV to 0.078 / 0.121 / 0.190 / 0.071 / 0.069 (Sem Corr / Low-level / Obj Aff / Art Style / 3D-Aware) — substantially below ViT-FT and Projector-FT on every task except Depth Order, where the original VLM already matched ground truth.

Why it’s interesting

This is the cleanest filed evidence so far that the “VLM can’t see” failure mode is an integration failure rather than a representation failure — the vision tokens are intact at every LLM layer; the LLM just doesn’t attend to them. It directly attacks the methodological premise of CV-Bench-style “use the VLM to benchmark the encoder” workflows that the wiki’s VLM-as-Evaluator cluster takes as a given: when VLMs invert the encoder ranking and bottom out at chance on tasks the encoders solve, VLM scores stop being a measurement of the encoder.

It also recasts a debate playing out across the Unified Multimodal Models page. Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation argues “remove the pretrained vision encoder entirely” and reports better fine-grained understanding; NEO-unify: Building Native Multimodal Unified Models End to End (NEO-unify) makes the same encoder-free + VAE-free bet via Mixture-of-Transformer; Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation adds generation supervision to push understanding up. Fu et al. provide a complementary diagnosis: even the existing encoder + LLM stack already contains the visual information needed — the gap is on the LLM side. That weakens the “vision encoder is the bottleneck” framing and strengthens the case for LLM-side training-time interventions (instruction-tuning for spatial reasoning, attention shaping over visual tokens) over architectural redesign. It also lines up with Laurençon et al.’s IDEFICS observation, cited inside the paper, that the LLM is more of a VLM bottleneck than the visual backbone.

For Luma’s research team specifically: the “blind baseline matches sighted output” experiment is a cheap diagnostic to run on any in-house VLM — if the TV distance between the two answer distributions is small on vision-centric VQA, your model isn’t reading the image regardless of what its benchmark score says.