Latent Visual Reasoning

TL;DR

LVR extends autoregressive CoT of a Qwen2.5-VL MLLM into the visual embedding space: when the model emits a <|lvr_start|> tag, it stops sampling text tokens and instead passes its last hidden states directly back as input embeddings, generating a fixed budget (empirically best: 4-16 steps) of “latent visual thoughts” before switching back to text with <|lvr_end|>. During SFT the latent hidden states are supervised by an MSE reconstruction loss against ground-truth ROI visual-patch embeddings from the frozen ViT (a teacher-forced “look at these specific patches” objective). A second RL stage adapts GRPO to policy-optimize text answers while replaying the latent hidden states via teacher-forcing so importance ratios remain well-defined despite the lack of a token distribution on the latent segment. On Qwen2.5-VL-7B: MMVP 78.5 → 81.7, V*-Attribute 81.7 → 84.4, V*-Spatial 73.7 → 79.0, BLINK-Jigsaw 66.7 → 72.0 — beats PixelReasoner (which invokes external image-editing tools) on both V* subsets without any tool calls.

Key claims

Reasoning directly in visual embedding space (via last-hidden-state passthrough gated by <|lvr_start|> / <|lvr_end|>) beats both pure-text CoT (“Think about Images”) and tool-calling image-edit CoT (“Think with Images”) on the same Qwen2.5-VL-7B base: +3.2 MMVP over Qwen2.5-VL, +2.7 V*-Attribute, +5.3 V*-Spatial, +5.3 BLINK-Jigsaw [Table 1, §4.3].
The visual reconstruction target is query-relevant visual patches, not the whole image: for each training example an ROI bounding box picks which flattened ViT patches the latent hidden states must reconstruct via MSE — an explicit “look at this region while reasoning” supervision [§3.2.1, Eq. 1].
The joint SFT objective is L = L_reconstruction + α·L_NTP, a standard next-token cross-entropy on the text response combined with the MSE-to-ROI-patches loss on the latent segment; only the LLM is trained, ViT and projector stay frozen [§3.2.1, Eq. 3].
Standard GRPO cannot be applied directly to a latent-reasoning model because the policy-gradient importance ratio is defined over token distributions and the latent segment has no explicit token distribution; the paper’s fix (“Latent GRPO”) records the last hidden states during rollout and patches them back into the latent positions during the teacher-forcing forward pass used to compute log-probabilities on the text output [§3.2.2, Eq. 4-5].
Reward is text-only (format reward = 1 if both <|lvr_start|> and <|lvr_end|> appear; accuracy reward = 1 if answer correct); the format reward is what implicitly encourages the model to use the LVR machinery during RL rollouts. No supervision on the intermediate latent hidden states in the RL stage [§3.2.2].
Fixed-token decoding (constant latent-step budget of 4/8/16) empirically beats two smarter alternatives — a learnable “latent end token” tensor in the hidden-state space, and a mode-switching BCE loss that supervises the LM head’s <|lvr_end|> prediction during latent reasoning. Mode-switching loss “fails to work as intended” [§3.3, §4.5].
LVR trails on Relative-Reflectance (a multi-image BLINK task) — attributed to training data being single-image-only rather than to a fundamental limitation of the paradigm [§4.3].
RL further improves LVR: Table 2 reports RL-stage gains stacked on top of SFT-only LVR, though exact deltas are truncated in the retrievable snapshot. Format+accuracy reward suffices; no per-latent-step supervision needed [§4.4].
SFT training data is Visual CoT (438K image-question-ROI-bbox triples, Shao et al. 2024) — an off-the-shelf visual-COT SFT set with region-of-interest annotations already present, no new data collection required [§4].
Adaptive multimodal data packing (variable image + latent-token count → variable sequence length) is required to keep per-device batch size healthy (~3.2 average); 7B SFT is 2500 steps × ~40 hours on a 4× AMD MI250 cluster [§4].

Method

The base VLM is Qwen2.5-VL 3B/7B; the vision encoder and multimodal projector are frozen. Input image is patchified by the ViT into a flattened visual-token sequence V; text query is embedded as L. During SFT, each example carries an ROI bounding box, from which LVR extracts the indices of patches inside the ROI and gathers the corresponding embeddings from V. These embeddings become the target for a contiguous span of hidden states in the LLM decoder, framed by <|lvr_start|> and <|lvr_end|> tokens, followed by the ground-truth text answer. Two losses train jointly: an MSE reconstruction loss (Eq. 1) between the LLM’s last hidden states inside the framed span and the ROI patch embeddings, and standard next-token cross-entropy (Eq. 2) on the response tokens outside the span. Balance is a scalar hyperparameter α (Eq. 3).

At inference, whenever the LM head emits <|lvr_start|>, the decoder switches into “latent reasoning” mode: the last hidden state at each position is passed directly as the next-position input embedding (rather than sampling a token and re-embedding), for a fixed budget of k ∈ {4, 8, 16} steps. The LM head’s token predictions during this window are ignored. After the budget is exhausted, <|lvr_end|> is inserted and text generation resumes normally.

The RL stage uses GRPO-style rollouts on ViRL data (8 responses per input, temperature 1.0, KL coeff 0.001, learning rate 1e-6). The key adaptation: at policy-gradient computation time, the rollout’s recorded latent hidden states are patched into the current-policy forward pass so that log-probabilities on the text tokens after the latent segment are conditioned on exactly the same intermediate state as at rollout time — otherwise the importance ratio π_θ(text | context) / π_θ_old(text | context) cannot be computed because the current-policy would produce different latent hidden states, breaking the conditional. The reward is a scalar over the final text output only.

Results

Qwen2.5-VL-7B baselines vs LVR (Fixed-Token decoding, best step count):

MMVP: 78.5 → 81.7 (+3.2) with 8-step LVR
V*-Attribute: 81.7 → 84.4 (+2.7) with 4/8-step LVR
V*-Spatial: 73.7 → 79.0 (+5.3) with 16-step LVR
BLINK-Counting: 66.7 → 72.0 (+5.3) with 4-step LVR
BLINK-IQ-Test: 67.5 → 70.8 (+3.3) with 16-step LVR
BLINK-JigSaw: 26.0 → 29.3 (+3.3) with 8/16-step LVR
BLINK-Spatial-Relation: 87.4 → 89.5 (+2.1) with 4-step LVR

LVR beats PixelReasoner (which crops image sub-regions via external tools) on V*-Attribute (84.4 vs 81.7) and V*-Spatial (79.0 vs 77.6) without any tool calls [Table 1]. PAPO and Vision-R1 (pure-text CoT baselines) degrade on V* and MMVP relative to the same-base Qwen2.5-VL, isolating the modality choice — not the reasoning-format tax — as the load-bearing factor.

Relative-Reflectance is the one benchmark where LVR trails (52.7 for LVR-4/16 vs 52.7 for Qwen2.5-VL baseline — parity, not gain); attributed to single-image training data vs the multi-image nature of the task.

RL further improves over SFT-only LVR at the 3B scale (7B RL not run due to compute). Ablations in §4.5 show Fixed-Token decoding is the best of three strategies tested; the learnable “latent end token” and the mode-switching BCE loss both fail to converge on when to exit latent reasoning.

Why it’s interesting

LVR is the direct predecessor of Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens on the substrate (a) autoregressive-tagged-latent axis of Thinking with Modalities: same Qwen2.5-VL backbone, same “emit latent tokens inside CoT that reconstruct visual content”, same MSE reconstruction loss family, same frozen-ViT / trained-LLM stack, same argument that pure-text CoT is the wrong substrate for vision-centric reasoning. The differences sharpen the substrate:

LVR supervises against ViT patches of a query-relevant ROI from the input image itself — a single supervision source that requires bbox-annotated training data (Visual CoT). CoVT supervises against four heterogeneous vision experts (SAM masks, DepthAnything-v2 depth, PIDINet edges, DINOv2 features) — a richer signal that doesn’t need ROI bboxes but does need multiple frozen experts and per-expert alignment strategies (prompt-level for decoders, feature-space MSE for representation encoders).
LVR does the RL half. LVR adapts GRPO to latent-reasoning models via hidden-state replay in the teacher-forcing pass — a specific engineering solution to the “importance ratio has no token distribution” problem that any latent-reasoning follow-up will need. CoVT (per the current wiki page on it) does not run an RL stage; the Eq. 5 hidden-state-replay trick is genuinely new machinery.
LVR pins down inference-time decoding. The “fixed step budget” ablation, and the negative result on both learnable-end-token and mode-switching-loss decoding strategies, is a concrete finding that CoVT’s fixed-8+4+4+4 (Segmentation + Depth + Edge + DINO) budget quietly inherits without reproving.

Kyle’s Slack point — that CoVT does not cite LVR — reads as more than a citation gripe: the two papers stake out the same paradigm two months apart on the same base model, and LVR’s ROI-reconstruction supervision is arguably the minimal form of what CoVT scales up. The current concept-page framing of CoVT as “substrate (a) with expert-supervised tokens vs Video-Thinker’s self-invoked ones” needs a third node: LVR is substrate (a) with self-supervised tokens (the target is ROI patches from the same frozen ViT the LLM already reads), a middle position that is neither self-invocation nor external-expert supervision.

Also sharpens the VLM Perception Failures story: Hidden in plain sight: VLMs overlook their visual representations showed the visual features are already present at every LLM layer but overridden at the readout. LVR’s ROI-reconstruction loss trains the LLM to re-emit those very features mid-CoT, essentially forcing the readout to route through the visual embedding space at least once per reasoning step. This is a specific mechanistic prediction for why the paradigm works — different from CoVT’s “internalize a heterogeneous tool bank” framing.