Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

ListenThinking in Pixels: Continuous Visual Tokens for VLMs10:43

Show notes

This episode digs into Chain-of-Visual-Thought (CoVT), a framework that lets vision-language models reason through a small budget of continuous visual tokens supervised by lightweight vision experts — segmentation, depth, edges, and DINO features. We cover the mechanism, the four-phase training curriculum, and the load-bearing ablation showing that text-only chain-of-thought actually degrades the same vision-centric tasks. Reported gains run 3 to 16 percent on benchmarks like CV-Bench, with a standout plus-14 on depth. **Sources** - Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

CoVT augments a VLM’s autoregressive CoT with a small budget (~20) of continuous visual tokens — each group supervised to reconstruct the output of a lightweight vision expert (SAM masks, DepthAnything v2 depth, PIDINet edges, DINOv2 patch features). At training time, gradients from expert-reconstruction losses flow through the visual tokens and shape the VLM’s latent representations; at inference, the model reasons in the continuous visual space without decoding the tokens (retaining efficiency) or optionally decodes them for interpretability. Applied to Qwen2.5-VL-7B and LLaVA-v1.5-13B, CoVT reports 3–16% gains on vision-centric benchmarks (CV-Bench, MMVP, HRBench, MMStar-P), with the largest jump on CV-Bench-Depth (+14%). The paper’s clean ablation shows text-only CoT degrades the same tasks — the visual tokens, not the reasoning-format tax, are load-bearing.

Key claims

Adding 3 CoVT tokens (Segmentation + Depth + DINO) to Qwen2.5-VL-7B lifts CV-Bench overall by +5.5, with sub-task gains of +14.0 depth, +7.0 distance, +1.2 count; also +4.5 on HRBench8K, +3.7 on MME-RealWorld, +2.7 on MMStar-P, +3.0 on RealWorldQA [Table 2, §4.3].
Text-only CoT (LLaVA-CoT-100k formatting on the same training data) fails to improve vision-centric benchmarks and often degrades them; CoVT with the same base and data improves consistently — isolating the continuous visual tokens (not the format tax) as the load-bearing lever [§4.5, Fig. 6].
Alignment strategy depends on expert granularity: fine-grained task-oriented experts (SAM, DepthAnything v2, PIDINet) are aligned at the prompt level of their decoders (visual tokens are fed in as decoder prompts); representation experts (DINOv2) are aligned in feature space via MSE after projection [§3.3].
Token count matters non-monotonically: 8 Segmentation tokens + 4 Depth + 4 DINO is the balanced configuration; 32 Segmentation tokens hurts (harder to align a large mask token set); “16 empty” tokens with no visual supervision performs worse than 0 tokens — visual alignment, not reasoning-slot capacity, is what helps [§4.5, Table 4].
Cross-baseline generalization: same recipe on LLaVA-v1.5-13B beats Aurora (the prior latent-visual-token baseline that used VQ-VAE latents of depth/detection) by +12.9 on BLINK relative-depth and +26.6 on BLINK count when using Segmentation tokens [Table 3, §4.3].
Training pipeline is four staged phases: comprehension (visual tokens after <image>, teach the semantics) → generation (learn to emit them precisely) → reasoning (embed them inside CoT) → efficient reasoning (random dropout of token-type subsets so the model doesn’t rely on a fixed pattern) [§3.4, Fig. 4].
Qualitative decode of CoVT tokens shows the four experts contribute complementary cues: Segmentation localizes a specified point, Depth encodes camera-relative distance, Edge supplies boundary structure for closely-spaced objects, DINO provides patch-level semantics. Different tasks recruit different subsets [Fig. 5, §4.4].
Inference-time cost is preserved: the continuous tokens are consumed inside the LLM’s normal next-token prediction pass and are not decoded to pixel outputs unless interpretability is requested. Decoding to masks/depth/edges is optional [§3.4].

Method

The base VLM (Qwen2.5-VL-7B or LLaVA-v1.5-13B, LoRA-tuned; rank 16, alpha 32) is trained to emit a sequence that interleaves text tokens with contiguous groups of continuous visual tokens (8 Segmentation, 4 Depth, 4 Edge, 4 DINO in the four-expert setting). Each group is projected into the corresponding expert’s input space and used as follows: (i) Segmentation tokens become 8 SAM decoder prompts, each generating a mask; Hungarian matching + dice + focal loss aligns them to ground-truth instances [Eq. 2]. (ii) Depth tokens become 4 DepthAnything v2 decoder prompts that interact with the DepthAnything encoder’s mid-layer features via batch matrix multiplication to reconstruct a depth map under L1 loss [Eq. 3]. (iii) Edge tokens act as 3×3 convolutional kernels applied to PIDINet’s dense features, reconstructing edge maps under L1 loss. (iv) DINO tokens are MSE-aligned to DINOv2 patch features in shared feature space. The joint objective is the standard next-token cross-entropy plus these four reconstruction losses [Eq. 4]. Data mixes vision-centric LLaVA-OneVision subsets with TallyQA + ADE20K-Depth (spatial perception). The four-phase curriculum progressively teaches comprehension, generation, reasoning-with-visual-tokens, and finally random-dropout robustness.

Results

CV-Bench overall: 74.5 (Qwen2.5-VL-7B baseline) → 80.0 with 3 CoVT tokens → 79.8 with 4 CoVT tokens (+5.5 / +5.3) [Table 2]. Sub-tasks: Count 65.0 → 66.2, Depth 72.8 → 86.8 (+14.0), Distance 75.5 → 82.5 (+7.0). Other vision-centric benchmarks (3 tokens): BLINK 55.7 → 56.0, RW-QA 68.6 → 71.6, MMT 61.7 → 62.1, MMStar-P 67.1 → 69.2, MMVP 56.0 → 58.7, MME-RW 60.0 → 63.7, V* 76.4 → 78.0, HRBench 68.6 → 72.9. On LLaVA-v1.5-13B, CoVT-Depth beats Aurora-depth on every BLINK sub-task, including +12.9 on Relative-Depth (62.9 → 75.8) and +5.2 on Visual Correspondence [Table 3]. Ablation: “16 empty” reasoning tokens (no visual supervision) score 75.7 CV-Bench vs 80.0 with 8 supervised Segmentation + 4 Depth + 4 DINO — confirms it’s the alignment, not the extra latent slots [Table 4].

Why it’s interesting

CoVT stakes out a specific position on the Thinking with Modalities substrate axis: it’s autoregressive-tagged-latent (substrate (a), the same family as Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning) but the tags aren’t self-invocations of the model’s own perception — they’re supervised against external experts during training and then consumed without decoding at inference. This lands closest to Context Unrolling in Omni Models‘s “internalized tool” framing and to Do multimodal models imagine electric sheep?‘s finding that a small auxiliary visual-prediction loss (with no inference overhead) suffices to make the LLM stop ignoring its own visual features. It also complements Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs (input-side scaffolding) and Hidden in plain sight: VLMs overlook their visual representations (LLM-LoRA readout fix) as a training-side answer to the VLM Perception Failures “LLM head ignores image” diagnosis — same locus (the language side of the VLM), different intervention (add supervised latent visual tokens the LLM must produce, rather than lifting the language prior out of the readout). The clean text-CoT-degrades-while-visual-CoT-improves ablation is the cleanest such isolation on Qwen2.5-VL published to date; the concept page’s anti-pure-text-CoT thesis gets a new anchor from an established VLM base rather than from LLaVA-only or maze/Sudoku settings.