Skip to content
Bud's Wiki

Thinking with Modalities

TL;DR

Section titled “TL;DR”

Reasoning paradigms where the model’s chain of thought lives in or invokes a non-text modality, rather than emitting plain text tokens. Three filed instances split on who runs the modality and how it’s represented: (a) an autoregressive MLLM that invokes its own latent perception ops via tagged CoT segments and a single forward-pass loop (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning — <grounding>/<caption> over video segments); (b) a diffusion model that treats the solution image itself as the CoT, so the diffusion ODE is the inference-time computation (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models — Qwen-Image-Edit fine-tuned on solution-image flow matching); (c) an MLLM that invokes an external Python sandbox to manipulate the image (crop / rotate / annotate / plot) and appends the transformed image back to its own context across a Think-Act-Observe loop (Introducing Agentic Vision in Gemini 3 Flash — Gemini 3 Flash Agentic Vision). All three argue against pure text-CoT for vision-centric problems; they disagree on whether the modality should be a tagged latent skill, an end-to-end pixel generation, or an external tool action.

Key claims

Section titled “Key claims”

  • Pure-text chain-of-thought is the wrong substrate for vision-centric reasoning — three filed instances independently report large wins by moving the reasoning trace into a visual modality (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §1; DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §1, Table 1; Introducing Agentic Vision in Gemini 3 Flash §intro). The mechanism differs: Video-Thinker improves grounding fidelity (mIoU 27.47 → 48.22 on temporal grounding, Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §4.3); DiffThinker exploits diffusion’s native parallel exploration of candidate solutions in early denoising steps (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2, Fig. 4); Agentic Vision exploits deterministic Python computation on transformed crops (Introducing Agentic Vision in Gemini 3 Flash §“Visual math and plotting”). A fourth datapoint sharpens the negative result: (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs §4.6, Fig. 3) shows that prepending the standard step-by-step CoT prompt to GPT-4o on the same binding-heavy tasks where a visual scaffold gives +25% does not help and can degrade performance — language tokens conditioned on entangled visual features inherit the misbindings, so reasoning over those tokens cannot repair them.

  • The filed instances now span a four-position substrate axis: (a) tagged latent skills inside an autoregressive CoT — same MLLM emits structured <grounding>/<caption> tags it was trained to interpret as self-invocation of its own perception, no external execution (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §3); (b) end-to-end pixel-space CoT — a diffusion model maps (problem_image, instruction) → solution_image and the diffusion trajectory itself is the reasoning, with a downstream symbolic parser extracting the answer (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §3.1, Eqs. 1–4); (c) external-tool Think-Act-Observe loop — the MLLM writes Python that crops/rotates/annotates the image, the transformed image is appended back to context, and the loop continues (Introducing Agentic Vision in Gemini 3 Flash §“Agentic Vision: a new frontier AI capability”); (d) input-side visual scaffolding — fixed, pre-inference overlays (e.g. numbered horizontal lines) plus a sequential-scanning prompt that bias the frozen model’s attention toward region-by-region serial processing, no training and no per-query tool/agentic loop (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs §3). The four substrates differ on training cost (none → SFT → SFT+GRPO → flow-matching) and inference cost (single query → bounded iterations → unbounded loop). No filed paper isolates which substrate wins on a shared benchmark.

  • Training the modality-invocation behavior matters: Video-Thinker’s SFT-only checkpoint underperforms the Qwen2.5-VL-7B base on Video-Holmes (31.52 vs 34.02) until GRPO is applied on top, demonstrating that the structured-trace format is a tax without RL (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §4.2, Table 1). DiffThinker’s base Qwen-Image-Edit-2509 scores 4.0% average vs 87.4% for the same architecture fine-tuned on ~30k flow-matching samples per task (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.1, Table 1). Agentic Vision states only the “implicit zoom” behavior is currently trained-in; other behaviors (rotating, visual math) still require explicit prompt nudges and Google flags training them in as roadmap (Introducing Agentic Vision in Gemini 3 Flash §“What’s next”). Counter-datapoint: (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs §4) shows substrate (d) does not require training at all — a single fixed prompt + a single fixed visual overlay lifts GPT-4o counting accuracy by +26.83% at 0 training cost. So the four substrates also span the training-required axis: (a) and (b) need SFT (+ optional RL); (c) is partially trained, partially prompted; (d) is fully training-free. Refinement: Demystifying Video Reasoning shows the diffusion-native parallel-reasoning substrate doesn’t require fine-tuning at all — pretrained video DiTs exhibit Chain-of-Steps (multiple candidate solutions held simultaneously in early denoising steps) by default. So the “trained vs prompted” axis collapses for substrate (b) too: the substrate is there at pretrain time; fine-tuning just teaches the model which problems to solve in it.

  • Diffusion-based reasoning is the only substrate that gives native parallel exploration — by decoding the predicted clean latent at each diffusion step, the model is visibly holding multiple candidate solutions simultaneously in early steps (multiple maze trajectories spread across the grid) and consolidating only late (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2, Fig. 4). The other three substrates serialize: Video-Thinker emits one grounding/caption choice per token-prefix, Agentic Vision emits one Python action per Think-Act-Observe iteration, and the input-scaffold substrate explicitly enforces serialization (one row at a time) as its mechanism of action (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs §3). Generalization: the same parallel-exploration dynamic is observed in video diffusion (Demystifying Video Reasoning Fig. 2) — multiple robot paths, multiple placements, multiple object rotations superimposed in early x̂_0 decodes — independently confirmed for maze-solving in Video Models Reason Early (Newman et al., arXiv 2603.30043) which reports plan commitment within the first few denoising steps. The substrate (b) parallel-exploration property is therefore a property of diffusion generally, not of the image-only fine-tuned setting.

  • Inference cost shape differs across substrates: DiffThinker is fixed-budget (20 Euler steps, 1.1 s) and accuracy plateaus past the budget (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2, §4.3 Figs. 5b, 7); Video-Thinker scales linearly with the number of self-invoked tags inside its CoT; Agentic Vision scales with the number of Think-Act-Observe iterations and is unbounded above by how many crops the model decides to take; the input-scaffold substrate (d) is single-query, negligible overhead — one image-edit at preprocessing time and one extra prompt sentence (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs §3, §4). Productionizing the multi-iteration substrates requires a budget cap; (b) bakes it in; (d) doesn’t need one. Demystifying Video Reasoning §5 supplies a training-free way to spend more compute on substrate (b): ensemble latent trajectories across random seeds (each seed explores a different set of early-step candidates), then aggregate — i.e. step count is fixed but seed count is a tunable budget.

  • Modality-CoT composes with text-CoT verifiers when the two are split by role: DiffThinker × MLLM-verifier strictly beats either alone on Jigsaw-level-4 — DiffThinker proposes multiple candidate reconstructions, the MLLM picks against the original constraints (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2, Fig. 6). Agentic Vision’s productized form bakes this in: code-execution is the proposal, the MLLM’s final answer is the verification. The pattern is “pixel-space generator proposes; token-space reasoner verifies”.

  • Image-based reasoning beats video-based reasoning at current scale, even on tasks where video is the natural modality (a maze ball video reaching the target): DiffThinker-Video on Wan2.2-TI2V-5B inherits the substrate but yields lower accuracy with higher training overhead and ~2× inference latency (2.0 s vs. 1.1 s) (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.4, Fig. 11). This is a useful negative result for any team contemplating a “thinking video model” — the compute / sample-efficiency ratio is currently bad. Conflict: Demystifying Video Reasoning argues video diffusion already exhibits the parallel-reasoning substrate at pretrain time (no fine-tuning) — so the bad cost ratio in DiffThinker-Video is plausibly an artifact of fine-tuning a small (5B) base, not a structural problem with video as a reasoning substrate. The two findings are reconcilable: the substrate exists in pretrained video DiTs (Demystifying); fine-tuning a small video DiT on a narrow reasoning task is currently uneconomical (DiffThinker). The cost crossover where it becomes economical is open.

  • Frontier labs have begun productizing the substrate, which sets the apples-to-apples baseline going forward: Google reports a consistent 5-10% quality boost across most vision benchmarks from enabling code execution with Gemini 3 Flash (Introducing Agentic Vision in Gemini 3 Flash §intro). Per-benchmark numbers and ablations are not disclosed in the blog post, but the implication is that future vision-benchmark scores from frontier labs will be code-execution-enabled by default.

  • Inside a single denoising step, a DiT exhibits layer-wise functional specialization: early layers encode dense perceptual structure, middle layers execute reasoning, later layers consolidate latent representations (Demystifying Video Reasoning §Abstract). This is a mechanistic prediction with consequences for substrate (b) design: depth-targeted fine-tuning (or extra compute on middle layers only) is the natural lever for teaching new reasoning behaviors, rather than fine-tuning the whole stack.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-19] RynnBrain 1.1 — Open Embodied Foundation Models (Alibaba DAMO Academy): RynnBrain 1.1 adds two new first-class output modalities beyond text and 2D boxes: 3D bounding boxes conditioned on camera intrinsics, and instruction-conditioned contact points with in-plane orientation — reasoning outputs pushed past image-plane localization into metric-3D and action-relevant interaction grounding.
  • [2026-07-17] HDR: Hierarchical Denoising for Multi-Step Visual Reasoning: HDR is a new architectural realization of substrate (b) diffusion-native reasoning on video: latents are organized as a temporal tree, upper levels hold global hypotheses (early-denoising role), lower levels consolidate into concrete frames (late-denoising role), and a sparse hierarchical attention pattern (SHAP: local + parent + first-frame) lets coarse plans flow into fine generation through a shared KV cache. On six long-horizon reasoning tasks (maze, Hanoi, one-line drawing, sliding puzzle, Sokoban, water pouring) HDR beats a streaming AR-diffusion baseline 34.22 → 60.29 overall success at 0.70s-per-latent — a data point that the DiffThinker-Video “video reasoning is uneconomical” negative result reverses once the diffusion trajectory is structured rather than flat.
  • [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain interleaves reasoning text and flow-matched imagined frames in a single autoregressive sequence: a learned <Image> token decides when to imagine, each generated frame is re-encoded and appended to context, and RxBrain-Bench evaluates whether embodied plans genuinely couple textual and visual components.
  • [2026-07-15] Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models: FoveateR (Min, Kumar et al.) internalizes substrate (c) — the external Think-Act-Observe zoom loop that Gemini 3 Agentic Vision runs via Python — into a single stateful decoding pass by emitting non-linguistic foveation actions (spatial region selectors distinct from the text vocabulary) that inject high-res patch tokens back into the same trajectory. Contrasts with (c)‘s multi-pass overhead and text-token budget cost, and with input-side scaffolding (d) by being reasoning-driven rather than static. Two-stage SFT-then-RL trains the policy without human gaze labels; pseudo labels produce multi-fixation behavior (~1.8 fixations vs 1.0 for human-labeled control).
  • [2026-07-10] Context Unrolling in Omni Models: Arxiv version of ByteDance Seed’s Omni — the “internalized tool” substrate position, formalized as iterative context construction (Eq. 1) with atomic primitives (text-think, visual-token rollout, camera-pose, novel-view synthesis, depth) invoked by the same MoE. Ablations show textual and visual contexts are non-redundant, and task-relevant textual context (depth caption) helps while generic captions do not — sharpens the “constraint-like intermediate” claim on this page.
  • [2026-07-09] Introducing Muse Spark 1.1: Muse Spark 1.1 productizes visual-to-code artifact generation and image/video-grounded computer use — perception and action interleaved within one agent’s thought process, demoed via a Facebook Marketplace listing agent that extracts photos from smartphone video and drives a browser.
  • [2026-07-07] Muse Video preview — MSL video generation model announcement (Alexandr Wang): Muse Image productizes the external-tool Think-Act-Observe substrate on the generation side — Muse Spark reasons over the prompt, searches the web, and plans before Muse Image generates pixels. Closed-source frontier version of the pattern that InterleaveThinker (open) and Reve’s Layout Bet (planner-first) explore in open research.
  • [2026-06-29] Do multimodal models imagine electric sheep?: Ramakrishnan et al. (Anthropic / Columbia / TAU / UT Austin) train Qwen3.5 9B with open-loop behavior cloning on twelve spatial puzzles and show mental imagery emerges from action-only supervision: a frozen visual decoder reconstructs intermediate states from last-action-token hidden activations. Sharpening that latent imagery via either an auxiliary visual-prediction loss (no inference overhead, +3.4 points) or 16-64 interleaved VQ-VAE tokens per action (+6.5 points to 89.5% avg) beats a matched textual-JSON-scene-graph CoT by ~10 points on the same base model. The matched textual-CoT control is the cleanest published isolation of “visual modality beats test-time compute” for substrate (a), and the 16-tokens-per-state result anchors the compact end of the substrate-cost axis.
  • [2026-06-23] MetaPoint: Unlocking Precise Spatial Control in Agentic Visual Generation: MetaPoint extends the coordinate-token-as-primitive paradigm (Thinking-with-Visual-Primitives, MolmoPoint, LocateAnything) from the output / CoT side to the generation-conditioning side: a planner agent emits sequences of MetaPoint tokens (1 token = point, 2 tokens = bbox) that a generator consumes natively via the backbone’s existing positional encoding, with no architectural change. A minimal token-level realization of the planner→primitives→generator pattern that Reve’s “Layout Bet” argues for.
  • [2026-06-21] OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains: OmniVideo-100K operationalizes “evidence chains” as a data-construction primitive for AV-QA: Clue-Guided QA Generation first mines cross-segment multimodal clues from a structured script, then writes the QA pair grounded on those clues, biasing the resulting instruction-tuning set toward long-horizon cross-modal reasoning rather than localized intra-clip facts.
  • [2026-06-13] InterleaveThinker: Reinforcing Agentic Interleaved Generation: InterleaveThinker is a (c)-style external-tool Think-Act-Observe substrate where the “tool” is a powerful frozen image generator (FLUX.2-klein etc.) and the loop ranges over an entire planned narrative of generations rather than a single output — sharpens the substrate axis beyond Agentic Vision’s single-call Python crops and EditThinker’s single-edit refinement.
  • [2026-06-05] EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks: EgoTL provides minute-long egocentric video with human spoken CoT aligned at word-level timestamps to actions and metric spatial labels, offering a substrate for interleaved video-language reasoning fine-tuning grounded in real physical execution.
  • [2026-06-04] The Layout Bet: Reve’s Layout Bet argues for a third reasoning substrate — neither text CoT nor pixel/video CoT but a structured layout (hierarchical regions with per-element captions) — that the model derives in its internal thinking trace and then renders to pixels.
  • [2026-06-02] Swift Sampling: Selecting Temporal Surprises via Taylor Series: Swift Sampling is the static, query-agnostic end of the spectrum — a training-free, Taylor-expansion-based frame selector that picks “temporally surprising” frames before any VLM/CoT runs; contrasts with active-perception substrates like Video-Thinker where the reasoning model itself drives perception.
  • [2026-06-01] How Cosmos 3 Helps Physical AI Think Before It Acts: Cosmos 3’s mixture-of-transformers architecture explicitly stages reasoning first (a reasoning block “interprets what is happening in a scene”) before generation (an expert generation block “uses that context to create physically grounded outputs”). NVIDIA frames this as “think before acting” — a structural separation of perception/reasoning and generation within a single open model.
  • [2026-05-24] Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs: Adds a fourth substrate position — input-side visual scaffolding. Overlay equidistant numbered horizontal lines on the input image and prepend “Scan the image sequentially based on horizontal lines exists in the image.” No fine-tuning, no tools, no multi-pass inference. On GPT-4o: +25% on 2D visual search, +26.83% on 2D counting, +9.50% on spatial relations, −0.32 edit distance on scene description. Critically: pure-text step-by-step CoT prompting does not help and sometimes degrades; only the visual scaffold + sequential-scanning prompt jointly works. Anchors the lowest-cost corner of the substrate-axis (zero training, single query, negligible overhead) and sharpens the page’s anti-pure-text-CoT thesis with a clean ablation. Caveat: gains shrink on natural images and Qwen2.5-VL shows small natural-data regressions (-1.62% counting, -3.06% spatial).
  • [2026-05-23] Demystifying Video Reasoning: Shows that pretrained diffusion-based video models reason along the denoising trajectory (Chain-of-Steps, CoS), not across video frames (Chain-of-Frames as previously assumed). Decoding the predicted clean video at each diffusion step reveals multiple candidate solutions held simultaneously in early steps and progressive convergence to one final answer — the same parallel-exploration dynamic DiffThinker reported for fine-tuned image diffusion, but here observed natively in pretrained video DiTs without any reasoning fine-tuning. Also locates functional specialization across DiT depth (early=perception, middle=reasoning, late=consolidation) and demonstrates a training-free inference-time booster via random-seed latent ensembling. Tightly coupled to the A Very Big Video Reasoning Suite (VBVR) benchmark — analyses run on VBVR-Wan2.2.
  • [2026-05-22] Introducing Agentic Vision in Gemini 3 Flash: Google announces “Agentic Vision” in Gemini 3 Flash — a productized Think-Act-Observe loop with Python code execution as the universal action space (crop / rotate / annotate / plot). Reports a 5-10% aggregate vision-benchmark boost (no per-benchmark numbers) and one customer case (PlanCheckSolver, +5% on building-plan validation). Establishes the third substrate position on this page: external-tool code execution that appends transformed images back into context, complementing the autoregressive-tagged (Video-Thinker) and diffusion-native (DiffThinker) substrates. Notable detail: only the implicit-zoom behavior is currently trained-in; other behaviors (rotation, visual math, table parsing) require explicit prompt nudges and are roadmap items.
  • [2026-05-22] DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models: Reframes reasoning as image-to-image generation — fine-tuned Qwen-Image-Edit-2509/2511 emits a solution image (drawn maze path, filled Sudoku, TSP tour) and a symbolic parser extracts the answer. Beats GPT-5 by +314%, Gemini-3-Flash by +112%, same-data Qwen3-VL-32B-SFT by +39% averaged across 7 vision-centric tasks. Reveals the diffusion-substrate’s native parallel exploration (multiple candidates visible in early-step clean-latent). Negative result on DiffThinker-Video flags video-based reasoning as currently uneconomical.
  • [2026-05-22] Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning: Extends “Thinking with Images” to video by training Qwen2.5-VL-7B to interleave <grounding> (temporal localization) and <caption> (segment description) tags inside its CoT, with no external tools. Two-stage training (SFT → GRPO with correctness + format reward) sets a new 7B SOTA on Video-Holmes (+4.68), CG-Bench-Reasoning, and VRBench (+11.44). SFT-only underperforms the base; GRPO carries the result.

Open questions

Section titled “Open questions”
  • Which substrate wins on a shared benchmark? No filed paper compares (tagged autoregressive, diffusion-native, external-tool, input-scaffold) against each other on the same task. The four instances were all evaluated on different benchmarks (Video-Holmes / VRBench / Maze-Sudoku-TSP / vision-benchmark-aggregate / Binding-Problem-Generator), so the empirical comparison is open. The input-scaffold substrate (d) is the obvious cheap baseline that the other three should be compared against.
  • Should modality-CoT be a trained skill or a prompted behavior? Video-Thinker’s SFT-then-GRPO is necessary for that recipe to work; DiffThinker is pure flow-matching SFT on ~30k samples per task; Agentic Vision is partially trained (implicit zoom) and partially prompted (rotation, math); input-scaffolding is fully training-free. Demystifying Video Reasoning complicates this: the diffusion-native parallel-exploration substrate is present at pretrain time without any reasoning-specific training, so for substrate (b) the question becomes “what minimal fine-tuning routes the existing CoS dynamics at a target task?”.
  • Does the diffusion-substrate’s native parallel exploration (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2) carry over to non-image domains? Answered (video): Demystifying Video Reasoning confirms the same multi-candidate-then-prune behavior is observed in pretrained video DiTs (mazes, branching dominos, object rotations). Still open for audio / 3D / other diffusion modalities.
  • Where is the cost crossover between modality substrates? DiffThinker is fixed at 20 steps × 1.1 s; Agentic Vision is bounded by user/API-side iteration cap; Video-Thinker is bounded by token budget; input-scaffold is single-query ~free. A unified “iso-accuracy compute” comparison is missing. Demystifying Video Reasoning §5’s seed-ensembling offers a training-free extra axis for substrate (b) that hasn’t been integrated.
  • Should the pixel-space-generator-proposes / token-space-reasoner-verifies pattern (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2 Fig. 6; Introducing Agentic Vision in Gemini 3 Flash §“Image annotation”) become the default architecture for vision-centric agents? If so, what’s the right interface contract between the two halves (multi-candidate latent batches vs final-only image)?
  • For “thinking video models” specifically — DiffThinker-Video underperforms DiffThinker-Image at ~2× inference cost (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.4). Is this a current-scale artifact (Wan2.2-TI2V-5B vs Qwen-Image-Edit-20B) or structural to the video substrate? Demystifying Video Reasoning suggests the substrate itself is fine — the bottleneck is fine-tuning economics, not the underlying CoS mechanism.
  • Does the early/middle/late layer-wise specialization in DiTs (Demystifying Video Reasoning) suggest depth-targeted fine-tuning (middle layers only) as the right intervention for teaching new reasoning behaviors in substrate (b)? Untested.
  • Agentic Vision’s productization suggests vision benchmark scores from frontier labs will be code-execution-enabled by default. What’s the right open-benchmark protocol — score with and without code execution, only with, or relative-improvement-from-enabling?
  • Does input-side visual scaffolding (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs) generalize beyond horizontal lines? Numbered grids, attention heatmaps, segmentation overlays, depth-map overlays all sit in the same design space; only the horizontal-line variant has been tested at frontier-model scale. And why does it regress on natural images for Qwen2.5-VL but not GPT-4o — is the failure mode “smaller model gets confused by the overlay” or “the overlay competes with natural scene structure that GPT-4o can disentangle”?

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2024-11

Section titled “2024-11”

2024-07

Section titled “2024-07”

Unknown date

Section titled “Unknown date”