Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models

ListenLook Where You Think: Foveated Reasoning in VLMs9:55

Show notes

FoveateR — Foveated Reasoner — is an autoregressive vision-language model that folds coarse-to-fine visual focusing into a single stateful decoding pass: it starts from a low-res view and only pulls high-res evidence when its own reasoning trace fires a non-linguistic foveation action. We walk through the mechanism, the two-stage training (pseudo-labeled SFT cold-start plus RL with an anti-"see-everything" penalty), and why internalizing the zoom beats multi-pass and text-grounded baselines under tight token budgets. We also connect it to agentic vision loops and to VLM perception failures the team has been tracking. **Sources** - Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models (Juhong Min, Pulkit Kumar, and 3 others)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

FoveateR is an autoregressive VLM that unifies coarse-to-fine visual focusing with textual reasoning inside a single stateful decoding pass — the model starts from a low-resolution view, and only when its ongoing reasoning trace triggers a foveation action does it retrieve high-resolution evidence from a selected region and inject those tokens back into the same trajectory. Trained in two stages (SFT cold-start to bootstrap the foveation behavior, then RL that jointly rewards evidence acquisition and task accuracy while penalizing trivial “see-everything” solutions), it achieves stronger accuracy under tight visual-token budgets than both multi-pass (crop-then-rerun) and text-grounded (emit bboxes as output tokens) prior work. The framing is human-analogous: rather than paying the full high-resolution token cost up front, the model acts to acquire evidence when its reasoning needs it, and never breaks its decoding state to do so.

Key claims

Prior visual-focusing methods fall into two families and both leak the reasoning state: (a) multi-pass designs re-encode a crop and re-run the VLM from scratch, breaking the ongoing decoding trajectory, and (b) text-grounded designs emit coordinates/tool-call arguments as output tokens alongside reasoning tokens, consuming sequence budget and quantizing the action space via the text vocabulary [§2].
Human foveation is effective specifically because it is (i) coarse-to-fine, (ii) stateful (evidence acquired mid-thought stays in scope), and (iii) driven by non-linguistic actions inside a single ongoing process — the paper argues no prior method instantiates all three simultaneously [§2].
FoveateR is defined as a stateful, single-pass, non-linguistic action-based visual focusing mechanism: foveation is emitted as an action (a spatial region selector distinct from the text vocabulary) rather than as a text-formatted bbox, and the retrieved high-resolution tokens are appended in-line into the current decoding trajectory rather than triggering a fresh pass [§3].
Two-stage training: (1) SFT cold-start supervises when to foveate and where; (2) RL post-training jointly optimizes evidence acquisition and task accuracy, with an explicit penalty against the degenerate solution of always foveating everywhere (“see-everything”) that would collapse the accuracy–efficiency benefit [Abstract, §3].
The method learns effective foveation policies without human-annotated foveation trajectories — the cold-start supervision uses pseudo labels rather than expensive human gaze annotations [§Appendix, per D.2 discussion in related-work].
Reported result (per web-search preview of abstract/experiments): FoveateR achieves stronger accuracy under tight visual-token budgets across multiple vision-language benchmarks than the multi-pass and text-grounded baselines [Abstract, §Experiments].
The pseudo-label training signal encourages localized, multi-fixation evidence acquisition rather than a single large reveal — the model performs multiple targeted foveations, analogous to human progressive fixation, and this behavior emerges more reliably from pseudo labels than from human-annotated single-region labels [§Appendix].

Method

FoveateR operates on a low-resolution view of the input image by default. During generation, the model can emit either a text token (continuing reasoning normally) or a foveation action — a non-linguistic spatial selector that specifies a region to acquire at high resolution. When a foveation action is emitted, the corresponding high-resolution patch tokens are extracted and inserted in-line into the decoding trajectory; the model then continues generating from the same stateful position, now with the new evidence in context. This differs from multi-pass approaches (which would re-encode and re-run the whole prompt) and from text-grounded approaches (which emit <bbox>x1,y1,x2,y2</bbox>-style tokens that both quantize the action space and consume text-sequence budget).

Training is two-stage. Cold-start SFT teaches the model when and where to foveate using pseudo labels rather than human gaze annotations — pseudo supervision leads to more localized, multi-fixation behavior than human single-region labels. RL post-training then optimizes a joint reward over (a) evidence-acquisition quality and (b) downstream task accuracy, with an explicit anti-”see-everything” term that penalizes the degenerate policy of foveating on the entire image (which would trivially maximize task accuracy at the cost of the token budget the method is designed to save).

At inference, the accuracy–efficiency trade-off is set by the visual-token budget: FoveateR keeps its base tokens at coarse resolution and only spends the high-resolution budget on regions its reasoning trace decided were relevant, giving it a favorable curve against methods that spend the full budget uniformly.

Results

Headline claims (from paper abstract and web-search of published sections; the arxiv HTML was unavailable at filing time for verbatim numbers):

Outperforms recent visual focusing methods across multiple vision-language benchmarks under tight visual-token budgets [§Experiments].
Learns effective foveation policies without human-annotated foveation trajectories — pseudo-labeled cold-start plus RL is sufficient [§Appendix].
Produces more localized, multi-fixation behavior than baselines trained on human-annotated single-region labels; the fixation count is higher (T^{hmn}_{fov} ≈ 1.8 vs. 1.0 for the human-label control per the appendix analysis) [§Appendix B.2].

Specific benchmark numbers require the arxiv PDF; refer to the paper directly at the canonical source.

Why it’s interesting

Foveated Reasoning sits in the crossfire of three concept clusters the wiki has been tracking. First, on thinking with modalities (Thinking with Modalities), it sharpens substrate (c) — the external-tool Think-Act-Observe loop that Introducing Agentic Vision in Gemini 3 Flash productizes as Gemini 3 Agentic Vision — by internalizing the tool. Gemini 3 Flash writes Python that crops the image and appends the transformed image back to context (external, iterated, unbounded). FoveateR emits a non-linguistic action that appends high-res tokens back into the same decoding trajectory (internal, single-pass, stateful). Both attack the same problem — VLMs need to spend visual tokens where the reasoning decides they matter — but the internalized version avoids the multi-pass overhead and the text-token budget cost that Gemini’s external form pays.

Second, on VLM perception failures (VLM Perception Failures), the page already documents that VLMs answer vision-centric questions from the LLM prior rather than from the image (Vision Language Models are Biased, Hidden in plain sight: VLMs overlook their visual representations), and that visual scaffolding on the input side can partially recover perception (Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs). FoveateR is the active counterpart: instead of statically scaffolding the input for every query, the model itself decides which regions need pixel-precise evidence and acquires them on demand. It doesn’t fix the “answer from prior” bias per se — but it makes the pixel-precise evidence available exactly when the reasoning trace needs it, which is where prior work has argued the failure is most costly.

Third, contrasts with Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing (AutoGaze) are instructive. AutoGaze also picks a sparse patch subset before the ViT and trains it with NTP+GRPO — but it does so once, up front, per input, driven by a reconstruction-loss objective independent of the downstream task. FoveateR interleaves the decision with the reasoning trace, so foveation is caused by what the model is currently trying to compute rather than by a task-agnostic reconstruction target. The two are stackable in principle: AutoGaze prunes the up-front coarse view; FoveateR then dynamically re-injects high-res patches when reasoning demands them.