FoveaVLA: Foveation Actions Replace the Wrist Camera in VLAs

ListenLearning Where to Look: FoveaVLA Kills the Wrist Camera9:45

Show notes

FoveaVLA replaces the wrist camera on a π0.5-based VLA with a *learned foveation action* — the policy predicts a bounding box per external view, and the next observation is assembled from a resized full frame plus a full-resolution crop of that region. On four real-world DROID task suites it hits 70.8% (single external view) and 77.1% (dual), versus 52.1% for the wrist-equipped baseline. We dig into the mechanism, why reusing PaliGemma's location tokens means zero added latency, and what "decouple vision from action" might mean for visual generative models. **Sources** - FoveaVLA: Foveation Actions Replace the Wrist Camera in VLAs — Bingao Chen, Lihan Zha, C. Karen Liu, Guy Tevet

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

FoveaVLA (Stanford TML — Bingao Chen, Lihan Zha, C. Karen Liu, Guy Tevet) replaces the wrist camera on a π0.5-based VLA with a learned foveation action: alongside every action chunk, the policy predicts a 2D interaction-region bounding box per external view that it will need to see at the next policy query, and the next observation is assembled from the resized full view plus a crop of the predicted region taken at the original image resolution. The box is decoded from PaliGemma’s existing location-token vocabulary off the cached prefix while the chunk executes, so no latency is added at the policy-query boundary. On four real-world DROID-setup task suites, FoveaVLA-Single (one external RGB-D view, no wrist) reaches 70.8% average success and FoveaVLA-Dual (two external views) reaches 77.1%, versus 52.1% for the wrist-equipped π0.5 baseline — matching the wrist policy on plain pick-and-place and improving on distractor-heavy, cloth, and far-away-object settings.

Key claims

Replacing the wrist camera with a learned foveation action beats the wrist-equipped π0.5 baseline on average: 70.8% (Single, one external RGB-D) and 77.1% (Dual, two external) vs 52.1%, across four real-world DROID task suites [Abstract, §Results].
Single and Dual configurations match the wrist-equipped π0.5 baseline on plain pick-and-place and improve on distractor-heavy, cloth-manipulation, and far-away-object tasks [Abstract, §Results].
Interaction-region supervision reuses existing RoboInter annotations on DROID — the smallest rectangle enclosing the components each annotated skill requires (gripper, manipulated object, target region) — rather than requiring new labels [§Method].
The prediction target is the interaction region at the next policy query, not the current one, because the gripper and objects move while the chunk executes [§Method].
Foveation adds no wait at the chunk boundary: box tokens decode from the cached prefix during chunk execution, so the crop is ready when the next frame arrives [§Method].
Vision budget is a sharp lever here — the ViT consumes a 224×224 input as a 16×16 patch grid, so a resized 1280×720 full frame covers an interaction region with ~15 of 256 patch tokens vs ~256 of 256 for a same-sized crop from the original image [§Method].

Method

FoveaVLA is built on π0.5 in two configurations — Single (one external stereo camera) and Dual (two external stereo cameras) — with full and foveal RGB and depth as inputs, and no wrist camera. From one shared observation the model produces two outputs: a flow-matched action chunk from the π0.5 action expert, and a foveation action consisting of one 2D bounding box per external view, decoded as PaliGemma location tokens. The next observation is then assembled as the resized full external view(s) plus a crop of the predicted region taken at the original camera resolution, together with aligned depth crops.

Supervision for the foveation action comes from RoboInter interaction-region annotations on DROID: for each annotated skill, the smallest rectangle enclosing the components it requires (gripper, manipulated object, target region). Because the target is the interaction region at the next policy query rather than the current one, the model is trained to anticipate where the relevant visual detail will be after the current chunk executes. Box tokens are decoded from the cached PaliGemma prefix in parallel with chunk execution, so there is no added wait at the chunk boundary.

Both configurations are post-trained on DROID with RoboInter interaction annotations on top of the π0.5 initialization.

Results

FoveaVLA-Single: 70.8% average success across four real-world DROID task suites with one external RGB-D view and no wrist camera [Abstract].
FoveaVLA-Dual: 77.1% average success with two external stereo cameras [Abstract].
Wrist-equipped π0.5 baseline on the same four suites: 52.1% [Abstract].
Both variants match the wrist-equipped baseline on plain pick-and-place and improve on distractor-heavy, cloth-manipulation, and far-away-object task suites [Abstract].
Per-suite ablation reported on the project page (with one external camera and no wrist view) shows removing either the foveal crop or depth substantially lowers success, with the full FoveaVLA-Single at 70.8% [§Results].
Attention-share analysis on a Move episode: baseline π0.5 attends almost exclusively to the wrist view (and misses targets outside it), whereas FoveaVLA keeps a steady share on the full external views for workspace coverage and draws close-up detail from the foveal crops [§Results].

Why it’s interesting

FoveaVLA lands on the “where should robots look” axis that EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras EyeRobot 2.0 opened — same Stanford Karen Liu involvement on both — but takes a sharply different structural bet: EyeRobot 2.0 adds a physical active-gaze stereo ego camera driven by a two-stage RL pipeline with a fixation-centric action frame, while FoveaVLA keeps the cameras fixed and makes the crop coordinates themselves the predicted action, reusing π0.5’s existing PaliGemma location-token head rather than training a new gaze policy. The 70.8% / 77.1% vs 52.1% split against wrist-equipped π0.5 on four suites puts a second large-sample datapoint on the board for “decouple vision from action” as a lever (per the Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level sensorimotor counter-position) alongside EyeRobot 2.0’s 67% vs 52% on seven tasks — two independent recipes converging on the “wrist cameras are not the right primitive for VLA perception” conclusion. For the VLA Models recipe-lever board, this adds a reuse-the-VLM-location-tokens foveation cell distinct from EyeRobot 2.0’s RL + fixation-centric-action-frame cell and from the within-VLM foveation of Foveated Reasoning and AutoGaze.