EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras

ListenWhere Should Robots Look? Active Gaze Beats Wrist Cameras12:30

Show notes

EyeRobot 2.0 from Berkeley AI Research drops the wrist camera entirely and instead trains a robot to *actively gaze* with a stereo ego camera — then expresses every observation and action relative to where it's looking. Across seven real tasks and over a thousand rollouts, active gaze raises average success from 27% (passive stereo) to 67%, matching wrist cameras on clean tasks and beating them under tool occlusion. We dig into the two-stage RL gaze pipeline, the fixation-centric policy, and the ablation that shows the fixation frame is doing the real work. **Sources** - EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

EyeRobot 2.0 (Berkeley AI Research — Kerr/Hari/Kanazawa/Goldberg + Jitendra Malik + Karen Liu) replaces wrist cameras on a bimanual YAM robot with an active-gaze stereo ego camera controlled by a two-stage RL-trained policy, then expresses actions in a fixation-centric reference frame so the manipulation head sees a compact, canonicalized observation + action manifold. On seven real-world tasks and over 1,000 rollouts, the stereo-only “No Gaze” baseline hits 27% average success and an “Ego + Wrist” baseline hits 52%; EyeRobot 2.0 reaches 67% from the same source stereo images and surpasses wrist cameras on tool-occlusion tasks (pot-lid, boba straw). The direct successor to EyeRobot 1.0’s foveated mechanical eyeball + BC-RL loop, the recipe now runs on standard stereo hardware with no mechanical eye and no gaze demonstrations.

Key claims

Active gaze from a stereo ego camera matches wrist cameras on well-observable tasks and beats them when the gripper occludes the wrist view, raising average real-world success from 27% (no gaze) and 52% (ego + wrist) to 67% [§Comparisons, Experiments].
Gaze is learned in two stages with no gaze demonstrations: a per-task gaze controller trained with PPO against detected 3D object positions (effectively distilling slow foundation-model object detection into a fast real-time servoing policy), then a target-selector RL agent trained in a BC-RL co-training loop with the gripper policy (gaze is rewarded by the BC policy’s per-step prediction accuracy) [§Gaze training].
The RL gym for the gaze controller is synthesized from real-world stereo video by treating each eye as a partial 360° image: calibrated image geometry reproduces how the scene would look as the eyes rotate within the original cameras’ field of view, so a massively parallel gaze environment is bootstrapped without a simulator or extra data collection [§RL gym].
The gripper policy is an ACT-style transformer decoder in which all observations and actions are expressed relative to fixation — multi-resolution fixation-centric crops (foveated image tokens with high density at center) and end-effector proprioception + actions in a moving gaze reference frame — compacting the input image manifold and the action manifold simultaneously [§Policy architecture].
Expressing actions in the fixation frame is load-bearing, not cosmetic: a sim-ablation that keeps everything else but uses world-frame SE(3) actions/observations degrades to roughly the performance of the no-fixation policy, with the paper’s interpretation that without the fixation-frame canonicalization the model learns proprioceptive shortcuts without actually using the images [§Action-frame ablation].
Passive stereo at matched token count cannot substitute for foveated gaze: matching the fovea’s effective resolution with uniform tokens would require over 10× the token budget, and the marker task — the one requiring the most precise visual acuity — is near 0% success without fixation [§Comparisons].
Decoupling vision from action gives robustness to distractors that wrist cameras structurally lack: on three distractor-task evaluations under identical initial object positions, EyeRobot 2.0 reaches for the correct object when its gaze is directed at it and mostly misses grasps when it looks at the wrong place, while the wrist-camera baseline reaches for the largest/most distinct in-view object and cannot recover once committed [§Distractors].
A MuJoCo YAM twin was built explicitly for high-volume, low-noise ablations (not for sim-to-real transfer) with VR teleop on the same GELLO leader arms and SAM-3D-imported assets; identical method + hyperparameters + camera/robot config transfer in to sim, and the full simulation setup (environments, teleop, training, eval) will be open-sourced [§Simulation twin].
Dominant failure mode is narrowly missing a stage and entering an unrecoverable state — common for low-data policies without DAgger data — and gaze provides minimal lift when the most precise phase is already near the stereo camera (boba straw insertion ablation unaffected by foveation) [§Failures].

Method

The system has three trained components glued by RL. (1) A gaze controller continuously servos the two eyes to fixate on a specified 3D target, taking foveated input views + eye proprioception + a goal and emitting eye motion. It is trained with PPO using a dense reward derived from detected 3D object positions — the paper frames this as distilling slow foundation-model object detection into a fast real-time servoing policy. (2) A target selector is a tiny RL agent that, at each stage of the task, chooses which object to look at and hands that goal to the gaze controller. (3) A gripper policy is an ACT-style transformer decoder whose entire input/output representation sits in the moving gaze frame: images arrive as fixation-centric multi-resolution crops (center-high foveation), and end-effector proprioception + predicted action chunks are expressed relative to the current fixation pose.

The target selector and gripper policy co-train in a BC-RL loop inherited from EyeRobot 1.0: the gripper is supervised by behavior-cloning on teleop demonstrations, while the target selector’s reward is the gripper policy’s per-step BC prediction accuracy — so gaze choices that make actions easier to predict get reinforced. To train the gaze controller with RL at scale, the system treats each eye in real stereo video as a partial 360° panorama and synthesizes gaze trajectories by looking around within the cameras’ native field of view, using calibrated image geometry to reproduce what the scene would look like as the eyes rotate — reusing the EyeRobot 1.0 panoramic-RL-gym idea but with high-resolution stereo instead of 360° imagery.

Evaluation runs over 1,000 real-world rollouts on seven tasks, 25 trials per policy-task cell with the same initial object positions across policies for comparability, plus a MuJoCo YAM twin with six tasks for high-volume controlled ablations under matched hyperparameters/camera/robot config.

Results

Headline real-world average over seven tasks: No-Gaze stereo 27%, Ego + Wrist 52%, EyeRobot 2.0 67% [§Comparisons].
On tool-occlusion / grasped-object-blocks-wrist tasks (boba straw, pot lid), the wrist-camera baseline collapses to roughly the passive-stereo level because the wrist cannot see the manipulation point; EyeRobot 2.0 “widens the gap” (specific per-task numbers in the project page’s task breakdown) [§Comparisons].
On the subset of tasks where wrist cameras do have an unobstructed view (marker, tea, toaster, wrench), EyeRobot 2.0 matches or exceeds Ego + Wrist despite consuming only the stereo stream [§Comparisons].
Marker task — the acuity-bottlenecked task — shows near-0% success for passive stereo at matched token budget; a uniform-resolution replica of the fovea would need >10× the tokens [§Comparisons].
Three distractor-condition evaluations under identical initial object positions: EyeRobot 2.0 has higher success and higher stage-progression rates than the wrist-camera baseline, which commits to reaching for the largest/most distinct object and cannot recover [§Distractors].
Sim ablation isolating action-frame choice: swapping fixation-relative actions for world-frame SE(3) collapses performance roughly to the no-fixation policy, matching the sim ablation that removes stereo or foveation entirely — the three ablations degrade along the same curve, consistent with the paper’s “shortcut” interpretation [§Action-frame ablation].

Why it’s interesting

EyeRobot 2.0 is the sharpest filed answer so far to the open “where should robots look” question that sits upstream of every VLA recipe tracked in VLA Models — it treats gaze as a first-class action channel with no gaze demonstrations, which puts it on the same axis as LIME: Learning Intent-aware Camera Motion from Egocentric Video LIME (language-conditioned SE(3) viewpoint-target head decoupled from manipulation) and ActiveMimic: Egocentric Video Pretraining with Active Perception ActiveMimic (joint pretraining of viewpoint + wrist actions from body-worn RGB) but with real bimanual-manipulation + wrist-camera head-to-heads on seven tasks and 1000+ rollouts, rather than LIME’s single-step prior-to-VLA or ActiveMimic’s egocentric-pretraining framing. The result that foveated gaze matches wrist cameras at ~1/10 the uniform-token budget — and beats them under tool occlusion — gives the Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level sensorimotor counter-position a direct architectural lever (“decouple vision from action”) rather than a modality-addition lever (tactile, force). It also parallels the within-VLM foveation work of Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models Foveated Reasoning (foveation policies learned without human gaze annotation) and Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing AutoGaze (pre-ViT patch selection) by taking the same “concentrate compute where it matters” bet to the physical-camera layer.