LIME: Learning Intent-aware Camera Motion from Egocentric Video

TL;DR

LIME (ETH Zürich, Sun et al.) formulates language-conditioned camera-motion generation — given a current RGB view and a free-form intent, predict a relative SE(3) target pose for the next observation — as a first-class robot action, distinct from vision-language navigation (base motion) and VLA (manipulation actions). Supervision is mined at ~3M examples from passive egocentric video (RoomTour3D + Nymeria): temporally separated frame pairs supply the relative pose, and a hindsight VLM labeller adds plausible intents plus observation-gain descriptions. The model factorizes into an auto-regressive observation-gain text head (what the next view should reveal) and a continuous flow-matching pose head that models multi-hypothesis SE(3) targets over the fused vision-language representation. Reported downstream lift: a single LIME viewpoint step before an unchanged VLA policy raises LIBERO-Goal success from 26% → 74% across 10 tasks under randomized initial-pose perturbations, and the same interface drives multi-round scan/inspect behaviours, embodied QA on AVS-ProcTHOR, and real-robot deployment.

Key claims

Language-conditioned camera motion is under-explored as a first-class robot action, sitting between VLN (translates instructions to base motion) and VLA (translates instructions to manipulation actions) [Abstract].
Multi-intention camera-motion supervision can be mined from passive egocentric video without teleop demonstrations by pairing temporally separated frame pairs (which supply the relative SE(3) transform) with hindsight-VLM-generated intents and observation-gain descriptions, yielding ~3M intent-conditioned examples from RoomTour3D and Nymeria [§Method-b, project page].
Factorizing the output into (a) an auto-regressive observation-gain text head — what the next view should reveal — and (b) a continuous flow-matching SE(3) pose head over the fused vision-language representation lets a single model both explain why a viewpoint change is chosen and represent the multi-hypothesis set of poses that satisfy the same image-intent pair [§Method-a, Abstract].
A single LIME viewpoint step before an unchanged manipulation policy raises LIBERO-Goal overall success from 26% → 74% across all 10 tasks under randomized initial robot pose, isolating the lift to viewpoint choice rather than downstream policy changes [Project page, LIBERO-Goal panel].
The same interface supports multi-round iterative viewpoint policies (navigating toward a distant target, scanning an object from different aspects) and embodied question answering, where LIME predicts the view a question needs and a downstream VLM answers from the resulting frame [Project page, Multi-round + EQA panels].

Method

LIME implements language-conditioned camera-motion generation as two coupled interfaces over a shared vision-language representation. The language interface autoregressively generates an observation-gain description — a natural-language summary of what the next view should reveal beyond the current one — keeping part of the task in the VLM’s native output space so it can be supervised with standard next-token loss. The pose interface is a continuous flow-matching head that models the conditional distribution of relative SE(3) target poses given the fused representation, capturing the multiple valid viewpoints that satisfy the same image-intent pair rather than collapsing to a single regressed pose.

Supervision comes from mining passive egocentric corpora. For each pair of temporally separated frames in RoomTour3D (room-scale walkthroughs) and Nymeria (body-scale egocentric interactions), the relative camera transform between them is treated as a ground-truth SE(3) target. A hindsight VLM labeller then annotates the pair with a plausible intent that would have produced the transform, plus an observation-gain description of what the second frame reveals over the first. The pipeline yields roughly 3M intent-conditioned training examples. At inference, given an RGB image and a free-form language intent, LIME jointly samples the observation-gain description and the SE(3) pose target.

Results

LIBERO-Goal downstream lift. Under randomized initial robot pose, running the same VLA policy directly gives 26% overall success on LIBERO-Goal; prepending a single LIME viewpoint step to reposition the camera before the VLA acts raises this to 74% across all 10 tasks [Project page, LIBERO-Goal panel].
Multi-round and embodied QA. LIME runs iteratively for long-horizon behaviours (navigating toward and inspecting a distant target, multi-aspect scanning) and drives embodied question answering on AVS-ProcTHOR by converting each question into a view-selection intent, moving the camera, and letting a downstream VLM answer from the resulting frame [Project page, Multi-round + EQA panels].
Real-robot deployment. LIME is deployed on a real robot both before manipulation (repositioning the camera to acquire the visual evidence needed to act, with manipulation performed by an external policy, VidBot) and after manipulation (verifying the result), across pick-and-place and inspection intents [Project page, Real-world examples panel].
Per-task success breakdowns and quantitative numbers on multi-round tasks, EQA, and real-robot rollouts are shown on the project page but not extracted into a table in the arxiv abstract.

Why it’s interesting

LIME is the sharpest filed instance so far of camera motion as a first-class language-conditioned action separate from the VLA action space, and it complements two adjacent lines already tracked on the wiki. It shares ActiveMimic’s (ActiveMimic: Egocentric Video Pretraining with Active Perception) core thesis that egocentric human video’s continuous viewpoint repositioning is the missing supervision signal for robot policies, but factorizes the recipe differently: ActiveMimic learns viewpoint action jointly with manipulation as a VLA pretraining objective, whereas LIME cleanly decouples viewpoint policy from manipulation policy and shows the composition works as a plug-in on top of a frozen VLA — 26% → 74% on LIBERO-Goal is a much sharper isolation of the viewpoint-choice lever than ActiveMimic’s joint-pretraining comparison. Second, the observation-gain-plus-flow-matching-pose head is an unusual factorization for the Pose Estimation and Motion Capture cluster: rather than probabilistic monocular pose lifting (FMPose3D: monocular 3D pose estimation via flow matching), it predicts a target SE(3) pose as an action, and the multi-hypothesis property is load-bearing because “look under the shelf” admits many valid poses. Third, the ~3M-example hindsight-VLM-labeller pipeline is another datapoint for VLM-labelled synthetic supervision over real video (Synthetic Training Data) — same recipe family as Macrodata’s WGO-Bench robot-video labeller (Segmenting Robot Video into Actionable Subtasks (WGO-Bench)), applied to intent + observation-gain rather than subtask boundaries.