ActiveMimic: Egocentric Video Pretraining with Active Perception

TL;DR

ActiveMimic (Fudan, Yu-Gang Jiang group) argues that egocentric human video has consistently underperformed robot data for VLA pretraining because the active perception signal — the way humans continuously reposition their viewpoint during manipulation — is being thrown away as nuisance camera motion by standard pipelines. The method recovers synchronized camera and wrist trajectories from a single body-worn RGB stream, treats camera motion as a first-class viewpoint action, and jointly learns active perception and manipulation from in-the-wild egocentric video before adapting to a target robot. Real-world experiments report that ActiveMimic consistently beats human-video-pretrained baselines and matches state-of-the-art models pretrained on robot data on tasks with diverse active-perception demands, and provides ablation evidence that the active-perception capability originates from the egocentric pretraining stage rather than from robot fine-tuning.

Key claims

The gap between egocentric-human-video pretraining and robot-data pretraining is attributed to a missing signal: continuous viewpoint repositioning during manipulation, which standard pipelines treat as noise [Abstract].
Synchronized camera and wrist trajectories can be recovered from a single body-worn RGB camera, removing the need for paired sensor rigs (IMU, depth, second camera) on the human-data side [Abstract].
Camera motion is modeled as a viewpoint action and jointly learned with manipulation actions during pretraining, then adapted to the target robot [Abstract].
Across real-world tasks with diverse active-perception demands, ActiveMimic consistently surpasses baselines pretrained on human video [Abstract].
ActiveMimic matches state-of-the-art models pretrained on robot data on the same real-world task suite [Abstract].
Ablation evidence is provided that the active-perception capability originates from egocentric pretraining rather than from robot-specific fine-tuning — i.e. the inductive bias is upstream [Abstract].
Camera-motion supervision facilitates representational transfer from human perception to robot control [Slack note quoting the abstract].

Method

The high-level recipe is a three-stage pipeline. First, a perception module recovers, from a single body-worn RGB video, the synchronized 6-DoF camera trajectory and the wrist trajectory of the demonstrator — both treated as actions rather than observations. Second, an active-perception-aware policy is pretrained on in-the-wild egocentric human video to jointly predict the viewpoint action (where to look next) and the manipulation action (where to move the wrist). The viewpoint action plays the same role for the head/camera as the wrist action plays for the gripper. Third, the pretrained policy is adapted to a target robot, where the viewpoint action becomes either a head-camera command (if available) or an auxiliary representational supervision signal during fine-tuning.

The contribution is structural: prior egocentric-video-for-robot-policy work either ignored camera motion, suppressed it via image stabilization, or treated it as a confounder to be regressed out. ActiveMimic flips the sign — camera motion is the active-perception decision the demonstrator is making, and supervising on it is what unlocks transfer to robot control.

Results

ActiveMimic consistently surpasses baselines pretrained on human video across real-world tasks with diverse active-perception demands [Abstract].
ActiveMimic matches state-of-the-art models pretrained on robot data on the same task suite — closing the long-standing human-video-vs-robot-data gap that motivates the paper [Abstract].
The active-perception capability originates from egocentric pretraining and persists through robot adaptation, per the paper’s “further analysis” [Abstract].
Per-task success rates, task list, baseline identities, and quantitative deltas were not retrievable from the abstract alone at filing time; the project page (https://activemimic.github.io/) is up but the BibTeX is the only substantive content extracted and code is not yet released (Slack note: “Not released yet but Jiang lab has a reasonable track record to open source projects”).

Why it’s interesting

ActiveMimic is the cleanest filed argument that the data axis (EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World, HumanNet: Scaling Human-centric Video Learning to One Million Hours) is necessary but not sufficient for closing the human-video-to-robot-policy gap — there is a signal-design axis (what part of the egocentric stream you treat as the action) that all three concurrent egocentric corpora largely sidestepped. EgoVerse argues for cross-lab protocol control and dense Aria-derived annotations; HumanNet argues for ~700× more hours of internet video; ActiveMimic argues that the missing ingredient is treating camera motion as a learned action, and reports matching robot-data baselines as a result — a sharper outcome than either dataset paper’s qualitative “more data helps” finding. It also contrasts with the Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners thesis that causal video prediction is the right pretraining objective for robot policies: Rhoda’s DVA treats the entire next frame as the prediction target, while ActiveMimic factorizes the prediction into a viewpoint action and a wrist action and supervises them jointly. The sensorimotor framing complements Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level‘s standing critique of VLAs (“don’t skip the sensorimotor level”) by promoting one specific sensorimotor signal — head/camera motion — into a first-class action channel.