World Action Learning via Interaction-Centric Spectral Latent Guidance

TL;DR

WING is a human-to-robot retargeting recipe that transfers knowledge from egocentric human videos to robot policies through a two-stage design: (1) WING-LAM learns interaction-centric latent actions by explicitly separating ego-camera motion from hand-object interaction motion using point tracks + hand-object masks and distilling the interaction branch into an RGB-only student encoder; (2) at inference, the latent-action sequence is projected via DCT, the lowest-K frequency coefficients are retained as a predicted guidance target, and the robot action generator attends to those guidance tokens through gated residual updates. Reports 99.20% on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa-GR1 average success, plus strong real-world generalization on 4 tabletop tasks. Load-bearing claim: cross-embodiment task semantics live in slowly varying temporal structure, not in the full latent trajectory.

Key claims

Latent actions trained via next-frame reconstruction on egocentric video are dominated by ego-camera motion unless the two motion sources are explicitly decoupled; WING-LAM uses point tracks + hand-object region masks to estimate a global camera warp and extracts the residual as the interaction signal, then distills interaction-only latents into an RGB-only student [§Method, Motivation 1].
Human and robot behavior at the same task share low-frequency temporal structure but diverge in high-frequency content; retaining only the lowest-K DCT coefficients of the latent-action sequence yields a transfer target more consistent across embodiments than the full trajectory [§Method, Motivation 2].
K=4 DCT coefficients beats K=2 (loses dynamics) and K=8 (reintroduces high-frequency noise) across all four LIBERO suites [§Question 03 ablation].
Human latent actions are most useful as guidance for action generation rather than as direct action targets or pure video-prediction auxiliary signal — predicting spectral guidance beats video-only pretraining and action-target pretraining at matched 200-h egocentric budget [§Question 04].
At inference the spectral target is predicted from current image + language + robot state by a lightweight predictor (no future-frame access required), then projected into guidance tokens the action generator attends to via gated residual updates [§Method, step 3].
Average success: LIBERO 99.20%, RoboTwin 2.0 93.80%, RoboCasa-GR1 57.7%; dual-arm standard 75.0% [§Results].
Under generalization, WING is competitive with π0.5 on average success and slightly higher on average progress — i.e. gains concentrate on partial-execution credit as much as full completion [§Results].
Removing interaction-centric debiasing degrades performance in most settings, confirming the camera/interaction decoupling is not just a cosmetic preprocessing step [§Question 04 ablation].

Method

WING is built in two training stages plus an inference-time guidance mechanism.

WING-LAM (teacher-student latent action model). A teacher takes a frame pair plus point tracks + hand-object masks and splits motion into two branches: a camera branch that predicts a global warp from background tracks, and an interaction branch that predicts residual displacements on hand-object regions. The two decoded motions jointly reconstruct observed trajectories. An RGB-only student is then distilled to produce the interaction latent from frame pairs alone, with camera-like augmentation warps forcing viewpoint invariance of the interaction target. The result: an RGB-only encoder that outputs interaction-centric latent actions.

Spectral guidance training. The latent-action sequence over a window of future frame pairs is transformed by a 1D DCT along the time axis; the lowest-K coefficients form the training target for a predictor that sees only current image + language + robot state. The predictor outputs a frequency-domain guidance vector.

Action generation. The predicted frequency coefficients are projected into guidance tokens. The robot action generator (producing executable action chunks) attends to these tokens via gated residual updates, so low-frequency human-interaction structure guides generation while high-frequency fine-grained control remains free to match the robot’s own embodiment and dynamics.

Results

LIBERO average: 99.20% across the four suites.
RoboTwin 2.0 average: 93.80%.
RoboCasa-GR1 average: 57.7%.
Dual-arm bimanual (standard setting): 75.0% success; under generalization competitive with π0.5 on success, slightly higher on average progress score.
Four-suite average: +0.70 pp over strongest reported baseline per the project-page bar chart; 24-task tabletop average: +2.30 pp over strongest shown baseline.
DCT ablation (K coefficients retained): K=4 is Pareto-best; K=2 loses too much dynamic content, K=8 reintroduces noise. Four LIBERO suites consistent.
Pretraining-strategy ablation (200 h egocentric budget): spectral guidance > video-only pretraining > direct action-target pretraining; removing interaction-centric debiasing is net-negative in most settings.
Action classification (frozen WING-LAM features): highest average accuracy among compared representations, with reduced sensitivity to camera perturbations.
Shared low-frequency latent components separate matched human-robot interaction pairs more cleanly than full trajectories or high-frequency components in the paper’s correspondence analysis.

Why it’s interesting

WING stakes out a frequency-domain position on the human-to-robot retargeting interface-location axis that no prior filed paper occupies. The dominant recipes on Human-to-Robot Retargeting either reduce the action channel (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots 3-DoF wrist), reconstruct 4D kinematics (Do as I Do: Dexterous Manipulation Data from Everyday Human Videos), swap target hardware (Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1), Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training), or absorb mismatch via WAM supervision (EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data, Modality-Autoregressive World-Action Models). WING instead argues the shared signal is a spectral slice of the latent trajectory, with the robot free to generate high-frequency control appropriate to its own embodiment. The interaction-centric debiasing step is a direct counter to EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data / ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining‘s “pay for noisy latent actions with scale” position — WING argues the noise is structured (ego-camera motion) and can be removed cheaply with point tracks + masks before scaling matters.

The DCT guidance mechanism is also a sibling to LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition LUCID’s task-intent recipe and WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time WAM-TTT’s KV-memory recipe: all three keep the retargeting interface soft (an auxiliary conditioning channel) rather than hard (a reduced action space), but WING specifies the interface in a transform domain rather than a 3D-scene domain or an opaque KV-cache.