RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

TL;DR

RynnVLA-Latent attacks the scarce-action-label bottleneck for VLA foundation models by learning continuous, 3D-grounded latent actions from heterogeneous human + robot + sim video without native robot controls. RynnLAM, the latent-action model, combines residual future-feature prediction with camera-compensated 3D-flow supervision so the latents capture control-relevant motion rather than appearance or ego-camera drift. Trained on 30.1K hours of video (yielding 745.52M latent-action descriptors), the resulting 4B flow-matching policy reaches 98.8% on LIBERO, 88.6% on bimanual RoboTwin 2.0, and leads on six real-world tasks across dexterous-hand and mobile-gripper platforms before any embodiment-specific adaptation.

Key claims

• Latent actions learned by residual future-feature prediction plus camera-compensated 3D-flow supervision capture control-relevant motion rather than appearance change — the camera-motion decoupling is the load-bearing design choice that lets heterogeneous video sources (human + robot + sim) contribute compatible supervision [§Method].
• 30.1K hours of unlabeled interaction video can be converted into 745.52M latent-action descriptors usable as a shared pretraining substrate across embodiments [§Abstract, §Data].
• A flow-matching policy that predicts latent-action chunks from visual observations + language, then adapts the pretrained backbone + action expert to embodiment-specific native controls, reaches 98.8% success on LIBERO and 84.7% on LIBERO-Plus without benchmark-specific adaptation [§Results].
• Average success 88.6% on the RoboTwin 2.0 bimanual benchmark and 51.9% average across five VLABench tracks at 4B parameters [§Results].
• The same 4B model achieves the highest observed success rates among compared methods on six real-world manipulation tasks across both dexterous-hand and mobile-gripper platforms — the same checkpoint, no per-platform retrain [§Results].
• Representation ablations and data-scaling results jointly support the claim that latent-action supervision is a viable alternative to native action labels for scaling robot policy learning beyond the teleop corpus ceiling [§Ablations].

Method

RynnVLA-Latent has two training stages plus an embodiment adaptation step.

RynnLAM (latent action model). Trained on heterogeneous interaction video (human egocentric, robot teleop, simulation) with two coupled objectives: (a) residual future-feature prediction — predict the difference between current and future visual features rather than reconstructing pixels, so the latent is forced to encode what changed under the agent’s control; (b) camera-compensated 3D-flow supervision — estimate a global camera-motion component from scene geometry and supervise the latent only on the residual 3D flow of interacting entities (hands, tools, objects). The decoupling means ego-camera motion in the training video (common in head-mounted human capture) is subtracted before the latent-action target is formed, so human and robot sources contribute compatible gradients to the same latent space.

RynnVLA-Latent policy. A VLA that takes current visual observation + language goal and predicts a chunk of latent actions via flow matching. The policy is pretrained at scale on paired (observation, latent-action-chunk) tuples derived from the 30.1K-hour corpus — no native robot controls required at this stage.

Embodiment adaptation. For each target robot (LIBERO Franka, RoboTwin bimanual, dexterous hands, mobile grippers), the pretrained backbone + action expert are lightly adapted to the embodiment’s native control space via a small amount of embodiment-specific data. The latent-action channel acts as a frozen bridge; adaptation only re-fits the mapping from latent chunk → native controls.

Results

• LIBERO (Franka): 98.8% average success at 4B parameters.
• LIBERO-Plus (no benchmark-specific adaptation): 84.7% — a direct measurement of transfer without per-benchmark data.
• RoboTwin 2.0 (bimanual): 88.6% average success.
• VLABench: 51.9% average success across five tracks.
• Six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms: highest observed success rates vs the compared VLA baselines, from a single pretrained checkpoint.
• Data scaling: reported trajectory of success rate vs pretraining corpus size supports the claim that latent-action supervision continues to improve policy performance well into the 30.1K-hour regime (compare directly against Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models‘s million-hour but minimal-interface recipe at the next order of magnitude).
• Representation ablations: removing camera-motion decoupling, or substituting raw future-feature prediction for the residual target, degrades downstream policy performance — the 3D-grounded + residual + camera-compensated combination is the Pareto point.

Why it’s interesting

RynnVLA-Latent stakes out a 3D-grounded latent action position on the human-to-robot retargeting interface-location axis that sharpens World Action Learning via Interaction-Centric Spectral Latent Guidance (WING)‘s interaction-centric debiasing: both papers argue that the shared signal from heterogeneous video lives after camera motion is subtracted, but WING restricts the shared target to the lowest-K DCT coefficients of a 2D latent trajectory while RynnVLA-Latent keeps the full continuous latent and anchors it in 3D flow with camera compensation. The two together bracket the “soft spectral guidance” vs “full latent-action channel” alternative on the same interaction-centric debiasing principle.

Against the two scale-first filings — EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale (22-DoF joint retargeting + 20,854 h) and Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2 (wrist-only reduction + 1M+ h) — RynnVLA-Latent sits at a middle operating point: 30.1K hours (comparable to EgoScale, 30× below Dyna-2) but with a learned embodiment-agnostic interface rather than either joint-space retargeting or hand-engineered DoF reduction. The 4B scale and the “latent action descriptors as a reusable substrate” framing also connects to HuRo: Robotizing Human Videos for Scalable VLA Pretraining HuRo’s robotize-both-observations-and-actions recipe, but RynnVLA-Latent replaces explicit robotization with latent-space supervision — a candidate alternative to HuRo’s cross-source aggregation strategy.

See also

• Human-to-Robot Retargeting — RynnVLA-Latent adds a 3D-grounded latent-action channel position to the interface-location axis
• World Action Learning via Interaction-Centric Spectral Latent Guidance — WING, closest sibling on interaction-centric debiasing; RynnVLA-Latent keeps the full latent, WING keeps a low-K DCT slice
• EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data — competing “scale with full joint-space retargeting” recipe at comparable corpus size
• Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models — competing “minimal wrist-only interface + million-hour scale” recipe at the next order of magnitude of data
• HuRo: Robotizing Human Videos for Scalable VLA Pretraining — HuRo retargets both observations and actions explicitly; RynnVLA-Latent routes the gap through latent supervision instead
• LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition — LUCID retargets task intent as a soft channel; RynnVLA-Latent retargets a continuous latent-action channel at larger scale
• EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data — EgoWAM uses WAM supervision to absorb residual action mismatch — a complement rather than alternative to the latent-action channel approach
• VLA Models — RynnVLA-Latent is a 4B VLA foundation model
• World Foundation Models — the latent-action model is trained on heterogeneous interaction video with future-feature prediction objectives