Video2Robo: 3DGS-based Synthetic Data from One Video Enables Scalable Robot Learning

TL;DR

Video2Robo takes a single monocular human demonstration video and generates diverse robot training data from it, using 3D Gaussian Splatting (3DGS) as the scene representation. The pipeline reconstructs the demonstration scene as a 3DGS asset, tracks temporally consistent motion trajectories of task-relevant objects, extracts key task skills, and then re-renders the scene under augmented lighting, camera viewpoints, object swaps, and texture changes to produce diverse (image, action) training pairs — with kinematically plausible robot execution under novel object arrangements. Policies trained on Video2Robo data are reported to transfer to both simulation and real robots, avoiding the hardware and manual-labor cost of teleop-scale data collection.

Key claims

3D Gaussian Splatting is used explicitly as an editable scene representation, not just a rendering primitive — the framework performs high-fidelity rendering and explicit 3D scene editing (backgrounds, textures, lighting, camera views) on the reconstructed asset [Abstract].
One human demonstration video is sufficient input: task-relevant object motion trajectories are tracked temporally, key skills are identified, and the extracted skills drive a robot to execute the task “kinematically plausibly under novel object arrangements” [Abstract].
Data-diversity axes explicitly named are background augmentation, texture augmentation, lighting augmentation, and camera view augmentation — targeting the visual generalization gap rather than the dynamics gap [Abstract].
Policies trained on Video2Robo-generated data are evaluated in both simulation and real-world environments and report superior generalization and transfer performance vs baselines [Abstract].
Positioned as a mitigation for the “hard-to-access hardware and labor-intensive manual effort” cost structure of existing embodied-data generation approaches [Abstract].

Method

Given one monocular human demonstration video, Video2Robo (i) reconstructs the scene as a 3D Gaussian Splatting asset, (ii) tracks temporally consistent motion trajectories of task-relevant objects from the raw video, (iii) identifies key task skills from those trajectories, and (iv) applies the identified skills to a robot embodiment under novel object arrangements. The 3DGS representation is then used as an editable substrate: backgrounds, textures, lighting, and camera views are augmented to render additional training frames, and objects can be swapped, so the resulting synthetic dataset spans a much wider distribution than the single input video. The project webpage (linked from the CVPR page) hosts qualitative demonstrations.

Results

The abstract reports “extensive evaluations in both simulation and real-world environments” and that policies trained on Video2Robo data achieve superior generalization and transfer performance. Quantitative numbers are not surfaced in the openaccess HTML abstract; the full CVPR 2026 paper (pages 6695–6705) contains the tables and ablations.

Why it’s interesting

Video2Robo sits at an unusually direct intersection of three filed lines. First, it is a strong 3DGS-based analog of LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World, which isolated 3DGS photorealism as the load-bearing piece for sim-to-real VLA transfer (mesh-only rendering halved success 67% → 33%); Video2Robo takes the next step by making the 3DGS scene editable to synthesize the training corpus rather than serving as a static background. Second, it stakes out a “one-video digital twin + affordance-preserving cousins” position adjacent to SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation (which builds cousins from a reconstructed digital twin and reports +17/+21/+40% real-DROID success from object/scene/task variations), but starts from a single monocular human video rather than a reconstructed scene asset. Third, it is a distinct point on the interface-location debate the Human-to-Robot Retargeting concept page tracks: the interface here is not motion retargeting (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots), joint-pose bilevel physics (ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting), or scene-level intent (LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition) — it is scene-level 3DGS re-rendering with augmented viewpoints/lighting/objects, essentially producing a photorealistic sim from one human video and letting the robot practice in it.