Skip to content
Bud's Wiki

Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation

TL;DR

Section titled “TL;DR”

WANDA (CMU; Guo, Li, Shi) is a synthetic data engine that turns a single RGB-D human demonstration into thousands of training trajectories for open-world mobile manipulation. From the source demo it reconstructs the scene as background 3D Gaussian splats plus object meshes plus 6D interaction trajectories, then rearranges the contact-rich interaction segments into new spatial configurations, chains them with whole-body motion planning, and re-renders photorealistic observations by compositing rendered robot/object meshes onto the splat background. Corrective State Expansion adds off-demonstration recovery states, and cross-environment scenes come from Marble-generated 3D worlds built from single everyday photos. A π0.5 fine-tuned only on WANDA data outperforms one trained on 50 teleop demos and supports zero-shot cross-embodiment deployment on a different mobile manipulator (Agibot G1 → Linearbot) — the human input stays fixed at one demonstration; the dataset scales with compute.

Key claims

Section titled “Key claims”
  • One demonstration is sufficient input for open-world mobile manipulation training: WANDA reconstructs background Gaussian splats and robot-object interaction trajectories from source RGB-D observations and uses them as a world substrate for planning and rendering [§Abstract].
  • Contact-rich interaction segments are rearranged into extensive spatial configurations and chained via whole-body motion planning to form new trajectories, so spatial-generalization data comes from recomposing the source demo rather than re-collecting it [§Abstract].
  • Corrective State Expansion increases robot and object state diversity at different stages of mobile manipulation, targeting long-horizon robustness by teaching recovery from off-demonstration states [§Abstract].
  • Cross-environment generalization is unlocked by synthesizing trajectories on diverse generated 3D worlds — background Gaussians are produced by Marble from a single everyday photograph, not just reconstructed from the source demo [Project page, §Abstract].
  • Photorealistic observations are produced by compositing rendered robot and object meshes with Gaussian-splat backgrounds (not pure mesh rendering) [§Abstract].
  • Policies trained with WANDA achieve long-horizon robustness, broad spatial generalization, and cross-environment generalization from a single real demonstration across simulation and real-world tasks [§Abstract].
  • WANDA supports cross-embodiment data generation: from one Agibot G1 demo, it generates Linearbot trajectories by re-planning whole-body motion in the same reconstructed world and rendering from the new robot’s cameras, validated by zero-shot deployment on a physical Linearbot with distinct kinematics [Project page].
  • Reported real-world evaluation (project page, Agibot G1, five long-horizon tasks): π0.5 fine-tuned only on WANDA-generated data reaches 54.8% task progress and outperforms a policy trained on 50 teleoperated demonstrations [Project page, Real-world evaluation].

Method

Section titled “Method”

WANDA is a multi-stage pipeline that decomposes “one demo → many trajectories” into reconstruction, diversification, and corrective expansion.

Reconstruction extracts a world substrate from the source RGB-D demonstration: background 3D Gaussian splats via MAtCha, object geometry via a refined BundleSDF pipeline, and 6D object trajectories over the demo. Diversification instantiates the recovered contact-rich interaction across sampled spatial configurations (new object/receptacle placements) inside either the reconstructed scene or a Marble-generated scene built from an everyday photograph; whole-body motion planning chains the interaction segments into full trajectories that respect the new geometry. Corrective State Expansion samples off-demonstration states — perturbations of pose, object placement, and grasp — and computes recovery trajectories from them, so the trained policy learns to return to the interaction-relevant manifold rather than only replaying it.

Rendering composes photorealistic observations by compositing meshed robots and objects onto the Gaussian-splat background, so the training frames match downstream deployment cameras. For cross-embodiment generation, WANDA does not copy joint motion — it re-plans whole-body trajectories for the new robot in the same reconstructed world and renders from that robot’s camera viewpoints. Cloth/fluids are replayed as demonstrated rather than physically simulated; reconstruction handles rigid and articulated parts.

Results

Section titled “Results”

The abstract reports policies trained on WANDA-generated data achieve long-horizon robustness, broad spatial generalization, and cross-environment generalization from a single real demonstration, plus cross-embodiment transfer via zero-shot Linearbot deployment on a policy trained from an Agibot G1 demo [§Abstract, Project page].

Quantitative headline from the project page’s real-world evaluation (Agibot G1, five long-horizon tasks, 10 trials each): π0.5 fine-tuned only on WANDA-generated data reaches 54.8% task progress and outperforms a π0.5 trained on 50 teleoperated demonstrations — every WANDA number starts from one demonstration [Project page, Real world panel]. The project page additionally reports data-efficiency scaling on the BEHAVIOR Challenge Q-score for a 3-minute task and a wall-clock-hours-per-1h-of-data cost panel where WANDA is shorter than the teleop/UMI baselines shown. Detailed per-task tables are in the arxiv PDF.

Why it’s interesting

Section titled “Why it’s interesting”

WANDA is the closest sibling on the wiki to Video2Robo: 3DGS-based Synthetic Data from One Video Enables Scalable Robot Learning — both take a single input demo, reconstruct the scene as an editable 3DGS asset, and diversify it into a training corpus — but WANDA pushes further along two axes: contact-rich manipulation is preserved via BundleSDF-tracked 6D object trajectories rather than re-rendered scene edits, and whole-body motion planning is the diversification engine rather than viewpoint/texture/lighting augmentation. That distinction matters for mobile manipulation, where the interaction dynamics are the load-bearing supervision. It also extends the LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World “3DGS photorealism is the load-bearing piece” result from static backgrounds to a full data engine that composites moving robots and articulated objects into splat scenes. On the Synthetic Training Data concept, WANDA lands at the extreme end of the “prices data in compute, not human-hours” axis: one demonstration is the entire human input, and the reported π0.5 result beating 50 teleop demos is the sharpest filed datapoint for that framing on a mobile manipulator with long-horizon tasks. The cross-embodiment story (Agibot G1 → Linearbot via re-planning rather than joint retargeting) also gives it a shared substrate with the world-reconstruction leg of SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation‘s digital-cousins recipe, but starting from a demo rather than a scanned digital twin.

See also

Section titled “See also”