LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World

TL;DR

LEGS is a hybrid simulator for humanoid loco-manipulation training that composites a mesh foreground (robot, objects, props) over a photorealistic 3D Gaussian Splatting (3DGS) background reconstructed from a handheld scene capture. A procedural motion-primitive generator (Walk → Pick → Place) synthesizes labeled demonstrations at scale with no human teleoperation, and a two-stage color calibration aligns rendered 3DGS images to the robot’s deployment camera. Across 1,110 real-robot trials on a Unitree G1, three pick-and-place tasks of increasing whole-body difficulty, and three VLA backbones (π0, π0.5, GR00T N1.6), policies fine-tuned purely on LEGS data match or exceed teleoperation on every (backbone, task) cell and reach 67% TSR vs 33% for a mesh-only ablation — identifying photorealistic rendering as the key transfer enabler.

Key claims

Teleoperation-free synthetic data matches or exceeds human teleoperation for VLA fine-tuning on every (backbone, task) cell evaluated: LEGS (200) ≥ Teleop (50) across nine experiments; on long-horizon Task 3, teleop collapses to 0/10 across all three backbones while LEGS reaches up to 6/10 [Results, Q1].
The improvement is not from dataset size: at a budget-matched 50 demos, LEGS (50) still matches or surpasses Teleop (50) on every experiment, isolating the gain to the data pipeline rather than its scale [Results, Q2].
Photorealism approximately doubles end-task success. A mesh-only SAM3D (200) baseline that ablates the 3DGS background averages 33% TSR versus 67% for LEGS (200), with the gap concentrated at close-range pick and place stages [Results, Q3, Results table].
Each new appearance condition (object swap, scene swap, or both) re-renders the same recorded motion at ~0.1 GPU-hr vs >1.5 operator-hr for teleoperation — ~15× cheaper [Results, Q4].
Under combined object-and-scene appearance shift, the policy trained on re-rendered LEGS-AUG data reaches 100/80/40% on Tasks 1–3 while teleoperation and unaugmented LEGS fail entirely [Results, Q4, LEGS-AUG section].
Decoupling foreground motion from scene appearance is what makes augmentation tractable: motion is recorded once in MuJoCo and re-composited against arbitrary 3DGS backgrounds and object meshes without re-rolling-out the controller [Method].

Method

LEGS reconstructs a scene background as a 3D Gaussian Splat from a handheld phone capture, then composites a mesh-rendered foreground (Unitree G1 + manipulable objects + props) over that 3DGS background in MuJoCo. Each pick-and-place task decomposes into Walk → Pick → Place motion primitives; a procedural generator samples scene-level arguments (object positions, robot heading, base offset) per initial condition, executes the trajectory using the robot’s whole-body controller, and saves only verified successful episodes. A deterministic two-stage color calibration is applied to rendered frames to match the deployment camera’s color statistics (head-mounted Intel RealSense D435, 30 Hz RGB on real hardware). Because the foreground trajectory is recorded independently of the 3DGS background, the same motion can be re-rendered under novel scenes and object meshes to produce LEGS-AUG data for appearance robustness, without re-running the controller.

Results

Across 1,110 real-robot trials on a Unitree G1 with three pick-and-place tasks and three VLA backbones (π0, π0.5, GR00T N1.6), under ±5 cm object perturbation, ±10° robot heading, and ±10 cm robot base perturbation:

LEGS (200) achieves up to 10/10, 9/10, and 6/10 end-task success on Tasks 1, 2, 3 respectively (best per cell); average 67% TSR vs SAM3D (200) 33% and Teleop (50) 0/10 on Task 3 across all three backbones [Results table].
LEGS (50) at a budget-matched scale still beats Teleop (50) on every cell, including 9/10 vs 4/10 on Task 1 with GR00T N1.6 [Results table].
Under combined object-and-scene appearance shift, LEGS-AUG reaches 100/80/40% on Tasks 1–3 while both Teleop and unaugmented LEGS fail [Q4].
Re-rendering cost: ~0.1 GPU-hr per new appearance condition vs >1.5 operator-hr per teleop re-collection (~15× cheaper) [Q4].

Why it’s interesting

LEGS is a concrete demonstration that 3DGS-backed sim + procedural motion primitives can fully replace human teleoperation for humanoid VLA fine-tuning — a position that runs against the VLA Models cluster’s “clean teleop data is the lever” pole (Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models) and complements the action-pretraining-free embodied-VLM pole (Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models) on the data axis rather than the architecture axis. For the Synthetic Training Data cluster it is the first filed instance where photorealism is explicitly ablated (mesh-only SAM3D vs 3DGS) and shown to roughly double real-world transfer — a sharper isolation than the rendering-quality discussions in Nyx — GPU path-traced renderer plugin for Genesis World or the eval-only stance of The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward. The “record motion once, re-render against arbitrary backgrounds” recipe is the manipulation-side analog of NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos‘s decoupling of motion from scene appearance.