Skip to content
Bud's Wiki

Simulation Pre-training for Dexterity (SPD)

TL;DR

Section titled “TL;DR”

SPD (Kamat, Rashid et al.; Stanford / MIT / Scale AI — Chelsea Finn, Phillip Isola, C. Karen Liu labs) tests whether simulation-teleoperated data — humans manipulating virtual objects inside a VR headset — can serve as a viable pre-training substrate for real-world bimanual dexterous manipulation. Five operators collect 75 hours of multi-task demonstrations over one week in MuJoCo, on the same embodiment (two YAM Pro arms + 22-DoF Sharpa Wave dexterous hands) used at deployment; a diffusion transformer with sliding-window context is pre-trained on this data and fine-tuned with 1–2 hours of real teleop per task. On five real-world contact-rich tasks (plate racking, mug hanging, Jenga, cup stacking, bottle tossing), the pre-trained policy beats a from-scratch BC baseline of the same architecture, and the largest pre-training benefit lands on the w=32, c=8 configuration — 32-step history window + 8-step action chunks — which without history would collapse for shakiness.

Key claims

Section titled “Key claims”
  • Simulation VR teleoperation on the target embodiment yields action-labeled, on-embodiment dexterous pre-training data that transfers to real robots without any explicit retargeting or human-to-robot embodiment gap [§1, §3.1].
  • Five operators collect ~2,000 episodes (75 h) across six scenes (spelling blocks, dishes, mugs, bottles, cups, Jenga) in one week — task prompts, assets, physical properties, and initial positions are randomized per episode, yielding diverse contact-rich behavior [§3.1, Fig. 2].
  • The policy is a causal diffusion transformer with a shared trunk and separate action-denoising expert; training sequences span 256 timesteps at 30 Hz and all future action chunks are denoised in parallel under a causal mask, amortizing sequence cost [§3.3, Fig. 3].
  • Every attention layer uses sliding-window attention with a 32-timestep window, matched to a rolling KV cache at inference; a frozen ViT patch-encoder with cross-attention pooling (patches re-cross-attended every two blocks) plus 8-timestep visual subsampling handles image inputs [§3.3, Fig. 3].
  • Pre-training + 1–2 h fine-tuning beats matched-architecture from-scratch BC on all five real-world tasks, reaching nearly every task stage more often and matching lower training loss (which the ABC paper argues correlates with real-world success unlike validation loss) [§4.1, Fig. 5].
  • History conditioning + short action chunks is the configuration that most benefits from pre-training: w=32, c=8 gains +18 pp average task progress from pre-training vs ≤3 pp for w=1, c=32 (the π0-style setting) and other variants [§4.2, Table 1, Fig. 6].
  • Under a single-frame context (w=1), shrinking the chunk to c=8 collapses performance — the policy becomes visibly shaky and temporally incoherent — but adding a 32-step window removes this trade-off, drawing coherence from context and reactivity from the shorter chunk [§4.2].
  • The 75-hour dataset (spd-75h), the spd-vr VR teleoperation software, and six curated MuJoCo scenes with tuned contact parameters are released for downstream simulation-pre-training work [§1, §5].
  • Real-world teleoperation (spd-teleop) mirrors the VR setup but swaps headset hand-tracking for Manus gloves + Quest controller for smoother, occlusion-robust wrist tracking — necessary to control the robot through contacts and at distance [§3.2].

Method

Section titled “Method”

Operators wear a VR headset and control the simulated bimanual robot in MuJoCo directly: the sim steps at 480 Hz, hand poses are captured at 60 Hz, and detected wrist + fingertip poses drive the simulated arms via inverse kinematics. Objects are virtual and hand-object contact is physically simulated, so all trajectories are action-labeled on the exact target embodiment. After collection, non-contact segments are filtered out and trajectories are re-rendered in parallel via an adapted Madrona renderer, with segmentation-mask-driven color randomization (object tint, background/table textures) and a left–right symmetry augmentation applied as GPU transforms after data-loading.

The policy is a causal diffusion transformer following the π0-family pattern of a shared multimodal trunk + separate action-denoising expert. Non-visual modalities use modality-specific input/output projections; visual patches come from a frozen ViT pooled via cross-attention from learnable queries, with those queries re-cross-attending to the raw patches every two blocks (context-dependent pooling in the style of PerceiverIO / Show-o2: Improved Native Unified Multimodal Models). Attention is causal with rotary embeddings from timestamps; noised action chunks additionally receive absolute positional embeddings encoding the flow-matching timestep and position within chunk. Every layer is sliding-window with a 32-step window so a fixed-length KV cache suffices at deployment. All models are trained with Muon at a fixed 1e-3 learning rate with EMA weights for inference.

Real-world data collection uses two YAM Pro arms retrofitted with 40:1 J3/J4 motors (from 10:1) to handle the Sharpa Wave hand weight, three RealSense D405 cameras (top + wrist-mounted), Manus gloves for finger tracking, and an attached Quest controller for wrist tracking.

Results

Section titled “Results”
  • Real-world task progress (w=32, c=8, avg across 5 tasks): SPD 76.7 vs from-scratch BC 58.7 — +18 pp absolute [Table 1].
  • Per-task progress with pre-training (w=32, c=8): plates 80.6%, mugs 93.3%, Jenga 85.0%, cups 55.6%, bottles 68.8% [Table 1].
  • Per-task progress from scratch (w=32, c=8): plates 66.9%, mugs 80.0%, Jenga 65.0%, cups 35.0%, bottles 47.5% [Table 1].
  • π0-style baseline (w=1, c=32): SPD 55.6 avg / from-scratch 36.9 avg on plates alone — pre-training gain is smaller than the history-conditioned setting.
  • Single-frame + short chunk (w=1, c=8): collapses on Jenga, cups, bottles (0.0 across both regimes) — history is load-bearing for reactive short-chunk control [Table 1].
  • Training loss: SPD checkpoints start and converge to lower flow-matching loss than from-scratch on all five tasks, matching real-world stage-completion rankings [Fig. 5].
  • Data cost: 75 h simulation pre-train + 1–2 h real fine-tune per task; 20 trials per checkpoint at evaluation.

Why it’s interesting

Section titled “Why it’s interesting”

SPD is a distinctive point on the Human-to-Robot Retargeting recipe board: instead of retargeting human video (Do as I Do: Dexterous Manipulation Data from Everyday Human Videos), scaling egocentric data (EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data), or wearing an exoskeleton locked to the robot’s kinematics (Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1)), it puts the human into a physics simulator and treats simulation VR teleop as a first-class off-robot data substrate — no post-hoc retargeting, no embodiment gap, and reset costs go to zero. This complements the Human-to-Robot Retargeting “make the collection tool embodiment-invariant” thread (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training UMI + HandUMI — Robot-free bimanual data collection retargetable to any parallel-gripper arm (Brik H. Meza launch tweet) HandUMI) by moving the collection environment rather than the tool.

The w=32, c=8 finding is the sharpest empirical counter to π0’s w=1, c=32 design choice (π*0.6: a VLA That Learns From Experience (RECAP)) on file for dexterous contact-rich tasks — history conditioning is what unlocks short reactive chunks, and it is also where pre-training gains concentrate. That framing aligns with recent async-inference work on flow-matching VLAs (πR²: Reactive Real-time Flow Policies πR², Real-Time Robot Execution with Masked Action Chunking REMAC) which likewise argue the training-time treatment of temporal structure is a lever independent of scale.

The dataset + VR software release also gives the Synthetic Training Data cluster a rare action-labeled contact-rich dexterous corpus at 75 h that other groups can build on — most simulation-derived robot data on the wiki is either RL-trained skills or WFM-generated video, not on-embodiment human demonstrations.

See also

Section titled “See also”