ScheduleGen: Demonstration Generation via Skill Scheduling for Multi-Arm Imitation Learning

TL;DR

ScheduleGen (NVIDIA + UT Austin; Jawale, Pleines, Pouya, Zhu, Mandlekar, Garrett) is an integrated planning + scheduling algorithm for multi-arm demonstration generation: from a handful of source demonstrations it synthesizes collision-aware, time-coordinated trajectories across bimanual, four-arm, and fixed-waist humanoid platforms, extending the MimicGen lineage to the setting where prior demo-generation algorithms break because they ignore multi-arm constraints. The pipeline transfers object-relative contact skills to sampled scenes, plans connecting per-arm joint-space motions, then solves a MILP over collision / ordering / synchronization constraints to produce a joint schedule, and finally replays schedules in parallel simulation with success filtering. Reported numbers: 79.6% mean data-generation success and 65.5% mean policy-training success across 28 simulation task variants (seven externally-authored RoboDojo tasks included), plus 50–80% success across eight real-world variants via zero-shot sim-to-real and real-to-sim-to-real transfer.

Key claims

A single integrated planner+scheduler is required to generate valid multi-arm demonstrations: per-arm paths preserve contact-rich behavior while a joint MILP schedule resolves task-ordering, collision, and synchronization constraints that single-arm demo-generation methods (MimicGen-style) cannot express [§Method].
Object-relative skills, collision-aware connecting motions, and multi-arm timing are solved together in one generator rather than as separate stages [§At a glance].
The pipeline covers setups prior demo-generation systems do not reach: shared workspaces, clutter, long-horizon contact, four-arm systems, and a fixed-waist humanoid [§At a glance, §Simulation].
Across 28 simulation variants (including seven externally-authored RoboDojo tasks), ScheduleGen achieves 79.6% mean data-generation success and trains policies with 65.5% mean success — outperforming a state-of-the-art data-generation baseline under matched source demonstrations within each task family [§Simulation, §Shared-source evaluation].
Policies trained only on ScheduleGen data transfer to real hardware across eight variants at 50–80% success under both zero-shot sim-to-real (simulation-only policies evaluated directly on robots at 77.5% mean success across four) and real-to-sim-to-real (real demonstrations seed randomized simulation datasets, policy returns to the same robot) [§Real-world transfer].
RoboDojo per-task numbers show the generator reaches high data-generation yield well above the resulting policy success rate — e.g. 94.2% generation / 45% nominal policy, 96.5% / 50%, 100% / 40%, 88.0% / 65%, 75.5% / 65% — i.e. the generator is not the bottleneck in these experiments [§RoboDojo data generation].
The scheduler itself contributes a measurable efficiency effect: across 3,500 generated trajectories, scheduling increases simultaneous arm motion and reduces execution length relative to the released source demonstrations [§Scheduling effect].
On Bowl stacking (Cross-Arm, two arms), the generator reports 66.7% data-generation success and 80% policy success [§Bowl stacking header].

Method

ScheduleGen takes a small set of source demonstrations with object-relative contact skills and expands them into training-scale multi-arm demonstration data via four stages:

Transfer object-relative contact segments to sampled scenes — contact-rich behavior is preserved per-arm by replaying skills in the target object’s frame rather than world frame.
Connect skills with feasible per-arm joint-space paths — a planner fills the inter-skill motion with collision-free joint-space trajectories.
Schedule across arms by MILP — collision, task ordering, and synchronization constraints are encoded as a mixed-integer linear program over per-arm paths; the solver returns a joint schedule that aligns the arms in time.
Replay in parallel simulation and filter by success — scheduled trajectories are executed in sim; only successful ones are retained as training data.

The pipeline is applied across bimanual (two-arm Cross-Arm layouts), four-arm, and fixed-waist humanoid platforms, with the same scheduler stage adapted per-platform via its constraint graph. Real-robot transfer uses one of two regimes: pure zero-shot sim-to-real (train on sim only, deploy without finetuning) or real-to-sim-to-real (a handful of real seeds seed randomized simulation datasets, then the learned policy returns to the same robot).

Results

Overall simulation (28 task variants): 79.6% mean data-generation success, 65.5% mean policy success — outperforms a state-of-the-art data-generation baseline across the shared-source evaluation [§Shared-source evaluation].
RoboDojo (seven externally-authored tasks): data-generation rates 75.5–100%; nominal policy success 25–65% at 20 held-out eval episodes per task [§RoboDojo data generation].
Scheduling effect (3,500 trajectories): higher concurrent arm motion and shorter trajectories on average vs the released source demonstrations [§Scheduling effect].
Real-world transfer (8 variants, 20 hardware episodes each, policies trained in sim without real-world fine-tuning): 50–80% success. Zero-shot sim-to-real mean 77.5% across four simulation-only policies (Bowl stacking Normal + Cross-Arm layouts; Dishware stacking Normal + Cross-Arm layouts) [§Real-world transfer, Zero-shot].
Demonstration-seeded real-to-sim-to-real: Cube stacking (five real seeds, 20 episodes) and three Throwaway trash variants (Normal, Cup Cross, Cup + Bin Cross) all evaluated under the same protocol [§Demonstration-seeded].
Headline per-task: Bowl stacking Cross-Arm — 66.7% generation, 80% policy [§Bowl stacking header card].

Why it’s interesting

ScheduleGen plugs a specific gap in the demo-generation lineage tracked under Synthetic Training Data: MimicGen-style pipelines (cf. Behavior Prompting Policy: Demonstrations as Prompts for Manipulation, which uses BDDL+MimicGen to synthesize 174-task and 321-task extensions of LIBERO, and What Matters in Learning from Large-Scale Datasets for Robot Manipulation MimicLabs, which runs the sharpest DV-level ablation on ~1M synthetic trajectories) scale coverage along object/scene/motion axes but assume a single arm, so they cannot produce the shared-workspace, collision-aware, time-synchronized trajectories needed for bimanual / four-arm / humanoid platforms. ScheduleGen adds the planning+scheduling cell to that recipe board — per-arm trajectory synthesis is handled by prior methods, but the MILP over inter-arm constraints is the new piece that makes the outputs executable without self-collision.

The real-to-sim-to-real variant is the closer sibling to Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies R2S2R and Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation WANDA (one-demo scene reconstruction + trajectory synthesis), but attacks the orthogonal axis: WANDA/R2S2R synthesize scene variation around a single demo, while ScheduleGen synthesizes multi-arm scheduling variation around a small set of per-arm skills. The 50–80% real-robot success across eight variants with no real-world fine-tuning is the sharpest filed sim-to-real datapoint for bimanual + humanoid platforms specifically, where ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning ADEPT (pure-sim RL rollouts) and Simulation Pre-training for Dexterity (SPD) SPD (sim VR teleop) stake out the single-platform baselines.

The scheduling-effect measurement (concurrent arm motion ↑, trajectory length ↓ across 3,500 trajectories) is a quiet but important datapoint: the MILP is doing visible work beyond just avoiding collisions — it exploits parallelism that the source demonstrations don’t have, which matters because the generated data then teaches policies to execute faster than their human/scripted sources.