What Matters in Learning from Large-Scale Datasets for Robot Manipulation

TL;DR

The MimicLabs study systematically dissects which axes of dataset diversity matter when co-training robot manipulation policies with large-scale demonstration datasets. A procedural sim data generator built on BDDL + MimicGen produces ~1M trajectories across 3K+ task instances in 8 environments with controlled combinations of camera poses, textures, spatial arrangements, and motions. Two operating modes are analyzed — the collector (which dimensions of variation to prioritize when authoring new data) and the retriever (which subsets to pull from an existing corpus like DROID for a given target task). Camera pose and spatial arrangement dominate; object textures barely matter. Retrieval strategies transfer to DROID and beat full-dataset co-training by up to 70% on target tasks.

Key claims

Camera pose is the single dimension where both alignment between target and co-training and diversity in co-training are critical; misaligned + low-diversity camera poses actively prevent skill transfer, and adding camera-pose diversity recovers >2× success even when the target pose is not covered [§5.1, Table 1].
Spatial arrangement (of both objects and receptacles) must be either aligned or highly diverse in co-training; a co-training set with narrow, misaligned spatial coverage overfits and hurts downstream performance [§5.1, Table 1; §5.2, Table 2].
Object textures are not a load-bearing dimension: co-training with completely misaligned and low-diversity object textures still yields >90% task success on the clear-table task, so long as object geometries match [§5.1, Table 1].
High camera-pose diversity acts as a general robustness lever — it substitutes for misaligned table or object textures because varied viewpoints implicitly cover more background variation [§5.1].
Motion-primitive alignment matters when the co-training set has only one motion; increasing motion diversity monotonically improves target-task success and removes the need for exact alignment [§5.1, Table 10].
Applying the retrieval strategy — align camera poses + retain spatial-arrangement diversity — on the real-world DROID dataset outperforms “co-train on all of DROID” by up to 70% on target real-robot tasks [§1, §5.3].
Retrieval-side takeaway: naive co-training with a large heterogeneous dataset can degrade downstream performance vs target-only training when misalignment on load-bearing DVs is not addressed [§5.2, Table 2].
The MimicLabs generator itself (BDDL task specs + auto-decomposed MimicGen source demos) is released as a controllable procedural pipeline that scales from ~500 human demos to ~1M synthetic trajectories with combinatorial DV control [§4].

Method

The authors formalize dataset composition through Dimensions of Variation (DVs) — independent, measurable axes such as camera pose, object texture, table texture, object spatial arrangement, and motion primitive — parameterized as distributions attached to a demonstration distribution. This lets them compare target and co-training datasets along each DV under four cases (aligned/misaligned × narrow/wide support).

To make controlled experiments tractable, MimicLabs extends LIBERO’s BDDL grammar for procedural task specification and MimicGen for automated demonstration synthesis: given ~500 source human demos, the pipeline decomposes trajectories into object-centric segments and stitches them across BDDL-specified variations in camera pose, textures, spatial arrangement, and object geometry, producing ~1M trajectories across 3K+ task instances in 8 scenes.

Two operating perspectives are then studied on this substrate:

Collector: fix target and co-training aligned on all DVs except one, then sweep that DV’s distribution in co-training to isolate its effect [§5.1].
Retriever: for each DV, build three co-training subsets (fully aligned, misaligned + narrow, misaligned + broad) plus a fully varied baseline, all at fixed size (10 target + 1000 co-training demos), and measure downstream success [§5.2].

Real-world validation runs 7 manipulation tasks on physical hardware, and the retriever recipe is applied to DROID’s ~76K episodes to select subsets aligned with target camera poses [§5.3].

Results

Camera pose in collection: on the clear-table task under a camera-pose misalignment, target-only reaches 16.7%; naive co-training baseline 43.3%; adding camera-pose diversity in co-training jumps to 90% [Table 1].
Object texture in collection: misalignment on object texture has essentially no cost — all co-training variants land at 90–96.7% success vs 93.3% baseline [Table 1].
Spatial arrangement in collection: under objSpat misalignment, target-only reaches 10%; naive co-training only 26.7%; adding spatial-arrangement diversity to co-training reaches 56.7% — the single DV that its own diversity recovers best [Table 1].
Retrieval on microwave-mug: aligned camera pose reaches 70% vs 60% for full-variation baseline; misaligned + low-diversity camera poses drop to 46.7%, showing that a bad retrieval strategy can underperform naive co-training [Table 2].
Retrieval on clear-table: aligned camera pose reaches 36.7% vs 30% baseline; retaining broad spatial arrangement diversity in retrieval reaches 40% [Table 2].
Retrieval on real DROID: informed retrieval outperforms “train on all of DROID” by up to +70% on real-robot target tasks [§1, §5.3].
Motion coverage: for both variants of the clear-table task (pickPlaceTopDrawer+push and pull+pickPlaceTopDrawer), the co-training subset that maximally covers the target motion consistently produces the best downstream success, and single-motion misaligned co-training can drop success by 20% on the easier variant [Table 10].

Why it’s interesting

This paper is the clearest filed evidence for a specific “retrieval > naive scale” position that several concurrent robotics-data works only argue at the aggregate level. It sharpens Synthetic Training Data‘s “curation gate is load-bearing” thesis by naming which axes of the target distribution the gate should align to — camera pose and spatial arrangement — and which it can safely ignore (object texture). Contrasts sharply with Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models‘s “clean data is the enemy of great robot foundation models” position: MimicLabs shows that on the retrieval side, targeted alignment beats broad co-training, whereas Spirit AI argues that broad, diverse real-world data beats filtering. Complements Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation‘s WANDA (which composes many synthetic trajectories from one demo) by identifying which dimensions of variation those compositions should span. The Table 1 finding that camera-pose diversity substitutes for texture alignment is the specific mechanism behind why R2S2R-style engines (Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies) that vary camera trajectories per scene transfer well without matching photometric properties exactly. The Ishaan-quoted retriever caveat — “random co-training with a large dataset can hurt performance” — is a direct rebuttal of the default DROID/OXE co-training pipeline.