Data Scaling Laws in Imitation Learning for Robotic Manipulation

ListenDiversity Is All You Need: Scaling Laws for Robot Data11:02

Show notes

A deep look at Lin, Hu, Sheng, Wen, You, and Gao's Chinchilla-style scaling study for real-world robotic manipulation. The headline: environment and object *diversity* — not raw demonstration count — is the load-bearing lever, and four people in one afternoon can collect enough data to hit roughly ninety percent success on unseen scenes and objects. We unpack the power-law fits, the "one object per environment, thirty-two pairs, fifty demos" recipe, and what it means for anyone building data pipelines for generative and embodied models. **Sources** - Data Scaling Laws in Imitation Learning for Robotic Manipulation (Lin, Hu, Sheng, Wen, You, Gao)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Lin, Hu, Sheng, Wen, You, and Gao (Tsinghua + Shanghai Qi Zhi + Shanghai AI Lab) run the first systematic Chinchilla-style scaling study on real-world imitation learning for robotic manipulation, using >40,000 UMI-collected demonstrations and >15,000 real-robot rollouts. Object generalization and environment generalization each follow clean power laws in the number of training objects / environments; the number of demonstrations per environment-or-object plateaus quickly. Diversity of environments and objects — not raw demonstration count — is the load-bearing lever, with the sharpest form of the recipe being to collect one unique manipulation object per environment and stop at ~32 environment-object pairs × 50 demos each to reach ~90% success on new environments with new objects. Foundational Oct-2024 paper that this wiki has been implicitly referencing throughout the VLA Models recipe-lever debate.

Key claims

Generalization to new objects scales approximately as a power law in the number of training objects; generalization to new environments scales approximately as a power law in the number of training environments; generalization to both simultaneously scales as a power law in the number of training environment-object pairs — with $R^2$ correlation coefficients close to 1 when fitting $y = a \cdot x^{-b}$ on the log-transformed optimality gap [§4.2, Fig. 5].
The number of demonstrations per environment or per object plateaus quickly: once past a low threshold, additional demonstrations yield minimal gains, and increasing environment/object diversity is far more effective than increasing demonstrations per collection unit [§4.1, Fig. 4; §4.3, Fig. 7].
Collecting data in as many diverse environments as possible with one unique manipulation object per environment matches or beats collecting multiple objects per environment once the environment count exceeds ~16 [§4.3, Fig. 6].
32 environment-object pairs × 50 demonstrations each is empirically sufficient to train a policy that reaches ~90% success on 8 new environments with unseen objects for two manipulation tasks (Pour Water, Mouse Arrangement); the recipe generalizes to two additional tasks (Fold Towels, Unplug Charger) with four data collectors in one afternoon [§4.3, §5, Table 1].
Simultaneously varying both environments and objects during training makes the demonstration-count-per-pair saturation faster than varying only one dimension — diversity across dimensions reduces the marginal value of demonstration count [§4.1, Fig. 4].
Validation-set action MSE does not reliably correlate with real-world task success, so tester-assigned normalized scores across multiple stages are used as the primary evaluation metric [§3 Evaluation, Appendix E.1].
The reported policy backbone is Diffusion Policy with a fine-tuned DINOv2 ViT-L/14 visual encoder — DINOv2 features outperform ImageNet-pretrained ResNet and CLIP ViT for this task class, and temporal ensemble smoothing (from ACT) is used at inference [§3 Policy learning].

Method

The empirical protocol fixes one manipulation task and collects human demonstrations using Universal Manipulation Interface (UMI) hand-held grippers, varying three data-collection dimensions: number of environments $M$ , number of manipulation objects $N$ within a category (each drawn from real everyday items), and demonstrations $K$ per object per environment. Environments are collected in-the-wild across diverse real scenes; objects vary in color, size, and geometry within a category (bottles, mice, towels, chargers). For each $(M, N, K)$ combination a Diffusion Policy is trained on the resulting corpus: DINOv2 ViT-L/14 visual encoder (fine-tuned end-to-end), CNN-based U-Net noise predictor, DDIM sampler for real-time control, and ACT-style temporal ensemble at inference. Evaluation is done exclusively on unseen environments and unseen objects (typically 8 held-out environments × 5 trials each). To reduce tester bias, multiple policies trained on different data configurations are evaluated in random order under matched initial conditions; per-stage normalized scores (max 1.0) are reported rather than binary success rates. Power-law fits use the optimality gap $y_{\max} - y$ as the dependent variable, with $M$ , $N$ , or $M{=}N$ (environment-object pairs) as the independent variable, fitted linearly in log-log space.

Results

Across two case-study tasks (Pour Water, Mouse Arrangement) with 3,765–3,820 valid demonstrations at max scale per collection axis:

Object generalization: normalized score exceeds 0.8 at just 8 training objects and 0.9 at 32 for both tasks — a very steep initial slope [§4.1, Fig. 2].
Environment generalization: slower initial slope than object generalization; both benefit monotonically from more environments even at fixed total demonstration count [§4.1, Fig. 3, and Appendix G.2 Fig. 22].
Both-dimension generalization: performance monotonically increases with environment-object pair count; 25% and 100% demonstration-fraction curves overlap, i.e. saturation happens faster than either dimension alone [§4.1, Fig. 4].
Demonstrations-per-pair threshold: performance plateaus at 400 demos for 8 pairs, 800 for 16 pairs, 1600 for 32 pairs — the paper recommends ~50 demos per pair as the operating point [§4.3, Fig. 7].
Power-law prediction: extrapolating the Mouse Arrangement fit predicts 1,191 environment-object pairs to reach normalized score 0.99 on new environments and new objects — a testable claim left to future work [§4.2].
Strategy validation on new tasks: applying the “32 env-object pairs × 50 demos, one object per environment” recipe to Fold Towels and Unplug Charger with four collectors over one afternoon yields normalized scores 0.922 (Pour Water) / 0.933 (Mouse Arrangement) / 0.95 (Fold Towels) / 0.887 (Unplug Charger), with success rates 85.0% / 92.5% / 87.5% / 90.0% on 8 unseen environments with unseen objects [§5, Table 1].
Preliminary model-scaling probe: briefly explored in §6 with the DINOv2 ViT-L/14 backbone chosen so encoder capacity does not bottleneck data scaling; specifics deferred.

Why it’s interesting

This is the foundational Oct-2024 paper that the wiki’s VLA Models recipe-lever board has been implicitly referencing for the “diversity beats demonstrations” position, made concrete a year before the more scaled follow-ups this wiki tracks. Directly complements EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale’s log-linear $R^2 = 0.9983$ egocentric-hours-vs-loss law and Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2’s 1M-hour cross-substrate scaling on the axis that Lin et al. establish here — diversity (environments × objects) as the primary axis rather than raw hours or teleoperation count. The recipe of “one object per environment, ~32 pairs, ~50 demos each” is the small-scale, single-task-generalist counterpart to Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training XR-1’s 100K-hour UMI-scale bet and to UMI-Bench 1.0: An Open and Reproducible Real-World Benchmark for Tabletop Robotic Manipulation with UMI Data‘s benchmark for evaluating UMI-style wrist-camera policies — same substrate (UMI + Diffusion Policy), opposite operating regime. Sharpens the Hyperparameter scaling laws page’s growing collection of embodied-data scaling fits with the earliest clean power-law entry on the manipulation side and puts a specific fit form ( $y = a \cdot x^{-b}$ in the optimality gap) that later embodied entries either follow or extend.