Neural Scaling Laws in Robotics

TL;DR

Sartor & Thompson meta-analyze 327 papers on foundation models for robotics — VLAs, VLMs, PVRs, and LLMs applied to manipulation/planning/reasoning — and fit power laws for downstream success rate against data size, model size, and compute across every study that reports at least 3 data points. Only 13% of surveyed papers report any scaling study at all (26 for data, 22 for model size, 1 for compute), and the vast majority of “scaling studies” have only 2-3 data points, so the resulting exponents are averaged across a small, heterogeneous population rather than derived from any single controlled sweep. The fitted average exponents are $\alpha_D = -0.276$ (95% CI [-0.317, -0.236], n=131 data scaling studies) and $\alpha_N = -0.246$ (95% CI [-0.337, -0.155], n=34 model-size scaling studies), with mean top task-success rate across the corpus of 67% (SD 24%). The paper’s methodological contribution is that it is the accounting — it quantifies how little rigorous scaling data the robotics-foundation-model literature has actually produced.

Key claims

Only 13% of 327 surveyed foundation-model-in-robotics papers report any scaling study; of those, most have 2 or 3 data points per curve, and only 1 paper analyzes how success rate varies with computational resources [§III-B, Table IV].
Data-size scaling dominates the reported studies (26 papers, 20 with ≥3 data points, 147 usable curves), model-size scaling is second (22 papers, 11 with ≥3 data points, 43 curves), compute is a single-paper wasteland [Table IV].
Power-law fits substantially outperform linear-model fits on the pooled scaling curves: median $R^2$ 0.93 (data), 0.86 (model size), 0.86 (compute) for power law vs 0.60 / 0.64 / 0.72 for linear [Table V].
Fitted average data-scaling exponent across 131 studies with $\alpha_D \in [-1, 0]$ is $\alpha_D = -0.276$ (95% CI [-0.317, -0.236]) [§IV-A, Fig. 4].
Fitted average model-size scaling exponent across 34 studies is $\alpha_N = -0.246$ (95% CI [-0.337, -0.155]) [§IV-B].
Reported dataset sizes span 1 to 1M demonstrations (6 OoM); model sizes span 3M to 1.8T parameters (6.8 OoM); compute studies span only 2.4 OoM — the smallest sweep width of any axis [Table VI].
Across the 424 total scaling studies (including 2-data-point curves), mean top task-success rate is 67% (median 70%, SD 24%) — far below the 99.X% reliability that real industrial and home applications require [§IV-A].
LLMs applied to robotics appear to scale more efficiently on data than VLAs, VLMs, or PVRs; the authors flag the small LLM sample size as a reason for caution [§IV-A, Fig. 4].
76% of the 327 papers were published in 2023 or 2024; Google DeepMind alone contributes 19% of the corpus [§III-A, Fig. 2].
Scaling law research applied to robotics is nascent — the four papers the authors identify as explicitly quantifying scaling laws in robotics (Villasevil et al. 2024, Lin et al. 2024, Duan et al. 2024, Pearce et al. 2024) all report power-law exponents in fundamentally different scaling regimes (data-diversity, dataset-size, compute-for-world-models), and no filed paper has a joint $(N, D, C)$ fit at the time of this meta-analysis [§II, Table III].
The paper advocates for reporting computational resource usage (FLOPs, PF-days) alongside every scaling result, since almost no surveyed paper does — a methodological gap that prevents compute-optimal analysis in the robotics regime [§III-B].

Method

The authors screen 327 papers on foundation models applied to robotics (RFMs — Robot Foundation Models, split into PVRs, VLMs, VLAs — plus LLMs used for robotic reasoning/planning), drawn from survey articles, GitHub repositories, citations of Open-X-Embodiment, and 2024 newsletters. Each paper is coded for whether it reports a scaling study on data size (demonstrations, trajectories, episodes, images, or frames), model size (parameters), or compute (FLOPs, PF-days, GPU-hours). Papers that report at least 3 data points per axis are used to fit power laws of the form L(x) = a·x^α + c; papers with only 2 data points are aggregated separately for descriptive statistics but not power-law fitting. Success rate is the only performance metric used because it is the only metric with a sufficient sample size across the corpus. For each fitted study, the exponent $\alpha$ is retained only if $\alpha \in [-1, 0]$ (87% of the studies), which the authors argue corresponds to the diminishing-but-positive-returns regime. Group means and 95% confidence intervals over the fitted exponents are reported per model type (VLA / VLM / PVR / LLM), sim vs real deployment, and pretraining vs fine-tuning. Because only one paper reports compute scaling (Liu et al.), the authors estimate training FLOPs for it from disclosed hardware specs and training duration using Epoch AI’s estimator.

Results

Data scaling: 131 usable power-law fits across data-size studies, average exponent $\alpha_D = -0.276$ (95% CI [-0.317, -0.236]). Median max/min ratio within a study is 10× (i.e., most sweeps span only one order of magnitude). VLAs dominate the corpus (n=93); LLM sub-population appears to scale more efficiently but has small sample size [§IV-A, Fig. 4].
Model-size scaling: 34 usable fits, average $\alpha_N = -0.246$ (95% CI [-0.337, -0.155]). Model sizes in the studies range from 3M to 1.8T parameters (6.8 OoM overall corpus, 10.3× median within-study sweep). VLA-specific and cross-type differences are noted but with small samples [§IV-B].
Compute scaling: 24 fitted curves derive from a single paper (Liu et al.); insufficient to draw cross-paper conclusions. Compute sweep width is only 2.4 OoM overall — a full order of magnitude less than data or model size [Table VI].
Absolute performance ceiling: mean top task-success across 424 scaling studies is 67% ± 24 (median 70%). Given the long-tail nature of power-law scaling and target reliability of 99.X% for industrial deployment, this implies enormous data/compute requirements to close the gap [§IV-A].
Emergence: the authors report qualitative evidence of emergent capabilities (novel behaviors appearing at scale) but note downstream capabilities remain difficult to predict from upstream loss improvements, consistent with the LLM literature [§II].
Publication trajectory: 76% of the 327 surveyed papers date to 2023-2024; the trend line is exponential, with Google DeepMind, Stanford, and UC Berkeley leading. Industrial/academic collaboration is the dominant configuration [§III-A].

Why it’s interesting

This is the accounting paper for the exact gap yixuan.ren’s Slack message names: 327 papers surveyed, only 22 report any model-size scaling study, only 1 reports compute scaling, and the median within-study parameter sweep is 10.3× — a single order of magnitude — with the fitted average exponent $\alpha_N = -0.246$ pooled from 34 heterogeneous studies. Compare this to the LLM side: Scaling Laws for Neural Language Models fits $\alpha_N$ on 1000× parameter sweeps of controlled architectures on a fixed dataset, and the modern LM literature has moved on to disentangling $(N, D, C, B)$ jointly (How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size, The Design Space of Tri-Modal Masked Diffusion Models). Robotics has essentially no controlled model-size sweep at parity with what Kaplan et al. published five years ago. The closest filed counter-examples are EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data (log-linear data-hours fit at $R^2 = 0.9983$ , but only sweeps data, holds model architecture fixed), Isaac 0.5: An Open-Weight Embodied Foundation Model (video-vs-teleop data-composition exchange, no model-size axis), and the announced-but-unpublished Sunday Robotics ran the Chinchilla scaling law with their VLA model (Tony Zhao follow-up) which would be the first VLA joint $(N, D)$ isoFLOP fit if it lands. Sartor & Thompson’s own headline numbers ( $\alpha_D = -0.276$ , $\alpha_N = -0.246$ ) should be read as lower bounds on the noise floor of the current literature, not as ground-truth robotics scaling exponents — they are means over dozens of uncontrolled small-N sweeps by different labs on different tasks, embodiments, and metrics. The meta-analysis’s real value is diagnostic: it establishes that as of mid-2024, the robotics-foundation-model literature has not produced a single clean $\alpha_N$ measurement of the sort LLM/diffusion research routinely publishes, and that computational-resource reporting is essentially absent.