Behavioral Cloning Mystery

ListenThe Behavioral Cloning Mysteries: Why Overfitting Wins11:31

Show notes

Seohong Park reports four counterintuitive behavioral-cloning phenomena that show up on real robot data but hide in standard sim benchmarks — then reproduces all four in a purpose-built MJWarp benchmark with an "infinite data" regime. We walk through why overfitting helps, why open-loop chunked policies crush closed-loop ones, why you seem to need a half-billion parameters for a 37-dim task, and why hand-scaling features still matters with unlimited data. Park's throughline: test-time distribution shift plus an expressivity mismatch between non-Markovian data and Markovian policies. **Sources** - Behavioral Cloning Mystery — Seohong Park (blog)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Seohong Park (recently moved from OGBench-era RL into robotics) reports four counterintuitive behavioral-cloning phenomena that appear on real robot data but do not show up in standard sim BC/RL benchmarks (D4RL, OGBench), then reproduces all four in a purpose-built MJWarp-based sim benchmark whose scripted policies mimic the properties of human demonstrations (narrow support, non-Markovian temporal correlation, smoothly random) via random piecewise Hermite splines. Every result uses state-based flow-matching action-chunk BC ( $\pi(a_{t:t+24} \mid s_t)$ ) with a large residual MLP and — crucially — an “infinite data” setting where fresh demo trajectories are streamed and consumed exactly once, so overfitting and data scarcity are ruled out as confounders. The four mysteries: (1) overfitting is often beneficial and larger datasets sometimes hurt performance; (2) open-loop chunked policies drastically beat closed-loop policies, and history-conditioning does not fix closed-loop; (3) even simple state-based single-task manipulation needs ≥[4096]×8 residual MLPs (~0.5B params) to learn well; (4) with infinite data, hand-scaling input features (especially amplifying object $xyz$ ) beats standardized features despite identical train-time flow-loss and MSE. Park’s cross-cutting thesis is that all four are surface expressions of test-time distribution shift combined with an expressivity mismatch between non-Markovian data and Markovian (or chunked-Markovian) policies — and speculates LLMs escape this because internet-scale data makes essentially every test-time state in-distribution.

Key claims

On a state-based block pick-and-place task with 10K episodes, flow-BC task success improves and stabilizes while validation flow loss keeps growing, and 10K-episode datasets can beat 50K-episode datasets from the same distribution (averaged over 4 seeds × 4 independently sampled datasets), evidence that overfitting is often not harmful and larger data can hurt [Mystery 1].
Validation flow loss is a poor proxy for policy performance; validation action-MSE stays stable while flow loss diverges, and the right metric — MSE between generated and target actions under the test-time state distribution induced by the policy — is generally inaccessible because target labels do not exist for test-time states [Mystery 1].
Even under infinite fresh data and a sufficiently large model, a pure closed-loop policy $\pi(a_t \mid s_t)$ “doesn’t even touch the cube” while an open-loop chunked policy $\pi(a_{t:t+24} \mid s_t)$ succeeds, with the reported chunk-length sweet spot around 25 steps for this task [Mystery 2].
History-conditioned closed-loop policies $\pi(a_t \mid s_{t-24:t})$ do not recover open-loop performance and in fact are worse than their non-history counterparts despite achieving lower flow losses and MSEs — proposed hypotheses are causal confusion (previous action is a spurious predictor of the next action) and an enlarged input space exposing more test-time distribution shift [Mystery 2].
A simple, fixed, state-based (37-dim) pick-and-place task requires at least $[4096] \times 8$ residual MLPs to learn well, and $[8192] \times 8$ further improves — nearly a 0.5B-param model — with autoregressive tokenized action prediction requiring similarly large models, so the effect is BC-recipe- rather than model-family-specific [Mystery 3].
Under infinite data + sufficient model, standardized vs handcrafted input features that carry identical information and give identical train-time flow-loss and MSE nevertheless deliver clearly different task success; amplifying the $xyz$ of the object and gripper (×10) significantly outperforms standardized or ×0.1-scaled variants — evidence that test-time generalization, not train-time fit, is what determines BC performance [Mystery 4].
All four mysteries reproduce even in state-based BC (not VLAs), so they are properties of the data distribution and the BC objective rather than of VLM/vision architectures [Setup, closing thoughts].
Hot take on why LLMs escape this: internet-scale pretraining makes essentially every test-time input in-distribution in the learned representation space, which is why train loss is a reasonable performance proxy for LLMs but not for BC — implying that scaling data + models in VLAs to LLM levels may make these mysteries disappear [Closing thoughts].
Two candidate architectural fixes for the non-Markovian-data / Markovian-policy expressivity mismatch: (a) fully history-conditioned autoregressive policies, or (b) hierarchical policies where a high-level plan kills multimodality so the low-level policy can be genuinely Markovian [Closing thoughts].

Method

Park builds a controlled sim benchmark on top of MJWarp (GPU-accelerated MuJoCo) so that fresh demonstration data can be generated continuously and consumed exactly once, yielding an effectively infinite-data training regime. Tasks include block stacking, Tower of Hanoi, and bowling; action space is 7-DoF joint velocity control at 50 Hz with a standard normalization scheme, matched to real robot data. Scripted “human-like” demonstrators are built around random piecewise Hermite splines as trajectory backbones with heavy randomization of control points, grasping angles/directions, contact points, motion speed, and gripper roll/pitch/yaw, plus injected mistakes and recoveries — the goal is not to solve robotics but to preserve the properties of real human demonstrations: narrowly distributed, highly temporally correlated (non-Markovian), and smoothly random. On top of this data pipeline, standard flow matching is used to train length-25 action-chunk policies $\pi(a_{t:t+24} \mid s_t)$ parameterized as (deliberately huge) residual MLPs, since everything is state-based. The four mysteries are then diagnosed by holding data properties + model class fixed and sweeping one axis at a time: dataset size vs training steps (Mystery 1), open-loop chunk length vs closed-loop step-by-step vs history-conditioned closed-loop (Mystery 2), MLP width/depth (Mystery 3), and input-feature scaling (Mystery 4).

Results

Flow-BC on block pick-and-place with 10K episodes: task success monotonically improves and stabilizes as training proceeds; validation flow loss keeps growing across the same window; validation action MSE stays stable [Mystery 1 plots].
Same-distribution 50K-episode datasets underperform 10K-episode datasets under 4-seed / 4-dataset averaging on the same task [Mystery 1 plots].
Closed-loop $\pi(a_t\mid s_t)$ under infinite data fails to even contact the target cube; open-loop 25-step chunked policy succeeds on the same task with the same data [Mystery 2 videos].
History-conditioned closed-loop policies achieve lower validation flow loss and MSE than their non-history counterparts but perform worse at task success [Mystery 2 plots].
State-based single-task pick-and-place: $[512]\times3$ MLPs underperform; $[4096]\times8$ residual MLPs learn the task well; $[8192]\times8$ further improves — approximately 0.5B parameters for a 37-dim state space and a fixed task [Mystery 3 sweep].
Standardized vs handcrafted-scale inputs yield essentially identical train-time flow loss and MSE but clearly different task success; further ×10 amplification of object + gripper $xyz$ under standardized features significantly outperforms baseline standardization or ×0.1 scaling [Mystery 4 plots].
All results are reproduced under state-based flow BC with no vision encoder or VLM backbone in the loop [Setup].

Why it’s interesting

This is a controlled-sim substrate for exactly the phenomena the VLA Models page has been tracking piecemeal across roughly a dozen recipes — and it delivers the sharpest single-source articulation yet of the why behind three recent findings. Park’s Mysteries 1–2 are the causal story that Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control? (Lazzati, Stachowicz, Chen, Metelli, Wagenmaker, Levine) formalized theoretically (non-Markovian expressivity + implicit ensembling) and that Revisiting Open-Loop Execution in Robotics: Toward Reactive, Higher-Performing Policies (Zeng, Ancha, Tedrake) sharpened into a prescription (long-context + short-chunk beats short-context + long-chunk as data scales) — the three papers agree that action chunking is compensating for missing non-Markovian expressivity plus test-time distribution shift, and Park now shows the same effect is reproducible even in state-based BC with no vision at all, which strengthens the Revisiting Open-Loop Execution in Robotics: Toward Reactive, Higher-Performing Policies and Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control? conclusions by ruling out perception as a confounder. Mystery 4 (feature scaling matters even with infinite data) is a concrete counterexample to a naive Bitter-Lesson reading and complements Simulation Pre-training for Dexterity (SPD) SPD’s w=32, c=8 finding — both argue that interface design around the BC policy (history window, coordinate frame, feature scaling) is load-bearing even in high-data regimes. Mystery 3’s “you need ~0.5B params for state-based single-task BC” is the most speculative claim on file but, if reproducible, would explain why the VLA action-expert component tracked on VLA Models cannot shrink even when perception + language are handled elsewhere. The MJWarp infinite-data benchmark itself, when released, will be the first shared controlled substrate for isolating BC/RL mysteries from real-world noise — the Slack thread reads this as an important complement to the real-hardware evaluation surfaces already on file.