Dominique Paul — two months of π0.5 finetune ablations on a real manufacturing task, 63% → 98%

TL;DR

Dominique Paul (founder of Dream Machines / @dm_robots) summarizes two months of π0.5 finetuning ablations on a single real manufacturing task. Reported ranking of levers, in order of marginal lift per hour of demonstration: inference settings > data quality > data diversity > raw data quantity. Headline numbers: 5× more homogeneous data lifts task success 63% → 76%; a single clean +1h on top of the 21h corpus lifts it 76% → 90%; and inference-side settings alone push the 21h+1h model to 98% with no retraining. Learning rate, batch size, relative-vs-absolute joint targets, and image augmentation are called out as less important than expected.

Key claims

Scaling homogeneous data is the weakest lever they found. 5× more demonstrations of the same task in the same scene raised success 63% → 76% [tweet §1].
Scene diversity beats scene concentration at matched hours. 4 h collected across five scenes + two workstations outperformed 4 h collected entirely in the target eval lightbox by ~30 pp [tweet §2, cross-referenced from the same author’s follow-up thread indexing this result on the Dream Machines actuator-unboxing task].
Quality beats quantity by a wide margin at their scale. One clean, slower hour added on top of the 21h corpus lifted success 76% → 90% — more marginal gain than the 17 hours before it [tweet §3].
Quality-only recipe outperforms quantity-only recipe. 1 h clean data + 240 human-intervention (DAgger-style) rollouts, 1.7 h total, took the policy 28% → 88% — 12 points above the 21h model at 1/12 the data [tweet §4].
Inference-time settings add substantial success with no retraining. They lifted the 21h model 76% → 93% and the 21h+1h model 90% → 98% [tweet §5].
Learning rate, batch size, relative vs. absolute joint targets, and image augmentation mattered less than expected across the ablation sweep [tweet, closing paragraph].

Method

The tweet reports the results-side of a two-month structured ablation program on π0.5 finetunes for a single real-world manufacturing task (actuator unboxing, per the author’s related tweets) run in a lightbox eval environment. The reference corpus is 21 hours of demonstration data. The comparison design isolates one lever at a time: raw hour count (holding scene and quality constant), scene/workstation diversity (holding hours constant), demonstration quality (adding a small clean subset on top of the base corpus), quality-in-isolation (small clean set + human-in-the-loop DAgger interventions instead of scaling passive teleop), and a final inference-side sweep applied to the trained checkpoints without further training. No architectural changes; the base model is π0.5 throughout. The tweet points at a full write-up “linked below” but does not include the write-up URL in the extracted content — flagged for /bud refresh once the write-up URL is recovered.

Results

Raw data scaling: 63% → 76% at 5× data in the same scene [tweet §1].
Diversity: ~30 pp absolute lift from 4 h spread across five scenes + two workstations vs 4 h in the target eval scene alone [tweet §2].
Quality on top of 21 h: +14 pp (76% → 90%) from a single clean hour, more than the preceding 17 h combined [tweet §3].
Quality-alone recipe (1 h clean + 240 human-intervention rollouts, 1.7 h total): 28% → 88%, i.e. +12 pp above the 21 h model at 1/12 the data [tweet §4].
Inference-side settings (no retraining): 21 h: 76% → 93%; 21 h + 1 h clean: 90% → 98% [tweet §5].
Explicitly “did not matter much”: learning rate, batch size, relative vs. absolute joints, image augmentation [tweet, closing paragraph].

Why it’s interesting

This is a rare decision-usable real-hardware ablation on the concrete recipe levers the VLA Models page has been enumerating from a distance, and it lands on the data-quality row rather than the data-scale, RL post-training, or unified-VLM rows. It matches and quantifies the Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models Spirit-v1.5 position (“clean teleop is the decisive lever”) on a very different substrate — one manufacturing task, one operator, no proprietary base model swap — and it supplies the operational grounding for the earlier What Matters in Learning from Large-Scale Datasets for Robot Manipulation MimicLabs finding that targeted alignment (camera-pose alignment + spatial diversity) beats naive corpus scaling. The inference-side lift (+7 pp and +8 pp on already-strong policies with zero retraining) is a datapoint the concept page did not previously have quantified for a real deployment. The “less important than we expected” list is unusual candor about the Hyperparameter scaling laws surface for VLAs at this scale: LR, batch size, action-space (rel vs abs), and image augmentation were all first-order suspects and none of them dominated. Ishaan’s Slack note (“helpful for demomaxxing”) captures the pragmatic value: this is a working recipe-lever priority list for anyone shipping a π0.5 finetune to a customer demo.