Empirical results fine-tuning π0.5 on a real manufacturing task

ListenThe Deployment Lever Board: Fine-Tuning π0.5 in a Real Factory10:43

Show notes

A deep look at Dream Machines' eleven controlled fine-tuning experiments on Physical Intelligence's π0.5, run on a real bimanual pick-and-place task from a German manufacturer. We unpack why LoRA fell far short of full fine-tuning, why scene diversity and data quality beat raw quantity, how 240 intervention rollouts hit 88 percent, and how tuning only the serving scheduler lifted a fixed checkpoint from 76 to 93 percent. Along the way we draw parallels to data-pipeline and inference-time decisions in image and video generation. **Sources** - Empirical results fine-tuning π0.5 on a real manufacturing task — Dream Machines (blog)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Dream Machines documents 11 controlled fine-tuning experiments on π0.5 for a bimanual actuator-into-tray pick-and-place task from a German manufacturer (21 hours of teleop, 7,271 episodes, evaluated at n=40 per configuration with binomial CIs). Headline findings: LoRA at rank 64/256 falls 23–43 pp short of full fine-tuning even under Schulman’s all-linear + LR-multiplier protocol; scene diversity across five collection scenes beats homogeneous evaluation-scene data by 30 pp at matched 4h budget; a single clean 1h dataset lifts the 21h baseline from 76% → 90%; 240 DAgger-style intervention rollouts on a 1h seed policy reach 88% (vs 28% base); tuning only RTC async-serving parameters lifts the same checkpoint from 76% → 93% (and 90% → 98% on the 1h-fine-tuned model); and training from scratch (PaliGemma init only) produces plausible motions but 0/40 successful insertions. Every dataset is released on HuggingFace, and validation error was found essentially useless for checkpoint selection beyond an early plateau.

Key claims

LoRA on the VLA underperforms full fine-tuning by wide margins: rank-64 EBS-256 hits 45%, rank-64 EBS-64 hits 32.5%, rank-256 EBS-64 hits 52.5%, vs 76% for full FT — even when applied to all linear layers with a 10× LR multiplier per Schulman’s recipe [§2.2, Fig. 8]. Author hypothesis: adapting to a new embodiment absent from π0.5 pretraining is not the low-rank edit LoRA assumes.
LoRA is not even a compute win on their setup: at matched EBS=64 on 4×H100s, rank-64 LoRA runs 0.90 steps/s and rank-256 runs 0.65 vs 0.95 for full fine-tuning [§2.2].
Scene diversity dominates data quantity at fixed budget: a 4h dataset sampled across four desks + a lightbox + two operators beats a 4h dataset collected entirely in the evaluation lightbox by 30 pp [§2.7, Fig. 17].
Data quality dominates quantity at the top end: one additional clean hour (consistent motion, tray-edge-clearing insertions) fine-tuned on top of the 21h/50k baseline lifts success 76% → 90%; the same 1h alone from scratch reaches 28% within 5pp of the 4h lightbox-only model [§2.8, Fig. 19].
DAgger-style intervention is extremely sample-efficient: 240 intervention rollouts (212 with corrections) added to the 1h seed policy lift standard-eval success 28% → 88% and out-perform the 21h full-data model by 12 pp [§2.9, Fig. 20]. Diverse-position eval also lifts 11% → 48%.
Real-Time Chunking (RTC) serving parameters are load-bearing: at fixed 21h checkpoint, immediate reinference + blend=20 lifts success 76% → 93%; applied to the 21h+1h model it reaches 98% (1 failure in 40) [§2.10, Fig. 22]. Reinference frequency mattered less than blend length; more frequent reinference plateaus.
Training-time interpretation: the RTC gain and the clean-data gain both point to the same mechanism — with slower motions or shorter prediction horizons the arms are closer together when the handover chunk is predicted [§2.10].
Pretrained π0.5 weights are decisively load-bearing: initializing only the PaliGemma backbone yields plausible motions but 0/40 successful insertions, with 29/33 non-grasp failures occurring at the handover [§2.11, Fig. 25]. Contradicts the smaller pretrain-vs-scratch gap reported in the original π0 paper.
Sensitivity to learning rate (5e-5 default vs 1e-4) and batch size (128 vs 256 vs 512) is inside the ~±15pp n=40 noise floor [§2.3, Fig. 9, Fig. 11]. The lowest-training-loss run (1e-4) had the lowest success rate.
Absolute vs relative joint targets and LeRobot’s default image augmentation pipeline produced no measurable difference vs the baseline (73% and 75% vs 76%) [§2.4, Fig. 12].
Two-operator ablation: single-operator VR-teleop data outperforms single-operator Gello-leader-arms data by >30pp; the mixed model inherits the VR operator’s handover strategy rather than combining both [§2.5, Fig. 14]. Handover style depends on collection interface form factor.
Validation action error (L2 on held-out 5%) is useful only for spotting saturation, not checkpoint selection: OLS fit against rig success explains ~20% variance across all checkpoints and almost none once the two earliest are dropped [§2.1, Fig. 7]. Validation flow-matching loss actively rises during runs where success stays flat.
Success plateaus at ~30k steps on 21h of data at batch 256 with WSD schedule — further training and cosine-schedule considerations did not help [§2.1, Fig. 6].

Method

The team fine-tunes Physical Intelligence’s open-source π0.5 on bimanual TRLC-DK1 workstations with three fisheye cameras (2× wrist, 1× context, centre-cropped to 224×224 for SigLIP), controlled at 50 Hz with 1-second (50-step) action chunks. Baseline recipe: WSD learning-rate schedule (quadratic-anneal last 10k of 50k steps), batch size 256, max LR 5e-5, all-episode filtering to rating ≥3 (per an internal 1–5 quality rubric), evaluation via 40 rollouts (one per top-layer tray position) with 45s timeout. Every experiment varies one axis and evaluates the resulting checkpoint on the same rig. RTC serving controls “reinfer” (queue length triggering the next chunk) and “blend” (how many first actions of a new chunk are guided toward the previous chunk’s overlap); with 140ms inference latency, the first 7 actions of each new chunk are pre-committed regardless. The one-hour “clean” dataset was collected under strict constraints (consistent motion, insertions clearing tray edges), roughly doubling mean episode length. The intervention dataset uses only intervention segments plus full trajectories of the 28 fully-successful rollouts (out of 240). All ten training datasets and the model checkpoints are released on HuggingFace.

Results

Full fine-tune baseline (21h, 50k steps): 76% success (n=120 across 3 rounds).
210k-step WSD ladder: success plateaus at ~30k steps with strong oscillation thereafter [Fig. 6].
LoRA rank 64 EBS 256: 45% (5e-4 LR, 10× multiplier). LoRA rank 64 EBS 64: 32.5%. LoRA rank 256 EBS 64: 52.5% [Fig. 8].
Relative joint targets: 73%. Image augmentation: 75%. Both within noise of 76% baseline [Fig. 12].
Operator ablation (10h each): VR-teleop operator 1 > Gello operator 2 by >30 pp; mixed model tracks operator 1 [Fig. 14].
Data scaling (4h/10h/21h nested subsets): 4h ≈ 10h; 21h clearly better [Fig. 15]. But diversity beats scale: 4h across 5 scenes > 4h lightbox-only by 30 pp [Fig. 17].
Quality lever: 21h + 1h clean → 90% (from 76%). 1h clean from scratch → 28% ≈ 4h lightbox-only (25%) [Fig. 19].
DAgger: 1h base → 88% standard eval and 48% diverse-position eval, using only 1.7h total training data [Fig. 20].
RTC tuning on 21h/50k checkpoint: reinfer=50, blend=20 → 93%. Same settings on 21h+1h checkpoint → 98% (39/40) [Fig. 22]. Compilation reduces latency but did not help further.
Scratch training (PaliGemma init only, no π0.5 weights, 50k steps on 21h): 0/40 insertions; 7 no-grasp, 29 handover failures, 4 insertion failures [Fig. 25].

Why it’s interesting

This is the sharpest filed empirical characterization of the deployment lever board for π0.5 fine-tuning to date, complementing the RECAP recipe in π*0.6: a VLA That Learns From Experience (RECAP) with orthogonal data-side and serving-side knobs on a fixed base checkpoint. The LoRA-underperforms-full-FT-on-new-embodiment finding is a concrete counter-datapoint to LoRA Without Regret‘s “LoRA matches FullFT when applied to all linear layers”— extending the sub-1B-LM regime that paper stakes to a 3.35B VLA and a genuinely new embodiment, and consistent with the paper’s information-content-vs-rank threshold argument (adapting a new robot is a bigger edit than a small SFT dataset). The DAgger 240-rollout → 88% result reinforces the intervention-scaling thesis running through Enact launch — post-training infrastructure that generates targeted recovery data for robotics VLAs Enact, MicroFactory — 99.9% reliability via $5 human-in-the-loop DAgger retraining on Jetson (Ilir Aliu × Igor Kulakov podcast) MicroFactory, and Robo Robotics launch — Robo-T bimanual humanoid, sub-$10/hour with Roboport teleoperation platform (Kyle Noble) Robo Robotics — but with a per-experiment cost curve concrete enough to plan against. The RTC-serving lever result (single checkpoint, 76% → 93% from parameter change alone) sharpens the Real-Time Robot Execution with Masked Action Chunking REMAC and πR²: Reactive Real-time Flow Policies πR² line by showing how much headroom sits in the base serving scheduler before invoking training-time async fixes. And the 0/40 from-scratch result is the sharpest filed rebuttal of the original π0 paper’s claim that pretraining is only marginally load-bearing — likely reflecting the architectural distance between π0 (PaliGemma) and π0.5 (newer Gemma, 2× resolution, 2× larger diffusion head).