Full-Weight VLA Fine-Tuning Without an fp32 Master Copy Silently Erases Most Updates

TL;DR

A PSA-style technical note identifying a widespread numerical pathology in open VLA fine-tuning: when trainable weights are stored in bfloat16 with no fp32 master copy, AdamW writes updates directly into bf16, silently rounding away any update smaller than ~1/512 of a weight’s magnitude. A probe on LIBERO runs finds at least 79% of per-step updates to bf16 weights erased. The paper surveys 12 open VLA codebases and reports 9 train in the broken configuration by default — including LeRobot’s SmolVLA recipe and the π0/π0.5 guides. The one-line fix (keep trainable weights in fp32 during training) lifts SmolVLA on LIBERO-Object from 4.3% → 49.3% and π0.5 from 89.0% → 97.7%; on seven controlled comparisons every fp32-copy run beats every no-copy run (p < 0.05).

Key claims

bf16-stored trainable weights under AdamW silently round away any update smaller than ~1/512 of the weight’s magnitude; a per-100-step probe on LIBERO measures ≥79% of updates to bf16 weights erased [Abstract, §probe].
Nine of twelve surveyed open VLA codebases train in the broken no-fp32-master-copy regime by default or by documented guide, four of them on full weight matrices rather than just adapters [Abstract].
On the Hugging Face Hub, 99% of sampled full-weight SmolVLA fine-tunes store the action expert in bf16, and 84% of recent π₀.₅ fine-tunes with a published training configuration have the bf16 flag set [Abstract].
Adding an fp32 master copy lifts LIBERO-Object success for SmolVLA 4.3% → 49.3% and for π₀.₅ (at 10× lower LR, where both models are undertrained) 5.1% → 80.9% [Abstract].
At π₀.₅’s default LR, the fp32 copy lifts LIBERO-Object 89.0% → 97.7% [Abstract].
For SmolVLA at its default rate, the gain is sensitive to execution horizon: fp32 adds +10.2 points with open-loop 50-action chunks but only +4.2 when re-planning every 10 actions [Abstract].
Across seven controlled comparisons (five on SmolVLA with every trainable weight in bf16, plus the π₀.₅ pair), every fp32-copy run beats every no-copy run at p < 0.05; all gaps except the 8.7- and 4.2-point ones clear a pre-registered 10-point threshold [Abstract].
The fix is a one-line change for SmolVLA inside LeRobot: keep the trainable weights in fp32 during training (AdamW master copy), without altering the activation/compute precision [Abstract].

Method

The paper is a short diagnostic + intervention note rather than a new algorithm. The underlying numerical claim is standard: bf16 has 8 bits of mantissa (ULP ≈ w · 2⁻⁸ ≈ w/256), so a per-step update Δw with |Δw| ≲ w/512 round-trips to zero when written back into the bf16 weight. The intervention is the textbook mixed-precision recipe: keep the AdamW master copy in fp32, perform the update in fp32, round only when materializing weights for the forward/backward (activations and compute may remain in bf16).

The diagnostic is two-pronged. (1) An erasure probe instruments AdamW inside a LIBERO fine-tuning run: every 100 steps it inspects per-parameter proposed updates vs. realized weight deltas and reports the fraction silently rounded to zero — ≥79% on bf16 weights. (2) A codebase + Hub audit: 12 open VLA codebases are read for default precision choices, and the paper surveys sampled SmolVLA (99% bf16 action expert) and π₀.₅ (84% bf16 flag) fine-tunes on the Hub.

The empirical half runs seven matched comparisons on LIBERO-Object across SmolVLA (default + lowered-LR variants, with the horizon-10 vs horizon-50 chunking split as a secondary axis) and π₀.₅ (default LR and 10× lower). All runs match seed, data, schedule, and compute other than the fp32-vs-no-fp32 master-copy flag.

Results

Erasure probe: ≥79% of per-step updates to bf16 weights erased every 100 steps on LIBERO fine-tuning runs [Abstract].
SmolVLA on LIBERO-Object: 4.3% → 49.3% with fp32 master copy at matched LR [Abstract].
π₀.₅ on LIBERO-Object at 10× lower LR (both undertrained): 5.1% → 80.9% [Abstract].
π₀.₅ on LIBERO-Object at default LR: 89.0% → 97.7% [Abstract].
SmolVLA at default LR: +10.2 points with 50-action open-loop chunks, +4.2 with re-planning every 10 actions [Abstract].
Statistical pattern: seven controlled comparisons, every fp32 run beats every no-fp32 run (p < 0.05); gaps except 8.7 and 4.2 pts clear a pre-registered 10-pt threshold [Abstract].
Prevalence: 9/12 open codebases default-broken, 99% of sampled Hub SmolVLA fine-tunes bf16 action expert, 84% of π₀.₅ fine-tunes with bf16 flag [Abstract].

Why it’s interesting

The paper lands in the gap between two concepts the wiki already tracks and sharpens a shared assumption behind both. On the VLA side, VLA Models‘s recipe-lever board catalogues ~50 distinct levers (clean teleop, action-pretraining, unified-VLM pointing, frozen-WFM+action-expert, flow-matching RL, LoRA-vs-full-FT) — and this note argues at least some of the SmolVLA- and π₀.₅-based entries (Empirical results fine-tuning π0.5 on a real manufacturing task, Empirical results fine-tuning π0.5 on a real manufacturing task‘s LoRA-vs-FullFT split, and most Hub-sampled fine-tunes) have been operating in a regime where ≥79% of the optimizer’s actual signal was being silently discarded. The π₀.₅ 89% → 97.7% at the default LR is the most alarming datapoint: a configuration that already looks “working” was leaving 8-9 points of task success on the table.

It complements Training stability at scale‘s low-precision-locus thread (CroissAttn / FP16-fixes-RL-mismatch / NVFP4 RL / fp32-RMSNorm defaults) by extending the “where does precision silently break modern training” question from LLM pretraining into the VLA fine-tuning regime, where the dominant practice is single-run short-horizon post-training on a small dataset — exactly the regime where a 1/512 ULP threshold truncates most meaningful updates. The LoRA-vs-FullFT gap Empirical results fine-tuning π0.5 on a real manufacturing task reports (LoRA 45% vs FullFT 76% on π₀.₅) now has a confound to check: were both runs using an fp32 master copy, or was the FullFT baseline silently handicapped and the LoRA adapters (typically trained in fp32 by default under peft) escaping the pathology?

See also Parameter-Efficient Finetuning on LoRA-vs-FullFT comparisons that would need re-running under matched fp32-master-copy flags to be interpretable.