Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models

ListenMake Every Trajectory Teach More: VLAct and Representation-Centric Robot Pretraining10:32

Show notes

VLAct reframes vision-language-action continued pre-training as **representation learning**, not action fitting — freezing the vision encoder and lower LLM layers, training through three action heads that get thrown away, and unifying cross-embodiment action spaces at the loss level. Under a fixed robot-data budget on just 16 GPUs, it reports 82.6% on LIBERO-Plus Total, 92.5% Clean on RoboTwin, and beats 100%-data baselines using only 20% of fine-tuning data on a held-out embodiment. We dig into what's load-bearing, what the controlled comparison actually shows, and why the parallels to image and video model training are hard to miss. **Sources** - Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

VLAct (CUHK MMLab / SmartMore / HKU) reframes VLA continued pre-training as representation learning rather than action fitting: under a fixed robot-data budget, the goal is to convert limited trajectories into transferable visual-action knowledge while keeping the underlying VLM prior intact. Three principles instantiate this — Preserve (freeze the entire vision encoder and lower half of LLM layers during continued pre-training, plus a caption-supervision anchor; unfreeze everything for downstream fine-tuning), Diversify (train against OFT + PI + GR00T action heads jointly, then discard the pre-training heads and attach a freshly initialized task-specific head at fine-tune time), and Unify (physically aligned control dimensions share coordinates, embodiment-specific dimensions stay separate, inactive dims are masked, periodic joints use a wrap-aware loss). Continued pre-training runs on Franka data (DROID + MolmoAct) and AgileX data (InternData-A1 + RoboCoin) using 16 GPUs; fine-tuning is done on RoboCasa-GR1 (an embodiment held out from continued pre-training). Reports 82.6% on LIBERO-Plus Total (+7.6 pp over a matched Qwen3VL-OFT baseline), 92.5% Clean / 90.8% Randomized on RoboTwin in a Data Scaling setting, and exceeding the 100%-data baselines with only 20% of RoboCasa-GR1 fine-tuning trajectories. Preprint (Aug 2026) with no arXiv id yet.

Key claims

Continued pre-training under a fixed robot-data budget should be treated as representation learning, not action fitting — the goal is transferable visual-action knowledge preservation, not raw action fit on the pre-training corpus [§01].
Preserve: the entire vision encoder and the lower half of the LLM layers are frozen during continued pre-training, and caption supervision anchors broad visual/semantic knowledge; the full model is unfrozen again during downstream fine-tuning [§03].
Diversify: OFT, PI, and GR00T action heads jointly supervise the same shared backbone during continued pre-training; those pre-training heads are then discarded and a freshly initialized task-specific head is attached for fine-tuning [§03].
Unify: physically aligned control dimensions share coordinates across embodiments, embodiment-specific dimensions remain separate, inactive dimensions are masked, and periodic joints use a wrap-aware loss [§03].
LIBERO-Plus Total success 82.6%, +7.6 pp over a controlled Qwen3VL-OFT baseline with the downstream head, data, optimizer, and fine-tune budget held fixed; surpasses ABot-M0 in the reported comparison [§04, Total across seven perturbation axes].
Default VLAct-OFT under Data Scaling on RoboTwin: 92.5% Clean and 90.8% Randomized [§02].
With only 20% of RoboCasa-GR1 fine-tuning trajectories, VLAct exceeds the reported 100%-data baselines on the same benchmark (GR1 was held out from continued pre-training) [§02].
Cross-body real-hardware setup: stationary Franka Research 3 arms, one fixed external RealSense D435 + one wrist-mounted D405 per arm, 50 demos per single-arm task / 100 per dual-arm task, separate single-arm and dual-arm fine-tunes each for 50k steps on 8× H800, 10 rollouts per task [§05].
Mean success across four in-domain single-arm tasks: 92.5% vs 77.5% for the Qwen3VL-4B-OFT baseline without VLA continued pre-training [§05].
Continued pre-training uses only 16 GPUs on open datasets — the recipe is designed to be reproducible outside industrial-scale labs [§01 sidebar].
Released artifacts include a general-purpose continued-pretraining backbone (StarVLA/VLAct_Qwen3_Pretrain) plus four downstream-task checkpoints (RoboDojo/OFT, RoboTwin/GR00T, VLA-Arena/PI, LIBERO-Plus/PI). Different released cards use different action heads than the paper’s headline evaluations (e.g. RoboTwin 92.5% uses OFT; the released RoboTwin card uses GR00T) [§06].

Method

VLAct is a continued pre-training recipe on a Qwen3-VL backbone (the paper’s headline is VLAct-Qwen3VL-4B) rather than a new architecture. Under a fixed robot-data budget, the recipe partitions the model into a preserved substrate and a learned substrate: the vision encoder and the bottom half of the LLM layers stay frozen throughout continued pre-training, while the upper LLM layers plus three parallel action heads (OFT, PI, GR00T) are jointly trained on a heterogeneous robot corpus that mixes Franka (DROID + MolmoAct) and AgileX (InternData-A1 + RoboCoin) trajectories. Caption supervision from the underlying VL data is retained on the frozen half to anchor the pretrained VLM prior against representational drift.

A key structural choice: the three pre-training action heads are discarded at the end of continued pre-training. Their role is not to serve as the deployment policy but to force the shared backbone to learn a head-agnostic action-relevant representation. For downstream fine-tuning, a freshly initialized task-specific head (OFT, PI, or GR00T depending on the benchmark) is attached and the full model — including the previously frozen halves — is unfrozen.

The Unify component handles the cross-embodiment action-space problem at the loss level rather than through a separate retargeting stage: physically corresponding dimensions across embodiments (e.g. shared 6-DoF end-effector coordinates) are placed in the same coordinate slots so the same output neurons see aligned targets across Franka and AgileX data; embodiment-specific dimensions remain in separate slots; inactive dimensions are masked out of the loss; and periodic joints use a wrap-aware loss to avoid discontinuities.

Continued pre-training runs on 16 GPUs — small enough that the recipe is claimed to be reproducible without industrial-scale compute. Downstream fine-tuning on the held-out GR1 embodiment uses a standard head-specific loop.

Results

LIBERO-Plus Total: 82.6%, +7.6 pp over the controlled Qwen3VL-OFT baseline with all downstream variables matched; surpasses ABot-M0 in the reported comparison [§04].
RoboTwin (Data Scaling setting, VLAct-OFT default): 92.5% Clean success, 90.8% Randomized [§02].
Data-efficiency crossover: 20% of RoboCasa-GR1 fine-tuning trajectories is enough to exceed the 100%-data baselines shown in the paper’s figures — an especially stringent test because GR1 is not an embodiment seen during continued pre-training [§02].
Real-hardware Franka single-arm (in-domain, 4 tasks × 10 rollouts): 92.5% mean success vs 77.5% for a Qwen3VL-4B-OFT baseline without VLA continued pre-training [§05].
Dual-arm and long-horizon tasks: 11 VLAct rollout videos plus 2 matched baseline videos released on the project page (no single aggregate number). Separate models are fine-tuned for single-arm and dual-arm evaluation, each for 50k steps on 8× H800.
Compute: continued pre-training on 16 GPUs against open datasets only [§01 sidebar].
Released checkpoints: backbone StarVLA/VLAct_Qwen3_Pretrain, plus four fine-tuned cards on RoboDojo (OFT, 100k steps), RoboTwin (GR00T, 50k steps), VLA-Arena (PI), and LIBERO-Plus (PI). Reported RoboDojo leaderboard-official numbers are 7.60% success / 10.66 score (50-episode-per-task snapshot); the repo’s local scaled evaluation reports different numbers [§06].
No arXiv id at filing time — preprint URL is the project page; a placeholder BibTeX entry is provided by the authors [§07].

Why it’s interesting

VLAct is a direct sibling to Anchor-Align: Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment on the VLA Models recipe-lever board: both target the observation that BC-style action learning on top of a pretrained VLM tends to erode the VLM’s pretrained visual/semantic priors, but they intervene at different points in the pipeline. Anchor-Align is the fine-tuning-time, loss-side answer — keep the model fully trainable and pin its representation via on-observation distillation against a frozen self-copy. VLAct is the continued-pretraining-time, freezing + diversification-side answer — freeze the vision encoder and lower LLM layers outright, keep caption supervision anchored on the frozen half, and force the trainable half to learn a head-agnostic representation by joint supervision through three different action heads that are then thrown away. Both report largest gains in the memorized-trajectory / OOD-generalization regimes (Anchor-Align on LIBERO-PRO Position Swap, VLAct on LIBERO-Plus Total’s seven-axis perturbation), which is consistent with the shared thesis that BC’s failure mode is representational drift rather than raw capacity.

The other close comparison on the board is OpenDM — DM0.5: An Open-World Foundation Model for General-Purpose Embodied Intelligence DM0.5, which likewise targets “keep VL capability strong while learning to act” — but through a separate-learning-rate group on the VLM backbone plus an internal automatic VL data generation pipeline. VLAct’s contribution here is that the anchoring signal is architectural (freeze the substrate) plus a supervision-diversification mechanism (three action heads at pre-train, discarded at fine-tune) rather than an optimizer knob or an extra data pipeline. The head-diversification move is also novel on the board: no other filed recipe uses “train through multiple heads to force head-agnostic representation, then throw the heads away” as an explicit anti-overfitting device.

The 16-GPU claim, if it holds, matters for a Luma-adjacent open question — reproducibility of VLA recipes outside frontier compute — and the paper’s controlled comparison (Qwen3VL-OFT with matched downstream head/data/optimizer) is the kind of ablation that is missing from most VLA papers filed this year.