Skip to content
Bud's Wiki

Anchor-Align: Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment

TL;DR

Section titled “TL;DR”

Anchor-Align (UIUC + Meta FAIR; project page for arXiv:2607.13429) augments standard behavior-cloning VLA finetuning with two auxiliary objectives to preserve the pretrained VLM’s semantic and visual generalization: (1) Vision-Language Anchoring — a layer-wise representation-distillation loss against a frozen copy of the same VLM, computed on the robot observations themselves — and (2) Language-Action Alignment — the action target is discretized into a motion-direction word (e.g. “up”/“down”/“left”) and jointly trained as a language-prediction target on the same observation, so language and action losses share the robot input instead of being applied to different image-text batches (the co-training remedy). On a physical UFactory xArm7, both applied VLA backbones jump from 28→54% and 37→60% real-robot success, and Anchor-Align tops every LIBERO-PRO / LIBERO-Plus baseline including Knowledge Insulation (KI) and OpenVLA-OFT, with the biggest lift on LIBERO-PRO Position Swap (2.3 → 22.6% mean vs the strongest baseline).

Key claims

Section titled “Key claims”
  • Standard BC finetuning progressively overwrites the pretrained VLM’s visual/semantic representations, and standard co-training on separate image-text data does not fix it because the language and action losses land on different observations — a language-action misalignment that standard manipulation benchmarks do not expose [§overview].
  • Vision-Language Anchoring: keep a frozen copy of the original VLM and distill its layer-wise representations of the same robot observation into the trainable VLA, preventing representational drift during BC without requiring extra language data [§method overview].
  • Language-Action Alignment: convert each action target into a discrete motion-direction label and jointly train language and action prediction on the same robot observation, so the shared robot input carries both losses — a language-action alignment that co-training + KI-style objectives do not provide [§method overview].
  • Ablations attribute most of the gain to vision-language anchoring and report Anchor-Align beating a Knowledge-Insulation baseline “almost always” (per the sharer’s summary; the page’s Table 1 shows KI-style Co-training + KI at 43.8 LIBERO-PRO mean and 57.1 LIBERO-Plus mean vs Anchor-Align’s 71.9 and 90.3) [Table 1].
  • Real-robot xArm7 success on two VLA backbones: 28% → 54% and 37% → 60% [§overview].
  • LIBERO-PRO mean: 71.9 (Anchor-Align) vs 61.0 VLA-Adapter, 56.5 OpenVLA-OFT, 53.4 MolmoAct, 43.8 Co-training+KI [Table 1].
  • LIBERO-PRO Position Swap: 22.6 vs 2.3 (VLA-Adapter) and 0.0 for the other four baselines — a task regime where memorizing a training trajectory yields zero and only Anchor-Align meaningfully generalizes [Table 1].
  • LIBERO-Plus mean across seven independent OOD axes: 90.3 vs 85.1 VLA-Adapter, 74.1 OpenVLA-OFT, 60.8 MolmoAct, 59.9 VLA-Adapter [Frozen], 57.1 Co-training+KI [Table 1].
  • Real-world OOD evaluation covers four regimes that never appeared in training: Semantic Perturbation (pick pink mug after training only on green), Spatial Rearrangement, Cluttered Scene with distractors, and Compositional Object-Layout — every rollout tests a layout unseen during training [§real-world generalization].
  • Qualitatively, the Standard-BC baseline collapses to memorized motions in the Compositional Object-Layout regime and to its training-time green-mug prior on the Semantic Perturbation test, while Anchor-Align follows the current instruction and observation [§semantic perturbation, §compositional object-layout].

Method

Section titled “Method”

Anchor-Align keeps a standard VLA architecture (a pretrained VLM backbone + an attached action head — evaluated on two backbones, not specified in the project page body) and layers two auxiliary losses on top of vanilla BC:

  1. Vision-Language Anchoring: at training time, forward each robot observation through both (i) the trainable VLM being finetuned and (ii) a frozen copy of the same VLM. A layer-wise distillation loss aligns the two representations, so the finetuning gradient must trade off action prediction against staying close to the pretrained VLM’s own encoding of the same input. This is distinct from KI-style knowledge insulation, which typically freezes / gates sub-modules rather than distilling against a frozen reference on the robot observation.
  2. Language-Action Alignment: discretize each ground-truth action into a motion-direction word (“up”, “down”, “left”, “right”, “forward”, “back”, “close”, …) and add it to the target token sequence, so language prediction and action prediction share the exact same robot observation. This closes the observation-mismatch gap that co-training on separate image-text web data leaves open.

Real-world evaluation is on a UFactory xArm7 with four OOD regimes: Semantic Perturbation (train on green mug, test on pink mug via a new instruction), Spatial Rearrangement (relocated target/objects/distractors), Cluttered Scene (extra distractors, language-specified target), and Compositional Object-Layout (joint change of placement + orientation). Simulation evaluation is on LIBERO-PRO (Language Rephrasing / Object Swap / Position Swap) and LIBERO-Plus (Language Instruction / Background Texture / Robot Init / Camera View / Object Layout / Lighting / Sensor Noise).

Results

Section titled “Results”
  • Real-robot xArm7 (two backbones): 28% → 54% and 37% → 60% success after adding Anchor-Align [§overview].
  • LIBERO-PRO mean (higher is better): Anchor-Align 71.9 vs VLA-Adapter 61.0 / OpenVLA-OFT 56.5 / MolmoAct 53.4 / Co-training + KI 43.8 / VLA-Adapter [Frozen] 43.1 [Table 1].
  • LIBERO-PRO Position Swap (memorized-trajectory trap): Anchor-Align 22.6 vs VLA-Adapter 2.3, all other baselines 0.0 [Table 1].
  • LIBERO-PRO Language Rephrasing: Anchor-Align 97.0 vs VLA-Adapter 91.1 / OpenVLA-OFT 74.4 / MolmoAct 77.8 / Co-training+KI 54.0 [Table 1].
  • LIBERO-PRO Object Swap: Anchor-Align 96.2 vs 95.2 OpenVLA-OFT, 89.6 VLA-Adapter, 82.4 MolmoAct [Table 1].
  • LIBERO-Plus mean (seven independent OOD axes): Anchor-Align 90.3 vs VLA-Adapter 85.1 / OpenVLA-OFT 74.1 / MolmoAct 60.8 / VLA-Adapter [Frozen] 59.9 / Co-training+KI 57.1 [Table 1].
  • LIBERO-Plus per-axis wins: Background Texture 99.6 vs 95.7 (OpenVLA-OFT); Camera Viewpoint 96.3 vs 94.7; Lighting 99.0 vs 95.5; Sensor Noise 96.9 vs 89.5; Robot Init 59.1 vs 52.6; Object Layout 97.4 vs 93.2; Language Instruction 87.2 vs 85.1 [Table 1].
  • Qualitative failure attribution: Standard BC on Spatial Rearrangement erodes the VLM’s pretrained color grounding and can no longer distinguish mugs by color across scenes; on Compositional Object-Layout the baseline drifts toward a memorized motion trajectory that ignores the current object pose — both are named as representation-drift failure modes that anchoring specifically arrests [§spatial rearrangement, §compositional object-layout].

Why it’s interesting

Section titled “Why it’s interesting”

Anchor-Align stakes out a distinct row on the VLA Models recipe-lever board. Every representation-preservation recipe already on file addresses the drift-during-BC problem structurally — Harness VLA / RoboHarness (Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents, RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning) keep the VLA fully frozen and wrap it in an orchestrator; LFP (Learning What to Say to Your VLA: Mostly Harmless VLA Steering) freezes the VLA and steers via language rewrites; Action QFormer (Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models) inserts a structured query interface so action loss cannot broadly rewrite the shared multimodal pathway; Co-training + Knowledge Insulation (the KI baseline in Table 1) freezes/gates sub-modules and adds image-text data on separate observations. Anchor-Align is the first filed instance to keep the VLA fully trainable and pin its representation via on-observation distillation against a frozen self-copy — the reference observations are the robot observations themselves, so there is no language-action alignment gap for the co-training remedy to leave open. The paired language-action-alignment loss (discretize the action into a motion word, share the observation) is a small architectural cost that squares the recipe: language loss and action loss now co-supervise the same pixels the deployment policy will see.

The load-bearing empirical evidence sits on VLM Perception Failures: the Position-Swap regime is a memorized-trajectory trap where five of six baselines score exactly 0.0 and only Anchor-Align (22.6) and VLA-Adapter (2.3) score above the noise floor. That is the sharpest quantitative datapoint yet that BC finetuning of the current VLA generation is actively memorizing training trajectories rather than acting on the observation, and that on-observation representation-anchoring can restore observation-grounded behavior without giving up the ability to learn actions.

See also

Section titled “See also”