Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

TL;DR

Latent Interface Training (LIT) is a framework-agnostic two-stage strategy for training VLA and world-action-model action experts that mitigates vision-action shortcuts — spurious correlations between task-irrelevant visual cues (background, lighting, camera angle, distractors) and demonstrated actions that undermine generalization when scene conditions shift. Stage 1 trains the action expert to generate action chunks conditioned only on language, robot state, and each demonstrated chunk’s terminal SE(3) end-effector pose — no images — learning goal-directed action generation with no opportunity to pick up visual shortcuts. Stage 2 introduces a latent interface that aggregates visual and semantic representations and serves as the pretrained action expert’s only visual conditioning pathway; the interface is supervised to reconstruct the terminal pose used to condition Stage 1, encouraging it to retain only goal-relevant spatial information. Across four VLA/WAM architectures (π0.5, MolmoAct2, FAST-WAM, ImageWAM), LIT improves LIBERO-Plus success by 3.87–10.70 pp while preserving or improving average LIBERO; real-world evaluations show 13.30–16.70 pp gains aggregated across three tasks under unseen camera configurations, lighting variations, and distractors.

Key claims

Robot foundation models achieve strong in-distribution performance but degrade under visual distribution shifts because the action expert exploits task-irrelevant visual cues that correlate with demonstrated actions in training [Abstract].
Mitigating these shortcuts requires constraining how visual information reaches action generation while preserving task-relevant spatial information — a training-time restructuring rather than a loss-side or freeze-and-wrap patch [Abstract].
Stage 1 of LIT trains the action expert without images: conditioning on language, robot state, and the terminal SE(3) end-effector pose of each demonstrated chunk is sufficient to learn goal-directed action generation [Abstract].
Stage 2 introduces a latent interface that aggregates visual and semantic representations and serves as the pretrained action expert’s only visual conditioning pathway, supervised to reconstruct the Stage 1 terminal pose [Abstract].
LIT improves overall LIBERO-Plus success by 3.87–10.70 percentage points across four backbone architectures (π0.5, MolmoAct2, FAST-WAM, ImageWAM) while preserving or improving average LIBERO success [Abstract].
Real-world evaluations show 13.30–16.70 percentage-point gains in success aggregated across three manipulation tasks under unseen camera configurations, lighting variations, and distractors [Abstract].

Method

LIT restructures how the action expert receives conditioning across two training stages, framework-agnostic across four evaluated backbones (π0.5, MolmoAct2, FAST-WAM, ImageWAM).

Stage 1 — image-free action pretraining. The action expert is trained on language + robot state + a spatial goal (each demonstrated chunk’s terminal SE(3) end-effector pose) with no image input. Because the action expert never sees pixels in this stage, it cannot pick up correlations between background, lighting, or camera angle and actions; a spatial goal is treated as sufficient to teach goal-directed motion.

Stage 2 — latent interface for vision. A latent interface module aggregates visual and semantic representations and is trained to reconstruct the terminal pose that conditioned Stage 1. The action expert then consumes the latent interface’s output rather than raw visual features, so any visual information reaching the action expert must first pass through a bottleneck explicitly supervised to be goal-relevant. Under scene shift, the interface’s output stays close to the same recovered goal, so the action expert’s predictions do too.

Results

LIBERO-Plus average success (four backbones): 3.87–10.70 percentage-point improvements across π0.5, MolmoAct2, FAST-WAM, and ImageWAM [Abstract]. Per-backbone numbers reported by co-author announcement thread: π0.5 68.97 → 79.67% (+10.7 pp); MolmoAct2 63.62 → 71.92% (+8.3 pp); FAST-WAM 51.44 → 60.63% (+9.2 pp); ImageWAM 83.02 → 86.89% (+3.9 pp).
In-distribution LIBERO average: preserved or improved across all four backbones — the OOD fix does not cost ID performance [Abstract].
Real-world manipulation (three tasks, MolmoAct2 backbone, 300 demos): 13.30–16.70 pp gains aggregated across unseen camera configurations, lighting variations, and distractors [Abstract]. Reported per-condition: new lighting 53.3 → 70.0% (+16.7 pp); top-camera-only 30.0 → 46.7% (+16.7 pp); distractors 50.0 → 63.3% (+13.3 pp); in-distribution 74.7 → 88.0% (+13.3 pp).
Framework-agnostic across two families (VLA: π0.5, MolmoAct2; WAM: FAST-WAM, ImageWAM), gains widest at low-data regimes and largest on perturbation axes that most stress visual shortcuts (lighting, viewpoint, distractors).

Why it’s interesting

LIT stakes out a training-time architectural row on the VLA Models recipe-lever board that is orthogonal to the loss-side and freeze-and-wrap positions already filed. The load-bearing move is inverting the standard “give the action expert everything jointly” recipe: image-free pretraining first with a spatial goal, then a supervised latent bottleneck for vision. Structurally distinct from Anchor-Align: Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment Anchor-Align, which keeps joint training but adds a distillation loss against a frozen VLM copy on the same robot observation — Anchor-Align pins the VLM’s representation, LIT restructures when and how vision enters the action expert. Both attack the same underlying problem (BC training on joint observations induces representation shortcuts) at different layers of the stack.

The four-backbone evaluation grid is where LIT’s evidence is unusually clean — it lifts π0.5, MolmoAct2, FAST-WAM, and ImageWAM simultaneously under matched LIBERO-Plus protocol, suggesting the training-time restructuring is not backbone-specific. The real-world MolmoAct2 numbers are the sharpest datapoint: all three perturbation axes (lighting, top-camera-only, distractors) individually gain +13–17 pp and in-distribution performance also jumps +13.3 pp — the visual-shortcut fix buys both ID and OOD rather than trading them.