LIT — two-stage VLA/WAM training constrains visual conditioning via latent interface to prevent visual shortcuts

TL;DR

LIT (Latent Interface Training) is a two-stage recipe for training VLA and WAM action experts that argues standard joint vision+language+state training encourages the action expert to rely on visual shortcuts — spurious correlations between backgrounds, camera angles, or lighting and demonstrated actions. Stage 1 trains the action expert without images, on language + robot state + spatial goal only, so it learns goal-directed motion from information that cannot carry shortcuts. Stage 2 reintroduces vision through a latent interface trained to recover the spatial goal, so visual information enters only through a goal-relevant bottleneck rather than through the full action expert’s input. Applied to four backbones (π0.5, MolmoAct2, FAST-WAM, ImageWAM), LIT improves average LIBERO-Plus success by 3.9–10.7 pp and preserves in-distribution LIBERO gains; on a real-world MolmoAct2 with 300 demos across three tasks, LIT lifts new-lighting success from 53.3% → 70.0%, top-camera-only from 30.0% → 46.7%, distractors from 50.0% → 63.3%, and in-distribution from 74.7% → 88.0%. Attribution analysis: adding distractors or blurring the image leaves LIT’s predicted trajectories close to their originals while baselines shift substantially; changing the instructed goal produces the expected trajectory change.

Key claims

Standard VLA/WAM training that feeds language + robot state + images jointly to the action expert encourages reliance on visual shortcuts — background, camera angle, or lighting correlations with actions that break under scene shift [§thread 1/].
LIT restructures training into two stages: (1) train the action expert without images using language + robot state + spatial goal, (2) introduce vision through a latent interface trained to recover that spatial goal [§thread 2/, 3/].
Rationale: giving the action expert a spatial goal and training it to move there without images teaches goal-directed motion without any opportunity to pick up visual shortcuts; the latent interface added afterward is trained explicitly to recover a goal-relevant signal from images rather than to carry arbitrary visual information into the action expert [§thread 3/].
Cross-architecture LIBERO-Plus average success improvements: π0.5 68.97 → 79.67%, MolmoAct2 63.62 → 71.92%, FAST-WAM 51.44 → 60.63%, ImageWAM 83.02 → 86.89% [§thread 4/].
In-distribution LIBERO average also improves for all four backbones — the visual-shortcut fix does not cost in-distribution performance [§thread 4/].
Real-world MolmoAct2 (300 demos across three tasks), MolmoAct2 → MolmoAct2+LIT: new lighting 53.3 → 70.0%, top-camera-only 30.0 → 46.7%, distractors 50.0 → 63.3%, in-distribution 74.7 → 88.0% [§thread 5/].
Attribution: adding a distractor or blurring the image leaves LIT’s predicted trajectories close to their original paths while the baseline’s shift substantially; changing the instructed goal produces the expected trajectory change [§thread 6/].

Method

The thread describes LIT (Latent Interface Training) as a two-stage restructuring of how the action expert receives conditioning:

Stage 1 — image-free action pretraining. The action expert is trained on language + robot state + a spatial goal only, with no image input. The claim is that a spatial goal is sufficient to teach goal-directed motion, and because the action expert never sees pixels in this stage, it cannot pick up correlations between background/lighting/camera angle and actions.

Stage 2 — latent interface for vision. A latent interface module is trained to recover the spatial goal from images. The action expert consumes the latent interface’s output rather than raw visual features, so any visual information reaching the action expert must first pass through a bottleneck explicitly supervised to be goal-relevant. Under scene shift (new lighting, distractors, camera-view change), the interface’s output stays close to the same recovered goal, so the action expert’s predictions do too.

The thread does not disclose interface architecture, spatial-goal representation format, or per-stage training data specifics; those await the paper. Method is validated as an add-on to four existing action-expert-bearing architectures (π0.5, MolmoAct2, FAST-WAM, ImageWAM), and the same LIT weights are evaluated across LIBERO-Plus perturbation axes and on real MolmoAct2 hardware.

Results

LIBERO-Plus average success (four backbones): π0.5 68.97 → 79.67% (+10.7 pp); MolmoAct2 63.62 → 71.92% (+8.3 pp); FAST-WAM 51.44 → 60.63% (+9.2 pp); ImageWAM 83.02 → 86.89% (+3.9 pp) [§thread 4/].
In-distribution LIBERO average: improves for all four backbones (numbers not reported in the thread) — the OOD fix does not cost ID performance [§thread 4/].
Real-world MolmoAct2 (300 demos, three tasks), MolmoAct2 → MolmoAct2+LIT:
- New lighting: 53.3 → 70.0% (+16.7 pp)
- Top-camera-only: 30.0 → 46.7% (+16.7 pp)
- Distractors: 50.0 → 63.3% (+13.3 pp)
- In-distribution: 74.7 → 88.0% (+13.3 pp) [§thread 5/].
Attribution analysis: adding a distractor or blurring the image leaves LIT’s predicted trajectories close to their original paths while the baseline’s shift substantially; changing the instructed goal produces the expected trajectory change — the intended causal signature of “vision only enters through a goal-relevant bottleneck” [§thread 6/].
Author notes: gains are widest at low-data regimes and largest on the perturbation axes that most stress visual shortcuts (lighting, viewpoint, distractors); underlying paper not yet linked from the thread.

Why it’s interesting

LIT stakes out a training-time architectural row on the VLA Models recipe-lever board that is orthogonal to the loss-side and freeze-and-wrap positions already on file. The load-bearing move is inverting the standard “give the action expert everything jointly” recipe: image-free pretraining first with a spatial goal, then a supervised latent bottleneck for vision. Structurally distinct from Anchor-Align: Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment Anchor-Align, which keeps the joint training but adds a distillation loss against a frozen VLM copy on the same robot observation to prevent representational drift — Anchor-Align pins the VLM’s representation, LIT restructures when and how vision enters the action expert. Both are answers to the same underlying problem (BC training on joint observations induces representation shortcuts / memorization) but at different layers of the stack; the four-backbone evaluation grid is where LIT’s evidence is unusually clean — it lifts π0.5, MolmoAct2, FAST-WAM, and ImageWAM simultaneously, all under matched LIBERO-Plus protocol, suggesting the training-time restructuring is not backbone-specific.

The real-world MolmoAct2 numbers are the sharpest datapoint. All three perturbation axes (lighting, top-camera-only, distractors) individually gain +13–17 pp, and in-distribution performance also jumps +13.3 pp — the visual-shortcut fix is not trading ID for OOD, it is buying both. Combined with the trajectory-attribution result (distractors and blur leave LIT’s paths near-original while shifting the baseline’s), this reads as the sharpest filed evidence that the current VLA generation’s action experts are picking up spurious visual correlations even in short-horizon single-task real-world settings — and that a two-stage training restructuring can attack the problem at the source rather than after the fact.