Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

TL;DR

The paper attacks a problem orthogonal to the VLA-training debate that has dominated the wiki: a frozen pretrained VLA is hard to steer reliably via language because semantically similar instructions induce drastically different behaviors, and some capabilities cannot be elicited from prompting at all. The authors propose a three-stage recipe — interactively search for language sequences that improve closed-loop task success, distill those sequences into a test-time language feedback policy (LFP) that suggests instruction rewrites at runtime, and train an improvement head that predicts whether intervening on a given rollout will actually help. The improvement head is then conformalized so that, on out-of-distribution scenarios, the LFP is provably unlikely to make performance worse than the base instruction — a “mostly harmless” safety guarantee on top of a closed-loop steering loop.

Key claims

The framework operates on arbitrary frozen pretrained VLAs without access to the original training distribution or any fine-tuning of the underlying model [Abstract].
On seen environments, the conformalized LFP improves base VLA performance by 24.7% in simulation and 65.0% on hardware [Abstract].
On visual and semantic perturbations (OOD scenarios), the conformalized LFP has strong harmlessness guarantees and produces recovery behaviors not observed with open-loop prompting [Abstract].
Both human-written instructions and zero-shot LLM-generated instructions fail to reliably steer VLAs toward task success, motivating the need to learn the steering policy rather than rely on prompt engineering [Abstract].

Method

The pipeline has three stages. First, an interactive search over language sequences (presumably guided by closed-loop rollouts and a task-success signal) collects instruction sequences that demonstrably improve task performance for the frozen VLA. Second, these sequences are distilled into a language feedback policy — a model that, at test time, observes the current rollout and emits language feedback / instruction rewrites to the VLA. Third, an improvement head is trained to predict whether applying the LFP in a given state will actually improve performance over the original instruction; this head is then conformalized (in the conformal-prediction sense) so that its abstention behavior has finite-sample guarantees on OOD inputs, preventing harmful interventions where steering would decrease task success.

Results

Simulation: +24.7% absolute improvement in task success on seen environments with the conformalized LFP vs. base VLA on the original instruction [Abstract].
Hardware: +65.0% absolute improvement on real-robot seen environments [Abstract].
OOD robustness: conformalized LFP produces recovery behaviors — closed-loop course corrections — not seen under open-loop prompting, while the conformal harmlessness guarantee bounds the risk that steering hurts performance under visual/semantic distribution shift [Abstract].

Why it’s interesting

This is a genuinely new axis in the VLA-design space the wiki has been tracking. Every other VLA paper filed so far argues for a recipe that changes the model — RECAP advantage-conditions a flow-matching action expert (π*0.6: a VLA That Learns From Experience (RECAP)), Embodied-R1.5 substitutes pointing/grounding pretraining for action pretraining (Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models), µ₀ factors out a frozen action-free 3D trace world model (μ₀: A Scalable 3D Interaction-Trace World Model). This paper instead leaves the VLA frozen and treats the instruction stream as the optimization variable, which is the closest VLA-side analog to test-time scaffolding tricks the wiki tracks under Inference-Time Scaling. The structural sibling on the wiki is VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation — VoLo also wraps a frozen VLA inside a VLM-driven orchestration loop — but VoLo’s orchestrator is a hand-designed prompted controller, whereas this paper learns the steering policy and conformalizes its abstention behavior, which is a meaningfully sharper safety story.