RL²-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

TL;DR

RL²-VLA is an adaptive inference-time steering framework that trains a lightweight offline-RL flow-matching policy on the latents of a frozen base VLA’s action expert, then composes the two policies’ flow velocities at inference. The load-bearing empirical finding is that test-time steering follows fundamentally different scaling laws under predicted-success vs predicted-failure states — action diversity helps when the base VLA is likely to fail but perturbs already-accurate actions when success is likely — so RL² activates compositional steering only when failure is predicted. Reports up to +17.3 pp OOD gain on SIMPLER and PolaRiS, +26.7 pp on real hardware over the strongest Rephrase baseline, and adds up to +18.6 pp by combining 8 rephrases × 5 samples with compositional steering vs single-sample inference.

Key claims

• Existing test-time steering / scaling methods for VLAs leave action samples concentrated around similar behaviors and inherit correlated failure modes; existing methods also apply the same intervention strategy at every timestep regardless of whether the base policy is already likely to succeed [Abstract].
• Latents extracted from the base VLA’s action expert are expressive enough to condition a separately trained lightweight offline-RL flow-matching policy — no fine-tuning of the base VLA is required [Abstract].
• Compositional steering combines two policies at inference by composing action-expert flow velocities rather than reranking samples or selecting one policy over the other; this is the paper’s proposed way to combine the behavioral prior of large-scale imitation with the action diversity induced by offline RL [Abstract, project page “Overall architecture”].
• Test-time steering follows fundamentally different scaling laws in success vs failure states: on the BridgeV2 dataset with π₀ + an oracle verifier, action error e ≈ a·k^b improves with sample count k in the failure regime and degrades / stays flat in the success regime — so uniform steering can hurt on trajectories the base VLA would have handled [project page “Test-Time Scaling analysis”].
• The failure/success gating primitive is a conformal-prediction band around a failure-probability predictor; RL² triggers compositional steering only when the predicted failure probability crosses the CP band [project page “Failure Detection” panel].
• On the SIMPLER and PolaRiS benchmarks, RL² improves success rates by up to +17.3 pp in out-of-domain settings vs prior test-time steering baselines [Abstract].
• Real-world experiments show +26.7 pp on-robot over the strongest Rephrase baseline on OOD prompts and environments [project page].
• Scaling sample diversity and quantity (8 rephrases × 5 samples) with compositional steering improves success rates by up to +18.6 pp over single-sample inference [project page].
• Ablations show latent representations from the action expert and offline RL training on those latents are both load-bearing for the reported gains [Abstract, “importance of latent representations and RL training”].

Method

RL² keeps the base VLA (e.g. π₀) frozen and trains a second, lightweight flow-matching policy with offline RL. The second policy is conditioned on latents extracted from the base VLA’s action expert — the intermediate activations that the base VLA would already compute during a normal forward pass, so no extra base-model inference is required. Training data for the offline RL policy is presumably drawn from rollouts of the base VLA (annotated with reward), which lets the RL policy learn corrective diversifying velocities without ever updating the base policy’s weights.

At inference, RL² composes the two action-expert flow velocities. Rather than picking one policy’s output over the other, or reranking sampled candidates, the two velocity fields are combined during the same denoising integration the base flow-matching action expert would run — a compositional-steering primitive that borrows the base VLA’s expressiveness and adds the RL policy’s diversity as an additive push.

The gating mechanism is the second contribution. A learned failure-probability predictor reads the ongoing rollout, and a conformal-prediction band is calibrated around it; compositional steering activates only when the predicted failure probability crosses the CP band. The scaling analysis in the paper motivates this: on BridgeV2 with an oracle verifier, additional sample-diversification k lowers action error e ≈ a·k^b when the base policy is failing, but leaves already-accurate actions unchanged at best and can drive them off-manifold at worst. Adaptive activation converts “steering budget” into a scarce resource spent only on trajectories that need it.

Results

• Up to +17.3 pp OOD gain on SIMPLER and PolaRiS over prior test-time-steering baselines [Abstract].
• Up to +26.7 pp on real hardware over the strongest Rephrase baseline on OOD prompts and environments [project page].
• Scaling analysis on BridgeV2 with π₀ + oracle verifier confirms e ≈ a·k^b scaling in failure states and near-flat/negative response in success states — RL²-steered action error is lowest during failure and stays high during success (in green vs orange in the reproduced figures) [project page “Test-Time Scaling analysis”].
• 8 rephrases × 5 samples with compositional steering adds up to +18.6 pp over single-sample inference on In-Domain SIMPLER OOD Prompt — sample-diversity and sample-quantity axes combine multiplicatively rather than substitutively [project page].
• Ablations confirm both (a) using latent representations from the action expert and (b) training the second policy with offline RL are individually load-bearing [Abstract].

Why it’s interesting

RL² stakes out a genuinely new position on the “how to steer a frozen VLA at test time” axis that VLA Models and Inference-Time Scaling have been tracking. Prior entries have either (a) kept the VLA frozen and searched over language rewrites (Learning What to Say to Your VLA: Mostly Harmless VLA Steering LFP with a conformal improvement head), (b) kept the VLA frozen and put a test-time-training update on its own memory (WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time WAM-TTT), or (c) fine-tuned the VLA end-to-end with RL through the flow chain (VINE: Taming Generative Control Policies for Reinforcement Learning VINE, π*0.6: a VLA That Learns From Experience (RECAP) π*0.6/RECAP). RL² is the first filed instance that trains a separate offline-RL policy on the base VLA’s latents and composes their action-expert velocities at inference — a training-side + inference-side split that leaves the base VLA weights and inference cost unchanged while still injecting learned diversifying behavior.

The scaling-law finding is the more general contribution. Existing inference-time-scaling entries on Inference-Time Scaling treat “more compute at inference” as broadly monotone: MCTS (Planning at Inference: MCTS Test-Time Scaling for Long Video Generation), iterative refinement on flow generators (Self-Refining Video Sampling), and cache-compressed longer chains (Inference-Time Hyper-Scaling with KV Cache Compression) all argue for spending more. RL² shows that for embodied control the monotonicity assumption breaks: diversifying-at-success can degrade already-accurate actions, and gating steering on a predicted-failure signal is what recovers the monotonic curve. It’s the sharpest filed evidence that inference-time-scaling axes need per-state, not per-run, activation.

See also

• Learning What to Say to Your VLA: Mostly Harmless VLA Steering — LFP is the closest sibling: frozen-VLA + test-time steering, but the steering signal is language rewrites and the OOD-refusal is a conformal improvement head; RL² steers in latent/velocity space and gates on a conformal failure band
• WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time — WAM-TTT steers a frozen WAM via TTT on its own KV memory; RL² steers a frozen VLA via a separately-trained offline-RL companion policy composed at the velocity level
• VINE: Taming Generative Control Policies for Reinforcement Learning — VINE adds RL to a flow-matching action head by fixing the sampler and back-propagating value gradients end-to-end; RL² keeps the base head frozen and trains a second flow-matching policy in latent space
• π*0.6: a VLA That Learns From Experience (RECAP) — π*0.6/RECAP conditions supervised likelihood on advantage tokens to sidestep flow-matching’s no-log-likelihood problem; RL² sidesteps it by staying entirely offline-RL and composing at inference
• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning — QGF is the other filed “no policy update, all inference” answer; QGF steers via value-gradient guidance, RL² steers via a learned latent RL policy’s velocity
• VLA Models — adds a training-side + inference-side split (frozen base + latent-conditioned offline-RL companion + velocity composition + failure-gated activation) as a distinct row on the recipe-lever board
• Inference-Time Scaling — adds a state-dependent scaling-law axis: sample-diversification scaling is monotone in failure states and non-monotone (potentially negative) in success states, motivating per-state gating on any inference-time-scaling primitive
• Reasoning RL — adds an offline-RL-on-frozen-backbone-latents recipe distinct from end-to-end value-gradient RL (VINE), advantage-conditioning (RECAP), and inference-time value guidance (QGF)