Proxy Policy Steering: Adapting Frozen Robotic Foundation Models via Velocity-Space Proxy Residuals

TL;DR

Proxy Policy Steering (PPS) is an anonymous under-review adaptation recipe for frozen robotics foundation models: instead of fine-tuning the base policy or steering within its existing action distribution, train two lightweight proxy flow-matching policies — a reference proxy distilled on-policy from the base’s own denoising trajectories, and a task proxy initialized from the reference and fine-tuned on task demonstrations — then add their velocity-space residual γ · (v_task − v_ref) to the base’s velocity field at every denoising step. Reports +53% absolute success over the π0.5 base averaged across 8 real-world tasks and +51% averaged across 4 simulation tasks, outperforming LoRA fine-tuning, from-scratch specialists, residual policies, and DSRL on every simulation task, with 0-to-1 gains on tasks the base never solves alone. Because the proxies are separate networks with their own inputs, PPS can route new modalities (point clouds, audio) into the RGB-only frozen base through the task proxy.

Key claims

• Existing adaptation approaches trade off in a way PPS avoids: fine-tuning risks catastrophic forgetting of the base’s broad priors, while inference-time steering that samples only within the base distribution can’t inject genuinely new task behaviors [project page].
• PPS’s core structural move is to estimate the direction in which task supervision would shift the base’s action distribution without accessing base gradients, by training two proxies and taking their velocity-space difference [project page, “Our key insight”].
• The reference proxy v_ref is distilled on-policy from the base’s own denoising trajectories, so it reproduces base behavior on exactly the states the sampler visits at inference — this on-policy distillation is what makes the subsequent residual meaningful [project page, method].
• The task proxy v_task is initialized from v_ref and fine-tuned only on task demonstrations; sharing initialization and architecture with the reference is load-bearing because it isolates the task-induced change from architectural or initialization noise [project page, method].
• At inference PPS integrates the guided velocity field v_PPS(x, k, o, l) = v_base(x, k, o, l) + γ · [v_task(x, k, o) − v_ref(x, k, o)], so when the proxies agree the residual vanishes and the frozen base stays in control [project page, inference equation].
• Reports +53% absolute success over the π0.5 base averaged across 8 real-world manipulation tasks [project page, results].
• Reports +51% absolute success over the π0.5 base averaged across 4 simulation tasks, outperforming LoRA, from-scratch specialists, residual policies, and DSRL on every task [project page, results].
• Reports 0-to-1 gains on tasks the base never solves on its own — evidence that steering is not restricted to the base’s existing distribution [project page, results].
• The frozen-base combined policy recovers from failures never seen in demonstrations, while LoRA fine-tuning overfits to the same demonstrations and loses those priors under matched perturbations [project page, robustness discussion].
• Because proxies are separate networks with their own inputs, PPS can route extra modalities (point cloud for texture-less objects, audio for ringing-vs-silent phone) into the frozen, RGB-only base through the task proxy — no base retraining required to add a sensor [project page, “extra modalities” section].
• Paper is anonymous under review at filing time and the tweet thread runs across 8 posts by @tritiumac introducing the method — no arXiv ID or code release available at filing.

Method

The base π0.5 is treated as a frozen flow-matching action expert with velocity field v_base(x, k, o, l) over noised action x at denoising step k, observation o, and language l. Two lightweight proxies of the same architecture are trained:

1. Reference proxy v_ref — distilled on-policy from the base’s own denoising trajectories on target-task observations. Crucially the distillation states are exactly the states the base sampler visits at inference, not a separate demonstration distribution, so v_ref reproduces base behavior on the deployment support.
2. Task proxy v_task — initialized from v_ref and fine-tuned on the task demonstrations. Because it shares initialization + architecture + observation inputs with the reference, the only distributional difference between them comes from task supervision — the paper’s identifiability claim is that the difference v_task − v_ref therefore isolates the change task supervision would have induced on the base itself.

At inference the base sampler runs unchanged except the velocity field is replaced with the guided field v_PPS = v_base + γ · (v_task − v_ref), integrated over the same denoising trajectory the base would have used. The coefficient γ sets the steering strength; the residual vanishes exactly when the proxies agree, so the frozen base retains control on states unchanged by task supervision.

The extra-modality extension takes advantage of v_task having its own encoder: point clouds or audio can be fed into the task proxy even when the frozen base only ingests RGB. Because the residual is what enters the base’s sampler, the base doesn’t need to be retrained or re-interfaced to consume the new modality — the modality routing lives inside the task proxy’s encoder.

Results

• +53% absolute success over the π0.5 base averaged across 8 real-world manipulation tasks.
• +51% absolute success over the π0.5 base averaged across 4 simulation tasks, outperforming LoRA, from-scratch specialists, residual policies, and DSRL on every task.
• 0-to-1 gains on tasks the base π0.5 never solves on its own — the sharpest evidence that the steering direction is not constrained to the base’s existing distribution.
• Robustness under perturbations: PPS recovers from failure modes it never saw in demonstrations while LoRA under matched-demo training breaks down on the same perturbations, per the project page’s robustness discussion.
• Modality routing: qualitative demonstrations of point-cloud-conditioned locating of a texture-less object and audio-conditioned selection between ringing vs silent phones, with the base π0.5 remaining RGB-only.
• Reported task set spans long-horizon, articulated, and deformable-object manipulation across the 12 total tasks (per the abstract’s summary of task diversity).
• No per-task-per-baseline table is enumerated in the retrieved page text; the reported margins are averages with the “every task” qualifier applied to the simulation set.

Why it’s interesting

PPS stakes out a training-side answer to the same problem Learning What to Say to Your VLA: Mostly Harmless VLA Steering LFP and RL²-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models RL²-VLA attack from the inference side: how to adapt a frozen robotics foundation model to a new task without eroding its priors. LFP steers by rewriting language mid-rollout; RL²-VLA composes velocities between the base and a separately-trained offline-RL companion policy conditioned on the base’s latents. PPS composes velocities between the base and a differentially-trained proxy pair (reference vs task-supervised), which lets it inject genuinely new behaviors — RL²-VLA’s companion is trained on rollouts of the base, so its steering support is bounded by what the base already tries; PPS’s task proxy is trained on task demonstrations directly, so 0-to-1 gains on tasks the base never solves are structurally admissible in a way they aren’t for RL²-VLA.

The most interesting position PPS stakes out relative to the rest of VLA Models is on the LoRA-vs-frozen-base recipe axis. Per the recipe-lever board, LoRA fine-tuning of π0.5-family bases is the dominant “cheap adaptation” default; PPS’s claim to beat LoRA on every simulation task and preserve robustness under perturbations LoRA breaks on is a direct challenge to that default. If the “shared initialization isolates the task-induced change” identifiability claim holds under scrutiny, PPS is the first filed adaptation recipe whose residual has a principled interpretation as “the direction task supervision would have moved the base” rather than just “a learned correction”. The modality-routing extension is a bonus — Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models Embodied-R1.5 and Isaac 0.5: An Open-Weight Embodied Foundation Model Isaac 0.5 both handle new modalities by retraining a unified backbone; PPS’s approach of routing new sensors into the task proxy while keeping the base frozen is a cheaper deployment story if it generalizes.

See also

• VLA Models — PPS adds a new row to the recipe-lever board: frozen base + on-policy-distilled reference proxy + task-supervised task proxy + velocity-space residual composition, distinct from LoRA / residual policies / DSRL / RL²-VLA / LFP / VoLo on the “adapt a frozen VLA” axis.
• RL²-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models — RL²-VLA is the closest sibling in inference-time mechanism (compose two flow-matching velocity fields at inference over the base’s action-expert integration), but trains the companion via offline RL on base rollouts rather than supervised on task demos, and gates activation on a conformal failure-probability band rather than always adding the residual.
• Learning What to Say to Your VLA: Mostly Harmless VLA Steering — LFP is the other filed “leave the VLA frozen and steer it” recipe, but the steering signal is closed-loop language rewrites with a conformal refusal head rather than a velocity-space residual; both accept the same “don’t touch the base weights” premise.
• π*0.6: a VLA That Learns From Experience (RECAP) — π*0.6 / RECAP is the direct counter-recipe: modify the same π0.5-family base via advantage-conditioned offline RL, updating the weights rather than composing a proxy residual around them.
• VINE: Taming Generative Control Policies for Reinforcement Learning — VINE also adds RL to flow-matching action heads but back-propagates value gradients through the entire denoising chain end-to-end; PPS keeps the base untouched and puts all learning into the two auxiliary proxies.
• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning — QGF is the value-gradient-guidance sibling on the “no base weight update” axis; PPS’s residual is a learned counterpart to QGF’s on-the-fly gradient guidance.
• Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering — original “Mostly Harmless VLA Steering” companion to LFP, similarly frozen-base + steering intervention.