Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

TL;DR

Q-Planning pairs a large frozen visuomotor Behavior Cloning policy with a small off-policy Q-function so a robot can improve from its own deployment rollouts without any RL fine-tuning of the multi-billion-parameter policy. At inference the BC policy samples N candidate action chunks and the executed chunk is a single-step Q-weighted average; failed and successful rollouts are appended to a replay buffer and only the Q-function is updated. On LIBERO-10 (93→99%), bimanual RoboTwin (83.8→91.4%), and two real-world contact-rich bimanual tasks (stack-cups 40→90%, insert-wallet 25→80% in five iterations, BC frozen throughout) Q-Planning is reported as the only method — among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR — that improves stably from failures without training an auxiliary actor at an identical online budget.

Key claims

Behavior Cloning is fundamentally limited by its inability to self-improve without additional human demonstrations, and RL fine-tuning of multi-billion-parameter BC policies is difficult to scale [Abstract].
A Q-function estimates value rather than imitating actions, so it can be trained on the same successful demonstrations as the BC policy and additionally absorb both successful and failed deployment rollouts — an asymmetry the BC policy itself does not have [Abstract].
Inference is a single-step Q-weighted average over N candidate action chunks sampled from the frozen BC policy; no separate actor is trained [Abstract, Fig. 1 caption].
Ten iterations of self-improvement lift LIBERO-10 93% → 99% and RoboTwin 83.8% → 91.4% [Abstract].
On the near-ceiling LIBERO-Object and LIBERO-Goal suites, self-improvement shortens successful-episode length rather than raising success rate — evidence the Q-function is learning something beyond binary success [Abstract].
On two contact-rich bimanual real-robot tasks with the BC policy frozen, five iterations of Q-Planning lift stack-cups 40% → 90% and insert-wallet 25% → 80%, with no human intervention [Abstract].
SFT on successful rollouts alone stalls at 55% and 30% on the same two real-world tasks — the ability to consume failures is load-bearing, not the extra data volume [Abstract].
Under an identical online budget, Q-Planning is reported as the only method among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR that improves stably from failures without training an auxiliary actor [Abstract].
On 100 teleop demos of a contact-rich real-world task the ceiling of a plain BC policy is ~25% success, and this ceiling is a property of BC as a paradigm (data is noisy, slow, fine-grained tasks are hard to teleoperate cleanly) rather than a property of the specific model [tweet thread §2/n].
Beyond raw success rate, successful episodes get faster and more confident across self-improvement iterations [tweet thread §5/n].

Method

Pre-train a BC policy $\pi_\theta$ and a Q-function $Q_\phi$ on the same expert demonstration set. Then repeat: at inference, sample $N$ candidate action chunks from the frozen BC policy, score each with $Q_\phi$ , and execute a single-step $Q$ -weighted average over the $N$ candidates. Both successful and failed episodes from these rollouts are appended to a replay buffer. Between iterations, fine-tune only $Q_\phi$ on the replay buffer; the BC weights are never updated. The next iteration then samples from the same BC policy but is scored by the updated $Q_\phi$ , and the loop repeats.

The load-bearing structural asymmetry, per the paper: because a Q-function estimates value rather than imitating actions, failed rollouts are legitimate training signal for $Q_\phi$ (they are low-value states), whereas the same failed rollouts are actively harmful supervision for a BC policy (they’d teach it to fail). This is what lets Q-Planning consume the mixed success/failure stream that natural deployment produces, without an auxiliary actor to interpret it.

Baselines specifically ruled out in the empirical comparison at identical online budget: Best-of-N (no updates), filtered SFT (updates BC on successes only), IBRL, DSRL, DAWR. Only Q-Planning is reported to improve stably from failures under the “no auxiliary actor” constraint.

Results

LIBERO-10: 93% → 99% over ten self-improvement iterations [Abstract].
Bimanual RoboTwin: 83.8% → 91.4% [Abstract].
LIBERO-Object / LIBERO-Goal: no headline success-rate jump (already near ceiling), but successful episodes shorten [Abstract]. In the tweet thread the author also reports LIBERO-10 at 99%.
Real robot, contact-rich bimanual stack-cups: 40% → 90% in five iterations with BC frozen; SFT-on-successes stalls at 55% [Abstract, tweet thread §5/n].
Real robot, contact-rich bimanual insert-wallet (credit card → wallet): 25% → 80% in five iterations with BC frozen; SFT-on-successes stalls at 30% [Abstract, tweet thread §5/n].
Real-robot BC ceiling from 100 teleop demos: 25% on the harder of the two tasks — quantifies the “BC has a ceiling by construction” framing [tweet thread §2/n].
Head-to-head at matched online budget across Best-of-N, filtered SFT, IBRL, DSRL, and DAWR: Q-Planning is reported as the only method that improves stably from failures without training an auxiliary actor [Abstract].
Setup detail (from tweet §3/n): FastWAM base policy, 100 on-policy rollouts per iteration in the reported real-world runs.

Why it’s interesting

Q-Planning is the sharpest filed instance of a distinct answer on the VLA Models recipe board to the “how do we make a large frozen VLA improve at deployment” question that also drives π*0.6: a VLA That Learns From Experience (RECAP) (π*0.6 / RECAP), VINE: Taming Generative Control Policies for Reinforcement Learning (VINE), and UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning (UniIntervene). Where RECAP folds the value signal into the BC policy itself by binarizing advantages into a text token in the prefix and fine-tuning the whole VLA via CFGRL-style advantage conditioning, and where VINE does end-to-end value-gradient BPTT through the flow-matching denoising steps, Q-Planning takes the opposite position: leave the BC policy weights untouched, put all the learning into a small off-policy Q-function, and use it as an inference-time re-scorer of BC candidates. That factorization slots it onto this concept’s “spend more inference compute for capability the BC can’t deliver one-shot” axis — sibling to Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning (QGF) as another value-guided-BC-sample-selection recipe, but with the added self-improvement loop where the scorer itself gets sharper over rollouts. Contrasts most sharply with World Value Models for Robotic Manipulation (WVM), which argues value models should sit on a world-model backbone rather than the policy backbone — Q-Planning stays on the policy side and demonstrates that a small policy-side Q-head suffices for the closed-loop-from-failures story.