Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement

TL;DR

Studies whether RL post-training of a generalist robot policy can improve both training-task performance and generalization to unseen scenes/objects, rather than specializing the policy to a narrow deployment task. Across a controlled sweep from 1 to 144 training tasks on MolmoSpaces, critic-guided RL recipes (OGPO, Best-of-N) show a transition from forgetting at low task counts to generalization at high task counts, reaching up to +58 pp on unseen scenes and +39 pp on unseen objects over the base policy. Filtered SFT (FSFT) also improves training-task success but transfers less reliably. Task diversity and value learning are named as the two load-bearing ingredients, with the type of diversity (scenes vs objects) determining which axis of generalization is unlocked.

Key claims

Single-task RL post-training specializes and forgets, but scaling the number of training tasks causes critic-guided recipes (OGPO, BoN) to transition from forgetting to generalization — up to +58 pp on unseen scenes and +39 pp on unseen objects over the base policy [Abstract, Results].
Scene diversity transfers primarily to unseen scenes and object diversity primarily to unseen objects; only scaling object diversity surpasses the base policy on both generalization axes in the head-to-head sweep [Results §What kind of diversity matters?].
At 144 training tasks (64 → 144 adds four object categories: salt shakers, soap dispensers, mugs, fruit & vegetables), BoN improves most on atomizers and wine bottles — objects sharing upright, bottle-like shapes with the newly added training categories — while shape-dissimilar held-out objects see little additional improvement [Results, Transfer to unseen objects 64→144].
OGPO’s largest gain over the same 64→144 range is on atomizers (≈+16 pp), while wine-bottle success stays low (≈9% → 12%) and bowls, forks, and remote controls reach high success rates [Results, Transfer to unseen objects 64→144].
Filtered SFT (FSFT) improves training-task performance but generalizes less reliably than critic-based RL (OGPO, BoN) across unseen scenes and objects [TL;DR, Results].
To the authors’ knowledge, this is “among the first evidence for online generalist improvement for robot foundation models” — both the training-task-distribution breadth and the use of value-based RL are named as critical [Abstract].

Method

Three post-training recipes are compared on a shared generalist base policy, under a structured evaluation that varies two axes of task diversity on MolmoSpaces — the object manipulated and the scene it appears in — scaling training-task count from 1 to 144 and holding out scenes and objects for generalization measurement:

FSFT (Filtered SFT) — supervised fine-tune on successful rollouts.
BoN (Best-of-N) — learn a critic, sample N action candidates at test time, execute the highest-ranked.
OGPO — learn a critic and use it to update the policy toward it (critic-guided policy extraction).

Design choices sweep value estimation, policy extraction, and resource allocation. Training-task counts of 1, 4, 16, 64, 144 are reported; the 64 → 144 step adds four object categories (salt shaker, soap dispenser, mug, fruit & vegetables), which makes it the diagnostic window for shape-based transfer.

Results

Critic-guided recipes (OGPO, BoN) show a phase transition from forgetting (single-task RL hurts other tasks) to generalization (RL improves unseen tasks) as task count scales.
Headline deltas vs the base policy: +58 pp on unseen scenes, +39 pp on unseen objects at the scaled training-task distribution.
Object-diversity scaling surpasses the base on both unseen scenes and unseen objects; scene-diversity scaling only surpasses on unseen scenes in the controlled head-to-head.
64 → 144 training-task shape-transfer: BoN’s biggest gains concentrate on atomizers and wine bottles (shape match to the four added training categories); OGPO’s largest gain is +16 pp on atomizers, with wine bottles moving only ≈9% → 12%; bowls, forks, remote controls stay high.
FSFT improves training-task performance but delivers weaker and less reliable transfer than BoN / OGPO across the generalization axes.

Why it’s interesting

Sharpest filed answer yet to the “does RL post-training generalize or specialize?” question on generalist robot policies — complements VLA Models‘s existing RECAP (π*0.6: a VLA That Learns From Experience (RECAP)) and VINE (VINE: Taming Generative Control Policies for Reinforcement Learning) entries, which both demonstrated that RL can improve a flow-matching VLA on target tasks but did not control for the number of RL training tasks as the primary scaling axis. Reframes “forgetting under RL” as a task-count-dependent regime rather than an intrinsic failure mode, and localizes the generalization lift to critic-based methods specifically — directly relevant to Reasoning RL‘s open question of when value-based signals beat filtered supervision, now with robot policies as an application domain. The shape-match transfer pattern (atomizers/wine-bottles lifted by adding salt-shakers/soap-dispensers to training) argues that task diversity in robot RL post-training acts along concrete perceptual axes (object shape) rather than as an undifferentiated “more data” lever, parallel to MimicLabs’s dimension-level findings on VLA co-training (What Matters in Learning from Large-Scale Datasets for Robot Manipulation).