World Value Models for Robotic Manipulation

TL;DR

ByteDance Seed (with PKU + Tsinghua) argues the right backbone for a generalist robotic value model is a world model rather than a VLM: value estimation requires deep temporal grounding and future planning, which VLMs trained on static / temporally-sparse images lack but world models naturally provide. World Value Model (WVM) marries the two — a world-model backbone produces accurate task-progression scores that act as both a data-quality filter and a policy-learning signal across mixed-quality demonstration corpora. On the standard Value-Order Correlation (VOC) benchmark over OXE / RoboCOIN / EgoDex / self-collected data WVM hits 0.95 average vs. 0.88 for the next-best (Robo Reward) and 0.78 for GVL; on a new Suboptimal-Value-Bench (800 human-frame-annotated suboptimal trajectories across AgileX / ARX / RoboSuite) WVM averages 0.78 VOC and 0.05 hesitation RMSE vs. 0.62 / 0.14 for the strongest baselines, and downstream policy extraction improves in both sim and real-world deployment.

Key claims

The mismatch the paper attacks: most existing robotic value models inherit a VLM backbone (Gemini-VL family, Qwen-VL, etc.) pretrained on static or temporally-sparse images, but accurate value estimation requires grounding the current belief against historical context and planning over future outcomes — capabilities that world models, not VLMs, naturally provide [Abstract].
WVM’s proposal: build the value model on a world-model backbone so the same temporal-modeling capacity that powers next-frame prediction is reused for value estimation; the marriage yields a generalist value function that produces accurate task-progression scores across embodiments and data sources [Abstract, project page Overview].
On the standard expert-data benchmark (Value-Order Correlation, higher is better) WVM hits 0.94 / 0.95 / 0.92 / 0.99 on OXE / RoboCOIN / EgoDex / self-collected (3 embodiments) for an average of 0.95, vs. 0.88 for the next-best Robo Reward and 0.78 for GVL [project page Expert VOC table].
The paper introduces Suboptimal-Value-Bench, an 800-trajectory multi-embodiment (AgileX / ARX / RoboSuite) benchmark with high-fidelity human-labeled frame annotations of suboptimal behavior — the first filed benchmark explicitly evaluating value models on suboptimal (not just expert) data [Abstract, project page Suboptimal-Value-Bench].
On Suboptimal-Value-Bench WVM holds SOTA across all three embodiments: 0.78 average VOC (next-best GVL at 0.62, VLAC at -0.37) and 0.05 average hesitation RMSE (next-best GVL/Robo meter tied at 0.14) [project page Suboptimal-Value-Bench tables].
Baselines split sharply on suboptimal data: GVL and Robo meter degrade gracefully from their expert numbers (0.78 → 0.62, 0.81 → -0.16 VOC), while VLAC collapses (0.59 → -0.37 VOC), suggesting VLM-backbone value models can be tuned to be robust but a world-model backbone removes the trade-off [project page tables].
When deployed for policy learning, WVM improves manipulation performance across various policy-extraction approaches in both simulated and real-world deployment, framed as providing “robust guidance for learning from mixed-quality data” [Abstract] — i.e. it’s positioned as a generic value head rather than tied to a specific RL recipe.

Method

The paper’s central design choice is the backbone swap: instead of attaching a value head to a VLM (the GVL / VLAC / Robo Reward / Robo-Dopamine lineage), WVM puts the value head on a world model — a model already trained to predict future frames conditioned on history. Mathematically, the authors argue this matches the structure of value estimation, which is V(o_t) = E[Σ γ^k r_{t+k}] and therefore requires both temporal grounding (the conditioning on history o_{≤t}) and forward planning (the expectation over future trajectories) — capabilities that a VLM’s pretraining objective doesn’t supervise.

The paper introduces two evaluation primitives the cluster previously lacked:

Value-Order Correlation (VOC) — the standard rank-correlation metric on expert demos, applied here across four diverse datasets (OXE, RoboCOIN, EgoDex, self-collected across 3 embodiments).
Suboptimal-Value-Bench — 800 trajectories spanning AgileX / ARX / RoboSuite, each with human-labeled frame annotations marking suboptimal segments. Two metrics: VOC (as above) and hesitation RMSE — the error in localizing where the trajectory becomes suboptimal.

The paper’s abstract claims gains across “various policy-extraction approaches” both in sim and on real robots, suggesting the value model is used in the standard offline-RL way (advantage-weighted regression, reward shaping, sample filtering) downstream rather than via a single bespoke policy recipe. Full architecture, training data composition, and the specific world-model backbone family are in the body of the arxiv PDF (not retrieved at filing time).

Results

Numbers from the project page leaderboards:

Expert VOC (higher better, average over 4 datasets): WVM 0.95 > Robo Reward 0.88 > Robo-Dopamine 0.82 > Robo meter 0.81 > GVL 0.78 > Top Reward 0.42.
Suboptimal-Value-Bench VOC (average over 3 embodiments): WVM 0.78 > GVL 0.62 > Top Reward 0.00 > Robo meter -0.16 > VLAC -0.37.
Suboptimal-Value-Bench Hesitation RMSE (lower better, average): WVM 0.05 < GVL / Robo meter 0.14 < Robo Reward 0.21 < Top Reward 0.31 < Robo-Dopamine 0.49 < VLAC 0.51.
Downstream policy learning: improvements reported across multiple policy-extraction approaches in simulation and real-world deployment [Abstract] — specific numbers and tasks in the PDF body (not retrieved at filing).

The headline numbers say two things: (a) on expert data WVM is incrementally better than the VLM-backbone Robo Reward family but not by a huge margin (0.95 vs. 0.88); (b) on suboptimal data the gap blows open (0.78 vs. 0.62 on VOC, 0.05 vs. 0.14 on hesitation RMSE) — which is the regime the paper argues actually matters for scaling policy learning from mixed-quality data.

Why it’s interesting

WVM stakes out a new deployment pattern for the World Foundation Models cluster that the page hasn’t seen before: WFM-as-value-model. The existing patterns the page enumerates are WFM-as-backbone (Cosmos + GR00T), WFM-as-reward via predictive surprise (Inference-time Physics Alignment of Video Generative Models with Latent World Models, where VJEPA-2’s frozen physics-plausibility score steers video generators), WFM-as-interruptible-tool (VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation VoLo), WFM-as-policy-internal-substrate (Causal World Modeling for Robot Control (LingBot-VA) LingBot-VA), WFM-as-edit-prior consumed via KV cache (ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing? ImageWAM), and µ₀’s frozen-trace-WM + small action expert (μ₀: A Scalable 3D Interaction-Trace World Model). WVM is structurally distinct from all of these: the world model isn’t a generator, isn’t a reward by surprise, isn’t a tool, isn’t part of the policy — it’s the substrate of a learned value function that supervises downstream policy learning.

This complements π*0.6: a VLA That Learns From Experience (RECAP) (RECAP / π*0.6) from a different angle: RECAP’s distributional value head shares the VLA’s own VLM-derived architecture (Gemma 3 4B base) and produces per-task value estimates that are then binarized into a CFGRL-style advantage token. WVM is the orthogonal recipe — a separate, generalist value model, world-model-backbone-based rather than VLM-backbone-based, that produces task-progression scores designed to be reused across policy-extraction recipes. The two designs aren’t mutually exclusive (RECAP’s binarization step would still apply to a WVM output), and the paper’s “improves manipulation performance across various policy extraction approaches” framing suggests it’s designed for exactly that interoperability.

The Suboptimal-Value-Bench contribution also fills a gap the VLA Models page has flagged implicitly: most filed VLA papers (Spirit-v1.5, Embodied-R1.5, LingBot-VLA) emphasize either action-pretraining scale or clean-teleop data quality, and don’t measure how well value/reward models localize suboptimal segments. WVM’s hesitation RMSE is the first filed metric for that question.