TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

TL;DR

TOPReward is a zero-shot, training-free progress estimator for robotic manipulation that reads task progress directly from a video-VLM’s next-token log-likelihood on an affirmative answer (“True”) to a completion query. The extracted per-frame log-probabilities are aligned across time to produce a dense temporal reward without any reward-model fine-tuning or numeric-generation prompting, sidestepping known VLM weaknesses in instruction following and numeric reasoning. Evaluated on Open-X-Embodiment (39 datasets, 780 episodes) and a new ManiRewardBench (113 tasks, 497 episodes across Franka, YAM, SO-100/101), TOPReward hits 0.945 mean Value-Order-Correlation on ManiRewardBench with Qwen3-VL and outperforms the GVL baseline on open-source VLMs; it also improves success detection ROC-AUC and, deployed on a real SO-100, enables advantage-weighted regression from 50 noisy demos per task to consistently beat behavior cloning across 6 tabletop tasks.

Key claims

Task progress can be read from a frozen video-VLM’s log-probability on the token “True” for a completion query, avoiding the model’s numeric-generation and instruction-following weaknesses [Method].
On ManiRewardBench (mean VOC), TOPReward with Qwen3-VL-8B reaches 0.857 vs GVL 0.194, and with Molmo-2-8B 0.417 vs GVL −0.016 — a large open-source margin; on Gemini-2.5-Pro the pattern flips (0.433 vs 0.541 for GVL) [Results, mean VOC table].
On per-embodiment splits with Qwen3-VL-8B, TOPReward reaches 0.954 LeRobot, 0.942 Franka, 0.947 Bimanual YAM, 0.945 Single-arm YAM — consistently >0.94 across four platforms vs GVL 0.164–0.544 [Results, per-embodiment table].
Failed-trajectory success detection (23 tasks, ROC-AUC): TOPReward outperforms GVL on Qwen3-VL-8B (0.654 vs 0.519) and matches on Gemini-2.5-Pro (0.826 vs 0.823); the authors argue VOC only measures rank consistency, so plateauing-early failures can score high on GVL but not on an instruction-satisfaction-likelihood readout [Success detection §, ROC-AUC table].
Deployed as advantage weights for AWR on a real SO-100 with only 50 noisy demos per task, TOP-AWR beats BC across all 6 tasks (e.g. Place doll in box 10 vs 7; Pick up cube 10 vs 7; Stack red cube on green cube 2.3 vs 1; Put pen into cup 6.3 vs 5.7) [Real-robot AWR table].
The reward signal is smooth and monotonically increasing during active manipulation with visible plateaus at subtask completion, versus noisier non-monotonic Gemini-GVL predictions [Qualitative §].

Method

For each frame in a robot trajectory, TOPReward queries a frozen video-VLM with the observed trajectory-so-far and a language instruction, asking whether the instruction has been completed. Rather than sampling a numeric score or a categorical answer, the method reads the model’s log-likelihood on the token “True” (or the equivalent affirmative token) — a single teacher-forced log-probability per frame. Frames are sampled through time and their extracted log-probabilities aligned to form a dense temporal reward signal. Success detection uses the average log-likelihood over the last three sampled frames as a scalar classifier. The downstream real-robot pipeline is advantage-weighted regression (AWR): TOPReward’s per-frame scores are converted into advantage weights on top of a pretrained policy, and behavior cloning is reweighted by those advantages using only 50 noisy demonstrations per task on an SO-100.

Results

ManiRewardBench mean VOC (Value-Order Correlation). Qwen3-VL-8B: TOPReward 0.857 vs GVL 0.194; Molmo-2-8B: 0.417 vs −0.016; Gemini-2.5-Pro: 0.433 vs 0.541 (only setting where GVL wins) [mean VOC table].
Per-embodiment VOC with Qwen3-VL-8B. LeRobot 0.954 (GVL 0.332); Franka 0.942 (0.242); Bimanual YAM 0.947 (0.164); Single-arm YAM 0.945 (0.544). Gemini-2.5-Pro TOPReward numbers are lower (0.448–0.578) than its GVL counterparts on the same splits, so the win depends on backbone [per-embodiment table].
Success detection ROC-AUC on ManiRewardBench failure split (23 tasks). Qwen3-VL-8B: 0.654 vs 0.519 (GVL); Gemini-2.5-Pro: 0.826 vs 0.823 [ROC-AUC table].
Real SO-100 (50 demos/task, average successes / N trials). TOP-AWR vs BC vs Pretrained: Place toy car in box 3/2/1; Stack red cube on green cube 2.3/1/1.3; Put pen into cup 6.3/5.7/1.7; Place doll in box 10/7/0; Pick up cube 10/7/4; Put cube in cup 9/6/4 [AWR table].
Datasets released. Open-X-Embodiment sweep (39 datasets, 780 episodes) and ManiRewardBench (113 tasks, 497 episodes across Franka, YAM, SO-100/101) [Datasets §].

Why it’s interesting

TOPReward is the concrete robot-manipulation instantiation of the continuous-logit-expectation verifier pattern LLM-as-a-Verifier: A General-Purpose Verification Framework identifies as a general primitive — but with two sharpenings. First, instead of an expectation over a scoring-token distribution, it uses a single affirmative-token log-probability, deliberately bypassing the score-token elicitation step that fails on VLMs weak at numeric generation; the authors report that this is exactly the readout that makes open-source video-VLMs (Qwen3-VL, Molmo-2) usable as reward models where GVL-style rank scoring collapses. Second, unlike DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation, which trains a dedicated dense-reward VLM on simulator-synthesized failure trajectories, TOPReward is entirely training-free — it complements DenseReward’s synthesis-heavy answer with a prompting-only answer to the same dense-reward bottleneck, and provides the first filed head-to-head evidence that the choice of readout (log-likelihood on “True” vs GVL rank scoring vs VOC) drives most of the open-source-VLM performance gap. The real-robot AWR result also connects directly to π*0.6: a VLA That Learns From Experience (RECAP) / Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline) — advantage-conditioning is exactly the RECAP-style lever those recipes use, and TOPReward supplies the reward signal without requiring a separately trained value model or Q-head. Note the backbone-dependence: TOPReward wins on open-source VLMs but ties or loses on Gemini-2.5-Pro, an important calibration point that echoes the judge-backbone-sensitivity finding IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation documented for image-side evaluation.