DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation

TL;DR

DenseReward is a vision-language reward model that predicts dense frame-level reward scores for robot manipulation from visual observations and language instructions, aimed at the two-part bottleneck that keeps RL from routinely improving VLA-style policies past their imitation-learning plateau: (1) diverse failure trajectories are hard to collect and pseudo-failure relabeling of successes doesn’t capture the physical failure modes that show up at execution time, and (2) most existing robot reward models emit sparse binary or trajectory-level scores that don’t give a policy optimizer per-step gradient. The training pipeline synthesizes physically realistic failure trajectories in simulation without human labeling, spanning collisions, missed grasps, object drops, and recovery behaviors, and the resulting reward model beats general-purpose VLMs and prior robotic reward models on dense-reward prediction in both simulation and real. The authors then show DenseReward is usable as the reward signal for downstream model-predictive control and reinforcement learning, and release dataset, weights, and evaluation suite.

Key claims

Robot RL is bottlenecked by two joint deficits — scale of failure data, and granularity of the reward signal — and prior work addresses at most one at a time; DenseReward targets both together [Abstract].
Pseudo-failure trajectories constructed by relabeling successful demonstrations fail to cover the physical failure modes (collisions, missed grasps, object drops, recovery) that actually arise during robot execution, motivating a simulator-driven synthesis pipeline [Abstract].
An automated failure-data generation pipeline in simulation produces physically realistic failure trajectories without human labeling, covering diverse failure modes [Abstract].
DenseReward predicts dense, frame-level reward scores from visual observations plus language instruction, enabling fine-grained task-progress estimation across an episode [Abstract].
On dense reward prediction, DenseReward beats general-purpose VLMs and prior robotic reward models across simulated and real-world manipulation [Abstract].
The learned reward provides effective guidance for both model-predictive control and reinforcement-learning downstream policies [Abstract].
Dataset, trained reward models, and an evaluation suite are released to support “failure-aware dense reward modeling” as a research subarea [Abstract].

Method

DenseReward is a VLM-based reward model whose training data is a synthesized corpus of failure trajectories generated by a simulation pipeline; the pipeline injects physical failure modes (collisions, missed grasps, drops, recovery) without human annotation, complementing existing success trajectories. The reward head consumes visual observations and a language instruction and outputs a per-frame scalar reward, so the signal is available at each timestep rather than only at episode end. Downstream, the same reward can be used as the return signal for either MPC-style short-horizon planning or on-policy / off-policy RL fine-tuning of a base policy.

Results

Reported dense-reward-prediction wins over both frozen general-purpose VLMs and dedicated robotic reward models, in simulated and real-world manipulation settings [Abstract]. Numerical comparisons and the exact downstream MPC/RL policy improvements are in the paper body and were not extractable from the abstract at filing time; the paper explicitly claims “effective reward guidance for downstream MPC and RL” [Abstract].

Why it’s interesting

Sits at the intersection of two threads this wiki actively tracks. First, it is the robot-manipulation instance of VLM-as-Evaluator — a domain that had been visible mostly as productized success detectors (Instance Labs — Verifying Robot Learning Episode Success, Instance — a VLM success detector for robot rollouts (Lucy Cai launch tweet)) or as retrospective analysis pipelines (RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies §3.3), with the dense reward shape less represented; DenseReward is a candidate to be the robot analog of Unified Personalized Reward Model for Vision Generation (T2I) or LLM-as-a-Verifier: A General-Purpose Verification Framework (general-purpose verifier). Second, the failure-synthesis pipeline directly addresses the Synthetic Training Data pattern where a verification-gate + procedurally-diverse rollouts substitute for scarce real labels — the same shape PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation and Instance Labs — Verifying Robot Learning Episode Success use on the policy-training side. It also contrasts with E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learning / UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning, which push efficiency of real-world human intervention rather than replace failure data with sim.