Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X)

TL;DR

ReVidGen (DAGroup PKU) introduces a paired benchmark + dataset ecosystem for robot-oriented video generation. RBench scores 25 representative video models (open, commercial, robotics-specific) across five task domains (manipulation, spatial, multi-entity, long-horizon, reasoning) and four embodiments (single-arm, dual-arm, quadruped, humanoid), with automatic sub-metrics for structural consistency, physical plausibility, and action completeness. RoVid-X is a 4M-clip 720p open-source robotic video dataset with optical flow, diverse robotic forms, and diverse captions — built via a four-stage refinement pipeline and the largest filed robotic dataset for video generation. The headline finding is that even the top commercial models top out at ~0.61 average score, robotics-specific models (Cosmos 2.5, DreamGen, Vidar, UnifoLM-WMA-0) trail or finish at the bottom, and the automatic metric achieves Spearman 0.96 with human evaluators.

Key claims

RBench evaluates 25 representative video generation models across five robot-oriented task domains (manipulation, spatial, multi-entity, long-horizon, reasoning) and four embodiments (single arm, dual arm, quadruped, humanoid), reporting both task-level correctness and visual fidelity via reproducible sub-metrics including structural consistency, physical plausibility, and action completeness [Abstract, Table 1].
Wan 2.6 leads the RBench leaderboard at 0.607 average; Seedance 1.5 Pro 0.584, Wan 2.5 0.570, Hailuo v2 0.565, Veo 3 0.563, Seedance 1.0 0.551, Kling 2.6 Pro 0.534 [RBench Quantitative Results, Table 1].
Best open-source model is Wan2.2_A14B at 0.507 (rank 8), followed by HunyuanVideo 1.5 at 0.460 and LongCat-Video at 0.437 [Table 1].
Robotics-specific models trail general video generators: Cosmos 2.5 reaches 0.464 (rank 9); DreamGen variants score 0.420 / 0.405; Vidar 0.206; UnifoLM-WMA-0 0.123 (rank 25, last) [Table 1].
Sora v2 Pro ranks 17 at 0.362 and Sora v1 ranks 22 at 0.266, both well behind the leading Chinese commercial models on robotics-oriented evaluation [Table 1].
RBench achieves Spearman correlation 0.96 with human evaluations, validating it as a faithful automatic proxy [Abstract].
RoVid-X provides 4M annotated 720p video clips covering 1300+ skills, with optical flow, diverse robotic forms, and diverse captions — the largest filed robotic video-generation dataset, ~3× Open X-Embodiment (1.4M) and 4× Agibot World (1M) [Comparison of Representative Robotic Video Datasets].
Among prior robotic video datasets, RoVid-X is the first to combine optical flow + diverse robotic forms + diverse captions in one corpus; predecessors (OXE, RoboMIND, Agibot World, InternData-A1) lack one or more of these axes [Comparison of Representative Robotic Video Datasets].
The data pipeline is four-stage and explicitly motivated by RBench’s diagnostic finding of “significant deficiencies in generating physically realistic robot behaviors” — evaluation drives the data-curation design [Abstract].
Task-level dimensions are unevenly difficult across models: manipulation, multi-entity, and long-horizon tasks are the hardest; embodiment-level scores cluster higher than task-level scores for nearly every model (e.g., Wan 2.6 humanoid 0.723 vs reasoning 0.531) [Table 1].

Method

RBench is constructed as a reproducible robotics video-generation benchmark with two evaluation axes — five tasks (manipulation, spatial, multi-entity, long-horizon, reasoning) and four embodiments (single arm, dual arm, quadruped, humanoid). For each (task × embodiment) combination, the benchmark scores a model’s generated video using three automatic sub-metrics — structural consistency, physical plausibility, and action completeness — combined into the nine reported per-cell scores plus an averaged headline. The benchmark explicitly targets both task-level correctness (does the robot do the requested thing?) and visual fidelity (is the video physically and structurally plausible?). Validation against human ratings yields Spearman 0.96.

RoVid-X is built via a four-stage data refinement pipeline designed in response to RBench’s diagnostic findings. The resulting corpus has 4M clips at 720p covering 1300+ skills with optical flow, multiple robotic forms, and diverse free-form captions. The full pipeline diagram and per-stage statistics are on the project page (Overview of RoVid-X Construction and Statistics).

The paper packages the benchmark, dataset, code, leaderboard, and a public Huggingface Space, framed as a “synergistic ecosystem of evaluation and data” for accelerating robotics-oriented video models.

Results

Headline leaderboard (Avg. score, top 7 of 25): Wan 2.6 0.607, Seedance 1.5 Pro 0.584, Wan 2.5 0.570, Hailuo v2 0.565, Veo 3 0.563, Seedance 1.0 0.551, Kling 2.6 Pro 0.534 [Table 1].
Robotics-specific models underperform general commercial video models: Cosmos 2.5 (0.464) is best of the robotics-specific group but ranks 9th overall; UnifoLM-WMA-0 (0.123) and Vidar (0.206) finish last [Table 1].
Task dimension hardest → easiest (averaged across all 25 models): reasoning and long-horizon tasks are hardest; spatial and manipulation easier; embodiment dimension shows quadruped/humanoid generally scored higher than dual-arm/single-arm — likely reflecting how prevalent each form is in pretraining corpora [Table 1].
Sora v2 Pro at rank 17 (0.362) underperforms multiple open-source models on robotics tasks despite leading on general-purpose video generation arenas [Table 1].
RoVid-X: 4M clips at 720p with 1300+ skills, optical-flow annotations, diverse robotic forms, and diverse captions — first dataset to combine all four properties [Comparison of Representative Robotic Video Datasets].
Automatic-metric validation: Spearman 0.96 with human ratings [Abstract].

Why it’s interesting

ReVidGen fills a specific gap in the World Foundation Models cluster — the wiki has filed several physical-realism benchmarks (Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench), Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning, RISE-Video: Can Video Generators Decode Implicit World Rules?) and robotics-specific WFM systems (Cosmos, GR00T, Veo-as-simulator at Evaluating Gemini Robotics Policies in a Veo World Simulator), but no benchmark previously evaluated video generators on robot-oriented tasks across both task and embodiment axes. RBench is also notable as the first filed benchmark to score the robotics-specific model line (Cosmos, DreamGen, Vidar, UnifoLM-WMA-0) head-to-head with general video models — and the finding that the robotics-specific line trails general commercial models on a robotics-oriented benchmark is sharp. On the data side, RoVid-X at 4M 720p clips with optical flow + diverse forms + diverse captions extends the Synthetic Training Data / open-data thesis of EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World from the egocentric-human side to the multi-embodiment robot side. The 0.96 Spearman-to-human number is comparable to the strongest prior VLM-judge results (e.g. PhyGenEval at ~0.65) and worth interrogating for methodology if the pipeline is reusable internally.