RISE-Video: Can Video Generators Decode Implicit World Rules?

TL;DR

RISE-Video is a reasoning-oriented benchmark for Text-Image-to-Video (TI2V) generators: 467 human-annotated samples across eight knowledge/capability categories (Commonsense, Subject, Perceptual, Societal, Logical, Experiential, Spatial, Temporal), scored along four metrics — Reasoning Alignment (RA), Temporal Consistency (TC), Physical Rationality (PR), Visual Quality (VQ) — by an LMM-driven automated pipeline that emulates human raters. The headline finding from evaluating 11 representative TI2V models is that even frontier closed systems (Hailuo 2.3, Veo 3.1, Sora-2) hit a hard ceiling on the strict reasoning-accuracy axis: top-1 accuracy is 22.5%, and the open Wan/Hunyuan/Kandinsky/CogVideoX tier sits well below 15%. The benchmark explicitly targets the gap between surface-level aesthetics (where weighted scores cluster near 60–80%) and the deeper question in the title — whether a video generator has actually decoded implicit world rules — by separating a relaxed weighted score (W.Score) from a strict accuracy number.

Key claims

TI2V generators are deficient on a reasoning axis that is not exposed by aesthetic / per-frame quality benchmarks: across 11 models, Visual Quality clusters between 74.5% and 96.2% while strict accuracy stays at 22.5% or below [Scoreboard].
The benchmark spans 467 human-annotated samples covering eight categories — Commonsense Knowledge, Subject Knowledge, Perceptual Knowledge, Societal Knowledge, Logical Capability, Experiential Knowledge, Spatial Knowledge, Temporal Knowledge — providing per-category breakdowns rather than a single scalar [Introduction].
The evaluation protocol is four-dimensional — Reasoning Alignment (RA, weight 0.4), Temporal Consistency (TC, 0.25), Physical Rationality (PR, 0.25), Visual Quality (VQ, 0.1) — with RA dominating the weighted score, deliberately under-weighting VQ [Scoreboard footnote].
Reasoning Alignment is operationalized via per-sample human-designed questions plus a manual frame-sampling strategy (extra_frame, questions fields); Physical Rationality optionally evaluates time-reversed sequences (reverse flag) [HF dataset card].
Frontier closed models lead but do not dominate: Hailuo 2.3 ranks 1st at W.Score 79.4 / Acc 22.5; Veo 3.1 2nd at 76.4 / 22.3; Sora-2 3rd at 77.0 / 21.3; Wan 2.6 is competitive at 77.8 / 21.3 [Scoreboard].
Open-weight models trail substantially on accuracy despite competitive VQ: Wan 2.2-I2V-A14B at Acc 11.4 (W.Score 63.9), HunyuanVideo-1.5-720P-I2V at 8.6 (60.4), Kandinsky-5.0-I2V-Lite at 7.5 (59.3), Wan 2.2-TI2V-5B at 5.4 (57.8), CogVideoX 1.5-5B at 1.9 (49.5) [Scoreboard].
The evaluation pipeline is LMM-as-judge — an OpenAI-API-compatible endpoint (GPT_URL, GPT_KEY) consumes extracted frames + per-sample questions and emits both a relaxed weighted score and a strict accuracy [README Evaluation; eval.py].

Method

RISE-Video defines a TI2V task: given a first-frame image and a text prompt, generate a video that satisfies an implicit world rule (e.g. commonsense physics, subject knowledge, logical consequence, temporal causality). The dataset ships 467 (image, prompt, task_id) triplets organized into eight knowledge categories, with each sample carrying additional supervision metadata used at evaluation time: an extra_frame sampling schedule that says which output frames to inspect, a questions list of human-written reasoning probes tied to that sample, and a reverse flag indicating whether Physical Rationality is to be checked on the reversed playback.

The evaluation pipeline runs in three stages. (1) Generated videos are organized as {MODEL}/{CATEGORY}/{TASK_ID} and paths written into the dataset JSON’s video_path field. (2) fps_clip.py reads extra_frame to extract the human-chosen frames into frame_path. (3) eval.py calls an LMM (GPT-style API) against the extracted frames + per-sample questions to score Reasoning Alignment; Temporal Consistency, Physical Rationality, and Visual Quality are scored alongside. Two aggregates are reported: a relaxed weighted score W.Score = 0.4·RA + 0.25·TC + 0.25·PR + 0.1·VQ, and a strict per-sample Accuracy (a sample is “correct” only when all reasoning checks pass). The 0.4 weight on RA is the design choice that lets the benchmark privilege reasoning over aesthetics.

Results

The 11-model leaderboard (Scoreboard table) places three frontier closed systems at the top — Hailuo 2.3 (RA 76.6 / TC 87.2 / PR 71.0 / VQ 92.0 → W.Score 79.4, Acc 22.5%), Veo 3.1 (64.9 / 86.0 / 78.9 / 91.9 → 76.4, 22.3%), Sora-2 (64.0 / 92.2 / 76.3 / 92.2 → 77.0, 21.3%) — followed by Wan 2.6 (70.0 / 88.8 / 72.5 / 94.5 → 77.8, 21.3%) which matches Sora-2 on accuracy. Kling 2.6 (W.Score 72.1, Acc 19.5) and Seedance 1.5-pro (72.0, 17.6) fill the next tier.

Open-weight models drop sharply on strict accuracy while keeping VQ high. Kandinsky-5.0-I2V-Pro: 65.5 / 14.8. Wan 2.2-I2V-A14B: 63.9 / 11.4. HunyuanVideo-1.5-720P-I2V: 60.4 / 8.6. Wan 2.2-TI2V-5B: 57.8 / 5.4. CogVideoX 1.5-5B: 49.5 / 1.9. The Visual Quality column compresses everything to a 74.5–96.2 band — i.e., the discriminative signal lives almost entirely in RA (range 30.7–76.6) and Acc (range 1.9–22.5), not in VQ.

The implied message: even the best TI2V model in early 2026 fails the strict reasoning check on three out of four samples, while looking visually plausible on >90% of them.

Why it’s interesting

For Luma, this is the first filed benchmark that separates surface fidelity from reasoning on video generators — and the separation is large enough to be useful as a development signal rather than a marketing axis. RISE-Video sits in the same conceptual slot as VideoPhy and PhysicsIQ (Inference-time Physics Alignment of Video Generative Models with Latent World Models) but probes a wider set of knowledge categories (logical, societal, experiential) beyond physics. It also instantiates the VLM-as-Evaluator pattern at video scale — the same LMM-as-judge primitive the wiki has been tracking on the image side (ERNIE-Image-Aes, UnifiedReward-Flex, DiffThinker × MLLM-verifier) is now packaged as a 467-sample TI2V benchmark with per-sample questions. The open question the wiki has been carrying — does the VLM-as-evaluator paradigm survive at video scale? — gets a partial answer: yes for relative ranking (it discriminates 1.9% to 22.5% accuracy across 11 models), but the absolute reliability of LMM-driven scoring on video reasoning is itself unvalidated against human raters in the public artifact.