RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligence

TL;DR

RULER-Bench is a benchmark designed to evaluate the rule-based reasoning ability of video generation models, complementing existing TI2V benchmarks that focus on aesthetics, instruction adherence, and temporal coherence. The benchmark spans 40 representative tasks across six cognitive-rule categories with 622 high-quality annotated instances under both text-to-video and image-to-video paradigms. Evaluation uses a per-instance checklist scored by GPT-o3 across four metrics, achieving 85% alignment with human judgments. Across 10 SOTA video generators, the best model reaches only 48.87% on the rule-coherence metric — pinning a large reasoning gap that fidelity-side progress has not closed.

Key claims

Existing video-generation benchmarks (VBench, EvalCrafter, AIGC-Bench, UI2V-Bench) restrict evaluation to perceptual / basic instruction-following criteria and do not probe causal or abstract consequence inference [§1].
RULER-Bench covers 40 tasks across six rule categories under two paradigms (T2V, I2V), with 622 annotated instances [§Abstract, §3].
Evaluation is checklist-driven: per-instance objective questions across four metrics, rated on a discrete three-point scale (good / medium / bad) rather than subjective continuous scores [§4.1].
The GPT-o3 judge agrees with human annotators on 85% of 813 checklist questions, validating the LMM-as-judge protocol for this task family [§Abstract, Fig. 1].
The SOTA model achieves only 48.87% on rule coherence; all 10 evaluated models exhibit consistent limitations in rule coherence even when their fidelity-side metrics are high [§Abstract, §5].

Method

RULER-Bench is constructed around the claim that “rule-based reasoning” is a distinct, fine-grained axis of video-generation evaluation, separate from visual aesthetics, instruction adherence, and temporal coherence. The benchmark organizes 622 hand-annotated instances under six cognitive-rule categories spanning 40 representative tasks, split across the two dominant generation paradigms (T2V and I2V). Each instance is paired with a checklist of objective yes/no-style questions targeting four scoring metrics (including rule coherence). The judge is GPT-o3, which assigns one of three discrete tiers (good / medium / bad) per question. Reliability of the protocol is validated by manually annotating 813 questions and measuring per-question agreement between GPT-o3 and humans (85%).

Results

Top-line: best model 48.87% on the rule-coherence metric — i.e. SOTA video generators fail rule reasoning on more than half of probed instances [§Abstract].
10 SOTA T2V/I2V models evaluated; all show consistent rule-coherence limitations, with the gap between the best and worst models smaller than the gap between any of them and a human-aligned ceiling [§5].
Judge calibration: 85% question-level alignment between GPT-o3 and human annotators on the 813-question validation set [Fig. 1].

Why it’s interesting

RULER-Bench is the direct rule-based-reasoning sibling of World Foundation Models probes like RISE-Video — both released in the same window, both find SOTA video generators bunched well below human strict-accuracy ceilings (RULER’s 48.87% rule-coherence ceiling vs RISE-Video’s 22.5% strict accuracy across 11 TI2V models). The two together separate “video-as-WFM” claims into a fidelity axis (largely solved) and a reasoning axis (largely open). The 85% GPT-o3–human alignment number is also a useful datapoint for VLM-as-Evaluator — it sits between IGenBench’s reported r=0.90 Pearson for Gemini-2.5-Pro on infographic reliability and RISE-Video’s unmeasured judge calibration, and is the strongest filed evidence so far that a frontier LMM judge can carry benchmark-time evaluation on a video reasoning task.