Spotlight: Identifying and Localizing Video Generation Errors Using VLMs

TL;DR

Spotlight is a benchmark + task for fine-grained, temporally localized error detection in modern T2V outputs — 600 videos from Veo 3, Seedance, and LTX-2 generated from 200 diverse prompts, with 1,604 human-annotated errors across six categories (physics, appearance/disappearance, logical, motion, anatomy, adherence) each tagged with start/end time, severity (1-5), and natural-language reason. The authors evaluate Qwen3-VL (8B and 30B-A3B) and Gemini 2.5 Pro on the task; a multi-agent inference-time strategy (one query per error type) nearly doubles Qwen3-8B’s zero-shot score and matches zero-shot Gemini 2.5 Pro. Humans still outperform the best VLM by roughly 2× on the combined Similarity+Precision metric (0.508 vs 0.254), so frontier VLMs are not yet usable as fine-grained T2V error localizers.

Key claims

Modern T2V generators (Veo 3, Seedance, LTX-2) produce on average ~2.7 human-annotated errors per 6.5s clip, with mean error-segment length 2.49s — local, transient errors are the dominant failure mode at the current quality frontier, not global artifacts [§4.1, §4.2].
Among the three generators, Veo 3 produces the fewest annotated errors (507), Seedance close behind (517), and the open-weight LTX-2 the most (580); mean severity scores are nearly identical (Veo 3 3.48, Seedance 3.38, LTX-2 3.36) [§4.2].
Adherence errors are the most frequent category and have the longest duration, followed by physics and logical errors; appearance/disappearance, motion, and anatomy errors are shorter [§4.2, Fig. 2].
Zero-shot, Gemini 2.5 Pro reaches S+P = 0.250 / 12.6% match coverage on Spotlight, roughly 2× Qwen3-VL-8B-Instruct (0.148 / 8.0%) and Qwen3-VL-30B-A3B-Instruct (0.137 / 6.1%) [Table 2].
A multi-agent inference-time strategy — querying the model six times, once per error category, then merging — lifts Qwen3-8B from S+P 0.148 to 0.254, matching zero-shot Gemini 2.5 Pro and beating it on precision (49.8% vs 44.3%); sliding-window and two-stage (reason-then-localize) baselines also help but less [Table 2, §6.2].
High recall in zero-shot Qwen3 (≥0.9) is an artifact of localization failure — the model often predicts the entire clip duration as the error segment, so coverage is high but precision and S+P are low; the paper makes the case that S+P, not R, is the metric to track [§7, Fig. 7b].
Humans still beat the best VLM by ~2× on the combined Similarity+Precision metric (0.508 vs 0.254) and have markedly higher severity-3-to-5 detection accuracy than any model [§7, Fig. 6].
Models and humans disagree on which generator is worst: humans find LTX-2 most error-prone (≈600 errors), Qwen3-MA finds Seedance least (1,136 errors), Gemini 2.5 Pro finds Veo 3 least (722 errors) — i.e. T2V model leaderboards built on different VLM judges will not agree [§8.1, Fig. 5].
Error-type sensitivity differs systematically between judges: Qwen3-MA is stronger on physics (0.205) and logical (0.132) errors; Gemini 2.5 Pro is stronger on appearance/disappearance (0.099 vs 0.017) and adherence (0.246) [Table 3].
Dataset quality: human validators rated 93% of error reasons and 99% of segments correct on a 100-sample audit [§4.1].

Method

The Spotlight task takes an AI-generated video plus its source prompt and asks for a set of errors, each a tuple (start time, end time, category from six fixed labels, natural-language reason). Ground truth comes from a two-round crowdsourced annotation pipeline (CloudConnect by CloudResearch, $13/hr): round 1 labels all 600 videos, then the authors filter to “high-quality” annotators (≥3 errors per video, ≥8-word descriptions) for round 2; semantically equivalent overlapping annotations (IoU ≥ 0.6) are merged with an LLM. Prompts are sourced from StoryEval, VBench2, VidProM, BlackSwan, and LVD-2M to span creative and realistic content.

Evaluation uses maximum-weight bipartite matching between predicted and ground-truth error sets. Three pairwise scores are defined per (pred, gt) pair: temporal Precision P (fraction of prediction inside ground truth), temporal Recall R (fraction of ground truth inside prediction), and Reason Similarity S (GPT-4o-mini rates 1-10 how similar the predicted reason is to the ground-truth reason). The headline metric is S+P @ threshold 0.7, which requires both reason similarity and temporal precision to clear the threshold — picked because ground-truth segments are short (~2.49s mean), so precise localization matters more than coverage. The final score averages matched-pair scores; “match coverage” is the fraction of ground-truth errors successfully matched.

Baselines on Qwen3-VL-{8B, 30B-A3B}-Instruct and Gemini 2.5 Pro: (a) zero-shot — full video, JSON list of errors; (b) sliding-window — 2-second sub-clips at 4 fps with global timeline reassembly and same-VLM text-only deduplication; (c) sequential reason-then-localize — first identify errors without timestamps, then re-process the video per query to localize; (d) multi-agent — six independent queries, one per error type, results concatenated. Human upper bound comes from expert annotation of a random 80-video subset.

Results

Zero-shot @ S+P threshold 0.7: Qwen3-VL-8B-Instruct 0.148 / 8.0% coverage; Qwen3-VL-30B-A3B-Instruct 0.137 / 6.1%; Gemini 2.5 Pro 0.250 / 12.6% [Table 2].
Inference-time strategies on Qwen3-8B: sliding-window 0.189 / 10.0%; sequential 0.189 / 9.3%; multi-agent 0.254 / 13.3% — multi-agent essentially matches Gemini 2.5 Pro on S+P with 8B parameters, and beats it on Precision (0.675 vs 0.691 P at threshold; 49.8% vs 44.3% precision coverage) [Table 2].
Human baseline: 0.508 / 26.8% on S+P — roughly 2× the best VLM result [Table 2].
Per-category breakdown (S+P match coverage %): adherence is easiest for everyone (Gemini 20.8%, Qwen3-MA 21.4%, Human 45.0%); appearance/disappearance is the largest VLM-vs-human gap for Qwen3-MA (1.5% vs 12.1% human) and the largest VLM-vs-VLM split (Gemini 9.0% vs Qwen3-MA 1.5%) [Table 3].
Severity sensitivity: all methods improve with higher severity; the human-vs-VLM gap widens at severity levels 3-5 — VLMs do not gain as much from “more obvious” errors as humans do [§8.1, Fig. 6].
Generator ranking by detected-error count diverges by judge: Human → LTX-2 worst; Qwen3-MA → Veo 3 and LTX-2 both ~1,800, Seedance 1,136; Gemini 2.5 Pro → Veo 3 best (722) [§8.1, Fig. 5].

Why it’s interesting

Spotlight directly answers an open question on Video Generation Benchmarks — “how well do VLM judges actually localize errors in the new generation of T2V output?” — with a hard “not very well.” It is the first filed dataset that combines temporal localization, error-type labels, severity, and natural-language reasons in one ground truth, and the human-vs-VLM 2× gap on a small (200-prompt, 600-video) suite is a concrete datapoint for the “frontier VLMs are not yet calibrated T2V verifiers” thesis floated under VLM-as-Evaluator. It complements VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation (action-centric physics commonsense, holistic VLM judge) and RISE-Video: Can Video Generators Decode Implicit World Rules? (implicit world rules, LMM-judge with no localization) by isolating where in the clip the error lives — making it the localization-aware counterpart to the existing benchmarks. The multi-agent result is also useful evidence for the “decompose the rubric per category and aggregate” pattern that VBVR and IGenBench gesture at in their rule-based and atomic-yes/no variants — at 8B a category-sharded prompt matches a frontier closed judge.