VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection

TL;DR

VideoEspresso is a fully-automated VideoQA dataset and accompanying hybrid-LVLM reasoning framework, presented as a CVPR 2025 Oral. The construction pipeline replaces frame-by-frame analysis with a semantic-aware redundancy filter (InternVL2-8B per-frame captions, BGE-M3 cosine-similarity de-duplication), then uses GPT-4o to generate QA pairs over 15-caption windows, followed by multimodal Chain-of-Thought annotations grounded with GroundingDINO bounding boxes and BGE-M3-retrieved temporal indices. The companion framework is a tiny LVLM (1B caption + 0.5B selector) that picks ~2.36 question-relevant core frames, fed to a two-stage instruction-fine-tuned 8.5B LVLM that first emits multimodal evidence then answers. It beats GPT-4o by 7.7 average accuracy points on the VideoEspresso benchmark while using ~14% of LLaVA-Next-Interleave’s TFLOPs.

Key claims

The semantic-aware frame filter (per-frame InternVL2-8B captions → BGE-M3 cosine similarity → LIFO de-duplication of adjacent near-duplicates) reduces frame redundancy before any GPT-4o call, replacing per-frame analysis with caption-level operations [§3.2].
QA pairs are generated by GPT-4o over groups of 15 consecutive frame captions, then passed through an LLM-based validator that checks for hallucinations, factual accuracy, and filters subjective open-ended items [§3.3].
Multimodal CoT annotations add (a) core-frame captions selected by GPT-4o, (b) key objects extracted from those captions, (c) a natural-language evidence string, and (d) GroundingDINO bounding boxes verified by CLIP-ViT-B/32 plus BGE-M3-retrieved temporal indices [§3.4, Eq. 3].
The Lightweight Selector is a sequential 1B-LVLM-captioner + 0.5B-LLM-selector that picks ≈2.36 core frames per question on average, vs 16 uniform / 128 LongVA frames [§4.1, Table 3].
The reasoning LVLM is trained in two SFT stages with explicit instructions: stage 1 elicits visual evidence, stage 2 conditions answer generation on that evidence [§4.2].
On the VideoEspresso benchmark (14 tasks, GPT-4o-based open-ended scoring with semantic similarity + distractor verification), the proposed 8.5B model reaches 34.1 average accuracy vs GPT-4o 26.4 and InternVL2-8B 28.7 [Table 1].
The subjective evaluation (Logic / Factuality / Accuracy / Conciseness, GPT-4o rater) gives 65.84 overall, within 0.3 of GPT-4o’s 66.13, and the proposed model wins Conciseness by ≈5.7 points [Table 2].
Computational efficiency: 9.26 TFLOPs per video vs 62.78 for LLaVA-Next-Interleave at the same input setting; the Selector replaces 16 uniform frames with 2.36 selected ones and reduces memory by 26–28 GB at the cost of 3–5 GB for the selector [§5.2, §5.3, Table 3].
Selector ablation: the 1B/1.5B selector pairing outperforms 1B/0.5B and outperforms uniform sampling when injected in front of GPT-4o, InternVL2, LongVA, and LLaVA-Next-Interleave — i.e. the selector is a plug-and-play module, with a clear quality knee at the selector LLM size [§5.4, Table 3 / adapting-selector table].
CoT ablation: providing GT CoT evidence at inference further boosts accuracy, showing CoT capacity is currently bottlenecked by the model’s CoT generation, not its CoT consumption [§5.3, Table 4].

Method

The construction pipeline is the contribution that distinguishes VideoEspresso from earlier auto-VideoQA work. Raw videos are pulled from seven existing sources covering news, movies, documentaries, animation, and education; FPS is set per video type (2–4 fps for dynamic content, 1 fps for static). InternVL2-8B captions every sampled frame into language space, then BGE-M3 embeds each caption, and adjacent-caption cosine similarities above a preset threshold trigger a LIFO drop — i.e. the later of two near-identical captions is discarded. The resulting concise caption sequence and the corresponding frames are then grouped into 15-frame windows.

GPT-4o is prompted over each 15-caption window to generate complex reasoning QA pairs while maintaining inter-caption semantic continuity; outputs are validated by a separate LLM that filters hallucinations, factual errors, and subjective open-ended items. The CoT annotation stage then re-prompts GPT-4o with the QA pair plus the caption group to extract (i) the captions of the most question-relevant core frames, (ii) the key objects within those captions, and (iii) a natural-language evidence sentence. Spatial grounding is supplied by GroundingDINO bounding boxes verified against captions by CLIP-ViT-B/32; temporal grounding is supplied by BGE-M3 retrieving the matching original captions, since GPT-4o-generated captions don’t directly string-match the source.

The Hybrid LVLMs Collaboration framework at inference time is two-stage. The Lightweight Selector first captions video frames with a 1B LVLM, then uses a 0.5B (or 1.5B) LLM to pick the captions whose content is most relevant to the question; only those frames are passed to the reasoning model. The reasoning LVLM is a 7B LLaVA-Next-Interleave backbone (8B total once vision-encoder weights are counted; paper reports 8.5B param count for the full system) instruction-tuned on the VideoEspresso CoT traces in two stages: stage 1 SFT teaches the model to emit evidence given the question, stage 2 SFT teaches answer generation conditioned on extracted evidence.

Results

The headline is a 7.7-point average accuracy gap over GPT-4o (34.1 vs 26.4) on the 14-task VideoEspresso benchmark while using only 2.36 frames vs FPS=3 [Table 1]. Open-source baselines (InternVL2-8B, Qwen2-VL-7B, mPLUG-Owl3, LLaVA-Next-Video, LongVA-DPO) all cluster in the 24-28% accuracy range; the proposed 8.5B model is the only open-source entry to clear 30% and is the only system to outperform Qwen-VL-Max by a substantial margin. The subjective evaluation (Logic / Factuality / Accuracy / Conciseness) places the system within 0.3 of GPT-4o overall, with Conciseness as the only sub-dimension where the open model exceeds the closed model.

The Selector-adaptation table is the most directly transferable result: pre-pending the 1B/1.5B Selector to any host LVLM (GPT-4o, InternVL2, LongVA, LLaVA-Next-Interleave) consistently improves benchmark accuracy while cutting TFLOPs by 6–50× and reducing input frame count from 16-128 to ~2.36. The two-stage CoT SFT is the second-most-transferable: the gap between SFT-with-evidence-extraction-then-answer and direct-SFT-answer is large enough that the authors point at SFT-side evidence elicitation as the load-bearing piece, with the GT-CoT ablation in Table 4 showing further headroom from better-quality evidence generation.

Why it’s interesting

The dataset itself sits cleanly in the Synthetic Training Data lineage — GPT-4o is the generator stack, an auxiliary LLM is the validation gate over QA pairs, and the CoT-annotation phase uses GroundingDINO + CLIP-ViT-B/32 as cross-model verifiers for spatial grounding. It predates and arguably prefigures the recipes used by Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning (DeepSeek-V3 reverse-curates traces; Qwen2.5-VL-7B is the hindsight gate) and Action100M: A Large-scale Video Action Dataset (Llama-3.2-Vision + Perception-LM-3B + GPT-OSS-120B Self-Refine over Tree-of-Captions) by about a year, with a structurally similar bet: a cross-model verification stack over frontier-LLM-generated annotations can substitute for human review at scale.

The Lightweight Selector also sits adjacent to Thinking with Modalities but as a learned compressor rather than an active-perception CoT — VideoEspresso picks frames once up-front, while Video-Thinker emits <grounding> tags interleaved with reasoning. The contrast is informative: VideoEspresso’s selector is task-agnostic relative to the reasoning model and is a feed-forward classifier; Video-Thinker’s grounding is RL-trained to be invoked by the same model that consumes it. The two designs bound the space — static compressor vs autoregressive self-invocation — and the apples-to-apples comparison (same base model, same eval) is missing in the wiki so far.