OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains

TL;DR

OmniVideo-100K is a 100K-sample instruction-tuning dataset for audio-visual question answering on omni-modal LLMs (VITA-1.5, Qwen2.5-Omni-7B, Qwen3-Omni-30B), produced by an automated two-stage pipeline that fixes two failure modes of prior video-caption-QA pipelines: (1) decoupled audio/visual captioning that severs cross-modal entity associations, and (2) single-step QA synthesis that yields only localized, intra-clip questions. The dataset and its human-verified companion benchmark OmniVideo-Test deliver up to +20.59% on OmniVideo-Test and +12.64% transfer to Daily-Omni and JointAVBench after fine-tuning open omni models.

Key claims

Standard video-caption-QA pipelines segment videos into short clips and describe audio and visual modalities independently, severing inherent sound–source associations and producing inconsistent entity descriptions across segments [§1].
Coupling long-text comprehension with QA synthesis in one step localizes questions to single events and prevents long-horizon temporal or deep cross-modal reasoning [§1].
Entity-Anchored Video Scripting restructures a video into a script containing a global summary, a main entity list, and segment-wise audio-visual descriptions; the entity list acts as a global prior that enforces cross-segment referential consistency and reconstructs the audio-visual association severed by decoupled captioning [Abstract, §1].
Clue-Guided QA Generation is a two-step prompting procedure that first mines cross-segment multimodal clues from the script and then generates QA pairs grounded in those clues, lifting questions from local-event to multi-clue evidence-chain reasoning [Abstract, §1].
Fine-tuning VITA-1.5, Qwen2.5-Omni-7B, and Qwen3-Omni-30B on OmniVideo-100K improves OmniVideo-Test scores by up to 20.59% absolute [Abstract].
The same fine-tuning transfers off-distribution to established audio-visual benchmarks, with up to 12.64% absolute gains on Daily-Omni and JointAVBench [Abstract].

Method

The data engine is a two-stage prompted pipeline over uncurated long-form video. Stage 1 — Entity-Anchored Video Scripting segments each video and writes a structured script with three fields: a top-level summary, a main entity list (the global prior), and segment-wise descriptions that fuse the audio and visual streams. The entity list is constructed once and referenced from every segment, which forces cross-segment referential consistency (the same person/object/sound source has the same name everywhere) and re-binds sounds to their visual sources within each segment. Stage 2 — Clue-Guided QA Generation prompts the QA generator to first extract a small set of “clues” — cross-segment, multimodal facts derived from the script — and only then write the QA pair conditioned on those clues. This separates what to ask about (clue mining) from how to phrase it (QA writing), and by construction biases generated questions toward long-range and cross-modal evidence chains rather than localized intra-clip facts. The pipeline is applied at scale to produce the 100K-sample instruction-tuning corpus OmniVideo-100K plus a smaller human-verified evaluation split OmniVideo-Test.

Results

Up to +20.59% absolute on OmniVideo-Test after fine-tuning each of VITA-1.5, Qwen2.5-Omni-7B, and Qwen3-Omni-30B on OmniVideo-100K [Abstract].
Up to +12.64% absolute transfer to existing benchmarks Daily-Omni and JointAVBench, indicating the gains are not OmniVideo-Test-specific [Abstract].
The three target models span ~7B to ~30B and three different omni architectures, so the gain is reported as architecture-agnostic rather than tied to a single model family [Abstract].

Why it’s interesting

This is another instance of the “verification + multi-step synthesis gate carries more weight than raw data scale” pattern documented in Synthetic Training Data — here the gate is the entity list (cross-segment consistency check) plus the clue-mining step (forces long-range cross-modal questions), not a downstream verifier. It is structurally close to Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning‘s hindsight curation and HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning‘s multi-hop synthesis, but applied to audio-visual QA rather than visual-only or text-only QA — a regime where decoupled per-modality processing is exactly the documented failure mode in Do Audio-Visual Large Language Models Really See and Hear?, which showed audio-visual LLMs largely ignore audio when video is informative. OmniVideo-100K supplies the instruction-tuning data that the Unified Multimodal Models omni-understanding line (Qwen2.5-Omni, Qwen3-Omni, VITA-1.5) needs to fix that very gap. A natural follow-up is whether the gains here actually move the AVLLMs in Do Audio-Visual Large Language Models Really See and Hear?‘s counterfactual-audio probe — i.e., whether the new questions force the omni models to use audio rather than just be evaluated on it.