Vidi2: Large Multimodal Models for Video Understanding and Creation

TL;DR

Vidi2 is ByteDance’s 12B multimodal model (Gemma-3 backbone) for long-form video understanding aimed at video-editing workflows. It introduces end-to-end spatio-temporal grounding (STG) — given a text query, predict both time ranges and per-second bounding boxes in a single decoder output — and adds open-ended video QA on top of the prior temporal-retrieval capability. On the team’s own VUE-STG and VUE-TR-V2 benchmarks (manually annotated, videos up to >1h), Vidi2 substantially outperforms Gemini 3 Pro (Preview), GPT-5, and Qwen3-VL-32B on temporal and spatio-temporal grounding while staying competitive on standard video QA. The Vidi2.5 follow-up (project page only) adds RL post-training, a Vidi2.5-Think plot-reasoning variant, and a Vidi-Edit post-trained model that emits structured editing plans (narrative, voiceover, audio, visual edits).

Key claims

Vidi2 outputs full spatio-temporal tubes (timestamp → bounding box at 1 fps) in a unified text format, which Gemini 3 Pro (Preview), GPT-5, and Qwen3-VL-32B cannot do in one shot under their native APIs [§4.1.1].
On VUE-STG overall, Vidi2 reaches vIoU 32.57 vs 4.61 (Gemini 3 Pro Prev.), 5.47 (GPT-5), 5.12 (Qwen3-VL-32B); tIoU 53.19 vs 27.50 / 16.40 / 25.91 [Tables 5–6].
On VUE-TR-V2 overall, Vidi2 reaches tIoU 48.75 vs 37.58 (Gemini 3 Pro Prev.) and 17.15 (GPT-5); margin grows on Long (tR 62.56 vs 39.36 / 16.05) and Ultra-Long (>30 min) buckets [Table 7].
The architecture re-designs an adaptive token-compression strategy so the same 12B model handles seconds to >1h videos; a single image is treated as a 1-second silent video for a unified interface [§2 Architecture].
VUE-STG is a new benchmark: 982 videos / 1,600 queries / 12,147 boxes spanning 204.79 hours, manually annotated at 1 fps; queries are mostly noun-phrase rewrites to reduce ambiguity; tubes can be temporally fragmented to reflect cuts/occlusions [§3.1.1].
The proposed primary metric is vIoU (spatial IoU averaged over the temporal union of predicted and GT tubes), with vIoU-Int (over the temporal intersection), vP, and vR for diagnostic decomposition [§3.1.2].
Vidi2.5 (project page only, not in the arxiv v1) adds RL training, a Vidi2.5-Think variant that beats Gemini 3 Pro Preview on fine-grained character understanding on the new VUE-PLOT benchmark, and a Vidi-Edit model that emits machine-readable editing plans for downstream rendering [project page].
Proprietary VLMs degrade sharply on medium-length videos and small-object scenarios: Gemini 3 Pro Prev. drops to vIoU 3.05 on Medium-length videos, GPT-5 to 1.44 — Vidi2 holds 28.18 [Table 6]. GPT-5’s vIoU-Int (33.64) being far higher than its vIoU (5.47) shows it can localize when it commits to a frame but only covers a tiny fraction of the ground-truth time range.

Method

Vidi2 stacks a multimodal encoder (text + visual + audio) on a Gemma-3 12B LLM backbone, inheriting the temporal-aware multimodal-alignment pretraining recipe from Vidi 1 but with a redesigned adaptive token-compression strategy that keeps token budgets bounded across ultra-short to ultra-long videos. Training data combines (a) synthesized STG data built by lifting image-level spatial grounding annotations into spatio-temporal tubes, (b) a curated real-video STG corpus with manual time/bbox labels, (c) expanded temporal-retrieval SFT data, and (d) generic image+video QA data added during post-training. Inference produces a sequence of (timestamp, normalized box) pairs at 1 fps; for fair comparison the authors design task-aligned prompts so Gemini 3 Pro Preview, GPT-5 (≤256 sampled frames), and Qwen3-VL-32B (interleaved timestamp/frame pattern) all emit comparable outputs, then normalize timestamps to integer seconds and boxes to [0,1] before scoring. Vidi2.5 extends the post-training with RL on grounding rewards and a thinking-mode variant; Vidi-Edit is a further post-trained head that emits structured editing plans (narrative structure, voiceover text, audio attributes, visual edit intents).

Results

STG (VUE-STG, primary metric vIoU): Vidi2 vIoU 32.57 vs Gemini 3 Pro Prev. 4.61 / GPT-5 5.47 / Qwen3-VL-32B 5.12; vIoU-Int 60.30 vs 16.59 / 33.64 / 18.47 [Table 6].
TR (VUE-TR-V2): Vidi2 overall tIoU 48.75 vs 37.58 / 17.15; tR 64.93 vs 56.30 / 26.63. On Ultra-Long videos (>30 min), Vidi2 tIoU 38.65 vs Gemini 21.19 / GPT-5 12.49 [Table 7].
Video QA (LVBench / LongVideoBench / VideoMME): Vidi2 is described as “competitive” with similar-scale open-source models — i.e. the win is on grounding, not on benchmark video QA scores [§4 / Abstract].
Failure modes: baselines degrade most on Medium-length videos (Gemini 3 Pro Prev. vIoU 7.77 Ultra-Short → 3.05 Medium) and on Small objects (Gemini 2.33, GPT-5 3.66, Qwen3-VL-32B 1.49 vs Vidi2 23.31) [Table 6].
Vidi2.5 (project page): improved STG and Plot-Reasoning; Vidi2.5-Think outperforms Gemini 3 Pro Preview on the Character track of VUE-PLOT with comparable Reasoning-track performance. No comparable Vidi2.5 numbers are in the arxiv v1.

Why it’s interesting

The Vidi line is one of the few public efforts framing VLMs explicitly around an editing workflow — the model’s “output” is a structured spec for a downstream NLE, not a paragraph. That a 12B Gemma-based model beats Gemini 3 Pro Preview and GPT-5 by 10–30 absolute IoU points on long-video spatio-temporal grounding is a useful data point: it suggests the frontier proprietary systems are not yet optimized for (or trained on) the dense-tube output format, and that a focused open-source effort with a benchmark in hand can pull ahead on a narrow capability. The closest prior on the wiki is Qwen3-VL Technical Report (Qwen3-VL also pushes long-video and grounding; Vidi2 specifically beats its 32B variant on STG) and VideoCoF: Unified Video Editing with Temporal Reasoner (which executes editing given temporal reasoning — Vidi-Edit’s plans are the natural upstream input). Note the strong caveat that VUE-STG and VUE-TR-V2 are also built by the Vidi team, and the public artifact is mostly a benchmark + numbers paper — outside reproduction has not happened yet.