LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence

TL;DR

LLaVA-OneVision-2 (LLaVA-OV-2) is the 2.0 release of the LMMs-Lab LLaVA-OneVision VLM series, built on the OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence vision backbone. Its key architectural move is codec-stream tokenization: compressed video is treated as a continuous bit-cost stream, where decoded bit-cost dynamics define adaptive temporal groups and motion-residual cues pick salient spatial evidence into compact visual canvases — concentrating a fixed token budget on event-bearing frames rather than uniformly sampled ones. A shared 3D RoPE places codec canvases, sampled frames, and images in a unified spatiotemporal coordinate system. The model is trained on ~8M re-captioned video samples for pretraining and a 4M-sample spatial corpus for fine-tuning; a new JumpScore benchmark targets temporal localization in high-frequency repeated motion (the regime where uniform sampling fails most). LLaVA-OV-2-8B reaches 74.9 JumpScore mAP vs Qwen3-VL-8B’s 30.1, and wins +4.3 on video, +5.3 on spatial, and +15.6 J&F on tracking benchmarks at matched scale.

Key claims

Codec-stream tokenization treats compressed video as a continuous bit-cost stream, with bit-cost dynamics determining adaptive temporal groups and motion-residual cues selecting salient spatial evidence into compact visual canvases [Abstract].
Codec canvases, sampled frames, and images share one 3D RoPE coordinate system, so the model handles all three input modes under a single positional scheme [Abstract].
Pretraining uses approximately 8M re-captioned video samples; spatial fine-tuning uses a 4M-sample spatial corpus [Abstract].
JumpScore is a new temporal-localization benchmark for fine-grained grounding in high-frequency, densely repeated motion — a regime underrepresented by existing video evaluations [Abstract].
On JumpScore mAP, LLaVA-OV-2-8B scores 74.9 vs Qwen3-VL-8B’s 30.1 — a +44.8-point gap [Abstract].
Under matched visual-token budgets on JumpScore, codec-stream inputs improve temporal grounding over uniform frame sampling by +9.7 points [Abstract].
vs Qwen3-VL-8B at matched 8B scale across standard benchmarks: +4.3 average on video tasks, +5.3 on spatial tasks, +15.6 average J&F on tracking tasks [Abstract].
The OneVision-Encoder backbone uses Windowed Attention for efficient local computation while maintaining native input resolution [Abstract].

Method

LLaVA-OV-2 is the language-model half of a stack that sits on top of the OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence visual backbone. The vision encoder ingests video through an HEVC codec front-end: GOP boundaries and per-coding-unit bit-cost / motion-vector / residual signals are read from the compressed bitstream and used to (a) define adaptive temporal groups (instead of fixed GOPs or uniform frame sampling) and (b) pick which spatial patches inside each group carry the most “event” content, which are then packed into a compact visual canvas per group. Static frames and individual images go through the same encoder with the spatial paths only. All three modes — codec canvases, sampled frames, single images — share a 3D RoPE that exposes a unified (t, x, y) coordinate for every patch the LM sees, so the LM does not need separate position schemes per input type.

Training is staged. Pretraining covers ~8M re-captioned video samples; supervised fine-tuning adds a 4M-sample corpus focused on spatial reasoning (grounding, tracking, manipulation traces). Evaluation is done on existing video, spatial, and tracking benchmarks plus the new JumpScore benchmark, which targets temporal localization in clips with high-frequency repeated motion (e.g. jumping, drumming) where uniformly-sampled frames provably miss most events.

Results

JumpScore (new, temporal-localization mAP): LLaVA-OV-2-8B 74.9 vs Qwen3-VL-8B 30.1 — +44.8 points [Abstract].
Codec-stream vs frame sampling on JumpScore at matched visual-token budget: +9.7 mAP for codec-stream [Abstract].
Video tasks (standard benchmarks, 8B scale): +4.3 average over Qwen3-VL-8B [Abstract].
Spatial tasks: +5.3 average over Qwen3-VL-8B [Abstract].
Tracking tasks: +15.6 average J&F over Qwen3-VL-8B [Abstract].

Why it’s interesting

This is the LM that completes the codec-native VLM stack LMMs-Lab has been building. OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence showed the encoder side — codec motion vectors + residuals are a stronger redundancy detector than pixel-L1 (Don't Look Twice: Faster Video Transformers with Run-Length Tokenization) or uniform frame sampling — and OV-2 shows the same idea wins downstream in an 8B LM: +44.8 mAP over Qwen3-VL-8B on the temporal-grounding regime where uniform sampling is structurally broken (high-frequency repeated motion). It also lines up with the codec-primitives angle CoPE-VideoLM: Codec Primitives For Efficient Video Language Models explored on the VideoLM side, but pushed end-to-end through a frontier-scale VLM with public weights. For Luma this matters as a candidate perception backbone for video-evaluator VLMs and as evidence that “decode-time signals the codec already computed” beat “uniformly sampled frames + a bigger LM” on temporal grounding at fixed visual-token budgets. The dependence on having an actual codec bitstream (HEVC/H.264 motion vectors) remains the open caveat — AI-generated video without a faithful bitstream may not expose useful motion-vector statistics.