OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence

TL;DR

OneVision-Encoder (OV-Encoder) is a ViT-based image+video encoder from LMMs-Lab that replaces uniform frame sampling with codec-aligned sparse patch selection: HEVC motion vectors and residuals decoded from the video bitstream pick the 3–25% of patches with the most temporal “surprise,” and only those are encoded. A shared 3D-RoPE handles the resulting irregular token layouts, and training uses a large-scale cluster-discrimination objective over ~1M concept centroids (no paired captions required). Plugged into a Qwen3-4B LMM, it beats Qwen3-ViT and SigLIP2 across 16 image/video/document benchmarks while using far fewer visual tokens and ~20× less pretraining data than Qwen3-ViT.

Key claims

Codec-derived motion vectors and residual energy give a principled, training-free saliency signal at the ViT-patch level: each P-frame patch is scored by aggregating decoded motion magnitude and luma-residual energy, and a fixed top-k fraction is kept [§2.1].
A 64-frame clip with GOP=32 and a 2048-token budget yields an 87.5% reduction vs dense processing of all 16,384 patches while preserving full temporal coverage of the clip [§4.1, §2.2].
Codec Patchification unifies three input modes under one ViT with shared 3D-RoPE: dense video-codec (I-frames full + P-frames sparse), chunk-wise patchification (1 frame per uniform chunk), and single-image spatial patchification [§2.2, §2.5].
The training objective is a multi-positive cluster-discrimination loss over a unified bank of ~2M image centroids and ~400k video centroids, derived offline by clustering frozen MetaCLIP-h14 features on LAION-400M + COYO-700M (images) and HowTo100M + Panda-70M + K710 + SSV2 (videos) [§2.3, §3].
Under a Qwen3-4B LMM with identical instruction-tuning corpus, OV-Encoder beats SigLIP2 across all 16 video/image/document benchmarks tested [§4.2, Table 2].
OV-Encoder outperforms Qwen3-ViT (decoupled, post LLaVA-OneVision-1.5 adaptation) on 16 benchmarks despite ~100B pretraining caption tokens vs Qwen3-ViT’s >2.1T — a >20× pretraining-data gap [§1, §4.2].
On video LMM tasks the average improvement over Qwen3-ViT is 4.1%; on VideoMME, MVBench, NExT-QA, PerceptionTest, LongVideoBench-Val-Video the codec variant wins [Table 2].
Under attentive probing on Diving-48 at a 2048-patch budget, OV-Encoder beats SigLIP2 by 17.1% and DINOv3 by 8.1% Top-1 [§1, §4].
Codec-style patch redistribution dominates dense frame sampling at every fixed token budget tested (512 / 1024 / 2048 / 4096 patches on Diving48 and Perception Test), and the gap widens as the budget tightens — codec at 512 patches (46.5% on Diving48) already beats SigLIP2 at 4096 patches (56.7%) is closed substantially, with codec at 2048 patches reaching 67.2% vs SigLIP2’s 50.1% [Table on project page].
Efficiency–accuracy are positively correlated under codec patchification: aggressive token reduction (96.9% on Diving48) yields higher accuracy than the dense baseline, not lower [project page].

Method

OV-Encoder is built around three input formulations consumed by a single bidirectional ViT with 3D-RoPE. (1) Dense Video-Codec Patchification: the input video is decoded through an HEVC pipeline that partitions it into Groups of Pictures (GOP=32). Each I-frame is patchified fully (RGB), and for each P-frame the codec exposes per-CU motion vectors and a luma residual; these are broadcast to pixel-level fields, aggregated per patch, and used to select a fixed top-k of “surprising” patches. (2) Chunk-wise Patchification: the video is split into N equal chunks, one frame per chunk is randomly sampled, and full spatial patchification is applied — a sparse-temporal complement to codec-mode for cases where motion vectors are unavailable. (3) Single-Image Spatial Patchification: standard row-wise patchification for static images. 3D-RoPE assigns each patch a (t, x, y) relative offset that adapts per input type (temporal offset under codec mode, chunk-index offset under chunk mode, pure spatial offset for images).

Training is self-supervised cluster discrimination. Offline, a frozen MetaCLIP-h14 encoder embeds images and 16-frame video clips, and embeddings are clustered into 2M image centroids and 400k video centroids using single-step k-means. Each sample is assigned its Top-10 nearest centroids as multi-positive pseudo-labels, and the loss is a softmax over a sampled subset of negatives per granularity, computed separately for object-level (image) and motion-level (video) clusters and summed. The encoder is trained in two stages on 128 A800 GPUs: Stage 1 is 13B image samples at 224² with 2M classes; Stage 2 adds OCR + video data at 448² (image) and 224² (video), with a 1:1 video-to-image ratio and a fixed 2048 visual-token budget per 64-frame clip (512 tokens to two I-frames, 1536 distributed across 62 P-frames by motion saliency).

Results

vs SigLIP2 (Qwen3-4B LMM, identical setup): OV-Encoder wins across all 16 benchmarks. Headline video deltas: MVBench 52.4 vs 47.2, NExT-QA MC 75.6 vs 70.6, VideoMME 54.1 vs 46.8, PerceptionTest 60.3 vs 56.0, LongVideoBench-Val-Video 50.4 vs 45.2 [project page Table].
vs Qwen3-ViT (decoupled, after LLaVA-OneVision-1.5 adaptation): OV-Encoder-Lang variant beats Qwen3-ViT on 14/16 benchmarks (Qwen3-ViT wins on DocVQA 85.1 vs 83.2, and OCRBench 706 vs 657 where Qwen3-ViT’s massive caption pretraining still helps). Average video improvement: +4.1% [§1, Table 2].
Attentive probing (ViT-L, 8-frame, 10 epochs, action recognition): AVG 60.2 (codec) / 58.4 (frame) vs DINOv3 58.0, SigLIP2 53.1, MetaCLIP2 50.2, AIMv2 53.8. Diving-48 specifically: 67.2 codec / 57.6 frame vs DINOv3 58.6, SigLIP2 50.1 [project page Table].
Efficiency × accuracy (Diving-48, fixed token budget): codec at 512 patches (96.9% reduction) = 46.5; codec at 2048 patches (87.5% reduction) = 67.2 — both above dense SigLIP2 at the same token budget (28.1 and 50.1 respectively). Same pattern on Perception Test [project page Table].

Why it’s interesting

For Luma this matters on two axes. First, video-token efficiency: OV-Encoder is the strongest “principled” reply to the question RLT raised — once you accept that video tokens are massively redundant, what’s the right detector for which tokens to keep? RLT used pixel-L1 on consecutive patches as a cheap heuristic; OV-Encoder uses motion vectors and residuals that the codec is already computing for free at decode time, and trains a representation around the resulting irregular layout (rather than just pruning at inference). This is the strongest “use what the codec already knows” architectural play on file, and the efficiency–accuracy correlation (more aggressive pruning → better accuracy) suggests the saliency signal is actually identifying the supervisory signal, not just throwing away cheap tokens. Second, perception backbone for VLAs/world-model evaluators: a public, open-weights ViT-L that beats SigLIP2 / DINOv3 / Qwen3-ViT under controlled LMM probes at 1/20th the caption-token budget is a strong candidate to replace SigLIP2 as the default vision encoder in any Luma system that needs to understand video (judge/evaluator MLLMs, conditioning encoders, agent perception). Worth tracking whether the codec-decoding step becomes a bottleneck at inference and whether the saliency signal generalizes to AI-generated video (where the H.264/HEVC bitstream may not be available or may have degenerate motion-vector statistics).