CoPE-VideoLM: Codec Primitives For Efficient Video Language Models

TL;DR

CoPE-VideoLM replaces dense per-frame RGB tokenization in a VideoLM with codec-native primitives: I-frames are kept as full RGB tokens, P-frames are encoded into compact “δ-tokens” derived from H.264-style motion vectors and residuals. Two lightweight transformer encoders (motion branch + residual branch) are first pre-trained to align with a frozen SigLIP image-encoder space via patch-wise regression against the warped+residual reconstruction of the target frame, then plugged into LLaVA-Video-7B for end-to-end fine-tuning. The result reduces token usage and time-to-first-token substantially while matching or beating the LLaVA-Video-7B baseline across general VQA, temporal-reasoning, long-form, and spatial-scene benchmarks — the codec’s GOP structure is repurposed as an inductive bias for efficient temporal coverage.

Key claims

VideoLMs that decode every frame to RGB ignore the redundancy that the codec already factored out; encoding only I-frame RGB tokens + δ-tokens from motion vectors and residuals yields equivalent task performance at a fraction of the token count [§1, §3].
The δ-encoder uses a multi-layer MLP + motion transformer for motion vectors (aggregated via learnable query tokens) and a ResNet-18 + residual transformer for residuals, producing N_m motion tokens and N_r residual tokens per P-frame, with N_m = N_r in the default config [§3.2, Fig. 3].
P-frame fusion lets consecutive P-frames be encoded jointly relative to the GOP’s I-frame instead of frame-to-frame, trading temporal resolution for token budget; at 8 FPS native with GOP=16 and fusion=8 the system runs at effectively 1 FPS for fused tokens [§3.2].
δ-encoder pre-training uses a “reference” transformer (image tokens + motion tokens, analogous to warping) followed by a “warped” transformer (warped tokens + residual tokens, analogous to residual addition) and minimizes patch-wise L2 against frozen SigLIP features of the target frame [§3.3, Eqs. 9–12].
The reference-conditioned branches are dropped after pre-training — at VideoLM fine-tuning the δ-encoder runs without any RGB reference frame for P-frames, which is what gives the inference-time compute and memory reduction [§3.3].
Token-stream layout in the LLM input is an ordered interleave of I-frame and P-frame tokens (z₁_I, z₁_{P,1}…z₁_{P,k}, z₂_I, …), so no architectural change to the LLM is needed; standard instruction-tuning next-token loss is reused [§3.2 Eq. 8, §3.3].
On PerceptionTest, NextQA, and ActivityNet-QA the codec-aware tokenization matches the same-sampling LLaVA-Video baseline and closes the gap to the next denser sampling tier while using over an order of magnitude fewer tokens; the largest relative gains are on PerceptionTest and NextQA, which the authors read as stronger reasoning under tight token budgets [§4.2, Table 1].
Among open-source 7B VideoLMs, CoPE-VideoLM is top on PerceptionTest, ActivityNet-QA (general VQA) and on TempCompass, TOMATO, CVRR-ES (temporal reasoning), and best on Video-TT, Video-MMMU, LVBench (long-form / instruction-following) [§4.3, Tables 2–4].
Compared to prior codec-based VideoLM work — Video-LaVIT discretizes motion vectors into language-like tokens, EMA discards residuals and aggregates I-frame + motion vectors per GOP into a fixed-length summary — CoPE-VideoLM keeps both motion and residuals as a unified, variable-length, temporally ordered token sequence that adapts to content dynamics [§2 “Compressed Video Representation”].
Native frame-rate processing is mandatory because codec primitives are defined relative to the previous frame; arbitrary frame skipping breaks dependencies. P-frame fusion is the codec-correct way to subsample [§3.2].

Method

A standard MPEG-4 / H.264 / HEVC bitstream organizes frames into a Group of Pictures (GOP): one I-frame (full RGB, encoded independently) followed by P-frames (block-wise motion vectors + residuals relative to the previous frame), optionally B-frames (bidirectional, dropped here for causality). CoPE-VideoLM keeps this structure as-is. Each I-frame I_t passes through the frozen SigLIP vision encoder f to produce dense tokens z^I_t. Each P-frame P_t = (M_t, R_t) — where M_t is the block-wise motion-vector tensor and R_t the residual tensor — passes through the δ-encoder g, producing compact δ-tokens z^P_t.

The δ-encoder has two branches. The motion branch processes M_t with a multi-layer MLP over a grid of size G_m, then a small “motion transformer” with N_m learnable query tokens that cross-attend to the MLP features and aggregate them into N_m motion tokens. The residual branch embeds R_t with a lightweight ResNet-18 over the same grid, then a “residual transformer” with N_r query tokens that aggregate to N_r residual tokens. Default: N_m = N_r, and only these compressed tokens enter the LLM.

Pre-training (stage 1) is the conceptual heart of the method. The δ-tokens are useless to the VideoLM unless they live in the same embedding space as the SigLIP image tokens. To force alignment, two extra transformers are stacked on top of the δ-encoder only during pre-training: a “reference” transformer that takes (image tokens of the reference frame, compressed motion tokens) and produces warped tokens (the analogue of motion compensation, Eq. 1’s warping step), and a “warped” transformer that takes (warped tokens, residual tokens) and produces final reconstructed tokens (the analogue of adding the residual). These final tokens are regressed patch-wise against the frozen SigLIP features of the actual target P-frame’s RGB image, via L2. Pre-training corpus: 5s clips from PerceptionTest’s training set.

Integration (stage 2) drops the reference/warped transformers. Only the δ-encoder (motion branch + residual branch + their query aggregators) remains; it is plugged into LLaVA-Video-7B with SigLIP + Qwen2 as the LLM, and the whole pipeline is fine-tuned end-to-end on LLaVA-Video-178K (~1.6M VQA samples) for ~5k GPU-hours (16× A100-80G × 13 days). Token stream into the LLM is interleaved I/P tokens; no architectural change to the LLM.

Training-time defaults: re-encode videos to MPEG-4 at 8 FPS, GOP=16, P-frame fusion=8 (so the effective fused rate is 1 FPS). For videos longer than 5 GOPs (~10s), uniform GOP sampling first, then encode only the I/P frames within those sampled GOPs.

Results

Token efficiency. Up to an order-of-magnitude reduction in token count vs. LLaVA-Video-7B at matched or improved accuracy; corresponding TTFT reduction is “up to” the same magnitude (specific percentages are in the abstract but the LaTeX-rendered numbers are missing in the extracted abstract — refer to the paper) [Abstract, §4.2].
General VQA. Top among open-source 7B VideoLMs on PerceptionTest and ActivityNet-QA; competitive on NextQA and VideoMME (w/o and w/ subtitles), despite training on a smaller corpus than several baselines (LLaVA-OneVision, Apollo, Oryx) [Table 2].
Temporal reasoning. Top among open-source models on TempCompass (test MCQ), TOMATO, and CVRR-ES. On MVBench, beats LLaVA-Video-7B base but trails some baselines — the authors attribute this to MVBench’s appearance-heavy character, where the LLaVA-Video-7B base itself underperforms peers [Table 3, §4.3 “Temporal Reasoning”].
Long-form / instruction-following. Best open-source result on Video-TT, Video-MMMU, and LVBench; competitive on LongVideoBench. Compressing P-frames into compact δ-tokens lets the same context window cover substantially more temporal content [Table 4, §4.3 “Long-form and Instruction-following”].
Effectiveness ablation (Table 1). At “1 keyframe / GOP + 7 P-frames / GOP”, CoPE-VideoLM closes the gap to LLaVA-Video’s “2 keyframes / GOP” sampling while using a fraction of the tokens. At “4 keyframes / GOP + 4 P-frames / GOP” it often surpasses the dense “64-keyframes-total” LLaVA-Video baseline. The Pareto-optimal frontier (accuracy × token-count) is pushed outward [§4.2, Table 1].
Spatial scene understanding (Supp. Sec. B): evaluated on ScanQA and SQA3D, results not fully visible in the fetched abstract but listed in the paper’s contribution claims.

Why it’s interesting

This is the third filed paper exploiting video temporal redundancy as architecture rather than as a post-hoc compression trick — alongside Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (RLT, discriminative side, pixel-L1 pre-network drop) and Pretraining Frame Preservation in Autoregressive Video Memory Compression (PFP, generation side, learned memory encoder). What CoPE adds is that the redundancy detector is already computed by every consumer device — H.264 motion vectors and residuals come for free out of the demux, so the “encoder” upstream of the LLM is a lookup, not a learned model. RLT does pixel-L1 in PyTorch; CoPE-VideoLM just uses what the codec produced. The pre-training trick — using a reference+warped transformer pair to enforce that δ-tokens live in image-encoder space, then dropping those auxiliary heads at integration — is a clean way to do modality alignment without touching the LLM. The structural argument that codec primitives are themselves the inductive bias for temporal reasoning is the more interesting framing: token-reduction methods like ToMe or RLT detect-and-drop redundancy; CoPE skips detecting it because the codec already did. For Luma’s long-video understanding work, this collapses the “video tokenizer” research direction with the H.264 spec — a strong prior on what video redundancy looks like is sitting in every MP4 file.