Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing

TL;DR

AutoGaze is a 3M-parameter lightweight module that selects a minimal multi-scale subset of video patches before the ViT processes them, so that downstream ViTs and MLLMs can skip the spatiotemporal redundancy that dominates long, high-resolution videos. It is trained in two stages — next-token prediction on greedy-search gazing sequences, then on-policy GRPO with reconstruction-loss reward — and decides its own gazing length on the fly via a learned reconstruction-loss head. At a 0.7 reconstruction-loss target, it cuts visual tokens 4–100× and accelerates ViTs/MLLMs up to 19×/10×, enabling NVILA-8B-Video to scale to 1024-frame 4K videos and hit 67.0% on VideoMME and 52.6% on the new HLVid (5-minute, 4K) benchmark — beating GPT-4o and Qwen2.5-VL-7B.

Key claims

All prior MLLM token-reduction methods prune tokens inside the LLM or between ViT and LLM, leaving the ViT itself running on full-resolution input and bottlenecking scalability — AutoGaze removes patches before the ViT [§2, §4.4].
AutoGaze autoregressively decodes patch indices over frames, conditioning each frame’s selection on previously selected patches so it can avoid re-selecting redundant content; the decoder vocabulary is patch indices instead of words [§3.1].
A learned reconstruction-loss head predicts $L(\hat{x}_t, x_t)$ after each new patch and stops gazing for that frame once the predicted loss falls below a user-specified threshold — making gazing length adaptive per frame [§3.1].
Multi-scale gazing: the decoder can pick patches from multiple resolutions, allowing coarse coverage in low-detail regions and fine patches in detailed ones; ablating multi-scale roughly doubles gazing ratio and latency [§3.1, Table 4].
Post-training uses a simplified on-policy GRPO with reconstruction loss as reward; combining NTP pre-training + RL post-training reduces the gazing ratio at fixed reconstruction loss from 0.263 (no training) → 0.102 (NTP only) → 0.094 (both) [§3.2, Table 3].
Pre-training data: 250K videos with greedy-search gazing sequences from a larger 800K-video pool, sampled at 16 frames and 224 resolution; the same model handles any-resolution/any-duration videos at inference via 16×224×224 spatiotemporal tiling [§3.2, §3.3].
ViT integration requires only two changes: interpolating positional embeddings to multiple scales for multi-scale patch input, and repurposing the image ViT as a video ViT by attending across all 16 frames jointly [§3.3].
At a reconstruction-loss target of 0.7, gazing ratios drop with higher FPS/resolution; 30-FPS 4K video needs only 1% of patches [§4.2, Fig. 6].
On VideoMME (no subs), NVILA-8B-Video + AutoGaze at 1024 frames / 3584 resolution scores 67.0% vs. 64.2% for the base model — a +2.8 point gain at 4× frames and 8× resolution [Table 1].
On HLVid (the new 5-minute, 4K benchmark introduced in this paper), NVILA-8B-Video + AutoGaze scores 52.6%, +10.1 over base NVILA and +4.5 over VideoChat-Flash — and surpasses GPT-4o (49.3%) and Qwen2.5-VL-7B (48.1%) [Table 1].
vs. token-reduction baselines at 6.25% selection ratio: AutoGaze cuts ViT latency from 2.20s → 0.55s (4× speedup) while all other methods (S-Pool, ToMe, VisionZip, FastV, AKS, STORM, FastVID, LongVU, PruneVID, VideoChat-Flash) leave ViT latency essentially unchanged because they prune only inside/after the ViT [Table 2].
vs. heuristic gazing baselines (Random, RGB-Diff, Optical-Flow via SEA-RAFT): AutoGaze reaches reconstruction loss 1.0 with ~5% of patches vs. ~15% for Random; RGB-Diff and Optical-Flow underperform Random because they fixate on the first frame’s padding artifact [§4.4, Fig. 9].
AutoGaze prioritizes high-optical-flow patches (motion = new information) and selects finer scales in high-Laplacian-variance regions (more detail); generalizes OOD to CCTV, robot, identity-swap (created with Luma Ray2 Flash), and style-transferred videos [§4.1, Figs. 3-5].

Method

AutoGaze is a 3M-parameter conv-encoder + autoregressive transformer-decoder. Given a video, it interleaves per-frame encoding and patch-index decoding: encode frame 1, decode its patch indices autoregressively (vocab = patch indices, not words), encode frame 2 conditioned on frame-1 features and selected indices, and so on. A per-step head predicts the reconstruction loss of the partially-reconstructed current frame from selected patches; once predicted loss falls below threshold, gazing for that frame stops. The reconstructor used during training is a custom VideoMAE with block-causal attention; the distance is a weighted sum of pixel reconstruction and perceptual loss.

Training: (a) NTP pre-training on 250K videos with greedy-search ground-truth gazing sequences, supervising both the cross-entropy on patch indices and an MSE on the loss-prediction head. (b) On-policy GRPO post-training with negative reconstruction loss as reward over the rest of the video (γ-discounted), pushing AutoGaze beyond its sub-optimal greedy teacher. To handle arbitrary video size at inference without retraining, the input is split into 16×224×224 tiles, each tile is gazed independently, and positions are merged back. The downstream ViT (SigLIP2-SO400M in the experiments) is modified to (i) take patches at multiple scales by interpolating positional embeddings and running patch-embed per scale, and (ii) attend across 16 frames jointly. Encoded tokens then feed into the MLLM (NVILA-8B-Video) as usual.

Results

Headline scaling. NVILA-8B-Video alone is capped at 256 frames × 448 resolution; with AutoGaze it scales to 1024 frames × 3584 resolution, a 4× / 8× expansion of the input envelope, with consistent improvement across all six tested benchmarks [Table 1, Fig. 8].
Efficiency. Up to 19× ViT speedup and 10× MLLM speedup at 0.7 reconstruction-loss target; ViT and MLLM baselines OOM around 30 FPS / 896 resolution and 30 FPS / 224 resolution respectively, but AutoGaze enables scaling well past those points [§4.2, Fig. 7].
HLVid (new benchmark). 268 QA pairs on 5-minute, 4K videos where each question is manually verified to require 1K–2K-pixel resolution to answer. NVILA-8B-Video: 42.5 → 52.6 (+10.1) with AutoGaze; Gemini 2.5 Flash-Lite: 52.2; GPT-4o: 49.3; Qwen2.5-VL-7B: 48.1; VideoChat-Flash: 46.6 [§3.4, Table 1].
vs. MLLM token-reduction at parity. At 128-frame video and 6.25% retention: AutoGaze 52.3 on V-MME / 50.3 on L-Vid with 0.55s ViT + 0.10s LLM latency; the best in-LLM pruner (STORM) gets 52.7/51.5 but at 2.18s ViT — AutoGaze is the only method that touches ViT latency [Table 2].
Training ablations. Pre-train only: 0.102 gazing ratio at 0.7 recon loss. RL only: 0.209. Both: 0.094. No training: 0.263. Multi-token prediction at 10 tokens/step balances gazing ratio (0.094) and latency (0.193s); ablating multi-scale roughly doubles both [Tables 3, 4].

Why it’s interesting

The wiki’s existing pre-ViT video token reducer is Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (RLT) — a pixel-L1 run-length encoding with no training, no parameters, and τ=0.1 as the only knob. AutoGaze is its learned counterpart: same architectural slot (drop patches before the ViT) but with a tiny 3M-param policy trained via NTP+GRPO to make content-aware multi-scale decisions and pick its own gazing length. The interesting tension is whether the learned policy’s 4–100× reduction is enough better than RLT’s free pixel-L1 prune to justify a separate training stage; AutoGaze’s 19× ViT speedup vs. RLT’s 1.4–2.1× wall-clock training speedup suggests yes for the high-FPS/high-res regime that Luma’s video-understanding workloads care about, but the comparison is not run directly. The other useful contrast is with TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generation which lives on the generation side of the same redundancy story — TokenTrim evicts unstable KV-cache tokens during AR rollout, AutoGaze prevents them from ever being computed. Both fit under the video temporal redundancy umbrella but operate at different ends of the encode/generate pipeline. The multi-scale aspect — letting low-detail regions be coarsely tokenized — is also a concrete instance of the bigger “adaptive tokenization” theme that connects to FlexTok and learned tokenizer ideas.