Swift Sampling: Selecting Temporal Surprises via Taylor Series

TL;DR

Swift Sampling is a training-free, query-agnostic frame-selection algorithm for long-form video understanding. It models a video as a differentiable trajectory in visual-feature space, computes per-frame velocity and acceleration, and uses a Taylor expansion to predict where the next feature vector should be — frames that diverge sharply from that prediction are flagged as “temporally surprising” and kept. The method adds only ~0.02× compute over the underlying VLM (≈30× cheaper than other training-free baselines), beats uniform sampling and prior query-agnostic methods across three long-video QA benchmarks and 10 downstream tasks, and gives up to +12.5 accuracy points when the frame budget is tight.

Key claims

Most frames in long-form video are redundant; the load-bearing information sits at temporal surprises — frames whose visual features deviate from the predicted continuation of the recent trajectory [§1].
A video can be treated as a differentiable trajectory in a frozen visual-encoder’s feature space, with per-frame velocity (first finite difference) and acceleration (second finite difference) used to Taylor-extrapolate the expected next feature vector [§3].
“Temporal surprise” is operationalized as the divergence between the observed feature vector and the Taylor-predicted one; frames with the largest divergence get sampled [§3].
The method is training-free, query-agnostic, and adds only ≈0.02× compute over the baseline VLM, ≈30× cheaper than leading prior training-free selectors that depend on auxiliary networks or per-video hyperparameter tuning [§1, §4].
Across three long-video QA benchmarks and 10 downstream tasks, Swift Sampling outperforms both uniform sampling and prior query-agnostic baselines [§4].
The accuracy gap is largest in the small-budget regime — long videos with limited frame budgets get up to +12.5 accuracy points over uniform sampling [§4, Abstract].

Method

The selector treats the sequence of frame embeddings produced by a frozen visual encoder as a discrete sample of a smooth trajectory f(t) in latent space. At each timestep it estimates f'(t) and f''(t) via finite differences, then projects the expected next position using a low-order Taylor expansion f̂(t+1) ≈ f(t) + f'(t) + ½ f''(t). The “surprise score” for frame t+1 is the distance between the observed embedding and f̂(t+1). Sampling reduces to a top-k over surprise scores, with no learnable parameters, no per-video tuning, and no query input — the selector runs once per video and feeds whichever VLM consumes it.

The conceptual handle is predictive coding: the human visual system is hypothesized to allocate attention to moments where prediction error spikes, and Swift Sampling is that mechanism in the visual-feature latent space.

Results

Outperforms uniform sampling and prior training-free query-agnostic frame selectors on three long-video QA benchmarks, across 10 downstream task categories [§4].
Up to +12.5 accuracy points over uniform sampling when frame budgets are tight on long videos [Abstract, §4].
~0.02× overhead on top of baseline VLM compute, ≈30× cheaper than leading training-free baselines that require auxiliary networks [§1, §4].

Why it’s interesting

This sits next to VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection as the static, query-agnostic counterpart to VideoEspresso’s caption-based redundancy filter — both compress a video to ~a handful of frames before the reasoning VLM sees it, but Swift Sampling does so with no captioner, no language model, and no question; the signal comes entirely from the trajectory geometry in the visual encoder. It also contrasts with Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning, where the reasoning model itself emits grounding tokens to actively pick frames during a CoT — Swift Sampling is the cheap front-end alternative to that whole loop, and the natural ablation is “active perception vs predictive-coding surprise”.

The Taylor-expansion-of-feature-trajectory primitive is also a candidate building block for any system that needs to budget tokens over redundant temporal streams (autoregressive video DiT memory compression, video super-resolution keyframe selection, etc.), not just video QA. The closest piece of wiki machinery is Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (RLT, run-length-tokenization of redundant patches inside video ViTs) — Swift Sampling is the between-frame analog of RLT’s within-token-grid redundancy detection, and both are pre-network reductions that beat the inside-attention-sparsity recipes on overhead.