TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generation

TL;DR

TokenTrim is an inference-time, training-free method to suppress temporal drift in autoregressive long-video generation by identifying and removing corrupted latent conditioning tokens from the KV cache before they propagate into future rollout steps. The hypothesis is that drift is not a capacity problem but an inference-time error-propagation problem: at each AR step, a small number of latent tokens drift sharply away from their predecessor batch’s representations, and reusing them as conditioning is what compounds the error. The detector is a per-token L2 distance between time-averaged latent summaries of consecutive batches; the trigger is an adaptive threshold on the top-pN drift severity; the action is to mask those positions in the temporal KV cache and regenerate the current batch from the pruned cache. No architecture, training, or latent-space changes are required.

Key claims

Temporal drift in autoregressive video generation stems primarily from inference-time error propagation via uncontrolled reuse of corrupted latent conditioning tokens, not from insufficient model capacity [project page §1].
Unstable tokens can be defined locally as latent tokens whose representations deviate significantly from those of the previously generated batch — flagging potential corruption or semantic drift [project page §1].
Per-token drift is computed as $d_i = \lVert Z_t(i) - Z_{t-1}(i) \rVert_2$ , where $Z_t$ is the latent summary obtained by averaging latents across the $F$ frames of batch $\mathcal{X}_t$ ; the top- $pN$ largest drifts form the unstable set $S_t$ and yield a per-step drift severity $D_t$ [project page Fig. overview(a)].
The pruning decision is gated by an adaptive threshold $\mu_t + \lambda \sigma_t$ on running drift-severity statistics: if $D_t$ is below threshold the cache is left alone; otherwise the selected token positions are masked in the temporal KV cache and the current batch is regenerated conditioned on the pruned cache $(\tilde{K}, \tilde{V})$ [project page Fig. overview(b)].
Running statistics and the KV cache are updated only with the accepted batch $\mathcal{X}_t^{\star}$ , so threshold drift cannot itself be polluted by rejected steps [project page Fig. overview(b)].
Explicitly removing corrupted latent tokens from the autoregressive context — rather than modifying entire spatial regions or model parameters — keeps unreliable latent information out of future conditioning without leaving latent space [project page §1].

Method

At AR step $t$ , the model has just produced candidate batch $\mathcal{X}_t$ on top of the previous batch $\mathcal{X}_{t-1}$ . TokenTrim encodes each batch and forms a latent summary by averaging the per-frame latents over the $F$ frames in the batch, yielding $Z_t, Z_{t-1} \in \mathbb{R}^{N \times d}$ where $N$ is the spatial token count. It then computes a per-token L2 drift $d_i = \lVert Z_t(i) - Z_{t-1}(i) \rVert_2$ and selects the top- $pN$ largest as the unstable set $S_t$ (so $p$ is the per-step pruning budget as a fraction of spatial tokens). The drift severity $D_t$ is a summary statistic over $S_t$ — large $D_t$ means this step’s unstable tokens are unusually unstable relative to recent history.

The decision rule is a one-sided adaptive z-test: $D_t \le \mu_t + \lambda \sigma_t$ accepts the batch with the current KV cache untouched. Otherwise the KV cache is surgically edited — the token positions in $S_t$ are masked out of the temporal KV cache to produce $(\tilde{K}, \tilde{V})$ and the current batch is regenerated conditioned on the pruned cache. Running statistics $(\mu_t, \sigma_t)$ and the KV cache are updated using only the accepted batch $\mathcal{X}_t^{\star}$ , which means rejected/regenerated steps don’t pollute the threshold.

Conceptually the method is closest to FlowAct-R1’s inference-time Memory Refinement (periodic re-noise+denoise of short-term memory) and to surprisal-based cache promotion in Context Forcing (Context Forcing: Consistent Autoregressive Video Generation with Long Context) — but where Context Forcing uses key-vector cosine surprisal to promote fast→slow memory at training time, TokenTrim uses latent-summary drift to evict unstable tokens at inference time, and where FlowAct-R1 re-noises an entire memory component, TokenTrim selects individual token positions.

Results

The project page does not provide quantitative benchmark numbers, ablations, or comparisons against prior streaming-AR methods (Self-Forcing, LongLive, Self-Forcing++, Rolling Forcing, FramePack-F1, Context Forcing, PFP, FlowAct-R1). Headline claims are qualitative: “significantly improves long-horizon temporal consistency without modifying the model architecture, training procedure, or leaving latent space” [project page §1]. No code, paper PDF, model checkpoint, or arxiv link was visible on the project page at filing time; the page may be a teaser preceding an arxiv release.

Why it’s interesting

This is the cleanest statement so far that AR-video drift correction can be done purely at inference, at the level of individual KV-cache token positions, with no reward model and no re-noising step. Compared to the other inference-time drift fix in the wiki — WMReward’s VJEPA-2-surprise BoN/guidance on a frozen MAGI-1 (Inference-time Physics Alignment of Video Generative Models with Latent World Models) — TokenTrim needs no external world model and edits the cache directly rather than steering sampling. Compared to FlowAct-R1’s Memory Refinement (FlowAct-R1: Towards Interactive Humanoid Video Generation), it acts at finer granularity (individual token positions vs. an entire memory component) and uses simpler signals (L2 drift on time-averaged latents vs. a re-noise+denoise pass). And compared to Context Forcing’s Slow-Fast Memory design (Context Forcing: Consistent Autoregressive Video Generation with Long Context), it’s training-free — making it potentially stackable with any of the training-time approaches.

The big unknown is whether the latent-summary L2 distance is actually a useful drift detector. Context Forcing reports that key-vector cosine surprisal beats uniform-interval cache promotion; TokenTrim’s choice of spatial averaging over the $F$ frames in a batch (collapsing temporal info before measuring) and L2 (not cosine, not key-space) is a different bet. Without numbers it’s hard to tell if this is a strict improvement or a different point on the same Pareto curve.