End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

TL;DR

Resampling Forcing is an end-to-end, teacher-free training framework for autoregressive video diffusion that mitigates exposure bias by training the model on its own degraded history rather than ground-truth history or a bidirectional teacher’s outputs. The core trick — autoregressive self-resampling — corrupts ground-truth history frames to a sampled timestep, then completes denoising with the online model weights (gradient-detached) to synthesize degraded history that matches the model’s inference-time error distribution. A logit-normal timestep schedule with shifting factor σ=0.2 biases the corruption toward low-noise regimes. A separately introduced history routing mechanism replaces dense causal attention with a parameter-free top-k frame router (mean-pool descriptor, head-wise / token-wise), keeping attention complexity near-constant for long-horizon rollout. Built on WAN2.1-1.3B, trained from-scratch-style (no 14B teacher), and achieves VBench Temporal 91.20 / 90.44 / 89.74 at 5s / 10s / 15s — beating Self Forcing 90.03 / 84.27 / 84.26 at the same parameter count.

Key claims

The root cause of error accumulation in AR video diffusion is the train–test mismatch between ground-truth history (training) and self-generated history (inference); the fix is to train under degraded history while keeping the clean prediction target [§3.1, Fig. 2].
Two distinct error sources need simulation: (1) intra-frame errors from imperfect score estimation, dominant at high-frequency details, and (2) inter-frame errors that accumulate through the AR loop [§3.2].
Autoregressive self-resampling — corrupt frame xⁱ to timestep τ via flow-matching interpolation, then denoise with the online model weights to produce a degraded x̂ⁱ — captures both error modes simultaneously when applied sequentially across frames with degraded prior frames as condition [§3.2, Eq. 5, Fig. 3(a)].
Gradients are detached from the resampling pass to prevent shortcut learning [§3.2].
The resampling timestep τ is drawn from a logit-normal distribution with a shifting factor σ controlling the bias toward low-noise (faithful history) vs. high-noise (more error-correction flexibility); σ=0.2 is recommended [§3.2, Eq. 6–7, Fig. 6].
Resampling Forcing requires a teacher-forcing warmup phase (10k steps) before transitioning to self-resampling, because the early-stage model has not yet acquired meaningful AR capability and self-resampling produces uninformative signals [§3.2, §4].
Comparison of error-simulation strategies on a 5s benchmark: noise augmentation (87.15 Temporal / 61.90 Visual / 21.44 Text) < parallel resampling (88.01 / 62.51 / 24.51) < autoregressive resampling (90.46 / 64.25 / 25.26) [Table 2].
Parallel resampling captures only per-frame degradation; autoregressive resampling additionally captures inter-frame accumulation, and the gap (≈2.4 Temporal points) quantifies the inter-frame-accumulation contribution [§4.2, Table 2].
A dynamic top-k history-routing mechanism replaces dense causal cross-frame attention with content-adaptive frame selection via dot-product affinity between query tokens and mean-pooled frame descriptors, head-wise and token-wise [§3.3, Eq. 8–9, Fig. 4].
Routing is implemented as a two-branch attention (intra-frame dense + cross-frame top-k sparse) fused via global log-sum-exp, both via flash_attn_varlen_func() from FlashAttention [§3.3].
Top-k=8 history routing achieves 75% attention sparsity with negligible quality drop relative to dense causal attention; top-k=4 (87.5% sparsity) causes only minor degradation [Fig. 7, Table 1].
Routing selection frequency exhibits a hybrid “frame-sink + sliding-window” pattern that emerges naturally — the router preferentially attends to initial frames and the most recent frames, with the bias more pronounced at higher sparsity [§4.2, Fig. 8].
Final headline numbers on VBench Temporal / Visual / Text at 5s / 10s / 15s for a 1.3B AR model trained without a teacher: 91.20 / 64.72 / 25.79, 90.44 / 64.03 / 25.61, 89.74 / 63.99 / 24.39 [Table 1].
Self Forcing (the prior distillation-based 1.3B baseline using a WAN2.1-14B teacher) drops from 90.03 Temporal at 5s to 84.27 / 84.26 at 10s / 15s, while Resampling Forcing holds at 90.44 / 89.74 — showing distillation-free training beats teacher-distilled baselines at longer horizons [Table 1].
LongLive (concurrent work using a short bidirectional teacher for sub-clip distillation) is shown qualitatively to violate strict causality — in a “milk pouring” example, the liquid level rises then falls despite continuous pouring; the authors attribute this to future-context leakage from the bidirectional teacher [§4.1, Fig. 5 lower panel].

Method

The framework reuses a WAN2.1-1.3B DiT backbone, modified for causal attention with per-frame noise levels and a chunk-size of 3 latent frames per AR unit. Training has three phases: (1) teacher-forcing warmup on 5s videos for 10k steps to establish causal AR competence; (2) Resampling Forcing on 5s → 15s sequentially (15k + 5k steps); (3) fine-tuning with sparse history routing enabled for 1.5k iterations on 15s videos.

The self-resampling step uses a 1-step Euler solver for efficiency. The training objective remains the standard flow-matching velocity regression against ground-truth clean frames; the modification is purely in the condition — the model sees x̂¹⋯x̂ⁱ⁻¹ (autoregressively resampled, gradient-detached) instead of x¹⋯xⁱ⁻¹ (ground-truth). At inference, the resampling machinery is dropped; the model performs standard AR rollout with KV cache.

The history-routing module is parameter-free: each frame is represented by the mean of its key tokens; per query token, the top-k frames by dot-product affinity are retained, and the attention is split into an intra-frame dense pathway plus a cross-frame sparse pathway, fused by aligning log-sum-exp terms.

Results

VBench at 15s: Temporal 89.74 / Visual 63.99 / Text 24.39 — beats Self Forcing (84.26 / 63.04 / 24.29) and matches LongLive (84.57 / 67.17 / 24.44, but with teacher) at the same 1.3B param count [Table 1].
Temporal-axis dominance: at 5s/10s/15s the model reports 91.20 / 90.44 / 89.74, a degradation of only 1.46 points across 3× length, vs. CausVid (89.35 → 87.14, 2.21 drop) and Self Forcing (90.03 → 84.26, 5.77 drop) [Table 1].
Sparsity vs. quality: dense attention vs. 75%-sparse routing differ by only 1.02 Temporal / 0.77 Visual / 1.67 Text at 5s [Table 1]; top-k=4 (87.5% sparsity) shows further but still minor drop [Fig. 7].
Ablation: shifting factor σ=0.2 outperforms extremes — very small σ → error accumulation; very large σ → content drifting [Fig. 6].
Native-long-video training beats extrapolated baselines at longer horizons, confirming the value of training on the target generation length directly rather than relying on sliding-window extrapolation [Table 1, §4.1].

Why it’s interesting

Resampling Forcing is the cleanest stake yet in the ground claimed by Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation and Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation: those papers fix the initialization of the few-step AR student in the Self-Forcing pipeline but still rely on a bidirectional teacher. This paper kills the teacher entirely — the model’s own online weights produce the degraded history, gradient-detached, removing both the future-context-leakage failure mode (which LongLive demonstrably suffers from per Fig. 5) and the 14B-teacher compute bill. Within the Autoregressive Video Generation cluster it occupies a previously empty operating point: end-to-end, from-scratch, single-model, native-long training.

The history-routing component sits orthogonal to the resampling idea and links to PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference (mid-buffer compression + top-k retrieval) and Context Forcing: Consistent Autoregressive Video Generation with Long Context (functional cache partitioning + surprisal promotion). The emergent “frame-sink + sliding-window” selection pattern is empirical validation of the heuristic mask design used by Infinity-RoPE / LongLive / Rolling Forcing — but achieved via a parameter-free content-aware router rather than a fixed mask, suggesting an alternative path to attention sparsity that none of the Context Length / Quality Trade-off in Video Generation papers has explored.

The closest filed analogue to the self-resampling idea is MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation‘s noise-augmented memory and FlowAct-R1: Towards Interactive Humanoid Video Generation‘s generated-GT memory swap — both inject perturbations into history to look like rollout — but neither uses the current online model to produce the perturbations, which is what makes Resampling Forcing teacher-free and self-evolving.