Context Forcing: Consistent Autoregressive Video Generation with Long Context

TL;DR

Context Forcing trains a long-context autoregressive video DiT by distilling from a long-context teacher rather than the standard 5s memoryless teacher used in Self-Forcing / LongLive / Self-Forcing++. The paper names the “forgetting–drifting dilemma”: short context windows lose subjects, long context windows accumulate error because the teacher can’t see (and therefore can’t correct) accumulated drift. Their fix is two-fold — a Context Teacher pretrained on video continuation with error-recycling fine-tuning that stays reliable under student-generated context, plus a Slow-Fast Memory KV-cache architecture (attention sink + surprisal-promoted slow memory + FIFO fast memory + bounded RoPE) that extends effective context past 20s while keeping the cache small. On Wan2.1-T2V-1.3B, they hit 60s generation at 17 FPS with VBench subject-consistency 94.88 vs. LongLive’s 93.88.

Key claims

The fundamental failure mode of streaming-tuning approaches (Self-Forcing, LongLive, Self-Forcing++) is a student-teacher mismatch: the student rolls out for 20–60s but the teacher only sees 5s windows, capping the student’s learnable context length [§1, Fig. 2(b)].
The result is a forgetting–drifting dilemma: short memory → minimal error accumulation but identity/scene loss; long memory → identity preserved but drift away from the real video manifold [§1, Fig. 1].
A long-horizon KL objective decomposes by chain rule into a local term (Stage 1, 1–5s windows, standard DMD) and a contextual continuation term (Stage 2, supervised by a long-context teacher) [§3, Eq. 2].
Contextual Distribution Matching Distillation (CDMD) trains the student on its own rollouts using fake-score / real-score evaluations of the teacher on the same student-generated context — mitigating exposure bias [§3.2, Eq. 5–6].
The KV cache is partitioned into three functional regions — Attention Sink, Slow Memory (long-term, surprisal-promoted), Fast Memory (FIFO local) — inspired by dual-process memory theories [§3.3].
Surprisal-based consolidation promotes a Fast-Memory token to Slow Memory when its key-vector cosine distance from its predecessor exceeds threshold τ, ensuring Slow Memory stores high-information state transitions rather than redundant frames [§3.3, Eq. 7].
Bounded Positional Encoding constrains all temporal RoPE indices to a fixed range regardless of generation step, preventing the distribution shift that unbounded RoPE creates on long sequences [§3.3, Eq. 8].
Error-Recycling Fine-Tuning trains the Context Teacher on perturbed contexts (real prefix + Bernoulli-injected past-residual noise), so the teacher remains reliable as a proxy for the real-data continuation even when the student’s context degrades [§3.4].
Effective context exceeds 20s during inference vs. 1.5–9.2s for prior SOTA (Infinity-RoPE 1.5s, LongLive 3.0s, Rolling Forcing 6.0s, FramePack-F1 9.2s) [§1, Table 2 caption].
At 60s generation, Context Forcing hits VBench subject-consistency 94.88 and background-consistency 95.34 vs. LongLive 93.88 / 94.62 and Rolling Forcing 93.10 / 94.12 [Table 2].
On the DINO-score / CLIP-F / CLIP-T metrics measured at 10–60s windows, Context Forcing is the only baseline that maintains >87 DINO and >37 CLIP-T at 60s [Table 1 (referenced as Table 4 in §4.3)].
A training-free ablation that applies the Slow-Fast Memory + Bounded RoPE on top of a Stage-1-only model (no CDMD) significantly degrades both semantic and temporal consistency — the memory architecture and the distillation procedure both matter [Table 3].
Bounded Positional Encoding is essential: removing it drops VBench Total from 82.45 → 73.52 [Table 3].

Method

The system is a two-stage curriculum over a Wan2.1-T2V-1.3B base. Stage 1 matches the student’s short-window distribution to a bidirectional teacher’s via standard DMD on 1–5s clips. Stage 2 addresses long-range coherence by introducing a separately pretrained Context Teacher — a long-context-aware video continuation model fine-tuned with Error-Recycling Fine-Tuning (ERFT), where past model residuals are stochastically injected into the teacher’s context so it learns to recover from drift. Stage 2’s loss is Contextual DMD: the student rolls out (length sampled from Uniform(1, t) with t growing linearly during training to avoid early-stage distribution shift), then both fake-score and real-score models score the student’s own generated continuation conditioned on the student’s own generated context, and the gradient pushes the student toward the teacher’s distribution.

The memory system that makes this run at long horizons partitions the KV cache into an Attention Sink (first tokens, always retained), Slow Memory (up to N tokens of high-surprisal history), and Fast Memory (FIFO of the most recent M tokens). When a new token is enqueued into Fast Memory, its key-vector is compared to its predecessor’s; if the cosine distance exceeds threshold τ, it’s promoted to Slow Memory. Bounded Positional Indexing remaps all temporal RoPE positions to a fixed range so that long sequences don’t drift out of the model’s training-time positional distribution. Setting cache size to 21 latent frames with consolidation every 2 chunks extends effective context past 20s. A “Clean Context Policy” decouples diffusion-step supervision: context frames are fully denoised at every training step (to match inference), while target frames retain the random-timestep exit policy of Self-Forcing (to preserve gradient coverage across diffusion steps).

Results

60s text-to-video on VBench (Wan2.1-T2V-1.3B base, 17 FPS): Total 82.45 / Quality 83.55 / Semantic 76.10 / Background-Consistency 95.34 / Subject-Consistency 94.88. LongLive at the same FPS budget: 83.64 / 84.53 / 74.97 / 94.62 / 93.88. Self-Forcing++: 83.11 short-only, no 60s reported. Rolling Forcing: 79.31 / 81.87 / 67.69 / 94.12 / 93.10 [Table 2].
5s text-to-video on VBench: Total 83.44 / Quality 84.98 / Semantic 77.29 — comparable to bidirectional Wan2.1 (84.26 Total) and ahead of CausVid (81.20) at 17 FPS [Table 2].
Long-context fine-grained consistency on 100 MovieGenBench prompts, measured by window-averaged DINOv2 / CLIP-F / CLIP-T at 10–60s intervals: Context Forcing student is the only method to hold DINO ≥ 87.45 and CLIP-T ≥ 36.25 across all six time buckets. FramePack-F1 collapses by 30s (DINO 73.10 at 40s, 68.50 at 60s); LongLive at 3s context holds 86–87 DINO but with abrupt scene resets noted qualitatively [Table 1 / §4.3, Fig. 8].
Ablations (Table 3): full model VBench Total 82.45; remove CDMD → 80.36; remove Bounded Positional Encoding → 73.52 (catastrophic); replace surprisal-based slow-memory sampling with uniform interval-1 → 80.82, interval-2 → 81.11.
Effective context 20+ seconds, vs. SOTA range 1.5–9.2 s [§1, Table 2].
Throughput 17 FPS at 1.3B (parity with Self-Forcing, CausVid, Infinity-RoPE; LongLive runs at 20.7 FPS with a 3s context cap) [Table 2].

Why it’s interesting

This paper sharpens a problem the long-video AR community has been talking around — that streaming-tuning style fine-tuning (Self-Forcing, LongLive, Rolling Forcing) effectively trains a long-horizon model with a short-horizon supervisor — and gives the cleanest fix yet: build a teacher that is itself long-context-aware, then use error-recycling fine-tuning so it stays informative on the student’s own drifted outputs. This is a complementary axis to the PFP recipe filed last week: PFP keeps context cheap via a pretrained compression encoder; Context Forcing keeps context useful via a long-context teacher and a surprisal-driven cache. The Slow-Fast Memory + Bounded RoPE ablations are the most actionable for Luma’s long-video stack — Bounded Positional Encoding alone is responsible for ~9 VBench Total points and is essentially free to add. The forgetting–drifting framing is also a useful diagnostic vocabulary for evaluating any future AR video model.