Streaming Autoregressive Video Generation via Diagonal Distillation

TL;DR

DiagDistill distills a multi-step AR video diffusion teacher (Wan2.1-T2V-1.3B) into a few-step streaming student by allocating denoising steps asymmetrically across chunks — more steps for early chunks (which establish global structure), progressively fewer for later chunks (which inherit structural priors via KV cache). Three pieces: (1) a Diagonal Denoising step schedule (e.g. 4-3-2-2-2-2-2), (2) Diagonal Forcing — conditioning each chunk on the partially-noised final frame of the previous chunk (KV cache from a non-final denoising step) rather than the clean output, to align training with inference, (3) Flow Distribution Matching — a flow-distribution KL added to DMD that aligns optical-flow statistics between student and teacher, restoring motion amplitude lost at low step counts. 5-second video in 2.61 s on a single H100 (~31 FPS), 277.3× speedup over Wan2.1, beating Self-Forcing on VBench Total (84.48 vs 84.31) with ~1.5× lower latency. ICLR 2026.

Key claims

A fixed per-chunk step budget is suboptimal for AR video distillation; allocating more steps to early chunks and progressively fewer to later ones (e.g. 4-3-2-2-2-2-2 over 7 chunks) trades a small early-latency cost for a large quality-at-low-NFE-later gain, because later chunks inherit appearance/structure priors from heavily-denoised early chunks via the KV cache [§3.2, Table 3].
Standard AR distillation conditions each new chunk on the clean output of the previous chunk, but at inference the previous chunk is generated by the few-step student (effectively a partially-noised state); this train/inference mismatch causes “implicit next-noise-level prediction,” accumulating into over-saturation and quality decay over long rollouts [§1, §3.2].
Diagonal Forcing closes this gap by conditioning chunk k on a controlled-noise-injected version of chunk k-1’s output, x_{k-1}^{noised} = α_t · x_{k-1}^{clean} + σ_t · ε, with t≈100 (out of 1000) found optimal — high enough to mimic inference-time partial denoising, low enough to preserve structural priors [§3.2, Eq. 6; §4.3 Fig. 5(a)].
Few-step denoising attenuates motion amplitude because truncated noise estimation paths lose motion-relevant gradient signal; standard DMD (a spatial reverse-KL) does not capture this [§3.3].
Flow Distribution Matching adds a flow-field reverse-KL on top of DMD using a lightweight learnable motion module (latent-frame difference → conv → MLP, teacher updated via EMA), with weight λ=1.0 optimal on an 8-config ablation; this restores motion under strict step constraints without an external optical-flow network [§3.3 Eqs. 7–12; §4.3 Fig. 5(b)].
On VBench (5-second T2V), DiagDistill at 31 FPS / 0.37 s first-frame latency hits Total 84.48 vs Wan2.1 teacher 84.26 at 0.78 FPS / 103 s, vs Self-Forcing 84.31 at 17 FPS / 0.69 s — i.e. matches or beats teacher quality at 277.3× speedup, 1.83× faster than Self-Forcing [Table 1].
Ablation: removing Diagonal Denoising drops the student to Self-Forcing-equivalent inference cost (same NFE per chunk) with comparable quality — i.e. the Diagonal Denoising contribution is efficiency, not quality; the quality contributions come from Diagonal Forcing and Flow Distribution Matching, both of which significantly degrade VBench scores when removed [§4.3, Table 3].
The asymmetric schedule degrades gracefully: configuration 5-3-3-3-3-3-3 maximizes quality, 4-2-2-2-2-2-2 maximizes throughput; the chosen 4-3-2-2-2-2-2 sits near the Pareto knee with the second-lowest total NFE and quality comparable to higher-NFE configs [§4.3, Table 3].
Long-video user study (93 participants, 150 comparisons per model pair, first 50 MovieGenBench prompts) shows baselines (Self-Forcing, Causvid, MAGI-1) suffer saturation/distortion drift over time while DiagDistill maintains visual quality, with dynamic-prompting support along the timeline for evolving narrative scenes [§4.4, Figs. 7–9].

Method

DiagDistill operates on top of LongLive-1.3B (initialized from Wan2.1-T2V-1.3B with Self-Forcing- or Causal-Forcing-style ODE init available), uses a rolling KV cache with 3-frame chunks and 4-chunk context window (17.5 GB constant memory footprint), and is trained in two stages: Stage 1 base distillation (exp_stage1_all4_odeinit) and Stage 2 diagonal-denoising fine-tune (exp_stage2_diag_from_stage1).

The DMD objective is extended along two orthogonal axes. Diagonal Denoising assigns steps {n_0, n_1, …, n_6} per chunk with n_k decreasing in k; for the first three chunks the student uses progressively reduced distilled models (e.g. 4→3 steps), then a fixed 2-step process from chunk 3 onward, reusing the KV cache from the final noisy frame of the preceding chunk rather than the final clean frame. Diagonal Forcing explicitly simulates this trajectory during training: the conditioning latent for chunk k is x_{k-1}^{cond} = α_{t_d} · x_{k-1}^{0} + σ_{t_d} · ε with t_d a controlled noise schedule along the diagonal path (t_d≈100 optimal), so the student learns to denoise given a noisy past chunk — matching the inference-time state of the KV cache. Flow Distribution Matching adds L_flow = KL(p_flow_real || p_flow_gen) where the flow field F(x) is extracted by a learnable latent-frame-difference module; the gradient follows the DMD form with student/teacher flow score functions, plus an L1 flow regression loss for feature alignment. The overall objective is L = L_DMD + λ · L_flow with λ=1.0. Training uses VidProM prompts (LLM-extended), Flow Matching parameterization from Wan2.1, evaluated on VBench at 832×480, 16 FPS.

Results

Throughput / latency: 31 FPS, 0.37 s first-frame latency, 277.3× speedup over Wan2.1 teacher [Table 1]. 1.53× latency improvement over Self-Forcing (0.69 s) and 1.83× throughput improvement (17 FPS).
5-second VBench: Total 84.48 / Quality 85.26 / Semantic 81.73 — beats Self-Forcing (84.31 / 85.07 / 81.28) and Causvid (81.20 / 84.05 / 69.80), and slightly exceeds the Wan2.1 teacher (84.26 / 85.30 / 80.09) on Total and Semantic at a fraction of the cost [Table 1].
1.88× speedup on 5-second short-video generation on a single H100 vs prior fastest AR baseline (Self-Forcing) for the comparable Diagonal-Denoising-only configuration; full DiagDistill is 277.3× vs the undistilled Wan2.1 teacher [GitHub highlights, §4.2].
Step-schedule ablation (Table 3): 5-3-3-3-3-3-3 best quality; 4-2-2-2-2-2-2 best throughput; 4-3-2-2-2-2-2 chosen as Pareto knee. Removing Diagonal Denoising keeps quality but kills the speedup. Removing Diagonal Forcing or Flow Distribution Matching significantly degrades all VBench dimensions.
Diagonal Forcing noise level: t_d=100 maximizes all three VBench axes (temporal, frame, text-alignment); t_d→0 (clean conditioning) causes over-saturation from implicit next-noise-level prediction; t_d→1000 (full noise) blurs structural priors and reduces motion amplitude [§4.3, Fig. 5(a)].
Flow loss weight: λ=1.0 optimal across 8 weights, balancing motion guidance against the spatial DMD signal [Fig. 5(b)].
Long video: maintains perceptual quality over time vs baselines that exhibit rapid quality decay (saturation drift). User study (93 participants × 150 comparisons / pair) confirms superiority on overall visual quality, text faithfulness, long-term consistency [§4.4]. Supports dynamic prompting (new text descriptions injected along the timeline).

Why it’s interesting

DiagDistill sits at the exact intersection of two heavily-trafficked threads on this wiki — the Diffusion Distillation thread (TMD, ArcFlow, DP-DMD, RMD, Causal Forcing, Causal Forcing++) and the Autoregressive Video Generation thread (Context Forcing, PFP, FlowAct-R1, Causal Forcing++) — and contributes a distinctively temporal axis that none of the existing entries explore: non-uniform step allocation across chunks. Where Cross-Resolution Distribution Matching for Diffusion Distillation cascades resolutions along the denoising trajectory (low-res for high-noise, high-res for low-noise), DiagDistill cascades step counts along the AR-chunk axis — a directly analogous “more compute where it matters most” idea applied to the orthogonal time dimension. Its Diagonal Forcing trick is also a different solution to the same train/inference-distribution-gap problem that Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation attacks via teacher-forced AR fine-tuning and that Transition Matching Distillation for Fast Video Generation attacks via rolling out the student’s flow head inside DMD; DiagDistill instead injects exactly the noise level the next-chunk KV cache will see at inference into the conditioning during training. The Flow Distribution Matching component is a video-specific complement to ArcFlow’s geometric trajectory fix and DP-DMD’s diversity preservation — addressing motion amplitude, which neither of those targets.