PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inference

TL;DR

PyramidalWan converts a pretrained text-to-video diffusion model (Wan2.1-1.3B) into a pyramidal model — one that runs early, high-noise denoising steps at low spatial-temporal resolution and only escalates resolution as noise decreases — using low-cost finetuning rather than from-scratch training, and then layers step distillation on top. The recipe restructures Wan’s diffusion process into three stages (81×448×832, 41×224×416, 21×112×208) under a pyramidal flow matching loss, and the authors compare adversarial vs. DMD distillation on both conventional and pyramidal teachers. The final model (PyramidalWan-DMD-PT*) reaches quality comparable to the multi-step Wan baseline at a fraction of the compute, and the paper also reports the first successful few-step training of Pyramidal Patchification (an alternative to Pyramidal Flow). A side theoretical contribution generalizes the pyramidal resolution-transition operators to arbitrary orthogonal-transform up/downsampling, recovering the original Haar-wavelet operator as a special case.

Key claims

Existing open-source pyramidal video models are trained from scratch and underperform SOTA on visual plausibility; converting a pretrained Wan2.1-1.3B to pyramidal via finetuning closes that quality gap [project page abstract].
The pyramidal restructuring splits Wan2.1’s diffusion into three spatiotemporal stages at 81×448×832, 41×224×416, and 21×112×208 resolutions, with each stage handling its corresponding noise band [project page §model recipe].
The PyramidalWan recipe (20-20-10 steps across the three stages) matches the quality of 50-step Wan in qualitative comparisons while requiring substantially less compute [project page qualitative grid].
Adversarial distillation produces more realistic-looking videos but suffers from motion artifacts; DMD distillation produces more stylized/cartoon-like videos but with better and more consistent motion [project page §step distillation analysis].
4-step and 2-step distilled Wan retain good quality; 1-step degrades noticeably — locating the practical floor of step distillation for this model class [project page §step-count ablation].
Pyramidal Patchification (an alternative formulation to Pyramidal Flow) is shown for the first time to be trainable for few-step video generation [project page summary].
The resolution-transition operators in the original Pyramidal Flow paper (average pooling, nearest-neighbor upsampling) are special cases of a more general class of orthogonal-transform operators — specifically scaled Haar wavelet operators [project page §theoretical contribution].
Training-free acceleration via Jenga, while strong on automated metrics, exhibits severe scene and motion artifacts in qualitative inspection; the authors recommend their final recipe PyramidalWan-DMD-PT* over Jenga-augmented baselines [project page §Jenga comparison].

Method

The model starts from a pretrained Wan2.1-1.3B text-to-video diffusion transformer. Its diffusion trajectory is partitioned into three resolution stages — full (81×448×832), half (41×224×416), and quarter (21×112×208) along all three spatiotemporal axes — with the highest-noise portion of the trajectory routed to the lowest-resolution stage and the lowest-noise portion to full resolution. A pyramidal flow matching loss finetunes the model to denoise at each stage and to handle the resolution transitions between them.

On top of the pyramidal teacher, the authors run step distillation, comparing two families: adversarial distillation and Distribution Matching Distillation (DMD), each applied either to the conventional (non-pyramidal) Wan teacher or to the pyramidal teacher. The DMD-PT* recipe (DMD distillation against a pyramidal teacher with a specific time-schedule choice) is identified as the final winner. A separate Pyramidal Patchification configuration is also distilled to demonstrate the recipe generalizes beyond Pyramidal Flow. The theoretical contribution rewrites the up/down sampling between resolution stages as a scaled orthogonal transform, with Haar wavelets as one instance, enabling principled alternatives in future work.

Results

The headline qualitative result is that PyramidalWan-DMD-PT* with very few effective sampling steps matches the quality of 50-step and 25-step Wan baselines on a battery of text prompts (animals, people, fantasy scenes), while PyramidalWan in its undistilled 20-20-10 form already represents a substantial compute reduction over the 50-step baseline. Adversarial-distilled variants are visually sharper but motion-glitch-prone; DMD-distilled variants are smoother but more stylized; PyramidalWan-DMD-OT (which closely follows a concurrent work) produces oversaturated colors. Wan-DMD at 4 and 2 steps holds up; at 1 step quality collapses. Jenga (training-free step caching) wins on automated metrics but loses qualitatively. No quantitative table from the paper is reproduced on the project page; for FVD/VBench numbers consult the arxiv preprint.

Why it’s interesting

Pyramidal diffusion is a structural lever for video efficiency that’s complementary to step distillation: it cuts the per-step cost at the high-noise end of the trajectory, while distillation cuts the step count. PyramidalWan is the first recipe to land both on a pretrained open-source video model (Wan2.1) without retraining from scratch, which is the only economical path for most teams. For Luma, the practical interest is twofold: (1) the finetuning-to-pyramidal recipe is a transferable blueprint for any pretrained DiT-style video stack, and (2) the head-to-head adversarial-vs-DMD comparison on a pyramidal teacher is one of the few public datapoints on how distillation families interact with multi-resolution structure — including the observation that automated metrics (where Jenga wins) sharply diverge from qualitative inspection, a recurring problem in the few-step video literature. The paper also flags Pyramidal Patchification as a viable alternative substrate, which expands the design space beyond Pyramidal Flow.