Cross-Resolution Distribution Matching for Diffusion Distillation

TL;DR

RMD (Cross-Resolution Distribution Matching Distillation) folds a coarse-to-fine multi-resolution cascade into the DMD few-step training objective rather than treating cascading and distillation as independent levers. The diffusion trajectory is partitioned by logSNR thresholds into resolution-specific segments — early high-noise steps run at low resolution, later low-noise steps at the target resolution — with a learned logSNR-based timestep mapping that compensates for the SNR shift resolution induces. Distribution matching is then performed only along resolution-wise stages, after upsampling the student’s low-resolution state into the teacher’s high-resolution space using a mixture of predicted noise (to inherit the teacher’s ODE trajectory) and Gaussian noise (to bridge larger resolution gaps). On Wan2.1-T2V-14B the resulting 2+2+2 or 3+3 schedule reaches 25.6× speedup over the 50-step base model while VBench Total = 82.51 — beating 6-NFE DMD2 (80.30) and 6-NFE TDM (80.48) at roughly 50% less compute than those 6-step baselines.

Key claims

The “saturation of step reduction” framing is explicit: aggressive reduction of generation steps to 1–3 results in catastrophic degradation, so further efficiency must come from cutting per-step cost rather than step count [§1].
Same diffusion model produces distinct data distributions at different resolutions; naively dropping resolution at selected timesteps introduces distributional inconsistencies because production models (Hunyuan, Wan, Waver) are trained on a multi-stage curriculum of low-res heterogeneous data then high-res curated data [§1, Fig. 2].
The role of each denoising step depends on the noise level — early high-noise steps reconstruct global structural layout; later low-noise steps refine fine-grained detail — so multi-resolution cascading is justified: high-resolution computation during the high-noise stage is wasted on details not yet required [§1].
Resolution scaling intrinsically modifies the effective noise schedule because higher resolutions suffer relatively weaker pixel corruption at identical noise magnitude; this can be characterized via logSNR, and the growth rate of logSNR differs across resolutions [§3.1].
RMD’s logSNR-invariance mapping (Eq. 1–2, following SD3’s resolution-dependent scaling): student timestep intervals at a given resolution are derived from teacher segments by adjusting logSNR by a resolution-shift term, so distillation occurs over identical denoising states regardless of spatial resolution [§3.1].
Multi-step alignment across resolutions is computationally demanding and supervision is imbalanced; RMD therefore performs distribution matching only along resolution-wise stages (uniformly sampled timesteps within each stage), not point-wise along the trajectory [§3.2, Eq. 6].
The upsampling transformation that lifts the low-resolution generator state into the teacher’s high-resolution space is a mixture of (i) upsampled predicted noise — inherits the teacher ODE trajectory — and (ii) Gaussian noise — bridges large resolution gaps; the mixing factor α is resolution-aware, with more Gaussian weight as the resolution gap grows [§3.3, Eq. 10].
Pure-Gaussian upsampling (α=0) and pure-predicted-noise upsampling (α=1) both underperform the mixture; predicted-noise-only degrades severely under large resolution gaps because upsampling artifacts are treated as hard guidance [§4.3.2, Fig. 8].
Warm-up phase: the low-logSNR (semantic) interval is distilled first to provide stable initialization, then the full trajectory is trained end-to-end; without this warm-up the cascaded trajectory does not converge well [§3.4].
On SDXL, 2+2 NFE (two low-res, two high-res steps) gives 33.4× speedup, HPS 31.99, AeS 6.73, CLIP 35.13 — outperforming SDXL-Turbo, SDXL-Lightning, DMD2, and TDM at the same 4-NFE [Table 1].
On Wan2.1-T2V-14B, the multi-stage schedule achieves 25.6× speedup with VBench Total 82.51, Quality 84.37, Semantic 75.05, T2V-Compbench 54.00 — vs. 6-NFE DMD2 at 16.7× speedup / VBench 80.30 and 6-NFE TDM at 16.7× / VBench 80.48 [Table 2]. The headline is matching the 50-step base model (83.75) much more closely than fixed-NFE distillation baselines.
Ablation isolates the two components: cross-resolution distribution matching (RM) alone gives substantial gains over the baseline; naive cascaded upsampling without RM performs worse than baseline because the low-resolution stage fails to establish robust semantic structures; combining RM+UP achieves the best result (HPS 31.99 vs. 30.28 RM-only, 29.57 UP-only) [Table 3, §4.3.1].
Resolution-stage allocation ablation (NFE = N_lo + N_hi for a total of 4 steps on PixArt-α): 2+2 is the sweet spot at 1.68× speedup, HPS 32.13, CLIP 33.86; pushing to 3+1 (only one high-res refinement step) collapses quality — HPS 24.79, CLIP 32.22 [Table 4, §4.4].
Training is data-free: only prompts from JourneyDB are used; bilinear interpolation for upsampling; CFG scales 7.5 (SDXL), 3.5 (PixArt-α), 4.5 (SD3.5), 5.0 (Wan2.1); single A100 for benchmarks [§4.1].

Method

A teacher diffusion model defines a logSNR trajectory; RMD partitions the trajectory at predefined logSNR thresholds into K segments and assigns each segment a resolution H_k (monotonically increasing toward the teacher’s full resolution at the lowest-noise end). Under Rectified Flow, each logSNR threshold is converted to a teacher timestep boundary (Eq. 1); the resolution-dependent SNR shift (Eq. 2, following SD3) maps each teacher segment onto a student timestep interval at the corresponding lower resolution, so distillation happens over matched denoising states regardless of spatial scale.

Within each segment, RMD performs DMD-style distribution matching, but lifted across resolutions: the student rolls out a cascaded trajectory producing low-resolution states z_t^k; each is upsampled into the teacher’s full-resolution space via the differentiable transformation in Eq. 10, which mixes (a) the student’s predicted noise upsampled to high resolution (to inherit the teacher ODE) and (b) i.i.d. Gaussian noise scaled by the time-dependent σ_t (to bridge cross-resolution distribution shift). The mixing weight α is reduced when the resolution gap is larger.

The DMD-style gradient (Eq. 8) compares the upsampled student distribution against the teacher’s high-resolution distribution using the standard true-score-minus-fake-score trick — a fake diffusion model is fine-tuned online on student outputs (Eq. 9) to estimate the generator score. Training proceeds in two phases: a warm-up that distills only the low-logSNR (semantic) interval to establish stable low-resolution layout, then end-to-end training across all stages.

At inference, denoising starts from Gaussian noise at the lowest resolution; the schedule decides at each step whether to upsample. When the resolution remains constant, a standard conditional denoising step is taken; when the resolution changes, the current state is projected into the next stage via the Eq. 10 transformation and noise is re-injected to maintain consistency with the high-resolution diffusion trajectory. For SDXL the default schedule is 2 low-res + 2 high-res = 4 NFE; for Wan2.1-T2V-14B the cascade extends from 480p to 720p across multiple stages with a default 2+2+2 / 3+3 distribution.

Results

Image text-to-image (Table 1, HPS v2.1 prompt set):

SDXL @ 2+2 NFE: 33.4× over base; HPS 31.99 vs DMD2 31.30, TDM 31.93; AeS 6.73 vs DMD2 6.71, TDM 6.70; CLIP 35.13 vs DMD2 35.10, TDM 35.03. RMD wins on all three at the same NFE budget.
PixArt-α @ 2+2 NFE: 21.0× speedup; HPS 32.23, AeS 6.89, CLIP 33.89 — all best in class.
SD3.5 medium @ 2+2 NFE: 32.0× speedup; HPS 30.56, AeS 6.38, CLIP 32.51 — beats DMD2 and SD3.5-Turbo.

Video text-to-video (Table 2, Wan2.1-T2V-14B base model, VBench + T2V-Compbench):

Base model (502 NFE): 1.0× / 83.75 / 85.44 / 76.95 / 54.17.
DMD2 (6 NFE): 16.7× / 80.30 / 81.98 / 73.60 / 52.81.
TDM (6 NFE): 16.7× / 80.48 / 81.85 / 75.00 / 52.27.
RMD: 25.6× / 82.51 / 84.37 / 75.05 / 54.00. RMD recovers most of the base-model quality while being >50% faster than 6-NFE TDM/DMD2 baselines. Qualitative comparison (Fig. 6) reports better motion detail and semantic coherence than DMD2.

Ablations:

Component (Table 3): RM-only HPS 30.28 / AeS 6.63 / CLIP 34.81 / VBench 80.81; UP-only HPS 29.57 / AeS 6.58 / CLIP 33.68 / VBench 81.11; RM+UP HPS 31.99 / AeS 6.73 / CLIP 35.13 / VBench 82.51. Naive cascaded upsampling without RM does worse than the 480p baseline on HPS/AeS/CLIP — establishes that the RM objective is doing the heavy lifting and that UP is a complementary efficiency lever, not an isolated quality lever.
Mixing factor α (Eq. 10) sweeps: pure Gaussian fails to inherit teacher ODE trajectory → weak structural consistency; pure predicted noise fails under large resolution gaps; optimal is an intermediate α [§4.3.2, Fig. 8].
Stage allocation (Table 4): N_lo=2, N_hi=2 best at 1.68× speedup and HPS 32.13; N_lo=3, N_hi=1 collapses (HPS 24.79, AeS 6.03) — the high-res refinement budget can’t be compressed to a single step.

Why it’s interesting

RMD is the cleanest published statement of “step distillation has saturated; cascade per-step cost instead” — the thesis Terrance highlighted in posting it. Two threads on the wiki converge here: (a) Diffusion Distillation — RMD slots in alongside DMD2 / TDM / DP-DMD / ArcFlow as a head-to-head Wan2.1-14B datapoint, and it directly beats both DMD2 and TDM at fewer effective NFE; the train/inference-gap solution from Transition Matching Distillation for Fast Video Generation (TMD) operates on a single resolution, so TMD’s flow-head rollout idea and RMD’s resolution-trajectory partitioning attack independent failure modes and look composable. (b) PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inference (PyramidalWan) is the wiki’s other “pyramid + distillation” datapoint and uses a similar three-stage 81×448×832 → 41×224×416 → 21×112×208 partition on Wan2.1-1.3B, but with a pyramidal flow-matching finetune followed by adversarial or DMD distillation as separate stages — RMD instead folds the cross-resolution alignment into the DMD objective itself via the logSNR-invariance mapping and the predicted-noise-vs-Gaussian mixing, which is a cleaner formulation; the two papers should now be read together as the “pretrained-teacher → pyramidal student” recipe space. Terrance’s note that megapatch combined with Luma’s distillation could give further speedup is exactly the right read: RMD is orthogonal to per-step token-count reduction (megapatch / patchifier rate / VAE compression) — both operate on different terms of the cost product.