DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers

TL;DR

DDiT lets a pretrained Diffusion Transformer (FLUX.1-dev, Wan-2.1) operate on multiple latent patch sizes — original p and 2p, 4p — and picks the patch size per denoising step at inference time, swapping in larger (coarser) patches when the latent’s local “acceleration” is low and smaller (finer) patches when it’s high. The architectural delta is tiny: per-patch-size embedding/de-embedding layers, a shared LoRA branch in each transformer block, a residual bypass around patchify, and a learnable patch-size identifier token; LoRA is distilled against the frozen base model on synthetic data from the base model itself. At a matched ~5.5 s/image budget on FLUX, DDiT reports a 2.2× speedup over the 50-step baseline with effectively unchanged FID and CLIP, and stacks with TeaCache to a 3.5× end-to-end speedup while still slightly beating caching-only baselines at similar wall-clock.

Key claims

The acceleration of the denoising latent (third-order finite difference of $z_t$ ) is the signal that best predicts whether the current step is producing coarse or fine detail; first/second-order differences are too short-horizon and Table 3 shows monotonically improving FID/CLIP/ImageReward as the difference order rises from 1 to 3 [§3.3, Table 3].
A patch-size-adaptive DiT can be obtained from a frozen pretrained DiT with only added per-patch-size patch-embedding layers + a rank-32 LoRA + a learnable patch-size token + a residual bypass; training is distillation against the frozen base model’s noise prediction on synthetic prompts [§3.2, Eq. 1].
Per-pixel acceleration is aggregated to per-patch standard deviation (not mean) and reduced across patches by the 80th percentile, not the mean, to avoid smoothing out localized high-detail regions in scenes with mixed background/foreground content [§3.3].
The scheduler is parameter-free at inference modulo a single threshold τ on per-patch variance that exposes a clean speed/quality knob; larger τ degrades FID/CLIP/ImageReward only slightly while raising speedup from 1.88× (τ=0.6) to 3.52× (τ=1.0) [Table 4].
On FLUX-1.dev / COCO at 1024² × 50 steps, DDiT delivers 2.18× speedup (12.0 → 5.5 s) with FID 33.42 vs base 33.07, CLIP 0.317 vs 0.314, ImageReward 1.0284 vs 1.0291 — i.e. quality essentially preserved [Table 1].
At matched inference time, DDiT beats both TeaCache and TaylorSeer on every reported quality metric (FID, CLIP, ImageReward, LPIPS, SSIM) on FLUX [Table 1, rows 4/6/8].
DDiT composes with TeaCache: DDiT + TeaCache(δ=0.3) reaches 3.52× speedup at FID 33.60, still ahead of TeaCache-only and TaylorSeer-only at similar speed [Table 1, row 9].
The method extends to text-to-video on Wan-2.1 1.3B (480p, 81 frames, 50 steps) and reduces inference time while preserving VBench scores [§4.3].
User study: DDiT vs FLUX-50-step is preferred 17% / no-preference 61% / FLUX 22% — generations are visually indistinguishable a majority of the time despite >2× speedup [§4.4].
The scheduler is content-aware in a strong sense: for prompts describing simple/uniform scenes it spends more steps at coarse patches (4p/2p); for prompts with fine-grained textures it spends more steps at the native patch size p [§4.4, Fig. 10].

Method

DDiT modifies a pretrained DiT minimally. The native patch size is $p$ (the size the model was trained with); DDiT adds support for $p_k = k \cdot p$ with $k \in \{2, 4\}$ by introducing per- $p_k$ patch-embedding weights $(W_k, b_k)$ that linearly project a $p_k \times p_k$ latent patch into the same embedding dimension $D$ . Positional embeddings are bilinearly interpolated from the native size. A single learnable $D$ -vector is added to every token as a patch-size identifier, so the same transformer trunk can disambiguate which $p_k$ is currently in use. A rank-32 LoRA branch is inserted into the FFN of every transformer block plus one residual block; LoRA is the only trainable adaptation, distilled against the frozen base model’s noise prediction on synthetic data (T2I-2M for FLUX, base-model-generated videos for Wan). A residual connection wraps the patch embed → transformer → patch de-embed pipeline so the new manifold (for $p_k > p$ ) doesn’t have to be learned from scratch.

At inference the scheduler picks $p_k$ per step. For each step it computes the third-order finite difference $a_t$ pointwise on the latent, divides $a_t$ into candidate patches of every supported $p_k$ , computes the per-patch standard deviation, and takes the 80th percentile across patches as a “spatial-variance” score $\sigma_k$ for that $p_k$ . It then selects the largest $p_k$ whose $\sigma_k$ is below a fixed threshold $\tau$ ; if none satisfies, it falls back to native $p$ . Empirically: when the latent is evolving slowly (low acceleration) coarse patches suffice; when acceleration spikes (fine-detail synthesis stage) the scheduler reverts to native $p$ . The scheduler is training-free and has only one hyperparameter, $\tau$ , which directly knobs the speed/quality tradeoff.

Results

FLUX-1.dev, COCO/DrawBench/PartiPrompts at 1024², 50 inference steps, guidance 3.5: DDiT 2.18× speedup (12.0 → 5.5 s/image) at FID 33.42 vs 33.07 base, CLIP 0.317 vs 0.314, ImageReward 1.0284 vs 1.0291 [Table 1].
At matched ~5.5–6 s/image: DDiT beats TeaCache (δ=0.4) FID 33.42 vs 34.95 and CLIP 0.317 vs 0.303; beats TaylorSeer (N=8) FID 33.42 vs 34.74 and CLIP 0.317 vs 0.303 [Table 1].
DDiT + TeaCache(δ=0.3): 3.52× speedup at FID 33.60, still ahead of step-reduction baseline FLUX-15-step (FID 34.02) at similar wall-clock [Table 1].
Wan-2.1 1.3B T2V at 480p × 81 frames × 50 steps: reduced inference time at competitive VBench [§4.3].
Threshold τ ablation: τ=0.6 → 1.88× speedup at CLIP 0.3148 / ImageReward 1.0271; τ=1.0 → 3.52× at CLIP 0.3082 / ImageReward 1.0124 — degradation small even at the aggressive end [Table 4].
Difference-order ablation: 1st-order CLIP 0.2927 / ImageReward 0.9782; 3rd-order CLIP 0.3136 / ImageReward 1.0284 — third-order is materially better [Table 3].
User study: DDiT preferred 17% / tie 61% / base 22% [§4.4].

Why it’s interesting

DDiT introduces an axis of efficient diffusion that is largely orthogonal to the two already on the wiki. (1) Algorithmic step-reduction (distillation): NVIDIA FastGen: Fast Generation from Diffusion Models compresses the number of denoising steps; DDiT keeps the step count but varies the cost per step by changing token count via patch size. (2) Approximate step caching: Run FLUX.1-dev three times faster reuses previous-step activations when First Block Caching / TEACache detects low residual change; DDiT’s signal (third-order latent acceleration) is a strict relative — both exploit smooth temporal stretches of the denoising trajectory, but caching reuses computation while DDiT re-quantizes it to a coarser spatial grid. The fact that DDiT and TeaCache stack to 3.5× without quality loss [Table 1] is the cleanest evidence yet that these are independent levers. The content-adaptive aggregation (80th-percentile per-patch std-dev, not mean) is also conceptually adjacent to the pixel-space content-aware token reduction in Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (RLT) — both decide locally where compute should go, but RLT does it in the encoder path of video ViTs by dropping temporally-redundant patches, while DDiT does it in the generation path of DiTs by upsizing patches in low-acceleration steps. For Luma’s video DiTs the obvious composition is: DDiT-style dynamic patch size × FBC/TeaCache step-caching × FastGen-style step distillation — three independent multipliers on top of one base model.