Skip to content
Bud's Wiki

Diffusion training efficiency

TL;DR

Section titled “TL;DR”

Diffusion training efficiency tracks how to reach a target generation quality (FID, GenEval, FVD) in fewer epochs, fewer GPU-days, or fewer real-data samples. Three converging recipes in late-2025/early-2026: (1) supervise diffusion with strong external vision-encoder features (REPA, RAE, P-DINO) so the model doesn’t waste capacity rediscovering visual structure DINO already knows; (2) transfer a pretrained latent diffusion model into a new representation (pixel space, larger patches, etc.) by retraining only the boundary layers on synthetic source-model outputs; (3) replace pixel-space MSE with perceptual losses that put gradient signal where it matters. The shared headline: order-of-magnitude convergence speedups over the standard “train a DiT from scratch with MSE on VAE latents” baseline, with no architectural exotica.

Key claims

Section titled “Key claims”
  • Multi-layer aggregation of a frozen vision encoder (sum the last k DINO layers, not just the final layer) combined with REPA-style intermediate-layer alignment yields >10× convergence speedup over the original RAE recipe, reaching gFID 1.06 on ImageNet-256 in 80 epochs versus 800 epochs for prior best, with gFID ≤ 2 reached in 35 epochs vs 177 (Improved Baselines with Representation Autoencoders §2.1, EP_FID@2 metric).
    • Conflict: Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis reports that external alignment with DINO does not scale: replacing DINOv2-B with progressively stronger DINOv2-L / DINOv3-B / DINOv3-H+ in REPA monotonically degrades FID, and a 625M Self-Flow model beats a 1B REPA model at T2I (§3.2, §4.1, Fig. 6). The disagreement is not about ImageNet-class-conditional at fixed encoder, where RAEv2’s numbers stand, but about whether the recipe extends to bigger encoders, bigger generators, and modalities beyond ImageNet.
  • A frozen LDM can be transferred into pixel space by training only its shallow layers on synthetic data from the source LDM itself, recovering 93% of source-model GenEval and full DPG-Bench performance on 8 GPUs and unlocking 4K generation (L2P: Unlocking Latent Potential for Pixel Generation abstract).
  • Pixel-native diffusion with LPIPS + Perceptual-DINO losses reaches FID 5.11 on ImageNet-256 without CFG in 80 epochs, framed by the authors as beating latent diffusion at parity epochs — the headline efficiency claim is “better FID in far fewer iterations than latent models” (PixelGen: Improving Pixel Diffusion with Perceptual Supervision Abstract; Slack note).
  • DINOv2 features are the common substrate across all three efficient-training recipes — used as the encoder (RAE), as an intermediate-layer alignment target (REPA), and as an output-space perceptual loss on x-prediction (P-DINO). The same external prior is being plumbed in at different points in the network (Improved Baselines with Representation Autoencoders §2.1; PixelGen: Improving Pixel Diffusion with Perceptual Supervision §3).
  • RAE (encoder-side) and REPA (intermediate-layer alignment) are complementary, not substitutive — combining them gives further gains and simplifies guidance by reusing the REPA head (Improved Baselines with Representation Autoencoders §2.1). Self-Flow further stacks on top of RAE: applied to RAE on ImageNet it pushes FID 3.24 → 2.95 at 1M steps (Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis Tab. 1).
  • Training-time fixes that remove the need for CFG at inference are an orthogonal efficiency lever: folding the guided velocity into the Flow Matching objective (SoFlow) or augmenting the training noise distribution so the model self-corrects (Ostris) both halve per-step inference cost and simplify few-step distillation (Random noise augmentation during diffusion training eliminates the need for CFG; see Classifier-Free Guidance variants).
  • Latent design — not just diffusion-side recipes — is itself an efficiency lever: tying the encoder’s noise floor to the diffusion prior’s minimum noise level reduces the KL to a noise-weighted MSE and provides explicit bits/pixel control via a “loss factor” hyperparameter, beating Stable Diffusion latents at matched architecture and training FLOPs on ImageNet-512 and reaching SOTA FVD 1.3 on Kinetics-600 (Unified Latents (UL): How to train your latents §3.1, §5.3, §5.7).
  • Optimal latent bitrate is base-model-capacity-dependent: smaller models prefer lower-bitrate (easier-to-model) latents, larger models can absorb more informative ones — there is no universal “right” latent compression ratio, only a model-size-conditional one (Unified Latents (UL): How to train your latents §5.3, Table 2).
  • Parameterization choice gates which simplifications of the training signal are admissible: dropping noise conditioning (a potential efficiency lever — one fewer input the network must plumb) is structurally safe only under velocity prediction (Flow Matching, EqM); it provably fails under noise-prediction (DDPM/DDIM) because the effective gain diverges as 1/σt1/\sigma_t and amplifies a finite Jensen Gap (The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning §6, Table 2; §7.1 empirical on CIFAR-10/SVHN/Fashion-MNIST).
  • External alignment fails to generalize across modalities: V-JEPA 2 and Depth Anything 3 hurt video generation FVD relative to vanilla flow matching, and MERT gives no benefit over vanilla flow for audio. External-encoder REPA gains are largely an ImageNet+DINOv2 artifact; self-supervised internal alignment recovers gains across all three modalities simultaneously (Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis §4.1, Tab. 3, Tab. 4).
  • Self-Flow scales with compute and model size; REPA does not: across 290M / 420M / 625M / 1B T2I models the Self-Flow-vs-REPA FID gap widens with scale — REPA flattens after ~625M while Self-Flow continues to improve (Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis §4.1, Fig. 6). This is the first reported scaling failure of external alignment and is the key practical reason to consider internal-self-distillation over REPA for >1B-parameter regimes.
  • Internal-self-distillation already beat REPA on T2I before Self-Flow: SRA (May 2025) reports FID 3.70 on T2I (FLUX.2 backbone, 1M steps), beating REPA 3.92 and vanilla 4.08, by aligning the DiT’s earlier-layer-noisier-view features to later-layer-cleaner-view features in a single network — no EMA teacher, no second forward pass (No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA) project page; numbers from Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis Tab. 2 reproduction). Self-Flow then extends this with an EMA-decoupled teacher and Dual-Timestep Scheduling, pushing T2I FID further to 3.61 and improving ImageNet from SRA’s 7.27 to 5.70.
  • Spatial token-count reduction via foveated post-training is a sixth, orthogonal efficiency axis: rather than reducing steps, epochs, or latent bitrate, Foveated Diffusion reduces the token count per forward pass by 4× in the periphery, using a LoRA-rank-32 post-training pass on FLUX.2 Klein 4B / Wan2.1 1.3B. Net 2× image / 4× video speedup at perceptual parity with full-resolution generation in a 2AFC user study (Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation §4.2, Table 1, Table 2). The training-free version of the same recipe fails — naïve mixed-resolution denoising produces duplicated objects — so this lever requires post-training, unlike caching/serving-side speedups.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Flow Matching in Feature Space for Stochastic World Modeling: FlowWM contributes a one-step endpoint-projection trick for training flow-matching models with downstream-task supervision: rather than backpropagating a frozen detector/reward through the full ODE integration (costly, gradient-unstable, per prior differentiable-reward work), approximate the endpoint linearly as ẑ_1 ≈ z_t + (1 − t)·v_θ(z_t, t) and apply the task loss on ẑ_1 alone. In FlowWM this gives a stable temporal-consistency loss (matching predicted-endpoint temporal derivatives against ground truth DINOv3 feature derivatives, +5.2% relative AP(3) at no additional training FLOPs) and a stable detector-loss backprop through the world model. Also confirms the SD3-style dimension-aware noise-schedule shift transfers beyond high-resolution image generation to high-dimensional-feature video world modeling.
  • [2026-07-19] Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation: SpectraReward eliminates the reward-model-training step from diffusion RL post-training entirely — no preference labels, no verifier fine-tuning, no reward-head, just a single teacher-forced forward pass through a pretrained MLLM to compute the prompt log-likelihood. Reported to outperform prior MLLM-derived reward training methods across two diffusion policies, three RL algorithms, nine MLLM backbones (4B–235B), and five OOD T2I benchmarks. Frames a training-efficiency argument for skipping the reward-model-scaling arms race.
  • [2026-07-16] VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders: VideoRAE reports ~5× faster autoencoder convergence than baselines and faster downstream generator convergence when a frozen VFM (V-JEPA 2 / VideoMAEv2) latent — compressed by a 1D self-attention projector — replaces LTX-VAE at 2B-scale T2V under matched settings. Extends the image-domain RAE convergence-speed argument to video.
  • [2026-07-04] RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation: Step Forcing reduces AR video-diffusion train–test mismatch by matching the training denoising schedule to the 4-step inference schedule and interleaving data-grounded anchor steps, keeping action–observation dynamics without the cost of full self-rollout training.
  • [2026-07-04] From SRA to Self-Flow: Attention Separation Reveals the Real Mechanism Is Data Augmentation: Attention Separation ablation shows Self-Flow’s gain over SRA comes from data augmentation along the noise dimension, not from cross-noise token interaction as Self-Flow claimed — reframes the lever inside self-alignment recipes as an augmentation operator.
  • [2026-06-29] Qwen-Image-2.0-RL Technical Report: Qwen-Image-2.0-RL is the post-training recipe for Qwen-Image-2.0 — a fully-documented GRPO + on-policy-distillation pipeline (split into two task specialists then merged) showing +2.61 Qwen-Image-Bench and +78/+93 arena Elo without changing the base architecture. Datapoint on how much headroom remains in RL post-training of an already-strong unified gen+edit model.
  • [2026-06-27] Introducing Un-0: Generating Images with Coupled Oscillators: Un-0 applies the drifting-loss + DINOv2 perceptual-judge recipe to a Kuramoto-oscillator backbone (not a neural network), reaching FID 6.74 on ImageNet 64×64 — evidence the loss family generalizes beyond conventional architectures.
  • [2026-06-24] Is Noise Conditioning Necessary for Denoising Generative Models?: Sun et al. (Kaiming He) show timestep conditioning is not indispensable: posterior p(t|x_t) concentrates as O(1/d) on high-dim natural images, so most diffusion variants (iDDPM, EDM, FM, Consistency Models) degrade only gracefully when t-input is removed; uEDM achieves FID 2.23 vs 1.99 t-conditional baseline on CIFAR-10. Only DDIM-ODE fails catastrophically (FID 40.90), predicted by a training-free accumulated-error bound.
  • [2026-06-24] Krea 2 Technical Report: Krea 2 technical report stacks several efficiency moves from this concept on a production T2I MMDiT: iREPA enabled only for the first epoch of 256 px pretraining and then removed; 8-bit training (tensorwise then rowwise) delivers 15–20% wall-clock speedup at 256/512 px vs bf16 with negligible quality loss; per-block AdaLN-MLP (20–30% of params) replaced by a tunable bias term, freeing parameters for attention/MLP; PMA model merging used in place of EMA at no quality cost and without EMA’s memory overhead.
  • [2026-06-24] Next Forcing: Causal World Modeling with Multi-Chunk Prediction: Next Forcing adds multi-chunk prediction auxiliary heads (next¹/²/³) to causal-AR video denoisers, giving dense multi-scale temporal supervision; 2.3× faster convergence and >50% FVD reduction on general video pretraining vs. single-current-chunk supervision.
  • [2026-06-23] i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models: i1 contributes three controlled-ablation findings to the efficiency repertoire at 3B / 1024² scale: (a) AdaLN removal saves ~0.25B params while improving DPG/PRISM/LongText, (b) a 2-block transformer adapter on a single text encoder captures most of the gain from using multiple encoders without increasing text sequence length, (c) equal weighting of 12 datasets beats threshold-based mixing at all four tested thresholds. Not a new training recipe, but the strongest filed validation that “remove AdaLN, grow the adapter, equal-weight the datasets” is the right default.
  • [2026-06-18] MiniT2I: Pixel-Space Text-to-Image with MM-JiT at 258M Parameters: MiniT2I (258M params, MM-JiT) reports T2I training cost equivalent to a single ImageNet diffusion experiment — pixel-space + x-prediction + MM-DiT-style text fusion as a budget-T2I recipe, undergraduate-scale.
  • [2026-05-28] DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation: DiffusionBlocks (Sakana AI, ICLR 2026) adds a seventh orthogonal efficiency axis: block-to-step assignment with no end-to-end backprop. Reinterprets residual blocks as Euler steps of the reverse PF-ODE and partitions noise levels across K independent denoiser blocks via equi-probability partitioning under the EDM prior + 10% controlled overlap. Reports K-fold training-memory reduction and ~K× inference speedup with FID 15.55 vs 16.62 (ImageNet-256, DiT-L, K=4) and MAUVE 0.779 vs 0.595 (LM1B). Distinct from the existing axes — doesn’t change supervision (REPA/SRA/Self-Flow), latent design (UL/RAE), perceptual loss (PixelGen), spatial tokens (Foveated), or step count; it removes end-to-end backprop itself.
  • [2026-05-25] SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers: SEGA is an inference-only, training-free RoPE-component scaling recipe that lets frozen pretrained DiTs (FLUX, Qwen) generate at 4096² by dynamically rescaling rotary dimensions according to per-axis spectral energy in the current latent — complements training-time recipes like NiT.
  • [2026-05-23] Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation (Wetzstein group, Stanford): adds spatial token-count reduction to the efficient-training repertoire. Post-trains FLUX.2 Klein 4B (image) and Wan2.1 1.3B (video) with LoRA rank 32 to operate on a mixed-resolution token grid — HR near a foveation mask, LR (k×k merged) in the periphery — with a modified RoPE that aligns key/query phases across resolutions. Headline: 2× image / 4× video speedup at perceptual parity with full-res in a 2AFC user study. Notable negative result: the training-free version of the same recipe fails with duplicated objects and scale mismatches, which is what motivates the post-training step. Also a sharp FID-vs-user-study disagreement: the naïve baseline beats full-res on FID despite visible structural artifacts, reinforcing concerns about FID for evaluating efficiency tradeoffs at the high end.
  • [2026-05-23] No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA) (SRA, Jiang et al., May 2025): the predecessor of the internal-self-distillation strand. Aligns the DiT’s earlier-layer output features at higher noise to later-layer output features at lower noise within the same network — no external encoder, no EMA teacher copy. Reports comparable-or-superior to REPA on DiTs/SiTs across model sizes. Third-party numbers from Self-Flow’s tables: T2I FID 3.70 (beats REPA 3.92), ImageNet SiT-XL FID 7.27 (trails REPA 5.89). Crucial historical context for the Self-Flow result — establishes that the internal-supervision idea predates Self-Flow’s EMA-teacher mechanism, and Self-Flow’s contribution is to scale that idea past REPA.
  • [2026-05-23] Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis (Self-Flow, BFL + MIT): EMA-teacher self-distillation inside the generator via Dual-Timestep Scheduling — two noise levels mixed across tokens, teacher sees the cleaner view, student aligns to teacher features via cosine similarity at layer ℓ=8. Beats REPA on ImageNet (FID 5.70 < 5.89 at 4M steps), T2I (3.61 < 3.92), T2V (FVD 47.81 < 49.59), and T2A (best CLAP-FAD). Stacks on RAE (FID 3.24 → 2.95). Critically, REPA degrades with stronger DINO encoders and stops scaling past 625M params, while Self-Flow recovers expected scaling laws (625M Self-Flow > 1B REPA). Strongest evidence to date that external-DINO supervision is a transitional recipe, not the Bitter-Lesson endpoint.
  • [2026-05-23] The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning: theoretical paper from Google Research (Sahraee-Ardakan / Delbracio / Milanfar) showing that noise-conditioning-free (“blind”) diffusion implements a Riemannian gradient flow on a Marginal Energy landscape. Proves that only velocity-prediction parameterizations (Flow Matching, EqM) are structurally stable when noise conditioning is dropped — DDPM noise-prediction fails (gain diverges, amplifies Jensen Gap) and EDM signal-prediction succeeds via exponential error decay. Empirically, DDPM-blind generates incoherent samples while FM-blind matches its conditional counterpart on CIFAR-10/SVHN/Fashion-MNIST. Relevant to this concept page as a constraint on which training-simplification levers are safe, not as a direct speedup claim.
  • [2026-05-23] Unified Latents (UL): How to train your latents (UL): jointly trains encoder + diffusion prior + diffusion decoder, with the encoder’s noise floor equal to the prior’s minimum noise level so the KL collapses to a tractable bitrate budget. Two hyperparameters (loss factor, sigmoid bias) control reconstruction-vs-modelability. Beats SD latents on ImageNet-512 at matched FLOPs; SOTA FVD 1.3 on Kinetics-600.
  • [2026-05-23] Random noise augmentation during diffusion training eliminates the need for CFG: per-channel additive + log-normal noise augmentation during training reportedly eliminates the need for CFG at inference, halving per-step compute and avoiding the “AI look.” Qualitative LoRA result on Z-Image; no quantitative metrics yet.
  • [2026-05-23] PixelGen: Improving Pixel Diffusion with Perceptual Supervision: pixel-native DiT + LPIPS + P-DINO + REPA reaches FID 5.11 in 80 epochs without CFG. Locates the source of latent-vs-pixel-space training inefficiency in the MSE loss, not the representation.
  • [2026-05-22] Improved Baselines with Representation Autoencoders (RAEv2): >10× convergence speedup over original RAE via multi-layer encoder aggregation and complementary use of REPA. SOTA gFID 1.06 on ImageNet-256 in 80 epochs.
  • [2026-05-22] L2P: Unlocking Latent Potential for Pixel Generation (L2P): trains only shallow layers of a frozen LDM on the source model’s own synthetic outputs, achieving on-par DPG-Bench and 93% GenEval on 8 GPUs.

Open questions

Section titled “Open questions”
  • Three independent late-2025/early-2026 recipes all converge on “80 epochs to strong ImageNet-256 FID” — is 80 epochs the right benchmark, or is it where the metric saturates and a deeper efficiency frontier exists past it?
  • DINOv2 features appear in all three external-supervision recipes (encoder / intermediate alignment / output-space loss). Is there a unified theory of “where DINO supervision is best applied” or is it task/scale-dependent? PixelGen + RAEv2 hint that all three locations can be stacked. SRA and Self-Flow’s counter-claim is that none of these locations is necessary if the generator self-distills.
  • External-DINO vs internal-self-distillation is now an active fork with two filed papers on the internal side. RAEv2 / RAE / P-DINO all report SOTA with external supervision at moderate scale; SRA and Self-Flow report SOTA without external supervision, with Self-Flow showing a widening gap with scale. The disagreement is most likely about scale and modality, not about ImageNet-class-conditional FID at fixed encoder size. Resolving this needs (a) RAEv2 + scaling-encoder ablation, and (b) Self-Flow at the 1B+ video scale Luma cares about.
  • What is the structural difference between SRA’s and Self-Flow’s internal-teacher recipes that produces the ImageNet gap? SRA: same-network earlier-layer-noisier → later-layer-cleaner; Self-Flow: EMA-decoupled teacher on cleaner-uniform-noise view + student on Dual-Timestep mix. Both work; Self-Flow wins by ~1.5 FID points on ImageNet. The natural ablation is “SRA + EMA teacher” vs “Self-Flow without Dual-Timestep” — neither paper runs it.
  • How do these training-efficiency recipes interact with few-step generation (SoFlow et al.) and Classifier-Free Guidance variants? Most reported numbers are at multi-step sampling with no CFG; the practical question is end-to-end inference cost, not just training cost.
  • Do the recipes transfer to video diffusion, where the analog of LPIPS / P-DINO needs temporal extension and the analog of REPA needs a video-foundation-model target (V-JEPA? frozen video DINO?)? UL’s Kinetics-600 SOTA at FVD 1.3 is the first datapoint that “latent-design efficiency” transfers to video. Self-Flow now provides a second: it improves T2V FVD by ~2 points over REPA and it actively repairs the case where external video encoders (V-JEPA 2, Depth Anything 3) underperform vanilla FM.
  • Synthetic-data-only training (L2P) raises a self-improvement question: can the same recipe iterate — pixel model trained on LDM outputs becomes the source for a next-gen pixel model trained on its own outputs — without manifold collapse?
  • UL frames latent bitrate as scale-dependent: smaller models prefer simpler latents, larger models can absorb richer ones. Does the DINO-as-encoder family (RAE) have an analogous capacity-dependent feature-depth choice? Do PixelGen’s perceptual-loss weights need rescheduling with model size?
  • Does dropping noise conditioning (per The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning) compose with the DINO/REPA/perceptual-loss recipes here, or do those recipes implicitly require noise conditioning to align features at specific noise scales? Untested across all filed papers — but a clean ablation given the parameterization compatibility constraint is now known.
  • Self-Flow needs an extra teacher forward per step. SRA, by sharing the network between teacher and student views, may avoid that overhead — at the cost of weaker alignment. What is the compute-matched head-to-head between SRA and Self-Flow?
  • Does spatial token-count reduction (Foveated Diffusion) stack with the other recipes? In principle a foveated DiT post-trained from a RAE or Self-Flow base inherits the upstream training-efficiency gains, but no filed paper has measured the compose. Also unknown: whether saliency-guided foveation can replace gaze input for non-interactive batch inference at meaningful speedups.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-01

Section titled “2025-01”

2024-12

Section titled “2024-12”

2024-02

Section titled “2024-02”