Skip to content
Bud's Wiki

Diffusion training efficiency

TL;DR

Section titled “TL;DR”

Diffusion training efficiency tracks how to reach a target generation quality (FID, GenEval, FVD) in fewer epochs, fewer GPU-days, or fewer real-data samples. Three converging recipes in late-2025/early-2026: (1) supervise diffusion with strong external vision-encoder features (REPA, RAE, P-DINO) so the model doesn’t waste capacity rediscovering visual structure DINO already knows; (2) transfer a pretrained latent diffusion model into a new representation (pixel space, larger patches, etc.) by retraining only the boundary layers on synthetic source-model outputs; (3) replace pixel-space MSE with perceptual losses that put gradient signal where it matters. The shared headline: order-of-magnitude convergence speedups over the standard “train a DiT from scratch with MSE on VAE latents” baseline, with no architectural exotica.

Key claims

Section titled “Key claims”
  • Multi-layer aggregation of a frozen vision encoder (sum the last k DINO layers, not just the final layer) combined with REPA-style intermediate-layer alignment yields >10× convergence speedup over the original RAE recipe, reaching gFID 1.06 on ImageNet-256 in 80 epochs versus 800 epochs for prior best, with gFID ≤ 2 reached in 35 epochs vs 177 (Improved Baselines with Representation Autoencoders §2.1, EP_FID@2 metric).
    • Conflict: Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis reports that external alignment with DINO does not scale: replacing DINOv2-B with progressively stronger DINOv2-L / DINOv3-B / DINOv3-H+ in REPA monotonically degrades FID, and a 625M Self-Flow model beats a 1B REPA model at T2I (§3.2, §4.1, Fig. 6). The disagreement is not about ImageNet-class-conditional at fixed encoder, where RAEv2’s numbers stand, but about whether the recipe extends to bigger encoders, bigger generators, and modalities beyond ImageNet.
  • A frozen LDM can be transferred into pixel space by training only its shallow layers on synthetic data from the source LDM itself, recovering 93% of source-model GenEval and full DPG-Bench performance on 8 GPUs and unlocking 4K generation (L2P: Unlocking Latent Potential for Pixel Generation abstract).
  • Pixel-native diffusion with LPIPS + Perceptual-DINO losses reaches FID 5.11 on ImageNet-256 without CFG in 80 epochs, framed by the authors as beating latent diffusion at parity epochs — the headline efficiency claim is “better FID in far fewer iterations than latent models” (PixelGen: Improving Pixel Diffusion with Perceptual Supervision Abstract; Slack note).
  • DINOv2 features are the common substrate across all three efficient-training recipes — used as the encoder (RAE), as an intermediate-layer alignment target (REPA), and as an output-space perceptual loss on x-prediction (P-DINO). The same external prior is being plumbed in at different points in the network (Improved Baselines with Representation Autoencoders §2.1; PixelGen: Improving Pixel Diffusion with Perceptual Supervision §3).
  • RAE (encoder-side) and REPA (intermediate-layer alignment) are complementary, not substitutive — combining them gives further gains and simplifies guidance by reusing the REPA head (Improved Baselines with Representation Autoencoders §2.1). Self-Flow further stacks on top of RAE: applied to RAE on ImageNet it pushes FID 3.24 → 2.95 at 1M steps (Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis Tab. 1).
  • Training-time fixes that remove the need for CFG at inference are an orthogonal efficiency lever: folding the guided velocity into the Flow Matching objective (SoFlow) or augmenting the training noise distribution so the model self-corrects (Ostris) both halve per-step inference cost and simplify few-step distillation (Random noise augmentation during diffusion training eliminates the need for CFG; see Classifier-Free Guidance variants).
  • Latent design — not just diffusion-side recipes — is itself an efficiency lever: tying the encoder’s noise floor to the diffusion prior’s minimum noise level reduces the KL to a noise-weighted MSE and provides explicit bits/pixel control via a “loss factor” hyperparameter, beating Stable Diffusion latents at matched architecture and training FLOPs on ImageNet-512 and reaching SOTA FVD 1.3 on Kinetics-600 (Unified Latents (UL): How to train your latents §3.1, §5.3, §5.7).
  • Optimal latent bitrate is base-model-capacity-dependent: smaller models prefer lower-bitrate (easier-to-model) latents, larger models can absorb more informative ones — there is no universal “right” latent compression ratio, only a model-size-conditional one (Unified Latents (UL): How to train your latents §5.3, Table 2).
  • Parameterization choice gates which simplifications of the training signal are admissible: dropping noise conditioning (a potential efficiency lever — one fewer input the network must plumb) is structurally safe only under velocity prediction (Flow Matching, EqM); it provably fails under noise-prediction (DDPM/DDIM) because the effective gain diverges as 1/σt1/\sigma_t and amplifies a finite Jensen Gap (The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning §6, Table 2; §7.1 empirical on CIFAR-10/SVHN/Fashion-MNIST).
  • External alignment fails to generalize across modalities: V-JEPA 2 and Depth Anything 3 hurt video generation FVD relative to vanilla flow matching, and MERT gives no benefit over vanilla flow for audio. External-encoder REPA gains are largely an ImageNet+DINOv2 artifact; self-supervised internal alignment recovers gains across all three modalities simultaneously (Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis §4.1, Tab. 3, Tab. 4).
  • Self-Flow scales with compute and model size; REPA does not: across 290M / 420M / 625M / 1B T2I models the Self-Flow-vs-REPA FID gap widens with scale — REPA flattens after ~625M while Self-Flow continues to improve (Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis §4.1, Fig. 6). This is the first reported scaling failure of external alignment and is the key practical reason to consider internal-self-distillation over REPA for >1B-parameter regimes.
  • Internal-self-distillation already beat REPA on T2I before Self-Flow: SRA (May 2025) reports FID 3.70 on T2I (FLUX.2 backbone, 1M steps), beating REPA 3.92 and vanilla 4.08, by aligning the DiT’s earlier-layer-noisier-view features to later-layer-cleaner-view features in a single network — no EMA teacher, no second forward pass (No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA) project page; numbers from Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis Tab. 2 reproduction). Self-Flow then extends this with an EMA-decoupled teacher and Dual-Timestep Scheduling, pushing T2I FID further to 3.61 and improving ImageNet from SRA’s 7.27 to 5.70.
  • Spatial token-count reduction via foveated post-training is a sixth, orthogonal efficiency axis: rather than reducing steps, epochs, or latent bitrate, Foveated Diffusion reduces the token count per forward pass by 4× in the periphery, using a LoRA-rank-32 post-training pass on FLUX.2 Klein 4B / Wan2.1 1.3B. Net 2× image / 4× video speedup at perceptual parity with full-resolution generation in a 2AFC user study (Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation §4.2, Table 1, Table 2). The training-free version of the same recipe fails — naïve mixed-resolution denoising produces duplicated objects — so this lever requires post-training, unlike caching/serving-side speedups.

Recent contributions

Section titled “Recent contributions”
  • [2026-10-19] Generative Representation Learning for Scalable Latent Tokenization: RePR demonstrates that spending 20–35% of total (tokenizer + diffusion) compute on tokenizer encoder training is a Pareto improvement over raw pixel-space JiT at matched budget, with FID 2.18 on ImageNet-256 using JiT-L. First filed instance of a self-supervised tokenizer whose scaling law co-fits cleanly with the downstream generator’s budget without external encoder dependencies.
  • [2026-10-16] Dino Forcing Flow Models: Do not denoise what you can predict: Dino Forcing proposes a fourth location on the DINO-supervision axis: predict the clean DINO representation from the denoising state and condition the main image denoiser on that self-prediction, removing the second ODE and the representation-specific noise schedule that co-denoising recipes (RAE) require. Reports SOTA on ImageNet latent-space at 2× fewer epochs than prior methods and >20% FID improvement in pixel space — a convergence-speed datapoint arguing the second denoising trajectory in RAE-style recipes is substitutable by a prediction head.
  • [2026-10-16] Are We Overthinking One-Step Generation?: Priming Models hit ImageNet 256² one-step FID 1.19 at ~1/10 the training compute of comparable one-step generators (6.2× fewer FLOPs than Drifting-L, 12.2× fewer than Drifting-B) via a two-stage recipe: pixel-space reconstruction pretraining at a single fixed noise level, then feature-space distributional post-training. Just 1,250 post-training updates on Priming-B reach FID 2.26 from a pretrained reconstruction init that itself samples poorly from Gaussian noise — a sharp compute-efficiency datapoint for the “distributional training from scratch” axis.
  • [2026-10-13] Correction Training for Diffusion Models: Correction Training (Ahn, project page) adds a corrector network trained alongside the base denoiser with the same clean-target objective, then applied iteratively at inference. The parameter-matched LightningDiT ablation (14-layer denoiser + 14-layer corrector at K=3 compute-matched vs 28-layer monolithic denoiser) drops FID 8.93 → 5.66 on ImageNet-256 at 80K updates — the gain comes from repeating the corrector, not from adding parameters. A two-sample classifier shows the corrector pulls inference-time states back toward the training manifold even though training-input MSE rises; a complementary training-signal design to REPA / self-flow / correction-via-regression recipes already in this cluster.
  • [2026-10-09] SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation: SemanTok demonstrates that where the DINO signal enters a tokenizer determines how much efficiency it buys: REPA on an early decoder layer (VideoFlexTok) is partially satisfied from the decoder’s noised input, while feeding DINO into the encoder plus reconstruction heads over each retained token prefix lands the signal in the token sequence itself — enabling a 3.4× downstream AR-model-size reduction at matched quality.
  • [2026-10-08] Level-of-Token Diffusion: LoT Diffusion (Nakayama, Chao, Yariv, Wetzstein et al.) generalizes the Foveated Diffusion recipe from the same group to arbitrary multiresolution token layouts — semantic masks, bounding boxes, texture variance, depth-of-field, or agentic plans. Introduces a patch-wise asymmetric flow parametrization that keeps full-resolution flow prediction at every denoising step while the network processes only the reduced token sequence, with multiresolution token embeddings handling the mixed-granularity rectangular patches. Third filed paper on the spatial-token-count-reduction axis alongside Foveated Diffusion and ETC, and the one that removes the “which cue” commitment — any layout works with the same checkpoint.
  • [2026-10-05] Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training: Prism project page (Shuyuan Tu / francis-rings, likely Tencent-affiliated per Slack post, unverified) claims “dynamic sparse attention” applied during training to enable native-2K joint audio-video DiT training — i.e. a trainable-sparse-attention recipe at the resolution frontier rather than training-free inference-time sparsification. Zero technical disclosure, no benchmarks, no code at filing; bookmark pending paper/weights.
  • [2026-10-01] Improving Video Sparse Attention with Fine-grained Router and Sparse Rebasing: VSA2’s sparse rebasing lets progressive low-to-high resolution pretraining consume full-attention early-stage checkpoints mid-training — the first filed design point where sparse attention and resolution-progressive pretraining are explicitly co-designed rather than treated as independent levers. Paired with the Hard-to-Easy curriculum (train at higher sparsity than inference), this positions trainable sparse attention as an efficiency primitive for the entire DiT development cycle rather than an inference-time retrofit.
  • [2026-10-01] Looped Diffusion Transformer: Looped-DiT scales T2I DiTs via depth not parameters: iterates a shared Transformer block L times per denoising step with deep supervision across intermediate loops and self-modulating attention to prevent attention from washing out local information. A 260M looped model beats a 6.5× larger non-looped baseline at 4.9× lower inference compute — a parameter-efficiency datapoint complementary to ELT: Elastic Looped Transformers for Visual Generation (ELT), which attacked the same weak-intermediate-supervision problem via prefix-as-student self-distillation (ILSD) rather than explicit deep supervision.
  • [2026-09-17] Training Text-to-Image Models 3.6× Faster (JiT-DDT): Linum reports JiT-DDT trains a 512² pixel-space T2I model in 3.6× fewer GPU-hours than their Linum v2 LDM at 256² (i.e., 4× more pixels). Attributes the win to (a) an encoder-decoder split predicting low-res x₀ early to accelerate structure learning, (b) x-prediction + v-loss to escape the curse of dimensionality that breaks aggressive patchification under v-prediction, (c) Z-Image refiners + Muon on 2D weights + sigmoid attention gating + noise-schedule widening — the last being described as the single decisive change over vanilla JiT.
  • [2026-09-11] Elastic Token Compression for Pixel-Space Diffusion Transformers: ETC (Zamfir et al.) is the second filed spatial-token-count-reduction datapoint on this page after Foveated Diffusion. Differs from Foveated on three axes: (a) content-derived layout via Hilbert-curve reordering + feature-change cuts rather than saliency-/gaze-derived masks, (b) connected regions are guaranteed by construction, (c) a random-region-count fine-tune produces one elastic checkpoint serving every compression ratio from 1× to 2.6× rather than one post-training per ratio. Matches dense-token quality at 2.0× MiniT2I inference speed, close at 2.6×. Like Foveated, the naive training-free variant fails and the recipe requires a fine-tune — corroborating that spatial-token reduction is a training-side, not serving-side, lever.
  • [2026-09-03] GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling: GenFirst attacks the reconstruction–generation dilemma via joint end-to-end VAE + prior training with a “generation before reconstruction” curriculum, arguing the load-bearing pieces are (i) the KL entropy term against latent collapse and (ii) staging weak-then-strong reconstruction against generation’s slower learning dynamic. Claims first diffusion model < FID 1.0 (gFID 0.97 CFG / 1.45 no-CFG) on ImageNet-256 without an FD loss and ≥70× fewer training steps than a matched from-scratch SiT baseline. Sits alongside UL as an end-to-end joint-training answer to the “two-stage is leaving efficiency on the table” thesis but uses a curriculum schedule rather than UL’s bitrate-budget KL coupling.
  • [2026-08-31] There and Back Again: Bidirectional Diffusion Bridges for Multimodality Translation: BIT bets that bridging directly between source and target modality endpoints — rather than denoising from Gaussian noise conditioned on the source — is the load-bearing prior for multimodal generation, derived through stochastic calculus into scalable-loss SDEs. Reports competitive quality vs denoising-diffusion and deterministic-flow baselines on vision-language tasks and outperforms them on several natural-science evals. Design-axis complement to noise-conditioning-free diffusion and source-aware CFG-alternative recipes.
  • [2026-08-28] LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics: LeVJEPA (Kuhn et al., LeCun/Balestriero lab) is an SSL-side datapoint on the “aggressive token dropping is a training accelerator, not just a computational shortcut” thread: 95% uniform random patch-token dropping raises frozen-probe ImageNet-1K accuracy from 33.9 → 47.6% and delivers 5.6–20.8× less total compute vs V-JEPA 2 at matched epochs on identical data. Complements Less-to-More Generalization: Unlocking More Controllability by In-Context Generation / TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training on the diffusion side by showing the same monotone drop-rate-vs-accuracy curve on a joint-embedding predictive objective — different loss family, same lever.
  • [2026-08-26] On-Policy Self-Distillation in Diffusion Models: DiffusionOPSD’s outer-loop self-distillation cuts training GPU-hours vs DiffusionNFT by 40% on SD 3.5-M and 63% on Z-Image-Turbo at reward-matched quality, with the compute win larger on the already step-distilled backbone. Concrete evidence that decoupling target construction from finite fitting (with detached supervision + EMA refresh) is a materially cheaper post-training recipe than direct policy-gradient RL on the reverse trajectory.
  • [2026-08-18] Equilibrium Forcing (EqF) — extending Equilibrium Matching to budget-adaptive long-horizon video generation: Equilibrium Forcing (EqF) announcement — video extension of EqM (Equilibrium Matching: Generative Modeling with Implicit Energy-Based Models) claiming budget-adaptive sampling and long-horizon video gains. No paper/code at filing; watchlist entry.
  • [2026-08-05] CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization: CMuon adds an optimizer-side efficiency axis to the DiT-training playbook: rather than changing supervision (REPA/SRA/Self-Flow), latent design (UL/RAE), spatial tokens (Foveated), perceptual loss (PixelGen), or step assignment (DiffusionBlocks), it fixes Muon’s interaction with the fused AdaLN and QKV tensors ubiquitous in DiT codebases via a one-line chunk-before-orthogonalize change. 675M DiT / ImageNet 256² / 200 epochs → FID 1.18 with >2× reported speedup over AdamW at matched FID, past the late-stage plateau vanilla Muon exhibits on the same architecture. Adds a ninth distinct efficiency axis to this page and is the first filed paper here that ties the DiT training-efficiency literature to the training-stability-at-scale Muon lineage.
  • [2026-08-04] FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks: FACT (Parés-Morlans et al., tweet thread) surfaces a control-side use case for non-uniform timestep sampling: swapping the standard Beta sampler for a Logit-Normal schedule during flow-matching post-training of a VLA reallocates ~6× more gradient to the low-noise regime, which is where the sub-mm action deltas of contact-phase corrections live. Reports a jump from 41% to 66% average success on 5 contact-rich manipulation tasks (~2,500 real rollouts) attributable in part to this sampler change alone, with no new parameters/data/architecture — the payoff of a better noise schedule showing up as task-space precision in a specific control regime rather than as generic sample quality.
  • [2026-08-03] Scaling Properties of Text Conditioning in Visual Generation: Chen et al. identify prompt structuredness as a compute-efficiency lever orthogonal to model/data scaling: swapping raw natural-language captions for image-derived semantically + geometrically structured prompts moves converged diffusion loss along a measurable scaling axis (linear in GPG, power-law in ED). Prescriptively: attach object/attribute/relation labels and layout/spatial annotations to captions, and train an SFT → cold-start → verifier-gated on-policy prompter to bring inference-time short prompts back onto the training distribution.
  • [2026-07-31] Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples: Top-k GAN (Sinha et al., ICLR 2020) is the pre-diffusion-era ancestor of “one-line batch-level sample-selection tricks that improve generative-model training efficiency.” Discards the k lowest-critic-scored samples from the generator gradient each mini-batch, annealed. On a 25-Gaussian toy the discarded (bottom-k) updates actively push samples away from the nearest mode — bad samples are anti-signal, not just weak signal. Improves FID across six GAN variants at fixed hyperparameters and set CIFAR-10 conditional SOTA (9.21 → 8.57) on ICR-BigGAN. Filed as historical context: Terrance flagged it in a thread reply as “basically did this years ago with GANs” against the recently-filed Immiscible Diffusion recipe.
  • [2026-07-31] Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignment: Immiscible Diffusion (Li et al., 2024) adds an eighth axis distinct from the seven already tracked: batch-level noise-image reassignment (Hungarian matching, quantized to negligible overhead) that preserves the Gaussian noise marginal while forcing each image to be diffused only toward nearby noise. Reports ~3× Consistency Model / DDIM speedup on CIFAR-10, 1.3× on CelebA, larger FID gains at fewer inference steps (ΔFID -2.57 at 20 steps vs -0.15 at 1000 for DDIM). Filed as historical context flagged by Terrance as a comparison anchor in a thread reply.
  • [2026-07-31] Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers: Chimera reports 1.7× (dense backbone alone) and 7.3× (full KDA+MLA+MoE system) compute-efficiency at matched pretraining diffusion loss against a Wan-2.1 2B full-attention baseline. First filed matched-baseline compute-efficiency measurement of a hybrid-linear + MoE video DiT against a mainstream open-source full-attention DiT, and one of the largest wall-clock efficiency margins reported on the wiki for a visual generation backbone at this scale.
  • [2026-07-28] USP: Unified Self-Supervised Pretraining for Image Generation and Understanding: USP (Meituan, March 2025) adds a warm-start axis distinct from external-alignment (REPA) and internal-self-distillation (SRA / Self-Flow): masked latent modeling in a frozen VAE’s latent space pretrains a ViT whose weights initialize DiT / SiT directly. Reports 11.7× DiT-XL/2 and 46.6× SiT-XL/2 convergence speedup vs. from-scratch, and 15× / 5× total training cost reduction vs. DiT-from-scratch and REPA respectively at FID 9.6 parity. Naive MAE-weight loading actually hurts DiT (FID 43.55 vs. from-scratch 42.62); the load-bearing pieces are VAE-latent-space pretraining, VAE normalization matching, and PatchConv alignment.
  • [2026-07-28] From SRA to Self-Flow: Data Augmentation or Self-Supervision?: arxiv version of the SRA→Self-Flow Attention Separation ablation (blog previously filed): confirms via block-diagonal masking that Self-Flow’s gain over SRA is data augmentation along the noise dimension, not cross-noise reasoning; the combined self-alignment + dual-timestep + Attention Separation recipe sets new ImageNet 256²/512² CFG bests over SRA / Self-Flow / REPA.
  • [2026-07-24] SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation: Trains a 5B/14B hybrid video DiT from scratch (rather than linearizing a pretrained softmax model), with 25% softmax anchors and AttnRes chosen as the compute-optimal recipe via reduced-resolution proxy sweeps — an explicit “train the hybrid as a unit” datapoint for the linearize-vs-train-from-scratch question.
  • [2026-07-22] SANA — Efficient High-Resolution Image and Video Generation family (NVlabs/Sana, sol-engine branch): NVlabs/Sana bundles Sol-RL (NVFP4 rollout + BF16 optimization for diffusion RL post-training, claiming 4.64× faster convergence with training recipes for SANA / FLUX.1 / SD3.5-L) alongside the full SANA training stack (FSDP, TAR WebDataset, multi-scale training, DC-AE 32× tokenizer, sCM distillation). Concrete open-source implementation of the training-efficiency levers this concept tracks.
  • [2026-07-22] Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing: Mage-Flow reports ~2.5× end-to-end training throughput improvement attributed jointly to native-resolution variable-length packing (avoiding rigid resolution buckets + padding across 512–2048 pixels) and stack-level CUDA kernel fusion — a systems-side complement to the algorithmic efficiency levers this page tracks. Adds a Microsoft 4B open-source datapoint alongside Krea 2 and Z-Image on the “compact model + throughput engineering beats scale” trajectory.
  • [2026-07-20] Flow Matching in Feature Space for Stochastic World Modeling: FlowWM contributes a one-step endpoint-projection trick for training flow-matching models with downstream-task supervision: rather than backpropagating a frozen detector/reward through the full ODE integration (costly, gradient-unstable, per prior differentiable-reward work), approximate the endpoint linearly as ẑ_1 ≈ z_t + (1 − t)·v_θ(z_t, t) and apply the task loss on ẑ_1 alone. In FlowWM this gives a stable temporal-consistency loss (matching predicted-endpoint temporal derivatives against ground truth DINOv3 feature derivatives, +5.2% relative AP(3) at no additional training FLOPs) and a stable detector-loss backprop through the world model. Also confirms the SD3-style dimension-aware noise-schedule shift transfers beyond high-resolution image generation to high-dimensional-feature video world modeling.
  • [2026-07-19] Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation: SpectraReward eliminates the reward-model-training step from diffusion RL post-training entirely — no preference labels, no verifier fine-tuning, no reward-head, just a single teacher-forced forward pass through a pretrained MLLM to compute the prompt log-likelihood. Reported to outperform prior MLLM-derived reward training methods across two diffusion policies, three RL algorithms, nine MLLM backbones (4B–235B), and five OOD T2I benchmarks. Frames a training-efficiency argument for skipping the reward-model-scaling arms race.
  • [2026-07-16] VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders: VideoRAE reports ~5× faster autoencoder convergence than baselines and faster downstream generator convergence when a frozen VFM (V-JEPA 2 / VideoMAEv2) latent — compressed by a 1D self-attention projector — replaces LTX-VAE at 2B-scale T2V under matched settings. Extends the image-domain RAE convergence-speed argument to video.
  • [2026-07-04] RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation: Step Forcing reduces AR video-diffusion train–test mismatch by matching the training denoising schedule to the 4-step inference schedule and interleaving data-grounded anchor steps, keeping action–observation dynamics without the cost of full self-rollout training.
  • [2026-07-04] From SRA to Self-Flow: Attention Separation Reveals the Real Mechanism Is Data Augmentation: Attention Separation ablation shows Self-Flow’s gain over SRA comes from data augmentation along the noise dimension, not from cross-noise token interaction as Self-Flow claimed — reframes the lever inside self-alignment recipes as an augmentation operator.
  • [2026-06-29] Qwen-Image-2.0-RL Technical Report: Qwen-Image-2.0-RL is the post-training recipe for Qwen-Image-2.0 — a fully-documented GRPO + on-policy-distillation pipeline (split into two task specialists then merged) showing +2.61 Qwen-Image-Bench and +78/+93 arena Elo without changing the base architecture. Datapoint on how much headroom remains in RL post-training of an already-strong unified gen+edit model.
  • [2026-06-27] Introducing Un-0: Generating Images with Coupled Oscillators: Un-0 applies the drifting-loss + DINOv2 perceptual-judge recipe to a Kuramoto-oscillator backbone (not a neural network), reaching FID 6.74 on ImageNet 64×64 — evidence the loss family generalizes beyond conventional architectures.
  • [2026-06-24] Is Noise Conditioning Necessary for Denoising Generative Models?: Sun et al. (Kaiming He) show timestep conditioning is not indispensable: posterior p(t|x_t) concentrates as O(1/d) on high-dim natural images, so most diffusion variants (iDDPM, EDM, FM, Consistency Models) degrade only gracefully when t-input is removed; uEDM achieves FID 2.23 vs 1.99 t-conditional baseline on CIFAR-10. Only DDIM-ODE fails catastrophically (FID 40.90), predicted by a training-free accumulated-error bound.
  • [2026-06-24] Krea 2 Technical Report: Krea 2 technical report stacks several efficiency moves from this concept on a production T2I MMDiT: iREPA enabled only for the first epoch of 256 px pretraining and then removed; 8-bit training (tensorwise then rowwise) delivers 15–20% wall-clock speedup at 256/512 px vs bf16 with negligible quality loss; per-block AdaLN-MLP (20–30% of params) replaced by a tunable bias term, freeing parameters for attention/MLP; PMA model merging used in place of EMA at no quality cost and without EMA’s memory overhead.
  • [2026-06-24] Next Forcing: Causal World Modeling with Multi-Chunk Prediction: Next Forcing adds multi-chunk prediction auxiliary heads (next¹/²/³) to causal-AR video denoisers, giving dense multi-scale temporal supervision; 2.3× faster convergence and >50% FVD reduction on general video pretraining vs. single-current-chunk supervision.
  • [2026-06-23] i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models: i1 contributes three controlled-ablation findings to the efficiency repertoire at 3B / 1024² scale: (a) AdaLN removal saves ~0.25B params while improving DPG/PRISM/LongText, (b) a 2-block transformer adapter on a single text encoder captures most of the gain from using multiple encoders without increasing text sequence length, (c) equal weighting of 12 datasets beats threshold-based mixing at all four tested thresholds. Not a new training recipe, but the strongest filed validation that “remove AdaLN, grow the adapter, equal-weight the datasets” is the right default.
  • [2026-06-18] MiniT2I: Pixel-Space Text-to-Image with MM-JiT at 258M Parameters: MiniT2I (258M params, MM-JiT) reports T2I training cost equivalent to a single ImageNet diffusion experiment — pixel-space + x-prediction + MM-DiT-style text fusion as a budget-T2I recipe, undergraduate-scale.
  • [2026-05-28] DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation: DiffusionBlocks (Sakana AI, ICLR 2026) adds a seventh orthogonal efficiency axis: block-to-step assignment with no end-to-end backprop. Reinterprets residual blocks as Euler steps of the reverse PF-ODE and partitions noise levels across K independent denoiser blocks via equi-probability partitioning under the EDM prior + 10% controlled overlap. Reports K-fold training-memory reduction and ~K× inference speedup with FID 15.55 vs 16.62 (ImageNet-256, DiT-L, K=4) and MAUVE 0.779 vs 0.595 (LM1B). Distinct from the existing axes — doesn’t change supervision (REPA/SRA/Self-Flow), latent design (UL/RAE), perceptual loss (PixelGen), spatial tokens (Foveated), or step count; it removes end-to-end backprop itself.
  • [2026-05-25] SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers: SEGA is an inference-only, training-free RoPE-component scaling recipe that lets frozen pretrained DiTs (FLUX, Qwen) generate at 4096² by dynamically rescaling rotary dimensions according to per-axis spectral energy in the current latent — complements training-time recipes like NiT.
  • [2026-05-23] Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation (Wetzstein group, Stanford): adds spatial token-count reduction to the efficient-training repertoire. Post-trains FLUX.2 Klein 4B (image) and Wan2.1 1.3B (video) with LoRA rank 32 to operate on a mixed-resolution token grid — HR near a foveation mask, LR (k×k merged) in the periphery — with a modified RoPE that aligns key/query phases across resolutions. Headline: 2× image / 4× video speedup at perceptual parity with full-res in a 2AFC user study. Notable negative result: the training-free version of the same recipe fails with duplicated objects and scale mismatches, which is what motivates the post-training step. Also a sharp FID-vs-user-study disagreement: the naïve baseline beats full-res on FID despite visible structural artifacts, reinforcing concerns about FID for evaluating efficiency tradeoffs at the high end.
  • [2026-05-23] No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA) (SRA, Jiang et al., May 2025): the predecessor of the internal-self-distillation strand. Aligns the DiT’s earlier-layer output features at higher noise to later-layer output features at lower noise within the same network — no external encoder, no EMA teacher copy. Reports comparable-or-superior to REPA on DiTs/SiTs across model sizes. Third-party numbers from Self-Flow’s tables: T2I FID 3.70 (beats REPA 3.92), ImageNet SiT-XL FID 7.27 (trails REPA 5.89). Crucial historical context for the Self-Flow result — establishes that the internal-supervision idea predates Self-Flow’s EMA-teacher mechanism, and Self-Flow’s contribution is to scale that idea past REPA.
  • [2026-05-23] Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis (Self-Flow, BFL + MIT): EMA-teacher self-distillation inside the generator via Dual-Timestep Scheduling — two noise levels mixed across tokens, teacher sees the cleaner view, student aligns to teacher features via cosine similarity at layer ℓ=8. Beats REPA on ImageNet (FID 5.70 < 5.89 at 4M steps), T2I (3.61 < 3.92), T2V (FVD 47.81 < 49.59), and T2A (best CLAP-FAD). Stacks on RAE (FID 3.24 → 2.95). Critically, REPA degrades with stronger DINO encoders and stops scaling past 625M params, while Self-Flow recovers expected scaling laws (625M Self-Flow > 1B REPA). Strongest evidence to date that external-DINO supervision is a transitional recipe, not the Bitter-Lesson endpoint.
  • [2026-05-23] The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning: theoretical paper from Google Research (Sahraee-Ardakan / Delbracio / Milanfar) showing that noise-conditioning-free (“blind”) diffusion implements a Riemannian gradient flow on a Marginal Energy landscape. Proves that only velocity-prediction parameterizations (Flow Matching, EqM) are structurally stable when noise conditioning is dropped — DDPM noise-prediction fails (gain diverges, amplifies Jensen Gap) and EDM signal-prediction succeeds via exponential error decay. Empirically, DDPM-blind generates incoherent samples while FM-blind matches its conditional counterpart on CIFAR-10/SVHN/Fashion-MNIST. Relevant to this concept page as a constraint on which training-simplification levers are safe, not as a direct speedup claim.
  • [2026-05-23] Unified Latents (UL): How to train your latents (UL): jointly trains encoder + diffusion prior + diffusion decoder, with the encoder’s noise floor equal to the prior’s minimum noise level so the KL collapses to a tractable bitrate budget. Two hyperparameters (loss factor, sigmoid bias) control reconstruction-vs-modelability. Beats SD latents on ImageNet-512 at matched FLOPs; SOTA FVD 1.3 on Kinetics-600.
  • [2026-05-23] Random noise augmentation during diffusion training eliminates the need for CFG: per-channel additive + log-normal noise augmentation during training reportedly eliminates the need for CFG at inference, halving per-step compute and avoiding the “AI look.” Qualitative LoRA result on Z-Image; no quantitative metrics yet.
  • [2026-05-23] PixelGen: Improving Pixel Diffusion with Perceptual Supervision: pixel-native DiT + LPIPS + P-DINO + REPA reaches FID 5.11 in 80 epochs without CFG. Locates the source of latent-vs-pixel-space training inefficiency in the MSE loss, not the representation.
  • [2026-05-22] Improved Baselines with Representation Autoencoders (RAEv2): >10× convergence speedup over original RAE via multi-layer encoder aggregation and complementary use of REPA. SOTA gFID 1.06 on ImageNet-256 in 80 epochs.
  • [2026-05-22] L2P: Unlocking Latent Potential for Pixel Generation (L2P): trains only shallow layers of a frozen LDM on the source model’s own synthetic outputs, achieving on-par DPG-Bench and 93% GenEval on 8 GPUs.

Open questions

Section titled “Open questions”
  • Three independent late-2025/early-2026 recipes all converge on “80 epochs to strong ImageNet-256 FID” — is 80 epochs the right benchmark, or is it where the metric saturates and a deeper efficiency frontier exists past it?
  • DINOv2 features appear in all three external-supervision recipes (encoder / intermediate alignment / output-space loss). Is there a unified theory of “where DINO supervision is best applied” or is it task/scale-dependent? PixelGen + RAEv2 hint that all three locations can be stacked. SRA and Self-Flow’s counter-claim is that none of these locations is necessary if the generator self-distills.
  • External-DINO vs internal-self-distillation is now an active fork with two filed papers on the internal side. RAEv2 / RAE / P-DINO all report SOTA with external supervision at moderate scale; SRA and Self-Flow report SOTA without external supervision, with Self-Flow showing a widening gap with scale. The disagreement is most likely about scale and modality, not about ImageNet-class-conditional FID at fixed encoder size. Resolving this needs (a) RAEv2 + scaling-encoder ablation, and (b) Self-Flow at the 1B+ video scale Luma cares about.
  • What is the structural difference between SRA’s and Self-Flow’s internal-teacher recipes that produces the ImageNet gap? SRA: same-network earlier-layer-noisier → later-layer-cleaner; Self-Flow: EMA-decoupled teacher on cleaner-uniform-noise view + student on Dual-Timestep mix. Both work; Self-Flow wins by ~1.5 FID points on ImageNet. The natural ablation is “SRA + EMA teacher” vs “Self-Flow without Dual-Timestep” — neither paper runs it.
  • How do these training-efficiency recipes interact with few-step generation (SoFlow et al.) and Classifier-Free Guidance variants? Most reported numbers are at multi-step sampling with no CFG; the practical question is end-to-end inference cost, not just training cost.
  • Do the recipes transfer to video diffusion, where the analog of LPIPS / P-DINO needs temporal extension and the analog of REPA needs a video-foundation-model target (V-JEPA? frozen video DINO?)? UL’s Kinetics-600 SOTA at FVD 1.3 is the first datapoint that “latent-design efficiency” transfers to video. Self-Flow now provides a second: it improves T2V FVD by ~2 points over REPA and it actively repairs the case where external video encoders (V-JEPA 2, Depth Anything 3) underperform vanilla FM.
  • Synthetic-data-only training (L2P) raises a self-improvement question: can the same recipe iterate — pixel model trained on LDM outputs becomes the source for a next-gen pixel model trained on its own outputs — without manifold collapse?
  • UL frames latent bitrate as scale-dependent: smaller models prefer simpler latents, larger models can absorb richer ones. Does the DINO-as-encoder family (RAE) have an analogous capacity-dependent feature-depth choice? Do PixelGen’s perceptual-loss weights need rescheduling with model size?
  • Does dropping noise conditioning (per The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning) compose with the DINO/REPA/perceptual-loss recipes here, or do those recipes implicitly require noise conditioning to align features at specific noise scales? Untested across all filed papers — but a clean ablation given the parameterization compatibility constraint is now known.
  • Self-Flow needs an extra teacher forward per step. SRA, by sharing the network between teacher and student views, may avoid that overhead — at the cost of weaker alignment. What is the compute-matched head-to-head between SRA and Self-Flow?
  • Does spatial token-count reduction (Foveated Diffusion) stack with the other recipes? In principle a foveated DiT post-trained from a RAE or Self-Flow base inherits the upstream training-efficiency gains, but no filed paper has measured the compose. Also unknown: whether saliency-guided foveation can replace gaze input for non-interactive batch inference at meaningful speedups.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-01

Section titled “2025-01”

2024-12

Section titled “2024-12”

2024-02

Section titled “2024-02”