Skip to content
Bud's Wiki

Pixel-space diffusion

TL;DR

Section titled “TL;DR”

Pixel-space diffusion trains the generative model directly on RGB pixels, bypassing the VAE encoder/decoder pair that latent diffusion uses to compress images before generation. After several years of being the perceived-loser to latent diffusion on quality-per-compute, a wave of 2025-2026 work argues the gap was never intrinsic to pixel space — it came from (a) optimizing MSE on a high-dimensional manifold full of perceptually irrelevant signal (PixelGen) and (b) the prohibitive cost of training pixel-native from scratch (L2P). Fixes include perceptual losses replacing or augmenting MSE, large-patch tokenization replacing the VAE compressor, and transfer-learning recipes that move a pretrained LDM into pixel space cheaply. A separate “decoder-only” strand (One-DVA, PiD) keeps a latent space for the LDM but does the latent→pixel step with a pixel-space DiT instead of a deterministic decoder.

Key claims

Section titled “Key claims”
  • The classical pixel-vs-latent gap is attributable to the loss function and training cost, not to pixel space being intrinsically harder to model — replacing MSE with perceptual losses on a pixel-native DiT (JiT backbone) reaches FID 5.11 on ImageNet-256 without CFG in 80 epochs, beating strong LDM baselines (PixelGen: Improving Pixel Diffusion with Perceptual Supervision Abstract, Fig. 5).
    • Sharpened by CAFM: post-training JiT-H/16 with a learned discriminator (adversarial criterion in continuous-time flow) for just 10 epochs drops guidance-free FID 7.17 → 3.57 and best guided FID 1.86 → 1.80 (Continuous Adversarial Flow Models Table 4). The PixelGen thesis (“loss, not encoder, was the bottleneck”) holds; CAFM adds that a learned manifold-aware loss can beat fixed perceptual losses (LPIPS / P-DINO) at fewer training epochs from a pretrained checkpoint.
    • Sharpened again by FD-loss: post-training JiT-H/16 with closed-form Fréchet Distance in Inception space (queue/EMA decoupled from batch, no discriminator) drops 1-NFE FID to 0.75 on JiT-H and 1.89 on pMF-H ImageNet-256, repurposing the multi-step JiT-L (291.59 → 0.77 at 1 NFE) without any teacher signal (Representation Fréchet Loss for Visual Generation Tables 2-4). Closed-form distributional moment-matching is competitive with the adversarial criterion at the same backbone.
    • Conflict / qualifier: UL (Unified Latents (UL): How to train your latents Table 1, §5.3) argues the gap closes the other way: a properly-designed latent (encoder noise tied to prior noise floor, bitrate-budgeted by a loss-factor hyperparameter) beats pixel-space diffusion at matched architecture (gFID@30k 4.1 vs 5.0 on internal T2I) and Pareto-dominates Stable Diffusion latents. Both sides agree the vanilla LDM-with-SD-latents recipe was the bottleneck; they disagree on whether the right fix lives in the diffusion model (PixelGen / CAFM / FD-loss / AsymFlow) or in the latent (UL).
    • Re-located by AsymFlow: the loss-side fork (PixelGen / CAFM / FD-loss) and the latent-redesign fork (UL) both leave the prediction target alone — they predict u=ϵ−x0u = \epsilon - x_0 or x0x_0 and change what’s wrapped around it. AsymFlow argues the prediction target itself is the bottleneck: full-dimensional noise prediction wastes capacity. Replacing it with rank-asymmetric velocity uA=Pϵ−x0u_A = P\epsilon - x_0 on the same JiT-H/16 backbone gives FID 1.57 (with REPA) / 1.76 (without) on ImageNet-256, beating PixelGen-XL/16 1.83 and the same-architecture JiT-H/16 x₀-pred at 1.86 (Asymmetric Flow Models Table on project page). The recipes are mutually compositional in principle: AsymFlow changes the parameterization, the loss-side methods change the criterion on top of it.
  • The VAE decoder is a memory bottleneck as much as a quality bottleneck: removing it unlocks native 4K generation on 8 GPUs that would be prohibitive in a latent pipeline (L2P: Unlocking Latent Potential for Pixel Generation abstract).
  • Pixel-native training does not have to start from scratch: an existing LDM’s intermediate layers can be frozen and only the shallow layers retrained to map latent priors to pixels, with the source LDM’s own synthetic outputs as the training corpus (L2P: Unlocking Latent Potential for Pixel Generation abstract).
    • AsymFlow gives an analytical alternative: by aligning the patch-wise low-rank pixel subspace with a pretrained latent space, an entire latent flow model’s denoising trajectory becomes a valid pixel-space initialization without retraining shallow layers — finetuning AsymFLUX.2 klein from FLUX.2 klein 9B beats the latent base on HPSv3 (10.66 vs 9.50), DPG-Bench (86.8 vs 85.2), and GenEval (0.82 vs 0.80), and the authors describe this as the “first-ever route for finetuning pretrained latent flow models into pixel-space models” (Asymmetric Flow Models Results table on project page).
  • DINOv2 features supervise pixel-space training effectively when applied as an output-space perceptual loss (P-DINO) on the predicted clean image, complementing local-pattern losses like LPIPS (PixelGen: Improving Pixel Diffusion with Perceptual Supervision §3, ablation: 23.67 → 10.00 with LPIPS → 7.46 with +P-DINO).
  • x-prediction + flow-matching reparametrization is the training paradigm of choice for pixel-native diffusion: predict x^0\hat{x}_0 to make perceptual losses well-defined, then convert to velocity to retain flow-matching’s sampling properties (PixelGen: Improving Pixel Diffusion with Perceptual Supervision §3 Method).
    • Refined by AsymFlow: x0x_0-prediction is the rank-0 special case of a rank-asymmetric velocity family, and uu-prediction is the full-rank special case; a small but non-zero noise-prediction rank (e.g. rank 8 of 768) Pareto-dominates both, keeping the numerical stability of uu-prediction in a meaningful subspace while reducing the capacity wasted on high-dimensional noise (Asymmetric Flow Models §Method).
  • A learned manifold-aware criterion outperforms fixed perceptual losses for pixel-space flow models. CAFM’s discriminator-as-criterion (continuous-time adversarial training in the JVP space of the flow) beats JiT-H/16 + MSE (FID 7.17 → 3.57 no-CFG) in 10 epochs of post-training, indicating the data-adaptive criterion captures aspects of the pixel manifold that LPIPS + P-DINO miss (Continuous Adversarial Flow Models §1, §3.1, Table 4).
  • A closed-form distributional criterion can match or beat the learned-discriminator route at a comparable scale, without GAN-style training instabilities. FD-loss in Inception space post-trains pMF-H and JiT-H on the same ImageNet-256 setup to 1-NFE FID 0.72 / 0.75 respectively (Representation Fréchet Loss for Visual Generation Tables 3-4). The criterion is just Tr-cov + ‖Δμ‖² over a 50k feature queue / EMA — no per-sample regression target, no adversarial signal.
  • FID has saturated as a quality signal at the current performance tier. Real ImageNet-256 validation images score FID 1.68 against train-set statistics, yet multiple recent generators report ≤ 1.5; FD-loss post-trained models optimizing modern representations (DINOv2 / MAE / SigLIP) produce visibly better samples while scoring worse on Inception-FID than Inception-optimized ones (Representation Fréchet Loss for Visual Generation §3.4, Fig. 4, Table 1c). Authors propose FDr^K — normalized FD ratio averaged over K representations — as a more diverse automatic metric.
  • Pixel-space diffusion also works as a decoder inside an otherwise-latent autoencoder pipeline: a pixel DiT decoder conditioned on a hybrid 1D+2D latent matches 3D-CNN VAEs on reconstruction (PSNR 36.48 vs HunyuanVideo’s 35.54 at identical compression) and recovers detail at aggressive 1D-only compression where a deterministic decoder cannot (Adaptive 1D Video Diffusion Autoencoder Table 1).
    • Sharpened by PiD at megapixel image scale: the same “pixel-diffusion decoder, latent generator unchanged” pattern, with two new ideas — (a) a sigma-aware adapter that injects noise-corrupted latents into the pixel-diffusion backbone, so the decoder can run on partially-denoised upstream latents and lets the LDM exit its own diffusion early; (b) DMD2 4-step distillation, getting decode+4× upsample of a 512² latent to a 2048² image down to ~210 ms on GB200 / under 1 s on RTX 5090 (PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion Abstract, §3). Crucially, the same decoder works on classic VAE latents and on semantic (SigLIP / DINOv2) latents used in RAE-style pipelines.
  • Train/inference distribution drift in the latents fed to a pixel-space diffusion decoder produces patch-like artifacts; fine-tuning the decoder on LDM-sampled latents (not encoded ground-truth) is the load-bearing fix and is the difference between gFVD 210.9 and a noticeable degradation on class-conditional video generation (Adaptive 1D Video Diffusion Autoencoder §3.5, §4.4).
    • PiD addresses the same train/inference gap structurally, not via fine-tuning data: the sigma-aware adapter takes a noise-corrupted latent as input at every diffusion step, so it never needs to assume the latent is clean (PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion §3, Abstract). Whether this is strictly stronger than One-DVA’s LDM-sampled-latent fine-tune is not directly compared.

Recent contributions

Section titled “Recent contributions”
  • [2026-06-24] Is Noise Conditioning Necessary for Denoising Generative Models?: Sun et al. show on CIFAR-10 / ImageNet-32² / FFHQ-64² (all pixel-space) that timestep conditioning can be removed from EDM/FM/iDDPM with at most graceful degradation; their proposed uEDM (constant c_in, no t-input) reaches FID 2.23 vs 1.99 t-conditional. Result is most clearly demonstrated in pixel space where the posterior-concentration argument has the cleanest form.
  • [2026-06-18] MiniT2I: Pixel-Space Text-to-Image with MM-JiT at 258M Parameters: MiniT2I (Kaiming He’s MIT group, undergrad team) is the first T2I extension of the JiT line — MM-JiT architecture, 258M params, no VAE/tokenizer/alignment, reportedly trained at ~one-ImageNet-experiment cost. Smallest filed T2I instance of the pixel-space recipe; complements PixelDiT-T2I (1.3B params, NVIDIA) and AsymFLUX.2 klein (9B fine-tune) at the opposite scale extreme.
  • [2026-05-25] PixelDiT: Pixel Diffusion Transformers for Image Generation: PixelDiT (NVIDIA — Yu/Xiong/Nie/Sheng/Liu/Luo, same group as PiD): the architecture-side recipe parallel to the loss-side (PixelGen/CAFM/FD-loss), latent-redesign (UL), and parameterization-side (AsymFlow) recipes already filed. Dual-level transformer — patch DiT for semantics + pixel DiT for detail — with pixel-token compaction (compress P² pixel tokens into one before global attention) and pixel-wise AdaLN (independent modulation params per pixel within a patch). Vanilla DiT/16 baseline maxes at gFID 9.84 (80 ep); full PixelDiT-XL hits 2.36 at 80 ep and 1.61 at 320 ep on ImageNet 256², beating PixelFlow-XL 1.98, PixNerd-XL 1.93, EPG 2.04, and JiT-G (2B) 1.82 at smaller scale. Notably the T2I variant trains natively at 1024² in pixel space (1.3B params, Gemma-2 text encoder, MM-DiT patch blocks) and gets GenEval 0.74 / DPG 83.5 vs FLUX-dev’s 0.67/84.0 at 12B params, at 0.33 samples/s on a single A100 (vs FLUX-dev 0.04) — putting serious pressure on the conventional wisdom that pixel-space can’t scale past 512² without cascading. Released under NVIDIA non-commercial research license.
  • [2026-05-25] PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion (PiD, NVIDIA — Lu/Wu/Wu/Wang/Ling/Fidler/Ren): pixel-diffusion decoder for high-resolution latent decoding. Slots in as a drop-in replacement for the deterministic VAE decoder + cascaded super-resolution stages of a standard LDM pipeline, and decodes both classic VAE latents and semantic SigLIP/DINOv2 latents (RAE-style). Two novel ideas relative to One-DVA: (a) a sigma-aware adapter injects noise-corrupted latents into the pixel-diffusion backbone, which lets PiD decode partially-denoised latents and lets the upstream LDM terminate early; (b) DMD2 4-step distillation reduces inference to ~210 ms on GB200 and under 1 s at 13 GB on a consumer RTX 5090 for a 512²→2048² decode, ~6× faster than cascaded diffusion-VSR pipelines and 5.9× faster than SeedVR2 at the same task with better visual fidelity per project-page side-by-sides. First concrete instance of the “decoder-only pixel diffusion” pattern at megapixel image scale (One-DVA was video autoencoder). Apache 2.0, weights on HuggingFace.
  • [2026-05-24] Asymmetric Flow Models (AsymFlow, Stanford — Chen, Ackermann, Kim, Wetzstein, Guibas): a parameterization-side recipe, distinct from the loss-side (PixelGen / CAFM / FD-loss) and latent-redesign (UL) recipes already on the page. Predicts uA=Pϵ−x0u_A = P\epsilon - x_0 where PP is a fixed patch-wise rank-8 projector; the full velocity is recovered analytically, so architecture/optimizer/sampler are unchanged. Same JiT-H/16 (953M, 363 GFLOPs) goes from FID 1.86 (x₀-pred) to 1.57 (rank-8 AsymFlow + REPA), beating PixelDiT-XL/16 (1.61), PixelGen-XL/16 (1.83), and JiT-G/16 at 2B params (1.82). The same low-rank projector enables analytical latent→pixel finetuning: aligning PP‘s image with a pretrained latent’s per-patch lift means a latent flow model’s trajectory is, at initialization, a valid asymmetric-velocity prediction in pixel space. AsymFLUX.2 klein (finetuned from FLUX.2 klein 9B) beats the latent base on HPSv3 (10.66 vs 9.50), DPG-Bench (86.8 vs 85.2), and GenEval (0.82 vs 0.80). Authors note the recipe assumes a good patch-level linear lift and may fail when the latent doesn’t preserve pixel structure.
  • [2026-05-24] Representation Fréchet Loss for Visual Generation (FD-loss, USC + CMU + GDM): post-training criterion family parallel to PixelGen’s perceptual losses and CAFM’s adversarial discriminator. Make Fréchet Distance a training loss by decoupling the 50k-sample population (queue or EMA of feature moments) from the gradient batch (1024). On JiT-H/16 the same pixel-space backbone CAFM uses, FD-loss + Inception reaches FID 0.75 at 1 NFE; FD-SIM (SigLIP+Inception+MAE) reaches FDr^K 2.65. Also repurposes multi-step JiT-L (1-NFE FID 291.59 → 0.77 with FD-Inception) into a one-step generator without any teacher signal. Independently introduces FDr^K, which exposes that single-Inception FID has saturated — modern-representation-trained models produce visibly better samples while scoring worse on Inception-FID.
  • [2026-05-24] Continuous Adversarial Flow Models (CAFM, Bytedance/ByteSeed): adversarial post-training of pixel-space JiT-H/16 with a continuous-time learned discriminator (JVP-based) drops guidance-free FID 7.17 → 3.57 and best guided FID 1.86 → 1.80 in 10 epochs, beating PixelFlow-XL/4 (1.98) and matching JiT-G/16 (2B params, 1.82) at smaller scale. Strengthens PixelGen’s “loss, not encoder, was the bottleneck” thesis by replacing fixed perceptual losses with a learned manifold-aware criterion.
  • [2026-05-23] Unified Latents (UL): How to train your latents (UL): jointly-trained encoder + diffusion prior + diffusion decoder beats both SD latents and a pixel-space UNet baseline on ImageNet-512 at matched FLOPs. Adds a contrary datapoint to the pixel-space-wins narrative — argues the right fix is a bitrate-budgeted latent, not abandoning the latent.
  • [2026-05-23] Adaptive 1D Video Diffusion Autoencoder: One-DVA uses a pixel-space DiT as the decoder of a video autoencoder, while the LDM stays in latent space. Pixel-space diffusion-decoder buys back reconstruction quality at aggressive compression, and decoder fine-tuning on LDM-sampled latents closes the train/inference distribution gap.
  • [2026-05-23] PixelGen: Improving Pixel Diffusion with Perceptual Supervision: pixel-native DiT (JiT) + LPIPS + Perceptual-DINO losses reaches FID 5.11 on ImageNet-256 in 80 epochs without CFG, beating strong latent baselines. Argues the loss function — not the encoder — was the bottleneck.
  • [2026-05-22] L2P: Unlocking Latent Potential for Pixel Generation: L2P transfers a pretrained LDM into pixel space by freezing intermediate layers and retraining shallow ones on the source LDM’s synthetic outputs, eliminating the VAE memory bottleneck and enabling 4K generation on 8 GPUs.

Open questions

Section titled “Open questions”
  • Does the perceptual-loss recipe (PixelGen) scale to video, where temporal LPIPS / P-DINO need to be defined carefully and frame-level DINO features may not capture temporal coherence?
  • Is the right pixel-tokenization scheme large patches (L2P), JiT-style patches, or learned patch tokenizers? What’s the patch-size vs. attention-cost tradeoff at high resolution?
  • Are PixelGen’s perceptual losses and L2P’s transfer-learning recipe composable — i.e., can one finetune an LDM into pixel space with perceptual supervision and skip both the VAE and the from-scratch training cost? AsymFlow is now an analytical answer in spirit to this question — its latent→pixel finetune is parameterization-driven rather than recipe-driven — but the composition of AsymFlow’s initialization with PixelGen’s perceptual losses is untested.
  • How do pixel-space methods compare to RAE-style alternatives (frozen DINO as encoder, still in latent space) at parity compute? Both routes share the “DINO features as the right inductive bias” story but apply it in different places. PiD partially answers this for the decoder side — the same pixel-diffusion decoder works for both VAE and SigLIP/DINOv2 latents — but does not compare end-to-end generation quality between the two encoder choices with the decoder held fixed.
  • Is decoder-only pixel diffusion (One-DVA, PiD) strictly dominated by end-to-end pixel diffusion (PixelGen / L2P), or does the hybrid pay off when downstream LDM compute is the binding constraint? The framings agree that the deterministic decoder is the weak link but disagree on whether the LDM should still operate on compressed latents. PiD’s sigma-aware adapter sharpens this: the hybrid plus partial-latent coupling lets the LDM exit early, which makes the decoder-only design strictly more compute-flexible than pure end-to-end pixel diffusion. Whether the resulting total compute beats end-to-end pixel diffusion at matched quality is unmeasured.
  • UL vs PixelGen vs AsymFlow now reframes the central question three ways: is the fix in the encoder + prior coupling (UL), in the loss on the pixel manifold (PixelGen / CAFM / FD-loss), or in the velocity parameterization itself (AsymFlow)? All three reach roughly comparable headline numbers on ImageNet but none have been compared head-to-head at matched compute and matched backbone.
  • Learned (CAFM) vs fixed (PixelGen) vs closed-form distributional (FD-loss) vs parameterization (AsymFlow) manifold-aware levers on the same JiT backbone. AsymFlow’s 1.57 FID beats PixelGen’s 5.11 and CAFM’s 3.57 (no-CFG numbers; CAFM hits 1.80 with guidance). The recipes are in principle compositional — AsymFlow changes what the network predicts, the others change the criterion on top of that — but no filed paper has run the stack.
  • Does AsymFlow’s latent→pixel finetune compose with PixelGen-style perceptual losses, or with FD-loss / CAFM post-training? AsymFLUX.2 klein already uses a “variance-reduced flow-matching objective with perceptual correction” but the paper does not ablate which perceptual loss matters and at what weight.
  • What latent spaces don’t admit a good patch-level linear lift? AsymFlow explicitly limits its latent→pixel recipe to latents that preserve pixel structure; this is a load-bearing assumption for FLUX.2 klein but would presumably fail for heavily-compressed video VAEs like 3D-CNN encoders. The boundary is unmeasured.
  • Is FDr^K the right replacement metric, or is it just a less-saturated FID? It still inherits the Gaussian moment-matching assumption, and its value depends on the chosen K representations. The authors explicitly disclaim it as a north star (Representation Fréchet Loss for Visual Generation §3.4). What would a more principled multi-representation metric look like — perhaps a precision-recall variant in each feature space?
  • Does PiD’s partial-latent / sigma-aware coupling generalize to video? The image case caps at 4×/8× upscale; the video case would have to handle temporal consistency across the decode+upsample step. One-DVA is the closest video datapoint and does not couple to a partially-denoised upstream LDM.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-01

Section titled “2025-01”