DiP: Taming Diffusion Models in Pixel Space

TL;DR

DiP is a pixel-space diffusion framework from the NJU + Tencent PCALab group that decouples generation into a coarse global stage and a fine local stage: a standard DiT backbone operates on large 16×16 patches for efficient global structure, then a co-trained lightweight Patch Detailer Head restores fine-grained pixel-level detail. The split reduces the global-stage sequence length by 16× relative to small-patch pixel diffusion, achieving up to 10× faster inference than PixelFlow-XL while adding only 0.3% extra parameters via the detailer head. Reports FID 1.79 on ImageNet 256×256, framed as beating leading latent baselines like DiT-XL and SiT-XL without needing a VAE.

Key claims

A two-stage decoupling — global structure on large 16×16 patches via a DiT backbone plus a lightweight Patch Detailer Head for fine-grained local detail — resolves the pixel-space efficiency/quality trade-off without a VAE [§1, Fig. 1(c)].
The Patch Detailer Head adds only 0.3% to total parameter count while significantly improving fine-detail fidelity over the bare large-patch DiT [Abstract, §1].
Inference speed is up to 10× faster than the prior pixel-space SOTA PixelFlow-XL, and overall compute cost is comparable to mainstream LDMs despite operating end-to-end in pixel space [§1, Fig. 2].
Reaches FID 1.79 on ImageNet 256×256, outperforming DiT-XL and SiT-XL latent baselines [Abstract].
The framing positions large patch size — not VAE compression — as the load-bearing efficiency lever: a 16×16 patch DiT cuts global-stage sequence length by 16× over a 4×4-patch pixel DiT, recovering LDM-scale latency without giving up end-to-end pixel-space training [§1, Fig. 1(c)].

Method

DiP keeps the standard DiT architecture but operates directly on RGB patches rather than on VAE latents. The decisive design choice is to push the patch size large (16×16 in the headline configuration) so the DiT backbone sees a sequence length comparable to a latent-space DiT operating on a 32×-compressed feature map. A 16×16-patch DiT alone, however, blurs fine detail because each token must represent 768 RGB values directly. DiP fixes this by attaching a co-trained Patch Detailer Head — a lightweight module that takes the DiT’s per-patch contextual features and refines the local pixel content within each patch, restoring high-frequency detail that the large-patch DiT itself cannot.

The two stages are co-trained end-to-end rather than chained as a separate cascade, and the detailer head is small enough (0.3% of parameters) that the parameter and compute budget is dominated by the DiT backbone. The result is a single end-to-end pixel-space diffusion model whose backbone runs at LDM-comparable cost.

Results

ImageNet 256×256: FID 1.79, reported as the SOTA pixel-space figure on this benchmark and as outperforming latent-space DiT-XL and SiT-XL [Abstract, §1].
Up to 10× inference speedup over PixelFlow-XL — the previous pixel-space SOTA [§1, Fig. 2 caption].
Parameter overhead from the detailer head: only 0.3% relative to the DiT backbone [Abstract].
The Fig. 2 efficiency plot positions DiP on the Pareto frontier of FID-vs-compute relative to both pixel-space (PixelFlow-XL) and latent-space (DiT-XL, SiT-XL) baselines [Fig. 2].

Why it’s interesting

Kyle pointed at this as “more work related to those two He papers from last week” — referring to Back to Basics: Let Denoising Generative Models Denoise (Li & He, Nov 17) and the surrounding pixel-space-diffusion thread Kaiming He’s group has been pushing. DiP is from a different group (NJU + Tencent PCALab, not MIT/Meta) but lands on the same thesis from a different angle: the pixel-vs-latent gap was never intrinsic to pixel space — JiT (He) argues the fix is the prediction target ( $x_0$ vs $\epsilon$ ), PixelGen: Improving Pixel Diffusion with Perceptual Supervision argues it’s the loss function (LPIPS + P-DINO), L2P: Unlocking Latent Potential for Pixel Generation (L2P, the same Zhennan Chen first author) argues it’s the training cost, and DiP argues it’s the sequence length — push the patch size up to 16×16 and the global-stage cost becomes LDM-comparable, then patch the resulting detail loss with a 0.3%-parameter local head. The recipes are not mutually exclusive: DiP’s large-patch DiT plus detailer-head split is in principle compatible with JiT’s $x_0$ -prediction, PixelGen’s perceptual losses, and AsymFlow’s rank-asymmetric velocity (Asymmetric Flow Models) — none of which fight DiP’s sequence-length lever.

DiP is also the same Zhennan Chen as L2P: Unlocking Latent Potential for Pixel Generation (L2P), so this is a follow-up from that group: L2P showed that a pretrained LDM can be retrained into pixel space cheaply by freezing intermediate layers; DiP shows that you can train from scratch in pixel space at LDM cost if you accept the patch-size/detail decoupling. The contrast with PixelGen: Improving Pixel Diffusion with Perceptual Supervision is worth noting: PixelGen and DiP both reach competitive ImageNet-256 FID via a pixel-native DiT, but PixelGen pays the cost in extra loss-function complexity (LPIPS + P-DINO + REPA) while DiP pays it in extra architecture (the detailer head). At reported numbers — PixelGen FID 5.11 (no CFG, 80 epochs) vs DiP 1.79 — the two are not strictly comparable since DiP’s headline is presumably with CFG and likely longer training, but the architectural recipe is the cleaner of the two if it holds at scale.

The 16×16-patch claim is also worth flagging for the wiki’s Diffusion training efficiency thread, which already tracks token-count reduction as an efficiency axis via Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation (foveation reduces tokens spatially) — DiP reduces tokens by patch-size scaling, which is the simpler and more universal lever.