PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion

TL;DR

PiD replaces the deterministic VAE decoder of a latent-diffusion pipeline with a pixel-space diffusion model that simultaneously decodes and upsamples. The latent is injected into a pixel diffusion backbone via a lightweight sigma-aware adapter so partially denoised latents can be decoded, letting the upstream LDM stop early. A 4-step DMD2 distillation gets inference down to 4 NFE; on a consumer RTX 5090 the 512²→2048² decode runs in under 1 s (~210 ms on a GB200) — about 6× the throughput of cascaded diffusion-based super-resolution at better visual fidelity. PiD also accepts semantic (SigLIP, DINOv2) latents, so it slots into both classic VAE pipelines and RAE-style ones.

Key claims

The latent-to-pixel decoder in standard LDMs is reconstruction-optimized (inverts the encoder) rather than synthesis-optimized, and becomes a megapixel-scale bottleneck that motivates replacing it with a generative decoder [§1, Abstract].
Reformulating decoding as conditional pixel diffusion unifies the latent→pixel decode and the subsequent super-resolution step into a single generative module that synthesizes 4× and 8× upscaled images directly in pixel space [§3, Abstract].
A sigma-aware adapter injects noise-corrupted latents into the pixel-diffusion backbone, allowing PiD to decode partially denoised latents — which lets the upstream LDM terminate its own diffusion process early [§3, Abstract].
DMD2 distillation reduces inference to 4 steps; on a consumer RTX 5090, 512×512 latents decode to 2048×2048 pixels in under 1 second at 13 GB peak memory, and as fast as 210 ms on a GB200 — about 6× faster than cascaded diffusion-based super-resolution pipelines at better visual fidelity [Abstract, Quantitative Results].
The recipe applies to both conventional VAE latents and semantic (SigLIP / DINOv2) latents used by RAE-based models, so the same decoder slots into both latent-design families [§3, Abstract].
Project-page side-by-side judges prefer PiD over each tested baseline on a per-category basis (project page, Baseline Comparison section), and PiD is up to 5.9× faster than SeedVR2 at the same decode+upsample task (211.2 ms vs 1237.5 ms, project page Quantitative Results).

Method

PiD is a pixel-space diffusion decoder. Instead of mapping a clean latent $z$ to pixels through a deterministic VAE decoder followed by an external super-resolution stage, PiD treats $z$ as a conditioning signal for a diffusion model that operates directly in high-resolution pixel space. A lightweight sigma-aware adapter takes a noise-corrupted version of $z$ (with corruption level $\sigma$ aligned with the pixel-diffusion timestep) and injects it as conditioning into the pixel diffusion backbone. Because the adapter is sigma-aware, PiD can be conditioned on partially-denoised latents from the upstream LDM — when the LDM is still on noisy latents at intermediate steps, PiD can already start producing usable pixels, so the LDM diffusion process can be terminated early and the two diffusion processes are coupled rather than strictly sequential.

To make the decode practical at megapixel scale, the trained multi-step PiD is distilled with DMD2 (Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals is the closest filed relative of the DMD2 procedure; see also Diffusion Distillation) down to a 4-step student. The pixel diffusion backbone is generic enough to condition on either a conventional VAE latent or a semantic latent from a self-supervised encoder (SigLIP, DINOv2) — the same backbone serves both pipeline families with a swap of the adapter input. The pipeline does decode and upsample as one operation: a 512² latent goes straight to a 2048² (4×) or 4096² (8×) image without a separate VSR network downstream.

Results

512² latent → 2048² image in under 1 second at 13 GB peak memory on a consumer RTX 5090 [Abstract].
Same decode in ~210 ms on a GB200, ~6× faster than cascaded diffusion-based super-resolution pipelines while reporting better visual fidelity [Abstract].
Up to 5.9× faster than SeedVR2 at the matched decode+upsample task (211.2 ms vs 1237.5 ms) per the project page Quantitative Results section.
Decoding generalizes across VAE latents and semantic (SigLIP / DINOv2) latents [§3, Abstract], i.e., works on both classic LDM and RAE-style pipelines without retraining the upstream generator.
Inference reduced from multi-step to 4 NFE via DMD2 distillation [§3].

Why it’s interesting

PiD is the clearest production-oriented instance yet of a pattern the wiki has been tracking: a pixel-space diffusion decoder dropped into an otherwise-latent pipeline. This is exactly the design point that Adaptive 1D Video Diffusion Autoencoder (One-DVA) staked out for video autoencoders — the LDM stays in latent space, but the decoder is a pixel-space DiT rather than a deterministic VAE decoder — and PiD generalizes it to images at megapixel scale and adds the partial-latent coupling that lets the upstream LDM stop early. The Pixel-space diffusion TL;DR already flagged this as a “decoder-only” fork distinct from the end-to-end pixel-native programs of PixelGen/L2P/JiT.

The semantic-latent path links it directly to Representation Autoencoders: PiD lets an RAE-style pipeline (frozen SigLIP/DINOv2 encoder, e.g. Diffusion Transformers with Representation Autoencoders or Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders) keep its high-information encoder while skipping the bespoke RAE-decoder design. And by collapsing the latent-decode + cascaded-VSR pipeline into one diffusion model, it competes head-on with the Video Super-Resolution / Restoration via Diffusion Priors cluster’s one-step distilled VSR models (the 5.9× SeedVR2 number is a direct claim against that line, e.g. SparkVSR, FlashVSR, DUO-VSR). The DMD2 4-step distillation puts it on the same Diffusion Distillation trajectory as the rest of the 2025–26 fast-diffusion frontier and on the same Diffusion serving optimization curve as TurboDiffusion / FastVideo, but applied to the often-overlooked decode step rather than the main denoise loop.