PixNerd: Pixel Neural Field Diffusion

TL;DR

PixNerd is a pixel-space diffusion transformer that replaces the VAE+latent stack with a per-patch neural field: each output token predicts the weights of a small MLP that, when queried at pixel coordinates inside its patch, produces the RGB output. This makes the entire pipeline single-stage and end-to-end, removes the VAE’s accumulated reconstruction errors, and as a side effect inherits the standard neural-field perks — training-free arbitrary-resolution generation and alias-free continuous rendering. The XL/16 variant reaches FID 2.15 on ImageNet 256² and 2.84 on ImageNet 512²; the XXL/16 text-to-image variant scores 0.73 on GenEval and 80.9 on DPG.

Key claims

VAE-free pixel diffusion at competitive quality: PixNerd-XL/16 reaches FID 2.15 on ImageNet 256² and 2.84 on 512² with no cascade and no VAE [Abstract].
The unit of prediction is changed: instead of predicting pixels (or latents) directly, each transformer token predicts the parameters of a small MLP neural field whose input is pixel coordinates within the corresponding patch [§Method].
Neural-field decoding improves high-frequency modeling relative to flat per-pixel prediction at the same patch size [§Method, §Ablation].
Row-wise normalization of the neural-field parameters (both weights and output features) stabilizes training and improves quality [§Ablation].
A 64-channel neural-field width is the sweet spot — 36 channels degrade quality, 72 channels add cost for marginal gain [§Ablation].
The method extends to text-to-image: PixNerd-XXL/16 scores 0.73 GenEval / 80.9 DPG-Bench [§Experiments, text-to-image].
The framework supports training-free arbitrary-resolution sampling, since the neural field can be queried on any pixel grid post-hoc [Fig. 3].
The codebase is built on top of FlowDCN and DDT (a flow-matching DiT with REPA-style representation alignment); REPA loss curves are reported as a training signal [README; ablation figures].

Method

A standard DiT-style transformer operates on a coarse grid of patch tokens (e.g. 16×16 patches over a 256² image, giving 256 tokens at XL scale). In a conventional pixel-space DiT, each output token would directly emit the 16×16×3 RGB values of its patch. PixNerd instead has each token emit the weights of a per-patch MLP (a “neural field” with row-wise-normalized parameters, ~64 channels wide). At decode time the MLP is queried at every pixel coordinate inside the patch and the outputs are stitched into the final image. Training is single-stage flow matching with REPA-style representation alignment (lifted from FlowDCN/DDT), end-to-end from noise → pixels. Because the MLP takes continuous coordinates, the same trained model can be queried on any grid resolution at inference, and the high-frequency detail that vanilla flat-patch prediction tends to blur is captured by the field’s frequency-encoded positional inputs rather than by the transformer’s coarse tokens.

Results

ImageNet 256²: PixNerd-XL/16 reaches FID 2.15 with compute comparable to its latent counterpart [Abstract; §Experiments].
ImageNet 512²: PixNerd-XL/16 reaches FID 2.84 with comparable compute to the latent baseline at the same resolution [Abstract; §Experiments].
Text-to-image (PixNerd-XXL/16): GenEval overall 0.73, DPG-Bench average 80.9 — competitive with latent T2I models at similar scale [§Experiments T2I].
Loss curves: PixNerd shows consistently lower REPA loss and flow-matching loss than the matched baseline DiT at the same training steps [Fig., aimodels.fyi summary].
Inference latency (per GitHub README corrigendum): PixNerd’s single-step inference time is close to the matched-architecture baseline; an earlier reported 10× speedup was a transcription error.
Caveat: The user’s TL;DR notes results still fall short of the leading LDMs at scale; the trajectory is “getting closer” rather than overtaking.

Why it’s interesting

PixNerd opens a third axis on the Pixel-space diffusion design space that’s distinct from the loss-side (PixelGen / CAFM / FD-loss), latent-redesign (UL), and parameterization (AsymFlow) recipes already on the wiki: change the prediction primitive itself. Where PixelGen: Improving Pixel Diffusion with Perceptual Supervision keeps the per-patch prediction flat and fixes the loss, and Asymmetric Flow Models keeps the architecture and changes what’s regressed, PixNerd swaps the per-token output head for a continuous neural field — which gives “for free” two properties the other recipes have to engineer separately: arbitrary-resolution sampling and alias-free decoding (cf. Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space, which had to add anti-aliasing to a latent pipeline). The Slack note flags a forward-looking thread worth tracking: timestep ↔ field-frequency-band coupling — early denoising steps could be restricted to low frequencies of the field’s positional encoding, matching the spectral-progressive intuition already explored in Spectral Progressive Diffusion for Efficient Image and Video Generation but realized through the field’s bandwidth rather than a frequency-aware noise schedule.