Latent Denoising Makes Good Visual Tokenizers

TL;DR

l-DeTok (Latent Denoising Tokenizer) trains a visual tokenizer with the same objective downstream generators use: reconstruct clean images from heavily corrupted latents. The corruption recipe is interpolative Gaussian noise on the encoder output plus optional random patch masking, with the decoder asked to reconstruct the original pixels. Across six generative families (MAR, RandomAR, RasterAR, SiT, DiT, LightningDiT) on ImageNet 256×256, swapping SD-VAE/MAR-VAE for l-DeTok yields a 37% average FID improvement (51% on AR models, 23% on non-AR), and MAR-L + l-DeTok sets a new SOTA at FID 1.32 with 479M parameters — beating MAR-H (943M) and VAR-d30 (2.0B).

Key claims

A tokenizer trained to reconstruct clean pixels from latents corrupted by interpolative Gaussian noise (mix ratio τ) and random masking (ratio m) is a plug-in replacement for VAE-style tokenizers and improves downstream FID on every generator tested [Method; Results].
Average FID improvement over SD-VAE across 6 generators on ImageNet-256 at 100 epochs: +51.3% on AR models, +22.8% on non-AR, +37.0% overall — drop-in tokenizer swap, no architectural changes [Generalizability table].
MAR-L + l-DeTok (decoder fine-tuned) reaches FID 1.32 / IS 304.1 on ImageNet-256 with CFG at 479M params, beating MAR-H + MAR-VAE (FID 1.55, 943M) and VAR-d30 (FID 1.92, 2.0B) [System-level table].
MAR-B + l-DeTok at 208M params reaches FID 1.55 with CFG, surpassing LlamaGen-3B (FID 2.18 at 3.1B) by ~15× fewer params [System-level table; Parameter efficiency].
On MS-COCO text-to-image, l-DeTok improves MAR-B FID 12.49 → 4.97 (+60%) and CLIP 22.07 → 24.82 over MAR-VAE; SiT-B FID 6.63 → 4.31 — both diversity and alignment improve without external semantic distillation [T2I table].
l-DeTok benefits from semantics distillation when stacked with it: adding distillation pushes l-DeTok further to FID 2.52 (MAR-B) / 5.57 (RandomAR-B), beating both VA-VAE and MAETok which use distillation [Generalizability table].
rFID is not the right tokenizer metric: l-DeTok has worse rFID (0.68) than MAR-VAE (0.53) and SD-VAE (0.61) but better downstream gFID across all six generators [Generalizability table].
Both AR and non-AR generative training are framed as a single “reconstruction-from-deconstruction” objective (AR fills masked context, non-AR removes injected noise), motivating the tokenizer-side corruption choice as alignment with that shared objective [Method].

Method

The training corruption has two knobs. Interpolative noise: given encoder latents $z$ and Gaussian noise $\epsilon$ , the tokenizer is trained on $z' = (1-\tau)\,z + \tau\,\epsilon$ for a randomly sampled mix strength $\tau$ scaled by a gamma hyperparameter (paper uses γ=3.0). Random masking: a random fraction $m$ of latent patches (paper uses m=0.7) is replaced with a learned [MASK] token. The decoder receives this corrupted-and-masked latent and is trained to reconstruct the original pixels with a standard reconstruction loss (plus GAN discriminator after 100 epochs). No external vision-encoder supervision is used in the base recipe; distillation is an optional add-on.

After tokenizer pretraining the decoder is fine-tuned for 100 more epochs with no corruption (γ=0, m=0) and a perceptual weight bump — this is the “decoder_ft” variant in the results tables. Downstream generators (MAR, SiT, DiT, etc.) are then trained on the frozen l-DeTok latents using a special “aligned schedule” that pairs the tokenizer’s training-time noise distribution with the diffusion sampler’s.

The full pipeline is implemented in <10 lines of corruption logic on top of a standard VAE training loop; the GitHub release includes pretrained DeTok-BB (172M) tokenizers and MAR-Base/Large generator checkpoints.

Results

Headline numbers on ImageNet 256×256 with CFG: MAR-B + l-DeTok → FID 1.61 / IS 289.7 (208M); MAR-B + l-DeTok decoder_ft → 1.55 / 291.0; MAR-L + l-DeTok → 1.43 / 303.5 (479M); MAR-L + l-DeTok decoder_ft → 1.32 / 304.1 [System table].

Without CFG, MAR-L + l-DeTok reaches FID 1.84 / IS 238.4 — competitive with REPA+SiT-XL+CFG (1.42) at smaller scale and beating SiT-XL+MAETok+CFG (1.67) outright [System table].

ImageNet 512×512: MAR-B + l-DeTok trained from scratch for 400 epochs at 208M params reaches FID 1.83 / IS 279.6, edging MAR-L + MAR-VAE at 479M (1.73 / 279.9). MAR-L + l-DeTok fine-tuned 200 epochs at 512×512 hits FID 1.61 / IS 315.7 [512 table].

MS-COCO 256×256 text-to-image: MAR-B + l-DeTok FID 4.97 / CLIP 24.82 vs MAR-B + MAR-VAE FID 12.49 / CLIP 22.07 (60% FID reduction, 12.5% CLIP increase). SiT-B + l-DeTok FID 4.31 / CLIP 24.61 vs SiT-B + MAR-VAE FID 5.74 / CLIP 23.50 [T2I table].

Plug-and-play generalization on ImageNet-256 at 100 epochs, ΔFID from SD-VAE baseline: MAR-B 4.64 → 2.43 (+47.6%), RandomAR-B 13.11 → 5.22 (+60.2%), RasterAR-B 8.26 → 4.46 (+46.0%), SiT-B 7.66 → 5.13 (+33.0%), DiT-B 8.33 → 6.58 (+21.0%), LightningDiT-B 4.24 → 3.63 (+14.4%). The gain is consistently larger on AR models than on non-AR — consistent with the paper’s framing that AR’s mask-filling objective is closer to the tokenizer’s training signal [Generalizability table].

Why it’s interesting

DeTok stakes out a distinct position in the tokenizer-redesign cluster the wiki has been tracking. The mechanism — corrupt the latent, decode to pixels — is the simplest of the recent recipes and the only one that explicitly matches the corruption process the downstream generator will see. Three sharp contrasts with filed neighbors:

Versus Making Reconstruction FID Predictive of Diffusion Generation FID (iFID metric): that paper argues rFID is uncorrelated with gFID because diffusion needs interpolable latents while reconstruction prefers disconnected ones. l-DeTok’s noise/mask corruption is one concrete way to manufacture interpolability — corrupted-latent reconstruction forces neighboring latents to remain decodable. DeTok is the training-side recipe iFID is the evaluation-side diagnostic for; the two should be tested together.
Versus Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders (RAE / frozen SigLIP-2 latents) and What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion (PAE): RAE inherits a frozen perception-encoder’s geometry; PAE shapes the manifold via VFM priors + perturbation regularization. l-DeTok argues you don’t need any external encoder — just train the tokenizer’s own decoder to handle the corruption budget downstream generators will inject. The semantic-distillation row in DeTok’s table makes the comparison direct: l-DeTok + distillation beats VA-VAE and MAETok, so the corruption objective is additive to encoder-inheritance recipes, not in tension with them.
Versus Unified Latents (UL): How to train your latents (UL): UL co-trains encoder + prior + decoder so the encoder’s noise floor equals the prior’s minimum noise level. l-DeTok freezes the tokenizer first and trains generators on top, but achieves a similar latent-corruption-budget alignment by training the decoder against a distribution of corruption strengths. Both arrive at “latent design = downstream-noise-aware” from different directions.

Practical note for the team: the recipe is famously cheap to try — γ and m are the only tokenizer-side hyperparameters, and the GitHub release ships pretrained l-DeTok-BB checkpoints. The 51% AR-side FID lift suggests this is worth ablating against any MAR/RasterAR-family training currently using VAE latents.