GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling

TL;DR

GenFirst (ByteDance Seed) attacks the two-stage recipe of latent diffusion — freeze a VAE, then train a generative model on its latents — by training encoder + prior + decoder end-to-end. The paper identifies two blockers: (1) reconstruction and prior fitting both shrink the posterior, so the entropy term of the KL is what prevents latent collapse in naive joint training; (2) reconstruction is fast and strongly supervised while generation is slow, so joint training with balanced losses lets reconstruction dominate. Their answer is “generation before reconstruction”: start with a generation-friendly latent shaped by the diffusion loss under weak reconstruction pressure, then progressively strengthen reconstruction to recover pixel detail. Reported: gFID 0.97 with CFG / 1.45 without on ImageNet-256 (SiT prior), GenEval 0.90 on T2I (MMDiT), and — per the announcement thread — the first diffusion model to reach FID < 1.0 without an FD loss, at ≥70× fewer training steps than a matched from-scratch SiT.

Key claims

Direct end-to-end joint training of a VAE with a latent generative prior is prone to latent collapse; the KL entropy term is the component that keeps the posterior non-degenerate against reconstruction and prior-fitting pressure [Abstract, §Introduction].
Reconstruction and generation have asymmetric learning dynamics under joint training — reconstruction converges fast under strong direct supervision, generation is slower and harder to optimize — so a symmetric-loss schedule causes reconstruction to dominate and yield a non-generation-friendly latent [Abstract].
GenFirst prescribes a two-phase schedule: first let the generative loss shape the latent under weak reconstruction pressure, then progressively strengthen reconstruction to restore visual detail [Abstract].
End-to-end SiT + VAE trained with GenFirst reaches gFID 0.97 with CFG and 1.45 without CFG on ImageNet-256 [Abstract].
End-to-end MMDiT + VAE with GenFirst reaches GenEval 0.90 for text-to-image [Abstract].
The framework is prior-agnostic: it is validated with continuous autoregressive priors (exact likelihood) and SiT priors (implicit likelihood) [Abstract].
The learned latent supports generation, representation learning, and reconstruction from a single shared space, and the recipe extends to continuous text latents to enable unified T2I and I2T diffusion modeling [Abstract].
Per the announcement thread, GenFirst is the first diffusion model to reach FID < 1.0 on ImageNet-256 without an FD loss, and reaches the final SiT result at ≥70× fewer training steps than the matched baseline [tweet §body].

Method

Two design axes. First, the training objective: the standard VAE KL objective is decomposed and the entropy term of $D_{KL}(q(z|x) \| p(z))$ is identified as the load-bearing component that prevents posterior collapse under joint training with a generative prior — because both reconstruction (which prefers deterministic latents) and prior-fitting (which prefers latents matching the prior’s low-entropy structure) push the posterior toward degeneracy. Retaining explicit entropy preservation stabilizes joint training. Second, the training schedule: the paper’s central empirical observation is that reconstruction and generation optimize at different rates, and if you weight them equally from step 0 the fast-moving reconstruction loss dictates the latent geometry. GenFirst inverts that — start with a weak reconstruction weight so the generative loss shapes the latent, then progressively strengthen reconstruction so the decoder can recover pixel detail on top of the generation-friendly latent. Validated with continuous AR priors (exact likelihood) and SiT priors (implicit likelihood), with an MMDiT variant for text-to-image and a further extension to continuous text latents for unified T2I / I2T.

Results

On ImageNet-256, the end-to-end SiT + VAE reaches gFID 0.97 with CFG and 1.45 without CFG — surpassing the previous best two-stage recipes reported on this benchmark and, per the announcement thread, being the first diffusion model to cross FID < 1.0 without relying on an FD loss. Convergence speedup: reaches the final SiT quality with ≥70× fewer training steps than a matched from-scratch SiT baseline [tweet §body]. For text-to-image, end-to-end MMDiT + VAE with GenFirst reaches GenEval 0.90 [Abstract]. Beyond these numbers, the paper reports that the shared latent supports representation learning as well as generation, and that the recipe extends to continuous text latents, enabling a single end-to-end framework for T2I and I2T diffusion modeling.

Why it’s interesting

Adds a fourth design point to the Representation Autoencoders cluster: the wiki has been tracking three positions — (i) frozen high-dim pretrained encoder + bespoke wide decoder (original RAE / RAEv2), (ii) frozen encoder + latent compression back to SD-VAE shape (FAE), (iii) from-scratch joint pre-training of an autoencoder + SSL (VTP). GenFirst is a fourth: from-scratch joint training of VAE + generative prior with a specific curriculum that resolves the “reconstruction–generation dilemma” flagged in Making Reconstruction FID Predictive of Diffusion Generation FID and named in Unified Latents (UL): How to train your latents (UL). Where UL’s answer is to couple the encoder noise floor to the prior’s minimum noise so the KL becomes a bitrate budget, GenFirst’s answer is a curriculum-based decoupling: let generation shape the latent first, then bolt reconstruction on. Also complements the internal-self-distillation strand (Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis, No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA)) in the Diffusion training efficiency cluster: both lines argue the standard two-stage recipe leaves capacity on the table, but GenFirst attacks the latent side while Self-Flow/SRA attack the generator’s internal supervision. The extension to continuous text latents ties this into Unified Multimodal Models as a candidate substrate for diffusion-based T2I ↔ I2T unification.