NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale

TL;DR

NextStep-1 is StepFun’s 14B autoregressive text-to-image model that generates images patch-by-patch over continuous VAE latents — no VQ, no heavy diffusion backbone — by attaching a small 157M MLP-based flow-matching head to a Qwen2.5-14B transformer trained end-to-end with next-token prediction. A modified Flux VAE (channel-wise normalization + stochastic perturbation on the latents) is the key ingredient that makes training over 16-channel continuous tokens stable. Headline numbers: 0.63 GenEval (0.73 with Self-CoT), 0.88 / 0.67 on GenAI-Bench basic/advanced (0.90 / 0.74), 0.54 WISE (0.67 / 0.83 with Self-CoT + rewrite), 85.28 DPG-Bench — beating every prior AR baseline (Janus-Pro, Emu3, Infinity, Fluid) and matching mid-tier diffusion (SD 3.5 Large, Flux.1-dev, BAGEL). Editing variant NextStep-1-Edit hits 6.58 GEdit-Bench-EN and 3.71 ImgEdit-Bench, parity with Flux.1-Kontext-dev. The paper’s central methodological claim is that the generative work is done by the Transformer, not the flow head: scaling the head from 40M → 528M barely moves quality, so the FM head functions like a lightweight LM-head-equivalent sampler over continuous tokens.

Key claims

Autoregressive image generation with continuous tokens at scale is competitive with diffusion: NextStep-1 14B+157M is the first AR model that closes the gap with mid-tier diffusion baselines (SD 3.5 Large, Flux.1-dev, BAGEL) on GenEval / GenAI-Bench / DPG-Bench / WISE without resorting to a heavy diffusion backbone or VQ tokenization [§1, Table 2].
The flow-matching head is a sampler, not the generative engine: ablating the head size (40M → 157M → 528M) produces nearly identical GenEval / GenAI-Bench / DPG-Bench numbers, supporting the claim that the AR transformer models the conditional distribution and the FM head only translates it into a continuous token [§6.1, Table 7].
Two image-tokenizer modifications are essential for stable training over 16-channel continuous latents at scale: (a) channel-wise normalization to zero mean / unit variance, and (b) stochastic perturbation z' = z + α·ε with α∼U[0, α_max] following σ-VAE — together preventing variance collapse and stabilizing convergence [§2.2, Image Tokenizer].
1D RoPE is sufficient for mixed text-image sequences in this design — 2D / multimodal RoPE variants were not needed for the reported results [§2.2, Causal Transformer].
Training recipe: 1.23T tokens at 256² (Stage 1), then dynamic-resolution Stage 2 with 256² and 512² buckets at 0.61T tokens, then 40B-token annealing on a 20M-sample high-quality subset, then SFT (5M samples including CoT data) and DPO (20K prompts × 16 candidates scored by ImageReward) [§4.2, §4.3, Table 1].
Two-track DPO: standard ImageReward-based preference pairs (top-4 vs bottom-12 of 16 candidates) plus a Self-CoT variant where the model emits a CoT prompt extension before image generation; Self-CoT consistently boosts GenEval by ~10 points and WISE by ~13 points [§4.3, Tables 2 / 4].
NextStep-1 reaches 0.54 / 0.67 on WISE (overall / with Self-CoT) and 0.79 / 0.83 under prompt-rewrite — the best AR-family result on WISE, exceeding most diffusion peers and approaching Qwen-Image’s 0.62 (no Self-CoT comparable) [Table 4].
On OneIG-Bench English prompts, NextStep-1 reaches an overall 0.417 — substantially above Emu3 (0.311) and Janus-Pro (0.267), and only behind frontier closed models (GPT-4o 0.533, Imagen4 0.515) and Qwen-Image (0.539) [Table 3].
NextStep-1-Edit (1M-sample finetune of NextStep-1 on Step1X-Edit / GPT-Image-Edit / proprietary data) reaches 6.58 GEdit-Bench-EN G_O and 3.71 ImgEdit-Bench, parity with Flux.1-Kontext-dev (3.71) and competitive with BAGEL (3.42) and OmniGen2 (3.44) — first AR editor at this level [Table 5].
Image tokenizer is initialized from Flux.1-dev VAE and fine-tuned with reconstruction + perceptual losses only (no GAN) for 50K steps at LR 1e-5, with AdamW β=(0.9, 0.95) on 512-batch [§4.1].
Total training corpus mix at pretraining: 20% text-only (400B tokens sampled from Step-3), 60% image-text pairs (550M filtered + recaptioned via Step-1o-turbo), 20% interleaved (80M video frames + tutorials + character-centric NextStep-Video-Interleave-5M + multi-view MV-ImageNet-v2 / Objaverse-XL) [§3, Table 1].

Method

NextStep-1 extends a decoder-only transformer (Qwen2.5-14B initialization) to image generation by appending two output heads to the same hidden-state stream: a standard cross-entropy LM head for discrete text tokens, and a 157M MLP-based flow-matching head for continuous image tokens. Images enter as continuous tokens from a modified Flux VAE — 8× spatial downsample, 16 channels, channel-wise normalization to zero-mean/unit-variance, and a σ-VAE-style stochastic perturbation z' = z + α·ε with α ~ U[0, α_max] applied during training to enforce a uniform latent distribution and prevent variance collapse. The 16-channel latents are pixel-shuffled with a 2×2 space-to-depth kernel into 64-channel tokens (a 256² image becomes a 16×16 grid of 256 tokens), then flattened to 1D for the causal transformer. Special tokens <boi>, <eoi>, and an <image_area>h*w metadata token bracket every image. 1D RoPE handles all positions. The training loss is a weighted sum L = λ_text · CE + λ_image · FM_MSE with the loss-weight ratio fixed at 0.01 : 1 throughout pretraining; the FM head regresses the velocity field that maps a noised patch to its clean target. At inference, text tokens are sampled from the LM head as usual; for each image token position, the FM head is integrated to convert the transformer’s predicted distribution parameters into a continuous patch. The whole stack — image tokenizer (Flux VAE) frozen during LM training, transformer + heads trained end-to-end — is then SFT’d on 5M instruction samples (including CoT text-to-image reasoning traces) and DPO’d on 20K diverse prompts × 16 ImageReward-scored candidates, with a Self-CoT variant that adds a model-generated CoT prompt extension before sampling.

Results

GenEval / GenAI-Bench / DPG-Bench [Table 2]: 0.63 / 0.73 (Self-CoT) on GenEval; 0.88 basic + 0.67 advanced on GenAI-Bench (0.90 / 0.74 with Self-CoT); 85.28 on DPG-Bench. Beats every AR baseline (Janus-Pro-7B 0.80 GenEval / 0.86-0.66 GenAI / 84.19 DPG; Emu3 0.54/0.65; Infinity 0.79). Matches mid-tier diffusion (SD 3.5 Large 0.71 / 0.88-0.66 / 83.38; Flux.1-dev 0.66 / 0.86-0.65 / 83.79; BAGEL 0.82/0.88).
WISE [Table 4]: 0.54 overall (0.67 with Self-CoT), 0.79 with prompt-rewrite (0.83 with both). Best AR-family WISE in the comparison; exceeds Flux.1-dev (0.50), SD 3.5 Large (0.46), Playground v2.5 (0.49), MetaQuery-XL (0.55) and approaches Qwen-Image (0.62) and BAGEL (0.52/0.70-rewrite).
OneIG-Bench EN [Table 3]: Alignment 0.826, Text 0.507, Reasoning 0.224, Style 0.332, Diversity 0.199, Overall 0.417. Substantially above Emu3 (0.311) and Janus-Pro (0.267); behind GPT-4o (0.533), Qwen-Image (0.539), Imagen4 (0.515), Seedream 3.0 (0.530).
NextStep-1-Edit [Table 5]: GEdit-Bench-EN 7.15 (G_SC) / 7.01 (G_PQ) / 6.58 (G_O), GEdit-Bench-CN 6.88 / 7.02 / 6.40, ImgEdit-Bench 3.71 — matches Flux.1-Kontext-dev (3.71), beats BAGEL (3.42) and OmniGen2 (3.44).
FM-head size ablation [§6.1, Table 7]: Small (40M, 6 layers, 1024 hidden), Base (157M, 12×1536), Large (528M, 24×2048) reach GenEval 0.55 / 0.55 / 0.56, GenAI-Bench 0.76 / 0.75 / 0.77, DPG-Bench 83.46 / 84.68 / 85.50 — variations within noise compared to the 14B baseline’s headline 0.59 GenEval.

Why it’s interesting

NextStep-1 is the first filed paper on the wiki that makes the continuous-tokens-with-next-token-prediction recipe work for text-to-image at SOTA-AR scale, and the architectural delta from prior continuous-AR work (MAR, Fluid) is small enough to be informative: a stabilized VAE plus a strong language backbone is enough. The closest sibling is Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression (TransDiff), which also pairs an AR transformer with a flow-matching head over continuous latents — but TransDiff’s head is a full DiT decoder conditioned on the entire AR output sequence at once, while NextStep-1’s head is a lightweight MLP applied per-patch, and the ablation here shows that even shrinking the head 4× barely hurts. This is the cleanest evidence yet for the “the transformer does the generative modeling, the head just samples” position that Unified Multimodal Models has been circling. Methodologically, it complements Latent Denoising Makes Good Visual Tokenizers (latent-denoising-as-tokenizer) and contrasts with VQ-token AR systems like UniTok: A Unified Tokenizer for Visual Generation and Understanding and Janus-Pro by keeping continuous latents and paying the price in a cheap auxiliary head rather than in quantization loss. The fact that the same Qwen2.5-14B trunk hosts text + image + edit + interleaved with a single 0.01 : 1 loss ratio also makes this a useful datapoint for the unified-MM debate — closer in spirit to the AR-side wing (Janus / Show-o) than to the AR+heavy-Diffusion wing (BAGEL / OmniGen2 / OmniGen2: Exploration to Advanced Multimodal Generation).