Scaling Properties of Text Conditioning in Visual Generation

TL;DR

Chen et al. identify a previously unmeasured scaling axis for visual generation: converged diffusion loss scales with the amount of structured language in the training prompt, not with raw prompt token count. They quantify structuredness with two complementary metrics — a white-box likelihood measure (GPG) and a black-box attribute measure (ED) — and find that converged loss decreases approximately linearly in GPG and follows a power law in ED across controlled training runs. Guided by the fit, they improve diffusability by constructing structured prompts with image-derived semantic and geometric annotations, and improve promptability by training a prompter via SFT → cold-start RL → verifier-gated on-policy distillation. The resulting system beats all evaluated open-weight models on nearly every compositional, reasoning, and world-knowledge benchmark, and matches or surpasses the strongest closed-weight models on most.

Key claims

Converged diffusion loss is essentially flat in raw natural-language prompt token count but scales with the amount of structured language in the prompt, as measured by two independent metrics [Abstract, §3].
The GPG metric (a white-box likelihood measure of structured content) has an approximately linear relationship with converged diffusion loss across controlled runs [§3].
The ED metric (a black-box attribute measure) follows a power-law relationship with converged diffusion loss [§3].
Structured prompts constructed by attaching image-derived semantic and geometric annotations to training captions improve diffusability — i.e., reduce converged loss at fixed compute [§4].
A prompter trained by SFT + cold-start on-policy + verifier-gated on-policy distillation improves promptability at inference — turning short user prompts into structured prompts the trained model was optimized for [§5].
The combined recipe outperforms all evaluated open-weight T2I models on nearly every compositional, reasoning, and world-knowledge benchmark, and matches or exceeds the strongest closed-weight models on most [Abstract, §6].

Method

The paper first establishes the scaling result on a controlled T2I training setup: fix the model, the image data, and the compute budget, and sweep the prompt distribution along the structuredness axis. Diffusion loss is measured at convergence. GPG is a likelihood-style score computed from a language model over the prompt (white-box because it needs LM access); ED is derived from prompt attributes visible without model access. Both metrics correlate with converged loss under the sweep — GPG approximately linearly, ED as a power law — even though prompt token count does not.

The diffusability recipe uses the fit prescriptively: image → semantic labels (objects, attributes, relations) + geometric annotations (layout, spatial relations) → structured caption, replacing or augmenting the original caption. The promptability recipe trains a separate prompter model on paired (short prompt → structured prompt) data via supervised fine-tuning, then cold-starts on-policy generation, then applies on-policy distillation gated by a verifier that scores structured-prompt quality (the paper’s structuredness metrics themselves are plausible verifier signals, though the abstract does not spell out the verifier).

Results

The final system is reported to beat all evaluated open-weight models on nearly every compositional, reasoning, and world-knowledge benchmark, and to match or surpass the strongest closed-weight models on most evaluations. Specific benchmark numbers and the identity of “evaluated” open- and closed-weight baselines are not visible in the abstract-level fetch.

Why it’s interesting

This paper introduces a text-conditioning axis to the diffusion scaling-law literature that the wiki’s existing scaling-law entries — Scaling Laws For Diffusion Transformers and Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers — do not have. Prior DiT scaling fits sweep (N, D, r) with prompt distribution held fixed; here the prompt distribution itself is the swept variable, and the resulting exponent is a first-class scaling quantity. It also gives a mechanistic reading of the “long caption + prompt rewriting at inference beats short captions” recipe already documented in i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models and HunyuanImage 2.1 — Efficient 2K Text-to-Image with 32× VAE + DINOv2-Aligned REPA and MeanFlow Distillation‘s PromptEnhancer — those papers report the empirical win; this paper argues the win is because structured captions move training along a measurable scaling axis, and structured prompting at inference realigns the input distribution with training. The verifier-gated on-policy distillation stage sits in the same family as Thinking Machines’ On-Policy Distillation but specialized to prompt rewriting for T2I.