When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators

TL;DR

A large-scale benchmark of thirteen open-source text-to-image diffusion models released between 2022 and 2025, evaluated purely as synthetic training-data generators on a controlled ImageNet-1K Synth→Real transfer setup. The headline result is a regression: as newer T2I models improve on GenEval and CLIPScore, classifiers trained solely on their outputs do worse on real ImageNet-1K test data, not better. The authors attribute the gap to distribution collapse toward an aesthetic-centric manifold, plus texture / high-frequency degradation that doesn’t show up in standard generation benchmarks. This is a sharp counterpoint to the “synthetic data scales like real data” thesis that has driven a lot of recent work.

Key claims

Across thirteen open-source T2I models from 2022–2025 evaluated on a controlled ImageNet-1K Synth→Real transfer setup, classifier accuracy on the real test set declines as the underlying T2I model gets newer and scores higher on GenEval / CLIPScore [§3, Fig. 3].
The downward trend “higher generation-benchmark score → lower transfer accuracy” holds across architectures, i.e. it’s not a single-model artefact [Fig. 3, §4].
The proposed mechanism is distribution collapse: newer T2I models produce a narrower, aesthetic-centric distribution that undermines diversity and label–image alignment relative to real ImageNet [Abstract, §4].
Probing with depth-based (structure) classifiers and local BagNet (texture) classifiers plus low/high-pass filtered inputs shows that global structure and low-frequency content are well-preserved across model generations, while texture quality and high-frequency detail degrade systematically with newer models [§4].
The regression is observed despite using low classifier-free-guidance settings intended to broaden the synthesized distribution [§3].
The benchmark isolates the generator’s effect on downstream training data utility from prompt engineering, by holding the prompting protocol and downstream classifier architecture fixed across all thirteen models [§3].

Method

The authors build a fixed-protocol pipeline: take a target real dataset (ImageNet-1K), for each of thirteen open-source T2I models from 2022–2025 generate a large synthetic training set using the same class-label prompting strategy and a low classifier-free-guidance setting, train standard image classifiers from scratch on each synthetic set, and evaluate on the real ImageNet-1K test set. The headline plot puts each model’s classifier on a scatter against that model’s GenEval / CLIPScore. To diagnose why newer models transfer worse, they run a battery of probes that decompose the synthetic images by frequency band and by structural vs textural cue: depth-based classifiers (which see only structure), local BagNet classifiers (which see only texture), and classifiers trained on low- vs high-pass-filtered inputs. The probes localise the regression to texture and high-frequency content rather than global structure.

Results

Across thirteen T2I generations, classifier accuracy on real ImageNet-1K declines as the generator’s GenEval / CLIPScore goes up; the trend is consistent across classifier architectures [§3, Fig. 3].
The baseline classifier trained on real ImageNet outperforms every synthetic-data classifier in the benchmark [Fig. 3].
Structural information (depth-classifier accuracy, low-frequency-only accuracy) is roughly preserved across model generations; textural information (local-BagNet accuracy, high-frequency-only accuracy) degrades systematically with newer models [§4].
The aesthetic-collapse signature is visible in standard distributional statistics: newer-model outputs are narrower in the kinds of image content they cover at fixed class-label conditioning [§4].

Why it’s interesting

This is the first paper filed under Synthetic Training Data that argues against the dominant framing on that page — every prior entry (L2P: Unlocking Latent Potential for Pixel Generation, Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning, Action100M: A Large-scale Video Action Dataset, Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing, Smol Training Playbook (GPU MODE talk on SmolLM3)) describes a recipe where stronger generators + a verification gate substitute for real data. This paper isolates a regime where stronger generators make things worse for downstream training: T2I improvement is tracked on benchmarks (GenEval, CLIPScore) that reward aesthetics and prompt-following, and optimization against those benchmarks drives distributional collapse on dimensions (texture, high-frequency content, intra-class diversity) that the data-utility task actually depends on. The empirical setup also complements ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization‘s observation that aesthetic-scoring pipelines themselves induce category bias, and 2406.05184-style “retrieved real beats synthetic” findings (not yet filed). The most interesting follow-up for the Luma stack is whether the same regression shows up on video T2V models, where the verification gate in Action100M: A Large-scale Video Action Dataset / Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning is doing visible work — if it does, those gates may be the only thing keeping the synth-for-training story alive at frontier model scale.