Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?

TL;DR

A Google DeepMind scaling-law study that asks how raw autoregressive next-pixel prediction — iGPT’s discarded recipe — scales when fit with IsoFlops profiles up to 7e19 FLOPs. The headline findings are (1) the optimal token-per-parameter ratio for raw pixels is 10–20× higher than for text, so pixel prediction is data-hungry but follows a clean power law; (2) the optimal scaling exponents are task-dependent — generation FID needs ~3–5× more data growth per FLOP than classification accuracy at the same resolution; (3) as resolution rises from 16² to 32² to 64², the model-size growth required outpaces data growth significantly. Projecting forward, the authors estimate that competitive ImageNet classification (>80% top-1) and generation quality (~90 FD) via pure next-pixel modeling are reachable within five years at current 4–5× annual compute-growth rates — the bottleneck is compute, not data.

Key claims

Raw next-pixel prediction on images follows a Chinchilla-style power-law $N^*, D^* \propto C^\alpha$ when fit with IsoFlops profiles, but requires 10–20× more tokens per parameter than language modeling — at 32² resolution the loss-optimal token-to-parameter ratio is well above the text-model ratio of ~20 even beyond a 10²² FLOPs budget [§1, §4.1, Fig. 1(a)].
The optimal scaling strategy depends on the downstream metric. At fixed 32² resolution the eval-loss-optimal exponents are $N^* \propto C^{0.55}$ / $D^* \propto C^{0.45}$ (model-scaling-heavy); the classification-optimal exponents roughly match; but the generation-quality-optimal exponents shift to $N^* \propto C^{0.38}$ / $D^* \propto C^{0.62}$ , requiring data to grow 3–5× faster than for classification [§4.2, Fig. 1(b), Fig. 7(a)].
The scaling-law predictions are quantitatively precise: an out-of-distribution 5×-compute run trained at the classification-optimal $(N, D)$ allocation hits 46.41% ImageNet top-1 vs a predicted 46.39%, and a generation-optimal 5×-compute run hits FD 240 vs predicted 244 [§4.2, Figs. 1(c)(d)].
The compute optimum is regime-dependent across resolutions: as image resolution increases from 16² → 32² → 64², the optimal model size must grow much faster than the optimal data size — the resolution-aware reparameterization $N_p^* = N^*/(H \cdot W)$ and $D_p^* = D^*/(H \cdot W)$ (parameters and images, not pixels) shows the parameter exponent rising consistently with resolution [§3.3, §4.2].
Forward projection: at current 4–5× annual frontier-compute growth, an autoregressive next-pixel Transformer trained at the appropriate optimum will reach ~80% ImageNet top-1 and competitive generation FID (~90 by Fréchet Distance against ImageNet) within roughly five years; the binding constraint is compute (~10²⁵ FLOPs needed), not the supply of training images — pixel modeling is “far from data bound” [§1, §4.2].
Modern Transformer details — RMSNorm pre-norm, 1D RoPE, GEGLU FFN — replace iGPT’s architecture but the model family is otherwise minimal (S-28M, S-77M, B-227M, B-449M with six IsoFlop perturbations each per FLOPs budget, totalling 24×7 training runs) [§3.2, Tables 1–2].
The fit follows Hoffmann et al.’s IsoFlops-profile methodology (fit a parabola at each FLOPs budget to identify the loss-minimum $(N^*, D^*)$ , then fit a power law across budgets), correcting the Kaplan-et-al. overestimation of parameter scaling that prior pixel-prediction scaling work (Henighan et al.) inherited [§2, §3.3].

Method

The setup is deliberately minimal: train a family of standard pre-norm Transformers (RMSNorm + 1D RoPE + GEGLU + GELU) on JFT-300M images at 16², 32², 64² resolutions, each pixel quantized to 256 values and emitted in raster order with negative log-likelihood. Four base architectures (S-28M through B-449M parameters) are each perturbed into six IsoFlops variants (varying width and depth at fixed FLOPs) and trained at four compute budgets, giving 24 runs per budget × 7 variants per architecture = ~168 runs feeding the parabolic fits per budget. Optimizer is AdamW, batch size 512, cosine LR with 1000-step warmup; the Simply codebase is used for the Transformer implementation. Standard Inception-style random crop + horizontal flip; JFT’s 300M images at 32² traverses ~3×10¹¹ pixels per epoch, so the infinite-data limit is approximated by single-epoch training.

Three downstream metrics are scaled independently. Next-pixel-loss-optimal uses the eval NLL directly. Classification-optimal uses best-layer linear-probing accuracy on ImageNet: features are extracted from every Transformer layer of the frozen backbone, a 1000-way linear head is fit by SGD for 600 epochs at batch 4096 with a learning-rate grid search, and the best (layer, LR) combination is reported per run. Generation-optimal uses image-completion Fréchet Distance: the top half of an ImageNet-val image is conditioned on, the model autoregressively generates the bottom half pixel-by-pixel, and FD is computed against 10k real bottom-halves using DINOv2 features (per Stein et al. on FID’s misalignment with human preference). For the 32² IsoFlops study FD is computed at 32²; for the resolution-comparison study FD is computed at native resolution with 10k references.

For each metric, parabolic fits at each FLOPs budget identify the loss-minimum compute-optimal $(N^*, D^*)$ ; these are then fit with a Hoffmann-style power law in $C$ . The resolution-comparison study reparameterizes to per-pixel quantities ( $N_p^* = N^*/HW$ , $D_p^* = D^*/HW$ ) so the exponent fits are comparable across resolutions; the exponents themselves are resolution-invariant under this reparameterization but the prefactor changes.

Results

Token-per-parameter ratio. At 32² and budgets up to 7e19 FLOPs, the optimal pixel-per-parameter ratio sits 10–20× above the ~20 ratio Chinchilla derives for text — i.e. raw pixel models stay data-hungry even at large compute. Visualized in Fig. 1(a).
Loss-optimal exponents at 32² are roughly $N^* \propto C^{0.55}$ , $D^* \propto C^{0.45}$ — slower parameter scaling and faster data scaling than Henighan et al.’s earlier estimates, which the paper attributes to Henighan’s use of intermediate-loss extrapolation (the same methodological flaw Hoffmann et al. identified in Kaplan).
Classification-optimal and generation-optimal exponents diverge at 32². Classification ≈ loss-optimal (model-scaling-heavy). Generation: $N^* \propto C^{0.38}$ , $D^* \propto C^{0.62}$ — data must grow 3–5× faster than parameters per FLOPs budget. Mechanistically the authors attribute this to generation needing to capture long-tail low-level texture statistics that classification can ignore.
Power-law precision. A 5×-compute classification-optimal run hits 46.41% ImageNet top-1 vs predicted 46.39%. A 5×-compute generation-optimal run hits FD 240 vs predicted 244. Both are within experimental noise of the extrapolated power law (Fig. 1(c)(d)).
Resolution scaling. Moving from 16² to 32² to 64² (at matched per-pixel compute), the resolution-aware model-size exponent rises consistently — i.e. doubling resolution requires more-than-quadrupling parameters per pixel for the optimum, while data growth per pixel can stay flat. The paper does not give native-resolution exponents directly but states the qualitative trend across the three studied resolutions.
Forward projection. Substituting the classification-optimal exponents and ~10²⁵ FLOPs (roughly 5 years of frontier-compute growth at 4–5×/yr) yields >80% ImageNet top-1. The generation projection at the same budget gives ~90 FD, which the authors describe as “compelling generation results.” Context: established methods like MAE (~3×10²² FLOPs) and DINO (~10²² FLOPs) already reach comparable classification accuracy with three to four orders of magnitude less compute, so the projection is a feasibility argument rather than a competitive one.

Why it’s interesting

This is the rigorous scaling-law counterpart to the renewed pixel-space movement on the wiki. Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation and NEO-unify: Building Native Multimodal Unified Models End to End argue empirically that throwing out the VAE and the vision encoder is a win at 7B scale; PixelGen: Improving Pixel Diffusion with Perceptual Supervision and Continuous Adversarial Flow Models argue the gap was the loss function, not the pixel-vs-latent dichotomy; this paper asks the orthogonal question — if you commit to raw pixels, how much compute does scaling actually need? The answer is “a lot, but it follows a predictable power law” with the load-bearing wrinkle that the optimal allocation depends on which downstream task you care about.

It also adds a clean new datapoint to the Hyperparameter scaling laws picture: the metric-dependence of the compute-optimal exponent (generation vs classification differing by 3–5× in data growth at the same backbone) is exactly the kind of regime split the page’s “Fitting vs Transfer vs Enforce” debate already needs to account for, but in a new axis — which downstream metric you target — rather than the architecture / optimizer / batch-size axes already on the page. Closest precedent on the page is the The Design Space of Tri-Modal Masked Diffusion Models result that masked-diffusion’s compute-optimal data-per-parameter exponent ( $D^* \propto N^{0.476}$ ) differs from AR-Chinchilla’s ( $D^* \propto N$ ); both are metric-or-objective-dependent scaling fork claims.

The Karpathy thesis (Karpathy: pixels may be better LLM inputs than text tokens (DeepSeek-OCR springboard)) — that pixels are a strictly better LLM input substrate than text tokens — gets a sober counter from this paper’s data-hunger result: even if pixels are the right input, the compute needed to scale raw pixel modeling to competitive performance is currently three to four orders of magnitude beyond DINO/MAE, so the pixel-input vision is a five-year bet, not a near-term one.