FARMER: Flow AutoRegressive Transformer over Pixels

TL;DR

FARMER is a pixel-space image generator from ByteDance Seed that unifies an invertible Autoregressive Flow (AF) with a Gaussian-mixture autoregressive Transformer into a single end-to-end likelihood-trained model. Instead of mapping pixels onto a fixed isotropic Gaussian (as JetFormer / TARFlow / STARFlow do), FARMER maps them to a latent sequence whose distribution is itself parameterized by an AR Transformer predicting GMM heads — exact likelihoods, no VAE, no diffusion sampler. Two practical pieces make this scale on raw 256² ImageNet: a self-supervised channel-split that disentangles structural (“informative”) channels from color/detail (“redundant”) channels so the AR Transformer only does per-token GMM on the small informative subset, and a one-step distillation that collapses the slow sequential AF reverse pass into a single forward of a non-causal student (≈22× faster). At 1.9B params with patch 8 the model hits FID 3.60 on ImageNet-256 — competitive with JetFormer / TARFlow / STARFlow in their own NF lane.

Key claims

Mapping image pixels to a learned AR-parameterized GMM target distribution beats mapping to a fixed standard Gaussian: FARMER-1.9B/patch-8 reaches FID 3.60 on ImageNet-256, vs JetFormer 6.64 and STARFlow patch-8 4.69 at comparable parameter count [Table 2].
A self-supervised channel split is the load-bearing trick for pixel-space AR: partition the per-token latent into a small “informative” group (modeled per-token with a K-component GMM) and a larger “redundant” group (modeled as a single shared GMM conditioned on the entire informative sequence) [§3.3, Eq. 14].
The split is not a manual feature: the model self-organizes contour/structure into the informative channels and color/fine-detail into the redundant channels, without an explicit prior or supervision signal [§3.3, Fig. 7].
The ablation isolates the contributions cleanly: starting from FID 61.17 (no dim reduction, no permute, no cond-repeat, no CFG), adding self-supervised dim reduction drops FID to 49.29, then condition-repeat → 45.34, permute → 44.56, naive CFG → 8.66, and resampling-based CFG → 5.67 (this is the 1.1B / K=1024 setting; the 1.9B / patch-8 / K=200 final config hits 3.60) [Table 3].
Resampling-based CFG fixes the fact that the standard CFG product-of-GMMs is not a tractable distribution: propose N candidates from the conditional GMM, weight by the ratio (p_cond / p_uncond)^w, resample categorically — the per-token marginal then equals the intended guided distribution [§3.4, Algorithm 1].
The AF reverse pass is the inference bottleneck (strict token-by-token causality at every block); one-step distillation trains a non-causal student AF whose forward path mimics the teacher’s reverse path, accelerating the reverse from 0.169s/img to 0.0076s/img (≈22×) with comparable generation quality and only ~60 additional training epochs [§3.5, Algorithm 2, Table 5].
A formal equivalence: when the GMM mixture count K=1, the AR Transformer’s GMM head reduces to a single affine block and FARMER collapses to a single deeper Autoregressive Flow — the GMM head is what gives FARMER expressivity beyond a plain AF [§3.1, §7.1].

Method

The model is purely pixel-space: a 256² image is dequantized with cosine-annealed Gaussian noise (σ 0.1 → 0.005), patchified at stride 16 or 8 with no dimension compression, then reshaped to a sequence of D-dim tokens. The Autoregressive Flow front-end is a stack of N affine blocks (28 for 1.1B, 32 for 1.9B), each block applying y_t = σ_t ⊙ x_t + μ_t where (σ_t, μ_t) are predicted by a causal Transformer over the previous tokens — invertibility is by construction, and the log-Jacobian determinant is a triangular sum of log σ. Between AF blocks, the token order is reversed (TARFlow-style permutation) to give effectively bidirectional context. The output latent sequence z is split channel-wise into informative (dim 128, K=200 GMM components) and redundant (dim 640, K=200 components for the whole sequence, shared). A deep causal AR Transformer (12 layers / 24 layers) predicts the per-token informative GMM autoregressively and the single shared redundant GMM from the full informative sequence as global context. The class label is repeated 64 times and prepended to amplify conditioning. Loss is the NLL composed of the AF Jacobian term plus both GMM heads. Sampling at inference uses the resampling-based CFG and, for the AF inverse pass, the one-step distilled student so the entire reverse is one parallel forward.

Results

ImageNet-256 class-conditional FID: FARMER-1.9B patch 8 = 3.60 (IS 269.21, Pre 0.81, Rec 0.51); FARMER-1.9B patch 16 = 3.96; FARMER-1.1B patch 8 = 5.02; FARMER-1.1B patch 16 = 5.40 [Table 2].
Compared to pixel-space NF baselines at comparable epochs: JetFormer (2.8B, 500 epochs) 6.64; TARFlow patch 8 (1.3B, 320 epochs) 5.56; STARFlow patch 8 (1.4B+86M, 320 epochs) 4.69 / 2.40 with finetuned decoder. FARMER 1.9B patch 8 (1.9B, 320 epochs) 3.60 is the best end-to-end pixel-space NF result at this scale [Table 2].
Latent-space comparison points (latent diffusion / latent AR): SiT-XL 2.06, REPA 1.42, REPA-E 1.12, MAR-L 1.78 — FARMER does not match the best latent models but closes a substantial gap vs prior end-to-end pixel NFs without any VAE or distillation-from-diffusion stage [Table 2].
Distillation: AF reverse from 0.1689s/image to 0.0076s/image (≈22×) with FID essentially preserved, requiring only 60 extra training epochs on the distilled student [§4.3, Table 5].
Qualitative claim: pixel-space training lets FARMER reconstruct fine-grained features (e.g. faces) that VAEs in latent pipelines blur or distort [§4.2, Figs. 4–5].

Why it’s interesting

FARMER is the third member of the 2025 normalizing-flow comeback for image generation, after JetFormer (Dec 2024) and the TARFlow/STARFlow lineage (STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis §3, STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flows §1) — and it is the only one of the three that operates fully end-to-end on raw pixels while also parameterizing the target distribution autoregressively rather than as a fixed Gaussian. That places it in a different cell of the Pixel-space diffusion design space from JiT-based methods like PixelGen / CAFM / AsymFlow: those keep flow-matching but change the loss / criterion / parameterization on a non-invertible backbone, while FARMER keeps invertibility-by-construction and exact MLE. The self-supervised channel-split is the most reusable idea here — it is a principled answer to the “pixel space wastes most of its dimensions on low-frequency redundancy” complaint that PixelGen and L2P each address differently. The one-step distillation also sits adjacent to Diffusion Distillation but with a different target (collapse a sequential invertible reverse pass, not a multi-step ODE), which makes it a clean test of whether NF-style inverses are amenable to the same compression that diffusion samplers have already absorbed. The most natural follow-up question is whether FARMER’s pixel-AR backbone could replace the latent AR backbone in STARFlow-V — at 3.60 FID it’s the strongest end-to-end pixel NF candidate to date for that role.