Scalable GANs with Transformers

TL;DR

GAT (Generative Adversarial Transformers) is a pure-transformer GAN in SD-VAE latent space that reaches FID 2.96 on class-conditional ImageNet-256 in 40 epochs of one-step generation — competitive with strong diffusion/flow baselines at far less training. It borrows the DiT recipe wholesale (ViT generator+discriminator, AdaLN-style modulation, VAE latent space) and adds two GAN-specific fixes: Multi-level Noise-perturbed image Guidance (MNG) to keep early generator layers active, and a width-aware learning-rate scaling rule (η ∝ 1/d_channels) to prevent divergence as the model grows from S to XL. The discriminator carries an auxiliary REPA-style alignment loss against DINOv2 features, only on real data.

Key claims

GAT-XL/2 reaches FID 2.96 on ImageNet-256 class-conditional single-step generation in 40 epochs, 6× fewer than GigaGAN’s 480 epochs and competitive with multi-step DiT-XL/2 (2.27 at 250 NFE × 2) and MeanFlow-XL/2 (3.43 at 1 NFE, 240 epochs) [Table 1, §3.1].
Naive GAT training without MNG leaves early generator layers inactive — PCA of intermediate features shows high redundancy in early blocks and block-ablation LPIPS shows perceptual contribution concentrated in late blocks [§3.3, Fig. 4].
MNG injects multi-level noise-perturbed intermediate supervision: each generator stage emits an output, perturbed by an exponentially-decreasing-with-depth Gaussian, and the discriminator evaluates all levels — coercing a coarse-to-fine division of labor across depth without explicit multi-scale resolution [§2.3, Eqs. 1–3].
An exponential noise schedule (MNG-exp) beats both no-MNG and a linear schedule (MNG-lin) on FID-5K; MSG-GAN-style resize-based multi-scale supervision is the worst of the four — overemphasizing cross-scale consistency suppresses generation quality [§3.3, Fig. 5(a)].
Naively scaling identical hyperparameters from GAT-S/2 to GAT-B/2 causes divergence; the width-aware rule η = η_base · (d_base / d_channels) restores stable convergence by holding constant the per-step magnitude of network-output change [§2.4, Eq. 4].
Cross-checking learning rates between scales is the load-bearing experiment: training GAT-B/2 with GAT-S/2’s LR diverges, training GAT-S/2 with GAT-B/2’s LR converges too slowly — neither config transfers without the channel-aware rescaling [§3.3 “Adaptive learning rate”].
REPA-style alignment of discriminator last-layer features to a frozen DINOv2 teacher (cosine similarity, real images only) gives a “significant and consistent” generator-quality gain even though the alignment objective never touches the generator [§2.5, Eqn. 9; §3.3, Fig. 5(c)].
Scaling the discriminator individually improves FID more than scaling the generator individually — discriminator capacity dominates the gain [§3.4, Fig. 6(a)].
Generator-fake-data and discriminator-real-data CKNNA against DINOv2-g rise in lockstep across training, suggesting generation quality is gated by how well the discriminator’s representations align with a VFM [§3.4, Fig. 6(b)].
GAT scales monotonically: larger capacity → consistently lower FID throughout training (not just at convergence) and a -0.95 correlation between FID-50K and generator GFLOPs across S/B/L/XL [§3.2, Fig. 3].

Method

GAT instantiates generator and discriminator as plain ViTs operating on SD-VAE latents (32×32, patch size 2). The generator removes the patchify input and uses an unpatchify linear head; both networks use AdaLN-style modulation conditioned on a style vector (mapped from the latent code + class label via an MLP, à la StyleGAN), with RMSNorm and LayerScale initialized small. The discriminator appends a [CLS] token before block 1 and reads out a real/fake logit through a linear head.

The loss stack is the R3GAN combo — relativistic pairing loss with an approximated two-sided gradient penalty (R1+R2) — augmented by (a) MNG, where each generator stage emits an intermediate residual-accumulated output that is perturbed with stage-specific Gaussian noise (strongest at the earliest stage, exponentially decreasing) before being scored by the discriminator, and (b) REPA-style alignment of the discriminator’s last-layer [CLS]+patch tokens against a frozen DINOv2 teacher via cosine similarity through a small learnable MLP, applied to real images only.

For scaling, hyperparameters are fixed across capacities except the learning rate, which follows η = η_base · (d_base / d_channels). All experiments run 50K iterations at batch 512 (20 epochs of ImageNet-1k) unless otherwise noted; GAT-XL/2 trains for 100K iterations (40 epochs).

Results

GAT-XL/2: 602M params, NFE=1, 40 epochs, FID-50K = 3.02 (no guidance) / 2.96 (with latent-space guidance from Zhang et al. 2024) on ImageNet-256 [Table 1].
Beats prior 1-NFE GAN baselines at parity-or-better epoch count: GigaGAN 3.45 at 569M / 480 epochs, BigGAN 6.95, StyleGAN-XL 2.30 (but uses ImageNet-pretrained discriminator, which deflates FID per Kynkäänniemi et al. 2022) [Table 1].
Beats 1-NFE diffusion-from-scratch: MeanFlow-XL/2 3.43 at 240 epochs, Shortcut-XL/2 10.60 at 250, iCT-XL/2 34.24 [Table 1].
Per-block LPIPS ablation contribution under MNG decreases monotonically from early → late blocks (with a final-block spike), versus a no-MNG model where early blocks contribute negligibly [Fig. 4(b)].
Strong negative correlation (-0.95) between generator GFLOPs and FID-50K across S/B/L/XL — Pareto-clean scaling [Fig. 3(c)].

Why it’s interesting

This is the cleanest demonstration to date that GANs can adopt the DiT recipe end-to-end (VAE latent space, plain ViT, AdaLN modulation, REPA-style external-encoder alignment) and recover the one-step efficiency advantage that diffusion gave up — the result puts a 602M one-step GAN at 2.96 FID on ImageNet-256 in 40 epochs, against MeanFlow’s 3.43 at 1-NFE / 240 epochs. The REPA-on-discriminator trick is the natural mirror of REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training / No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA) for adversarial training — DINOv2 features lifting the discriminator’s representation quality propagates through to the generator without ever touching the generator’s gradients, which connects this paper to the broader “Diffusion training efficiency = better external feature priors” thread.

The channel-aware LR rule η ∝ 1/d is a minimal, hand-crafted slice of the Hyperparameter scaling laws / Training stability at scale debate. It is essentially the µP rule for the LR coordinate without the full µP machinery; the paper notes the parallel to equalized-LR (StyleGAN-era) and obtains it from a “constant update magnitude” argument. Worth comparing against How to Set the Learning Rate for Large-Scale Pre-training?‘s Fitting paradigm and Controlled LLM Training on Spectral Sphere‘s SSO — GAT’s recipe is the simplest possible width-axis fix, but it works because GAT keeps depth fixed across model sizes (only width changes), so the harder depth-coupled µP corrections never arise.

MNG is the paper’s most genuinely novel contribution: a pure-transformer way to inject a coarse-to-fine division of labor without resolution hierarchies or explicit pyramid losses (cf. MSG-GAN’s resize-based variant, which the ablation shows performs worst). The “noise-as-implicit-pyramid” framing is conceptually adjacent to diffusion’s noise schedule, but used here as a supervision lever rather than as the generative process itself.