Skip to content
Bud's Wiki

Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples

TL;DR

Section titled “TL;DR”

Top-k GAN adds one line to the GAN generator update: when computing the generator gradient over a mini-batch of fake samples, zero out the gradient contributions from the k samples the critic scored lowest. This is annealed — early in training k = batch size (nothing is thrown away), then k decays each epoch to a floor. The intuition: bad samples’ gradients actually point away from the nearest true data mode, and updating on them is anti-signal, not just weak signal. Improves FID across DCGAN, WGAN-GP, MS-GAN, SN-GAN, SA-GAN, and BigGAN with no architectural or hyperparameter changes, setting a new CIFAR-10 conditional FID SOTA of 8.57 (from 9.21) at time of publication.

Key claims

Section titled “Key claims”
  • The critic in a trained GAN implicitly acts as a scoring function for how realistic each sample in the generator’s mini-batch is; using only the top-k highest-scored samples’ gradients to update the generator gives materially better FID than using the full batch [§3.1, Table 3].
  • On a 2D 25-mode Gaussian toy problem, gradient updates computed only on the worst-scoring (bottom-k) samples push those samples away from their nearest mode (negative cosine similarity with the oracle direction to nearest mode), even when they were within 4σ of that mode — so bad samples aren’t just noisy, they’re actively harmful signal [§4.2, Fig. 3].
  • Top-k discarding must be annealed: at initialization the critic can’t yet distinguish good from bad samples, so aggressive early filtering is equivalent to using a smaller effective batch size. The paper decays k by factor γ per epoch to a floor k_min [§3.2].
  • The technique transfers as a drop-in: across DCGAN, WGAN with gradient clipping, WGAN-GP, MS-GAN, SN-GAN, and SA-GAN on CIFAR-10, top-k improves FID at fixed baseline hyperparameters [§5, Table 3].
  • Achieves state-of-the-art conditional-CIFAR-10 FID 8.57 on top of ICR-BigGAN (previously 9.21) [§5 BigGAN paragraph, Table 5].
  • Effect persists across batch sizes 64–256 for SAGAN on CIFAR-10, so it isn’t just a batch-size confound [§5 Effect of Batch-Size, Table 6].
  • Applying top-k to the critic in addition to the generator degrades performance [§5 Examining Main Hyper-parameters, Table 7].

Method

Section titled “Method”

Standard GAN generator update: gradient ∇_θ (1/B) Σᵢ L(D(G(zᵢ))). Top-k modification: compute the critic score D(G(zᵢ)) for all B samples, keep only the top-k indices by score, and take the gradient only over those k samples. In PyTorch this is a single mask-then-select line before the loss backward. k starts at B and decays geometrically (γ ≈ 0.99 per epoch on CIFAR-10, slower on ImageNet) to a floor k_min (typically B/2). Critic training is unchanged.

Results

Section titled “Results”
  • CIFAR-10 FID with top-k on top of baselines (from Table 3): DCGAN 38.09 → 35.62, WGAN+GC 37.33 → 34.41, WGAN+GP 31.80 → 29.83, MS-GAN 27.33 → 26.54, SN-GAN 21.36 → 19.80, SA-GAN 19.02 → 17.93.
  • ICR-BigGAN CIFAR-10 conditional: FID 9.21 → 8.57 (SOTA at time of publication, Table 5).
  • ImageNet SAGAN: consistent improvement at batch sizes 64, 128, 256 (Table 6).
  • Toy 25-Gaussians mixture: bottom-k updates move samples away from nearest mode (negative oracle-direction cosine); top-k updates move them toward it. The distributional recovery table shows top-k recovers the target’s per-σ mass much more faithfully than a vanilla GAN, which produces long-tailed mode-smearing [§4.1-4.2, Fig. 3-4, Table 2].

Why it’s interesting

Section titled “Why it’s interesting”

Filed as historical context flagged in Terrance’s thread comparison. The connection to Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignment is worth noting: both papers are “one-line, batch-level sample-selection tricks that materially improve training efficiency of a generative model,” but they touch different levers — Top-k GAN discards low-quality samples from the gradient computation, while Immiscible Diffusion re-assigns which noise pairs with which image but keeps all samples. Both connect to Diffusion training efficiency as members of a family of “batch-level structure matters for gradient quality” recipes that operate on the training loop rather than the model, and both illustrate that trivially-small algorithmic changes (one line of code) can produce substantial gains — reinforcing the recurring theme on the wiki that training-side interventions often beat architectural ones at fixed FLOPs.

See also

Section titled “See also”