Dynamic Classifier-Free Diffusion Guidance via Online Feedback

TL;DR

Replaces the standard static CFG scale with a per-timestep, per-prompt, per-sample CFG schedule chosen online by a greedy search over candidate scales against a suite of small latent-space evaluators (CLIP for alignment, a discriminator for fidelity, a Bradley-Terry reward model, plus capability-specific evaluators for text rendering and numerical reasoning). Because the conditional and unconditional model passes already exist, testing K candidate CFG values per step adds only ~1% FLOPs vs ~400% for a pixel-space equivalent. Demonstrated on a StableDiffusion-equivalent LDM and on Imagen 3, reaching up to 53.8% human-preference win-rate over default Imagen 3 (55.5% on text-rendering prompts) with no model retraining.

Key claims

Static CFG is “one-size-fits-all” and fails to adapt to per-prompt requirements: complex compositions need strong alignment guidance, artistic prompts benefit from low guidance, and text rendering responds poorly to standard scales [§1].
Latent-space evaluators recover most of the signal of pixel-space equivalents: filtering by latent CLIP at 25% of NFEs already discards poorly-aligned LDM samples, and the gap to pixel-space CLIP is small (LDM Gecko 43.0 vs 45.1 at 100% filtering) [§5.1, Table 1].
Greedy per-step search reuses the existing conditional + unconditional predictions and only re-combines them with K candidate scales, so K trials cost ~1% extra FLOPs vs ~400% for pixel-space rejection [§3.3, Appendix A.3].
On LDM with Gecko + MS-COCO, alignment+VQ adaptive weighting reaches 47.2 Gecko / 24.8 FID, dominating gradient guidance (46.1 / 25.6), Limited Interval (43.0 / 26.1), and Annealing (47.0 / 28.9) — improving both alignment and fidelity simultaneously where prior methods trade one for the other [Table 2].
On Imagen 3, alignment+reward adaptive weighting wins 53.6% / 53.8% / 54.7% / 53.6% against the default-CFG baseline on Gecko / GenAI-Bench / MARIO-eval / GeckoNum; adding text-rendering or numerical-reasoning evaluators pushes capability-specific wins to 55.5% / 54.1% [Table 3].
Adaptive evaluator weighting (time-dependent, triggered by score-change detection) is critical: static linear weighting underperforms across the board because coarse alignment matters early while text/artifact concerns dominate late [§3.3, §5.2].
Heuristic CFG schedules don’t transfer between model families: Limited Interval (Kynkäänniemi et al.) and Annealing (Sadat et al.) give modest LDM gains but degrade Imagen 3 below baseline on most prompt sets, with Limited Interval failing completely on MARIO-eval text rendering (19.6% win-rate) [Table 3].
The discovered median CFG schedules on LDM and Imagen 3 are different shapes, but in both cases the alignment evaluator alone pushes to high CFG and the visual-quality / reward evaluator alone pushes to low CFG — adaptive weighting navigates the alignment-vs-fidelity Pareto explicitly per sample [§5.3, Fig. 2].
The latent evaluators are constructed by replacing the pre-trained CLIP image-tower embedding layer with a randomly-initialized one matching the diffusion encoder’s latent dimensionality and fine-tuning with time-conditioning on noised latents — they operate directly on noisy intermediate states, not on decoded images [§3.2].
The reward evaluator is a Bradley-Terry fine-tune of the latent CLIP on human preference pairs; the text-rendering evaluator is an MSE-supervised fine-tune against OCR-model scores; the numerical-reasoning evaluator is a contrastive fine-tune on WebLI-100B filtered for countable entities [§3.2, Eqs. 3–4].

Method

A diffusion model is trained as usual to support both conditional and unconditional sampling. At inference, each denoising step computes both the conditional and unconditional score estimates exactly once, then evaluates K candidate CFG scales by simply re-combining those two estimates — no extra denoiser forward passes. For each candidate scale w_k, the resulting predicted latent x̂₀(w_k) is scored by an ensemble of latent-space evaluators (alignment / visual quality / reward / capability-specific). Evaluator scores are combined with a time-dependent adaptive weighting that detects when each evaluator’s signal becomes meaningful (significant cross-timestep score change) and amplifies its contribution accordingly. The argmax candidate scale is applied for that step; the schedule is recomputed for the next step. Total inference overhead is ~1% FLOPs vs vanilla CFG because (a) only the K-fold linear re-combination is repeated per step, and (b) the evaluators are small networks operating in the diffusion latent space rather than on decoded pixels.

The latent evaluators are themselves built by retrofitting a pre-trained CLIP onto noisy diffusion latents: the original embedding layer is replaced to match latent dimensionality, the encoder is made time-conditional, and fine-tuning uses the standard CLIP contrastive objective on (image latent + noise, caption) pairs. The discriminator (visual quality) reuses this latent CLIP vision encoder with a classification head trained on MSCOCO real vs generated. The reward evaluator further fine-tunes the latent CLIP with the Bradley-Terry loss on human preference pairs. Capability-specific evaluators (text rendering, numerical reasoning) are MSE-against-OCR or contrastive-on-filtered-data fine-tunes of the same base.

Results

LDM (StableDiffusion-equivalent), Gecko prompts: Alignment+VQ adaptive weighting: 47.2 Gecko score / 24.8 FID, vs Default CFG 43.8 / 25.6, Annealing 47.0 / 28.9, Limited Interval 43.0 / 26.1, Gradient guidance (alignment+VQ) 45.3 / 25.5. Adaptive weighting is the only schedule that improves both axes simultaneously [Table 2].
Imagen 3, Gecko prompts: Alignment+Reward adaptive: 53.6% win-rate over default Imagen 3. Alignment alone: 50.9%. Reward alone: 52.1%. Annealing schedule (which helped LDM): only 46.4% [Table 3].
Imagen 3, GenAI-Bench: Alignment+Reward: 53.8% win-rate; Limited Interval: 33.1% (loses to baseline) [Table 3].
Imagen 3, MARIO-eval (text rendering): Text-rendering + Alignment + Reward adaptive: 55.5% win-rate; Limited Interval: 19.6% (catastrophic) [Table 3].
Imagen 3, GeckoNum (counting): Numerical + Alignment + Reward adaptive: 54.1% win-rate; Annealing: 50.8% [Table 3].
Compute overhead: ~1% FLOPs increase vs ~400% for pixel-space equivalent evaluation; NFEs unchanged [§3.3, Appendix A.3].
Schedule shape: Median dynamic CFG schedules on LDM follow an arc avoiding extremes at both ends (matches Wang et al. 2024 heuristic); Imagen 3 schedules differ substantially in shape from LDM’s, confirming that per-model heuristics don’t transfer [§5.3, Fig. 2].

Why it’s interesting

This is the first paper in the Classifier-Free Guidance variants cluster to attack the CFG-scale-choice problem as a per-prompt, per-step online search rather than as a static-vs-dynamic schedule design problem — orthogonal to the training-side fixes (SoFlow: Solution Flow Models for One-Step Generative Modeling folding CFG into the loss; Random noise augmentation during diffusion training eliminates the need for CFG noise augmentation; Continuous Adversarial Flow Models adversarial post-training) and to the attention-space patches (Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models). It complements Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance (ALG): both schedule something per-timestep, but ALG fixes the schedule shape and varies the content entering the conditional, while this paper varies the guidance scale via online search. The “schedules don’t transfer between model families” finding (Limited Interval helps LDM but breaks Imagen 3) is direct evidence for one of the open questions on the CFG-variants concept page: prior heuristics are brittle, and the right answer is model-agnostic online adaptation. The latent-evaluator construction also connects to VLM-as-Evaluator: small, fast judges operating in latent space, not on decoded images, are the practical primitive that makes inference-time search affordable.