Skip to content
Bud's Wiki

Classifier-Free Guidance variants

TL;DR

Section titled “TL;DR”

Classifier-Free Guidance (CFG) — the two-pass conditional/unconditional velocity mix used by virtually every modern diffusion sampler — is increasingly treated as a training-time problem rather than an inference knob. Recent work either (a) folds the guided velocity into the training objective so the inference-time two-pass is unnecessary (SoFlow), (b) decouples guidance into multiple modality-specific scales for joint generation (LTX-2’s bimodal CFG), or (c) argues CFG is patching a noise-distribution miscalibration that can instead be cured by augmenting the training noise (Ostris on Z-Image). The shared headline: a single text-guidance scalar applied at inference is the wrong design point — guidance should be re-distributed across training, modalities, or the noise schedule.

Key claims

Section titled “Key claims”
  • CFG can be incorporated during training via a Flow Matching loss that regresses against a guided (mixed conditional + unconditional) marginal velocity target, eliminating the two-pass overhead at inference — demonstrated for one-step-from-scratch generation (Random noise augmentation during diffusion training eliminates the need for CFG tweet body; SoFlow: Solution Flow Models for One-Step Generative Modeling §3).
  • For joint multi-modal generation, a single CFG scalar conflates text guidance and cross-modal guidance; LTX-2 instead uses per-stream, per-source scales — (w_T, w_M) = (4.0, 0.5) for video and (5.0, 1.0) for audio — with stronger cross-modal guidance improving temporal sync (LTX-2: Efficient Joint Audio-Visual Foundation Model §4.1).
  • For triplet-modality generation (text + image + audio), HuMo decomposes CFG into per-modality scales (w_T, w_I, w_A) with null-token substitution for missing modalities, then schedules between two configurations across the denoising trajectory (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning §3.4, Eq. 4).
  • CFG can be interpreted as a runtime patch for a training-time miscalibration: the denoiser has only seen perfectly normalized N(0, I) noise and so cannot correct off-distribution intermediate states; CFG amplifies the model’s correction signal at the cost of oversaturation (the “AI look”) (Random noise augmentation during diffusion training eliminates the need for CFG tweet body).
  • A per-channel additive + log-normal augmentation of the training noise — ε' = (ε + U(-0.2, 0.2)) · exp(N(0, 1)) — reportedly cures the miscalibration directly, so the resulting model needs no CFG at inference and avoids the oversaturated look (Random noise augmentation during diffusion training eliminates the need for CFG tweet body; Slack note).
  • Eliminating inference-time CFG roughly halves per-step compute (one forward pass instead of two) and removes the guidance-scale hyperparameter — important for distillation and few-step sampling, which both interact awkwardly with CFG (SoFlow: Solution Flow Models for One-Step Generative Modeling §3; Random noise augmentation during diffusion training eliminates the need for CFG).
  • The CFG mix can be redefined over a non-text conditioning channel: SparkVSR’s Reference-Free Guidance linearly combines a “keyframe-reference” branch and a “zero-latent (blind SR)” branch with the standard CFG formula, giving the user a runtime knob to trade external prior strength against the model’s internal prior. Trained via reference-dropout (p=0.5) so the same network parameters serve both branches (SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation §3.3, Eq. 3).
  • CFG also patches generalization miscalibration, not just noise miscalibration: replacing flow matching’s MSE criterion with a learned discriminator (adversarial post-training, Continuous Adversarial Flow Models) drops guidance-free ImageNet-256 FID on SiT-XL/2 from 8.26 → 3.63 and on JiT-H/16 from 7.17 → 3.57, and the post-trained model reaches its best FID at a lower CFG scale than the baseline (SiT optimal CFG drops 1.5 → 1.3). On Z-Image T2I, the no-CFG GenEval lift is 0.60 → 0.71 (+0.11) while the with-CFG lift is 0.81 → 0.85 (+0.04) — i.e. fixing generalization recovers most of CFG’s gain (Continuous Adversarial Flow Models Tables 2, 5; §1).
  • In few-step regimes the positive and negative branches diverge after one inference step, so output-space CFG extrapolation is structurally ill-posed and produces artifacts rather than negative control (Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models §4, Fig. 3 left). The fix is to move extrapolation into attention feature space: Z̃ = Z+ + s·(Z+ − Z−), then L1-norm-clip against Z+ and α-blend back toward Z+. The L1 (vs L2/max) clip preserves low-magnitude activations that linear projections downstream amplify (Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models §4.1, Eqs. 7–10).
  • Attention-space negative guidance is markedly cheaper than CFG because only attention blocks see extra work, not the whole forward pass: per-step latency overhead is +13% (SANA) / +12% (Wan2.1) / +43% (SD3.5-Large) vs CFG’s +90% / +100% / +95% — a near-9× compute saving on Wan2.1 at matched negative-prompting capability (Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models Table 4).
  • Attention-space guidance composes additively with CFG and PAG when CFG works (multi-step models): on 25-step SDXL + CFG, NAG adds +0.8 CLIP / −2.35 FID / +0.115 ImageReward; on 25-step SDXL + CFG+PAG, +0.8 CLIP / −2.72 FID / +0.166 ImageReward (Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models Table 2). The earlier NASA attempt at attention-space guidance lacked normalization/refinement and is unstable on DiTs at any guidance scale (Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models §5.2, Fig. 2).
  • The content of the conditional input itself can be scheduled across timesteps: ALG low-pass filters the I2V conditioning image at early denoising steps and exposes the original image at later steps, with an asymmetric CFG mix (Eq. 3) that uses the filtered image in the conditional + one unconditional term while keeping the original in the other unconditional term. Algebraically this decomposes into standard CFG over the filtered image (which restores T2V-level motion) plus a high-frequency restoration term (which preserves image fidelity) (Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance §3.2, Eq. 3). Diagnostic motivation: I2V denoisers “lock onto” reference-image fine details within the first denoising step, confining the sampling trajectory before motion can develop (Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance §3.1, Fig. 2). Yields +36% average Dynamic Degree across CogVideoX, Wan 2.1, HunyuanVideo, LTX-Video on VBench-I2V with τ=0.06 (filter active for first 6% of timesteps) and a bilinear down-up filter at factor 2.5 (Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance Table 2).
  • Richer conditioning input itself reduces required CFG scale: DLC conditions a DiT on a length-L sequence of discrete SSL-derived tokens (DINOv2 → Simplicial Embeddings → argmax) rather than class labels or single CLIP/DINOv2 embeddings. The longer, more informative conditional makes the conditional branch already close to the target distribution, lowering the marginal benefit of extrapolating against the unconditional — empirically reaching SOTA unconditional ImageNet-256 FID 1.59 with reduced reliance on CFG (Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models Table 1, Fig. 5).
  • Per-timestep guidance-scale switching. HuMo observes that early denoising steps build semantic structure / spatial layout (text-dominant) while later steps refine identity and audio-visual sync (audio/image-dominant), so a static CFG configuration is suboptimal across the trajectory. The fix: define two complete (w_T, w_I, w_A) configurations and switch between them at a hard cutoff — text/image-dominant from timestep 1.0 → 0.98, then audio/image-dominant from 0.98 → 0. The schedule is orthogonal to ALG’s per-timestep content schedule (which keeps the formula fixed and varies what enters the conditional) — here the formula’s scales vary while the inputs are fixed (HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning §3.4).

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline): Learning to Fold operationalizes RECAP-style advantage conditioning + CFG at inference on a flow-matching VLA and demonstrates per-garment-type CFG scale tuning via a Thompson-sampling bandit during rollout collection — a runtime optimization of the guidance scalar over a discrete condition space (garment type ∈ {long top, short top, long pants, shorts}) rather than a global hyperparameter. The advantage token sits in its own attention group invisible to state/images/garment-type and is stochastically masked during training (25→10% unconditional-dropout rate ramp); the same gate drives AdaRMS advantage vectors into every action-expert layer, so the guidance signal reaches the expert through both attention and layer-norm modulation simultaneously. Data point for the RECAP-as-CFG-generalization branch already seeded by π*0.6 and Evo-RL, with the tightest reproduction of the recipe on a competition benchmark filed so far.
  • [2026-07-10] Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio: TR-Adaptive Guidance modulates the compositional video-conditioning strength of a flow-matching action head at each replan step based on a runtime internal-state signal (Temporal Ratio, the ratio of action-token attention on future vs current video frames), rather than a fixed schedule or a text-derived signal. Amplifies conditioning when the policy is future-reliant (planning phase) and reduces it when the policy is anchored to the current frame (precise manipulation), improving compositional OOD success on LIBERO and real-world YAM bimanual tasks while preserving ID precision — a VAM/robotics-side counterpart to the dynamic-CFG literature that reads guidance strength off of internal statistics rather than external timers.
  • [2026-07-10] Emerging Properties in Unified Multimodal Pretraining: BAGEL’s interleaved-inference CFG drops three modalities independently during training — text at 0.1, ViT tokens at 0.5, and clean-VAE tokens at 0.1 — enabling multi-axis classifier-free guidance across text/image conditioning at inference. This is the training-side recipe that Hyper-Bagel later folds into a single forward pass via FLUX-style timestep embeddings.
  • [2026-06-29] Qwen-Image-2.0-RL Technical Report: Qwen-Image-2.0-RL introduces a hybrid CFG strategy specifically as an RL-time stabilizer — the role of guidance shifts from inference-time quality knob to a training-time mechanism for preserving pretrained knowledge during GRPO post-training. New axis for this cluster: CFG as a regularizer against RL-driven distribution drift, distinct from the inference / distillation / generalization-miscalibration framings already on this page.
  • [2026-06-26] Evo-RL: Open Real-World Offline RL on SO-101 and AgileX PiPER: Evo-RL operationalizes CFGRL-style advantage conditioning as a CLI flag (--acp.indicator_dropout_prob=0.3 defaulting to a 30% unconditional-dropout rate) for VLA policy extraction on commodity SO-101 / AgileX PiPER hardware — first open-source implementation of the π*0.6: a VLA That Learns From Experience (RECAP) recipe outside Physical Intelligence.
  • [2026-06-25] InfiniteDiffusion: Bridging Learned Fidelity and Procedural Utility for Open-World Terrain Generation: InfiniteDiffusion is a sampler-level (not CFG-level) intervention in the same family as Z-Sampling and Time-to-Move: a training-free generalization of MultiDiffusion that runs overlapping tiles with order-invariant averaging from a bounded LRU cache, converting bounded diffusion into an unbounded, deterministic, embarrassingly parallel sampler.
  • [2026-06-24] Is Noise Conditioning Necessary for Denoising Generative Models?: Sun et al. analyze a different but adjacent “always-on diffusion input” removal: dropping the timestep/noise-level conditioning entirely. Their effective-target framework (per-step error governed by the posterior variance of p(t|x_t)) is structurally similar to CFG-flexibility analyses and helps explain why FM-style v-prediction is robust to modulation-pathway changes.
  • [2026-06-23] i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models: i1’s design-space sweep includes CFG schedule choices as part of its inference ablations, providing a 3B-scale fully open reference for which CFG variants pay off at T2I scale.
  • [2026-06-10] Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning: QGF (Q-Guided Flow) extends test-time guidance from text/image scores to RL value functions: at each denoising step of a flow control policy, the velocity prediction is nudged by ∇_a V(s), with the Jacobian of the denoising chain approximated by identity. Structurally CFG-shaped (steer the sampling trajectory at inference toward a different distribution), but the steering signal is a learned critic rather than an unconditioned vs. conditioned score difference — and the goal is policy improvement on offline-RL benchmarks rather than text-image alignment.
  • [2026-06-09] LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories: LeapAlign is a direct-gradient (non-RL) flow-matching post-training recipe that keeps standard CFG (scale 3.5) at sampling time during fine-tuning — unlike FDFO and CAFM, it does not claim or test removal of CFG at inference. Relevant here as another data point in the post-training/CFG interaction space: alignment gains can be obtained without changing the guidance recipe at all, by instead reaching early generation steps via a two-step leap trajectory with α-discounted nested gradients.
  • [2026-06-05] Ideogram 4.0 Technical Details: Open model at the forefront of design: Ideogram 4.0 uses asymmetric classifier-free guidance — the unconditional pass drops text tokens entirely instead of swapping in a learned null embedding. A simple architectural variant of standard CFG worth comparing against the learned-null-embedding default.
  • [2026-06-02] PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence: PoseAnything uses CFG anchors as a control-channel separator: positive anchor carries the subject-pose condition, negative anchor carries an opposite-direction camera-motion condition (a “leftward rectangle” negative steers background flow rightward, i.e. a leftward camera pan). Enables a model trained only on subject-pose data to obey camera-trajectory instructions at inference, without an explicit camera-control training signal.
  • [2026-06-03] Bootstrap Your Generator: Unpaired Visual Editing with Flow Matching: BYG (NVIDIA, ICML 2026): unpaired visual editing built on top of a flow-matching generator’s existing guidance/velocity structure — claims six-style zero-shot transfer beating paired editors (FLUX-Kontext, Qwen-Image-Edit) without any paired supervision on the target styles.
  • [2026-06-01] RenderFlow: Single-Step Neural Rendering via Flow Matching: RenderFlow introduces “sparse keyframe guidance” — conditioning a neural renderer on a small subset of fully path-traced reference frames as a guidance signal, alongside dense G-buffer conditioning. A new guidance shape: the auxiliary signal is a sparse set of physically-correct samples from the same scene, not a paired unconditional pass or a negative prompt.
  • [2026-05-25] SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers: SEGA extends the “dynamic per-sample, per-step inference-time knob” pattern (also seen in Dynamic CFG and VLM-Guided Adaptive Negative Prompting) to RoPE-component attention temperature, driven by the latent’s spatial spectral energy rather than guidance strength.
  • [2026-05-25] HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning (HuMo): adds the ninth axis — time-adaptive guidance-scale switching in the triplet-modality setting. Decomposes CFG into per-modality scales (w_T, w_I, w_A) per Eq. 4, then schedules between two configurations across denoising. The split point (timestep 0.98 → 0) is empirical: 2% of trajectory for semantic layout, 98% for fine detail. Combines with LTX-2’s per-stream-scale framing (both decompose CFG by modality) but adds the temporal-schedule dimension LTX-2 lacks. Orthogonal to ALG: both schedule something across timesteps, but ALG schedules the content of the conditioning input (filtered → unfiltered) while HuMo schedules the scales of the guidance formula.
  • [2026-05-24] Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (DLC, Lavoie et al., July 2025): adds an eighth axis — richer conditioning input — to this cluster. Discretizes DINOv2-L features via Simplicial Embeddings into a length-L token sequence (L=512, V=256), conditions DiT-XL/2 on the DLC, and models p(c) with a small autoregressive prior. SOTA unconditional ImageNet-256 FID 1.59 with explicitly reduced CFG reliance. Distinct from the existing axes because it neither retrains, redesigns guidance, nor reshapes the unconditional branch — it just makes the conditional branch carry more information, which empirically substitutes for guidance scale. Compositional bonus: two DLCs can be mixed at inference for OOD class compositions (jellyfish + mushroom).
  • [2026-05-24] Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance (ALG, Choi et al., June 2025; CVPR 2026): training-free inference-time fix for the I2V motion-suppression problem. Diagnoses that I2V models fine-tuned from T2V backbones lose 17-19% Dynamic Degree because the denoiser locks onto the reference image’s high-frequency details within step 1 of denoising, biasing the trajectory toward a static shortcut. Schedules the content of the conditioning image via a low-pass filter strength κ(t) that is high early and decays to 0; uses an asymmetric CFG mix where only the conditional and one unconditional term consume the filtered image. The asymmetry decomposes algebraically into (a) CFG over the filtered image, restoring motion, plus (b) a high-frequency restoration term, preserving fidelity. Adds a new axis to this cluster: where SparkVSR redefines the unconditional branch and NAG moves the extrapolation into attention space, ALG schedules the conditional branch’s content across timesteps in the frequency domain.
  • [2026-05-24] Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models (NAG, Chen et al., May 2025): training-free, inference-time negative guidance that moves CFG’s extrapolation from noise-prediction space into attention feature space, then stabilizes the result with L1-norm clipping and α-blending against the positive baseline. Restores negative prompting in 1–8-NFE distilled models (Flux-Schnell, SD3.5-Large-Turbo, SANA-Sprint, DMD2-SDXL, NitroSD-Realism, SDXL-Lightning) where CFG collapses; stacks additively with CFG and PAG on multi-step models. Latency overhead 10–50% of CFG’s on most stacks (Wan2.1 +12% vs CFG +100%); transfers to video on Wan2.1-T2V-14B without modification. The first inference-only, no-retraining answer filed in this cluster — complementary to SoFlow / Ostris / CAFM, which all attack the same problem at training time.
  • [2026-05-24] Continuous Adversarial Flow Models (CAFM): adversarial post-training of flow models substantially reduces the guidance scale needed and lifts the no-CFG numbers more than the with-CFG numbers, suggesting CFG and generalization quality are interchangeable knobs. SiT-XL/2 best-FID CFG drops 1.5 → 1.3; Z-Image GenEval no-CFG lift (+0.11) exceeds with-CFG lift (+0.04). Adds a “generalization-miscalibration” axis to the Ostris “noise-miscalibration” framing.
  • [2026-05-23] SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation: Reference-Free Guidance — the CFG formula applied over a keyframe-reference vs. zero-latent branch pair (not text-conditional vs. unconditional). Trained with reference-dropout p=0.5 plus an explicit zero-latent image branch in Stage 2. Lets the user tune adherence to potentially-imperfect external keyframe priors at inference. Demonstrates that the “unconditional” side of CFG is a free design slot — anything trained jointly via dropout can fill it, not just “no text”.
  • [2026-05-23] Random noise augmentation during diffusion training eliminates the need for CFG: argues CFG is a patch for a training-time noise-distribution miscalibration; demonstrates that per-channel additive + log-normal noise augmentation during training removes the need for CFG at inference (Z-Image LoRA, qualitative).
  • [2026-05-22] LTX-2: Efficient Joint Audio-Visual Foundation Model: bimodal CFG for joint audio-video — independent text and cross-modal guidance scales per stream.
  • [2026-05-22] SoFlow: Solution Flow Models for One-Step Generative Modeling: folds CFG into the Flow Matching training objective for one-step-from-scratch generation; no two-pass at inference.

Open questions

Section titled “Open questions”
  • Does Ostris’s noise-augmentation recipe reproduce on a video diffusion stack, on a different base model (not Z-Image), and with controlled quantitative metrics (FID, prompt-adherence benchmarks)? The mechanism is plausible but the evidence is currently one qualitative thread.
  • If CFG is “a patch for noise miscalibration,” do the SoFlow training-time-CFG recipe and the noise-augmentation recipe achieve the same effect by different routes? If so, do they stack?
  • For joint-modality models (LTX-2-style, HuMo-style), can the per-stream guidance scales be learned per-sample or per-timestep rather than chosen as global hyperparameters? HuMo hard-codes the schedule at timestep 0.98; a learned cutoff or a smooth interpolation between configurations is the obvious next step.
  • How do CFG-free models behave under distillation to one- and few-step samplers? CFG is a known pain point for distillation (each guidance scale → a different distilled model); eliminating CFG should simplify the distillation target.
  • What is the right metric for “AI look”? Saturation/contrast statistics are coarse; a quantitative benchmark that captures the qualitative artifact would let CFG-replacement claims be measured rather than asserted.
  • Reference-Free Guidance generalizes the “unconditional branch” slot: what other branches can be plugged in? E.g. negative-style conditioning, identity-vs-no-identity, audio-vs-no-audio. SparkVSR is the first filed concrete instance; HuMo demonstrates per-modality decomposition naturally enables exactly these (image-vs-no-image, audio-vs-no-audio via null-token substitution).
  • Noise-miscalibration (Ostris) vs generalization-miscalibration (CAFM) vs both: are these the same underlying problem (the model has been trained against the wrong implicit metric on the data manifold) seen from different angles, or are they additive? CAFM’s adversarial criterion in principle subsumes a learned noise model; Ostris’s noise augmentation in principle increases coverage of the data manifold. Composability is untested.
  • Does the NAG attention-space patch compose with the training-side fixes (SoFlow / Ostris / CAFM)? In principle yes — NAG only touches attention at inference and is orthogonal to the training objective — but no one has tried (SoFlow’s whole point is no negative branch is needed at inference; NAG requires one). On a CAFM-post-trained student that still uses CFG at lower scale, NAG should compose trivially.
  • Is the L1-clip + α-blend stabilizer specific to feature-space extrapolation, or does an analogous regularizer recover CFG in few-step regimes if applied at the noise-prediction level? The paper assumes feature space is necessary; the implicit ablation against “stabilized output-space CFG” is missing.
  • ALG’s diagnostic — that I2V conditioning collapses the trajectory within step 1 of denoising — predicts that other “static shortcut” problems (e.g. ID-conditioned video, last-frame-conditioned interpolation, depth-conditioned generation) should respond to the same frequency-scheduling trick. Untested.
  • Does ALG’s frequency-schedule compose with HuMo’s time-adaptive scale switch and NAG’s attention-space negative guidance? All three are inference-only and operate on disjoint mechanisms (input content vs guidance scales vs attention features); stacking is plausible but unmeasured.
  • Is the asymmetric CFG mix in ALG (filtered image in conditional + one unconditional, original in the other) a general design pattern, or specific to the high-frequency-restoration use case? An analogous “asymmetric branch” mix appears nowhere else in the cluster — worth pulling on.
  • DLC’s “richer conditioning reduces CFG reliance” claim suggests an interaction with the bitrate-of-conditioning. Is there a clean scaling law: required-CFG-scale ↓ as conditioning-bitrate ↑? If so, DLC’s L=512 / V=256 design exposes a tunable knob that all the other recipes lack.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2024-12

Section titled “2024-12”

Unknown date

Section titled “Unknown date”