Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models

TL;DR

NAG is a training-free, inference-time mechanism that restores negative prompting in few-step diffusion models where CFG breaks. Instead of extrapolating between conditional and unconditional noise predictions (which diverge after one step in 1–8 NFE samplers), NAG extrapolates between attention features under positive and negative prompts, then constrains the result by L1-norm clipping and α-blending against the positive baseline. It works as a drop-in across UNet and DiT, image and video, few-step and multi-step, and consistently improves CLIP / FID / PFID / ImageReward with much lower extra latency than CFG on most architectures. Project page also hosts a ComfyUI node, which is what has driven the recent OSS interest.

Key claims

In few-step regimes (1–8 NFE), the positive and negative noise predictions diverge after one inference step, making CFG’s extrapolation ill-posed and producing artifacts rather than negative control [§4, Fig. 3 left]. This is the structural reason CFG-based negative prompting is broken on distilled models.
NAG applies extrapolation in attention feature space — Z̃ = Z+ + s·(Z+ − Z−) — then (a) L1-norm-ratio clips the extrapolated features against the positive baseline to bound magnitude, and (b) α-blends back toward Z+ to keep the result on the feature manifold [§4.1, Eqs. 7–10].
L1 (rather than L2 or max) normalization is chosen specifically because low-magnitude activations encode subtle semantics that downstream linear projections amplify; L2/max disproportionately shrink them and lose precise guidance [§4.1].
On 7 few-step models spanning DiT (SANA-Sprint, Flux-Schnell, SD3.5-Large-Turbo, Flux-Dev) and UNet (NitroSD-Realism, DMD2-SDXL, SDXL-Lightning), NAG improves CLIP by +0.4 to +0.7, lowers FID by 0.2–2.9, lowers PFID by 1.5–4.3, and raises ImageReward by 0.07–0.17 [Table 1].
NAG stacks with CFG and with PAG on multi-step models: on 25-step SD3.5-Large and SDXL with CFG and CFG+PAG, CLIP and ImageReward improve in all four configurations (e.g. SDXL CFG: +0.8 CLIP, +0.115 ImageReward; SDXL CFG+PAG: +0.8 CLIP, +0.166 ImageReward) [Table 2].
The closest prior approach, NASA, also modifies cross-attention but without normalization/refinement; NASA is unstable on DiTs (e.g. Flux-Schnell) at any guidance scale and degrades on UNets as scale grows [§5.2, Fig. 2].
Compute overhead is much lower than CFG on most stacks because only cross-attention / MM-DiT blocks see extra work: SD3.5-Large per-step latency add is +43% (NAG) vs +95% (CFG); SANA +13% vs +90%; Wan2.1 +12% vs +100%. On Flux the overhead is similar to CFG (+87% vs +100%) [Table 4].
NAG transfers to video on Wan2.1-T2V-14B without modification, both suppressing unwanted attributes (facial features) and increasing motion (negative “static”) — and user-study preference is +20.5% text, +8.7% visual, +14.3% motion vs same model without NAG [§5.4, Table 3].
Ablations confirm both stabilizers are load-bearing: removing refinement causes distortion at s≥10 and failure at s≥15; removing normalization causes artifacts that intensify with scale [§5.6, Fig. 8].

Method

NAG inserts itself inside each attention block (cross-attention for UNets, MM-DiT for SD3/Flux/Wan). For every block:

Two attention passes. Compute the standard attention output Z+ using the positive prompt’s keys/values, and a second Z− using the negative prompt’s KV.
Extrapolation in feature space. Form Z̃ = Z+ + s·(Z+ − Z−) with guidance scale s [Eq. 7]. This is the same recipe as CFG but applied to features, not noise predictions.
L1 normalization with clipping. Compute the per-token L1 ratio r = ‖Z̃‖₁ / ‖Z+‖₁; if r > τ rescale Z̃ by τ/r to bound magnitude while preserving direction [Eqs. 8–9].
Refinement. Output Z = α·Z̃_norm + (1 − α)·Z+, pulling the guided feature toward the on-manifold positive baseline [Eq. 10].

Because step 1 doubles attention work but the rest of the network only runs once, the marginal cost is mostly the second attention pass plus a few elementwise ops — versus CFG, which doubles the entire model forward pass. This is why on MM-DiT models (where attention is one of several heavy blocks) NAG’s latency overhead is ~40–90% lower than CFG’s.

The geometric picture (Fig. 4 in the paper): raw extrapolation Z̃₀ can leave the feature manifold; L1-clip produces Z̃₁ bounded inside a “guidance boundary”; α-blend produces final Z inside a smaller “refinement manifold” close to Z+. The two regularizers together keep negative guidance directionally faithful without out-of-manifold drift.

NAG is training-free — it ships as a wrapper around attention modules in existing pipelines, with ComfyUI nodes for Flux-Schnell, Flux-Dev, Flux Kontext, SD3.5, SDXL, SANA-Sprint, NitroSD, Wan2.1, LTX-Video, Hunyuan Video, and HiDream as of the GitHub README.

Results

Headline numbers on COCO-5K with universal negative prompt “Low resolution, blurry”:

Few-step DiTs. Flux-Schnell @ 4 steps: CLIP 31.4 → 32.0 (+0.6); FID 25.47 → 24.46 (−1.01); PFID 38.26 → 34.95 (−3.31); ImageReward 1.029 → 1.099 (+0.070) [Table 1].
Few-step UNets. NitroSD-Realism @ 1 step: CLIP 31.8 → 32.4 (+0.6); FID 26.21 → 23.98 (−2.23); ImageReward 0.847 → 0.948 (+0.101). SDXL-Lightning @ 8 steps: ImageReward 0.730 → 0.842 (+0.112) [Table 1].
Multi-step + CFG (no replacement, addition). SD3.5-Large @ 25 steps with CFG: CLIP +0.2, FID +0.35, ImageReward +0.101. SDXL @ 25 steps with CFG: CLIP +0.8, FID −2.35, ImageReward +0.115. NAG strictly helps even when CFG already works [Table 2].
Compute. On Wan2.1 a single denoising step is 10.7s baseline; CFG adds +10.7s (+100%) and NAG adds +1.3s (+12%) — a near-9× compute saving at matched negative-prompting capability [Table 4]. Flux is the unfavorable case (NAG +87% vs CFG +100%) because attention dominates Flux’s per-step cost.
User study. Preference scores (Lin et al. formula, range −100% to +100%) for NAG vs no-NAG: Flux-Schnell +25.0% text / +33.9% visual; SD3.5-Large +9.2% / +15.5%; Wan2.1-14B +20.5% text / +8.7% visual / +14.3% motion [Table 3].

The single most relevant headline for Luma’s distill work: on 4-step Flux-Schnell, NAG gets full negative-prompt control with ~the same latency as CFG would have added (Flux is the worst case), while on Wan2.1 the speed advantage is dramatic.

Why it’s interesting

This is the cleanest concrete answer the wiki has filed to the question “what replaces CFG when you’ve distilled the model down to 1–4 NFE.” Classifier-Free Guidance variants currently catalogs three other answers: SoFlow folds guidance into training (SoFlow: Solution Flow Models for One-Step Generative Modeling), Ostris’s recipe replaces CFG with training-noise augmentation (Random noise augmentation during diffusion training eliminates the need for CFG), and CAFM argues better generalization shrinks the needed CFG scale (Continuous Adversarial Flow Models). NAG is the inference-only, no-retraining answer in that family — and the only one of the four that ships as a plug-in to existing distilled checkpoints, which is why ComfyUI uptake has been immediate.

Sibling-relationship to Diffusion Distillation: every paper in that cluster (DMD2, ArcFlow, TMD, Causal Forcing, RMD, DiagDistill, …) inherits “negative prompting is broken at low NFE” as a downstream pain point. NAG is orthogonal to the distillation objective and the trajectory parameterization — it operates at sampling time on whatever student the distillation produced — so in principle it composes with all of them. The Slack note’s empirical claim (“works really well, way better than just turning up CFG on a distill model”) is the field-observation analog of Table 1 above.

Caveats worth flagging: the paper’s only comparison to other attention-space methods is NASA, the headline metric set leans on CLIP/FID/ImageReward (text alignment + IR-proxy aesthetic preference), and the user studies are small. Two comparisons are absent: against simply increasing CFG scale on a CFG-distilled model (the baseline Nathan called out), and against the SoFlow / Ostris / CAFM training-side answers in the same concept page.