Beyond the Noise: Aligning Prompts with Latent Representations in Diffusion Models

TL;DR

NoisyCLIP fine-tunes the CLIP image encoder on noisy intermediate latents from the reverse diffusion process so that text-to-image alignment can be measured during generation rather than only on the final decoded image. The training signal is contrastive InfoNCE over (prompt, noisy-latent) pairs synthesized with SDXL, with the text encoder frozen so its embedding space stays compatible with the diffusion model’s conditioning. Used as a ranker in a Best-of-N (BoN) setting on GenAI-Bench, NoisyCLIP can pick the winning seed at the midpoint of denoising — cutting compute by 50% while retaining 98% of full-CLIP BoN’s VQAScore. The work also ships two open benchmarks (Noisy-Conceptual-Captions, Noisy-GenAI-Bench) with stored per-step latents, which previously did not exist in open form.

Key claims

Off-the-shelf CLIP cannot discriminate aligned from misaligned candidates on intermediate noisy latents: the score gap between best and worst seeds stays below 1% until past latent step 30 of a 50-step SDXL schedule, only reaching ~2.5% at the final decoded image [§5.1, Fig. 3].
NoisyCLIP, fine-tuned on latents from steps 20–29, opens a >3% best-vs-worst score gap as early as latent step 20 and maintains the separation through the rest of the denoising trajectory [§5.1, Fig. 3].
In a best-of-6 BoN setting on GenAI-Bench, selecting at latent step 25 (half the steps) gives VQAScore 0.83 vs the full-generation CLIP baseline’s 0.85 — 98% of the score at 50% of the compute (cost 150 vs 300 denoising steps) [§5.2, Fig. 5].
To match the no-early-stopping NoisyCLIP best-of-6 VQAScore, the CLIP baseline needs at least 20% more total denoising steps [§5.2, Fig. 5c].
At identical mid-denoising selection (latent step 20), CLIP-as-ranker provides essentially no lift over the per-seed average alignment, while NoisyCLIP-as-ranker adds ~1 VQAScore point — the gain is from training on noisy latents, not from the BoN framing itself [§5.2, Fig. 5a].
Training-noise range governs inference-time behavior: training on latent ranges 10–19 wins early but degrades sharply near the final image; training on 40–49 wins late but is weak early; training on 20–29 is Pareto-best across the trajectory [§5.4.1, Fig. 8].
On a 1-correct-vs-4-non-factual caption classification task (Noisy-Conceptual-Captions), NoisyCLIP’s Recall@1 crosses 0.5 by latent step 25 while frozen CLIP plateaus below 0.4 even at the final image [§5.3, Fig. 7].
Only the image encoder is fine-tuned; the text encoder is frozen to share representational space with the diffusion model’s cross-attention conditioning, which the paper argues is what lets the encoder “mimic the cross-attention interaction during denoising” [§3.3].
A linear “latent-RGB” transform $T$ maps noisy latents into a CLIP-consumable RGB-like space, sidestepping the expensive VAE decode at every denoising step [§3.1, Fig. 2].
The released benchmarks (Noisy-Conceptual-Captions: 1000 prompts × 4 images + 4 non-factual prompts each; Noisy-GenAI-Bench: 1600 prompts × 10 images) store full per-step latent sequences — addressing a real gap in open evaluation infrastructure for any “judge during denoising” research [§3.4, §4.2].

Method

NoisyCLIP extends a standard CLIP twin-tower setup so the image encoder operates on latents from any reverse-diffusion step $t$ , not just decoded RGB. A linear transform $T$ converts the SDXL latent $z_t$ into a 3-channel “latent-RGB” tensor that the existing CLIP visual backbone consumes without architectural changes. Training data is synthesized by running SDXL on the rewritten CC12M captions and storing each prompt with its full latent sequence; 50K latents uniformly sampled from intermediate steps form the training set. The image encoder is fine-tuned with InfoNCE — each (prompt, noisy-latent) pair is positive against in-batch negatives — while the text encoder is frozen to preserve compatibility with diffusion-conditioning text embeddings.

At inference, NoisyCLIP is used as a ranker: $N$ seeds are denoised in parallel, scored at some intermediate step $t$ , and all but the top-1 are dropped — the rest of the denoising budget is spent only on the surviving seed. “Cost” is reported as total denoising steps across the live seeds; a best-of-6 that culls at step 25 costs $6 \cdot 25 + 1 \cdot 25 = 175$ steps vs $6 \cdot 50 = 300$ for full-completion BoN.

Results

Headline numbers on Noisy-GenAI-Bench basic prompts (SDXL, 50-step DDIM):

Best-of-6, full completion + CLIP rerank: VQAScore 0.85, cost 300.
Best-of-6, NoisyCLIP cull at step 25: VQAScore 0.83, cost 175 — 98% of the score at 58% of the cost.
Best-of-6, NoisyCLIP cull at step 30: VQAScore exceeds the CLIP baseline’s best-of-4 final-image rerank at matched cost 200.
Recall@1 on factual-vs-non-factual captions at latent step 10: NoisyCLIP > frozen CLIP by 20+ denoising steps’ worth of “wait time” [§5.3].
Ablation: training on noise range 20–29 dominates 10–19, 30–39, and 40–49 across the inference-time latent sweep [§5.4.1, Fig. 8].
Backbone ablation (CLIP-base vs SigLIP-base) in the appendix shows the recipe is encoder-agnostic.

The paper does not directly compare against VLM-as-judge baselines (VQAScore is used as ground truth for ranking, not as a competing ranker) — so the comparison is “fine-tuned CLIP” vs “frozen CLIP”, not “fine-tuned CLIP” vs “a large VLM that consumes decoded RGB at extra cost”.

Why it’s interesting

This is the cheapest credible answer the wiki has filed for online alignment feedback during diffusion sampling. The dominant filed line on inference-time T2I quality has been spend-more-compute approaches — full-completion BoN scored by an external VLM judge (VLM-as-Evaluator, e.g. UnifiedReward-Flex used inside GRPO), best-of-N rejection sampling against a reasoner verifier (Self-Refining Video Sampling vs Cosmos-Reason1-7B in Inference-Time Scaling), or training-time fixes to remove guidance miscalibration entirely (Classifier-Free Guidance variants — SoFlow, Ostris, CAFM). NoisyCLIP slots between them: a small fine-tune of an existing dual encoder that lets you prune the BoN tree early without changing the generator, training objective, or sampler. The closest filed cousin is Dynamic Classifier-Free Diffusion Guidance via Online Feedback — also reads from intermediate latents during generation, also uses CLIP-on-latents — but applies the signal to modulate CFG on the surviving seed rather than to kill losing seeds. The two recipes are orthogonal and stackable; combining them would be the obvious next experiment for any team running BoN-style inference at scale (luma’s video-gen serving stack is one).

The Kyle-asks-about-eval framing is the more interesting handle: the same recipe re-purposed as an eval metric would give per-step prompt-adherence curves rather than a single final-image score. That’s a measurement primitive nobody else in the VLM-as-Evaluator cluster is producing — RISE-Video / IGenBench / UnifiedReward-Flex all judge final outputs only. The honest caveat is that NoisyCLIP is calibrated to SDXL latents on a specific sampler (DDIM, 50 steps); transferring to Wan, Flux.2, or Z-Image would need a re-fine-tune at minimum, possibly a new latent-RGB transform.