Contrastive Flow Matching

TL;DR

Contrastive Flow Matching (ΔFM) adds a single auxiliary term to the standard conditional flow-matching objective: for each (sample, condition, noise) triplet in a batch, pair it with an independently sampled (sample, condition, noise) triplet from the same batch and maximize the L2 distance between their predicted velocity fields. The intuition is that vanilla conditional FM only guarantees flow uniqueness within a class; ΔFM enforces it across classes, fixing the “averaging” failure mode where conditional flows from different classes overlap and produce blurry/ambiguous outputs. On SiT-XL/2 + REPA at ImageNet-256, ΔFM reaches the same FID as the baseline in ~9× fewer training iterations and at 50 NFE matches the baseline’s quality at 250 NFE — a 5× inference speedup. It stacks with REPA and with CFG. Code: github.com/gstoica27/DeltaFM.

Key claims

Vanilla conditional flow matching guarantees flow uniqueness only within a single class condition, not across conditions — overlapping conditional distributions cause flows to collide and produce “average” samples [§3, Eq. 4].
The contrastive term maximizes $\|v_\theta(x_t, c) - (x_1' - x_0')\|_2^2$ between the model’s predicted velocity for one (sample, condition, noise) triplet and the target velocity of an independently drawn triplet from the same batch [§4, Eq. 5].
Total objective is $\mathcal{L}_{\Delta FM} = \mathcal{L}_{FM} - \lambda \mathcal{L}_{contrast}$ with $\lambda = 0.05$ stable across all reported model/dataset settings; ΔFM reduces to FM at $\lambda = 0$ [§4, Eq. 6; §5.5, Table 5].
On ImageNet-256 with SiT-B/2, ΔFM lowers FID from 42.28 → 33.39 (Δ −8.89) and sFID from 11.35 → 5.67 with no architectural change [§5.1, Table 1].
On ImageNet-256 with SiT-XL/2 + REPA, ΔFM lowers FID from 11.14 → 7.29 (Δ −3.85) and IS from 115.83 → 129.89 [§5.2, Table 2].
On CC3M text-to-image with MMDiT + REPA at 400K iterations, ΔFM lowers FID from 24 → 19 on the full validation set [§5.3, Table 3].
ΔFM with CFG further improves SiT-XL/2 + REPA: FID 2.09 → 1.97, sFID 5.55 → 4.49 versus baseline FM + CFG at matched guidance hyperparameters [§5.4, Table 4].
Training-iteration efficiency: ΔFM matches the FM baseline’s FID-50K in up to ~9× fewer iterations [§5.5, Fig. 5 left; abstract].
Sampling-step efficiency: ΔFM at 50 NFE matches FM at 250 NFE on the same model — a 5× reduction in denoising steps for equivalent quality [§5.5, Fig. 5 right; abstract].
Larger contrastive batches help: at SiT-B/2 batch 256→1024 ΔFM FID drops 33.39 → 15.23 while baseline drops 42.28 → 22.00 — the gap widens with batch size, consistent with in-batch-negative contrastive losses generally [§5.5, Table 6].
Closed-form analysis: under the contrastive loss, the optimal velocity decomposes as $v^*_{\Delta FM}(x_t, c) = v^*_{FM}(x_t, c) - \lambda \cdot v^*_{unconditional}(x_t)$ — i.e. ΔFM at training time approximately steers each conditional flow away from the unconditional data-mean trajectory, with $\lambda$ controlling the strength [§5.4 derivation, App. B.1].
This gives a clean account of why ΔFM stacks with CFG: both push the conditional flow away from the unconditional flow, but ΔFM does it implicitly at training time (data-driven mean) while CFG does it explicitly at inference time (model’s unconditional estimate). The paper proposes a CFG-coupling formula that subtracts the ΔFM-implicit term to avoid double-counting [§5.4, App. B.2].
ΔFM produces structurally-coherent images earlier in the denoising trajectory than FM — class-iconic features (animal eyes, train windows, slanted bridges) are visible by step ~15-20 of a 30-step trajectory, vs only at the final step for FM [§5.5, Fig. 4].

Method

The objective is the standard conditional flow-matching loss plus a contrastive regularizer. For each batch element $i$ — a class $c_i$ , a real sample $x^i_1 \sim p(x_1 | c_i)$ , a noise sample $x^i_0 \sim \mathcal{N}(0, I)$ , and a timestep $t$ — the FM target is the ideal velocity $u^i = x^i_1 - x^i_0$ along the linear interpolant $x^i_t = (1-t) x^i_0 + t x^i_1$ . Now sample an independent batch index $j$ — independent class $c_j$ , independent sample $x^j_1$ , independent noise $x^j_0$ , independent timestep — yielding target $u^j$ . The contrastive term maximizes $\|v_\theta(x^i_t, c_i) - u^j\|_2^2$ , i.e. the model’s predicted velocity for example $i$ ‘s condition should be as far as possible from example $j$ ‘s independent target velocity. The total loss subtracts $\lambda$ times this distance from the FM loss.

Algorithmically this is a few extra lines around the existing FM training step — no additional forward passes, no extra data, no extra encoders. Pairing is done by shifting the batch indices by a random offset, so every batch element gets exactly one negative partner. Implementation is publicly released as a drop-in replacement for FM training loops at github.com/gstoica27/DeltaFM.

The closed-form derivation (App. B.1) shows that under the contrastive loss the optimal velocity becomes the FM-optimal velocity minus $\lambda$ times the unconditional data-mean velocity. This is the same functional form as CFG’s conditional-minus-unconditional update — except ΔFM bakes the unconditional-subtraction into training and uses the data distribution’s mean rather than the model’s implicit unconditional generator’s estimate. When both are applied, the paper modifies the CFG update from $v_{cond} + w(v_{cond} - v_{uncond})$ to a coupled form that subtracts ΔFM’s already-applied unconditional steerage to avoid double-counting; this coupling is only applied within a guidance interval $[t_a, t_b]$ and the unmodified ΔFM model is used outside it.

Toy 2D experiments (§4, Fig. 3) on two overlapping Gaussian class-distributions show vanilla FM generating samples in the overlap region (ambiguous class identity) while ΔFM produces clearly class-separated outputs whose flows visually avoid each other — a clean illustration of the mechanism.

Results

Quantitative headlines, all FID-50K on ImageNet-256 unless noted:

SiT-B/2 (base): FM 42.28 → ΔFM 33.39 (sFID 11.35 → 5.67) [§5.1, Table 1].
SiT-XL/2: FM 20.01 → ΔFM 16.32 (sFID 8.45 → 5.08) [§5.1, Table 1].
REPA SiT-B/2: FM 27.33 → ΔFM 20.52 (sFID 11.70 → 5.47) [§5.2, Table 2].
REPA SiT-XL/2: FM 11.14 → ΔFM 7.29 (sFID 8.25 → 4.93, IS 115.83 → 129.89) [§5.2, Table 2].
ImageNet-512 REPA SiT-XL/2: FM 11.32 → ΔFM 7.64 [§5.2, Table 2].
CC3M text-to-image (MMDiT + REPA, 400K iters): FID 24 → 19 on the full validation set [§5.3, Table 3].
REPA SiT-XL/2 + CFG: FM+CFG FID 2.09 → ΔFM+CFG 1.97; sFID 5.55 → 4.49 [§5.4, Table 4].
Batch-size scaling, REPA SiT-B/2: at batch 1024, FM 22.00 → ΔFM 15.23 (ΔFM gap widens from −8.89 at batch 256 to −6.77 at batch 1024) [§5.5, Table 6].
Training efficiency: ΔFM matches FM’s FID-50K at up to 9× fewer iterations [§5.5, Fig. 5 left; abstract].
Sampling efficiency: ΔFM at 50 NFE ≈ FM at 250 NFE — same model, 5× fewer denoising steps for the same FID [§5.5, Fig. 5 right; abstract].
Hyperparameter robustness: $\lambda = 0.05$ is stable across all reported settings; $\lambda$ too low collapses to FM, too high (>0.10) over-separates flows and degrades FID (FID jumps 7.29 → 19.21 at $\lambda = 0.50$ ) [§5.5, Table 5].

The headline efficiency claim — 9× training, 5× inference — is reported at SiT-XL/2 + REPA scale and stacks with both REPA and CFG. The ImageNet-256 + REPA + ΔFM number (7.29 without CFG, 1.97 with CFG) is competitive with the strongest reported numbers on the benchmark at the time of publication.

Why it’s interesting

This is the clearest filed instance of making the conditional flow-matching loss itself fix the problem CFG patches at inference time. The closed-form decomposition (§5.4, App. B.1) shows ΔFM’s training-time update is the same functional form as CFG’s inference-time update — conditional velocity minus a weighted unconditional velocity — which gives a principled account of (a) why ΔFM works at all, (b) why it stacks with CFG (the two unconditional terms are different — data-mean vs model’s implicit), and (c) why naïve coupling needs the offset App. B.2 derives. This makes it a direct theoretical companion to two threads on the wiki: Random noise augmentation during diffusion training eliminates the need for CFG (CFG-as-noise-miscalibration-patch) and SoFlow: Solution Flow Models for One-Step Generative Modeling (CFG folded into the training objective). All three papers diagnose CFG as a training-time-deferrable problem; ΔFM gives the cleanest closed-form account of what the training-time correction is doing.

The training-efficiency angle is also significant. ΔFM is the first filed entry under Diffusion training efficiency that does not use external supervision (DINOv2, V-JEPA, etc.), does not redesign the latent, and does not change parameterization — it adds two lines to the FM loss. That makes it a clean composability test for the page’s “external-DINO vs internal-self-distillation vs latent-design” framing: ΔFM stacks with REPA (the external-DINO recipe), giving 7.29 vs 11.14 at REPA SiT-XL/2, suggesting it’s an orthogonal lever. The natural unanswered question is whether it also stacks with Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis‘s internal-self-distillation EMA-teacher recipe and with Unified Latents (UL): How to train your latents‘s latent redesign.

Two practical caveats worth surfacing: (1) the contrastive loss is a batch-size-sensitive lever in the standard in-batch-negative sense — gains keep widening up to batch 1024 in the paper, so practitioners with small effective batches may see attenuated effects; (2) the closed-form derivation that justifies the CFG-coupling assumes the model exactly factorizes into ΔFM’s unconditional-subtraction form, which “cannot be guaranteed” by training dynamics — the recipe is principled-but-approximate.