FreeFlow: Flow Map Distillation Without Data

TL;DR

FreeFlow is a flow-map distillation framework that uses no external dataset — only samples from the prior. The authors identify Teacher-Data Mismatch: when distilling, the standard practice of constructing intermediate states $\tilde p_t$ by noising a real dataset diverges from the teacher’s own generating distribution $\hat p_t$ , and that mismatch caps student fidelity. By replacing the noised-data distribution with prior-anchored trajectories and combining a prediction objective (student’s generating velocity = teacher’s instantaneous velocity along its own path) with a correction objective (student’s noising velocity matches the teacher’s via a VSD-style auxiliary score), the 1-step student hits FID 1.45 on ImageNet-256 and 1.49 on ImageNet-512 with only 20 distillation epochs — beating pi-Flow, sCD-XXL, and DMF at matched or larger budgets.

Key claims

Distilling a flow map by noising an external dataset bakes in a Teacher-Data Mismatch: $\tilde p_t \neq \hat p_t$ whenever the teacher’s solution paths diverge from the data-noising paths; controlled augmentation experiments show monotonic FID degradation as the dataset is pushed further from the teacher [§“The Cost of Mismatch”, Fig. with augmentation sweep].
Sampling intermediate states only from the prior $\pi$ avoids the mismatch by construction — the prior is the one distribution where teacher and student are guaranteed to agree [§“Core insight”].
A local consistency identity $\vF_{\theta^*}(\vz,\delta) + \delta\partial_\delta\vF_{\theta^*}(\vz,\delta) = \vu(\vf_{\theta^*}(\vz,\delta),1-\delta)$ reduces flow-map distillation to aligning the student’s generating velocity $\vvG = \partial_\delta \vf_\theta$ with the teacher’s instantaneous velocity $\vu$ along the student’s own trajectory, computable via JVP or a finite-difference approximation [§“Method”].
The prediction objective alone suffers compounding drift: errors at small $\delta$ accumulate as the student integrates toward data, and the local velocity target at the drifted state cannot pull it back [§“The student functions effectively as an autonomous ODE solver”].
A VSD-derived correction objective aligns the student’s noising velocity $\vvN$ (estimated by an online auxiliary score net $g_\psi$ ) with the underlying $\vu$ — also purely prior-anchored, no data — and acts as the drift-rectifying stabilizer [§“To mitigate this drift”].
Prediction-only plateaus at suboptimal FID; correction-only suffers mode collapse and gradual degradation; only the combination keeps improving monotonically over training [§“Synergy” with three-curve training plot].
On ImageNet-256 with SiT-XL/2+REPA as teacher (teacher FID 1.37), FreeFlow-XL/2 reaches FID 1.84 at 1 NFE in 20 epochs and 1.45 at 300 epochs, vs. pi-Flow’s 2.85 (1 NFE, 448 epochs) / 1.97 (2 NFE) [Table — ImageNet-256, distillation block].
On ImageNet-512 with the same teacher, FreeFlow reaches 2.11 at 1 NFE and 1.49 at 200 epochs — strictly better than sCD-XXL (1.5B params, 320 epochs, 2.28 at 1 NFE / 1.88 at 2 NFE) at fewer epochs and roughly half the parameters [Table — ImageNet-512].
Distilling produces a fast proxy that can search noise for the expensive teacher: with 80 NFEs total (FreeFlow noise search + one teacher integration), the system beats classifier-free-guidance sampling of the teacher at 128 NFEs [§“Scaling compute at inference time”].

Method

The student parameterizes the flow map via an average-velocity head $\vF_\theta$ such that $\vf_\theta(\vz, \delta) = \vz + \delta \vF_\theta(\vz, \delta)$ , where $\vz \sim \pi$ is a prior sample and $\delta \in [0,1]$ is the integration duration from the prior. Differentiating the optimal-displacement identity $\delta \vF_{\theta^*}(\vz,\delta) = \int_1^{1-\delta} -\vu(\vx(\tau),\tau)\, d\tau$ in $\delta$ produces the local consistency condition above. The prediction objective asks the student’s generating velocity $\vvG = \vF_\theta + \delta \partial_\delta \vF_\theta$ to match the teacher’s velocity at the student’s own current state — computed with a stop-gradient on the residual and implemented via JVP or finite differences.

Because the student is an approximate ODE solver, its trajectory drifts off the teacher’s path as $\delta \to 1$ , and the local target $\vu$ at a drifted state does not correct that drift. FreeFlow adds a correction objective that aligns the student’s noising velocity $\vvN$ — estimated by a lightweight online auxiliary network $g_\psi$ trained alongside — with $\vu$ . Both objectives draw all of their intermediate states from the prior $\pi$ , never from a dataset. At inference the student is a one- (or few-) step generator; the auxiliary $g_\psi$ is discarded.

Results

ImageNet-256 (teacher SiT-XL/2+REPA, teacher FID 1.37): FreeFlow-XL/2 at 678M params, 1 NFE FID = 1.84 after 20 epochs, 1.45 after 300 epochs. Comparable rows: pi-Flow 2.85 / 1.97 (1 / 2 NFE, 448 epochs), DMF-XL/2 2.16 / 1.51 (1 / 4 NFE, 880 epochs from scratch), MeanFlow-XL/2 3.43 (1 NFE) / 2.20 (2 NFE, 1000 epochs) [Table — ImageNet-256].
ImageNet-256 (teacher SiT-XL/2, teacher FID 2.06): FreeFlow 2.24 at 1 NFE, 20 epochs / 1.69 at 300 epochs, beating SDEI (2.46, 8 NFE) and FACM (2.07, 2 NFE) [Table — ImageNet-256].
ImageNet-512 (teacher SiT-XL/2+REPA, FID 1.37): 2.11 at 1 NFE (20 ep) / 1.49 at 200 ep; vs. sCD-XXL (1.5B) at 2.28 / 1.88 (1/2 NFE, 320 ep), sCD-XXL+VSD at 2.16 / 1.89 (32 ep) [Table — ImageNet-512].
Inference-time noise search: total budget 80 NFE (FreeFlow proxy search + one teacher integration) outperforms teacher’s CFG sampling at 128 NFE [§“Scaling compute at inference time”].
Ablation: prediction-only plateaus; correction-only collapses; combined objective monotonically improves [§“Synergy” plot].

Why it’s interesting

FreeFlow reframes flow-map distillation as a question about the support of intermediate states, not the choice of loss or parameterization — directly addressing a failure mode the wiki’s distillation thread had not isolated. Wiki entries on the prediction/correction split each attack a different lever: ArcFlow: Unleashing 2-Step Text-to-Image Generation via High-Precision Non-Linear Flow Distillation (ArcFlow) optimizes the trajectory parameterization (analytic non-linear flow), Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis (DP-DMD) optimizes the role separation across student steps, and Cross-Resolution Distribution Matching for Diffusion Distillation (RMD) optimizes the resolution schedule. FreeFlow operates orthogonally: it changes where the intermediate states come from (prior vs noised data), which is invariant to all three of those axes — so it should in principle compose with them. The 20-epoch result at 1 NFE FID 1.84 on ImageNet-256 against pi-Flow’s 448-epoch 1 NFE FID 2.85 also substantially shifts the compute–quality frontier on the wiki’s distillation table.

The data-free framing is also a budget unlock: for proprietary or licensed teachers where redistribution of training data is restricted, distilling without ever touching it is operationally useful.