Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis

TL;DR

Self-Flow (BFL + MIT) is a self-supervised flow-matching paradigm that gets REPA-style representation alignment without any external encoder. The mechanism is Dual-Timestep Scheduling: two noise levels are mixed across tokens of one input, and an EMA teacher (run on the cleaner view) supervises the student’s deep-layer features via cosine similarity — effectively doing DINO-style self-distillation inside the generative model. Reported gains are consistent across class-to-image (ImageNet FID 5.70 < REPA 5.89 at 4M steps), text-to-image (3.61 < REPA 3.92), text-to-video (FVD 47.81 < REPA 49.59), and text-to-audio. Crucially, REPA’s quality degrades as the external DINO encoder gets stronger; Self-Flow recovers expected scaling laws and a 625M Self-Flow model beats a 1B REPA model at T2I.

Key claims

Stronger external alignment encoders hurt generation quality: replacing DINOv2-B with DINOv2-L → DINOv3-B → DINOv3-H+ in REPA monotonically degrades FID, exposing external alignment as a bottleneck rather than a Bitter-Lesson-style scaling lever [§3.2, Fig. 2(a)].
External alignment fails to generalize across modalities: V-JEPA 2 and Depth Anything 3 harm video generation versus vanilla flow matching, and MERT gives no benefit over vanilla flow for audio — REPA’s gains are dataset/encoder-specific to ImageNet+DINOv2 [§4.1, Tab. 3, Tab. 4].
Dual-Timestep Scheduling samples two timesteps from the noise distribution and applies them to disjoint token subsets via a Bernoulli mask, creating per-input information asymmetry while preserving the marginal per-token noise distribution; naive alternatives (full masking, per-token i.i.d. timesteps à la diffusion forcing) degrade quality due to the train/inference gap [§3.3, Fig. 2(b)].
The Self-Flow objective adds a cosine-similarity feature-alignment loss between a student (on the mixed-noise input) and an EMA teacher (on the cleaner-noise view of the same input) at layer ℓ=8 of a SiT-XL / FLUX.2 transformer, combined with the standard flow-matching loss [Eq. 6, Eq. 7].
On ImageNet without external representations, Self-Flow at 4M steps reaches FID 5.70 / IS 151.40, beating REPA’s 5.89 / 157.66 — the first reported case of self-supervised alignment outperforming external alignment on ImageNet [§4.1, Tab. 1].
Self-Flow is autoencoder-agnostic and stacks with representation autoencoders: applied on top of RAE it improves ImageNet FID 3.24 → 2.95, sFID 6.73 → 5.50, IS 218.53 → 222.34 at 1M steps [§4.1, Tab. 1, Fig. 4(a)].
On T2I, Self-Flow wins on every reported metric — FID 3.61 vs REPA 3.92 / SigLIP-2 3.97 / SRA 3.70 / Vanilla 4.08, and also lowest FD-DINO (167.98) even though REPA is the one explicitly aligning to DINOv2 features [§4.1, Tab. 2].
Self-Flow follows expected scaling laws while REPA does not: across 290M / 420M / 625M / 1B T2I models, the Self-Flow vs REPA gap widens with scale, and 625M Self-Flow beats 1B REPA [§4.1, Fig. 6(a)].
Linear-probe quality of early- and mid-layer features increases substantially under Self-Flow at 2M steps, confirming the generative model itself learns better representations rather than relying on a frozen one [§4.1, Fig. 4(b)].
Ablation: removing the representation loss costs >4 FID points; removing the masking while keeping the rep loss costs >1 point; constraining the two timesteps to be close costs almost as much as removing masking; an ℓ₂ alignment objective is numerically unstable due to growing feature norms — cosine similarity is required [§4.4, Fig. 11(a)].
Mixed-modality training: on a single 625M model jointly trained on T2I + T2V + T2A under modality-loss-weighting sweeps, Self-Flow improves all three modalities simultaneously across the full weighting simplex [§4.2, Fig. 8(a)].
Joint video-action prediction on RT-1 / SIMPLER: Self-Flow initialization beats vanilla-FM initialization on robot manipulation tasks, with the gap concentrated on multi-object and sequential tasks (Move Near, Open-and-Place) — pointing toward better visual reasoning rather than just better pixels [§4.2, Fig. 8(b)].

Method

The generator is a standard flow-matching transformer (SiT-XL for ImageNet, FLUX.2 backbone for T2I/T2V/T2A; 625M params unless noted). The new training-time machinery sits entirely on top.

Dual-Timestep Scheduling. For each input x, sample two timesteps t ≥ t̃ from the FM noise schedule and a Bernoulli mask m (with masking ratio drawn from a fixed distribution). Construct one noised input where tokens with m=1 use the higher noise t and tokens with m=0 use the lower noise t̃, and a second “cleaner” input where all tokens are noised at t̃. Both keep the linear flow interpolation between data and noise, so the marginal per-token noise distribution matches vanilla FM — which is the key reason this beats per-token i.i.d. timestep schemes (diffusion forcing) that drift out of distribution at inference.

Self-Flow loss. A student network (the model being trained) is run on the heterogeneously-noised input. An EMA teacher (same architecture, slow-following weights) is run on the cleaner uniformly-noised input. At layer ℓ=8 of the student and a corresponding teacher layer, the student’s features are projected through a small MLP and aligned to the teacher’s features via cosine similarity. The total loss is the standard flow-matching velocity loss on the heterogeneously-noised input plus a scaled alignment term; the alignment term is the part that does the heavy lifting (removing it costs >4 FID).

This is what the Slack note flags as “doing DINO inside the diffusion model”: the EMA-teacher + student + feature-cosine-alignment skeleton is the DINO recipe, but the augmentation pair is “more-noised vs less-noised view of the same latent” rather than “two random crops of the same image”, and the network being supervised is the generator itself rather than a separate vision backbone. The teacher needs an extra forward pass per step (the paper’s main acknowledged cost), but no external encoder weights are loaded or queried.

Results

ImageNet class-to-image (SiT-XL, 4M steps). Self-Flow 5.70 FID / IS 151.40 / Pre 0.72; REPA 5.89 / 157.66 / 0.70; SRA 7.27 / 143.06; SiT-XL/2 baseline 8.30 (at 7M steps) [Tab. 1]. Stacked on RAE at 1M steps: FID 3.24 → 2.95 [Tab. 1].

T2I (FLUX.2 backbone, 1M steps, 20M pairs). Self-Flow FID 3.61, sFID 8.14, IS 21.19, FD-DINO 167.98, CLIP 30.88 — best across the board, beating REPA (3.92, 173.35) even on the DINO-feature distance REPA explicitly optimizes [Tab. 2].

T2V (FLUX.2 backbone, Wan2.2 AE, 6M videos, 600K steps). Self-Flow FVD 47.81 / per-frame FID 8.92; REPA (DINOv2) 49.59 / 9.39; V-JEPA 2 53.55 / 9.91 — worse than vanilla 50.95 / 9.28 [Tab. 3]. Video-specialized external encoders actively hurt.

T2A (FLUX.2 backbone, Songbloom AE, FMA, 350K steps). Self-Flow CLAP 145.65 / CLAP-M 0.1634 — best FAD across all CLAP variants; MERT external alignment ≈ vanilla baseline [Tab. 4].

Scaling (T2I, 290M / 420M / 625M / 1B). Self-Flow-vs-REPA gap widens monotonically with scale; 625M Self-Flow > 1B REPA. REPA itself does not improve proportionally with parameter scale [Fig. 6].

Compute overhead. One extra teacher forward per step is the main cost; the paper argues convergence speedups (FID at fewer steps, scaling-law recovery) justify it.

Why it’s interesting

This is the strongest evidence yet that the “external DINO supervision” recipe that Diffusion training efficiency is currently organized around may be a transitional crutch rather than a Bitter-Lesson-final answer. The whole concept page’s premise — that DINOv2 is the common substrate plumbed in at the encoder (Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders / Improved Baselines with Representation Autoencoders), at intermediate layers (REPA), or at the output (PixelGen: Improving Pixel Diffusion with Perceptual Supervision) — gets undercut by Self-Flow’s evidence that the same network can grow its own DINO-quality features when given the right self-supervised objective, and that the external version actively scales worse with encoder strength.

It also slots beside DINO-SAE: DINO Spherical Autoencoder for High-Fidelity Image Reconstruction and Generation‘s critique from the autoencoder side: DINO-SAE argued you should drop magnitude matching with DINO (keep direction only), Self-Flow goes further and argues you shouldn’t need DINO at all if your generator’s own teacher signal is structured right. Both diagnose external alignment as architecturally fragile in different ways.

Concretely actionable for Luma: (a) the alignment loss is trivially additive to any flow-matching training loop — no encoder to host, no extra dependency, just an EMA copy of the model and a second forward; (b) the headline T2V gain (REPA-helps-image / external-encoders-hurt-video → Self-Flow-helps-both) is the directly Luma-relevant result, since video is where external visual encoders have been most awkward; (c) the scaling behavior (Self-Flow scales like normal compute, REPA flattens) is exactly the property you want before committing alignment infrastructure to a 10B-parameter video DiT.