Skip to content
Bud's Wiki

No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA)

TL;DR

Section titled “TL;DR”

SRA (Self-Representation Alignment) argues that a diffusion transformer can supply its own representation guidance — no external DINO/CLIP encoder, no auxiliary representation-pretraining stage. The recipe: at each training step, align the DiT’s earlier-layer output features at a higher noise level to the later-layer output features at a lower noise level of the same network, via self-distillation. The intuition is that a generative-only DiT already learns a coarse-to-fine discriminative process across layers and noise levels, so the cleaner, deeper view can supervise the noisier, shallower view. SRA reports comparable-or-superior accuracy to REPA and other external-encoder methods on DiTs and SiTs, across multiple model sizes, while removing the external dependency entirely.

Key claims

Section titled “Key claims”
  • Existing representation-guided diffusion recipes (REPA family) either need an additional and complex representation training framework or a large-scale pre-trained representation foundation model to provide guidance — SRA removes both [project page Body §1].
  • Diffusion transformers trained on generation alone already implement a roughly coarse-to-fine discriminative process across layers, so a within-network self-distillation signal exists without any external supervision [project page Body §3].
  • SRA aligns the DiT’s earlier-layer output latent representation at higher noise to the later-layer output latent representation at lower noise — i.e. the cleaner, deeper view is the teacher, the noisier, shallower view is the student, both inside the same network during the standard generative training pass [project page Body §4].
  • SRA both accelerates training and improves final generation FID/IS on both DiT and SiT backbones [project page Body §2].
  • SRA is robust across model size on the same backbone family — gains hold from smaller to larger DiTs/SiTs (no parity-only sweet spot) [project page Body §5].
  • SRA achieves comparable or superior performance versus methods that use an external representation-training paradigm or a representation foundation model (i.e. versus REPA’s external-DINO recipe family) [project page Body §6].
  • SRA genuinely enhances the representation capacity of the baseline model (not just sample quality), and the generative capability is reported to be strongly correlated with the strength of the representation guidance [project page Body §7].
  • Independent measurement from the Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis T2I benchmark (FLUX.2 backbone, 1M steps, 20M pairs) places SRA at FID 3.61’s neighborhood — specifically FID 3.70, beating REPA (3.92) and vanilla flow matching (4.08) but trailing Self-Flow (3.61) Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis §4.1, Tab. 2.
  • Same benchmark on ImageNet-class-conditional SiT-XL at 4M steps: SRA reaches FID 7.27 / IS 143.06 — better than the SiT-XL/2 baseline (8.30) but trailing both REPA (5.89) and Self-Flow (5.70) Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis §4.1, Tab. 1.

Method

Section titled “Method”

The artifact filed here is the project page; the full paper is arxiv 2505.02831 (Jiang et al., 2025). What follows summarises the project page’s account of the mechanism — the deeper algorithmic content lives in the full paper.

The setup is a standard generative-training pass through a DiT or SiT — no extra stages, no extra encoders, no extra losses on auxiliary data. During the normal denoising/flow step, the model is evaluated twice (or features at two layers are read out of one pass): once at a noisier input giving an earlier-layer output feature, and once at a cleaner input giving a later-layer output feature. The cleaner-deeper feature is treated as the target; the noisier-shallower feature is pushed toward it. The combined objective is the usual generative loss plus this self-alignment term.

Two distinctive choices, per the project page: (a) the alignment is across both layer depth and noise level — not just deep-to-shallow within the same noise schedule and not just clean-to-noisy at the same layer. The project page frames this as supervising “the same coarse-to-fine discriminative process the network already exhibits.” (b) There is no separate representation-pretraining stage and no external network — the teacher view is just a different evaluation of the same DiT/SiT during generative training.

This places SRA in the same family as the later Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis (BFL + MIT), which uses an EMA teacher on a cleaner-noise view and a student on a heterogeneously-noised view, with cosine-similarity alignment at a chosen mid-layer. The shared spirit is internal-self-distillation as a substitute for REPA-style external-DINO supervision; Self-Flow’s added machinery (EMA-teacher copy, Dual-Timestep Scheduling with Bernoulli token masking) is what differentiates the two recipes mechanically.

Results

Section titled “Results”

The project page reports qualitative claims rather than full numerical tables (the numbers live in the arxiv PDF, which is what one should consult for tables and ablations):

  • Accelerated convergence on both DiT and SiT baselines: SRA reaches comparable generation quality in fewer training steps than the same backbones trained with generation-only.
  • Improved final FID/IS at matched training budget on DiT/SiT.
  • Robust across model size within the DiT/SiT families.
  • Competitive with REPA family: comparable or superior to methods that use an external representation foundation model or a separate representation-training paradigm.
  • Representation capacity gain: SRA improves the discriminative quality of the model’s internal features, and this gain correlates with the generative-quality gain.

Independent third-party numbers from the Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis paper’s benchmark tables (filed alongside this page):

The headline reading: at the time of writing, SRA is the strongest reported encoder-free representation-alignment recipe before Self-Flow, and Self-Flow’s contribution is to take the same “internal teacher” spirit further with an EMA-decoupled teacher copy and explicit dual-timestep view construction.

Why it’s interesting

Section titled “Why it’s interesting”

This is the paper that planted the flag on “diffusion transformers can self-supply representation guidance without any external encoder” — the entire premise that Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis later sharpens and scales. The Slack note from Dejia frames the connection precisely: “similar to the spirit of self-flow.” Filing this is what lets Diffusion training efficiency resolve the internal-self-distillation thread chronologically rather than treating Self-Flow as a singular event.

Two specific reasons this matters for Luma’s understanding of the space:

  1. Priority on the internal-teacher idea. Self-Flow’s contribution is now better located as “SRA-style internal supervision outscales REPA” rather than “internal supervision was a novel invention.” That changes how the Diffusion training efficiency page should frame the internal-vs-external fork.

  2. Concrete gap to close. SRA’s ImageNet number (FID 7.27) is worse than REPA’s (5.89) at the same step count — but SRA on T2I (3.70) is better than REPA (3.92). Self-Flow then beats both on both benchmarks. The natural next question: what is the structural difference between SRA’s “earlier-layer-noisier vs later-layer-cleaner same-network” and Self-Flow’s “EMA-teacher cleaner-uniform vs student heterogeneous-mix” that produces the ImageNet gap? Both papers’ setups provide the controlled ablation lever.

It also pairs naturally with DINO-SAE: DINO Spherical Autoencoder for High-Fidelity Image Reconstruction and Generation‘s direction-only DINO matching critique — three concurrent papers all locating fragility in external DINO supervision and proposing different repairs (drop magnitude, drop encoder entirely, redesign noise/depth coupling).

See also

Section titled “See also”