What Matters for Representation Alignment: Global Information or Spatial Structure?

TL;DR

iREPA asks what specifically about a target representation makes REPA work, and answers: spatial structure (patch-to-patch similarity geometry), not global semantic quality (ImageNet linear-probe accuracy). Across 27 vision encoders the authors find linear-probe accuracy correlates with FID only at |r| = 0.26, while spatial self-similarity metrics correlate at |r| > 0.85. Acting on this, they replace REPA’s 3-layer MLP projection head with a small 2D convolution (kernel size 3) that preserves local spatial relationships, and apply spatial (instance-style) normalization to the DINO target features to sharpen patch-vs-background contrast. The combined recipe — termed iREPA, ~4 lines of code change — consistently improves convergence across DINOv2 / DINOv3 / CLIP / WebSSL / PE-Core targets, across SiT-B/L/XL, and across REPA-E and MeanFlow-with-REPA training recipes.

Key claims

• Linear-probe accuracy on ImageNet-1K is not predictive of REPA generation quality: across 27 encoders the correlation is only |r| = 0.26, while spatial-self-similarity metrics correlate with FID at |r| > 0.85 [§Spatial-Structure correlation analysis].
• Concrete inverse cases: PE-Core-G (1.88B, 82.8% acc) gives FID 32.3 vs PE-Spatial-B (80M, 53.1% acc) at FID 21.0; SAM2-S (46M, 24.1% acc) beats PE-Core-G despite ~59 points lower ImageNet accuracy [§Counter-examples].
• Larger encoders within the same family (DINOv2-L vs DINOv2-B; PE-Core-G vs PE-Core-B; C-RADIO) tend to give similar or worse generation than the smaller variant despite higher classification accuracy [§Counter-examples].
• A controlled CLS-token-mixing ablation isolates the causal direction: increasing α from 0 → 0.5 in target = (1-α)·patch + α·CLS lifts linear-probe accuracy from 70.7% → 78.5% but worsens generation FID from 19.2 → 25.4 — injecting more global information actively hurts REPA generation [§CLS-mixing ablation].
• Classical hand-designed spatial features (SIFT, HOG, intermediate VGG features) all yield REPA gains, providing further evidence that spatial structure alone is the operative signal — global semantic content is not required [§Hand-designed-features experiment].
• iREPA’s two changes individually each improve over baseline REPA, and combine: across four encoders (DINOv2-B, DINOv3-B, WebSSL-1B, PE-Core-G) iREPA-full beats both single-change ablations on every encoder [Table — ablation].
• iREPA reduces baseline-REPA FID from 19.06 → 16.96 with DINOv2-B, 21.47 → 16.26 with DINOv3-B, 26.10 → 16.66 with WebSSL-1B, and 32.35 → 18.19 with PE-Core-G — i.e., the bigger gains are on the previously-worse encoders, partially closing the encoder gap [Table — encoder sweep].
• iREPA stacks on top of REPA-E and MeanFlow-with-REPA: on REPA-E, iREPA-E lifts WebSSL-1B from FID 26.5 → 13.2 and PE-G from 25.9 → 16.4; on MeanFlow at 1-NFE / 4-NFE with CFG, iREPA halves the FID gap to DINOv3-B baselines [Table — recipe transfer].
• The spatial-structure correlation explains both baseline REPA and iREPA’s gains over baseline — i.e., the same metric that predicts which target encoders are good for REPA also predicts the magnitude of iREPA’s improvement [§Spatial-Structure correlation analysis, “explains both”].

Method

iREPA modifies REPA in two places, both targeting how spatial structure flows from encoder to student. (1) Convolutional projection head: standard REPA maps diffusion mid-layer features to encoder dimension via a 3-layer MLP applied token-wise; the authors argue this loses spatial contrast because the MLP cannot see neighbor tokens. They replace the MLP with a small 2D convolution of kernel size 3, preserving local spatial relationships explicitly. (2) Spatial normalization on target features: pretrained encoder patch tokens carry a global component that flattens contrast (tomato-patches and background-patches end up similar in cosine space). They apply instance-style normalization across the spatial dimension to the target features only: y = (x − E[x]) / √(Var[x] + ε), sacrificing the global mean signal to amplify between-region contrast. Both changes are total <4 lines of code on top of the official REPA training script. Loss and alignment depth are unchanged from REPA. Experiments use SiT-B/L/XL students with SD-VAE latents on ImageNet-256, plus REPA-E and MeanFlow-with-REPA training recipes for transferability.

Results

• ImageNet-256, baseline REPA → iREPA, DINOv3-B: FID 21.47 → 16.26, IS 63.4 → 78.8 [Table — encoder sweep].
• ImageNet-256, baseline REPA → iREPA, PE-Core-G: FID 32.35 → 18.19, IS 42.7 → 75.0 — the largest absolute improvement, on the worst-performing baseline encoder [Table — encoder sweep].
• REPA-E variant: WebSSL-1B 26.5 → 13.2 (FID halves), DINOv3-B 14.4 → 11.7, PE-G 25.9 → 16.4 [Table — REPA-E transfer].
• MeanFlow-with-REPA, 4 NFE with CFG 2.0, DINOv3-B → +iREPA: FID 15.6 → 11.1; at 1 NFE: 22.6 → 17.3 [Table — MeanFlow transfer].
• Spatial-similarity vs FID correlation: |r| > 0.85 across SiT-B / SiT-L / SiT-XL (consistent across model sizes); linear-probe-accuracy vs FID: |r| = 0.26 [§Correlation analysis].
• Convex CLS-mixing α=0 → α=0.5: ImageNet linear-probe acc 70.7% → 78.5%; FID 19.2 → 25.4 [§CLS-mixing ablation].

Why it’s interesting

This is the mechanism paper for the Representation Autoencoders and REPA threads — same author (Jaskirat Singh) and same group (Saining Xie’s lab) as Improved Baselines with Representation Autoencoders (RAEv2). RAEv2 reported that multi-layer aggregation + REPA-complementarity gives 10× speedup but treated which DINO features are useful as a black box; iREPA opens that box and identifies spatial-similarity geometry as the operative signal. Three concrete connections:

1. Re-interprets the HASTE / Self-Flow / SRA findings under one frame. REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training showed REPA helps early then hurts late and proposed termination; Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis showed REPA degrades with stronger DINO encoders and proposed internal self-distillation. Both observations are consistent with iREPA’s thesis: it is not more accurate representations that REPA needs, but representations with more useful spatial geometry. Stronger encoders gain accuracy by globally compressing patch features (CLS-like leakage into patch tokens), which iREPA’s ablation shows directly degrades generation. The “REPA stops scaling with encoder size” pathology is plausibly the same as “CLS-mixing degrades FID.”

2. Predicts what RAE / RAEv2 / FAE actually inherit from DINO. The Representation Autoencoders page’s existing open question is whether RAE-style frozen-encoder latents preserve the right thing. iREPA’s answer: spatial structure is the load-bearing property, which is exactly what single-attention-layer FAE encoders (One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation) preserve (and what deeper-encoder variants discard via the feature-reconstruction objective). iREPA’s “deeper projection hurts” finding mirrors FAE’s “deeper encoder overfits.”

3. Terrance’s note flags the right Luma-side question. The Slack message asks whether replacing LayerNorm with InstanceNorm or GroupNorm in the DiT would help convergence — i.e., whether iREPA’s “preserve spatial contrast” insight is also actionable on the student side, not just the teacher side. iREPA doesn’t run that ablation; it’s a clean follow-up the wiki should flag if anyone tries it.

The paper does not engage Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis‘s internal-self-distillation alternative — it accepts the external-DINO premise and patches it. Whether iREPA’s spatial-normalization trick is also necessary for Self-Flow’s internal teacher (which uses an EMA-decoupled view of the student’s own features, possibly with the same global-leakage problem) is an obvious open ablation.

See also

• Representation Autoencoders — same author lineage; iREPA explains why RAE-family representations work and where the limit lives.
• Diffusion training efficiency — adds a mechanistic claim (“what part of the encoder matters”) to a concept page that mostly tracked recipes.
• Improved Baselines with Representation Autoencoders — RAEv2, same author. RAEv2 reports the empirical 10× speedup; iREPA explains the underlying signal.
• Diffusion Transformers with Representation Autoencoders — original RAE; iREPA’s spatial-structure thesis is the strongest framework yet for choosing RAE encoders.
• One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation — FAE; the “deeper projection hurts” observation is the encoder-side mirror of iREPA’s “deeper projection head loses spatial contrast” observation.
• REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training — HASTE; complementary diagnosis of REPA failure modes (gradient-cosine vs spatial-structure).
• Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis — Self-Flow; the open question is whether iREPA’s spatial-normalization trick also fixes Self-Flow’s internal-teacher view.
• No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA) — SRA; internal-self-distillation alternative that iREPA doesn’t engage but probably interacts with.