Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Models

TL;DR

CREPA is REPA adapted for Video Diffusion Model fine-tuning, with one structural change: instead of aligning each frame’s DiT hidden state only with the pretrained image-encoder feature of that same frame (REPA*), each frame’s hidden state is additionally aligned with the pretrained features of neighboring frames, weighted by an exponential temporal decay. The motivation is empirical: under naive frame-wise REPA, hidden states drift stochastically across frames because the denoising autoencoder sees independent noise per frame, breaking semantic consistency. Visualizing learned hidden states under CKNN-A on pretrained feature manifolds shows that CREPA produces smoother cross-frame trajectories along the pretrained manifold than REPA*. The paper positions CREPA as a drop-in regularizer for fine-tuning rather than from-scratch training.

Key claims

A direct frame-wise adaptation of REPA to VDMs (REPA*) accelerates convergence but is suboptimal at preserving semantic consistency across frames, because hidden states extracted from independently-noised frames drift stochastically [project page Body §1].
CKNN-A (Continuous K-Nearest Neighbors Alignment) visualization shows that under REPA*, learned hidden states fail to follow a smooth trajectory across frames on the pretrained feature manifold; under CREPA they align more faithfully and follow smoother cross-frame trajectories [project page Body §2, motivation figure].
The CREPA loss adds, to REPA*‘s per-frame alignment term, an exponentially-decayed cross-frame term: for each frame $f$ , the hidden-state projection $h_\phi(h^f_t)$ is pushed toward the pretrained features $\bar{y}^k$ of each neighbor $k$ , weighted by $e^{-|k-f|/\tau}$ [project page §CREPA, alignment-loss equation].
The final training objective is the standard score / flow-matching loss plus $\lambda \cdot \mathcal{L}_{align}$ with the cross-frame alignment term [project page §CREPA, final-objective equation].
The method is framed as fine-tuning rather than from-scratch pretraining — CREPA is a regularization technique applied during the fine-tuning of an already-pretrained VDM [project page §title, §Body].

Method

CREPA modifies REPA in two places. First, it changes the alignment target set: a student frame at index $f$ is aligned not just with $\bar{y}^f$ (the frozen image-encoder feature of frame $f$ ) but also with $\{\bar{y}^k\}_{k \in K}$ for some neighbor set $K$ . Second, it adds an exponential temporal-decay weighting $e^{-|k-f|/\tau}$ so that close-in-time neighbors contribute more to the supervision than far ones. The student-side projection $h_\phi$ remains a small MLP head on top of a chosen DiT hidden state, same as REPA. The motivation, per the project page, is that hidden states of a VDM are extracted from per-frame stochastically-noised inputs — denoising autoencoders extract noise-dependent representations — so frame-wise REPA leaves the hidden-state trajectory free to drift along directions orthogonal to the pretrained manifold. Pulling each frame’s representation toward a window of neighbor pretrained features explicitly couples adjacent frames’ learned hidden states.

The paper visualizes this with CKNN-A on the pretrained feature manifold (the project page calls out “Continuous K-Nearest Neighbors Alignment” as the diagnostic): the trajectory of hidden states under REPA* is jagged, while under CREPA it is smooth. The arxiv abstract additionally states that CREPA is intended as a regularizer for fine-tuning a VDM rather than as a pretraining recipe.

The full paper (arxiv 2506.09229) contains the quantitative tables; the project page does not enumerate them.

Results

The fetched artifact (project page) reports qualitative claims rather than full tables — concrete numbers live in the arxiv PDF. The headline qualitative findings:

REPA* (frame-wise REPA) accelerates VDM fine-tuning convergence but degrades cross-frame semantic consistency relative to vanilla fine-tuning [project page §1].
CREPA recovers the convergence benefit and improves cross-frame semantic consistency over REPA*, evidenced by smoother hidden-state trajectories on the pretrained manifold under CKNN-A visualization [project page §2 + figure].

No FID / FVD / VBench numbers are reproduced on the project page.

Why it’s interesting

CREPA is the missing video-domain datapoint in the REPA-family vs. internal-self-distillation debate the wiki is already tracking. REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training (HASTE) showed REPA is a transient scaffold even in the image domain — gradient cosines flip from helpful to harmful past ~400K iterations on SiT-XL/2. No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA) (SRA) and the Self-Flow line argue the external scaffold isn’t needed at all, with Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis specifically reporting that external alignment with V-JEPA 2 and Depth Anything 3 hurts video FVD relative to vanilla flow matching. CREPA approaches the same video pathology from the opposite direction: keep REPA, but make the alignment cross-frame-aware so per-frame stochastic noise doesn’t desynchronize the supervision.

Two concrete observations Luma should care about:

Locates REPA’s video failure mode at the noise-stochasticity layer, not the encoder-quality layer. Self-Flow’s diagnosis is that the choice of video encoder (V-JEPA 2 / Depth Anything) is what breaks video REPA. CREPA’s diagnosis is that per-frame independent noise is what breaks it — frame-wise targets are themselves consistent (the encoder is the same image encoder applied per frame), but the student drifts because each frame sees independent stochastic input. These two diagnoses are not contradictory but they suggest different fixes, and they predict different things about whether SRA-style internal-self-distillation would also need a CREPA-like cross-frame coupling.
Framed as fine-tuning, not pretraining. This sidesteps the HASTE termination question — fine-tuning runs are short enough that the “REPA helps early, hurts late” gradient-cosine crossover may never fire. The natural unasked question is whether CREPA’s cross-frame weighting scheme would also be the right regularizer for pretraining a VDM from scratch, where the HASTE termination logic does apply.