Mean Mode Screaming: Mean-Variance Split Residuals for 1000-Layer Diffusion Transformers

TL;DR

A mechanistic audit of why deep Diffusion Transformers (DiTs) silently collapse at depth, plus an architectural fix that lets a single-stream DiT train stably at 400 and 1000 layers. The paper isolates the trigger as Mean Mode Screaming (MMS): a mean-coherent backward shock on residual writers that pushes the residual stream into a mean-dominated state, homogenizing token representations and suppressing centered (non-mean) variation. The diagnosis is an exact decomposition of writer gradients into a mean-coherent component and a centered component, plus the observation that once values homogenize, the Softmax Jacobian’s null space structurally suppresses attention-logit gradients — locking the collapse in. The fix, MV-Split Residuals, separates the residual update into a centered residual (with its own gain) and a leaky trunk-mean replacement; on a 400-layer DiT it prevents the divergent collapse that crashes the un-stabilized baseline, and a 1000-layer scale-validation run remains stably trainable.

Key claims

Naively scaling a single-stream DiT to hundreds of layers produces a silent failure mode (MMS) where the network enters a mean-dominated state that homogenizes token representations and suppresses centered variation, even while training loss looks stable [Abstract, §1].
The MMS trigger admits an exact gradient decomposition: residual-writer gradients split into a mean-coherent component and a centered component, and the mean-coherent component delivers a backward shock that opens deep residual branches into the mean-dominated regime [Abstract, §3 (decomposition)].
The collapse is self-reinforcing because once values homogenize, attention-logit gradients are structurally suppressed through the null space of the Softmax Jacobian, removing the gradient signal that would otherwise re-introduce token-wise variation [Abstract, §3].
MV-Split Residuals fix this by combining a separately-gained centered residual update with a leaky trunk-mean replacement, treating the mean and the centered components as distinct update channels rather than a single residual add [Abstract, §4 (method)].
On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline and tracks close to the baseline’s pre-crash trajectory [Abstract, §5].
MV-Split is substantially better than token-isotropic gating methods such as LayerScale across the full training schedule on the 400-layer DiT [Abstract, §5].
A 1000-layer DiT trained with MV-Split serves as a scale-validation run, establishing that the architecture remains stably trainable at extreme depth [Abstract, §6].

Method

The diagnosis is the load-bearing part. The paper decomposes each residual writer’s backward signal into the gradient of the per-token mean plus the gradient of the centered (zero-mean) component, and shows that when the mean-coherent component dominates, every residual branch deeper in the stack receives a coherent push in the same direction — the “mean mode screams.” Empirically this drives the token-stream variance away from the centered subspace and into a mean-dominated state, at which point the Softmax Jacobian (which has the all-ones direction in its null space) structurally suppresses the attention-logit gradient that would have re-injected centered variation. The collapse is therefore self-locking: nothing the optimizer sees in the loss signals the failure, but the representation has homogenized.

The intervention, MV-Split Residuals, treats the residual update as two channels rather than one. The centered component of the layer output is added back with its own learnable gain, and the trunk mean is replaced via a leaky (interpolating, not gating) rule rather than additively accumulated. This decouples the two failure modes — mean-coherent backward shock and centered-component starvation — so each can be controlled independently. The paper contrasts this with LayerScale-style token-isotropic gating (a single scalar gain per channel applied identically to mean and centered components), which cannot separate the two channels and so under-controls the dominant failure direction.

Results

The headline experimental claims are at two depths. On the 400-layer single-stream DiT, the un-stabilized baseline exhibits a divergent collapse mid-training that MV-Split prevents entirely; MV-Split tracks close to the baseline’s pre-crash trajectory while staying substantially better than LayerScale across the full schedule [Abstract, §5]. The 1000-layer DiT is a scale-validation run rather than a benchmark-chasing result — the paper’s claim is qualitative: at extreme depth, the architecture remains stably trainable with MV-Split, which is the property a depth-scaling primitive needs to demonstrate. No FID/CLIP numbers vs. shallow-DiT SOTA are foregrounded in the abstract; the contribution is the depth axis itself.

Why it’s interesting

This sits at the intersection of two active lines on the wiki. First, it’s a DiT-side answer to the open question raised under Training stability at scale — every architectural-stability paper there (mHC, Gated Attention, Canon Layers, SSO) is on LLM decoders, and a recurring open question is whether the same residual-geometry pathologies surface in DiT pretraining where AdaLN modulations are an additional outlier source. MV-Split says yes, with a specific mechanism (mean-coherent backward shock through residual writers) and a specific fix. Second, the diagnostic style — gradient decomposition into mean-coherent vs. centered, plus a Softmax-Jacobian null-space argument — mirrors A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training‘s outlier-driven-rescaling framing and The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks‘s anatomy of massive activations; together they’re building a vocabulary for “where stability lives in the forward/backward map.” It also contrasts directly with mHC: Manifold-Constrained Hyper-Connections: mHC constrains the residual-mixing matrix to bound composite gain across depth; MV-Split instead splits the residual into two update channels with different gain treatments. Both target the identity-mapping property at depth from different sides, and stacking them at extreme depth is an obvious untested combination.