When Does Sparsity Mitigate the Curse of Depth in LLMs

TL;DR

Reframes “the curse of depth” (later layers in pre-LayerNorm LLMs contribute less than earlier ones) as a variance-propagation problem and identifies sparsity — both implicit (weight-decay-induced weight sparsity, soft attention sparsity from data/training) and explicit (architectural MoE / sliding-window / sparse-attention patterns) — as a regulator of that variance. The mechanism is concrete: sparsity reduces the per-block output variance, which prevents the cumulative growth in Pre-LN that pushes deep blocks toward near-identity behavior, in turn improving layer utilization. The paper distills the findings into a rule-of-thumb training recipe that yields a 4.6% downstream-accuracy improvement, and argues sparsity is a previously-overlooked mechanism for effective depth scaling — orthogonal to QK-Norm / Gated-Attention / spectral-sphere fixes that target the same instability from other loci.

Key claims

The curse of depth in Pre-LN LLMs is mechanistically driven by accumulated growth of residual-stream variance, which makes deep blocks operate in a near-identity regime and waste capacity [Abstract, §1].
Sparsity — implicit (weight sparsity from weight decay; soft attention sparsity from data conditions) and explicit (architectural sparse-attention / MoE patterns) — regulates variance propagation and is therefore not just an efficiency feature but a stability/depth-utilization feature [Abstract, §1].
Prior reports of “signal collapse” in sparse networks (Dey et al. 2024/2025) — where variance shrinks as connection density falls — are reinterpreted here as evidence that sparsity inherently regulates variance, linking efficiency-motivated and stability-motivated sparsity lines [§1].
Across controlled depth-scaling experiments and targeted layer-effectiveness interventions, sparsity improves layer utilization by reducing output variance and promoting functional differentiation across blocks [Abstract].
A practical rule-of-thumb recipe distilled from the analysis yields a 4.6% accuracy improvement on downstream tasks for depth-effective LLM training [Abstract].
Explicit sparsity (GQA / MoE / Mixture-of-Depths-style designs) hard-codes a fraction of absent connections at initialization, in contrast to implicit sparsity which emerges dynamically during training — both regulate variance but via structurally different routes [§1].

Method

The paper studies two families of sparsity side-by-side. Implicit sparsity is the kind that emerges from training itself: weight sparsity induced by weight decay, and the soft attention sparsity that arises from data and training-condition choices. Explicit sparsity is architectural — patterns hard-coded at initialization that guarantee a predetermined fraction of connections or computational paths are absent (GQA, MoE, sparse-attention variants).

The analysis combines (a) variance-propagation arguments in the Pre-LayerNorm residual stack, showing how each block’s output variance accumulates and pushes deeper blocks toward identity, and (b) controlled depth-scaling experiments + targeted layer-effectiveness interventions that vary sparsity at multiple loci (weights, attention, architecture) while measuring per-layer effectiveness, output variance, and downstream task accuracy. Findings are then distilled into a recipe-style prescription for training depth-effective LLMs.

Results

A 4.6% accuracy improvement on downstream tasks results from applying the paper’s depth-effective training recipe [Abstract].
Across the studied settings, the relationship “more sparsity → lower output variance → higher per-layer effectiveness” holds consistently [Abstract].
Sparsity also promotes functional differentiation across blocks — i.e. deep layers do meaningfully different work rather than collapsing toward identity [Abstract].
Code is released for the rule-of-thumb recipe [Abstract].

(Full quantitative tables and ablations are in the full PDF; the arXiv abstract HTML available to filing summarizes the headline numbers above.)

Why it’s interesting

This is a fourth locus in the Training stability at scale story: previous wiki entries place the load-bearing fix at normalization (QK-Norm in How to Set the Learning Rate for Large-Scale Pre-training?; Gated Attention / GatedNorm in A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training), at the residual-mixing matrix (mHC: Manifold-Constrained Hyper-Connections), or at the optimizer’s spectral structure (Controlled LLM Training on Spectral Sphere). This paper locates it in sparsity-as-variance-regulator, with the explicit framing that the curse of depth and the variance accumulation that drives it are two sides of the same Pre-LN coin — sharpening the open question on the concept page about whether these fixes stack or substitute. It also reconnects efficiency-motivated sparsity choices (GQA, MoE, sparse attention) to the variance-control story that The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks documents from the opposite direction (RMSNorm sparsifies spike tokens into a low-dimensional subspace, and that geometric collapse is what makes sinks form) — i.e. both papers identify sparsity as central to the post-norm geometry, but Sun et al. frame the resulting sparsity as a side-effect of outliers that enables sinks, while Muhtar et al. frame architectural sparsity as a cure for the variance growth that wastes deep layers.