Spectral Scaling Laws in Language Models: How Effectively Do Feed-Forward Networks Use Their Latent Space?

TL;DR

Recasts feed-forward-network width selection in transformers as a spectral utilization problem: instead of asking “how big should the FFN be?”, ask “of the directions the FFN could span, how many does the model actually use?” Introduces a four-metric diagnostic suite (Hard Rank / Soft Rank / Spectral Concentration / Spectral Utilization Index) computed from the eigenspectrum of post-activation covariance, and uncovers an Asymmetric Spectral Scaling Law across GPT-2, LLaMA, and nGPT 70M–250M models trained on C4: soft rank scales near-linearly with FFN width $D$ ( $\beta_{\rm soft} \approx 0.97{-}1.0$ ) while hard rank grows only sub-linearly with high variance ( $\beta_{\rm hard} \approx 0.3{-}0.5$ ). Widening FFNs is tail-first growth — capacity expands by adding low-energy directions while the high-energy core saturates early, giving a concrete spectral rationale for why LLaMA’s $\alpha = 2.67$ width multiplier and GPT-2’s $\alpha = 4$ work where wider settings show diminishing returns.

Key claims

Soft spectral rank (Shannon entropy of the covariance eigenspectrum) follows a near-perfect power law $\text{SRank} \propto D^{\beta_{\rm soft}}$ with $\beta_{\rm soft} \approx 0.97$ at LLaMA-70M, $\approx 1.0$ at 130M, $\approx 0.94$ at 250M, all with $R^2 > 0.95$ [§4.1, Fig. 2].
Hard spectral rank (Participation Ratio) grows sublinearly with $\beta_{\rm hard} \approx 0.30{-}0.50$ and substantially noisier fits ( $R^2 \approx 0.4{-}0.7$ ); the gap between SRank and HRank slopes is the Asymmetric Spectral Scaling Law [§4.1, Fig. 2].
Normalized utilizations follow $u_{\rm hard}(D) \propto D^{-0.55}$ and $u_{\rm soft}(D) \propto D^{-0.05}$ — soft utilization is nearly width-invariant, hard utilization collapses with width [§4.2].
Two distinct failure modes show up in the normalized view: spectral dilution (flat $u_{\rm soft}$ , falling $u_{\rm hard}$ — pronounced at LLaMA-130M) and spectral collapse (both falling — pronounced at LLaMA-250M with large $\alpha$ ) [§4.2].
Effective dimension (eDim, harmonic mean of hard and soft rank) grows sub-linearly with width and remains a small fraction of $D$ — at LLaMA-250M, eDim only rises from 36 ( $\alpha=1$ ) to 153 ( $\alpha=8$ ) against a width of 6144 [Table 2].
LayerNorm placement controls which side of the tail/core asymmetry dominates: Pre-LN gives the canonical asymmetry ( $\beta_{\rm soft} \approx 0.97$ , $\beta_{\rm hard} \approx 0.3{-}0.5$ ); Post-LN dampens both ( $\beta_{\rm soft}$ drops to $\approx 0.5{-}0.6$ , $\beta_{\rm hard}$ rises modestly); Mix-LN restores near-linear soft scaling and improves hard scaling above Pre-LN [§5.1, Table 4].
At LLaMA-250M with vanilla Post-LN, scaling FFN width to $2.67d$ or $4d$ produces full spectral collapse: hard rank plunges below 10 and concentration saturates above 0.95 within a few thousand steps, with test perplexity exceeding 100 [§5.2, Fig. 6a, Table 5].
WeightNorm in the FFN partially rescues Post-LN-250M from collapse — hard rank stabilizes in the 80–120 range, concentration drops to 0.25–0.30, and perplexity at $\alpha = 4$ (24.3) beats the $\alpha = 1$ vanilla baseline (27.1) [§5.2, Table 5].
A truncated power-law fit $\lambda_i \propto i^{-\delta}$ to the eigenspectrum gives a single-knob description of the tail/core trade-off: $\delta < 1$ produces dilution, $\delta > 1.5$ produces collapse, and LLaMA in practice sits at $\delta \in [1.0, 1.2]$ — balanced [§4.3, Table 3].
The scaling exponents stabilize after ~5K training steps with $R^2 > 0.95$ ; transient crossovers between widths appear in raw ranks but do not affect the converged exponents [§4.4].

Method

The pipeline is a layer-wise spectral audit run during training. At each step $t$ a mini-batch of tokens is forwarded through each FFN layer; the post-activation $X \in \mathbb{R}^{B \times D}$ (after the SwiGLU/SiLU gate, before the down-projection) is collected from every layer with no sub-sampling. An unbiased covariance $\Sigma = X^\top X / (B-1)$ is formed and eigen-decomposed to give sorted eigenvalues $\lambda_1 \geq \lambda_2 \geq \ldots \geq \lambda_D$ .

Four metrics are computed from this spectrum, each $O(D)$ once eigenvalues are sorted and each lying in $[0, 1]$ after normalization. Hard Rank is the Participation Ratio $\text{PR} = (\sum_i \lambda_i)^2 / \sum_i \lambda_i^2$ , a hard count of dominant directions that collapses to 1 under spectral collapse. Soft Rank is the exponentiated Shannon entropy of $p_i = \lambda_i / \sum_j \lambda_j$ , sensitive to the long tail. Spectral Concentration measures the area between the cumulative eigenspectrum and a uniform baseline (high = front-loaded, collapse-leaning; low = uniform, dilution-leaning). SUI is the harmonic mean of hard and soft rank — a single number that drops sharply under either failure mode, used to define an integer-valued Effective Dimension eDim.

The empirical sweep spans LLaMA, GPT-2, and nGPT backbones at 70M, 130M, and 250M parameters, with FFN width multiplier $\alpha \in \{1, 2, 2.67, 4, 5, 6, 7, 8\}$ , trained from scratch on C4. Each metric is fit as a power law in $D$ via log–log linear regression on the layer-median values after step 5K.

Results

Asymmetric power law (Fig. 2): at LLaMA-70M / 130M / 250M, $\beta_{\rm soft} = 0.97 / 1.00 / 0.94$ vs $\beta_{\rm hard} = 0.31 / 0.50 / 0.30$ . The gap between slopes is invariant across model size; only the absolute level of each line moves.
eDim sub-linearity (Table 2): at LLaMA-250M, eDim rises from 36 ( $D = 768$ ) to 153 ( $D = 6144$ ) — eight-fold width increase yields only 4.25× active-direction increase, and eDim $/D$ falls from 4.7% to 2.5%.
LayerNorm trade-off (Table 4): Mix-LN at LLaMA-130M achieves $\beta_{\rm soft} \approx 0.97$ and $\beta_{\rm hard} \approx 0.50$ , both higher than Pre-LN (0.97 / 0.32) and Post-LN (0.55 / 0.42). Mix-LN is the only ablation that improves both.
Post-LN collapse rescue (Table 5): vanilla Post-LN LLaMA-250M at $\alpha = 4$ has perplexity > 100 (training instability); WeightNorm in the FFN brings this to 24.3, better than the $\alpha = 1$ baseline at 27.1 — a clean demonstration that the spectral metric predicts which configurations will train.
Spectral concentration table (Table 3): at decay exponent $\delta = 1.5$ , 96% of variance lives in the top 25% of dimensions and 99% in the top 50% — practical threshold above which tail directions can be pruned without measurable loss.

Why it’s interesting

The paper supplies the spectral diagnostic that the wiki’s hyperparameter-scaling-laws cluster has been arguing about indirectly. Controlled LLM Training on Spectral Sphere claims Muon is “half-aligned” with µP because it constrains update spectra but not weight spectra; this paper provides the dual measurement on the activation side that shows what the resulting representation looks like when the spectrum drifts (tail-first growth, dominant-mode saturation). Spectral Condition for μP under Width-Depth Scaling gives the spectral condition µP must satisfy for joint width–depth scaling; this paper measures whether the empirical scaling laws actually hit it. The Mix-LN result is also a quiet rebuttal to the “QK-Norm subsumes all stability tricks” framing in How to Set the Learning Rate for Large-Scale Pre-training? §6.1 — LayerNorm placement still controls the spectral asymmetry independent of attention-side normalization. The most immediately usable artifact for Luma is the eDim diagnostic: a cheap, layer-local signal that flags when to stop widening an FFN and reallocate the budget — applicable to DiT FFN blocks, MMDiT cross-attention FFNs, and MoE expert sizing without modification.