Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD

TL;DR

Wan, Zhu, Zhang & Sun analyze SGD/SGDM with weight decay on scale-invariant weights (produced by normalization) as motion on a sphere — “Spherical Motion Dynamics.” Prior work observed that the effective learning rate ends up determined by hyperparameters when weight norm converges, but never explained why the equilibrium condition is reached. This paper introduces mild assumptions on the unit-gradient-norm process, proves weight norm converges to its equilibrium value at a linear rate, defines the angular update (angle between consecutive weight vectors) as a scale-invariant measure of progress, and derives its closed-form theoretical value in both SGD and SGDM. Theory is verified on ImageNet ResNet-50 and MSCOCO Mask-RCNN with standard hyperparameter schedules.

Key claims

For scale-invariant weights, effective learning rate η/‖w‖² is scale-dependent and does not measure how much the represented function changes; the angular update Δθ = arccos(⟨w_t, w_{t+1}⟩ / (‖w_t‖‖w_{t+1}‖)) does [§3, Def. 2].
Under mild assumptions on the unit-gradient-norm distribution (locally constant expectation, bounded variance), SGD-with-weight-decay drives ‖w‖² to a theoretical equilibrium ‖w‖² → η · E[‖g_unit‖²] / (2λ) at a linear rate — the equilibrium is reached long before the loss converges [§4, Thm. 1].
SGDM (heavy-ball) admits an analogous convergence result with modified assumptions accommodating the momentum vector’s non-orthogonality to the weight; the equilibrium rate is faster than SGD’s at the same LR/WD but the variance is larger [§4, Thm. 2].
In equilibrium, the angular update in SGD is Δθ ≈ √(2ηλ) and in SGDM Δθ ≈ √(2ηλ / (1-β)); both depend only on hyperparameters, not on gradient magnitude [§4, Thm. 3].
When learning rate is dropped by factor α mid-training (multi-stage schedule), the equilibrium weight norm scales by √α and requires at least O(1/(αηλ)) iterations to reach — explaining why late-stage angular updates in ImageNet training fall below their theoretical value: the schedule ends before the new equilibrium is reached [Corollary 3.1].
Manually rescaling ‖w‖ ← ‖w‖ · √α at each LR drop lets angular update stay at its theoretical value throughout multi-stage schedules on ImageNet ResNet-50 and MSCOCO Mask-RCNN [§5.2, Fig. 3].

Method

The core object is a scale-invariant weight vector w for which L(cw) = L(w), so ∇L(w) ⊥ w and ‖∇L(cw)‖ = ‖∇L(w)‖ / c. Define the unit gradient g_unit = ‖w‖ · ∇L(w), which is invariant to weight norm. An SGD step with weight decay produces two competing effects on ‖w‖²: the gradient part increases it (perpendicular to w, so via Pythagoras), the WD part decreases it (proportional to -w). The paper writes a recursion for ‖w_{t+1}‖² in terms of ‖w_t‖², η, λ, and ‖g_unit,t‖², then proves that under Assumptions {LR-WD small; local constancy of E[‖g_unit‖²]; bounded variance / lower bound}, the recursion is a contraction toward the fixed point η·E[‖g_unit‖²]/(2λ) with contraction rate 1 - 2ηλ (SGD) or 1 - 2ηλ/(1-β) (SGDM). The angular update result then follows from small-Δθ approximation.

Results

On ImageNet + ResNet-50 with fixed LR (0.1) and WD (1e-4), SGDM reaches equilibrium ~10× faster than SGD, with slightly higher steady-state variance — matching the theoretical contraction-rate ratio (1-β)⁻¹ = 10 at β=0.9 [Fig. 2].
On the standard ImageNet multi-stage schedule (LR 0.1 → 0.01 → 0.001 → 0.0001), the empirical angular update in the last two stages falls below its theoretical value; the paper explains this as insufficient iterations to re-equilibrate after the LR drop, and verifies by showing that manually rescaling ‖w‖ ← ‖w‖ · √0.1 at each drop restores agreement [§5.2, Fig. 3(b,e)].
The same pattern replicates on MSCOCO Mask-RCNN (4× schedule) with SyncBN [§5.2, Fig. 3(d,e,f)].

Why it’s interesting

Provides the mechanistic foundation the rest of the wiki’s rotational-equilibrium / angular-update literature builds on. Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks extends SMD from SGDM to AdamW / Lion / Adam+L2 and identifies imbalanced rotation across neurons as the mechanistic reason AdamW beats Adam+L2. Downstream, the Fitting-paradigm paper How to Set the Learning Rate for Large-Scale Pre-training? explicitly cites the ≈0.2 per-module AdamW update magnitude “consistent with rotational equilibrium” as its explanation for why module-specific LRs no longer help at 4B-MoE scale, and the whole “constrain weight geometry so weight decay becomes redundant” cluster (Controlled LLM Training on Spectral Sphere SSO, Delphi: Scaling Laws That Extrapolate 300× Past the Fit AdamH, Modular Manifolds, Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model‘s WD∝LR² coupling) is a direct outgrowth of the SMD framing. The team-side skepticism (per the filing pointer) is worth naming explicitly: the angular-update interpretation studies one axis of the weight geometry (rotation) in isolation and may miss other dynamically important pieces; whether SMD analysis actually produces meaningfully better training or only ≈sub-percentage-point gains is a live question.