Effect of the initial configuration of weights on the training and function of artificial neural networks

TL;DR

A 2020 statistical-physics-flavored empirical study of how far SGD-trained two-hidden-layer ReLU networks travel from their random initialization, across MNIST / Fashion-MNIST / HASYv2 and a wide range of widths (10 → 1000). The headline claim: successful training leaves the network in the immediate neighborhood of its initial weights — and as width is reduced past a dataset-dependent threshold (~300 nodes here), the network crosses from a trainable regime (small RMSD from init, decreasing loss) to an untrainable one (large RMSD, diverging loss), with a sharp abrupt transition for midsize “unstable learners.” Frames this as evidence for a Lottery-Ticket-Hypothesis-style picture: training is fine-tuning of features that already exist in the initial random configuration; overparameterization works because random init contains good subnetworks, and shrinks the work each weight has to do.

Key claims

Successful SGD training of overparameterized two-hidden-layer ReLU networks leaves weights in close neighborhood of their random init; large initial weights tend to grow further in the same sign, with most weights concentrated near the $w_{\rm final} = w_{\rm init}$ line [§4, Fig. 3].
A network’s width determines its training regime — there are three classes: strong learners (large nets, always high-trainability, small RMSD), weak learners (small nets, low-trainability, monotonically growing RMSD, never reach low loss), and unstable learners (midsize nets, initially low-RMSD/trainable then abruptly crossing to untrainable with simultaneous loss spike and RMSD jump) [§5, Fig. 2, Fig. 5].
The crossover from trainability to untrainability for midsize networks coincides — in training time and in shape — with the abrupt RMSD jump, locating the failure mechanism in the loss of the initial configuration’s structure [§5, Fig. 5].
Width-dependent phase diagram of training regimes: the transition threshold is approximately 300 nodes per hidden layer for the dataset/architecture studied; above the threshold networks stabilize in trainability as $t \to \infty$ ; below it they always end up in untrainability given enough time [§5, Fig. 2(c)].
“Stamped letter” experiment: zeroing the initial weights outside a letter-shaped mask leaves the mark visible after 1000 epochs in large networks without degrading loss, and produces no measurable training penalty — the mark survives because most weights barely move [§4, Fig. 1(a)].
For midsize unstable networks, the same letter-mask is erased over the course of training, and its erasure tracks the loss divergence [§4, Fig. 1(b)].
Larger networks reach a given loss threshold with smaller average per-weight displacement — interpreted as overparameterization reducing the optimization “work” each parameter has to do [§5, Fig. 6].
Frames the picture in terms of the Lottery Ticket Hypothesis (Frankle & Carbin 2019) and the Li & Liang 2018 / Jacot et al. 2018 lazy-training / NTK lineage: trainability is the presence of good subnetworks in the random init; below the over-parameterization threshold, the random init no longer contains a winning ticket [§2.1, §2.2, Conclusions].

Method

The setup is small and deliberate: feedforward MLPs with two ReLU hidden layers of width $w \in [10, 1000]$ , Glorot-uniform init, biases at zero, softmax output, categorical cross-entropy, SGD with LR 0.1 and batch size 128, trained for 1000 epochs on MNIST / Fashion-MNIST / HASYv2. The key measurements are (i) the per-weight final-vs-initial distribution (Fig. 3), (ii) the RMSD trajectory $\sigma(t) = \sqrt{N^{-1}\sum_{ij}(w_{ij}(t) - w_{ij}(0))^2}$ vs. the loss trajectory across widths (Fig. 5), and (iii) the three characteristic training times (test-loss minimum, train-loss minimum, loss-divergence onset) plotted against width to build the phase diagram (Fig. 2). The “stamped letter” experiment is a qualitative visualization: rasterize a letter onto the weight matrix of one layer as a binary mask, zero everything outside, train, observe whether the mask is visible after training.

Results

Trainability threshold: ~300 nodes per hidden layer for Fashion-MNIST and HASYv2 on this architecture; below this width, networks always end up in untrainability given enough training time [§5, Fig. 2(c)].
At width 10 (deep weak-learner regime) on Fashion-MNIST, no network reaches loss 0.1 in 1000 epochs; at width 100 networks reach losses three orders of magnitude smaller [§5, Fig. 6].
Width-1000 (strong-learner) networks: RMSD remains small and approximately constant after the early training phase; train loss approaches zero; “letter” stamps remain clearly visible at epoch 1000 [§4, Fig. 1(a); §5, Fig. 5].
Midsize unstable networks on HASYv2: train loss decreases for hundreds of epochs, then jumps by orders of magnitude essentially in a single step, with the RMSD jumping simultaneously to a plateau [§5, Fig. 5, middle panels].
The result is consistent across all three datasets, ruling out task-specific artifacts and supporting the universality of the trainability/untrainability crossover [§4–5].

Why it’s interesting

The paper is a 2020 throwback in a 2026 context: the wiki’s training-stability cluster (Training stability at scale, Hyperparameter scaling laws) has located stability successively in architecture (QK-Norm in How to Set the Learning Rate for Large-Scale Pre-training?), in residual geometry (mHC: Manifold-Constrained Hyper-Connections), and in the optimizer (Controlled LLM Training on Spectral Sphere) — but every one of those papers takes the initial weight distribution as a given and tunes everything downstream of it. This paper sits one step earlier in the pipeline: it makes the empirical case that the random init itself is what determines whether a network is trainable, by showing that successful training does not move far from init, and that loss of init structure and loss of trainability happen simultaneously. The Slack note framing — given µP 4 and the other new optimisers in circulation, whether initialization deserves more attention — is the right read: the µP / Muon / SSO / mHC stack has been arguing about how to preserve a regime that this paper says is set at $t=0$ . Direct connections: complements Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning‘s “Init[B] + $\alpha=1$ gives rank-invariant optimal LR” by suggesting init scale is a load-bearing knob even in the LoRA regime; the Lottery-Ticket framing it builds on is also the conceptual ancestor of Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights‘s “diverse task experts are dense around pretrained weights” claim — the local-minimum-near-init story extended from random init to pretrained init.