Simplifying Adam: Bias Correction Debunked

TL;DR

Adam’s bias-correction term — the $1/(1-\beta^t)$ divisors applied to the first and second moment estimates — is empirically unnecessary when training with a proper learning-rate schedule and language-model-optimal smoothing $(\beta_1, \beta_2)$ . Through ablations on 124M-parameter language-model pretraining (SlimPajama, 10B tokens) and ResNet/ViT vision training, the authors show bias correction acts as an implicit, often-clumsy LR warm-up whose shape depends sharply on $\beta$ . At $\beta_1 = \beta_2 = 0.95$ (the LM-pretraining optimum from Orvieto et al.’s “Adam’s secret sauce” sweep), bias correction injects a large early-step LR spike that warmup-cosine scheduling absorbs but a constant LR does not — degrading final perplexity. They advocate removing the term from both implementations and convergence analyses.

Key claims

In the LM-optimal hyperparameter setting $(\beta_1, \beta_2) = (0.95, 0.95)$ with warmup-cosine scheduling, including or omitting bias correction produces indistinguishable final validation perplexity across the entire LR sweep on 124M-param SlimPajama pretraining [Fig. 1, §2 LM training].
Without explicit LR scheduling (constant LR), bias correction is detrimental at $\beta_1 = \beta_2 = 0.95$ — the early-step spike in the effective LR destabilizes training and worsens final loss [Fig. 1, §3].
The bias-correction factor $\sqrt{1-\beta_2^t}/(1-\beta_1^t)$ modulates the effective LR as an implicit schedule whose shape is highly $\beta$ -dependent: torch-default $(0.9, 0.999)$ produces a gradual thousand-step warm-up, while $(0.95, 0.95)$ produces a large initial spike that decays rapidly to baseline [§3, Fig. 2].
With explicit warmup-cosine scheduling, the bias-correction spike is absorbed for $\beta = 0.95$ but dampens the cosine peak LR for $\beta = 0.999$ — so even when bias correction “doesn’t hurt”, it modifies the schedule the practitioner actually sees [§3].
The torch-default $(0.9, 0.999)$ setting is precisely the regime where bias correction’s implicit warm-up helps with no scheduling, which the authors offer as the mechanistic source of bias correction’s reputation as a stabilizer [§3, §4].
The result generalizes across vision: ResNet9/CIFAR-10 and ResNet+ViT/TinyImageNet at $(0.9, 0.999)$ and $(0.95, 0.95)$ , both constant and warmup-cosine LR, three seeds each — bias correction can be removed without test-accuracy loss given a solid pipeline [Appendix A].
Across a sweep of $(\beta_1 = \beta_2) \in \{0.9, 0.95, 0.99, 0.999\}$ with constant LR, including bias correction always worsens optimum performance, with the discrepancy growing in $\beta$ — consistent with the bias-correction factor decaying more slowly toward baseline at higher $\beta$ [Appendix B, Figs. 6–7].
The standard textbook justification — that $E[\hat m_t] = E[g_t]$ under the assumption $E[g_\tau] \approx E[g_t]$ for $\tau < t$ — fails in practice unless the LR warm-up is so gradual that early steps barely move in the loss landscape, undermining the analytical motivation for the term [§1 “A Discussion of the Proof”].

Method

Two ablation sweeps on (a) a 124M-parameter, 12-layer transformer (hidden 768, RMSNorm, SwiGLU, FlashAttention, RoPE) trained on 10B SlimPajama tokens with decoupled weight decay, global gradient clipping, batch 480, and (b) ResNet9 on CIFAR-10 / ResNet+ViT on TinyImageNet. For each $(\beta_1, \beta_2)$ pair and each LR schedule (constant vs warmup-cosine), they run the LR grid with bias correction on and off and compare final validation perplexity / test accuracy. Appendix B extends the LM study to $\beta_1 = \beta_2 \in \{0.9, 0.95, 0.99, 0.999\}$ . The mechanistic explanation re-derives the bias-correction factor as $\sqrt{1-\beta_2^t}/(1-\beta_1^t)$ multiplying the effective LR (Eq. 3) and plots its time course at each $\beta$ .

Results

LM pretraining @ 124M, 10B tokens, warmup-cosine: best val perplexity is identical with/without bias correction at $\beta=0.95$ across the LR sweep [Fig. 1].
LM pretraining, constant LR: bias correction strictly worse at $\beta=0.95$ ; bias correction strictly better at $\beta=0.999$ — the crossover happens because the implicit-warmup shape (gradual ramp vs early spike) flips between the two regimes [Fig. 1, §3].
Across $\beta \in \{0.9, 0.95, 0.99, 0.999\}$ with constant LR, the no-bias-correction curve dominates everywhere; the gap grows monotonically in $\beta$ [Fig. 6].
Vision results (3 seeds each): ResNet9/CIFAR-10 and ResNet+ViT/TinyImageNet show no test-accuracy degradation from removing bias correction under either scheduling [Appendix A].

Why it’s interesting

This is a clean negative-result entry into the “remove the tuning knob entirely” lineage on the wiki — MuonClip / QK-Norm (QK-Clip: Taking Muon Further on the Scaleup Journey), Quantile Balancing for MoE load balancing (Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method), and the Fitting paradigm replacing µTransfer in How to Set the Learning Rate for Large-Scale Pre-training? — except instead of a knob being replaced it’s being deleted. The mechanistic framing (bias correction is implicit warmup whose shape depends on $\beta$ ) directly couples to Why Gradients Rapidly Increase Near the End of Training and the broader story that schedule-shape choices interact with optimizer state in ways the textbook derivation didn’t anticipate. Most directly relevant to Luma practice: any internal training stack still relying on bias correction and the torch-default $(0.9, 0.999)$ to provide warmup-by-accident is one schedule change away from the “constant LR @ 0.95” failure mode this paper isolates.