Skip to content
Bud's Wiki

Hyperparameter scaling laws

TL;DR

Section titled “TL;DR”

The recipe for setting learning rate (and adjacent hyperparameters: weight decay, init std, α\alpha-multiplier, weight-spectrum radius) at scale is contested across three positions on the wiki. Fitting: extrapolate a closed-form scaling law η∗(N,D)\eta^*(N, D) from cheap small-scale runs (How to Set the Learning Rate for Large-Scale Pre-training?). Transfer (µP and its variants): derive η\eta for the target run from asymptotic stability analysis — at fixed width via µTransfer, and at fixed adapter rank via µA (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning). Enforce alignment: constrain weights and updates spectrally so that µP’s stability guarantees actually hold during optimization, rather than relying on µP-prescribed LRs alone (Controlled LLM Training on Spectral Sphere). Each paper agrees that pure µP-prescribed LRs are insufficient at industrial scale; they disagree on whether the fix lives in the scaling law (Fitting), in the LoRA-config asymptotics (µA), or in the optimizer (SSO). Closely coupled with Training stability at scale — the two concepts share the µP-vs-modern-architecture debate but ask different questions (where does stability live? vs how do you pick the LR?).

Key claims

Section titled “Key claims”
  • The optimal LR under WSD’s constant-LR stable phase follows a clean power-law in model size NN and data size DD; a quadratic-in-log-LR fit at each (N,D)(N,D) combined with an L(D)L(D) power law gives a closed-form η∗(N,D)\eta^*(N, D) surface with R2>0.99R^2 > 0.99 (How to Set the Learning Rate for Large-Scale Pre-training? §3.1, Eqs. 2–6).
  • At 4B-total / 530M-active and 12B-total / 1.3B-active MoE pretraining on 500B tokens, the Fitting paradigm outperforms µTransfer at every measured checkpoint; post-WSD-decay gap is +1.28% MMLU / +2.23% CMMLU (How to Set the Learning Rate for Large-Scale Pre-training? Figs. 3, 4, 11).
    • Conflict: Controlled LLM Training on Spectral Sphere argues µP’s motivation still stands and that the Fitting-paradigm gap is symptomatic of Muon-style optimizers being only “half-aligned” with µP (controlling update spectra but letting weight spectra drift). The prescribed fix is to constrain both weights and updates to a per-module spectral sphere of radius R=Θ(dout/din)R = \Theta(\sqrt{d_\text{out}/d_\text{in}}), after which µP LR transfer holds again across widths (Controlled LLM Training on Spectral Sphere README §1, §4, Fig. mup_trans).
  • Under modern architectures the attention-logit blow-up that motivated µP is suppressed by QK-Norm alone; ablating QK-Norm restores the instability, so module-specific LRs (a µP prescription) yield negligible gain at 4B-MoE scale (How to Set the Learning Rate for Large-Scale Pre-training? §6.1–6.3, Figs. 5–9).
  • Per-module AdamW update magnitudes stay near 0.2 across all modules and layers throughout training (consistent with rotational equilibrium), giving a mechanistic explanation for why a single global LR suffices under modern normalization (How to Set the Learning Rate for Large-Scale Pre-training? §6.2, Fig. 7).
  • Training-data size DD affects internal-state stability more sharply than model size NN — the LR scaling exponent on DD is larger in magnitude than on NN, contradicting the typical NN-only framing (How to Set the Learning Rate for Large-Scale Pre-training? §6.4, Eq. 6, Fig. 10).
  • The µA framework extends µP-style asymptotic analysis to LoRA finetuning in the joint limit n,r→∞n, r \to \infty, yielding closed-form scaling rules for η\eta as a function of width and adapter rank under each (Init[A] vs Init[B]) × α\alpha configuration (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4, Corollaries 4.4, 4.6).
  • µA identifies a configuration — Init[B] (random BB, A=0A=0) with constant α=1\alpha=1 — where the optimal LoRA LR is rank-invariant and coincides with the optimal FFT LR, enabling direct LR transfer from a cheap LoRA sweep to FFT (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.2, §5.2, Fig. 5).
  • µA identifies the rank-dependence regime explicitly: under Init[A] with α=1\alpha=1, optimal LR scales as r−1/2r^{-1/2}, requiring re-tuning whenever rank changes — observed across Llama-3.2-1B, RoBERTa-large, Qwen2.5-3B, ViT-H/14, Qwen3-VL-2B (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.1, Fig. 3).
  • µA scaling rules generalize beyond SFT: same rank-invariant/-dependent patterns hold under RLVR and Stable Diffusion finetuning, and suboptimal LRs cost 4–6× wall-clock training time in RLVR (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §5.3–5.4, Figs. 6, 7).
  • Constraining both weights and updates to a spectral sphere via SSO outperforms AdamW and Muon on Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet, with µP LR transfer preserved across widths and bounded activations as a side-effect (Controlled LLM Training on Spectral Sphere README §4, Fig. mup_trans, Fig. control_act).
  • Muon is only “half-aligned” with µP: it controls update spectra via Newton–Schulz orthogonalization but places no constraint on weight spectra, so weights drift away from the µP-prescribed scale during training (Controlled LLM Training on Spectral Sphere README §1). The MuonSphere variant retains equivalent activation control with minimal compute overhead.
  • SDE reparameterization of AdamW eliminates the optimal-batch-size search. Under CompleteP + SDE, training loss is invariant to physical batch size up to a critical threshold BcritB_{\rm crit}, so batch size is no longer a tuned hyperparameter — and TcritT_{\rm crit} (iteration count below which the SDE approximation holds) is empirically independent of model size at fixed token budget (The Design Space of Tri-Modal Masked Diffusion Models §4.1, §5.1, Fig. 5). A new isonoise/isohorizon interpolation parameter γ∗≈0.50\gamma^* \approx 0.50 governs how extra tokens are split between reducing SDE drift and extending horizon — both endpoints prior literature used (γ=0\gamma=0 Kaplan, γ=1\gamma=1 CompleteP) are suboptimal (The Design Space of Tri-Modal Masked Diffusion Models §5.2, Eq. 4.1, Figs. 9–9).
  • Compute-optimal token-per-parameter ratio is regime-dependent: AR Chinchilla gives D∗∝ND^* \propto N (TPP constant); tri-modal MDM gives D∗∝N0.476D^* \propto N^{0.476}, so larger MDMs are asymptotically more data-efficient per parameter — a 3B MDM wants ~250B tokens vs ~60B for an AR LLM at the same parameter count (The Design Space of Tri-Modal Masked Diffusion Models §5.3, Eq. 5.2, Fig. 3).
  • The Kaplan multiplicative scaling-law form fits tri-modal MDMs better than the Chinchilla additive form. Across 262 isoFLOP runs spanning TPP ratios 1–2000, the Kaplan form yields R2=99.3%R^2 = 99.3\% / MRE 0.5%; the additive form fits poorly (The Design Space of Tri-Modal Masked Diffusion Models §5.3, Eq. 5.1, Fig. 12).
  • The optimal expert-vs-attention FLOPs ratio r∗r^* in MoE follows a power law in total compute, with sparsity-dependent coefficients. Prior MoE scaling laws (Ludziejewski 2024; Abnar 2025; Wang 2024) only vary expert count, granularity, and sparsity while implicitly fixing rr; making rr an explicit scaling dimension yields a closed-form r∗(C,S)r^*(C, S) that increases with CC — i.e., as compute grows, allocate proportionally more to expert (FFN) layers and less to attention (Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design §1, §3, Fig. 5).
  • Layer looping during training is a parameter-free FLOPs-scaling axis with its own predictable scaling law. Applying a shared block of layers iteratively scales FLOPs without scaling parameters; the headline qualitative claim is a 2× parameter-equivalent quality match at the same parameter budget, with the prescription that data and loop count should be scaled in tandem at fixed parameter budget (Layer-looping scaling laws — Hayden Prairie tweet). Visual-generation analog with the same qualitative scaling has been demonstrated at FID parity in ELT: Elastic Looped Transformers for Visual Generation (4× parameter reduction at iso-inference-compute on ImageNet-256; power-law scaling in unique-layer count LbL_b). Open whether the data × loop-count surface from Prairie generalizes to visual generation, and whether ELT’s LbL_b axis is the same axis as Prairie’s loop count.

Recent contributions

Section titled “Recent contributions”
  • [2026-08-27] Liam Fedus reflects on Switch Transformers — 1.6T-param / 2048-expert MoE was ahead on perplexity, behind on reasoning: Liam Fedus retrospective frames the compute-optimal tokens-per-parameter ratio as task-dependent (Switch Transformer 1.6T beat T5 on perplexity + TriviaQA but lost on SuperGLUE), invoking Shazeer’s “FLOPs are intelligence; parameters are knowledge” — a historical version of the multi-axis scaling-law point made by Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design and Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies.
  • [2026-08-27] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning: Ablates four independent scaling axes (data 2M→22M, model L→g, iterations 90k→252k, resolution 256²/16 → 384²/64 cool-down) that each contribute cumulatively to +4.0 avg understanding; progressive-resolution cool-down cuts total compute 8× vs full-resolution training.
  • [2026-08-27] LoRA Without Regret: Thinking Machines Lab measures the LoRA↔FullFT optimal-LR ratio empirically across 14 (model × Tulu3) sweeps on Llama/Qwen — fitted multiplier is 9.8× and the hidden-size exponent matches FullFT’s, so the ratio is model-independent. Preliminary evidence puts the multiplier at ~15× for short runs (<~100 steps), converging to ~10× for long runs. Under the standard Huggingface peft parametrization (Init[B] + α/r, both LRs equal) the optimal LoRA LR is (approximately) rank-invariant — varies by <2× between rank 4 and rank 512. Complements Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning‘s theoretical prescription (Init[B] + α=1 gives ratio 1×): both agree on rank-invariance; the 10× vs 1× discrepancy is exactly the α-schedule choice µA identifies as load-bearing.
  • [2026-08-26] Isaac 0.5: An Open-Weight Embodied Foundation Model: Isaac 0.5 measures the video-vs-teleop exchange on a well-calibrated offline action-loss surface (τ=2.50): scaling general action-free video from 1,000 h to 1,000,000 h at fixed 80:30:30 general:egocentric:UMI composition moves the within-grid teleoperation crossing from 5,884 h → 28 h — a 210.3× reduction, bounded 83× to 300× by adjacent measured teleop rungs. The exchange is conditional: empirical loss-vs-video slope is nearly flat at 1 teleop-h and stabilizes at −0.21 loss per 10× video from ~100 teleop-h onward. First embodied scaling law to (a) name general action-free video (not just egocentric) as the primary axis and (b) fit a well-calibrated loss-threshold rather than a downstream success rate — a methodologically sharper cousin of EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale’s R²=0.9983 log-linear egocentric-hours-vs-loss law and of Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2’s 1M-h scaling curve.
  • [2026-08-26] Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding: Qwen3.8-Flash-Next’s training recipe applies Muon and AdamW to category-specific weight groups and, “guided by refitted scaling laws”, eliminates traditional batch-size warmup and starts directly at the target batch size — Qwen’s team claims this substantially reduces total optimizer steps while safely admitting larger learning rates for robust convergence.
  • [2026-08-26] Behavioral Cloning Mystery: Park’s Mystery 3 stakes out an anomalous model-size datapoint on the scaling-laws board: even a fixed, single-task, state-based (37-dim) block pick-and-place task under infinite-data flow BC needs at least a [4096]×8[4096]\times 8 residual MLP to work well and [8192]×8[8192]\times 8 still improves — nearly 0.5B params for a task that classical BC/RL benchmarks (D4RL, OGBench) solve with [1024]×4[1024]\times 4 or smaller. The effect reproduces under autoregressive tokenized action prediction, so it is not a flow-matching artifact. Together with Park’s larger-dataset-can-hurt result (10K episodes beats 50K from the same distribution, 4 seeds × 4 datasets), this is a concrete anti-Chinchilla-shaped datapoint: for BC on non-Markovian narrowly-distributed demo data, the “smaller data ⇒ smaller model” heuristic fails and the required model size is decoupled from the state-space dimension. Suggests VLA action experts cannot be shrunk even when perception + language are handled elsewhere.
  • [2026-08-25] S1: In-Context Learning for Robotics: S1 (Skild AI) ships a controlled task-conditioning-interface scaling law between 1k and 100k pretraining hours at matched architectures, data, and compute — the first filed scaling law that treats the prompt interface (video demonstration vs language instruction) as the axis of interest rather than parameter count, data volume, or optimizer choice. Two headline findings: (i) crossover between language-conditioned VLA (53% at 1k hours seen tasks, ahead of 43% ICL) and ICL happens within the sweep, with ICL winning by 100k hours; (ii) language-conditioned prompting plateaus at 9% on unseen tasks by 100k hours while ICL reaches 66% — a 7× gap that grows with scale, arguing language is a wrong-turn interface for the OOD regime rather than a weaker one. Data-quality budget: 3onQCper3 on QC per 1 on collection (“every data point screened for low-level precision, task coherence, and annotation fidelity”). Complements the model/data/optimizer scaling-law entries on the page with an interface axis, and gives a data-side counterweight to the “language conditioning is a wrong-turn interface” position on VLA Models.
  • [2026-08-24] Scaling Domain Data Repetition in LLM Pretraining: Li et al. add a per-domain repetition axis to the fixed-TPP scaling recipe: at fixed tokens-per-parameter, the optimal repetition count for each domain rises mildly with model size, is strongly negatively correlated with the domain’s final validation loss (easier domains tolerate more repetition), and is only weakly related to unique-data volume. The practical claim is that a proxy-model repetition sweep at the target TPP transfers to larger runs — an analogue of µA’s “cheap-LoRA-sweep transfers to FFT” recipe on the data-mixture side. Complements Internal Data Repetition Destroys Language Models (the damage half) by covering the benefit half of the same repetition-structure design space.
  • [2026-08-24] Internal Data Repetition Destroys Language Models: Chudnovsky, Kazdan, Levi & Schaeffer et al. modernize Hernandez et al. 2022 for the Chinchilla era: at fixed 10% repeated-token fraction, eval loss is non-monotonic in per-document repeat count RR, and the peak location scales as R∗∝N−0.66R^\ast \propto N^{-0.66} (worst-case pool grows as U∗∝N1.66U^\ast \propto N^{1.66} — faster than compute). Translated via a fitted no-repetition Chinchilla law, the 344M worst-case repetition structure costs ~33% of training compute (a 2-4% loss bump maps to CEGCEG = 0.67-0.85 because the no-rep curve is shallow). A closed-form misspecified linear regression reproduces the same non-monotonic peak, suggesting the phenomenon is a generic statistical feature of duplicated samples rather than a transformer-specific artifact — a prediction directly testable on the diffusion-pretraining side.
  • [2026-08-20] Marin scaling laws simulate the entire training trajectory of a 67B MoE run within 1%: Marin team member Larry Dial reports the Delphi Fitting recipe now predicts full training trajectories to within 1% at 67B MoE scale — a strictly stronger claim than Delphi: Scaling Laws That Extrapolate 300× Past the Fit‘s terminal-loss result — and has already used the same ladder to forecast the trajectory of a 535B MoE run in flight. If validated, closes ground with the dynamics-based whole-curve prediction of Scaling with Collapse: Efficient and Predictable Training of LLM Families from the closed-form Fitting side.
  • [2026-08-20] GEN-0: Embodied Foundation Models That Scale with Physical Interaction: GEN-0 (Generalist AI, Nov 2025) reports the first filed observation of ossification in robotics — LLM-style loss of ability to absorb new information as training data grows — at ~O(1B) parameters, three orders of magnitude larger than the O(10M) scale prior LLM literature identified it at. Frames the ~7B ossification/absorption phase transition as consistent with Moravec’s paradox: physical commonsense has a higher compute activation threshold than abstract reasoning. Also fits a power law L(D)=a D−b+cL(D)=a\,D^{-b}+c on downstream post-training loss vs pretraining data size and shows the curve transfers to real-robot success under blind A/B — adds an embodied-data axis to the wiki’s Chinchilla-style entries and to Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models‘s parallel million-hour egocentric-video scaling result, on a real-manipulation substrate. No fit constants are disclosed in the post.
  • [2026-08-19] A lower bound for stepsize-based acceleration of gradient descent: Ma & Chen prove a new Ω(T−1.9319)\Omega(T^{-1.9319}) lower bound on last-iterate convergence of gradient descent with any predetermined nonnegative stepsize schedule on smooth convex problems — ruling out that stepsize-only scheduling can reach Nesterov’s Ω(T−2)\Omega(T^{-2}) optimal first-order rate. Draws a strict theoretical ceiling above the current O(T−log⁡2(1+2))≈O(T−1.2716)O(T^{-\log_2(1+\sqrt{2})}) \approx O(T^{-1.2716}) upper bound from Altschuler–Parrilo / Grimmer et al., leaving only a T 0.07T^{~0.07}-factor gap for further stepsize-alone improvement in the convex-worst-case model that the empirical LR-shape literature (e.g. What do near-optimal learning rate schedules look like?) implicitly baselines against.
  • [2026-08-17] Measuring Autonomous AI Research: 153 autonomous nanoGPT-speedrun runs converge on the same optimizer-scaling ingredients — better preconditioning, weight/update magnitude caps, longer hot LR schedules, end-of-training weight averaging — across 18 different frontier models with no internet access, providing evidence that the modded-nanogpt optimizer frontier is a stable attractor under LLM-driven search, not a set of tricks unique to any one team.
  • [2026-08-14] Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling: Over-Tokenized Transformer decouples input and output vocabularies and identifies a log-linear scaling law: training loss decreases linearly as input-vocabulary size increases exponentially, holding across 151M–1B dense models and OLMoE-1.3B/7B MoE up to Vin∼107V_{in} \sim 10^7 entries with no saturation (~0.015 nats per doubling). A 400M OT-12.8M matches a 1B baseline at the same wall-clock cost; OLMo2-1B convergence acceleration 5.7× on loss, 2.6–3.9× on five downstream benchmarks. Complements Tao et al.’s Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies joint-VV compute-optimal fit by showing the input axis alone has essentially unbounded scaling room, and refines Compute Optimal Tokenization‘s tokenizer-agnostic framing with a concrete architectural instantiation (hierarchical nn-gram tiled hashing at the input).
  • [2026-08-14] Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies: Tao et al. add vocabulary size VV as a first-class scaling-law axis under a fixed BPE tokenizer family. Three independent methods (IsoFLOPs, FLOP-derivative, parametric loss fit) converge on Nv∝C0.27N_v \propto C^{0.27} vs non-vocab Nnv∝C0.49N_{nv} \propto C^{0.49} — vocabulary scales strictly slower than non-vocabulary parameters. Predicts Llama 2-70B should have had ≥216K vocabulary (7× its 32K); verifies at 3B that switching 32K → 43K improves ARC-Challenge 29.1 → 32.0 at matched 2.3e21 FLOPs. Also identifies the regime-dependence: optimum VV rises under overtraining (Llama-style) and falls under undertraining (data-scarce), the empirical shape that Compute Optimal Tokenization later restates as “optimal compression rate decreases with compute” in the BLT frame.
  • [2026-08-11] Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization: Hyperball (Wen, Dang, Lyu, Ma, Liang) — the arxiv writeup of the AdamH/MuonH recipe already deployed in Delphi: Scaling Laws That Extrapolate 300× Past the Fit. Directly relevant to this page: LR transfer under Hyperball is essentially width- and depth-invariant (maximal drift ≈5% for AdamH/MuonH vs 24% AdamW / 40% MuonW at large depth on Qwen3-style pretraining) because pinning ‖W‖_F directly severs the WD-based coupling that made optimal LR shift with model geometry. Theory (§4.2) proves the mechanism: under scale-invariant weight blocks, decoupled WD sets an equilibrium radius ρ* which sets angular step η/ρ*; by fixing ρ, Hyperball turns η into an explicit angular LR that transfers by construction. Removes WD from the hyperparameter search entirely and applies to any base optimizer (Adam, Muon, NorMuon, KL-SOAP validated on modded-nanogpt Track 3).
  • [2026-08-10] Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models: Dyna-2 (Dyna Robotics) extends the embodied-data scaling-law axis to 1M+ hours of egocentric human video (50× EgoScale’s 20k h) and reports monotone power-law fits y = a·N^{-b} on four metrics (MSE, L1, acc@0.5, acc@0.1) with no plateau. First filed cross-substrate transfer scaling law: same human-only pretraining checkpoints show monotone improvement on 39 held-out YAM robot tasks with zero robot data in pretraining, plus an emergence-like inflection between 10k–100k h. Also reports the pretraining scale carries through to real hardware after post-training (20% → 28% → 45% → 53% of max normalized score across the ladder on 14 tasks). Uses Schaeffer’s “emergence-is-a-mirage” multi-metric guard on all curves.
  • [2026-08-07] Scaling Native Multimodal Pre-Training From Scratch: Decouples the multimodal isoFLOP fit into a separate language allocation law and multimodal allocation law, fitted via IsoFLOP profiles and training-curve envelopes; the two estimators agree. Headline: the language law is composition-invariant — language exponents barely move as rmmr_\mathrm{mm} varies — while the multimodal law is strongly composition-variant, with the parameter exponent aa collapsing (0.54 → 0.42) and the token exponent bb rising (0.46 → 0.58) as rmmr_\mathrm{mm} grows from 0.05 to 0.7. Sharper than Beyond Language Modeling: An Exploration of Multimodal Pretraining‘s single-fit “vision more data-hungry” result: the asymmetry lives entirely in the multimodal leg, so unified pretraining reduces to two decoupled Chinchilla-style problems once rmmr_\mathrm{mm} is chosen. Also gives a concrete joint Pareto frontier over compute × rmmr_\mathrm{mm} that outputs deployable (N∗,Dtext∗,Dmm∗)(N^*, D_\mathrm{text}^*, D_\mathrm{mm}^*) configurations.
  • [2026-08-07] Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes: Three-axis (Language / Understanding / Generation) data-mixture grid search at 1T tokens finds a sharply asymmetric compute-optimal split — language ~70%, understanding ~25%, generation ~5% — that validates at 13.5B-MoE / 2T tokens against an L50/U25/G25 balanced baseline (Full beats Balanced on every axis: PPL 11.67 vs 11.97, GenEval 0.482 vs 0.467, DPG 0.689 vs 0.676) despite 5× fewer generation tokens. Complements the IsoFLOP scaling-asymmetry finding of Beyond Language Modeling: An Exploration of Multimodal Pretraining (vision more data-hungry than language per parameter) with the data-mixture allocation counterpart: even though vision is data-hungry per parameter, generation quality is largely inherited from language + understanding foundations, so most of the token budget should go there.
  • [2026-08-06] Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks: Kosson, Messmer & Jaggi (2023) extend SMD to AdamW / Adam+L2 / Lion and derive per-optimizer closed-form equilibrium ‖w‖ and Δθ. Key HP-scaling consequence: AdamW’s equilibrium Δθ is HP-only, while Adam+L2’s depends on gradient RMS — explaining Adam+L2’s imbalanced per-layer rotation (~1000× effective-LR variation) and mechanistically grounding AdamW’s now-standard use in the Fitting-paradigm scaling recipes. Their Rotational Variant also demonstrates that LR warmup partially substitutes for enforcing rotational equilibrium — the transient rotation instability warmup was implicitly suppressing is eliminated by construction under RV, foreshadowing Delphi: Scaling Laws That Extrapolate 300× Past the Fit‘s AdamH weight-sphere retraction and the Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model WD∝LR² coupling.
  • [2026-08-06] Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD: Backfill of the SMD foundation the rotational-equilibrium framing on this page rests on. Wan et al. (2020) prove that for scale-invariant weights under SGDM+WD, ‖w‖² converges to η·E[‖g_unit‖²]/(2λ) at a linear rate independent of loss convergence, and the equilibrium angular update √(2ηλ/(1-β)) depends only on hyperparameters. How to Set the Learning Rate for Large-Scale Pre-training?‘s “per-module AdamW update magnitude ≈0.2” mechanistic explanation and the argument for a single global LR under modern normalization both trace back to this equilibrium.
  • [2026-08-03] Scaling Properties of Text Conditioning in Visual Generation: Chen et al. add a prompt-structuredness axis to the diffusion scaling-law taxonomy: converged T2I diffusion loss is flat in raw natural-language token count but decreases approximately linearly with a white-box structuredness metric (GPG) and as a power law with a black-box attribute metric (ED). Complementary to Scaling Laws For Diffusion Transformers‘s (N, D) isoFLOP fit and Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers‘s three-axis fit — those hold prompt distribution fixed; this paper sweeps the input distribution and reports the resulting scaling exponents as first-class quantities. Provides a scaling-law rationale for the “long caption + inference-time prompt rewriting” empirical recipe already documented in i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models and HunyuanImage 2.1 — Efficient 2K Text-to-Image with 32× VAE + DINOv2-Aligned REPA and MeanFlow Distillation‘s PromptEnhancer.
  • [2026-07-31] Scaling Laws For Diffusion Transformers: Liang et al. (Shanghai AI Lab, Oct 2024) run the first Chinchilla-style isoFLOP sweep on text-to-image DiTs across 1e17–6e18 FLOPs, fitting N∗∝C0.56N^* \propto C^{0.56}, D∗∝C0.43D^* \propto C^{0.43}, and a loss exponent of −0.0273-0.0273 (in-context DiT), then validate the fit by matching an extrapolated 1B / 1.5e21-FLOP run — a ~250× compute extrapolation. Model size grows slightly faster than data with compute, opposite to The Design Space of Tri-Modal Masked Diffusion Models‘s tri-modal MDM finding (D∗∝N0.476D^* \propto N^{0.476}) and closer to but not identical to LLM Chinchilla. Also proposes using scaling exponents themselves as a benchmark for architecture / data quality — the methodological ancestor of later “isoFLOP fit tells you if your change is good” framings on this page. Foundational reference; predates most of the current DiT scaling-law wave.
  • [2026-07-31] Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers: Chimera introduces HeteroP — a module-wise µP variant that transfers hyperparameters per tensor according to its functional fan-in and depth position, rather than applying a single width-transfer factor across the whole network. Designed to handle a heterogeneous KDA/MLA/conv/MoE stack where no single fan-in convention applies. HeteroP-tuned reference runs then fit a three-axis Chinchilla law over activated parameters, training tokens, and image-vs-video data ratio — one of the first filed instances of a modality-mixture dimension fit jointly with (N, D). Compute-optimal image pretraining splits nearly evenly between size and tokens; video pretraining modestly favors size at higher budgets.
  • [2026-07-30] A Defense of the Quadratic Model: Meterez et al. give a mechanistic account of the LR–B coupling that Fitting/Transfer scaling recipes measure phenomenologically: local linear-stability probes at intermediate checkpoints of a 150M/3B-token run show LLM training sits at a stochastic edge of stability at small batch and rotates onto the deterministic edge of stability (η · λ_max ≈ 2) at large batch, matching linear-stability theory. The Hessian’s power-law tail is invariant to both batch size and the Adam preconditioner, so what changes with B is which stability boundary the optimizer is pushed against, not the underlying spectrum. This suggests LR* scaling laws are implicitly measuring where along a fixed stability manifold the optimum lives — a cleaner framing than raw (η, B) sweeps.
  • [2026-07-28] Online KL Shampoo (OKLS) — Tilde Research's KL-optimal full-matrix AdaGrad approximation optimizer with 1.45× parameter efficiency over Muon: Tilde Research’s OKLS optimizer reports a scaling-law fit across 79M–1.2B non-embedding parameters × 20B tokens showing a median 1.46× parameter-efficiency advantage over Muon — restated as “an OKLS run matches a Muon run ~1.5× the parameter count.” First public Kronecker-factored-preconditioner-vs-Muon scaling comparison at ≥1B scale; the exact fit form and exponent are behind the login-walled X-article writeup at filing time.
  • [2026-07-27] Is Diversity All You Need for Scalable Robotic Manipulation?: Shi et al. fit a downstream-score power law y=A−Bx−Cy = A - Bx^{-C} over pre-training-data scale under fixed task diversity, reporting Pearson r=0.9998r = 0.9998 across {no-pretrain, 100K, 250K, 1M} demos on GO-1 with scores 0.28 → 0.58. Complements EgoScale’s log-linear egocentric-hours fit (R2=0.9983R^2 = 0.9983, EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data) and XR-1’s UMI-hour scaling (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training) with a third clean VLA data-axis fit — this one specifically on the teleoperated multi-task substrate. Cross-embodiment fine-tuning also shows a power-law relationship with fine-tuning data size on ManiSkill (RDT-AWB’s slope beats RDT-OXE’s), which is a rare filed instance of a fine-tuning-side scaling law on VLAs rather than a pre-training-side one.
  • [2026-07-27] Delphi: Scaling Laws That Extrapolate 300× Past the Fit: Delphi (Will Held / Marin, on TPU Research Cloud) is the strongest recent public Fitting-paradigm instance: a scaling law fit on IsoFLOP buckets from 3e18 to 3e20 FLOPs forecasts a 1e23-FLOP / 25B / 600B-token held-out run within 0.2%, a 300× extrapolation. The load-bearing recipe changes over an initial diverging attempt are (a) a (T0/T)0.3(T_0/T)^{0.3} token-horizon correction on projection LR alongside the usual B/B0\sqrt{B/B_0}, and (b) switching to AdamH — a Frobenius-norm-sphere retraction that removes weight decay from the recipe (nothing to regularize away) and, per Kaiyue Wen’s separate scaling sweeps, transfers hyperparameters across width more cleanly than Adam. Also releases the downstream forecasting pipeline: fit a scaling law on a soft metric (correct-answer log-prob for MMLU, reference-completion bits-per-byte for HumanEval), then compose with a sigmoid observational projection fit on public models to recover discrete benchmark scores — extending the Fitting recipe from pretraining loss to task performance and giving small-scale ablations a usable downstream signal even below emergence thresholds. Complements How to Set the Learning Rate for Large-Scale Pre-training? as an open-weights public realization, and the AdamH weight-sphere retraction sits alongside Controlled LLM Training on Spectral Sphere (spectral sphere) and How Muon Lost Its Geometry (µP-correct Muon scaling factor) as geometric-constraint variants of the same “remove degrees of freedom from the HP search” idea.
  • [2026-07-20] Scaling Behavior Foundation Model for Humanoid Robots: ScaleBFM (Zeng et al., Shanghai AI Lab + collaborators) is the first filed scaling study on RL-pretrained humanoid Behavior Foundation Models, extending Kaplan/Chinchilla-style analysis beyond LLM pretraining and manipulation VLAs to whole-body control. Identifies three coordinated levers: (a) on-policy rollout width × depth must scale jointly — single-axis scaling does not consistently help; (b) heterogeneous reference-motion diversity drives OOD gains while homogeneous scale drives only ID gains; (c) a Transformer policy beats MLPs at matched parameter counts with mode-dependent saturation. Complements the manipulation-side EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data “scale + diversity” result on egocentric video, and preempts the still-unreleased Sunday Robotics VLA Chinchilla claim (Sunday Robotics ran the Chinchilla scaling law with their VLA model (Tony Zhao follow-up)) with a concrete whole-body-control instance. Reports >10% MPKPE reduction in local mode and 82% in global mode over prior humanoid controllers.
  • [2026-07-20] Understanding Reasoning from Pretraining to Post-Training: Shen et al. extend the Fitting-style “predict downstream from cheap runs” methodology from pretraining loss to post-training RL return: across a 5M–1B chess-pretrained sweep, post-RL performance at matched RL compute is well-predicted from pretraining loss, and the slope of the RL reward curve improves approximately linearly with pretraining tokens. Reproduces on a 1B math-text run. Adds a post-training-return axis to this page (previously all entries are pretraining-loss or LR-transfer axes) and gives a chess-based controlled testbed where the pretraining corpus, reasoning traces, and RL rewards can each be swept independently — cleaner than the standard LLM setup for isolating each stage’s contribution to downstream capability.
  • [2026-07-17] Sunday Robotics ran the Chinchilla scaling law with their VLA model (Tony Zhao follow-up): Sunday Robotics co-founder Tony Zhao claims (no numbers, no writeup) that they ran the “exact chinchilla-scaling-law” on their VLA model. Would be the first filed VLA joint (N,D)(N,D) isoFLOP fit if delivered — a sharper result than the existing log-linear data-axis fits from EgoScale (EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data, R²=0.9983) and XR-1 (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training), and a direct test of whether AR LLMs’ D∗∝ND^*\propto N transfers to VLA training (versus e.g. tri-modal MDM’s D∗∝N0.476D^*\propto N^{0.476} from The Design Space of Tri-Modal Masked Diffusion Models).
  • [2026-07-16] Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training: Xiaomi-Robotics-1 (XR-1) reports the first filed instance where an LLM-style pre-train scaling curve on a VLA transfers cleanly through post-training to real-robot success — validation action error decreases monotonically with UMI pre-training data + model size, and real-robot success on unseen environments after post-training rises “steadily and predictably” with pre-training compute, “with no signs of saturation.” Sharpens EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data‘s log-linear loss-vs-hours result (R²=0.9983) by showing the scaling gain survives the embodiment-alignment stage rather than only appearing on the validation loss.
  • [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling’s stability recipe operationalizes an LR-schedule-invariant weight-decay rule: coupling weight decay to LR² (as suggested by prior modular-manifolds work) is reported to keep the overall size of model weights stable across training horizons. Concrete frontier-scale datapoint (975B/41B, 45T tokens) that hyperparameter coupling (rather than isolated per-knob tuning) is the stability lever teams are shipping in 2026.
  • [2026-07-13] Modular Duality in Deep Learning: Bernstein & Newhouse show that µP (Yang & Hu 2021) and Shampoo (Gupta et al. 2018) are both partial approximations to a single duality map: the √(d_out/d_in)·UVᵀ map induced by the RMS→RMS operator norm on Linear layers. The √(d_out/d_in) prefactor recovers µP’s spectral update scaling and the UVᵀ projection is Shampoo without accumulation. Frames µP not as a parametrization recipe but as the update-scaling half of the modular duality map; a paper’s µP-transfer behavior is then determined by which pieces of the composition (norm choice, module mass, orthogonalization primitive) it actually implements.
  • [2026-07-09] Scaling with Collapse: Efficient and Predictable Training of LLM Families: Bergsma et al. (Cerebras) extend Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks to LLM scale with joint width/depth/batch/weight-decay scaling, identifying three controls governing normalized-TLC shape under µP: AdamW normalized timescale τ=ηλBs/Ds\tau = \eta\lambda B_s/D_s, tokens-per-parameter ratio, and LR schedule. Collapse emerges only when τ\tau is fixed across sizes at its TPP-optimal value — Llama-2 (decreasing-TPP family) fails to collapse, Celerity 300M–3.9B at fixed TPP does. Adds a live HPO-tuning consequence: fixing τ\tau preserves loss-curve ordering throughout training, and a small-scale-fit parametric predictor L(f)=f−AG(f,sched;w)L(f)=f^{-A}G(f,\text{sched};w) with A(τ)A(\tau)/w(TPP)w(\text{TPP}) power laws enables reliable HPO winner-selection at 10-30% of training. Together with Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks, this makes the Cerebras/Google-Research “timescale + collapse” line an alternative to the Fitting scaling-law paradigm — same predictive goal, dynamics-based rather than closed-form.
  • [2026-07-09] Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks: Qiu et al. show compute-optimally trained models exhibit loss-curve scaling collapse: after affinely rescaling reducible loss and compute to unity at training end, curves across model sizes fall onto one universal shape. With LR decay, collapse tightens below the seed-to-seed noise floor (supercollapse). Derives collapse from sum-of-power-law scaling laws and a noisy-quadratic SGD model; shows supercollapse-deviation is a sharper mis-parameterization diagnostic than final-loss comparison (breaks under constant-LR-across-widths or wrong data exponent even where final losses look indistinguishable). Adds whole-loss-curve prediction as a validation axis on top of the scalar Fitting/Transfer/Enforce fork already on the page.
  • [2026-07-05] Implications of Large-Scale Test-Time Compute (Noam Brown): Noam Brown argues that inference-compute is the scaling-law axis current benchmark reporting practice under-represents: the capability ceiling for modern LLMs is often unknown because runs are cut off at time/infra limits rather than a genuine plateau, so single-scalar leaderboards misrepresent the model. Proposes replacing scalar reports with performance-vs-cost / tokens / time curves — the same “score as a function of compute” framing this concept applies to pre-training extended into the inference regime.
  • [2026-07-03] How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size: Schaipp proposes a three-term scaling law L(N, T, B) that keeps training steps T and batch size B as separate axes (rather than lumping them into D=T·B); recovers optimal-batch-size scaling and matches prior critical-batch-size empirical findings as downstream consequences of a single fit. Crucially, the fit consumes suboptimal-batch runs and needs significantly fewer runs than Chinchilla-style methodology. Complements Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling‘s schedule-equivalence proof and directly addresses the “batch size as a free hyperparameter” Open Question on this page.
  • [2026-07-01] To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters: Dragutinović, Zhang & Ranganath analyze Muon (via an exact-SVD, no-momentum abstraction called Spectral GD) on 2-layer deep linear networks and prove Spectral GD learns all singular components of the product matrix W₂W₁ in parallel at equal rate, while GD’s saddle-to-saddle dynamics sequentially learn largest-σ-first — explaining Muon’s wall-clock speedup as a loss of the rank-monotone simplicity bias GD naturally has. Constructs two toy settings where SGD beats Muon: (a) a Saxe-style multi-source “routing” task where SGD generalizes to unseen source pairs by learning the 4-D shared underlying task while stochastic Spectral GD memorizes each pair (high effective rank, heavy-tailed hidden spectrum); (b) MNIST + per-class spurious pixel where SGD’s dominant-first learning gives a longer clean-digit window before spurious features saturate. Adds an inductive-bias axis to the wiki’s Muon debate that sits below both the µP-alignment critique (Controlled LLM Training on Spectral Sphere SSO, How Muon Lost Its Geometry) and the “geometry doesn’t matter” critique (Muon is Not That Special: Random or Inverted Spectra Work Just as Well).
  • [2026-06-30] EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: EgoScale (NVIDIA GEAR) reports the first clean log-linear scaling law on the embodied-data axis: across {1k, 2k, 4k, 10k, 20k} hours of action-labeled egocentric pretraining video, optimal validation loss on held-out human action prediction fits R²=0.9983 with no saturation; offline loss strongly tracks real-robot dexterous-manipulation success rate (0.30 → 0.71 average task completion). Extends the Kaplan/Chinchilla style data-axis fits beyond text into the egocentric-video pretraining substrate and gives a predictable budget calculus for human-data-scale decisions.
  • [2026-06-30] How Muon Lost Its Geometry: Arda Göreci (Ligo) traces how Muon’s per-matrix scaling factor diverged from the µP-correct fan-out/fan-in\sqrt{\text{fan-out}/\text{fan-in}} as Muon scaled up: Keller’s speedrun variant clips the correction with max⁡(1,⋅)\max(1, \cdot), Moonshot’s Moonlight variant replaces it with max⁡(fan-in,fan-out)\sqrt{\max(\text{fan-in}, \text{fan-out})} (only equivalent when fan-in is uniform), and torch.optim.Muon ships both wrong options as user-selectable defaults. Microsoft’s Dion is the one popular variant that gets it right. A CIFAR-10 bottlenecked-MLP LR-transfer sweep across 5 widths shows the correct factor produces no LR drift while Moonshot’s drifts the optimum to lower LRs as width grows. Substantiates Controlled LLM Training on Spectral Sphere‘s “Muon is half-aligned with µP” claim with a sharper, one-line-fix version: the update-side half is itself wrong as implemented.
  • [2026-06-26] Scaling Laws, Carefully: Lilian Weng’s long-overdue Lil’Log primer on (N,D,C)(N, D, C) scaling laws — consolidates Kaplan vs. Chinchilla, the Pearce-Song embedding-fraction reconciliation, Hernandez/Muennighoff/Lovelace on the data-limited regime, and Besiroglu et al.’s Chinchilla method-3 replication bug. Useful canonical reference for the data-axis baseline that this concept page’s optimizer-axis and architecture-axis entries extend.
  • [2026-05-26] Laguna M.1/XS.2 Technical Report: Laguna fits a WSD-specific lr*(N,D) = 10^4.488·N^-0.4639·D^-0.2661 across 4 MoE sizes × 6 LRs × 5 token budgets at fixed 8M-token batch, then √(B/B₀)-scales to the 24M-batch production run; predicts 5.5e-4 for XS.2, deployed at 5e-4 — production-scale validation of the How to Set the Learning Rate for Large-Scale Pre-training? methodology.
  • [2026-05-24] Layer-looping scaling laws — Hayden Prairie tweet: tweet-form announcement of layer-looping scaling laws for transformer pretraining — predictable scaling laws exist for loop count, a looped transformer matches the quality of a 2× larger transformer at the same parameter budget, and the prescription is to scale data and loop count together at fixed parameter budget. Adds a parameter-free FLOPs-scaling axis to the page that complements the existing optimizer/LR axes (Fitting/Transfer/Enforce), the data axis (Kaplan/Chinchilla), and the architecture axis (MoE expert/attention FLOPs ratio). Loop count is also already an established scaling-law dimension in the visual-generation setting via ELT: Elastic Looped Transformers for Visual Generation.
  • [2026-05-23] Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design: introduces the expert-vs-attention FLOPs ratio rr as a first-class scaling-law dimension for MoE. Sweeps GPT-style MoE Transformers across compute scales and sparsity levels, finds r∗r^* follows a power law in total compute CC with sparsity-dependent coefficients, and derives an explicit closed-form r∗(C,S)r^*(C, S). Headline qualitative finding: r∗r^* rises with compute, contradicting the standard practice of holding a fixed FFN/attention dim ratio across scales. Sits in the architecture-axis of scaling laws (vs. the LR-axis covered by the existing entries) and is the cleanest scaling claim adjacent to the Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method / DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks “make MoE design less arbitrary” cluster.
  • [2026-05-23] The Design Space of Tri-Modal Masked Diffusion Models: Apple’s tri-modal MDM design-space study extends CompleteP with an SDE-based reparameterization that makes training loss invariant to physical batch size up to BcritB_{\rm crit} — eliminating the batch-size search entirely below threshold. Introduces an isonoise/isohorizon interpolation γ\gamma controlling how extra tokens are allocated between reducing SDE drift and extending horizon; finds γ∗≈0.50\gamma^* \approx 0.50, with both prior-literature endpoints suboptimal. Fits multiplicative Kaplan-form scaling laws on 262 runs at R2=99.3%R^2 = 99.3\%, deriving D∗∝N0.476D^* \propto N^{0.476} — i.e. larger MDMs are more data-efficient per parameter, opposite to AR LLMs.
  • [2026-05-23] Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning: introduces Maximal-Update Adaptation (µA), the LoRA analogue of µP, characterizing optimal-LR scaling jointly in model width and adapter rank. Identifies Init[B] + α=1\alpha=1 as the unique configuration where LoRA’s optimal LR transfers directly to FFT, plus a separate “learning-efficiency imbalance” diagnostic showing that one of the two LoRA factors contributes essentially nothing to feature learning under each named configuration. Verified across SFT (5 models), RLVR, and Stable Diffusion finetuning.
  • [2026-05-23] Controlled LLM Training on Spectral Sphere: introduces SSO, an optimizer that retracts both weights and updates onto a per-module spectral sphere; argues Muon is half-aligned with µP and that closing the loop restores µP transfer. Beats AdamW and Muon on Dense 1.7B / MoE 8B-A1B / 200-layer DeepNet with µP LR transfer preserved across widths.
  • [2026-05-22] How to Set the Learning Rate for Large-Scale Pre-training?: formalizes the Fitting paradigm (closed-form η∗(N,D)\eta^*(N, D) from cheap small-scale WSD runs); beats µTransfer at 4B and 12B MoE scale on 500B tokens. The QK-Norm ablation is the load-bearing experiment for the “µP’s stability motivation is stale” claim.

Open questions

Section titled “Open questions”
  • Fitting vs Transfer vs Enforce is a three-way fork that hasn’t been benchmarked end-to-end. Fitting wins at 4B/12B MoE pretraining when the architecture is QK-Norm-stable; SSO claims µP transfer holds when weight+update spectra are constrained; µA shows µP-style analysis pays off for LoRA finetuning. No filed paper has tested whether SSO restores µTransfer’s edge over Fitting at the 4B/12B scale where Fitting beats vanilla µTransfer.
  • Architecture-side scaling laws need to be co-fit with optimizer-side ones. Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design shows r∗r^* (FFN/attention FLOPs allocation) scales with CC, but every other entry on this page holds architecture fixed and tunes the optimizer. Open: does the Fitting paradigm’s η∗(N,D)\eta^*(N, D) surface depend on whether the architecture is at r∗r^* or off it? Conversely, does the prescribed r∗r^* depend on whether the LR is at η∗\eta^*? Joint scaling-law fits over (N,D,r,η)(N, D, r, \eta) have not been done.
  • Does loop count enter the Fitting scaling-law surface? Layer-looping scaling laws — Hayden Prairie tweet proposes a data × loop-count rule at fixed parameter budget; ELT: Elastic Looped Transformers for Visual Generation reports power-law scaling in unique-layer count LbL_b at fixed width. Whether the same LL axis governs both LLM pretraining and visual generation, and whether the optimal LR η∗\eta^* has a clean dependence on LL, is open.
  • Does µA transfer to DiT / MMDiT finetuning? Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning runs Stable Diffusion (UNet) and finds the Init[B] + α=1\alpha=1 rule holds with FFT being sharply LR-sensitive on UNets. The MMDiT case (with AdaLN modulations as an extra outlier source) is untested. This is the most directly testable claim from this concept on Luma’s stack.
  • What’s the right default LoRA init? The Slack thread that filed Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning notes that Luma’s current default is Init[B] with α=1/r\alpha=1/r — covered in µA’s Appendix A.5 but not one of the three named configurations in the paper’s headline workflow. The clean action is to switch defaults to Init[B] + α=1\alpha=1 for finetuning, then transfer LR to FFT — but the appendix’s analysis of the 1/r1/r variant should be read first.
  • Module-specific LRs: How to Set the Learning Rate for Large-Scale Pre-training? §6.1 says they don’t help at 4B-MoE scale under QK-Norm; µA’s “learning-efficiency imbalance” claim suggests one LoRA factor contributes negligibly to feature updates, raising the question of whether per-factor LRs (in the spirit of LoRA+) are still warranted even when per-module LRs are not. No filed paper has done the joint experiment.
  • µTransfer’s effective horizon is short (How to Set the Learning Rate for Large-Scale Pre-training? §3.2); extending it requires weight-decay scaling and token-horizon-aware init. Whether SSO’s spectral retraction implicitly handles this horizon issue is open.
  • Does the Fitting recipe transfer to diffusion/flow pretraining? η∗(N,D)\eta^*(N, D) quadratic-in-log-LR + power-law-in-D is cheap enough to run on diffusion pretraining sweeps; the Universal One-third Time Scaling in Learning Peaked Distributions universal-exponent result suggests softmax-CE-specific behaviors may not apply, so the slope and possibly the scaling-law form could differ for MSE / flow-matching objectives. Partial answer from The Design Space of Tri-Modal Masked Diffusion Models: for discrete-token masked diffusion (still softmax-CE), the Kaplan multiplicative form fits well and D∗∝N0.476D^* \propto N^{0.476} — strictly more data-hungry per-parameter than AR. Open whether the same shape carries to continuous-token DiT/flow-matching pretraining where the objective is MSE rather than CE.
  • Is “batch size as a free hyperparameter” robust to the optimizer choice? The Design Space of Tri-Modal Masked Diffusion Models‘s SDE-reparam result is derived for AdamW. Whether the same physical-vs-logical batch-size decoupling holds under Muon / SSO / Lion is open, and the result directly affects how the Fitting / SSO / SDE recipes interact.
  • The “learning-efficiency imbalance” in µA — that under both Init[A] and Init[B] (with r≪nr \ll n) one LoRA factor contributes negligibly — suggests that the standard 2-factor LoRA factorization is over-parameterized for the rank regime in actual use. Is there a single-factor equivalent (random projection + trainable head) that captures the same dynamics with half the parameters? Open across all filed papers.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-03

Section titled “2025-03”

2025-02

Section titled “2025-02”

2024-01

Section titled “2024-01”

2020-12

Section titled “2020-12”

Unknown date

Section titled “Unknown date”