Skip to content
Bud's Wiki

Hyperparameter scaling laws

TL;DR

Section titled “TL;DR”

The recipe for setting learning rate (and adjacent hyperparameters: weight decay, init std, α\alpha-multiplier, weight-spectrum radius) at scale is contested across three positions on the wiki. Fitting: extrapolate a closed-form scaling law η∗(N,D)\eta^*(N, D) from cheap small-scale runs (How to Set the Learning Rate for Large-Scale Pre-training?). Transfer (µP and its variants): derive η\eta for the target run from asymptotic stability analysis — at fixed width via µTransfer, and at fixed adapter rank via µA (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning). Enforce alignment: constrain weights and updates spectrally so that µP’s stability guarantees actually hold during optimization, rather than relying on µP-prescribed LRs alone (Controlled LLM Training on Spectral Sphere). Each paper agrees that pure µP-prescribed LRs are insufficient at industrial scale; they disagree on whether the fix lives in the scaling law (Fitting), in the LoRA-config asymptotics (µA), or in the optimizer (SSO). Closely coupled with Training stability at scale — the two concepts share the µP-vs-modern-architecture debate but ask different questions (where does stability live? vs how do you pick the LR?).

Key claims

Section titled “Key claims”
  • The optimal LR under WSD’s constant-LR stable phase follows a clean power-law in model size NN and data size DD; a quadratic-in-log-LR fit at each (N,D)(N,D) combined with an L(D)L(D) power law gives a closed-form η∗(N,D)\eta^*(N, D) surface with R2>0.99R^2 > 0.99 (How to Set the Learning Rate for Large-Scale Pre-training? §3.1, Eqs. 2–6).
  • At 4B-total / 530M-active and 12B-total / 1.3B-active MoE pretraining on 500B tokens, the Fitting paradigm outperforms µTransfer at every measured checkpoint; post-WSD-decay gap is +1.28% MMLU / +2.23% CMMLU (How to Set the Learning Rate for Large-Scale Pre-training? Figs. 3, 4, 11).
    • Conflict: Controlled LLM Training on Spectral Sphere argues µP’s motivation still stands and that the Fitting-paradigm gap is symptomatic of Muon-style optimizers being only “half-aligned” with µP (controlling update spectra but letting weight spectra drift). The prescribed fix is to constrain both weights and updates to a per-module spectral sphere of radius R=Θ(dout/din)R = \Theta(\sqrt{d_\text{out}/d_\text{in}}), after which µP LR transfer holds again across widths (Controlled LLM Training on Spectral Sphere README §1, §4, Fig. mup_trans).
  • Under modern architectures the attention-logit blow-up that motivated µP is suppressed by QK-Norm alone; ablating QK-Norm restores the instability, so module-specific LRs (a µP prescription) yield negligible gain at 4B-MoE scale (How to Set the Learning Rate for Large-Scale Pre-training? §6.1–6.3, Figs. 5–9).
  • Per-module AdamW update magnitudes stay near 0.2 across all modules and layers throughout training (consistent with rotational equilibrium), giving a mechanistic explanation for why a single global LR suffices under modern normalization (How to Set the Learning Rate for Large-Scale Pre-training? §6.2, Fig. 7).
  • Training-data size DD affects internal-state stability more sharply than model size NN — the LR scaling exponent on DD is larger in magnitude than on NN, contradicting the typical NN-only framing (How to Set the Learning Rate for Large-Scale Pre-training? §6.4, Eq. 6, Fig. 10).
  • The µA framework extends µP-style asymptotic analysis to LoRA finetuning in the joint limit n,r→∞n, r \to \infty, yielding closed-form scaling rules for η\eta as a function of width and adapter rank under each (Init[A] vs Init[B]) × α\alpha configuration (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4, Corollaries 4.4, 4.6).
  • µA identifies a configuration — Init[B] (random BB, A=0A=0) with constant α=1\alpha=1 — where the optimal LoRA LR is rank-invariant and coincides with the optimal FFT LR, enabling direct LR transfer from a cheap LoRA sweep to FFT (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.2, §5.2, Fig. 5).
  • µA identifies the rank-dependence regime explicitly: under Init[A] with α=1\alpha=1, optimal LR scales as r−1/2r^{-1/2}, requiring re-tuning whenever rank changes — observed across Llama-3.2-1B, RoBERTa-large, Qwen2.5-3B, ViT-H/14, Qwen3-VL-2B (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.1, Fig. 3).
  • µA scaling rules generalize beyond SFT: same rank-invariant/-dependent patterns hold under RLVR and Stable Diffusion finetuning, and suboptimal LRs cost 4–6× wall-clock training time in RLVR (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §5.3–5.4, Figs. 6, 7).
  • Constraining both weights and updates to a spectral sphere via SSO outperforms AdamW and Muon on Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet, with µP LR transfer preserved across widths and bounded activations as a side-effect (Controlled LLM Training on Spectral Sphere README §4, Fig. mup_trans, Fig. control_act).
  • Muon is only “half-aligned” with µP: it controls update spectra via Newton–Schulz orthogonalization but places no constraint on weight spectra, so weights drift away from the µP-prescribed scale during training (Controlled LLM Training on Spectral Sphere README §1). The MuonSphere variant retains equivalent activation control with minimal compute overhead.
  • SDE reparameterization of AdamW eliminates the optimal-batch-size search. Under CompleteP + SDE, training loss is invariant to physical batch size up to a critical threshold BcritB_{\rm crit}, so batch size is no longer a tuned hyperparameter — and TcritT_{\rm crit} (iteration count below which the SDE approximation holds) is empirically independent of model size at fixed token budget (The Design Space of Tri-Modal Masked Diffusion Models §4.1, §5.1, Fig. 5). A new isonoise/isohorizon interpolation parameter γ∗≈0.50\gamma^* \approx 0.50 governs how extra tokens are split between reducing SDE drift and extending horizon — both endpoints prior literature used (γ=0\gamma=0 Kaplan, γ=1\gamma=1 CompleteP) are suboptimal (The Design Space of Tri-Modal Masked Diffusion Models §5.2, Eq. 4.1, Figs. 9–9).
  • Compute-optimal token-per-parameter ratio is regime-dependent: AR Chinchilla gives D∗∝ND^* \propto N (TPP constant); tri-modal MDM gives D∗∝N0.476D^* \propto N^{0.476}, so larger MDMs are asymptotically more data-efficient per parameter — a 3B MDM wants ~250B tokens vs ~60B for an AR LLM at the same parameter count (The Design Space of Tri-Modal Masked Diffusion Models §5.3, Eq. 5.2, Fig. 3).
  • The Kaplan multiplicative scaling-law form fits tri-modal MDMs better than the Chinchilla additive form. Across 262 isoFLOP runs spanning TPP ratios 1–2000, the Kaplan form yields R2=99.3%R^2 = 99.3\% / MRE 0.5%; the additive form fits poorly (The Design Space of Tri-Modal Masked Diffusion Models §5.3, Eq. 5.1, Fig. 12).
  • The optimal expert-vs-attention FLOPs ratio r∗r^* in MoE follows a power law in total compute, with sparsity-dependent coefficients. Prior MoE scaling laws (Ludziejewski 2024; Abnar 2025; Wang 2024) only vary expert count, granularity, and sparsity while implicitly fixing rr; making rr an explicit scaling dimension yields a closed-form r∗(C,S)r^*(C, S) that increases with CC — i.e., as compute grows, allocate proportionally more to expert (FFN) layers and less to attention (Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design §1, §3, Fig. 5).
  • Layer looping during training is a parameter-free FLOPs-scaling axis with its own predictable scaling law. Applying a shared block of layers iteratively scales FLOPs without scaling parameters; the headline qualitative claim is a 2× parameter-equivalent quality match at the same parameter budget, with the prescription that data and loop count should be scaled in tandem at fixed parameter budget (Layer-looping scaling laws — Hayden Prairie tweet). Visual-generation analog with the same qualitative scaling has been demonstrated at FID parity in ELT: Elastic Looped Transformers for Visual Generation (4× parameter reduction at iso-inference-compute on ImageNet-256; power-law scaling in unique-layer count LbL_b). Open whether the data × loop-count surface from Prairie generalizes to visual generation, and whether ELT’s LbL_b axis is the same axis as Prairie’s loop count.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Scaling Behavior Foundation Model for Humanoid Robots: ScaleBFM (Zeng et al., Shanghai AI Lab + collaborators) is the first filed scaling study on RL-pretrained humanoid Behavior Foundation Models, extending Kaplan/Chinchilla-style analysis beyond LLM pretraining and manipulation VLAs to whole-body control. Identifies three coordinated levers: (a) on-policy rollout width × depth must scale jointly — single-axis scaling does not consistently help; (b) heterogeneous reference-motion diversity drives OOD gains while homogeneous scale drives only ID gains; (c) a Transformer policy beats MLPs at matched parameter counts with mode-dependent saturation. Complements the manipulation-side EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data “scale + diversity” result on egocentric video, and preempts the still-unreleased Sunday Robotics VLA Chinchilla claim (Sunday Robotics ran the Chinchilla scaling law with their VLA model (Tony Zhao follow-up)) with a concrete whole-body-control instance. Reports >10% MPKPE reduction in local mode and 82% in global mode over prior humanoid controllers.
  • [2026-07-20] Understanding Reasoning from Pretraining to Post-Training: Shen et al. extend the Fitting-style “predict downstream from cheap runs” methodology from pretraining loss to post-training RL return: across a 5M–1B chess-pretrained sweep, post-RL performance at matched RL compute is well-predicted from pretraining loss, and the slope of the RL reward curve improves approximately linearly with pretraining tokens. Reproduces on a 1B math-text run. Adds a post-training-return axis to this page (previously all entries are pretraining-loss or LR-transfer axes) and gives a chess-based controlled testbed where the pretraining corpus, reasoning traces, and RL rewards can each be swept independently — cleaner than the standard LLM setup for isolating each stage’s contribution to downstream capability.
  • [2026-07-17] Sunday Robotics ran the Chinchilla scaling law with their VLA model (Tony Zhao follow-up): Sunday Robotics co-founder Tony Zhao claims (no numbers, no writeup) that they ran the “exact chinchilla-scaling-law” on their VLA model. Would be the first filed VLA joint (N,D)(N,D) isoFLOP fit if delivered — a sharper result than the existing log-linear data-axis fits from EgoScale (EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data, R²=0.9983) and XR-1 (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training), and a direct test of whether AR LLMs’ D∗∝ND^*\propto N transfers to VLA training (versus e.g. tri-modal MDM’s D∗∝N0.476D^*\propto N^{0.476} from The Design Space of Tri-Modal Masked Diffusion Models).
  • [2026-07-16] Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training: Xiaomi-Robotics-1 (XR-1) reports the first filed instance where an LLM-style pre-train scaling curve on a VLA transfers cleanly through post-training to real-robot success — validation action error decreases monotonically with UMI pre-training data + model size, and real-robot success on unseen environments after post-training rises “steadily and predictably” with pre-training compute, “with no signs of saturation.” Sharpens EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data‘s log-linear loss-vs-hours result (R²=0.9983) by showing the scaling gain survives the embodiment-alignment stage rather than only appearing on the validation loss.
  • [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling’s stability recipe operationalizes an LR-schedule-invariant weight-decay rule: coupling weight decay to LR² (as suggested by prior modular-manifolds work) is reported to keep the overall size of model weights stable across training horizons. Concrete frontier-scale datapoint (975B/41B, 45T tokens) that hyperparameter coupling (rather than isolated per-knob tuning) is the stability lever teams are shipping in 2026.
  • [2026-07-13] Modular Duality in Deep Learning: Bernstein & Newhouse show that µP (Yang & Hu 2021) and Shampoo (Gupta et al. 2018) are both partial approximations to a single duality map: the √(d_out/d_in)·UVᵀ map induced by the RMS→RMS operator norm on Linear layers. The √(d_out/d_in) prefactor recovers µP’s spectral update scaling and the UVᵀ projection is Shampoo without accumulation. Frames µP not as a parametrization recipe but as the update-scaling half of the modular duality map; a paper’s µP-transfer behavior is then determined by which pieces of the composition (norm choice, module mass, orthogonalization primitive) it actually implements.
  • [2026-07-09] Scaling with Collapse: Efficient and Predictable Training of LLM Families: Bergsma et al. (Cerebras) extend Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks to LLM scale with joint width/depth/batch/weight-decay scaling, identifying three controls governing normalized-TLC shape under µP: AdamW normalized timescale τ=ηλBs/Ds\tau = \eta\lambda B_s/D_s, tokens-per-parameter ratio, and LR schedule. Collapse emerges only when τ\tau is fixed across sizes at its TPP-optimal value — Llama-2 (decreasing-TPP family) fails to collapse, Celerity 300M–3.9B at fixed TPP does. Adds a live HPO-tuning consequence: fixing τ\tau preserves loss-curve ordering throughout training, and a small-scale-fit parametric predictor L(f)=f−AG(f,sched;w)L(f)=f^{-A}G(f,\text{sched};w) with A(τ)A(\tau)/w(TPP)w(\text{TPP}) power laws enables reliable HPO winner-selection at 10-30% of training. Together with Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks, this makes the Cerebras/Google-Research “timescale + collapse” line an alternative to the Fitting scaling-law paradigm — same predictive goal, dynamics-based rather than closed-form.
  • [2026-07-09] Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks: Qiu et al. show compute-optimally trained models exhibit loss-curve scaling collapse: after affinely rescaling reducible loss and compute to unity at training end, curves across model sizes fall onto one universal shape. With LR decay, collapse tightens below the seed-to-seed noise floor (supercollapse). Derives collapse from sum-of-power-law scaling laws and a noisy-quadratic SGD model; shows supercollapse-deviation is a sharper mis-parameterization diagnostic than final-loss comparison (breaks under constant-LR-across-widths or wrong data exponent even where final losses look indistinguishable). Adds whole-loss-curve prediction as a validation axis on top of the scalar Fitting/Transfer/Enforce fork already on the page.
  • [2026-07-05] Implications of Large-Scale Test-Time Compute (Noam Brown): Noam Brown argues that inference-compute is the scaling-law axis current benchmark reporting practice under-represents: the capability ceiling for modern LLMs is often unknown because runs are cut off at time/infra limits rather than a genuine plateau, so single-scalar leaderboards misrepresent the model. Proposes replacing scalar reports with performance-vs-cost / tokens / time curves — the same “score as a function of compute” framing this concept applies to pre-training extended into the inference regime.
  • [2026-07-03] How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size: Schaipp proposes a three-term scaling law L(N, T, B) that keeps training steps T and batch size B as separate axes (rather than lumping them into D=T·B); recovers optimal-batch-size scaling and matches prior critical-batch-size empirical findings as downstream consequences of a single fit. Crucially, the fit consumes suboptimal-batch runs and needs significantly fewer runs than Chinchilla-style methodology. Complements Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling‘s schedule-equivalence proof and directly addresses the “batch size as a free hyperparameter” Open Question on this page.
  • [2026-07-01] To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters: Dragutinović, Zhang & Ranganath analyze Muon (via an exact-SVD, no-momentum abstraction called Spectral GD) on 2-layer deep linear networks and prove Spectral GD learns all singular components of the product matrix W₂W₁ in parallel at equal rate, while GD’s saddle-to-saddle dynamics sequentially learn largest-σ-first — explaining Muon’s wall-clock speedup as a loss of the rank-monotone simplicity bias GD naturally has. Constructs two toy settings where SGD beats Muon: (a) a Saxe-style multi-source “routing” task where SGD generalizes to unseen source pairs by learning the 4-D shared underlying task while stochastic Spectral GD memorizes each pair (high effective rank, heavy-tailed hidden spectrum); (b) MNIST + per-class spurious pixel where SGD’s dominant-first learning gives a longer clean-digit window before spurious features saturate. Adds an inductive-bias axis to the wiki’s Muon debate that sits below both the µP-alignment critique (Controlled LLM Training on Spectral Sphere SSO, How Muon Lost Its Geometry) and the “geometry doesn’t matter” critique (Muon is Not That Special: Random or Inverted Spectra Work Just as Well).
  • [2026-06-30] EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: EgoScale (NVIDIA GEAR) reports the first clean log-linear scaling law on the embodied-data axis: across {1k, 2k, 4k, 10k, 20k} hours of action-labeled egocentric pretraining video, optimal validation loss on held-out human action prediction fits R²=0.9983 with no saturation; offline loss strongly tracks real-robot dexterous-manipulation success rate (0.30 → 0.71 average task completion). Extends the Kaplan/Chinchilla style data-axis fits beyond text into the egocentric-video pretraining substrate and gives a predictable budget calculus for human-data-scale decisions.
  • [2026-06-30] How Muon Lost Its Geometry: Arda Göreci (Ligo) traces how Muon’s per-matrix scaling factor diverged from the µP-correct fan-out/fan-in\sqrt{\text{fan-out}/\text{fan-in}} as Muon scaled up: Keller’s speedrun variant clips the correction with max⁡(1,⋅)\max(1, \cdot), Moonshot’s Moonlight variant replaces it with max⁡(fan-in,fan-out)\sqrt{\max(\text{fan-in}, \text{fan-out})} (only equivalent when fan-in is uniform), and torch.optim.Muon ships both wrong options as user-selectable defaults. Microsoft’s Dion is the one popular variant that gets it right. A CIFAR-10 bottlenecked-MLP LR-transfer sweep across 5 widths shows the correct factor produces no LR drift while Moonshot’s drifts the optimum to lower LRs as width grows. Substantiates Controlled LLM Training on Spectral Sphere‘s “Muon is half-aligned with µP” claim with a sharper, one-line-fix version: the update-side half is itself wrong as implemented.
  • [2026-06-26] Scaling Laws, Carefully: Lilian Weng’s long-overdue Lil’Log primer on (N,D,C)(N, D, C) scaling laws — consolidates Kaplan vs. Chinchilla, the Pearce-Song embedding-fraction reconciliation, Hernandez/Muennighoff/Lovelace on the data-limited regime, and Besiroglu et al.’s Chinchilla method-3 replication bug. Useful canonical reference for the data-axis baseline that this concept page’s optimizer-axis and architecture-axis entries extend.
  • [2026-05-26] Laguna M.1/XS.2 Technical Report: Laguna fits a WSD-specific lr*(N,D) = 10^4.488·N^-0.4639·D^-0.2661 across 4 MoE sizes × 6 LRs × 5 token budgets at fixed 8M-token batch, then √(B/B₀)-scales to the 24M-batch production run; predicts 5.5e-4 for XS.2, deployed at 5e-4 — production-scale validation of the How to Set the Learning Rate for Large-Scale Pre-training? methodology.
  • [2026-05-24] Layer-looping scaling laws — Hayden Prairie tweet: tweet-form announcement of layer-looping scaling laws for transformer pretraining — predictable scaling laws exist for loop count, a looped transformer matches the quality of a 2× larger transformer at the same parameter budget, and the prescription is to scale data and loop count together at fixed parameter budget. Adds a parameter-free FLOPs-scaling axis to the page that complements the existing optimizer/LR axes (Fitting/Transfer/Enforce), the data axis (Kaplan/Chinchilla), and the architecture axis (MoE expert/attention FLOPs ratio). Loop count is also already an established scaling-law dimension in the visual-generation setting via ELT: Elastic Looped Transformers for Visual Generation.
  • [2026-05-23] Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design: introduces the expert-vs-attention FLOPs ratio rr as a first-class scaling-law dimension for MoE. Sweeps GPT-style MoE Transformers across compute scales and sparsity levels, finds r∗r^* follows a power law in total compute CC with sparsity-dependent coefficients, and derives an explicit closed-form r∗(C,S)r^*(C, S). Headline qualitative finding: r∗r^* rises with compute, contradicting the standard practice of holding a fixed FFN/attention dim ratio across scales. Sits in the architecture-axis of scaling laws (vs. the LR-axis covered by the existing entries) and is the cleanest scaling claim adjacent to the Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method / DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks “make MoE design less arbitrary” cluster.
  • [2026-05-23] The Design Space of Tri-Modal Masked Diffusion Models: Apple’s tri-modal MDM design-space study extends CompleteP with an SDE-based reparameterization that makes training loss invariant to physical batch size up to BcritB_{\rm crit} — eliminating the batch-size search entirely below threshold. Introduces an isonoise/isohorizon interpolation γ\gamma controlling how extra tokens are allocated between reducing SDE drift and extending horizon; finds γ∗≈0.50\gamma^* \approx 0.50, with both prior-literature endpoints suboptimal. Fits multiplicative Kaplan-form scaling laws on 262 runs at R2=99.3%R^2 = 99.3\%, deriving D∗∝N0.476D^* \propto N^{0.476} — i.e. larger MDMs are more data-efficient per parameter, opposite to AR LLMs.
  • [2026-05-23] Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning: introduces Maximal-Update Adaptation (µA), the LoRA analogue of µP, characterizing optimal-LR scaling jointly in model width and adapter rank. Identifies Init[B] + α=1\alpha=1 as the unique configuration where LoRA’s optimal LR transfers directly to FFT, plus a separate “learning-efficiency imbalance” diagnostic showing that one of the two LoRA factors contributes essentially nothing to feature learning under each named configuration. Verified across SFT (5 models), RLVR, and Stable Diffusion finetuning.
  • [2026-05-23] Controlled LLM Training on Spectral Sphere: introduces SSO, an optimizer that retracts both weights and updates onto a per-module spectral sphere; argues Muon is half-aligned with µP and that closing the loop restores µP transfer. Beats AdamW and Muon on Dense 1.7B / MoE 8B-A1B / 200-layer DeepNet with µP LR transfer preserved across widths.
  • [2026-05-22] How to Set the Learning Rate for Large-Scale Pre-training?: formalizes the Fitting paradigm (closed-form η∗(N,D)\eta^*(N, D) from cheap small-scale WSD runs); beats µTransfer at 4B and 12B MoE scale on 500B tokens. The QK-Norm ablation is the load-bearing experiment for the “µP’s stability motivation is stale” claim.

Open questions

Section titled “Open questions”
  • Fitting vs Transfer vs Enforce is a three-way fork that hasn’t been benchmarked end-to-end. Fitting wins at 4B/12B MoE pretraining when the architecture is QK-Norm-stable; SSO claims µP transfer holds when weight+update spectra are constrained; µA shows µP-style analysis pays off for LoRA finetuning. No filed paper has tested whether SSO restores µTransfer’s edge over Fitting at the 4B/12B scale where Fitting beats vanilla µTransfer.
  • Architecture-side scaling laws need to be co-fit with optimizer-side ones. Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design shows r∗r^* (FFN/attention FLOPs allocation) scales with CC, but every other entry on this page holds architecture fixed and tunes the optimizer. Open: does the Fitting paradigm’s η∗(N,D)\eta^*(N, D) surface depend on whether the architecture is at r∗r^* or off it? Conversely, does the prescribed r∗r^* depend on whether the LR is at η∗\eta^*? Joint scaling-law fits over (N,D,r,η)(N, D, r, \eta) have not been done.
  • Does loop count enter the Fitting scaling-law surface? Layer-looping scaling laws — Hayden Prairie tweet proposes a data × loop-count rule at fixed parameter budget; ELT: Elastic Looped Transformers for Visual Generation reports power-law scaling in unique-layer count LbL_b at fixed width. Whether the same LL axis governs both LLM pretraining and visual generation, and whether the optimal LR η∗\eta^* has a clean dependence on LL, is open.
  • Does µA transfer to DiT / MMDiT finetuning? Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning runs Stable Diffusion (UNet) and finds the Init[B] + α=1\alpha=1 rule holds with FFT being sharply LR-sensitive on UNets. The MMDiT case (with AdaLN modulations as an extra outlier source) is untested. This is the most directly testable claim from this concept on Luma’s stack.
  • What’s the right default LoRA init? The Slack thread that filed Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning notes that Luma’s current default is Init[B] with α=1/r\alpha=1/r — covered in µA’s Appendix A.5 but not one of the three named configurations in the paper’s headline workflow. The clean action is to switch defaults to Init[B] + α=1\alpha=1 for finetuning, then transfer LR to FFT — but the appendix’s analysis of the 1/r1/r variant should be read first.
  • Module-specific LRs: How to Set the Learning Rate for Large-Scale Pre-training? §6.1 says they don’t help at 4B-MoE scale under QK-Norm; µA’s “learning-efficiency imbalance” claim suggests one LoRA factor contributes negligibly to feature updates, raising the question of whether per-factor LRs (in the spirit of LoRA+) are still warranted even when per-module LRs are not. No filed paper has done the joint experiment.
  • µTransfer’s effective horizon is short (How to Set the Learning Rate for Large-Scale Pre-training? §3.2); extending it requires weight-decay scaling and token-horizon-aware init. Whether SSO’s spectral retraction implicitly handles this horizon issue is open.
  • Does the Fitting recipe transfer to diffusion/flow pretraining? η∗(N,D)\eta^*(N, D) quadratic-in-log-LR + power-law-in-D is cheap enough to run on diffusion pretraining sweeps; the Universal One-third Time Scaling in Learning Peaked Distributions universal-exponent result suggests softmax-CE-specific behaviors may not apply, so the slope and possibly the scaling-law form could differ for MSE / flow-matching objectives. Partial answer from The Design Space of Tri-Modal Masked Diffusion Models: for discrete-token masked diffusion (still softmax-CE), the Kaplan multiplicative form fits well and D∗∝N0.476D^* \propto N^{0.476} — strictly more data-hungry per-parameter than AR. Open whether the same shape carries to continuous-token DiT/flow-matching pretraining where the objective is MSE rather than CE.
  • Is “batch size as a free hyperparameter” robust to the optimizer choice? The Design Space of Tri-Modal Masked Diffusion Models‘s SDE-reparam result is derived for AdamW. Whether the same physical-vs-logical batch-size decoupling holds under Muon / SSO / Lion is open, and the result directly affects how the Fitting / SSO / SDE recipes interact.
  • The “learning-efficiency imbalance” in µA — that under both Init[A] and Init[B] (with r≪nr \ll n) one LoRA factor contributes negligibly — suggests that the standard 2-factor LoRA factorization is over-parameterized for the rank regime in actual use. Is there a single-factor equivalent (random projection + trainable head) that captures the same dynamics with half the parameters? Open across all filed papers.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-03

Section titled “2025-03”

2025-02

Section titled “2025-02”

2024-01

Section titled “2024-01”

2020-12

Section titled “2020-12”

Unknown date

Section titled “Unknown date”