Hyperparameter scaling laws

TL;DR

The recipe for setting learning rate (and adjacent hyperparameters: weight decay, init std, $\alpha$-multiplier, weight-spectrum radius) at scale is contested across three positions on the wiki. Fitting: extrapolate a closed-form scaling law $\eta^*(N, D)$ from cheap small-scale runs (How to Set the Learning Rate for Large-Scale Pre-training?). Transfer (µP and its variants): derive $\eta$ for the target run from asymptotic stability analysis — at fixed width via µTransfer, and at fixed adapter rank via µA (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning). Enforce alignment: constrain weights and updates spectrally so that µP’s stability guarantees actually hold during optimization, rather than relying on µP-prescribed LRs alone (Controlled LLM Training on Spectral Sphere). Each paper agrees that pure µP-prescribed LRs are insufficient at industrial scale; they disagree on whether the fix lives in the scaling law (Fitting), in the LoRA-config asymptotics (µA), or in the optimizer (SSO). Closely coupled with Training stability at scale — the two concepts share the µP-vs-modern-architecture debate but ask different questions (where does stability live? vs how do you pick the LR?).

Key claims

• The optimal LR under WSD’s constant-LR stable phase follows a clean power-law in model size $N$ and data size $D$; a quadratic-in-log-LR fit at each $(N,D)$ combined with an $L(D)$ power law gives a closed-form $\eta^*(N, D)$ surface with $R^2 > 0.99$ (How to Set the Learning Rate for Large-Scale Pre-training? §3.1, Eqs. 2–6).
• At 4B-total / 530M-active and 12B-total / 1.3B-active MoE pretraining on 500B tokens, the Fitting paradigm outperforms µTransfer at every measured checkpoint; post-WSD-decay gap is +1.28% MMLU / +2.23% CMMLU (How to Set the Learning Rate for Large-Scale Pre-training? Figs. 3, 4, 11).
  • Conflict: Controlled LLM Training on Spectral Sphere argues µP’s motivation still stands and that the Fitting-paradigm gap is symptomatic of Muon-style optimizers being only “half-aligned” with µP (controlling update spectra but letting weight spectra drift). The prescribed fix is to constrain both weights and updates to a per-module spectral sphere of radius $R = \Theta(\sqrt{d_\text{out}/d_\text{in}})$, after which µP LR transfer holds again across widths (Controlled LLM Training on Spectral Sphere README §1, §4, Fig. mup_trans).
• Under modern architectures the attention-logit blow-up that motivated µP is suppressed by QK-Norm alone; ablating QK-Norm restores the instability, so module-specific LRs (a µP prescription) yield negligible gain at 4B-MoE scale (How to Set the Learning Rate for Large-Scale Pre-training? §6.1–6.3, Figs. 5–9).
• Per-module AdamW update magnitudes stay near 0.2 across all modules and layers throughout training (consistent with rotational equilibrium), giving a mechanistic explanation for why a single global LR suffices under modern normalization (How to Set the Learning Rate for Large-Scale Pre-training? §6.2, Fig. 7).
• Training-data size $D$ affects internal-state stability more sharply than model size $N$ — the LR scaling exponent on $D$ is larger in magnitude than on $N$, contradicting the typical $N$-only framing (How to Set the Learning Rate for Large-Scale Pre-training? §6.4, Eq. 6, Fig. 10).
• The µA framework extends µP-style asymptotic analysis to LoRA finetuning in the joint limit $n, r \to \infty$, yielding closed-form scaling rules for $\eta$ as a function of width and adapter rank under each (Init[A] vs Init[B]) × $\alpha$ configuration (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4, Corollaries 4.4, 4.6).
• µA identifies a configuration — Init[B] (random $B$, $A=0$) with constant $\alpha=1$ — where the optimal LoRA LR is rank-invariant and coincides with the optimal FFT LR, enabling direct LR transfer from a cheap LoRA sweep to FFT (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.2, §5.2, Fig. 5).
• µA identifies the rank-dependence regime explicitly: under Init[A] with $\alpha=1$, optimal LR scales as $r^{-1/2}$, requiring re-tuning whenever rank changes — observed across Llama-3.2-1B, RoBERTa-large, Qwen2.5-3B, ViT-H/14, Qwen3-VL-2B (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.1, Fig. 3).
• µA scaling rules generalize beyond SFT: same rank-invariant/-dependent patterns hold under RLVR and Stable Diffusion finetuning, and suboptimal LRs cost 4–6× wall-clock training time in RLVR (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §5.3–5.4, Figs. 6, 7).
• Constraining both weights and updates to a spectral sphere via SSO outperforms AdamW and Muon on Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet, with µP LR transfer preserved across widths and bounded activations as a side-effect (Controlled LLM Training on Spectral Sphere README §4, Fig. mup_trans, Fig. control_act).
• Muon is only “half-aligned” with µP: it controls update spectra via Newton–Schulz orthogonalization but places no constraint on weight spectra, so weights drift away from the µP-prescribed scale during training (Controlled LLM Training on Spectral Sphere README §1). The MuonSphere variant retains equivalent activation control with minimal compute overhead.
• SDE reparameterization of AdamW eliminates the optimal-batch-size search. Under CompleteP + SDE, training loss is invariant to physical batch size up to a critical threshold $B_{\rm crit}$, so batch size is no longer a tuned hyperparameter — and $T_{\rm crit}$ (iteration count below which the SDE approximation holds) is empirically independent of model size at fixed token budget (The Design Space of Tri-Modal Masked Diffusion Models §4.1, §5.1, Fig. 5). A new isonoise/isohorizon interpolation parameter $\gamma^* \approx 0.50$ governs how extra tokens are split between reducing SDE drift and extending horizon — both endpoints prior literature used ($\gamma=0$ Kaplan, $\gamma=1$ CompleteP) are suboptimal (The Design Space of Tri-Modal Masked Diffusion Models §5.2, Eq. 4.1, Figs. 9–9).
• Compute-optimal token-per-parameter ratio is regime-dependent: AR Chinchilla gives $D^* \propto N$ (TPP constant); tri-modal MDM gives $D^* \propto N^{0.476}$, so larger MDMs are asymptotically more data-efficient per parameter — a 3B MDM wants ~250B tokens vs ~60B for an AR LLM at the same parameter count (The Design Space of Tri-Modal Masked Diffusion Models §5.3, Eq. 5.2, Fig. 3).
• The Kaplan multiplicative scaling-law form fits tri-modal MDMs better than the Chinchilla additive form. Across 262 isoFLOP runs spanning TPP ratios 1–2000, the Kaplan form yields $R^2 = 99.3\%$ / MRE 0.5%; the additive form fits poorly (The Design Space of Tri-Modal Masked Diffusion Models §5.3, Eq. 5.1, Fig. 12).
• The optimal expert-vs-attention FLOPs ratio $r^*$ in MoE follows a power law in total compute, with sparsity-dependent coefficients. Prior MoE scaling laws (Ludziejewski 2024; Abnar 2025; Wang 2024) only vary expert count, granularity, and sparsity while implicitly fixing $r$; making $r$ an explicit scaling dimension yields a closed-form $r^*(C, S)$ that increases with $C$ — i.e., as compute grows, allocate proportionally more to expert (FFN) layers and less to attention (Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design §1, §3, Fig. 5).
• Layer looping during training is a parameter-free FLOPs-scaling axis with its own predictable scaling law. Applying a shared block of layers iteratively scales FLOPs without scaling parameters; the headline qualitative claim is a 2× parameter-equivalent quality match at the same parameter budget, with the prescription that data and loop count should be scaled in tandem at fixed parameter budget (Layer-looping scaling laws — Hayden Prairie tweet). Visual-generation analog with the same qualitative scaling has been demonstrated at FID parity in ELT: Elastic Looped Transformers for Visual Generation (4× parameter reduction at iso-inference-compute on ImageNet-256; power-law scaling in unique-layer count $L_b$). Open whether the data × loop-count surface from Prairie generalizes to visual generation, and whether ELT’s $L_b$ axis is the same axis as Prairie’s loop count.

Recent contributions

• [2026-11-06] Introducing Inkling 2 Small — Thinking Machines' 594B/17B MoE with unified RL at 4e23 FLOPs: Inkling 2 Small is positioned as “part of a ladder of models” with the explicit framing that “improvements to our process allow us to train models at different sizes using similar recipes with predictable scaling and performance” — the v2 instantiation of Thinking Machines’ recipe-portability-across-size claim first made at Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model. The async-RL run reports “both training reward and evaluation performance grow log-linearly with compute” across the 9-day joint RL run at 4×10²³ FLOPs, matching v1’s claim that reasoning performance improves log-linearly throughout a long RL run and extending it from v1’s separate-runs-combined-via-distillation pattern to the single-joint-run case. Pretraining-side ladder-of-models claim is announced but not yet quantified — follow-on models are promised.
• [2026-11-05] Ripple-65B Preview — Open-Weight Dense 65B Co-Designed with Ricursive's Chip Platform: Ricursive’s six-step pretraining ladder from Olmo 3’s baseline (StepFun LR/BS scaling → architectural+depth scaling → pre+post norm → sigmoid gated attention → global NoPE + LR cooldown to zero → value residuals) stacks to 16× compute efficiency on code at 4B scale; notably documents a 2-stream mHC variant (ladder v1.4) that looked good at small scale but regressed at 4B and was rejected.
• [2026-11-03] Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap: Adds a data-budget-dependent optimizer-choice axis to the scaling-law board: Muon vs AdamW is not a scale-invariant ranking — Muon wins fine-tuning at small-to-moderate TPP but AdamW overtakes Muon beyond a sufficiently large pretraining data budget, with the crossover TPP decreasing as model size grows. Explained via a margin-to-spread ratio of output logits that mirrors the fine-tuning crossover in LMs and is proven for one-step gradient descent vs Muon in a linear-associative-memory model. Companion result to The Best Optimizer Depends on Batch Size on the batch-size axis — both argue the “pick one optimizer and extrapolate” default implicit in the Fitting / Transfer / Enforce fork is unsafe at long horizons, and that optimizer selection should be co-fit with (N, D) rather than treated as fixed.
• [2026-11-02] The Bitter Lesson: Canonical back-fill of Sutton 2019’s “general compute-leveraging methods win over human-knowledge-encoded methods” position — the normative frame behind the whole Fitting/Transfer/Enforce fork tracked on this page, which argues within the compute-scaling paradigm about which general method absorbs more compute cleanly.
• [2026-10-28] Tatsu Hashimoto — AI-assisted humans, not autoresearch, drive NanoGPT speedrun gains (Herman Brunborg 39.9s → 6.7s stamp): Herman Brunborg’s single-record 39.9s → 6.7s improvement (−83.2%) on modded-NanoGPT exceeds the cumulative 2025–2026 improvement (−81.4%) from retrieval-augmented n-gram hints — a datapoint for the “late-stage speedrun gains come from data-and-schedule tricks, not optimizer swaps” pattern tracked here.
• [2026-10-28] RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions: RynnVLA-Latent (Alibaba DAMO) reports data-scaling results for latent-action supervision on 30.1K hours of mixed human + robot + sim video, finding continued improvement in downstream policy performance at the 30K-hour regime — a data point positioned between EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale’s 22-DoF joint-space retargeting at ~21K h and Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2’s wrist-only reduction at 1M+ h, measuring the middle operating point on the latent-action-channel branch where both scale and interface design are jointly varied.
• [2026-10-23] Lucas Beyer — curb the enthusiasm on zeroth-order LM training; loss 4.2 is near the unigram floor: Lucas Beyer’s skeptical take on the Noise/Dust zeroth-order pretraining results sharpens the open scaling question — his “intrinsic size, not raw N” objection is not resolved by Dust’s 120× parameter sweep to 243M (bigger models being more population-efficient could reflect fixed-intrinsic-dim activations in overparametrized models), and his “loss 4.2 is near unigram floor on 30M-param LMs” point reframes the whole leaderboard as living in a regime where the real scaling test has not yet started.
• [2026-10-23] Noise: Speedrun for pretraining without backpropagation: Q Labs’ Noise benchmark operationalizes Dust’s population × tokens × N scaling frontier as a public leaderboard; after ten days of community PRs saturated the 10M-token setting (Track 2 1.76 h → 1.6 s, Track 1 4.167 → 3.676), Q Labs is lifting the training-token budget to 30M — explicitly treating the scaling-law surface as a knob to re-expose headroom rather than only as a prediction target.
• [2026-10-22] OPAL: a reactive optimizer with an agent in the loop: OPAL (Bai, Liang, Teoh, Sitzmann; MIT) is a direct counter-recipe to the Fitting paradigm’s “extrapolate LR*(N, D) from a scaling law and run it unchanged” default: a coding agent redesigns the LR schedule during the run from observed training dynamics, outperforming official recipes on SiT-B/2, RAEv2-B, LightningDiT-B/1, modded-nanogpt, and Goldfish-Swahili at single-run resolution. Interesting because the agent reportedly produces “sophisticated LR schedules” of the kind that were previously only affordable to hand-design for DeepSeek-scale runs — an existence proof that the LR*(N, D) surface the Fitting paradigm estimates may be leaving substantial headroom on the table at per-run resolution. No prediction of how much of the gain is from LR schedule choice vs. optimizer family choice vs. loss-spike guards is reported.
• [2026-10-20] spicylemonade — 2.3× pretraining speedup without backprop, inspired by Cammarata's 'Neural Networks Are Already Clean': Early signal that Dust’s zeroth-order pretraining frontier is already accelerating — a 2.3× speedup via “ignore the useless neurons” interpretability prior, if real, would shift the population × tokens × N surface in Dust: Pretraining Transformers Without Backpropagation by about a Chinchilla-tier compute multiplier (in-progress, single training curve, no numbers yet).
• [2026-10-19] Generative Representation Learning for Scalable Latent Tokenization: RePR reports the first filed tokenizer-encoder scaling law for diffusability: holding the downstream generator fixed and scaling only the encoder yields predictable, monotone FID improvement with no saturation, and the extrapolated fits imply 7× fewer encoder FLOPs than DINOv2 on ImageNet / 8× fewer than WebSSL on GPIC at matched representation quality. Adds the “tokenizer encoder compute” axis to the scaling-law board alongside the DiT isoFLOP fits of Scaling Laws For Diffusion Transformers and the (tokenizer + generator) joint-budget Pareto.
• [2026-10-19] Dust: Pretraining Transformers Without Backpropagation: Dust (Quest Labs) introduces a scaling-law surface orthogonal to Chinchilla: population × tokens × N for a zeroth-order pretraining algorithm. Reports two surprising empirical regularities — (1) backprop-alignment cosine saturates per layer-type with a two-parameter law c(K) = c_max K / (K + c) (RMSE < 0.06) and c_max stays flat across two orders of magnitude in training tokens (10M → 1B), suggesting the required population to match backprop may not grow with tokens; (2) at fixed 10M tokens, bigger models are more population-efficient — a 120× param sweep shows the loss gap to backprop barely moves with N while small models saturate earlier at large populations. Reframes overparameterization as “larger + better-conditioned search space” rather than variance liability.
• [2026-10-15] Frontier Pre-training at Microsoft AI (MAI-Base-2): MAI-Base-2 (Microsoft AI, 2.8T/80B MoE, 16K GB200) introduces source scaling — artificially restricting the set of unique samples on lower ladder rungs so each dataset is repeated the same number of times regardless of training horizon — as a prerequisite for scaling-ladder extrapolation at fixed TPP, arguing without it small-scale experiments reward under-regularized configurations that overfit at scale. Also fits a joint (log LR, log(LR·WD)) scaling law (rather than independent LR and WD): LR* · WD* = 17.04 / steps and log LR* = -1.51 + 0.175 log TPP - 0.573 log steps, with the ~-1 exponent on LR·WD matching the “decoupled WD time scale is a fixed training fraction” view and ~-0.5 on LR matching the √2-batch-doubling rule. Dropout fitted as sigmoid(0.526 (log ActiveParams - 27.2)). Reports 4.2–12.3× FLOPs-efficiency over DeepSeek-V4-Pro-Base via BPB-vs-FLOPs scaling-law extrapolation per evaluation. Extends the Fitting paradigm from decoupled single-axis laws (How to Set the Learning Rate for Large-Scale Pre-training?) to coupled multi-hyperparameter laws at frontier (2.8T) scale.
• [2026-10-13] Jiahua Dong — 30,000-hour egocentric world model learns hands cleanly, not objects; neither 100× data nor 4× model closes the gap: Jiahua Dong reports a joint data × model scan on an egocentric video WFM (300 → 30,000 h and Cosmos 3 Nano 16B → Super 64B) where neither axis closes the hand-object fidelity gap; the measured rate of object-fidelity improvement under model scaling implies ≈10⁴× more compute to close the remaining gap. Datapoint against the “scale alone will close the physics gap” position for pixel-rollout video WFMs — the limiting factor is identified as missing-in-pixels content (mass, friction, hidden forces) rather than compute or parameter budget.
• [2026-10-09] UniWAM: Unified World-Action Model: UniWAM characterizes a log-linear scaling law for unified human-robot co-training over 10,000+ hours of robot + human-egocentric data plus VQA — the first filed quantitative scaling law for the mixture axis in embodied training, distinct from EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale’s human-video-only log-linear law and Isaac 0.5: An Open-Weight Embodied Foundation Model Isaac 0.5’s measured video-vs-teleop exchange at a fixed composition. Joint-training objective runs a physical reasoner, world generator, and action predictor on one backbone with complementary-supervision routing of streams to heads.
• [2026-10-08] RoboJEPA: Scaling Robotic Latent World Models: RoboJEPA fits a second-order power law on latent-rollout imagination error vs compute for JEPA robotic world models, projecting model quality well beyond the fit range at up to 8B parameters across 12 real-robot embodiments — the first filed scaling-law study on the latent-predictive JEPA side of the robotics pretraining space. Reports imagination error strongly correlates with downstream real-robot planning success, giving an offline proxy methodology distinct from EgoScale’s loss-vs-hours law (EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data) and the DVA DINO-FD-vs-success proxy (Rhoda AI — Web-video pretraining scaling laws for robot policy learning (DVA)). Directly addresses the controlled-sweep gap documented by Neural Scaling Laws in Robotics at an order of magnitude above where any filed robotics-scale sweep had reached.
• [2026-10-08] The Best Optimizer Depends on Batch Size: Dang, Wen & Malladi show Muon’s optimizer-ranking position is not stable across batch sizes even after per-batch-size tuning, and no principled scaling rule for Muon transfers hyperparameters consistently across training settings. Argues the batch-size axis is a stability-at-scale axis Muon-family optimizers have not yet absorbed, orthogonal to the width/depth-axis fixes from Hyperball (Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization) and SSO (Controlled LLM Training on Spectral Sphere).
• [2026-10-06] OCBench — A Controllable Robotic Manipulation Benchmark for BC and RL: OCBench makes the companion blog’s “~0.5B params for state-based single-task BC” claim directly reproducible — JAX/Flax reference BC/RL implementations ship with the exact residual-MLP family used in the writeup’s model-size sweep.
• [2026-10-01] How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text: Russell et al. add a data-origin axis (human vs wild AI-generated web text, Pangram-labeled, 27.5–31.1% of FineWeb-filtered 2026 Common Crawl) to the Chinchilla data term via a two-term benefit+harm scaling law that reduces to Chinchilla at AI-fraction 0. Fit on 800 pretrained models, the new law extrapolates to models 3.6× larger with 41% lower error than the best existing law across all AI ratios. First filed scaling-law generalization on the data-quality axis (prior page entries add architecture axes — expert/attention ratio, loop count, vocabulary — or optimizer axes — µA, SSO, AdamH). The sign of the marginal AI-token value flips within a single (N, D) regime as the AI fraction grows, which Chinchilla’s additive form provably cannot capture.
• [2026-09-29] How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining: First head-to-head Chinchilla-style scaling comparison of encoder-based vs encoder-free MLLMs. Text loss–compute frontiers overlap between architectures; multimodal frontiers diverge at small scale but are projected to cross at ~$10^{22}$ FLOPs. Removing the encoder shifts the compute-optimal multimodal parameter allocation toward larger models while leaving the text allocation unchanged — evidence that encoder-vs-encoder-free is a modality-specific allocation-law question, not a global one. Directly extends the decoupled language-vs-multimodal isoFLOP framework of Scaling Native Multimodal Pre-Training From Scratch to a cross-architecture comparison.
• [2026-09-25] Neural Scaling Laws in Robotics: Sartor & Thompson (MIT / IIDS) meta-analyze 327 foundation-models-in-robotics papers and quantify how little rigorous scaling data the field has actually produced: only 13% of papers report any scaling study, only 22 report model-size scaling (median within-study sweep 10.3×, a single order of magnitude), only 1 reports compute scaling (2.4 OoM total sweep width). Pooled power-law fits over 131 data-scaling curves give $\alpha_D = -0.276$ (95% CI [-0.317, -0.236]) and over 34 model-size curves give $\alpha_N = -0.246$ (95% CI [-0.337, -0.155]) — both means over uncontrolled small-N heterogeneous sweeps rather than any single controlled experiment, so best read as the noise floor of the current literature. Meta-level accounting for the exact gap the more recent controlled-sweep entries on this page (EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale data-hours R²=0.9983, Isaac 0.5: An Open-Weight Embodied Foundation Model Isaac 0.5 video-vs-teleop exchange, Scaling Behavior Foundation Model for Humanoid Robots humanoid BFM joint sweep, unpublished Sunday Robotics ran the Chinchilla scaling law with their VLA model (Tony Zhao follow-up) Sunday Robotics VLA Chinchilla claim) are trying to close, and the direct answer to yixuan.ren’s Slack question about why controlled action-model-size sweeps in the LLM/diffusion style don’t exist yet in robotics.
• [2026-09-24] Contrastive Language Model (CLM) — System One Model for Fast and Generalizable Decision-Making: CLM adds retrieval-model geometry (projection-head size, encoder size) as first-class scaling-law axes alongside training compute, model size, and dataset size — fitting the test InfoNCE loss as a power law in each and reporting that these dimensions must be scaled jointly rather than any one alone. Extends the wiki’s multi-axis-co-fit line (Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design, Scaling Native Multimodal Pre-Training From Scratch) into the dual-encoder-verifier design space.
• [2026-09-23] Empirical results fine-tuning π0.5 on a real manufacturing task: Dream Machines’ π0.5 fine-tuning study contributes three concrete negative results at the VLA fine-tune scale: (1) success plateaus at ~30k steps for 21h of data at batch 256 with WSD schedule and does not benefit from longer training; (2) validation action error (L2 on held-out 5%) is useful only for spotting saturation — OLS against real-rig success explains ~20% of variance across all checkpoints and ~0% once the two earliest checkpoints are dropped, and validation flow-matching loss actively rises during runs where success stays flat; (3) sensitivity to learning rate (5e-5 default vs 1e-4) and batch size (128/256/512) sits inside the ±15pp n=40 noise floor — the lowest-training-loss run (1e-4) had the lowest success rate. Together they warn that “sweep LR and pick the checkpoint by validation loss” fails at the VLA-fine-tune scale where downstream success is expensive and noisy to measure, arguing for real-rig evaluation as the load-bearing signal even when it is n=40.
• [2026-09-13] Scaling Laws for Neural Language Models: Foundational back-fill of the original Kaplan et al. LM scaling laws paper this page has been implicitly referencing throughout (Kaplan-vs-Chinchilla, $B_\mathrm{crit}(L)$, isoFLOP methodology). Sharpens the An Empirical Model of Large-Batch Training cross-domain result to Transformer LMs: critical batch size depends on the loss alone (not model size) and follows $B_\mathrm{crit}(L) \approx B_*/L^{1/0.21}$ — the exact “grows steeply as loss falls” behavior Sid’s Slack post quotes. Also the origin of the (now-superseded) Kaplan-2020 compute-optimal recipe $N_\mathrm{opt} \propto C^{0.73}$ that most subsequent entries on this page either follow or explicitly contrast against (Chinchilla revision, tri-modal MDM $D^* \propto N^{0.476}$, DiT $N^* \propto C^{0.56}$).
• [2026-09-13] An Empirical Model of Large-Batch Training: Foundational back-fill of the [MKAT18] gradient-noise-scale / critical-batch-size paper this page has been referencing implicitly. Establishes that $B_\mathrm{crit}$ is predictable from a cheap statistic (simple noise scale $\mathrm{tr}(\Sigma)/|G|^2$) and grows during training as loss falls — the cross-domain result later specialized by Scaling Laws for Neural Language Models to Transformers ($B_\mathrm{crit}$ depends on loss alone) and reinvoked by How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size and The Design Space of Tri-Modal Masked Diffusion Models. Directly addresses Sid’s open question about whether CBS studies transfer to image/video models: the paper’s evidence spans MNIST → Dota → generative VAEs and finds the noise-scale prediction holds across all 8 tasks including a generative model, so the methodology transfers by construction — what remains open is the values of the exponents for diffusion/flow objectives.
• [2026-09-12] Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data: Jha et al. add the missing sparsity axis to the data-repetition scaling literature (Scaling Domain Data Repetition in LLM Pretraining, Internal Data Repetition Destroys Language Models): at matched active params 80M–1B, MoEs overfit to repeated data strictly faster than dense — degradation grows with sparsity and is governed by total rather than active parameters. Practical consequence: dense-derived per-domain repetition budgets do not transfer to MoE runs at the same active-param setting.
• [2026-09-11] RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training: Controlled 1B-vs-4B pretraining comparison shows that on harder MATH problems, pretraining data composition is a stronger lever for RL effectiveness than model scale: adding 10B math tokens to the 1B pretraining mix substantially increases the marginal RL gain, whereas scaling from 1B to 4B parameters at fixed 50B-token budget mostly lifts the base checkpoint without improving RL’s contribution. Also identifies base-model pass@k on the test set as a monotonic pre-run diagnostic for post-RL pass@k — a lightweight predictability tool distinct from the pretraining-loss scaling-law framing.
• [2026-09-10] Rhoda AI — Web-video pretraining scaling laws for robot policy learning (DVA): Rhoda AI’s Tongzhou Mu reports the first public quantitative scaling law for the DVA recipe: joint (model size × pre-training compute × post-training data fraction) sweep across 7 checkpoints on a single deployed industrial bearing-unpacking task, with Bayesian posteriors over real-robot at-speed completion. Model size scales 4% → 85% (still rising at L); pre-training compute matters most when robot demos are scarce (31-point spread at 12.5% data, p=0.07); and DINO Fréchet Distance on held-out web video ranks all 7 checkpoints in the same order as robot performance — a pre-training-measurable proxy that skips the post-training + real-robot eval loop. Sharper joint-axis sweep than the offline-loss fits from EgoScale, XR-1, Isaac 0.5, and Dyna-2.
• [2026-09-08] >10x More Efficient Pretraining: Magic reports a >10× compute-efficient pretraining recipe using per-domain BPB scaling-law fits as the change-selection oracle — train 3 models across 2 OOMs of compute per candidate, keep only if the power-law fit predicts help at scale. Matches DeepSeek V4 Pro Base at 50× fewer FLOPs ($0.5M on GB200) and scales 10× further to beat all evaluated public open base models (DeepSeek V4 Pro/Flash, Kimi K2, Nemotron 3 Ultra) on heldout BPB. Fits Fitting-paradigm in the How to Set the Learning Rate for Large-Scale Pre-training? lineage but as a frontier-scale full-training deployment rather than an LR-surface study. Also observes Nemotron 3 outperforming DeepSeek V4 Pro on pretraining BPB across domains — argues Nemotron’s weak post-RL benchmarks reflect weaker post-training rather than a weaker pretrain.
• [2026-09-03] Introducing K2 Horizon: Frontier Performance, Radically Open: K2 Horizon 3.7B, 7B, 32B, and 36B-A4B are trained on the same 22T tokens; when their normalized training-loss curves are aligned by training progress and normalized by median loss over the final 1%, they approximately collapse onto a single trajectory across dense/sparse architectures spanning ~10× total parameter range. IFM presents this as a production-scale extension of the “scaling-collapse universal-dynamics” observations from Scaling with Collapse: Efficient and Predictable Training of LLM Families and Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks — and, critically, ships the raw intermediate checkpoints and fine-grained training logs so it can be independently re-verified.
• [2026-08-27] Liam Fedus reflects on Switch Transformers — 1.6T-param / 2048-expert MoE was ahead on perplexity, behind on reasoning: Liam Fedus retrospective frames the compute-optimal tokens-per-parameter ratio as task-dependent (Switch Transformer 1.6T beat T5 on perplexity + TriviaQA but lost on SuperGLUE), invoking Shazeer’s “FLOPs are intelligence; parameters are knowledge” — a historical version of the multi-axis scaling-law point made by Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design and Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies.
• [2026-08-27] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning: Ablates four independent scaling axes (data 2M→22M, model L→g, iterations 90k→252k, resolution 256²/16 → 384²/64 cool-down) that each contribute cumulatively to +4.0 avg understanding; progressive-resolution cool-down cuts total compute 8× vs full-resolution training.
• [2026-08-27] LoRA Without Regret: Thinking Machines Lab measures the LoRA↔FullFT optimal-LR ratio empirically across 14 (model × Tulu3) sweeps on Llama/Qwen — fitted multiplier is 9.8× and the hidden-size exponent matches FullFT’s, so the ratio is model-independent. Preliminary evidence puts the multiplier at ~15× for short runs (<~100 steps), converging to ~10× for long runs. Under the standard Huggingface peft parametrization (Init[B] + α/r, both LRs equal) the optimal LoRA LR is (approximately) rank-invariant — varies by <2× between rank 4 and rank 512. Complements Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning‘s theoretical prescription (Init[B] + α=1 gives ratio 1×): both agree on rank-invariance; the 10× vs 1× discrepancy is exactly the α-schedule choice µA identifies as load-bearing.
• [2026-08-26] Isaac 0.5: An Open-Weight Embodied Foundation Model: Isaac 0.5 measures the video-vs-teleop exchange on a well-calibrated offline action-loss surface (τ=2.50): scaling general action-free video from 1,000 h to 1,000,000 h at fixed 80:30:30 general:egocentric:UMI composition moves the within-grid teleoperation crossing from 5,884 h → 28 h — a 210.3× reduction, bounded 83× to 300× by adjacent measured teleop rungs. The exchange is conditional: empirical loss-vs-video slope is nearly flat at 1 teleop-h and stabilizes at −0.21 loss per 10× video from ~100 teleop-h onward. First embodied scaling law to (a) name general action-free video (not just egocentric) as the primary axis and (b) fit a well-calibrated loss-threshold rather than a downstream success rate — a methodologically sharper cousin of EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale’s R²=0.9983 log-linear egocentric-hours-vs-loss law and of Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2’s 1M-h scaling curve.
• [2026-08-26] Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding: Qwen3.8-Flash-Next’s training recipe applies Muon and AdamW to category-specific weight groups and, “guided by refitted scaling laws”, eliminates traditional batch-size warmup and starts directly at the target batch size — Qwen’s team claims this substantially reduces total optimizer steps while safely admitting larger learning rates for robust convergence.
• [2026-08-26] Behavioral Cloning Mystery: Park’s Mystery 3 stakes out an anomalous model-size datapoint on the scaling-laws board: even a fixed, single-task, state-based (37-dim) block pick-and-place task under infinite-data flow BC needs at least a $[4096]\times 8$ residual MLP to work well and $[8192]\times 8$ still improves — nearly 0.5B params for a task that classical BC/RL benchmarks (D4RL, OGBench) solve with $[1024]\times 4$ or smaller. The effect reproduces under autoregressive tokenized action prediction, so it is not a flow-matching artifact. Together with Park’s larger-dataset-can-hurt result (10K episodes beats 50K from the same distribution, 4 seeds × 4 datasets), this is a concrete anti-Chinchilla-shaped datapoint: for BC on non-Markovian narrowly-distributed demo data, the “smaller data ⇒ smaller model” heuristic fails and the required model size is decoupled from the state-space dimension. Suggests VLA action experts cannot be shrunk even when perception + language are handled elsewhere.
• [2026-08-25] S1: In-Context Learning for Robotics: S1 (Skild AI) ships a controlled task-conditioning-interface scaling law between 1k and 100k pretraining hours at matched architectures, data, and compute — the first filed scaling law that treats the prompt interface (video demonstration vs language instruction) as the axis of interest rather than parameter count, data volume, or optimizer choice. Two headline findings: (i) crossover between language-conditioned VLA (53% at 1k hours seen tasks, ahead of 43% ICL) and ICL happens within the sweep, with ICL winning by 100k hours; (ii) language-conditioned prompting plateaus at 9% on unseen tasks by 100k hours while ICL reaches 66% — a 7× gap that grows with scale, arguing language is a wrong-turn interface for the OOD regime rather than a weaker one. Data-quality budget: $3 on QC per$1 on collection (“every data point screened for low-level precision, task coherence, and annotation fidelity”). Complements the model/data/optimizer scaling-law entries on the page with an interface axis, and gives a data-side counterweight to the “language conditioning is a wrong-turn interface” position on VLA Models.
• [2026-08-24] Scaling Domain Data Repetition in LLM Pretraining: Li et al. add a per-domain repetition axis to the fixed-TPP scaling recipe: at fixed tokens-per-parameter, the optimal repetition count for each domain rises mildly with model size, is strongly negatively correlated with the domain’s final validation loss (easier domains tolerate more repetition), and is only weakly related to unique-data volume. The practical claim is that a proxy-model repetition sweep at the target TPP transfers to larger runs — an analogue of µA’s “cheap-LoRA-sweep transfers to FFT” recipe on the data-mixture side. Complements Internal Data Repetition Destroys Language Models (the damage half) by covering the benefit half of the same repetition-structure design space.
• [2026-08-24] Internal Data Repetition Destroys Language Models: Chudnovsky, Kazdan, Levi & Schaeffer et al. modernize Hernandez et al. 2022 for the Chinchilla era: at fixed 10% repeated-token fraction, eval loss is non-monotonic in per-document repeat count $R$, and the peak location scales as $R^\ast \propto N^{-0.66}$ (worst-case pool grows as $U^\ast \propto N^{1.66}$ — faster than compute). Translated via a fitted no-repetition Chinchilla law, the 344M worst-case repetition structure costs ~33% of training compute (a 2-4% loss bump maps to $CEG$ = 0.67-0.85 because the no-rep curve is shallow). A closed-form misspecified linear regression reproduces the same non-monotonic peak, suggesting the phenomenon is a generic statistical feature of duplicated samples rather than a transformer-specific artifact — a prediction directly testable on the diffusion-pretraining side.
• [2026-08-20] Marin scaling laws simulate the entire training trajectory of a 67B MoE run within 1%: Marin team member Larry Dial reports the Delphi Fitting recipe now predicts full training trajectories to within 1% at 67B MoE scale — a strictly stronger claim than Delphi: Scaling Laws That Extrapolate 300× Past the Fit‘s terminal-loss result — and has already used the same ladder to forecast the trajectory of a 535B MoE run in flight. If validated, closes ground with the dynamics-based whole-curve prediction of Scaling with Collapse: Efficient and Predictable Training of LLM Families from the closed-form Fitting side.
• [2026-08-20] GEN-0: Embodied Foundation Models That Scale with Physical Interaction: GEN-0 (Generalist AI, Nov 2025) reports the first filed observation of ossification in robotics — LLM-style loss of ability to absorb new information as training data grows — at ~O(1B) parameters, three orders of magnitude larger than the O(10M) scale prior LLM literature identified it at. Frames the ~7B ossification/absorption phase transition as consistent with Moravec’s paradox: physical commonsense has a higher compute activation threshold than abstract reasoning. Also fits a power law $L(D)=a\,D^{-b}+c$ on downstream post-training loss vs pretraining data size and shows the curve transfers to real-robot success under blind A/B — adds an embodied-data axis to the wiki’s Chinchilla-style entries and to Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models‘s parallel million-hour egocentric-video scaling result, on a real-manipulation substrate. No fit constants are disclosed in the post.
• [2026-08-19] A lower bound for stepsize-based acceleration of gradient descent: Ma & Chen prove a new $\Omega(T^{-1.9319})$ lower bound on last-iterate convergence of gradient descent with any predetermined nonnegative stepsize schedule on smooth convex problems — ruling out that stepsize-only scheduling can reach Nesterov’s $\Omega(T^{-2})$ optimal first-order rate. Draws a strict theoretical ceiling above the current $O(T^{-\log_2(1+\sqrt{2})}) \approx O(T^{-1.2716})$ upper bound from Altschuler–Parrilo / Grimmer et al., leaving only a $T^{~0.07}$-factor gap for further stepsize-alone improvement in the convex-worst-case model that the empirical LR-shape literature (e.g. What do near-optimal learning rate schedules look like?) implicitly baselines against.
• [2026-08-17] Measuring Autonomous AI Research: 153 autonomous nanoGPT-speedrun runs converge on the same optimizer-scaling ingredients — better preconditioning, weight/update magnitude caps, longer hot LR schedules, end-of-training weight averaging — across 18 different frontier models with no internet access, providing evidence that the modded-nanogpt optimizer frontier is a stable attractor under LLM-driven search, not a set of tricks unique to any one team.
• [2026-08-14] Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling: Over-Tokenized Transformer decouples input and output vocabularies and identifies a log-linear scaling law: training loss decreases linearly as input-vocabulary size increases exponentially, holding across 151M–1B dense models and OLMoE-1.3B/7B MoE up to $V_{in} \sim 10^7$ entries with no saturation (~0.015 nats per doubling). A 400M OT-12.8M matches a 1B baseline at the same wall-clock cost; OLMo2-1B convergence acceleration 5.7× on loss, 2.6–3.9× on five downstream benchmarks. Complements Tao et al.’s Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies joint-$V$ compute-optimal fit by showing the input axis alone has essentially unbounded scaling room, and refines Compute Optimal Tokenization‘s tokenizer-agnostic framing with a concrete architectural instantiation (hierarchical $n$-gram tiled hashing at the input).
• [2026-08-14] Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies: Tao et al. add vocabulary size $V$ as a first-class scaling-law axis under a fixed BPE tokenizer family. Three independent methods (IsoFLOPs, FLOP-derivative, parametric loss fit) converge on $N_v \propto C^{0.27}$ vs non-vocab $N_{nv} \propto C^{0.49}$ — vocabulary scales strictly slower than non-vocabulary parameters. Predicts Llama 2-70B should have had ≥216K vocabulary (7× its 32K); verifies at 3B that switching 32K → 43K improves ARC-Challenge 29.1 → 32.0 at matched 2.3e21 FLOPs. Also identifies the regime-dependence: optimum $V$ rises under overtraining (Llama-style) and falls under undertraining (data-scarce), the empirical shape that Compute Optimal Tokenization later restates as “optimal compression rate decreases with compute” in the BLT frame.
• [2026-08-11] Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization: Hyperball (Wen, Dang, Lyu, Ma, Liang) — the arxiv writeup of the AdamH/MuonH recipe already deployed in Delphi: Scaling Laws That Extrapolate 300× Past the Fit. Directly relevant to this page: LR transfer under Hyperball is essentially width- and depth-invariant (maximal drift ≈5% for AdamH/MuonH vs 24% AdamW / 40% MuonW at large depth on Qwen3-style pretraining) because pinning ‖W‖_F directly severs the WD-based coupling that made optimal LR shift with model geometry. Theory (§4.2) proves the mechanism: under scale-invariant weight blocks, decoupled WD sets an equilibrium radius ρ* which sets angular step η/ρ*; by fixing ρ, Hyperball turns η into an explicit angular LR that transfers by construction. Removes WD from the hyperparameter search entirely and applies to any base optimizer (Adam, Muon, NorMuon, KL-SOAP validated on modded-nanogpt Track 3).
• [2026-08-10] Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models: Dyna-2 (Dyna Robotics) extends the embodied-data scaling-law axis to 1M+ hours of egocentric human video (50× EgoScale’s 20k h) and reports monotone power-law fits y = a·N^{-b} on four metrics (MSE, L1, acc@0.5, acc@0.1) with no plateau. First filed cross-substrate transfer scaling law: same human-only pretraining checkpoints show monotone improvement on 39 held-out YAM robot tasks with zero robot data in pretraining, plus an emergence-like inflection between 10k–100k h. Also reports the pretraining scale carries through to real hardware after post-training (20% → 28% → 45% → 53% of max normalized score across the ladder on 14 tasks). Uses Schaeffer’s “emergence-is-a-mirage” multi-metric guard on all curves.
• [2026-08-07] Scaling Native Multimodal Pre-Training From Scratch: Decouples the multimodal isoFLOP fit into a separate language allocation law and multimodal allocation law, fitted via IsoFLOP profiles and training-curve envelopes; the two estimators agree. Headline: the language law is composition-invariant — language exponents barely move as $r_\mathrm{mm}$ varies — while the multimodal law is strongly composition-variant, with the parameter exponent $a$ collapsing (0.54 → 0.42) and the token exponent $b$ rising (0.46 → 0.58) as $r_\mathrm{mm}$ grows from 0.05 to 0.7. Sharper than Beyond Language Modeling: An Exploration of Multimodal Pretraining‘s single-fit “vision more data-hungry” result: the asymmetry lives entirely in the multimodal leg, so unified pretraining reduces to two decoupled Chinchilla-style problems once $r_\mathrm{mm}$ is chosen. Also gives a concrete joint Pareto frontier over compute × $r_\mathrm{mm}$ that outputs deployable $(N^*, D_\mathrm{text}^*, D_\mathrm{mm}^*)$ configurations.
• [2026-08-07] Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes: Three-axis (Language / Understanding / Generation) data-mixture grid search at 1T tokens finds a sharply asymmetric compute-optimal split — language ~70%, understanding ~25%, generation ~5% — that validates at 13.5B-MoE / 2T tokens against an L50/U25/G25 balanced baseline (Full beats Balanced on every axis: PPL 11.67 vs 11.97, GenEval 0.482 vs 0.467, DPG 0.689 vs 0.676) despite 5× fewer generation tokens. Complements the IsoFLOP scaling-asymmetry finding of Beyond Language Modeling: An Exploration of Multimodal Pretraining (vision more data-hungry than language per parameter) with the data-mixture allocation counterpart: even though vision is data-hungry per parameter, generation quality is largely inherited from language + understanding foundations, so most of the token budget should go there.
• [2026-08-06] Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks: Kosson, Messmer & Jaggi (2023) extend SMD to AdamW / Adam+L2 / Lion and derive per-optimizer closed-form equilibrium ‖w‖ and Δθ. Key HP-scaling consequence: AdamW’s equilibrium Δθ is HP-only, while Adam+L2’s depends on gradient RMS — explaining Adam+L2’s imbalanced per-layer rotation (~1000× effective-LR variation) and mechanistically grounding AdamW’s now-standard use in the Fitting-paradigm scaling recipes. Their Rotational Variant also demonstrates that LR warmup partially substitutes for enforcing rotational equilibrium — the transient rotation instability warmup was implicitly suppressing is eliminated by construction under RV, foreshadowing Delphi: Scaling Laws That Extrapolate 300× Past the Fit‘s AdamH weight-sphere retraction and the Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model WD∝LR² coupling.
• [2026-08-06] Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD: Backfill of the SMD foundation the rotational-equilibrium framing on this page rests on. Wan et al. (2020) prove that for scale-invariant weights under SGDM+WD, ‖w‖² converges to η·E[‖g_unit‖²]/(2λ) at a linear rate independent of loss convergence, and the equilibrium angular update √(2ηλ/(1-β)) depends only on hyperparameters. How to Set the Learning Rate for Large-Scale Pre-training?‘s “per-module AdamW update magnitude ≈0.2” mechanistic explanation and the argument for a single global LR under modern normalization both trace back to this equilibrium.
• [2026-08-03] Scaling Properties of Text Conditioning in Visual Generation: Chen et al. add a prompt-structuredness axis to the diffusion scaling-law taxonomy: converged T2I diffusion loss is flat in raw natural-language token count but decreases approximately linearly with a white-box structuredness metric (GPG) and as a power law with a black-box attribute metric (ED). Complementary to Scaling Laws For Diffusion Transformers‘s (N, D) isoFLOP fit and Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers‘s three-axis fit — those hold prompt distribution fixed; this paper sweeps the input distribution and reports the resulting scaling exponents as first-class quantities. Provides a scaling-law rationale for the “long caption + inference-time prompt rewriting” empirical recipe already documented in i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models and HunyuanImage 2.1 — Efficient 2K Text-to-Image with 32× VAE + DINOv2-Aligned REPA and MeanFlow Distillation‘s PromptEnhancer.
• [2026-07-31] Scaling Laws For Diffusion Transformers: Liang et al. (Shanghai AI Lab, Oct 2024) run the first Chinchilla-style isoFLOP sweep on text-to-image DiTs across 1e17–6e18 FLOPs, fitting $N^* \propto C^{0.56}$, $D^* \propto C^{0.43}$, and a loss exponent of $-0.0273$ (in-context DiT), then validate the fit by matching an extrapolated 1B / 1.5e21-FLOP run — a ~250× compute extrapolation. Model size grows slightly faster than data with compute, opposite to The Design Space of Tri-Modal Masked Diffusion Models‘s tri-modal MDM finding ($D^* \propto N^{0.476}$) and closer to but not identical to LLM Chinchilla. Also proposes using scaling exponents themselves as a benchmark for architecture / data quality — the methodological ancestor of later “isoFLOP fit tells you if your change is good” framings on this page. Foundational reference; predates most of the current DiT scaling-law wave.
• [2026-07-31] Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers: Chimera introduces HeteroP — a module-wise µP variant that transfers hyperparameters per tensor according to its functional fan-in and depth position, rather than applying a single width-transfer factor across the whole network. Designed to handle a heterogeneous KDA/MLA/conv/MoE stack where no single fan-in convention applies. HeteroP-tuned reference runs then fit a three-axis Chinchilla law over activated parameters, training tokens, and image-vs-video data ratio — one of the first filed instances of a modality-mixture dimension fit jointly with (N, D). Compute-optimal image pretraining splits nearly evenly between size and tokens; video pretraining modestly favors size at higher budgets.
• [2026-07-30] A Defense of the Quadratic Model: Meterez et al. give a mechanistic account of the LR–B coupling that Fitting/Transfer scaling recipes measure phenomenologically: local linear-stability probes at intermediate checkpoints of a 150M/3B-token run show LLM training sits at a stochastic edge of stability at small batch and rotates onto the deterministic edge of stability (η · λ_max ≈ 2) at large batch, matching linear-stability theory. The Hessian’s power-law tail is invariant to both batch size and the Adam preconditioner, so what changes with B is which stability boundary the optimizer is pushed against, not the underlying spectrum. This suggests LR* scaling laws are implicitly measuring where along a fixed stability manifold the optimum lives — a cleaner framing than raw (η, B) sweeps.
• [2026-07-28] Online KL Shampoo (OKLS) — Tilde Research's KL-optimal full-matrix AdaGrad approximation optimizer with 1.45× parameter efficiency over Muon: Tilde Research’s OKLS optimizer reports a scaling-law fit across 79M–1.2B non-embedding parameters × 20B tokens showing a median 1.46× parameter-efficiency advantage over Muon — restated as “an OKLS run matches a Muon run ~1.5× the parameter count.” First public Kronecker-factored-preconditioner-vs-Muon scaling comparison at ≥1B scale; the exact fit form and exponent are behind the login-walled X-article writeup at filing time.
• [2026-07-27] Is Diversity All You Need for Scalable Robotic Manipulation?: Shi et al. fit a downstream-score power law $y = A - Bx^{-C}$ over pre-training-data scale under fixed task diversity, reporting Pearson $r = 0.9998$ across {no-pretrain, 100K, 250K, 1M} demos on GO-1 with scores 0.28 → 0.58. Complements EgoScale’s log-linear egocentric-hours fit ($R^2 = 0.9983$, EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data) and XR-1’s UMI-hour scaling (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training) with a third clean VLA data-axis fit — this one specifically on the teleoperated multi-task substrate. Cross-embodiment fine-tuning also shows a power-law relationship with fine-tuning data size on ManiSkill (RDT-AWB’s slope beats RDT-OXE’s), which is a rare filed instance of a fine-tuning-side scaling law on VLAs rather than a pre-training-side one.
• [2026-07-27] Delphi: Scaling Laws That Extrapolate 300× Past the Fit: Delphi (Will Held / Marin, on TPU Research Cloud) is the strongest recent public Fitting-paradigm instance: a scaling law fit on IsoFLOP buckets from 3e18 to 3e20 FLOPs forecasts a 1e23-FLOP / 25B / 600B-token held-out run within 0.2%, a 300× extrapolation. The load-bearing recipe changes over an initial diverging attempt are (a) a $(T_0/T)^{0.3}$ token-horizon correction on projection LR alongside the usual $\sqrt{B/B_0}$, and (b) switching to AdamH — a Frobenius-norm-sphere retraction that removes weight decay from the recipe (nothing to regularize away) and, per Kaiyue Wen’s separate scaling sweeps, transfers hyperparameters across width more cleanly than Adam. Also releases the downstream forecasting pipeline: fit a scaling law on a soft metric (correct-answer log-prob for MMLU, reference-completion bits-per-byte for HumanEval), then compose with a sigmoid observational projection fit on public models to recover discrete benchmark scores — extending the Fitting recipe from pretraining loss to task performance and giving small-scale ablations a usable downstream signal even below emergence thresholds. Complements How to Set the Learning Rate for Large-Scale Pre-training? as an open-weights public realization, and the AdamH weight-sphere retraction sits alongside Controlled LLM Training on Spectral Sphere (spectral sphere) and How Muon Lost Its Geometry (µP-correct Muon scaling factor) as geometric-constraint variants of the same “remove degrees of freedom from the HP search” idea.
• [2026-07-20] Scaling Behavior Foundation Model for Humanoid Robots: ScaleBFM (Zeng et al., Shanghai AI Lab + collaborators) is the first filed scaling study on RL-pretrained humanoid Behavior Foundation Models, extending Kaplan/Chinchilla-style analysis beyond LLM pretraining and manipulation VLAs to whole-body control. Identifies three coordinated levers: (a) on-policy rollout width × depth must scale jointly — single-axis scaling does not consistently help; (b) heterogeneous reference-motion diversity drives OOD gains while homogeneous scale drives only ID gains; (c) a Transformer policy beats MLPs at matched parameter counts with mode-dependent saturation. Complements the manipulation-side EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data “scale + diversity” result on egocentric video, and preempts the still-unreleased Sunday Robotics VLA Chinchilla claim (Sunday Robotics ran the Chinchilla scaling law with their VLA model (Tony Zhao follow-up)) with a concrete whole-body-control instance. Reports >10% MPKPE reduction in local mode and 82% in global mode over prior humanoid controllers.
• [2026-07-20] Understanding Reasoning from Pretraining to Post-Training: Shen et al. extend the Fitting-style “predict downstream from cheap runs” methodology from pretraining loss to post-training RL return: across a 5M–1B chess-pretrained sweep, post-RL performance at matched RL compute is well-predicted from pretraining loss, and the slope of the RL reward curve improves approximately linearly with pretraining tokens. Reproduces on a 1B math-text run. Adds a post-training-return axis to this page (previously all entries are pretraining-loss or LR-transfer axes) and gives a chess-based controlled testbed where the pretraining corpus, reasoning traces, and RL rewards can each be swept independently — cleaner than the standard LLM setup for isolating each stage’s contribution to downstream capability.
• [2026-07-17] Sunday Robotics ran the Chinchilla scaling law with their VLA model (Tony Zhao follow-up): Sunday Robotics co-founder Tony Zhao claims (no numbers, no writeup) that they ran the “exact chinchilla-scaling-law” on their VLA model. Would be the first filed VLA joint $(N,D)$ isoFLOP fit if delivered — a sharper result than the existing log-linear data-axis fits from EgoScale (EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data, R²=0.9983) and XR-1 (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training), and a direct test of whether AR LLMs’ $D^*\propto N$ transfers to VLA training (versus e.g. tri-modal MDM’s $D^*\propto N^{0.476}$ from The Design Space of Tri-Modal Masked Diffusion Models).
• [2026-07-16] Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training: Xiaomi-Robotics-1 (XR-1) reports the first filed instance where an LLM-style pre-train scaling curve on a VLA transfers cleanly through post-training to real-robot success — validation action error decreases monotonically with UMI pre-training data + model size, and real-robot success on unseen environments after post-training rises “steadily and predictably” with pre-training compute, “with no signs of saturation.” Sharpens EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data‘s log-linear loss-vs-hours result (R²=0.9983) by showing the scaling gain survives the embodiment-alignment stage rather than only appearing on the validation loss.
• [2026-07-15] Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model: Inkling’s stability recipe operationalizes an LR-schedule-invariant weight-decay rule: coupling weight decay to LR² (as suggested by prior modular-manifolds work) is reported to keep the overall size of model weights stable across training horizons. Concrete frontier-scale datapoint (975B/41B, 45T tokens) that hyperparameter coupling (rather than isolated per-knob tuning) is the stability lever teams are shipping in 2026.
• [2026-07-13] Modular Duality in Deep Learning: Bernstein & Newhouse show that µP (Yang & Hu 2021) and Shampoo (Gupta et al. 2018) are both partial approximations to a single duality map: the √(d_out/d_in)·UVᵀ map induced by the RMS→RMS operator norm on Linear layers. The √(d_out/d_in) prefactor recovers µP’s spectral update scaling and the UVᵀ projection is Shampoo without accumulation. Frames µP not as a parametrization recipe but as the update-scaling half of the modular duality map; a paper’s µP-transfer behavior is then determined by which pieces of the composition (norm choice, module mass, orthogonalization primitive) it actually implements.
• [2026-07-09] Scaling with Collapse: Efficient and Predictable Training of LLM Families: Bergsma et al. (Cerebras) extend Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks to LLM scale with joint width/depth/batch/weight-decay scaling, identifying three controls governing normalized-TLC shape under µP: AdamW normalized timescale $\tau = \eta\lambda B_s/D_s$, tokens-per-parameter ratio, and LR schedule. Collapse emerges only when $\tau$ is fixed across sizes at its TPP-optimal value — Llama-2 (decreasing-TPP family) fails to collapse, Celerity 300M–3.9B at fixed TPP does. Adds a live HPO-tuning consequence: fixing $\tau$ preserves loss-curve ordering throughout training, and a small-scale-fit parametric predictor $L(f)=f^{-A}G(f,\text{sched};w)$ with $A(\tau)$/$w(\text{TPP})$ power laws enables reliable HPO winner-selection at 10-30% of training. Together with Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks, this makes the Cerebras/Google-Research “timescale + collapse” line an alternative to the Fitting scaling-law paradigm — same predictive goal, dynamics-based rather than closed-form.
• [2026-07-09] Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks: Qiu et al. show compute-optimally trained models exhibit loss-curve scaling collapse: after affinely rescaling reducible loss and compute to unity at training end, curves across model sizes fall onto one universal shape. With LR decay, collapse tightens below the seed-to-seed noise floor (supercollapse). Derives collapse from sum-of-power-law scaling laws and a noisy-quadratic SGD model; shows supercollapse-deviation is a sharper mis-parameterization diagnostic than final-loss comparison (breaks under constant-LR-across-widths or wrong data exponent even where final losses look indistinguishable). Adds whole-loss-curve prediction as a validation axis on top of the scalar Fitting/Transfer/Enforce fork already on the page.
• [2026-07-05] Implications of Large-Scale Test-Time Compute (Noam Brown): Noam Brown argues that inference-compute is the scaling-law axis current benchmark reporting practice under-represents: the capability ceiling for modern LLMs is often unknown because runs are cut off at time/infra limits rather than a genuine plateau, so single-scalar leaderboards misrepresent the model. Proposes replacing scalar reports with performance-vs-cost / tokens / time curves — the same “score as a function of compute” framing this concept applies to pre-training extended into the inference regime.
• [2026-07-03] How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size: Schaipp proposes a three-term scaling law L(N, T, B) that keeps training steps T and batch size B as separate axes (rather than lumping them into D=T·B); recovers optimal-batch-size scaling and matches prior critical-batch-size empirical findings as downstream consequences of a single fit. Crucially, the fit consumes suboptimal-batch runs and needs significantly fewer runs than Chinchilla-style methodology. Complements Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling‘s schedule-equivalence proof and directly addresses the “batch size as a free hyperparameter” Open Question on this page.
• [2026-07-01] To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters: Dragutinović, Zhang & Ranganath analyze Muon (via an exact-SVD, no-momentum abstraction called Spectral GD) on 2-layer deep linear networks and prove Spectral GD learns all singular components of the product matrix W₂W₁ in parallel at equal rate, while GD’s saddle-to-saddle dynamics sequentially learn largest-σ-first — explaining Muon’s wall-clock speedup as a loss of the rank-monotone simplicity bias GD naturally has. Constructs two toy settings where SGD beats Muon: (a) a Saxe-style multi-source “routing” task where SGD generalizes to unseen source pairs by learning the 4-D shared underlying task while stochastic Spectral GD memorizes each pair (high effective rank, heavy-tailed hidden spectrum); (b) MNIST + per-class spurious pixel where SGD’s dominant-first learning gives a longer clean-digit window before spurious features saturate. Adds an inductive-bias axis to the wiki’s Muon debate that sits below both the µP-alignment critique (Controlled LLM Training on Spectral Sphere SSO, How Muon Lost Its Geometry) and the “geometry doesn’t matter” critique (Muon is Not That Special: Random or Inverted Spectra Work Just as Well).
• [2026-06-30] EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: EgoScale (NVIDIA GEAR) reports the first clean log-linear scaling law on the embodied-data axis: across {1k, 2k, 4k, 10k, 20k} hours of action-labeled egocentric pretraining video, optimal validation loss on held-out human action prediction fits R²=0.9983 with no saturation; offline loss strongly tracks real-robot dexterous-manipulation success rate (0.30 → 0.71 average task completion). Extends the Kaplan/Chinchilla style data-axis fits beyond text into the egocentric-video pretraining substrate and gives a predictable budget calculus for human-data-scale decisions.
• [2026-06-30] How Muon Lost Its Geometry: Arda Göreci (Ligo) traces how Muon’s per-matrix scaling factor diverged from the µP-correct $\sqrt{\text{fan-out}/\text{fan-in}}$ as Muon scaled up: Keller’s speedrun variant clips the correction with $\max(1, \cdot)$, Moonshot’s Moonlight variant replaces it with $\sqrt{\max(\text{fan-in}, \text{fan-out})}$ (only equivalent when fan-in is uniform), and torch.optim.Muon ships both wrong options as user-selectable defaults. Microsoft’s Dion is the one popular variant that gets it right. A CIFAR-10 bottlenecked-MLP LR-transfer sweep across 5 widths shows the correct factor produces no LR drift while Moonshot’s drifts the optimum to lower LRs as width grows. Substantiates Controlled LLM Training on Spectral Sphere‘s “Muon is half-aligned with µP” claim with a sharper, one-line-fix version: the update-side half is itself wrong as implemented.
• [2026-06-26] Scaling Laws, Carefully: Lilian Weng’s long-overdue Lil’Log primer on $(N, D, C)$ scaling laws — consolidates Kaplan vs. Chinchilla, the Pearce-Song embedding-fraction reconciliation, Hernandez/Muennighoff/Lovelace on the data-limited regime, and Besiroglu et al.’s Chinchilla method-3 replication bug. Useful canonical reference for the data-axis baseline that this concept page’s optimizer-axis and architecture-axis entries extend.
• [2026-05-26] Laguna M.1/XS.2 Technical Report: Laguna fits a WSD-specific lr*(N,D) = 10^4.488·N^-0.4639·D^-0.2661 across 4 MoE sizes × 6 LRs × 5 token budgets at fixed 8M-token batch, then √(B/B₀)-scales to the 24M-batch production run; predicts 5.5e-4 for XS.2, deployed at 5e-4 — production-scale validation of the How to Set the Learning Rate for Large-Scale Pre-training? methodology.
• [2026-05-24] Layer-looping scaling laws — Hayden Prairie tweet: tweet-form announcement of layer-looping scaling laws for transformer pretraining — predictable scaling laws exist for loop count, a looped transformer matches the quality of a 2× larger transformer at the same parameter budget, and the prescription is to scale data and loop count together at fixed parameter budget. Adds a parameter-free FLOPs-scaling axis to the page that complements the existing optimizer/LR axes (Fitting/Transfer/Enforce), the data axis (Kaplan/Chinchilla), and the architecture axis (MoE expert/attention FLOPs ratio). Loop count is also already an established scaling-law dimension in the visual-generation setting via ELT: Elastic Looped Transformers for Visual Generation.
• [2026-05-23] Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design: introduces the expert-vs-attention FLOPs ratio $r$ as a first-class scaling-law dimension for MoE. Sweeps GPT-style MoE Transformers across compute scales and sparsity levels, finds $r^*$ follows a power law in total compute $C$ with sparsity-dependent coefficients, and derives an explicit closed-form $r^*(C, S)$. Headline qualitative finding: $r^*$ rises with compute, contradicting the standard practice of holding a fixed FFN/attention dim ratio across scales. Sits in the architecture-axis of scaling laws (vs. the LR-axis covered by the existing entries) and is the cleanest scaling claim adjacent to the Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method / DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks “make MoE design less arbitrary” cluster.
• [2026-05-23] The Design Space of Tri-Modal Masked Diffusion Models: Apple’s tri-modal MDM design-space study extends CompleteP with an SDE-based reparameterization that makes training loss invariant to physical batch size up to $B_{\rm crit}$ — eliminating the batch-size search entirely below threshold. Introduces an isonoise/isohorizon interpolation $\gamma$ controlling how extra tokens are allocated between reducing SDE drift and extending horizon; finds $\gamma^* \approx 0.50$, with both prior-literature endpoints suboptimal. Fits multiplicative Kaplan-form scaling laws on 262 runs at $R^2 = 99.3\%$, deriving $D^* \propto N^{0.476}$ — i.e. larger MDMs are more data-efficient per parameter, opposite to AR LLMs.
• [2026-05-23] Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning: introduces Maximal-Update Adaptation (µA), the LoRA analogue of µP, characterizing optimal-LR scaling jointly in model width and adapter rank. Identifies Init[B] + $\alpha=1$ as the unique configuration where LoRA’s optimal LR transfers directly to FFT, plus a separate “learning-efficiency imbalance” diagnostic showing that one of the two LoRA factors contributes essentially nothing to feature learning under each named configuration. Verified across SFT (5 models), RLVR, and Stable Diffusion finetuning.
• [2026-05-23] Controlled LLM Training on Spectral Sphere: introduces SSO, an optimizer that retracts both weights and updates onto a per-module spectral sphere; argues Muon is half-aligned with µP and that closing the loop restores µP transfer. Beats AdamW and Muon on Dense 1.7B / MoE 8B-A1B / 200-layer DeepNet with µP LR transfer preserved across widths.
• [2026-05-22] How to Set the Learning Rate for Large-Scale Pre-training?: formalizes the Fitting paradigm (closed-form $\eta^*(N, D)$ from cheap small-scale WSD runs); beats µTransfer at 4B and 12B MoE scale on 500B tokens. The QK-Norm ablation is the load-bearing experiment for the “µP’s stability motivation is stale” claim.

Open questions

• Fitting vs Transfer vs Enforce is a three-way fork that hasn’t been benchmarked end-to-end. Fitting wins at 4B/12B MoE pretraining when the architecture is QK-Norm-stable; SSO claims µP transfer holds when weight+update spectra are constrained; µA shows µP-style analysis pays off for LoRA finetuning. No filed paper has tested whether SSO restores µTransfer’s edge over Fitting at the 4B/12B scale where Fitting beats vanilla µTransfer.
• Architecture-side scaling laws need to be co-fit with optimizer-side ones. Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design shows $r^*$ (FFN/attention FLOPs allocation) scales with $C$, but every other entry on this page holds architecture fixed and tunes the optimizer. Open: does the Fitting paradigm’s $\eta^*(N, D)$ surface depend on whether the architecture is at $r^*$ or off it? Conversely, does the prescribed $r^*$ depend on whether the LR is at $\eta^*$? Joint scaling-law fits over $(N, D, r, \eta)$ have not been done.
• Does loop count enter the Fitting scaling-law surface? Layer-looping scaling laws — Hayden Prairie tweet proposes a data × loop-count rule at fixed parameter budget; ELT: Elastic Looped Transformers for Visual Generation reports power-law scaling in unique-layer count $L_b$ at fixed width. Whether the same $L$ axis governs both LLM pretraining and visual generation, and whether the optimal LR $\eta^*$ has a clean dependence on $L$, is open.
• Does µA transfer to DiT / MMDiT finetuning? Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning runs Stable Diffusion (UNet) and finds the Init[B] + $\alpha=1$ rule holds with FFT being sharply LR-sensitive on UNets. The MMDiT case (with AdaLN modulations as an extra outlier source) is untested. This is the most directly testable claim from this concept on Luma’s stack.
• What’s the right default LoRA init? The Slack thread that filed Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning notes that Luma’s current default is Init[B] with $\alpha=1/r$ — covered in µA’s Appendix A.5 but not one of the three named configurations in the paper’s headline workflow. The clean action is to switch defaults to Init[B] + $\alpha=1$ for finetuning, then transfer LR to FFT — but the appendix’s analysis of the $1/r$ variant should be read first.
• Module-specific LRs: How to Set the Learning Rate for Large-Scale Pre-training? §6.1 says they don’t help at 4B-MoE scale under QK-Norm; µA’s “learning-efficiency imbalance” claim suggests one LoRA factor contributes negligibly to feature updates, raising the question of whether per-factor LRs (in the spirit of LoRA+) are still warranted even when per-module LRs are not. No filed paper has done the joint experiment.
• µTransfer’s effective horizon is short (How to Set the Learning Rate for Large-Scale Pre-training? §3.2); extending it requires weight-decay scaling and token-horizon-aware init. Whether SSO’s spectral retraction implicitly handles this horizon issue is open.
• Does the Fitting recipe transfer to diffusion/flow pretraining? $\eta^*(N, D)$ quadratic-in-log-LR + power-law-in-D is cheap enough to run on diffusion pretraining sweeps; the Universal One-third Time Scaling in Learning Peaked Distributions universal-exponent result suggests softmax-CE-specific behaviors may not apply, so the slope and possibly the scaling-law form could differ for MSE / flow-matching objectives. Partial answer from The Design Space of Tri-Modal Masked Diffusion Models: for discrete-token masked diffusion (still softmax-CE), the Kaplan multiplicative form fits well and $D^* \propto N^{0.476}$ — strictly more data-hungry per-parameter than AR. Open whether the same shape carries to continuous-token DiT/flow-matching pretraining where the objective is MSE rather than CE.
• Is “batch size as a free hyperparameter” robust to the optimizer choice? The Design Space of Tri-Modal Masked Diffusion Models‘s SDE-reparam result is derived for AdamW. Whether the same physical-vs-logical batch-size decoupling holds under Muon / SSO / Lion is open, and the result directly affects how the Fitting / SSO / SDE recipes interact.
• The “learning-efficiency imbalance” in µA — that under both Init[A] and Init[B] (with $r \ll n$) one LoRA factor contributes negligibly — suggests that the standard 2-factor LoRA factorization is over-parameterized for the rank regime in actual use. Is there a single-factor equivalent (random projection + trainable head) that captures the same dynamics with half the parameters? Open across all filed papers.

Papers

2026-05

• PhysicsLM4 — Canon Layers code release (Physics of Language Models Part 4) — PhysicsLM4 — Canon Layers code release (Physics of Language Models Part 4) (published 2026-05-24)
• Beyond MuP 3: Special Cases, Special Treatment (Embedding, LM Head, RMS Norm) — Beyond MuP 3: Special Cases, Special Treatment (Embedding, LM Head, RMS Norm) (published 2026-05-23)
• Compute Optimal Tokenization — Compute Optimal Tokenization (published 2026-05-02)
• MuonH (hyperball) sets new modded-NanoGPT speedrun record at 3325 steps — Keller Jordan announcing @wen_kaiyue's result — MuonH (hyperball) sets new modded-NanoGPT speedrun record at 3325 steps — Keller Jordan announcing @wen_kaiyue’s result (published 2026-05-01)

2026-04

• How GPT, Claude, and Gemini are actually trained and served – Reiner Pope — How GPT, Claude, and Gemini are actually trained and served – Reiner Pope (published 2026-04-29)
• Layer-looping scaling laws — Hayden Prairie tweet — Layer-looping scaling laws — Hayden Prairie tweet (published 2026-04-15)
• ELT: Elastic Looped Transformers for Visual Generation — ELT: Elastic Looped Transformers for Visual Generation (published 2026-04-10)
• Test-Time Scaling Makes Overtraining Compute-Optimal — Test-Time Scaling Makes Overtraining Compute-Optimal (published 2026-04-01)

2026-03

• Rethinking Language Model Scaling under Transferable Hypersphere Optimization (HyperP) — Rethinking Language Model Scaling under Transferable Hypersphere Optimization (HyperP) (published 2026-03-30)
• Path-Constrained Mixture-of-Experts — Path-Constrained Mixture-of-Experts (published 2026-03-18)
• OpenAI Model Craft: Parameter Golf — OpenAI Model Craft: Parameter Golf (published 2026-03-18)
• The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data — The Finetuner’s Fallacy: When to Pretrain with Your Finetuning Data (published 2026-03-17)
• Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design — Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design (published 2026-03-11)
• Lost in Backpropagation: The LM Head is a Gradient Bottleneck — Lost in Backpropagation: The LM Head is a Gradient Bottleneck (published 2026-03-10)
• Beyond Language Modeling: An Exploration of Multimodal Pretraining — Beyond Language Modeling: An Exploration of Multimodal Pretraining (published 2026-03-03)
• DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks — DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks (published 2026-03-02)

2026-02

• Spectral Condition for μP under Width-Depth Scaling — Spectral Condition for μP under Width-Depth Scaling (published 2026-02-28)
• Summer-22B: A Systematic Approach to Dataset Engineering and Training at Scale for Video Foundation Model — Summer-22B: A Systematic Approach to Dataset Engineering and Training at Scale for Video Foundation Model (published 2026-02-26)
• The Design Space of Tri-Modal Masked Diffusion Models — The Design Space of Tri-Modal Masked Diffusion Models (published 2026-02-25)
• Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning — Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning (published 2026-02-05)
• Universal One-third Time Scaling in Learning Peaked Distributions — Universal One-third Time Scaling in Learning Peaked Distributions (published 2026-02-03)
• Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method — Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method (published 2026-02-01)

2026-01

• Controlled LLM Training on Spectral Sphere — Controlled LLM Training on Spectral Sphere (published 2026-01-13)
• How to Set the Learning Rate for Large-Scale Pre-training? — How to Set the Learning Rate for Large-Scale Pre-training? (published 2026-01-08)

2025-12

• Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers — Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (published 2025-12-19)
• Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality — Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality (published 2025-12-19)

2025-11

• Simplifying Adam: Bias Correction Debunked — Simplifying Adam: Bias Correction Debunked (published 2025-11-25)
• Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction? — Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction? (published 2025-11-11)

2025-10

• Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling — Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling (published 2025-10-16)

2025-09

• Scalable GANs with Transformers — Scalable GANs with Transformers (published 2025-09-29)
• LongCat-Flash Technical Report — LongCat-Flash Technical Report (published 2025-09-01)

2025-07

• Diffusion Beats Autoregressive in Data-Constrained Settings — Diffusion Beats Autoregressive in Data-Constrained Settings (published 2025-07-21)

2025-06

• Diagnosing and Improving Diffusion Models by Estimating the Optimal Loss Value — Diagnosing and Improving Diffusion Models by Estimating the Optimal Loss Value (published 2025-06-16)
• Why Gradients Rapidly Increase Near the End of Training — Why Gradients Rapidly Increase Near the End of Training (published 2025-06-02)

2025-03

• Addition of Muon optimizer to torch.optim — PyTorch team welcomes PR (Issue #148819) — Addition of Muon optimizer to torch.optim — PyTorch team welcomes PR (Issue #148819) (published 2025-03-08)

2025-02

• Distillation Scaling Laws — Distillation Scaling Laws (published 2025-02-12)

2024-01

• How To Scale — How To Scale (published 2024-01-22)

2020-12

• Effect of the initial configuration of weights on the training and function of artificial neural networks — Effect of the initial configuration of weights on the training and function of artificial neural networks (published 2020-12-04)

Unknown date

• Marin Selected Experiment Reports — Marin Selected Experiment Reports (published unknown)
• Token Superposition Training (TST) — 2-3× wall-clock pretraining speedup without architectural change (Nous Research) — Token Superposition Training (TST) — 2-3× wall-clock pretraining speedup without architectural change (Nous Research) (published unknown)