LoRA Without Regret

TL;DR

An empirical characterization by Thinking Machines Lab of the regime in which LoRA matches full fine-tuning (FullFT) on both sample efficiency and final loss. Two conditions must hold: (i) LoRA is applied to all weight matrices (with MLP and MoE layers being the load-bearing ones, not attention), and (ii) the dataset is small enough that the number of LoRA parameters exceeds its information content. Under these conditions LoRA matches FullFT for supervised fine-tuning on instruction/reasoning-scale datasets, and fully matches FullFT for policy-gradient RL even at rank 1 — the information-theoretic argument being that policy gradients inject only ~1 bit per episode, so rank-1 LoRA (~3M params on Llama-3.1-8B) already exceeds capacity needs by an order of magnitude. The post also empirically fixes the LoRA-vs-FullFT LR ratio at ~10× and identifies a batch-size gap: LoRA pays a larger loss penalty than FullFT as batch size grows, and this gap is a property of the $BA$ parametrization, independent of rank.

Key claims

On instruction-tuning and reasoning datasets (Tulu3, OpenThoughts3), FullFT and high-rank LoRAs share the same learning curve — loss decreases linearly in log(steps); low- and medium-rank LoRAs fall off this curve at a step count that correlates with rank, when adapter capacity is exhausted [Fig. 1].
LoRA is less tolerant of large batch sizes than FullFT: a persistent LoRA-vs-FullFT loss gap opens as batch size grows, and the gap is not mitigated by increasing rank — it is a property of the $BA$ parametrization vs the full $W$ [Fig. 3].
Attention-only LoRA underperforms even at matched parameter count: attention-rank-256 (0.25B params) underperforms MLP-rank-128 (0.24B params) on Llama-3.1-8B; MLP-only ≈ MLP+attention > attention-only [Fig. 5, Table §Layers Where LoRA Is Applied].
LoRA matches FullFT for policy-gradient RL on MATH and GSM at ranks as low as 1, over both Llama-3.1-8B and Qwen3-8B-base on DeepMath, with matched peak performance on held-out AIME 2024/2025 [Fig. 6].
Information-theoretic argument for the RL result: policy gradient methods absorb $O(1)$ bit per episode (mutual-information upper bound via the data-processing inequality applied to a $B$ -bin-quantized advantage); for the MATH run (10k problems × 32 samples = 320k bits) rank-1 LoRA on Llama-3.1-8B already provides ~3M parameters — ~10× the information budget [§Reinforcement learning, §How much capacity is needed].
With Init[B] + $\alpha/r$ scaling, the LoRA learning curve is exactly the same at the start of training regardless of rank, and the optimal LR is approximately rank-invariant (varies by <2× between rank 4 and rank 512, with a slight downward shift at rank 1) [§Optimal learning rate and rank, Fig. 2].
Empirically, the optimal LR for LoRA is ~10× the optimal FullFT LR across 14 sweeps on Llama/Qwen models — with a slightly higher multiplier (~15×) for short (<~100-step) runs [§Optimal learning rates for LoRA vs. FullFT, §Learning rates in short and long runs].
The four LoRA hyperparameters — $\alpha$ , LR for $A$ , LR for $B$ , and init scale of $A$ — collapse to two invariant degrees of freedom under Adam training: an initial-update scale $\eta_A \eta_B s^2 / \alpha$ and a timescale for $A$ to evolve away from its init [§Parametrization invariances].
FLOP cost of LoRA is ~2/3 of FullFT per training pass — LoRA can therefore be more compute-efficient overall in addition to being memory-efficient [§Compute efficiency advantage of LoRA].
Beyond the rank threshold at which LoRA underperforms FullFT, the gap manifests as reduced training efficiency (a rank-dependent slowdown), not as a loss floor that LoRA cannot cross [§What matters for LoRA].

Method

Setup: Llama 3 series + Qwen3 (including a MoE), constant-LR schedule (no warmup or cooldown), per-condition LR sweep, LoRA rank spanning 1 to 512. Supervised fine-tuning experiments use Tulu3 (instruction following) and OpenThoughts3 (reasoning), scored by log loss on held-out completions. Log loss (rather than sampling evals) is the deliberate design choice, chosen for generality across dataset sizes and training-step counts. RL uses a policy-gradient algorithm with importance-sampling correction and a GRPO-like group-mean-centered advantage, evaluated on MATH and GSM (with Llama-3.1-8B as the base, deliberately avoiding Qwen bases because their pretraining data is known to contain math). A follow-up scale-up uses Qwen3-8B-base on DeepMath (harder, 8192-token cap) with AIME 2024 and 2025 as held-out evals.

For the MoE experiment, a separate LoRA is trained per expert with rank equal to total_rank / num_active_experts (8 for Qwen3 MoE), so LoRA-params/FFT-params stays constant across dense and MoE layers.

Results

Instruction / reasoning SFT (Tulu3, OpenThoughts3 subsets). FullFT and rank-≥256 LoRAs are indistinguishable on the learning curve. Rank-32 falls off ~partway through the run; rank-1 falls off almost immediately. The “fall-off point” tracks rank monotonically.
Batch-size scan (10k-example OpenThoughts3 subset). At batch 32 the LoRA↔FullFT loss gap shrinks over time; at large batches the gap is persistent and grows with batch size. Best absolute loss for both LoRA and FullFT is at smaller batch sizes.
Where to place LoRA. Applying LoRA to all layers (MLP+attention) is roughly equal to MLP-only; both beat attention-only at matched parameter count. This holds on the OpenThoughts3-rank-256 SFT run and on the MATH RL run.
RL (MATH, GSM, DeepMath). LR sweeps for LoRA (any rank ≥ 1) and FullFT peak at the same reward. LoRA also shows a wider range of performant learning rates — a robustness win at no cost. On Qwen3-8B DeepMath, LoRA and FullFT produce identical training curves and identical held-out AIME scores; both develop backtracking, self-verification, and in-context exploration.
Optimal LR ratio. Across 14 (model × Tulu3) sweeps, the fitted LoRA-vs-FFT LR multiplier is 9.8× (with matched exponent on hidden size — so the ratio is model-independent). Preliminary evidence puts the multiplier at ~15× for short runs (<~100 steps) and it converges to ~10× for long runs.
Compute. LoRA forward+backward pass costs ~2/3 the FLOPs of FullFT.

Why it’s interesting

Direct empirical companion to Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning — the µA paper on the wiki. µA gives a theoretical prescription (Init[B] + $\alpha=1$ makes the LoRA optimum coincide with FullFT’s optimum, i.e. LR-ratio = 1); this post uses the conventional Huggingface peft parametrization (Init[B] + $\alpha/r$ scaling, both LRs equal) and measures a stable ~10× LR ratio empirically. Both agree that the optimal LR is (approximately) rank-invariant, but they disagree about whether the ratio to FullFT is 1× or ~10× — the discrepancy is exactly the $\alpha$ -schedule choice µA identifies as load-bearing. This closes the loop on the Slack open question the µA page raised (“Should Luma’s default LoRA init be updated?”): either update to Init[B] + $\alpha=1$ (µA’s prescription; ratio = 1×) or keep the current $\alpha/r$ and use LR ≈ 10× FullFT (this post; direct empirical calibration).

The RL result also resolves a wiki-level question. Spurious Rewards: Rethinking Training Signals in RLVR and Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training both argue that RL post-training moves in a very-low-capacity subspace of the base model. This post gives the crispest form of that argument: policy gradients inject $O(1)$ bit per episode by a data-processing-inequality bound, and rank-1 LoRA already carries an order of magnitude more capacity than a typical RL post-training budget. That has practical implications for Luma RL post-training runs — the standard “use big LoRAs to be safe” default is dominated by rank-1 LoRA in this regime.

Regarding the batch-size caveat Ishaan flagged: this post finds the batch-size gap but also notes both LoRA and FullFT peak at smaller batch sizes anyway, so in practice the gap only matters when compute constraints force large-batch training. Complements Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning‘s claim that optimal LR scales with batch size — the Thinking Machines batch-size gap here is measured at tuned LRs per batch size, so it’s not a mis-tuning artifact.