Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning

TL;DR

A systematic re-evaluation of four “advanced” LoRA variants (PiSSA, MiLoRA, Init[AB], DoRA) against vanilla LoRA, run under proper per-method learning-rate sweeps across three model scales (Qwen3-0.6B, Gemma-3-1B, Llama-2-7B) and two tasks (math, code). The headline finding is that once each method’s LR is tuned, all five converge to within 1–2% of each other, with no systematic winner — most published gains over vanilla LoRA were artifacts of inheriting hyperparameters from the original LoRA setup rather than re-tuning per variant. A survey of 52 recent LoRA papers finds fewer than 30% tuned the learning rate. A Hessian-eigenvalue analysis explains why different variants want different LRs: PiSSA-style SVD initialization lands on a sharper region of the loss landscape, so its optimal LR is ≈10× lower.

Key claims

Of 52 surveyed LoRA papers (42 published + 10 high-impact preprints), only one tuned learning rate + batch size + rank jointly, and fewer than 30% tuned the learning rate at all, calling into question the methodological attribution of reported gains [§1, Fig. 2].
Once learning rate is properly tuned per method, vanilla LoRA, PiSSA, MiLoRA, Init[AB], and DoRA all peak within 1–2% of each other across all tested model × task combinations [§4.3.1, Fig. 1, Table 1].
On Qwen3-0.6B math, the top method (LoRA itself) beats the runner-up (DoRA) by 0.15% and the worst (MiLoRA) by 0.43% — well inside noise [§1, Fig. 1].
Optimal learning rates differ by up to an order of magnitude across methods — PiSSA consistently requires ≈10× lower LR than vanilla LoRA, while MiLoRA and Init[AB] sit within the same order of magnitude as LoRA [§4.3.1, Fig. 1].
Relative performance between variants flips with rank: MiLoRA outperforms LoRA at low ranks (e.g. +0.8% at r=8 Gemma math) but underperforms at high ranks (−0.63% at r=256); PiSSA shows the opposite trend [§4.3.2, Fig. 5].
DoRA’s “consistently better than LoRA” claim from the original paper does not survive — DoRA only beats LoRA at very low ranks (peak +1.1% at r=4 math), and the gain is smaller than reported in Liu et al. 2024a, which is attributed to insufficient LR tuning in the original [§4.3.2].
Learning rate is far more important than batch size — fixing LR and tuning batch size yields suboptimal results (DoRA at LR=2e-4 only reaches 11.16%); fixing batch size and tuning LR reaches 20.5–21.0% [Table 1, §4.3.1].
Optimal LR scales proportionally with batch size, consistent with classical SGD scaling rules; this explains away the LoRA-degrades-at-batch-128 result from Schulman & Lab 2025 as a suboptimal LR pairing [§4.3.1].
The top Hessian eigenvalue λ_max at initialization is up to ≈10× larger for PiSSA than for vanilla LoRA (computed via Lanczos + Hessian-vector products on the Query projection across layers), giving a theoretical justification (1/λ_max LR scaling under quadratic approximation) for PiSSA’s lower optimal LR [§5.2, Fig. 6].
MiLoRA and Init[AB] have slightly larger λ_max than vanilla LoRA on Qwen (factors consistent with the observed ≈0.5–0.71× LR ratios seen in Fig. 1), corroborating the inverse-eigenvalue prediction [§5.2, Fig. 6].

Method

The setup: five PEFT methods (vanilla LoRA, PiSSA, MiLoRA, Init[AB], DoRA), three decoder-only LLMs (Qwen3-0.6B, Gemma-3-1B, Llama-2-7B), two tasks (MetaMathQA → eval on GSM8K + MATH; CodeFeedback → eval on HumanEval + MBPP). For each (method, model, task) cell, the authors sweep learning rate on a log grid with four points per decade from 1e-5 to 1e-2 (16 grid points), and additionally vary rank ∈ {4, 8, 16, 32, 64, 128, 256} and batch size ∈ {16, 64, 128} for selected combinations. The LoRA scaling α = r is held fixed so γ = α/r = 1, factoring out scale tuning. Each method’s PEFT parameters are initialized per its own published recipe — PiSSA with top-r SVD components of W₀, MiLoRA with bottom-r minor components, Init[AB] with the theoretically derived dual-initialization, DoRA with vanilla LoRA + an additional magnitude vector.

For the Hessian analysis (§5), the authors run a layer-wise top-eigenvalue estimation via the Lanczos algorithm and Hessian-vector products on the trainable LoRA parameters at initialization, focusing on Query projection blocks. They report the per-layer ratio λ_max(variant)/λ_max(LoRA) and check it against the observed optimal-LR ratios. DoRA is deferred to future work since its architecture makes the Hessian evolve differently from its initialization.

Results

Performance parity under tuned LR. On Gemma-3-1B math, the spread across the five methods is 0.52%. On Llama-2-7B math and code, the spreads are 0.43% and 1.75% respectively. On Qwen3-0.6B at rank 128, all five methods land within 0.43% [§4.3.1, Table 1, Fig. 1].
Variant-specific optimal LR. PiSSA’s optimum is ≈1e-5–3e-5 where vanilla LoRA’s optimum is ≈2e-4–6e-4, a ≈10× gap that holds across all three models [Fig. 1, Fig. 4].
Robustness boundary. At LR = 1.1e-2 (deep in the unstable regime for LoRA-style methods), PiSSA still reaches 27.83% (math) and 26.90% (code) on Gemma while the other methods collapse to ≈0% on at least one task [§4.3.1, Fig. 4].
Rank-dependence inversions. MiLoRA at r=8 Gemma math: +0.8% over LoRA. MiLoRA at r=256 Gemma math: −0.63%. PiSSA at r=8 Gemma math: −1.67%. PiSSA at r=256: +0.33% [§4.3.2, Fig. 5].
DoRA’s true gain. Peak DoRA improvement over LoRA: +1.1% on math, +0.95% on code, both at r=4 only. The original paper’s reported improvements are larger; this is attributed to LoRA’s LR being undertuned in the original comparison [§4.3.2].
Hessian eigenvalue ratio. PiSSA’s λ_max is roughly an order of magnitude larger than LoRA’s across all layers of Qwen, consistent with its required 10× LR reduction. MiLoRA and Init[AB] sit ≈1.5–2× higher, matching their ≈0.5–0.71× optimal-LR ratios [§5.2, Fig. 6].

Why it’s interesting

Two takeaways that travel beyond LoRA. First, this is yet another data point in the “claimed methodological gain was actually a hyperparameter-tuning artifact” pattern that Melis et al. (LSTMs), Schmidt et al. (optimizers), and now this paper (LoRA variants) keep finding — relevant for anyone reading PEFT or fine-tuning benchmarks, especially the DoRA, PiSSA, and MiLoRA results that are quoted as “free wins over LoRA.” Practical implication for Luma’s own LoRA training on DiTs: if you switch initialization (e.g. SVD-based) and don’t re-tune LR, you may see a fake regression or a fake win — the LR for PiSSA-style init should be ≈10× lower than for vanilla LoRA out of the box.

Second, the Hessian-eigenvalue → optimal-LR connection is clean and predictive enough that it suggests a cheap pre-finetuning diagnostic: estimate λ_max on a handful of layers at initialization (Lanczos + HVP is single-batch-cheap) and use 1/λ_max as a prior for the LR sweep range. That’s much cheaper than a 16-point log-grid sweep on a multi-billion-parameter model. The result also connects to How to Set the Learning Rate for Large-Scale Pre-training?‘s thesis that LR tuning is the load-bearing knob for transformer training — both papers find that other hyperparameters (batch size, module-specific LRs there; rank, variant, batch size here) are secondary to getting LR right.