PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

TL;DR

PiSSA is a drop-in replacement for LoRA’s initialization scheme: instead of the “Noise & Zero” (Gaussian $A$ , zero $B$ ) init that leaves the adapter at zero and gradients small for many steps, PiSSA runs a (fast) SVD on each pretrained weight matrix and initializes the two adapter factors with the top- $r$ principal singular components while freezing the residual (low-singular-value) components. Same architecture as LoRA, same trainable parameter count, same deployment story — but the adapter starts life pointing along the directions in which $W$ has the largest stretching, so the first gradient steps do useful work. Empirically PiSSA beats LoRA across 11 models (184M-70B) on 5 NLG + 8 NLU tasks, and its 4-bit variant QPiSSA quantizes the residual rather than the full $W$ , cutting quantization error by ~20% vs QLoRA.

Key claims

LoRA’s Gaussian- $A$ /zero- $B$ init produces small, uninformative gradients in the first phase of finetuning, wasting many steps around the initial point; PiSSA’s principal-components init aligns the adapter with the most significant directions of $W$ from step 0 [§1, §3, Fig. 4].
PiSSA outperforms LoRA on all 3 tested 7B models (LLaMA 2-7B, Mistral-7B, Gemma-7B) across GSM8K, MATH, HumanEval, MBPP, MT-Bench, and often exceeds full fine-tuning — e.g. Gemma-7B on GSM8K: PiSSA 77.78% vs LoRA(gaussian) 75.11% vs Full-FT 72.09% [Table 2].
On GLUE with DeBERTa-v3-base, PiSSA beats LoRA on 7 of 8 tasks and averages 89.83 vs LoRA 88.50 / DoRA 88.98 / AdaLoRA 89.46 at matched 1.33M trainable params [Table 3].
QPiSSA (NF4 quantization applied to the residual $W - BA$ , not the full $W$ ) reduces the initial quantization error by ~20% averaged across projection matrices, and by ~49% on the LLaMA-3-70B “Key” projections at rank 128 [Table 4, §4].
QPiSSA’s loss reduction in the first 100 steps is faster than PiSSA and full fine-tuning on LLaMA-3-8B + MetaMathQA-395K, and its final GSM8K accuracy exceeds full-precision LoRA [Fig. 5].
The advantage holds across 9 models spanning 7B-70B including MoE models (DeepSeek-MoE-16B, Mixtral-8x7B) [§5.4, Fig. 6] and across all tested ranks 1-128 [§5.5, Fig. 7].
Fast SVD (Halko et al. 2011) makes PiSSA initialization complete in “a few seconds” per model, presenting negligible cost for transitioning from a LoRA pipeline [Appendix B].
Grad-norm curves during training show PiSSA tracks full fine-tuning’s shape while LoRA stays much lower, evidence that PiSSA is behaving as a “denoised” version of full-parameter finetuning [Fig. 4b].

Method

PiSSA computes the economy SVD $W = U \Sigma V^\top$ of each targeted linear-layer weight matrix (self-attention + MLP), sorts singular values in descending order, and splits into a top- $r$ principal block and a bottom- $(n-r)$ residual block. The trainable LoRA-shaped adapter is initialized so that $A \gets \Sigma_{[:r]}^{1/2} V_{[:,:r]}^\top$ and $B \gets U_{[:,:r]} \Sigma_{[:r]}^{1/2}$ , i.e. $BA$ reconstructs the top- $r$ SVD approximation of $W$ . The frozen “residual” matrix is $W^{\text{res}} \gets U_{[:,r:]} \Sigma_{[r:]} V_{[:,r:]}^\top$ . Forward is $y = (W^{\text{res}} + BA) x$ ; at initialization $W^{\text{res}} + BA = W$ exactly, so the pretrained function is preserved. Gradients w.r.t. $A$ and $B$ have the same algebraic form as LoRA, but because $B$ and $A$ start non-zero and aligned with the dominant directions of $W$ , the first steps move along informative directions instead of Gaussian noise. QPiSSA additionally applies NF4 quantization to $W^{\text{res}}$ ; because the residual has narrower spectral distribution than $W$ , NF4’s Gaussian-optimized codebook fits it much better (Figures 3(c)-(f)) — quantization error drops without touching the adapter, which stays in Float32.

Results

NLG (Table 2, 100K samples, 1 epoch): Gemma-7B on GSM8K: Full-FT 72.09 / LoRA(gaussian) 75.11 / PiSSA 77.78. MATH: Full-FT 22.71 / LoRA 30.41 / PiSSA 31.33. HumanEval: Full-FT 47.02 / LoRA 53.70 / PiSSA 54.31. Mistral-7B on MBPP: LoRA 58.46 / PiSSA 62.55 (vs Full-FT 51.46).
NLU (Table 3, DeBERTa-v3-base, GLUE): PiSSA 89.83 avg beats DoRA 88.98, AdaLoRA 89.46, LoRA(kaiming) 88.62, LoRA(gaussian) 88.50 at 1.33M params. Wins 7/8 tasks.
Quantization error reduction (Table 4): at rank 128, PiSSA cuts base-model quantization error by 19.4% avg on LLaMA-2-7B, 22.9% on LLaMA-3-8B; LoftQ achieves only 14.6% and 18.1% respectively. On LLaMA-3-70B “Key” projections: PiSSA 49.0% vs LoftQ 17.8%.
Rank scaling (Fig. 7): PiSSA beats LoRA at all tested ranks 1-128 on GSM8K and MATH, with the gap widening at low rank. As rank grows, PiSSA reaches and exceeds full fine-tuning.
Convergence (Fig. 4, LLaMA 2-7B, MetaMathQA-395K, 3 epochs): PiSSA’s loss drops steeply in the first 100 steps and stays below LoRA throughout; PiSSA’s grad-norm curve tracks full-FT’s shape while LoRA’s stays flat.

Why it’s interesting

The right framing for this cluster is that PiSSA is the initialization patch that closes most of the LoRA↔FullFT gap while keeping LoRA’s architecture, deployment story, and quantization compatibility. It complements — rather than competes with — the LoRA-scaling analyses tracked in Parameter-Efficient Finetuning: Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning‘s µA identifies Init[B] + $\alpha=1$ as the LR-transferable LoRA parametrization, and PiSSA identifies a different dimension of the init (which vectors, not which zero-out) that also transfers LoRA behavior toward full-parameter finetuning. It’s also the natural conceptual sibling to LoRA Without Regret‘s “when LoRA matches FullFT” recipe: Thinking Machines describes the data regime and layer coverage under which LoRA=FullFT, PiSSA changes the init so the equivalence shows up earlier in training and at a better local optimum. QPiSSA’s residual-quantization insight is particularly clean — it exploits the singular-value gap to shift quantization off the informative subspace onto the “noise floor,” which the LoRA family has always treated as low-signal anyway.