The Universal Weight Subspace Hypothesis

TL;DR

Mode-wise spectral (HOSVD) analysis of over 1,100 finetuned models — 500 Mistral-7B LoRAs (natural-instruction tasks), 500 Vision Transformers (medical / satellite / synthetic), 50 LLaMA3-8B, plus GPT-2 / Flan-T5 / SDXL LoRAs / ResNet-50 — finds that for any fixed architecture the per-layer weights of all models live in a small joint spectral subspace (the “Universal Subspace”), regardless of initialization, dataset, or task. Projecting any of these models onto a 16-dim layer-wise universal subspace recovers near-original accuracy on both IID and OOD ViT tasks, and using the subspace as a merging primitive hits 83.5% normalized accuracy across 8 tasks vs ~64–68% for Task Arithmetic / TIES / DARE-TIES / KnOTS variants. The paper proposes that the universal subspace is architecture-determined rather than data-determined, offers it as a partial explanation for why LoRA / weight-sharing / sparse-training all work, and shows that learning only subspace coefficients can adapt a network to new tasks with 100× fewer trainable parameters.

Key claims

Across ~500 publicly-available Mistral-7B-Instruct-v0.2 LoRAs (each ≥ rank 16) trained on disjoint natural-instruction tasks, HOSVD of the per-layer concatenated $A$ and $B$ submatrices shows the top few spectral components capture the dominant variance for every layer [§3.2.1, Fig. 3].
Reconstructing held-out IID and OOD task LoRAs by projection onto the joint top- $k$ subspace preserves task performance, while the same operation onto the orthogonal “Secondary Subspace” of leftover components performs drastically worse — i.e. the recovered subspace is the signal direction, not an artifact of dimension counting [§3.2.1, Fig. 4].
500 publicly-available finetuned Vision Transformers from HuggingFace (no curation, no access to original training data) yield a layer-wise universal subspace such that projecting an unseen sixth ViT onto a 16-dimensional per-layer subspace recovers IID accuracy 94.1 ± 2.0 vs 94.4 ± 1.7 full training and OOD 87.8 ± 1.5 vs 91.3 ± 2.1 [§3.3, Table 3].
The same shared subspace structure recurs for 50 finetuned LLaMA3-8B models, 177 GPT-2 models, and Flan-T5 models trained on GLUE — i.e. universality is not specific to a backbone or to LoRA-style updates [§3.3, Fig. 6].
Universal-subspace-based model merging beats six SOTA gradient-free baselines (RegMean, Task Arithmetic, TIES, DARE-TIES, KnOTS-TIES, KnOTS-DARE-TIES) by ≥ 14 normalized-accuracy points averaged over 8 tasks (88.1 / 82.3 / 65.9 / 61.3 / 88.3 / 98.1 / 98.5 / 85.1 = 83.5% avg vs 60.9–68.0% for baselines, against an 84.1% finetuned ceiling) and requires no validation-set hyperparameter search [§3.2.1, Table 2].
A 16-dim universal-subspace ViT compresses 500 ViTs (minus task-specific first/last layers) into a single model, claimed as ≈100× memory reduction [§3.3].
ResNet-50s trained from scratch on five disjoint vision datasets (CIFAR-10/100, ImageNet, Oxford-IIIT Pets, EuroSAT) collapse to a shared low-rank joint subspace; the Universal-R50 averages 87.58% vs ResNet-50 full-training 90.89% (a 3.3-pt drop), despite the subspace being estimated from far fewer models than the LoRA/ViT experiments [§3.2.1, Table — small N inducing coarse subspace estimation acknowledged].
SDXL LoRAs collected from HuggingFace and projected onto a universal subspace preserve and sometimes improve CLIP-style alignment for style transfer (avg 19.83 vs 19.73 raw LoRA across 10 styles), the authors attributing the small gain to denoising-by-projection [§3.2.1, Table 1].
Adapting to a new unseen task by freezing the universal-subspace basis and learning only its per-task coefficients yields substantially fewer trainable parameters than retraining a full LoRA, while preserving competitive accuracy [§3.3.1, Table 5].
Theoretical justification: under bounded-effective-rank and Hilbert-space realizability assumptions, the empirical second-moment operator built from learned predictors converges to the population operator with two-level rates — averaging across tasks (∝ $1/\sqrt{T}$ ) and per-task estimation error — and the top- $k$ subspace converges in operator norm, with the eigengap $\Delta_k$ controlling stability [Theorem 2.5, §2].
The framing claim: the universal subspace is architecture-determined — i.e. an “ideal” subspace exists per architecture toward which independently-trained models converge — and the fidelity of the recovered subspace tracks the quantity and quality of available models, not the diversity of their training data [§3, intro of §3].

Method

For each architecture, collect a large set of finetuned checkpoints (500 LoRAs, 500 ViTs, etc.). For each layer, concatenate the per-model weight matrices along a tensor mode and apply Higher-Order SVD (HOSVD) — typically Order 1–2 only — to extract a layer-wise top- $k$ spectral basis. This basis spans the “Universal Subspace” for that layer. Adapting a new model means freezing the basis and learning only the per-task coefficients (a $k$ -dimensional projection); merging $N$ models means computing analytic per-model coefficients in the basis and averaging; compressing a model means storing only its coefficients in the universal basis. There is no training cost to extract the subspace itself (spectral decomposition runs on CPU); the only assumption is that enough independently-trained checkpoints exist for the architecture in question.

The theoretical scaffolding treats each task’s ground-truth predictor as an element of a Hilbert space and analyzes the second-moment operator $S = \mathbb{E}_t[f_t \otimes f_t]$ . Two error sources — finite number of tasks $T$ and per-task estimation noise — yield separate rate terms; the recovered top- $k$ eigenspace converges to the population subspace at a rate governed by the eigengap $\Delta_k = \lambda_k - \lambda_{k+1}$ and the kernel/Rademacher complexity of the per-task hypothesis class.

Results

ViT subspace recovery (Table 3): IID 94.1 ± 2.0 (vs 94.4 ± 1.7), OOD 87.8 ± 1.5 (vs 91.3 ± 2.1) — five held-out finetuned ViTs reconstructed from a 16-dim universal subspace.
Model merging (Table 2): 83.5% avg normalized accuracy across 8 vision tasks (Cars/DTD/EuroSAT/GTSRB/MNIST/RESISC45/SUN397/SVHN) vs RegMean 60.9, TA 63.7, TIES 63.7, DARE-TIES 63.7, KnOTS-TIES 68.0, KnOTS-DARE-TIES 63.9. Finetuned ceiling 84.1%. The universal-subspace merge does not require validation-set tuning of mixing coefficients.
ResNet-50 (5-task disjoint, Table in §3.2.1): Universal R50 87.58% vs full ResNet50 90.89% — 3.3-pt gap despite small training corpus.
SDXL LoRA style transfer (Table 1, 10 styles): Universal-SDXL-LoRA 19.83 vs raw LoRA 19.73 CLIP score average.
Compression: 500 ViTs → 1 universal-subspace model + per-model coefficients ≈ 100× memory reduction.
Mistral-7B LoRAs: dominant variance in first few spectral components across all layers (visualized for top 100 components in Fig. 3a; total basis count larger).

Why it’s interesting

The result is a weight-space version of the Platonic Representation Hypothesis that the wiki’s Emergent Representation Geometry cluster has been tracking on the representation side. The cluster’s strongest existing claims — Universal SAE dictionaries shared across DinoV2 / SigLIP / ViT in Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment, and the cross-model kernel convergence elicited at inference time in Words That Make Language Models Perceive — are about activations. This paper makes the same claim about parameters and supplies the missing primary-source datapoint at the weight level: 500 models per architecture, no curation, low-rank shared structure, recoverable by uncurated HOSVD. It also lands directly on the open question that Effect of the initial configuration of weights on the training and function of artificial neural networks raised — “training stays near init” — and that Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights sharpened: if diverse task experts pack densely around the pretrained init, what direction is the packing in? This paper says: in the top eigenspace of the per-architecture weight covariance, and that direction is shared across architectures’ worth of finetunes. The model-merging beat over Task Arithmetic / TIES / KnOTS-TIES is the practical hook for Parameter-Efficient Finetuning — it suggests that LoRWeB-style “learned basis of LoRAs” (Spanning the Visual Analogy Space with a Weight Basis of LoRAs) could be replaced by an analytic basis recovered from existing public LoRAs, with no router training and no validation tuning. Worth a direct head-to-head against T2L (Text-to-LoRA: Instant Transformer Adaption) and Doc-to-LoRA (Doc-to-LoRA: Learning to Instantly Internalize Contexts).