Spanning the Visual Analogy Space with a Weight Basis of LoRAs

TL;DR

LoRWeB (LoRA Weight Basis) is a visual-analogy editing framework from NVIDIA built on Flux.1-Kontext. Instead of fine-tuning a single LoRA on analogy data — the standard recipe, which struggles to generalize to unseen transformations — LoRWeB jointly trains a basis of $N$ rank- $r$ LoRAs together with a lightweight CLIP-conditioned encoder that picks a softmax mixture of those basis adapters for each input triplet $(a, a', b)$ . The mixed LoRA is injected into the diffusion model at inference time; no test-time optimization is needed. The paper reports SOTA results against RelationAdapter, VisualCloze and EditTransfer on a custom unseen-task evaluation set, with the configuration $N=32, r=16$ landing on the editing-accuracy / preservation Pareto front.

Key claims

A single LoRA fine-tuned for visual analogy generalizes poorly because the diversity of transformations (style transfer, object insertion, layout edits, …) exceeds what a fixed-rank adapter can capture in one parameter blob [§1, §3.2 “Naive solutions and limitations”].
Jointly training a basis $\{(A_i, B_i)\}_{i=1}^{N}$ with associated learnable key vectors $k_i$ and a CLIP+projection encoder that emits softmax mixing coefficients $c_i$ produces a “Mixed LoRA” $\sum_i c_i \alpha B_i A_i$ specialized to the input analogy at inference time [§3.2, Eq. 2–4, Fig. 2].
Each weight matrix in the target network gets its own independent LoRWeB module (own LoRAs, own keys, own projection), so the basis is layer-specific; only the CLIP backbone is shared across layers [§3.2].
The conditioning input is the triplet $(a, a', b)$ encoded separately through CLIP and concatenated before projection; passing CLIP a $2{\times}2$ composite instead measurably hurts edit-accuracy [§4.3 “Layout of encoder input”, Tab. 1].
Softmax over basis coefficients beats Tanh by a wide margin; the authors hypothesize Tanh permits negative coefficients and much larger LoRA norms, pushing the model out of domain [§4.3 “Similarity normalizing function”, Tab. 1].
A large basis matters more than rank: shrinking to $N=16, r=16$ or $N=32, r=4$ both degrade results; bumping rank in isolation hurts editability, attributed to dataset-induced overfitting [§4.3 “Capacity effect”, Tab. 1].
At $N=32, r=16$ , LoRWeB pushes the Pareto front on the preservation × edit-accuracy plane: VLM-based Edit Accuracy 5.94 vs. 4.92 for the single-LoRA Flux baseline, with comparable preservation 7.87 vs. 8.13 [Tab. 1, Fig. 5].
In pairwise VLM judging (Gemma-3 2AFC) LoRWeB is preferred 57.9 % vs. the matched-capacity Flux LoRA, 70.4 % vs. EditTransfer, 68.1 % vs. VisualCloze and 58.5 % vs. RelationAdapter [Tab. 1, Fig. 6].
A 33-user, 45-image-pair user study agrees with the VLM verdict: users prefer LoRWeB over each baseline [§4.2 “User study”, Fig. 6].
Swapping the CLIP encoder for SigLIP2 changes results only marginally — the basis-of-LoRAs idea is robust to the choice of encoder backbone [§4.3 “Alternative image encoders”, Tab. 1].
Training uses the Relation252k corpus (16K analogy pairs across 208 tasks); the evaluation set is custom-built from 100+ Unsplash photos across animals/persons/objects plus 18 HuggingFace community Flux-Kontext LoRAs, yielding 270 analogy triplets across unseen tasks [§4 “Dataset”].

Method

LoRWeB sits on top of Flux.1-Kontext as a conditional flow model. For each targeted weight matrix in the network, the module maintains $N$ rank- $r$ LoRA factor pairs $(A_i, B_i)$ , a matching set of $d$ -dimensional learnable key vectors $\{k_i\}$ , and a small fully-connected projection head. At inference time the analogy triplet $(a, a', b)$ is passed through a frozen CLIP image encoder; the three resulting embeddings are concatenated and projected to a single query vector $q \in \mathbb{R}^d$ . Softmax similarity between $q$ and the key matrix $K$ produces mixing coefficients $c$ , and the layer’s effective adapter is the linear combination $\sum_i c_i B_i A_i$ (the “Mixed LoRA”). This single mixed LoRA is injected into the layer; in parallel, the model also receives a $2{\times}2$ composite image of $(a, a', b, \emptyset)$ as its conditioning input and outputs the same composite with $b'$ in the bottom-right quadrant. All components — basis LoRAs, keys, projections — are jointly trained end-to-end with the standard rectified-flow loss against analogy targets.

The default configuration ( $N=32$ , $r=16$ , $d$ as a single FC projection from CLIP output) matches RelationAdapter’s parameter capacity. Training resolution is capped at the long-edge resize used by Flux-Kontext.

Results

The custom evaluation set is split into 135 in-domain analogy triplets (generated via LLM-paraphrased Relation252k prompts that don’t appear in training) and 135 out-of-domain triplets generated using community Flux-Kontext LoRAs that the base model alone can’t reproduce. Headline numbers from Table 1 (full LoRWeB, $N=32$ , $r=16$ ):

Metric	LoRWeB	Single-LoRA Flux	Tanh-norm	$N=16, r=16$	$N=32, r=4$
VLM Preservation	7.87	8.13	7.94	7.82	7.74
VLM Edit Accuracy	5.94	4.92	4.49	5.49	5.95
LPIPS	0.31	0.20	0.18	0.29	0.31
CLIP Directional	0.21	0.11	0.09	0.19	0.23
Pairwise VLM vs. LoRA	—	51.8 %	48.2 %	59.9 %	60.4 %
Pairwise VLM vs. EditTransfer	70.4 %	63.9 %	58.3 %	73.1 %	70.5 %

The single Flux LoRA at matched capacity wins on Preservation/LPIPS but is well behind on Edit Accuracy and CLIP-directional similarity — i.e. it preserves the input image well because it barely edits it. Reducing basis size or rank closes the edit-accuracy gap but at the cost of preservation; the Tanh-norm ablation collapses on edit-accuracy. SigLIP2-as-encoder and the (Enc. Input) variant fall within ~1 VLM point of the main configuration, demonstrating robustness. The 33-user, 45-pair study mirrors the VLM rankings on every baseline pairing.

Why it’s interesting

Three angles relevant to Luma. First, test-time-free task specialization: LoRWeB replaces hypernetwork-style per-task LoRA generation (notoriously unstable) and inference-time LoRA optimization (slow) with a small softmax over a jointly-trained basis. For image/video editing pipelines that need to switch between many edit “skills” without per-call optimization, this is a cleaner deployment story than carrying a zoo of independent LoRAs.

Second, layer-specific LoRA bases. Every targeted weight matrix gets its own basis and its own router, which empirically matters. This is the visual-analogy specialization of the broader Dravid-et-al observation that LoRAs from independently fine-tuned models live on a meaningful semantic manifold; LoRWeB shows the manifold can be learned (rather than discovered post-hoc) and indexed per-layer.

Third, the parameter-allocation lesson: the ablations make a sharp claim that basis size beats per-LoRA rank under fixed total parameters. This is the inverse intuition to standard “scale up the adapter” recipes, and it dovetails with the LoRA-init/multiplier coupling that Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning studies (µA notes that under typical small- $r$ regimes one of the two LoRA factors barely contributes, suggesting the rank dimension is over-parameterized; LoRWeB independently lands on “many small adapters > one big one”).