Skip to content
Bud's Wiki

Parameter-Efficient Finetuning

TL;DR

Section titled “TL;DR”

Methods that finetune a small subset of weights or low-rank adapters instead of the full model — LoRA and its parametrization variants, learned bases of LoRAs with routers, and hypernetworks that generate LoRAs from task descriptions or documents. The current wiki cluster tracks four distinct sub-strands: (i) how the LoRA optimum’s learning rate scales with width / rank / init (µA), (ii) replacing one big LoRA with a learned basis of small ones plus a router (LoRWeB), (iii) producing task-specific LoRAs in a single hypernetwork forward pass from a text description (T2L), and (iv) producing document-specific LoRAs to internalize long context (D2L). These four positions disagree about whether hypernetwork-generated LoRAs are stable enough to ship.

Key claims

Section titled “Key claims”
  • Under Init[B] with α=1\alpha=1, the LoRA learning rate optimum is rank-invariant and aligns with full-finetuning’s optimum, enabling direct LR transfer from LoRA → FFT (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.2 Corollary 4.6, §5.2, Fig. 5).
  • Standard LoRA factorizations are likely over-parameterized at the small-rank regime in current use: under the µA-stable configurations one of the two factors (AA under Init[A], or BB under Init[B] with r≪nr \ll n) contributes effectively zero feature-learning (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §4.1–§4.2 “Learning efficiency”).
  • Replacing a single LoRA with a jointly-trained basis of small LoRAs + a CLIP-conditioned softmax router produces better visual-analogy generalization than a single capacity-matched LoRA, with basis size NN mattering more than per-LoRA rank rr (Spanning the Visual Analogy Space with a Weight Basis of LoRAs §3.2, §4.3 “Capacity effect”).
  • A hypernetwork conditioned on a text task description can emit a usable LoRA adapter for an unseen task in a single forward pass and matches per-task LoRA performance after training on 9 anchor tasks; it can also compress hundreds of LoRAs and zero-shot generalize (Text-to-LoRA: Instant Transformer Adaption Abstract).
  • A hypernetwork conditioned on a long document can emit a LoRA that internalizes that document, achieving near-perfect zero-shot accuracy on needle-in-a-haystack at sequence lengths >5× the target LLM’s native context window, while removing the KV cache for the document (Doc-to-LoRA: Learning to Instantly Internalize Contexts Abstract, §1).
  • Rank-32 LoRA inserted in DiT attention and feed-forward layers (with a fully frozen backbone) is sufficient to repurpose a 14B pretrained video diffusion model (Wan-2.1-VACE) for the out-of-distribution radiance domain of HDR video generation, without modifying the pretrained VAE (DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models §3.3.3, §4.1).
  • A LoRA adding less than 1% of parameters on LTX-Video, with a frozen VAE and a fixed logarithmic input transform (LogC3) + camera-mimicking SDR augmentations, suffices to retarget the model for HDR video — 10k steps on 300 clips in 8 GPU-hours (HDR Video Generation via Latent Alignment with Logarithmic Encoding (LumiVid) §3, §5).
  • LoRA finetuning on Flux.1-dev MMDiT, paired with a structured-conditioning embedding that replaces the text-token slot rather than augmenting it, is sufficient to retarget the model from text-to-image to a measurement-conditioned virtual-try-on task (FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On §Architecture).
  • LoRA-only finetuning on dual frozen SAM ViT-B encoders (RGB + depth) is sufficient to repurpose SAM for self-prompted depth-aware instance segmentation across twelve domain-shifted datasets, with +11 mAP over the best SAM-based baselines on Cityscapes (SPDA-SAM: A Self-prompted Depth-Aware Segment Anything Model for Instance Segmentation §III-A1, §IV-B2 Table II).
  • A community LoRA on Qwen/Qwen-Image-Edit-2509 adds language-prompted camera-axis control (translation / rotation / wide-angle / close-up) with no trigger word, distributed code-free via ModelScope’s training UI — a deployment-side data point that the “tiny adapter on a frozen image-edit DiT” recipe survives without a published training procedure and is compositional with an independently-trained few-step distillation LoRA at inference time (Qwen-Edit-2509-Multiple-angles LoRA — community camera-control adapter for Qwen-Image-Edit-2509 model card §usage).
  • A second community LoRA on Qwen/Qwen-Image-Edit-2509 repurposes the same backbone for a completely different task — photo restoration / upscaling across a wide degradation envelope (16× upscale, 5%-quality JPEG, 50% noise, 64 px motion blur, 3 px Gaussian blur, 16× pixelation) — and ships as an intentional two-LoRA A→B stack (asymmetric ranks 16 and 64 in V4) meant to be wired sequentially, plus a third Lightning LoRA at inference (Qwen-Edit-2509-Upscale-LoRA — community photo restoration adapter for Qwen-Image-Edit-2509 model card §current best, §inference recipe).
  • Unfreezing only the newly-inserted audio cross-attention layers + audio adapter on a 14B Wan2.1-I2V backbone (everything else frozen) is sufficient to add a new conditioning modality (driving audio) without losing the base model’s prompt-following — first shown by MultiTalk and reused by InfiniteTalk for sparse-frame V2V dubbing (InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing §Method, Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) §3.4).
  • Convergence on HDR recipe: Two concurrent April-2026 papers (DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models and HDR Video Generation via Latent Alignment with Logarithmic Encoding (LumiVid)) independently arrive at the same SDR→HDR template — frozen video DiT + frozen VAE + fixed logarithmic input transform + tiny LoRA, trained on PolyHaven-HDRI-rendered clips. The agreement across two backbones (Wan-2.1-VACE-14B vs LTX-Video) is strong evidence that the input-shaping + small-adapter primitive is the right structure when the domain gap is information-theoretic rather than semantic.
  • Convergence on Qwen-Image-Edit-2509 as community-LoRA substrate: two independently-developed community LoRAs (Qwen-Edit-2509-Multiple-angles LoRA — community camera-control adapter for Qwen-Image-Edit-2509 and Qwen-Edit-2509-Upscale-LoRA — community photo restoration adapter for Qwen-Image-Edit-2509) ship the same artifact shape on the same base — frozen Qwen-Image-Edit-2509 + small adapter + 8-step Lighting LoRA stacked at inference — for completely orthogonal task axes (camera-axis control vs photo restoration). Both have 10k+ downloads and 70-100 dependent HF Spaces, and the restoration LoRA was amplified by the Qwen team’s own account. Field evidence that Qwen-Image-Edit-2509 is functioning as the community’s preferred frozen substrate for the “tiny adapter repurposes the DiT” recipe.
  • Conflict: LoRWeB explicitly characterizes hypernetwork-generated per-task LoRAs as “notoriously unstable” and proposes a learned basis + router as the replacement (Spanning the Visual Analogy Space with a Weight Basis of LoRAs “Why it’s interesting”); T2L and D2L are concrete counter-recipes that argue the hypernetwork approach scales when paired with (a) a Perceiver + chunking architecture (D2L) or (b) a frozen text-embedding + learnable module/layer embeddings + two-stage training (T2L). The two camps have not been compared head-to-head on the same task.
  • LoRA-adjacent FFT-style methods are increasingly used as cheap LR-sweep proxies: the µA prescription is to sweep LR with LoRA (~½ GPU memory of FFT) and transfer the optimum to FFT (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning §5.3, Fig. 6d).

Recent contributions

Section titled “Recent contributions”
  • [2026-07-17] WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time: WAM-TTT (Feng, Lyu, Wang et al.) proposes a robotics-domain PEFT variant: freeze a pretrained World-Action Model and update only a small internal key–value memory at test time via self-supervised video prediction on raw human demonstration videos. Meta-training on paired human-robot data with a KV memory reconstruction objective is what makes the resulting memory updates useful for robot control rather than just video reconstruction. Complements adapter / LoRA-style PEFT with a “steering-via-KV-memory-TTT” recipe that requires no annotations and no robot data at deployment time.
  • [2026-07-15] Three things everyone should know about Vision Transformers: Earliest form of the “adapt attention, freeze FFN” PEFT pattern: fine-tuning only MHSA weights of a pretrained ViT to a new resolution (224 → 384) matches full fine-tuning within one standard deviation on ImageNet-val and ImageNet-V2 while saving ~10% memory and time and letting the 2/3-of-parameters FFN weights be shared across tasks. Attention-only beats full fine-tuning on small downstream classification sets (CARS, Flowers) and only lags on iNat-18 with small backbones (ViT-S) — regularizing effect of restricted parameter count.
  • [2026-07-03] GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction: GenRecon extends the frozen-backbone + tiny-LoRA + structured-input-transform recipe across the biggest domain gap on file so far — a 3D generative shape prior (Trellis.2) repurposed for scene-scale multi-view reconstruction, with a projection-based multi-view-to-voxel conditioning pathway playing the role that Log-Gamma / LogC3 / measurement-embedding slots play in DiffHDR / LumiVid / Fit-VTO. Ablation confirms the load-bearing piece is the conditioning geometry, not the LoRA fine-tune: fine-tuning without the 3D condition recovers appearance but loses pose alignment (SAGE-10k F-score@10cm 0.6312 with vs 0.3666 without).
  • [2026-07-02] Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training: Depth-axis PEFT: training a single mid-stack transformer layer during RL post-training (with all others frozen) recovers most of the gain of full-parameter RL across seven Qwen3/Qwen2.5 checkpoints and three RL algorithms. Complements the weight-matrix-axis PEFT (LoRA and variants) that dominates this page, and naturally composes with the off-principal weight-space constraint identified by The Path Not Taken: RLVR Provably Learns Off the Principals — mid-stack single-layer + off-principal LoRA-within-layer is the obvious next PEFT-for-RL recipe.
  • [2026-07-02] Learning to Replicate Expert Judgment in Financial Tasks: Full-parameter Qwen3-235B fine-tune on Tinker beats GPT-5.2/5.4 / Claude / Gemini variants on six financial information-triage tasks at 13.8× lower per-task inference cost — a domain-specific-fine-tune-vs-frontier-prompting data point that this cluster does not yet contain (all prior entries fine-tune with LoRA or hypernetwork-generated LoRAs, not full parameters). Deployment-side argument that the “customize an open-weight base” lever wins against frontier models when high-quality expert labels are available.
  • [2026-06-19] Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation: DKT reports a clean LoRA-vs-full-finetune ablation when adapting Wan (a 1.3B/14B video DiT trained with flow matching) into a transparent-object depth regressor: at 1.3B parameters, LoRA reaches REL 11.17 vs full FT’s 11.86 on ClearPose, and scaling LoRA to 14B compounds the gain to REL 9.72 — the authors frame LoRA as a forgetting-mitigation tool against catastrophic loss of the base VDM’s optical priors when porting the model from “generate refraction” to “predict depth from refraction” [Tab. III].
  • [2026-06-02] Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning: Stable-Layers uses LoRA as the adaptation surface for Flow-GRPO post-training of Qwen-Image-Layered — the entire RL fine-tune updates LoRA weights only, with reward signal coming from a VLM judge.
  • [2026-06-02] EgoX: Egocentric Video Generation from a Single Exocentric Video: EgoX adds another datapoint that lightweight LoRA on a frozen large video diffusion backbone (4DNeX) can span a very large domain gap — here, third-person→first-person view translation with near-zero source/target overlap — when paired with shaped input conditioning (width-concat exocentric video + channel-concat egocentric prior) and a geometry-guided attention mask. Same structural pattern as DiffHDR / LumiVid (LoRA + input-side domain transform for HDR retargeting), with the egocentric prior video playing the role the Log-Gamma / LogC3 transform plays in the HDR recipes.
  • [2026-06-01] RenderFlow: Single-Step Neural Rendering via Flow Matching: RenderFlow repurposes a pretrained forward-rendering flow-matching backbone for the inverse task (intrinsic decomposition) via a lightweight adapter module, without retraining the backbone — same one-backbone-many-tasks adapter pattern seen across the wiki, here applied across the forward/inverse rendering boundary.
  • [2026-05-24] Qwen-Edit-2509-Upscale-LoRA — community photo restoration adapter for Qwen-Image-Edit-2509: Qwen-Edit-2509-Upscale-LoRA — community LoRA on Qwen/Qwen-Image-Edit-2509 retargeting the backbone to photo restoration / upscaling across a broad degradation envelope (16× upscale, 5%-quality JPEG, 50% noise, 64 px motion blur, etc.). Ships as a two-LoRA A→B stack (V4 candidates have ranks 16 and 64 respectively) meant to be wired sequentially, plus an 8-step Lighting LoRA at inference — i.e. three LoRAs on a single frozen backbone in production. 12.6k downloads, 71 dependent HF Spaces, officially amplified by @Ali_TongyiLab (the Qwen team’s own account). The second independently-developed community LoRA on Qwen-Image-Edit-2509 the wiki tracks, providing convergent evidence that this base is functioning as the community’s preferred frozen substrate for tiny-adapter retargeting.
  • [2026-05-24] InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing: InfiniteTalk — direct sequel to MultiTalk by the same author group, reusing the same partial-parameter recipe on Wan2.1-I2V-14B (only the inserted audio cross-attention layers + audio adapter unfrozen). Adds a new sparse-frame video-to-video dubbing mode where a handful of source frames anchor identity and the audio drives every other frame. Another data point that “new conditioning modality = new cross-attention layers + adapter only” is a stable recipe for adapting a 14B foundation video model.
  • [2026-05-24] Qwen-Edit-2509-Multiple-angles LoRA — community camera-control adapter for Qwen-Image-Edit-2509: Community LoRA on Qwen/Qwen-Image-Edit-2509 exposing a closed set of natural-language camera commands (“Move the camera forward”, “Rotate 45° to the left”, “Turn to wide-angle / close-up”) with no trigger word, trained code-free via ModelScope’s hosted training UI. Recommends stacking at inference time with lightx2v/Qwen-Image-Lightning, a separately-trained few-step distillation LoRA. 87k downloads and 100+ HF Spaces in the month after release — a field-deployment data point that the “tiny adapter on a frozen image-edit DiT + structured language-prompt slot” recipe survives the no-published-procedure regime. Also a concrete instance of the open compositional question (does loading two independently-trained LoRAs onto the same backbone behave well?), with the answer empirically: yes, well enough for 100 Spaces to ship it.
  • [2026-05-24] SPDA-SAM: A Self-prompted Depth-Aware Segment Anything Model for Instance Segmentation: SPDA-SAM — LoRA-only finetuning of two parallel frozen SAM ViT-B encoders (one for RGB, one for an estimated depth map from Depth-Anything-v2). A Coarse-to-Fine RGB-D Fusion Module mixes the two streams and a Semantic-Spatial Self-Prompt Module replaces SAM’s external prompts with internally-generated semantic + spatial prompts. Extends the “frozen large backbone + tiny adapter + domain-shaping input” pattern from generative diffusion (DiffHDR, LumiVid, Fit-VTO) to a perception backbone, with depth taking the structural role that Log-Gamma / measurement embeddings play in those papers.
  • [2026-05-24] HDR Video Generation via Latent Alignment with Logarithmic Encoding (LumiVid): LumiVid — <1%-parameter LoRA on frozen LTX-Video + frozen VAE, with a fixed LogC3 transfer-curve pre-encoder and three SDR-side augmentations (MP4 compression, contrast scaling, highlight/shadow blur) forcing generative reconstruction. Mirrors DiffHDR’s recipe on a different backbone, providing the second datapoint that “input transform that minimizes KL against pretrained input distribution + tiny LoRA” is the load-bearing structure for cross-radiance-domain transfer.
  • [2026-05-24] FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On: Fit-VTO — LoRA finetune of Flux.1-dev MMDiT for fit-aware virtual try-on. Replaces the text-embedding slot with a learned measurement embedding m_embed (concatenated body + garment measurements), with layflat-garment and person latents concatenated to the noise tokens. Another instance of “swap the text-conditioning slot for structured task-conditioning + LoRA the backbone” as a recipe for repurposing large pretrained DiTs to domain-specific tasks (cf. DiffHDR’s Log-Gamma input transform + rank-32 LoRA for HDR).
  • [2026-05-24] DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models: DiffHDR — rank-32 LoRA finetune of frozen Wan-2.1-VACE-14B that retargets the model to HDR video generation. Demonstrates that a sufficiently aggressive input-side domain transform (Log-Gamma color mapping) lets a small adapter span a large domain gap, instead of requiring backbone or VAE finetuning.
  • [2026-05-23] Doc-to-LoRA: Learning to Instantly Internalize Contexts: Doc-to-LoRA — hypernetwork that emits a document-internalizing LoRA in one forward pass, with a Perceiver backbone + chunking that allows generalization to contexts >5× the target LLM’s native window.
  • [2026-05-23] Text-to-LoRA: Instant Transformer Adaption: Text-to-LoRA — hypernetwork that emits a task-specialized LoRA from a natural-language task description, matching per-task LoRA performance and zero-shot generalizing across the SNI task suite.
  • [2026-05-23] Spanning the Visual Analogy Space with a Weight Basis of LoRAs: LoRWeB — learned basis of N rank-rr LoRAs + CLIP-conditioned softmax router for visual-analogy editing on Flux-Kontext; basis size NN matters more than per-LoRA rank rr.
  • [2026-05-23] Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning: µA — joint width/rank µP-style scaling analysis of LoRA, identifying the Init[B] + α=1\alpha=1 configuration as the one that transfers directly to FFT LR.

Open questions

Section titled “Open questions”
  • Hypernetwork-generated vs basis-of-LoRAs: head-to-head benchmarking of T2L-style (one hypernetwork emits per-task LoRA) against LoRWeB-style (router over learned basis) on the same task suite. No filed paper does this comparison.
  • Should Luma’s default LoRA init be updated? Current Slack-stated default is Init[B] + α=1/r\alpha=1/r, which is not the configuration µA’s recommended LR-transfer workflow uses; switching to Init[B] + α=1\alpha=1 is the cheap experiment.
  • Is the LoRA factorization over-parameterized for the ranks in actual use? µA’s “learning efficiency imbalance” claim and LoRWeB’s “basis-beats-rank” ablation both suggest yes, from independent angles.
  • Hypernetwork output target: D2L emits LoRA factors directly; T2L emits per-(layer, module) factors via a shared text-conditioned hypernetwork; LoRWeB allocates a per-layer basis with a shared CLIP-conditioned router. Which factorization wins under matched capacity?
  • Compounding adapters: can D2L (document) and T2L (task) adapters be applied jointly to one LLM, or do they collide? No filed paper does this composition. The Qwen-Image-Edit-2509 community releases are deployment-side data points that two-LoRA (Multiple-angles + Lightning) and even three-LoRA (Upscale-A + Upscale-B + Lightning) stacks compose well enough to ship in 70-100+ user-facing Spaces — but no controlled measurement of degradation vs. any LoRA in isolation exists.
  • Hand-wired two-LoRA decomposition vs single LoRA at matched capacity: the Qwen-Edit-2509-Upscale-LoRA ships intentionally factored as an A→B pair (asymmetric ranks 16 and 64) rather than a single capacity-matched LoRA. This is adjacent to LoRWeB’s “basis-of-LoRAs beats single LoRA” claim (Spanning the Visual Analogy Space with a Weight Basis of LoRAs) but the LoRWeB result uses a learned router rather than a fixed sequential pipeline. No filed controlled experiment confirms or refutes the hand-wired decomposition.
  • Backbone-vs-conditioning lift attribution: DiffHDR and LumiVid both achieve LDR→HDR retargeting with rank-tiny LoRA plus a domain-shaping input transform (Log-Gamma / LogC3). Across these two papers, the backbone differs (Wan-2.1-VACE-14B vs LTX-Video) but the recipe is otherwise the same — strong indirect evidence that the input transform is most of the lift. A controlled ablation that holds the transform fixed while varying LoRA rank (and vice versa) would settle it directly. Same question applies to Fit-VTO: how much of the gain over IDM-VTON comes from the LoRA finetune vs from replacing the text-prompt slot with a structured measurement embedding?
  • LogC3 vs Log-Gamma vs PQ: LumiVid prefers LogC3 over PQ on perceptual metrics (JOD 7.86 vs 7.62) but the gap is small; DiffHDR uses Log-Gamma. Is there a principled reason to pick one cinematic transfer curve over another, or are they roughly interchangeable in this regime?
  • Dual-modality LoRA: SPDA-SAM finetunes two parallel LoRA-adapted copies of the same backbone (one for RGB, one for depth) and fuses them at multiple depths. Does this generalize? Could the same recipe retarget a video DiT to consume per-frame depth/segmentation maps as a parallel conditioning stream rather than via channel-cat with the RGB latent?
  • Closed-set language-prompt control as a “conditioning slot”: the Qwen-Edit-2509-Multiple-angles LoRA (Qwen-Edit-2509-Multiple-angles LoRA — community camera-control adapter for Qwen-Image-Edit-2509) does not swap the text-conditioning slot (cf. Fit-VTO’s measurement embedding); it leaves the prompt encoder intact and surfaces camera-axis control on a small closed set of natural-language commands. How much of the result is the LoRA learning a new behavior vs. surfacing latent structure already present in Qwen-Image-Edit-2509? A useful ablation: same prompts against the base model with no LoRA. The Upscale-LoRA uses a similar pattern with "Enhance image quality, ..." prefix — same ablation applies.
  • New-modality-as-cross-attention recipe: MultiTalk and InfiniteTalk both add a new conditioning modality (audio) to a 14B Wan2.1-I2V base by inserting new audio cross-attention blocks and unfreezing only those + the audio adapter. The base model’s full-parameter finetune is empirically worse under limited audio data, but this hasn’t been tested against rank-tiny LoRA inserted in the existing text-cross-attention layers — i.e. is the new-layer-insertion the load-bearing structure, or could LoRA on existing layers carry the same conditioning information?

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2024-01

Section titled “2024-01”

Unknown date

Section titled “Unknown date”