HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing

TL;DR

HY-WU (“Weight Unleashing”) is a Tencent Hunyuan framework that treats adaptation as functional memory: instead of fine-tuning a single LoRA, a transformer-based generator synthesizes instance-conditioned LoRA updates on-the-fly from a hybrid image+instruction condition, and injects them into a frozen image-editing backbone for the forward pass. The generator is trained end-to-end through the downstream editing loss (no checkpoint collection, no reconstruction targets), which the authors argue is what makes the approach scale to large editing backbones where prior hypernetwork recipes stalled. Part I instantiates the framework on text-guided image editing (TI2I) as a stress test: in pairwise human preference (GSB), HY-WU wins 67–78% against leading open-source editors (Step1X / Qwen-Image-Edit / LongCat / FLUX), 55.6% vs. Seedream 4.5, 55.5% vs. GPT Image 1.5, and is roughly tied with Nano-Banana 2 / Pro. It ranks #1 on GEdit-Bench and #2 on ImgEdit-Bench among open-source models. The conceptual claim is bigger than the editing numbers: adaptation should target a conditional family of operators indexed by instance, not a single shared parameter point.

Key claims

Static-adaptation interfaces (full fine-tuning, single LoRA) commit inference to one fixed operator; under heterogeneous editing objectives this yields two structural failure modes — infeasible sharing (Fig. 1a) and over-specialization (Fig. 1b) [§1, §1.2].
HY-WU reframes adaptation as a memory-interface problem and proposes a third interface beyond static-parameter memory and context/retrieval memory: functional memory implemented as a neural module $g_\theta(c)$ that emits operator updates per instance [§1.1, Eq. 4].
Crucially, HY-WU is trained on-the-fly directly from the downstream editing loss, with the backbone frozen and gradients flowing through the generated LoRA factors into the generator — avoiding the checkpoint-collection / reconstruction-loss bottleneck that prior hypernetwork approaches relied on [§3.1, Fig. 2].
A rank-anchored 2D parameter tokenization uses the (fixed) LoRA rank dimension as a stable axis and slices the heterogeneous layer dimension into fixed-size segments, producing uniformly-shaped “parameter tokens” that scale to large backbones [§3.4, Fig. 4].
Hybrid condition: a frozen SigLIP2 encoder ingests the input image and the editing instruction together; the concatenated features condition the parameter generator via cross-attention [§3.3, Eq. 6, §3.5].
The method is reported practical even for very large backbones (80B parameters), enabled by the parameter-tokenization choice and a factorized-attention design over parameter-token sequences [Abstract, §3].
Instance-alignment ablations show that gains come from correct condition→update alignment, not raw capacity: matched-capacity controls that scramble the alignment collapse the gains [§2.4, §5.3, Tab. 7].
The learned set $\{g_\theta(c)\}_c$ exhibits emergent semantic structure in weight space — generated updates cluster by editing semantics, supporting the framing as a structured conditional family rather than a bag of per-instance perturbations [§1.1, §5].
Headline GSB win-rates: 67–78% vs. open-source editors (Step1X/Qwen-Image-Edit/LongCat/FLUX), 55.6% vs. Seedream 4.5, 55.5% vs. GPT Image 1.5; 47.6% vs. Nano Banana 2 and 46.2% vs. Nano Banana Pro [Abstract].
Public-benchmark standing: #1 on GEdit-Bench and #2 on ImgEdit-Bench among open-source models; additional gains on the team’s internal WU-Eval [Abstract].

Method

HY-WU pairs a frozen diffusion editing backbone with a learned parameter generator. For each editing instance $(x, p)$ (image + instruction), three stages run: (1) Hybrid condition extraction — a frozen SigLIP2 encoder embeds image and instruction jointly, producing a single condition feature $c$ [§3.3]. (2) Parameter generation — a transformer cross-attends to $c$ and emits a sequence of parameter tokens. The trick that makes this scale to heterogeneous-dimension backbone weights is rank-anchored 2D tokenization: each LoRA factor pair $(A, B)$ for a given backbone weight $W \in \mathbb{R}^{m\times n}$ is reshaped so the rank dimension $r$ (which is constant across modules) becomes the anchor axis, and the $n$ dimension is sliced into fixed-size segments of width $C$ , yielding tokens of uniform shape $(r, C)$ across all layers and modules [§3.4]. Tokens for all adapted modules in a layer are concatenated; tokens across layers form the full parameter sequence. (3) Detokenize + inject — the generated tokens are detokenized back into per-layer $(A, B)$ LoRA matrices and inserted into the frozen backbone for that instance’s forward pass.

Training is fully end-to-end: only the generator (and the light projection/cross-attention modules) receive gradients; the backbone and SigLIP2 are frozen. The training signal is the standard rectified-flow / diffusion editing loss on the output of the LoRA-modified backbone — no checkpoint bank, no reconstruction target, no per-task collection step. The architectural designs the paper highlights as necessary for large-scale on-the-fly synthesis are: rank-anchored parameter tokenization, factorized attention over the (potentially long) parameter-token sequence, and a stable-initialization scheme that keeps the generated $\Delta W$ near zero at the start of training.

Baselines designed to separate confounders: SFT and Shared-LoRA (static single-point memory), Single-LoRA-per-objective (modular isolation without conditional routing), and diagnostic controls that preserve trainable-parameter count but remove instance↔update correspondence [§2.4, §5.3].

Results

Stress-test setup: TI2I editing under deliberately heterogeneous, sometimes directionally-opposed objectives (e.g. restoration vs. aging, blur vs. deblur, add vs. remove) — the paper argues these expose static-adaptation’s “single compromise point” failure mode quantitatively rather than as an anecdote [§2.2].

Headline numbers from the abstract:

Pairwise human preference (GSB): 67–78% win-rate against open-source editors (Step1X-Edit / Qwen-Image-Edit / LongCat-Image / FLUX-Kontext); 55.6% vs. Seedream 4.5; 55.5% vs. GPT Image 1.5; 47.6% vs. Nano Banana 2; 46.2% vs. Nano Banana Pro [Abstract].
Public benchmarks: #1 on GEdit-Bench among open-source models; #2 on ImgEdit-Bench among open-source models [Abstract].
Internal WU-Eval: substantial gains across all evaluation dimensions [Abstract].
Mechanism: instance-alignment ablations (Tab. 7) show that scrambling the instance→update correspondence while preserving parameter count collapses the gains, isolating conditional routing (not capacity) as the source of improvement [§5.3].
Geometry: generated updates in weight space exhibit semantically-coherent clustering (edits with similar intent map to nearby parameter neighborhoods), supporting the “structured conditional family” framing rather than an unstructured bag of per-instance perturbations [§5].

Why it’s interesting

HY-WU is the image-editing counterpart of Text-to-LoRA: Instant Transformer Adaption (T2L, task-description → LoRA) and Doc-to-LoRA: Learning to Instantly Internalize Contexts (D2L, document → LoRA): three papers from three different groups, all generating LoRA adapters via a hypernetwork conditioned on different signals (task text, document content, image+instruction). HY-WU’s specific contribution to that lineage is the rank-anchored parameter tokenization — a structural recipe for scaling weight generation to large editing backbones, where the previous bottleneck for hypernetwork approaches was checkpoint-supervised training. This is also a direct counter-recipe to Spanning the Visual Analogy Space with a Weight Basis of LoRAs‘s LoRWeB, which explicitly characterized hypernetwork-generated LoRAs as “notoriously unstable” and proposed a learned basis of LoRAs with a CLIP-conditioned router as the alternative; HY-WU and LoRWeB are now the cleanest head-to-head framing of the “generate-the-LoRA” vs. “route-over-a-basis” design choice in image editing (though they have not been compared on the same benchmark). Finally, the framing language — “functional memory” as a third interface beyond static-parameter memory and retrieval — is unusually close to the taxonomy proposed in the Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey survey, and gives a clean vocabulary for talking about “context-as-weights” methods like End-to-End Test-Time Training for Long Context under the same umbrella.