Less-to-More Generalization: Unlocking More Controllability by In-Context Generation

TL;DR

UNO (ByteDance Intelligent Creation, April 2025) attacks subject-driven image generation as a model-data co-evolution problem instead of an architecture problem. It uses a base T2I DiT (FLUX.1-dev) to synthesize high-resolution subject-consistent paired data — single-subject pairs first, then a trained single-subject model bootstraps multi-subject pairs — passed through VLM-based quality filtering. On top, it trains a LoRA-rank-512 adapter on FLUX.1-dev with two technical pieces: progressive cross-modal alignment (stage 1 single reference, stage 2 multi reference) and UnoPE, a Rotary Position Embedding scheme that places reference-image tokens at diagonal positions beyond the noisy latent’s max H/W. UNO reports new SOTA on DreamBench (DINO 0.760, CLIP-I 0.835, CLIP-T 0.304) and multi-subject DreamBench (DINO 0.542, CLIP-I 0.733), with the UnoPE ablation showing the position-encoding choice is the dominant lever on subject similarity.

Key claims

Existing customized-generation data is the bottleneck, not the model: real paired data is too restrictive (limited subject variation) and prior synthetic data was low-resolution (typically 512²) and narrow-domain [§1].
A T2I DiT can be used as its own in-context data synthesizer: with crafted text prompts and a grid-target format, FLUX.1-dev generates subject-consistent paired outputs that can be split into (reference, target) pairs at high resolution [§3.2, Fig. 3].
Pipeline is staged single → multi: a single-subject S2I model trained on stage-1 synthetic data is then used (with an open-vocabulary detector + cropping) to synthesize stage-2 multi-subject pairs, avoiding the copy-paste failure that arises if cropped patches are used directly as references [§3.2 Multi-Subject In-Context Generation].
VLM-based filtering on appearance, details, and attributes is the verification gate; the score distribution maps monotonically to downstream DINO and CLIP-I gains, so filtered-by-score is itself the lever [§3.2, Fig. 5].
UNO is a LoRA-rank-512 adapter on FLUX.1-dev trained in two stages: ~5k steps single-subject, then ~5k steps multi-subject, total batch 16, lr 1e-4, 8× A100 [§4.1].
UnoPE assigns reference-image RoPE indices on the diagonal starting beyond the noisy-latent’s (H, W) max — preventing the model from inheriting the reference image’s spatial layout while still seeing it through attention [§3.3 UnoPE, Eq. 8].
Without UnoPE, the model “follows text but barely references the input image” (DINO drops 0.760 → 0.386 at single-subject; 0.542 → 0.386 at multi-subject) — UnoPE is the single largest lever in the ablation [Tab. 5, Tab. 6, Fig. 9].
Progressive cross-modal alignment (single-subject stage before multi-subject) is necessary: skipping it drops DINO 0.542 → 0.511 and CLIP-I 0.733 → 0.721 on multi-subject, with subjects “merging” qualitatively [Tab. 3, Fig. 9].
Using cropped sub-images as references instead of newly generated I_target triggers copy-paste behavior — model pastes the reference instead of obeying text prompts (DINO 0.542 → 0.529; visually worse in Fig. 9) [Tab. 3].
On DreamBench (single-subject, tuning-free), UNO sets new SOTA: DINO 0.760, CLIP-I 0.835, CLIP-T 0.304 — beating OminiControl (DINO 0.684), OmniGen (0.693), RealCustom++ (0.702), and FLUX.1 IP-Adapter (DINO 0.582 / CLIP-I 0.820) [Tab. 1].
On multi-subject DreamBench, UNO leads DINO (0.542) and CLIP-I (0.733), with competitive CLIP-T (0.322) — beating MS-Diffusion (0.525/0.726/0.319), OmniGen (0.511/0.722/0.331), MIP-Adapter (0.482/0.726/0.311) [Tab. 2].
Width-offset variants of UnoPE (placing references off only in W, not on the diagonal) recover most but not all of UnoPE’s gain — height-offset is worse than width-offset, and full-diagonal is best [Tab. 5, Tab. 6].

Method

UNO’s two technical halves are the synthetic data curation framework and the customization model framework.

The data curation framework runs three stages over a taxonomy of 365 Object365 super-classes (further sub-divided by age / profession / attire when relevant). Single-subject in-context generation: an LLM expands each category into millions of (subject, setting, text-template) tuples; a T2I model (FLUX.1-dev) renders each as a grid that visually pairs the same subject in two contexts. The grid is split into reference I_ref and target I_target; pairs with low DINOv2 similarity between the two halves are dropped immediately; surviving pairs go through a VLM scorer that emits a list of scores across appearance / details / attributes (Eqs. 2–3), and only high-aggregate-score pairs are kept. Multi-subject in-context generation: an open-vocabulary detector finds additional subjects in I_ref beyond the primary one; the cropped patches are re-fed into the stage-1-trained S2I model with their subject prompts to re-synthesize I_target containing those additional subjects in plausible interaction. This re-synthesis (rather than direct paste) is the explicit mitigation for the copy-paste failure — the paper argues training on cropped patches directly teaches the model to copy rather than to reference.

The customization model framework is UNO itself — a LoRA-rank-512 finetune of FLUX.1-dev MMDiT trained in two stages. Stage 1 (single-reference): one reference image’s tokens are concatenated with the text and noisy-latent tokens; the DiT runs its standard MM-attention over the union. Stage 2 (multi-reference, ≤4 images): the same construction generalized to multiple [X_text; X_ref^1; …; X_ref^N; X_noisy_latent] token blocks. The progressive aspect is that the model first learns “look at one reference and follow text” before it has to disambiguate multiple references. UnoPE handles the position-encoding problem: FLUX uses 2D RoPE indexed by (H, W) for image tokens and (0, 0) for text. UNO’s reference images can’t reuse the noisy-latent’s (H, W) grid (the model would over-condition on spatial layout) and can’t reuse text’s (0, 0) (the model would lose all spatial information). UnoPE’s fix: place reference-image tokens at positions starting beyond the noisy-latent’s max H and max W, on the diagonal — different references get different diagonal offsets (Eq. 8). The intuition the paper offers is that the diagonal-beyond-max placement decouples the reference’s spatial structure from the target’s while preserving the implicit “this is image” position-class. Training: FLUX.1-dev frozen except the inserted LoRA, lr 1e-4, total batch 16, 8× A100, ~5k stage-1 + ~5k stage-2 steps. Inference is tuning-free per new subject.

Results

On the standard tuning-free DreamBench single-subject benchmark (Tab. 1), UNO reports DINO 0.760, CLIP-I 0.835, CLIP-T 0.304 — best DINO and best CLIP-I across both tuning-based and tuning-free baselines. The closest tuning-free competitor on subject similarity is RealCustom++ (DINO 0.702, CLIP-I 0.794) which UNO beats by +0.058 DINO and +0.041 CLIP-I; CLIP-T is competitive but not leading (RealCustom++ 0.318 vs UNO 0.304), consistent with the standard subject-similarity vs text-fidelity tradeoff. On multi-subject DreamBench (Tab. 2), UNO leads both DINO (0.542 vs MS-Diffusion 0.525) and CLIP-I (0.733 vs MIP-Adapter / MS-Diffusion 0.726), with CLIP-T 0.322 just below OmniGen’s 0.331 — again the similarity-vs-text tradeoff. The user study (Fig. 8) covers five axes (text fidelity at subject level, text fidelity at background level, subject similarity, composition quality, visual appeal) and UNO leads on all five against the comparison set.

Ablations are the more informative read. (Tab. 3 / Fig. 9) Removing the synthesized I_target and substituting cropped patches drops DINO 0.542 → 0.529 and CLIP-I 0.733 → 0.730 — small in absolute terms but qualitative copy-paste behavior is obvious in Fig. 9. Removing progressive cross-modal alignment (no stage-1 single-subject pretraining) drops DINO 0.542 → 0.511 and CLIP-I 0.733 → 0.721. Removing UnoPE is catastrophic: DINO 0.542 → 0.386 and CLIP-I 0.733 → 0.674 — the model essentially stops referencing the input image. Tab. 4 shows progressive cross-modal alignment also lifts the single-subject ceiling (DINO 0.730 → 0.760, CLIP-I 0.821 → 0.835 vs single-subject-only training). The UnoPE variant study (Tabs. 5–6) decomposes its contribution: no offset (just clone the noisy-latent indices) is the worst (DINO 0.470 single, 0.386 multi); width-offset recovers most of the gain (0.717 single, 0.508 multi); height-offset is intermediate; full diagonal is best.

Why it’s interesting

UNO is the cleanest published instance of “the right amount of synthetic data, made by the model you’re training, with a verification gate” for paired customization tasks — a recipe the wiki has been triangulating from three angles. The closest sibling on the wiki is Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset (Phantom-Data, ByteDance, same lab cluster, same year): Phantom-Data attacks subject-consistent video generation with cross-pair retrieval over a 53M-clip corpus + 3B LAION images and explicitly identifies the copy-paste failure as a data problem. UNO attacks subject-consistent image generation with self-generated paired data and identifies the same copy-paste failure with the same “cropped patches as targets teaches copy-paste” diagnosis — but goes one step further by also fixing the position-encoding side of the problem with UnoPE, which Phantom-Data does not touch (because video DiTs handle reference frames through different mechanisms). Both papers share the model-data co-evolution framing; UNO is the earlier, cleaner image-side instance. The synthetic-data gate (DINOv2 + VLM scoring) is structurally identical to Phantom-Data’s two-stage verification (per-pair identity consistency + contextual diversity).

The UnoPE result is also a useful contrast point for the wiki’s position-encoding cluster. The page Camera-Controlled Video Diffusion has accumulated RoPE-as-relative-position recipes (Cameras-as-RPE, GenDoP) where the position index encodes a control axis; UnoPE is the inverse — it places reference tokens at positions specifically chosen to avoid encoding any meaningful spatial relationship between reference and target. The width-offset / height-offset / diagonal ablation (Tabs. 5–6) is a clean controlled experiment that the camera-control papers don’t run for their own RoPE designs. On the parameter-efficient side, UNO is the largest LoRA rank (512) on the Parameter-Efficient Finetuning page so far — the existing cluster runs at rank 16–64 (DiffHDR rank 32, Fit-VTO unspecified-small, Qwen-Edit community LoRAs 16/64). UNO’s rank 512 on FLUX.1-dev for the substantial T2I→S2I retargeting is a data point that the rank scales with the magnitude of the task shift, not just with backbone size.