Reconstruction Alignment Improves Unified Multimodal Models

TL;DR

RecA is a resource-efficient post-training method for Unified Multimodal Models (UMMs) that replaces sparse text captions with dense embeddings from the model’s own visual understanding encoder, then optimizes the UMM to reconstruct the input image conditioned on those embeddings. The result is a self-supervised “understanding → generation” realignment loss that needs no captions, no human preferences, and no GPT-4o distillation data. With only 27 GPU-hours of post-training, a 1.5B-parameter Harmon improves from GenEval 0.73 → 0.90 and DPGBench 80.93 → 88.15, surpassing 7B–24B open-source UMMs. The recipe applies to AR (Show-o), AR+MAR (Harmon), and AR+Diffusion (OpenUni, BAGEL) backbones, and also boosts BAGEL’s editing scores (ImgEdit 3.38 → 3.75, GEdit 6.94 → 7.25). Accepted ICLR 2026.

Key claims

Standard UMM training relies on image–text pairs in which captions are sparse and miss fine-grained visual detail even at hundreds of words per image; understanding-encoder embeddings preserve richer, more faithful semantics and can act as dense “text prompts” [§1, Fig. 1].
The RecA objective conditions the UMM on its own visual-understanding embeddings of an input image and applies the model’s native generative loss (cross-entropy on discrete tokens for AR/MAR, diffusion loss for AR+Diffusion) to reconstruct that same image, with no captions involved [§3].
The recipe is broadly applicable: it gives consistent gains across four open-source UMM families — Show-o (AR), Harmon (AR+MAR), OpenUni (AR+Diffusion), BAGEL (AR+Diffusion) [§4, Table 2].
On Harmon-1.5B, RecA lifts GenEval 0.73 → 0.90 and DPGBench 80.93 → 88.15 in 27 GPU-hours; the 1.5B post-trained model exceeds 7B–24B open-source UMMs on these benchmarks without GPT-4o distillation or RL [§4, Table 1].
On BAGEL, RecA also boosts editing benchmarks (ImgEdit 3.38 → 3.75, GEdit-Bench-EN 6.94 → 7.25), showing that the alignment effect transfers to instruction-guided editing, not just text-to-image [§4, Table 3].
Training-order matters: SFT → RecA gives the best results, while RecA → SFT degrades performance — RecA is best positioned as a final refinement stage after standard SFT establishes broad image–language alignment [§4, Table 5].
The choice of conditioning encoder is critical: embeddings from understanding encoders (ViT-style) produce consistently better gains than embeddings from generation encoders (VAE), indicating that RecA benefits from high-level semantic features rather than raw visual detail [§4, ablation].
Replacing SFT entirely with RecA (rather than appending it) on Harmon delivers +10.93 GenEval / +6.54 DPGBench, suggesting reconstruction tuning alone is a viable post-training objective even without paired captions [reported ablation].
The authors deliberately avoid GPT-4o-Image distillation data — which can inflate benchmark scores via template leakage — and instead use open-source MidjourneyV6 + LLaVA Mix-665K, making the gains attributable to the method rather than to teacher-data quality [§4].

Method

A UMM is decomposed into a visual understanding encoder $E_u$ (ViT-style for the four backbones studied) and a generative head $G$ (token decoder for AR/MAR, diffusion head for AR+Diffusion). Standard post-training applies $G$ to caption embeddings; RecA replaces the caption with $E_u(x)$ , the model’s own understanding-encoder output on the input image $x$ , and trains $G$ to reconstruct $x$ under the model’s native generative loss. The loss is therefore self-supervised — no captions, no human preference labels, no teacher model in the loop.

Because the same backbone supplies both the conditioning embedding and the generator, the recipe explicitly closes the understanding ↔ generation loop inside a single model: whatever the understanding encoder produces is graded directly by the generator’s ability to reconstruct from it. This makes “understanding embedding is a useful prompt” a trainable invariant rather than an unmeasured assumption. The intended use is at the very end of training: a UMM that has been SFT-ed on captions is then post-trained for ~27 GPU-hours on the unlabeled reconstruction objective. The two-stage SFT → RecA ordering is reported as both empirically best and conceptually clean — SFT first installs broad language–image alignment, then RecA sharpens fine-grained semantic grounding.

Results

Headline numbers on Harmon-1.5B: GenEval 0.73 → 0.90, DPGBench 80.93 → 88.15 with 27 GPU-hours of post-training [Table 1]. The post-trained 1.5B model surpasses 7B–24B open-source UMMs on GenEval/DPGBench without any GPT-4o-Image distillation data or RL. On BAGEL, ImgEdit moves from 3.38 → 3.75 and GEdit-Bench-EN from 6.94 → 7.25 [Table 3]. Cross-architecture results [Table 2] show non-trivial gains for all four UMM families, with the most notable being Harmon-1.5B’s +12.8 GenEval. Training-order ablation: SFT-then-RecA gives the best score (0.90 GenEval); reversing the order degrades performance; replacing SFT entirely with RecA still gives +10.93 GenEval / +6.54 DPGBench. Encoder-choice ablation: VAE embeddings produce only marginal gains; ViT-style understanding embeddings drive the improvements.

Why it’s interesting

RecA is the paper the Unified Multimodal Models concept page repeatedly cites as the canonical “understanding-improves-generation” recipe — it’s the baseline that Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation (UniMRG) explicitly inverts. The two papers form a clean pair: RecA uses the understanding side to grade and refine the generation side via self-supervised reconstruction; UniMRG uses additional generation targets (depth, segmentation) to refine the understanding side. UniMRG also reports that RecA matches its generation gains on Harmon (GenEval 83.86 ≈ UniMRG’s 85.26) but provides no understanding lift, confirming RecA’s effect is generation-side. The Show-o codebook-bottleneck observation in UniMRG is also attributed to RecA’s earlier finding here — a useful concrete handle for the “representational-capacity ceiling” discussion under Parametric memory.

In the broader frame, RecA fits naturally next to REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training (REPA) and Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (representation entanglement) as another “use a pretrained vision encoder’s embeddings as a training signal for a generative model” move, but specialized to the unified setting where the encoder and generator live in the same backbone. Together they make the encoder-as-supervision pattern look like a default rather than a trick — and the absence of GPT-4o-Image distillation data here (a deliberate methodological choice) is a useful counter-example to the recent trend toward teacher-data-inflated benchmark numbers flagged on Unified Multimodal Models.