EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture

TL;DR

EMMA is a 4B-LLM-backed unified multimodal architecture for understanding, text-to-image generation, and instruction-based editing. Its central design is to align both the understanding encoder (SigLIP2 + 2×2 pixel shuffle) and the generation encoder (DCAE) to the same 32× spatial compression so the two branches operate on the same grid, then fuse them by channel-wise concatenation instead of token-wise concatenation — cutting visual token count without losing modality-specific features. Shallow LLM layers are shared across understanding and generation; deep layers are decoupled. A STEM expert added via MoE on the understanding encoder targets charts/math/diagrams. The 4B model reportedly outperforms BAGEL-7B and matches Qwen3-VL / Qwen-Image while using ~1/5 the visual tokens for editing.

Key claims

A 32× autoencoder (DCAE) on the generation side, matched to a 32×-compressed understanding tokenizer (SigLIP2 patchify + 2×2 pixel shuffle), aligns the two branches on a single spatial grid and rebalances per-task token counts so neither branch dominates [§ Method, Architecture].
Channel-wise concatenation of understanding and generation tokens (rather than BAGEL-style token-wise concatenation along the sequence axis) keeps the visual sequence length flat while fusing both feature streams [§ Method; contrast with BAGEL].
A shared-and-decoupled trunk — cross-task parameter sharing in shallow LLM layers, parameter decoupling in deep layers — is the configuration the paper attributes the joint understanding+generation gains to [§ Method, Architecture].
A STEM expert routed via an MoE gate on the understanding encoder targets science/math/diagram inputs and is added in a late training stage after the rest of the stack is trained [§ Training stages].
Hybrid attention masking — pure causal masking on Und tokens and intra-image bidirectional attention on Gen tokens — is used (following Transfusion) [§ Method, Attention strategy].
Training is staged: visual feature alignment → pre-training → SFT → quality tuning (1:1:1 mixture) → STEM expert + router training [§ Training stages].
EMMA-4B scores 73.0 on MMVet vs 67.2 for BAGEL-7B (smaller LLM, higher MLLM score) [Table 1].
On GenEval, EMMA reaches 0.91 without prompt rewriting and 0.93 with rewriting; on DPG-Bench it is competitive with Qwen-Image [Table 1].
On GEdit-Bench-EN, EMMA edges past unified baselines (BAGEL, OmniGen2, UniWorld-V1) while using ~20% of their visual tokens for editing — roughly 5× fewer tokens [Table 1, § Efficiency].
The authors note current editing benchmarks insufficiently measure subject-consistency — they explicitly exclude GPT-Image-Edit-1.5M from training because it degrades subject preservation in their pipeline [§ Editing data].

Method

EMMA is built around four design choices targeting token efficiency in a unified architecture. (1) Two matched 32× encoders. Und-Enc is SigLIP2-so400m-patch16-512 followed by a 2×2 pixel shuffle (the standard InternVL trick), yielding a 32× spatial compression. Gen-Enc is DCAE (deep compression autoencoder) frozen at 32×. Because both branches now produce tokens on the same grid, they can be aligned position-for-position. (2) Channel-wise fusion. Where BAGEL concatenates Und and Gen tokens along the sequence axis (doubling the visual token count), EMMA stacks them along the channel dimension — same number of tokens, fused features per position. (3) Shared-and-decoupled trunk. Shallow LLM layers are shared across Und and Gen tasks (cross-task parameter sharing), deeper layers split into task-specific parameters (parameter decoupling). (4) MoE STEM expert. A second expert is added to the understanding encoder with a learned router that selects STEM vs general expert per input.

Training objectives: next-token prediction for understanding, flow matching with velocity prediction for generation and editing. Attention mask: causal everywhere for Und; for Gen, visual tokens within a single image attend bidirectionally. Native-resolution support is achieved by interpolating SigLIP2’s positional embeddings. 2D positional encoding is applied to visual tokens before they enter the LLM (which uses 1D RoPE for the sequence axis).

Training proceeds in five stages: visual feature alignment (only the Und adapter trained), pre-training, supervised fine-tuning, quality tuning under a 1:1:1 Und:T2I:Edit mixture, then a final stage that introduces the STEM expert and trains the MoE router. Editing data is built from a VLM-instruction → edit → filter → reverse-pair pipeline; the authors filter aggressively via face similarity for portraits.

Results

Understanding (MLLM benchmarks): EMMA-4B reports MMBench / MMMU / MMVet that beat or match BAGEL-7B at smaller LLM size — with the largest gap on MMVet 73.0 vs BAGEL 67.2 [Table 1].
T2I generation: GenEval 0.91 (no prompt rewriting) / 0.93 (with rewriting) — competitive with Qwen-Image and ahead of unified baselines BAGEL, OmniGen2, UniWorld-V1, BLIP3-o, Janus-Pro on the same benchmark [Table 1]. DPG-Bench is also reported competitive with specialized T2I experts.
Editing: GEdit-Bench-EN edges past OmniGen2 (6.42) and UniWorld-V1 (4.85) while using roughly 1/5 the visual tokens those baselines need for editing [Table 1, § Efficiency].
Token efficiency: at the editing setting, EMMA’s channel-wise fusion + matched 32× compression yields ~5× fewer visual tokens than token-wise-fused 7B unified peers — the paper’s headline efficiency claim.
The authors flag that current editing metrics overweight VLM-judged adherence and underweight subject/identity preservation, and structure their data pipeline accordingly.

Why it’s interesting

EMMA is a direct entrant in the Unified Multimodal Models design-space race — and it picks a specific, falsifiable architectural bet that is interesting precisely because it differs from the page’s current cluster: rather than the AR+Diffusion decoupled-MLLM-then-DiT recipe used by DuoGen: Towards General Purpose Interleaved Multimodal Generation (DuoGen) and the encoder-free Mixture-of-Transformer route of NEO-unify: Building Native Multimodal Unified Models End to End (NEO-unify), EMMA keeps both pretrained encoders, matches their compression ratios, and fuses on the channel axis inside one LLM trunk. It also reads as a natural successor to Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation‘s “pixel embeddings as the unified token” thesis from the opposite direction — Tuna-2 ablates encoders entirely, EMMA keeps two and aligns their grids. The MoE STEM-expert detail is the kind of late-stage capability bolt-on that connects this paper to MoE Routing Design and to the broader “specialized experts for difficult visual subdomains” trend (Routing-Free Mixture-of-Experts).