LightBagel: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation

TL;DR

LightBagel (arxiv title: LightFusion) is a token-efficient unified multimodal model built by stitching — not pretraining — a frozen VLM understanding stream (Qwen2.5-VL-7B) and a video-DiT generation stream (Wan2.2-TI2V-5B), then interleaving zero-initialized multimodal self-attention blocks after every block on both sides. The original blocks are kept intact so the base models’ AR and diffusion capabilities are preserved; the inserted blocks carry the cross-modal fusion. With only ~35B training tokens the resulting model reaches 0.91 GenEval, 82.16 DPG-Bench, 6.06 GEdit-Bench, and 3.77 ImgEdit-Bench — outperforming UniWorld V1, OmniGen2, UniPic, and BLIP3-O8B that all consumed substantially more training tokens. All code, weights, and data are released.

Key claims

Strategically fusing publicly available specialized models (a VLM for understanding + a DiT for generation) can match leading UMMs trained from scratch at a fraction of the token budget — ~35B training tokens vs OmniGen2 / UniPic / BLIP3-O8B’s much larger (estimated) budgets [§1, Fig. 1].
The Double Fusion mechanism interleaves zero-initialized multimodal self-attention blocks after each understanding block and after each generation block, keeping the original blocks unchanged so the base models’ AR and diffusion strengths survive fusion [§3, §1].
Anchoring to QWen2.5-VL-7B (understanding) + Wan2.2-TI2V-5B (generation) — rather than duplicating LLM weights into a parallel generation branch as LMFusion / BAGEL do — removes the order-of-magnitude token budget those recipes need [§2, §3].
The double pathway naturally integrates high-level ViT semantic features (from the understanding stream) with low-level VAE spatial latents (from the generation stream), giving a semantically grounded generation pipeline without a separate alignment loss [§1, §3].
Compositional T2I (GenEval) 0.91 with ~35B training tokens, outperforming UniPic and OmniGen2 (both 0.86) and UniWorld V1 / BLIP3-O8B (both 0.84) [Fig. 1, Table 2].
Complex T2I (DPG-Bench) overall 82.16 [§4, Table 3].
Image editing: GEditBench 6.06, ImgEdit-Bench 3.77 — beating UniPic 3.49, OmniGen2 3.44, and UniWorld V1 3.26 on ImgEdit-Bench [Fig. 1, Table 4].
Full open release of code, weights, and datasets to support future UMM research [§1, §6].

Method

LightBagel’s architecture is a frozen-VLM + frozen-DiT pair with trainable bridge layers. The two pathways run in parallel: the understanding pathway is QWen2.5-VL-7B, kept end-to-end; the generation pathway is Wan2.2-TI2V-5B, also kept end-to-end. After each transformer block on either side, a multimodal self-attention block is inserted. Each inserted block is zero-initialized in its output projection — so at initialization the fused model behaves exactly as the two base models running side-by-side, and the bridge learns to add cross-modal interaction during finetuning. The inserted block on the understanding side reads both modalities’ hidden states; the inserted block on the generation side does the same, in the other direction. The base block weights can be either frozen or finetuned; the paper reports the configuration that yields ~35B tokens of total trainable updates.

The structural contrast with BAGEL and LMFusion is sharp: those methods duplicate LLM weights into a parallel generation branch and treat the fused result as a modality-specialized mixture-of-experts trained from scratch — demanding orders of magnitude more tokens and private data. LightBagel never duplicates weights and never pretrains from scratch; its generation branch is an off-the-shelf video DiT, which also brings spatial-detail inductive bias from VAE latents that an LLM-initialized generation branch lacks.

Results

GenEval compositional T2I: 0.91 overall vs UniPic 0.86, OmniGen2 0.86, UniWorld V1 0.84, BLIP3-O8B 0.84 [Fig. 1, Table 2].
DPG-Bench complex T2I: 82.16 overall [§4, Table 3].
GEditBench image editing: 6.06 [§4].
ImgEdit-Bench image editing: 3.77 vs UniPic 3.49, OmniGen2 3.44, UniWorld V1 3.26 [Fig. 1, Table 4].
Token efficiency: all of the above with ~35B training tokens — the paper’s headline metric is GenEval / ImgEdit-Bench performance per training token, on which LightBagel sits in a Pareto-better region than every filed comparator [Fig. 1].

Why it’s interesting

LightBagel is the cleanest filed counterexample to the “unified-from-scratch” recipe that anchors most of Unified Multimodal Models — including the team’s own prior BAGEL. Where BAGEL initializes the generation branch with duplicated LLM weights, LightBagel bolts a video DiT (Wan2.2-TI2V-5B) onto a frozen VLM (Qwen2.5-VL-7B) with zero-initialized bridge layers, and gets to GenEval 0.91 at 35B tokens. This makes it the second filed paper (after DuoGen: Towards General Purpose Interleaved Multimodal Generation) to argue that a pretrained video DiT is the right image-generation head for a UMM, and the first filed paper to push the “freeze both base models, train only the bridge” recipe to GenEval ≥ 0.9. It also fits the Dual-stream diffusion transformer family by construction — two parallel transformer streams, one per modality function, coupled at every block — but with a fusion primitive (zero-init multimodal self-attention inserted between blocks) rather than the bidirectional cross-attention recipe that LTX-2: Efficient Joint Audio-Visual Foundation Model, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model, and MOVA: Towards Scalable and Synchronized Video-Audio Generation converged on, and with the modality split being understanding-vs-generation rather than audio-vs-video. The Hyper-Bagel direction (Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generation) is the natural follow-on if LightBagel needs few-step inference.