Lance: Unified Multimodal Modeling by Multi-Task Synergy

TL;DR

Lance is a 3B native unified multimodal model from USTC + ByteDance covering image and video understanding, generation, and editing, trained from scratch on ≤128 GPUs through a staged multi-task recipe. The architecture keeps a single interleaved text/image/video sequence and splits the network into dedicated understanding and generation experts. At 3B parameters Lance matches or beats much larger unified models on GenEval (0.90), DPG-Bench (84.67), GEdit-Bench (7.30 avg), VBench (85.11 total — best in the unified-model group), and MVBench (62.0 avg — best in the unified-model group), positioning it as the compact-end point of the AR+Diffusion UMM cluster that already includes BAGEL, Mogao, and TUNA on this wiki.

Key claims

Lance is a 3B unified model trained from scratch within a ≤128-GPU budget, covering image+video understanding, generation, and editing in one architecture [Framework].
A shared interleaved text/image/video sequence is processed by a backbone that separates semantic understanding and visual generation through dedicated experts [Framework].
GenEval Overall 0.90 ties the best unified model (TUNA-7B) and beats larger generation-only baselines (FLUX.1-dev 12B at 0.82, Qwen-Image 20B at 0.87) at 3B parameters [GenEval table].
DPG-Bench Overall 84.67 — strongest on the Relation dimension at 93.38, weaker on Other at 80.80 [DPG-Bench table].
GEdit-Bench Avg/G-O 7.30 — the best score among the listed unified models (Lumina-DiMOO 3.91, Ovis-U1 6.42, BAGEL 6.52, InternVL-U(+CoT) 6.88), still trailing the generation-only Qwen-Image-Edit 20B at 8.01 [GEdit-Bench table].
VBench Total Score 85.11 — best in the unified group and above all listed generation-only video baselines (Wan2.1-T2V 14B at 83.69, Hunyuan Video at 83.43), with particularly strong Spatial Relation (93.61) and Color (92.61) [VBench table].
MVBench Avg 62.0 — best in the listed unified-model group (Show-o2 7B 55.7, UniVideo 7B 46.3), still below understanding-only specialists in the same parameter band (Qwen2.5-VL 3B at 67.0) [MVBench table].
Demonstrated capabilities include multi-turn consistency video editing (a source video followed by four linked edits preserving the same subject) and “intelligent” structured-planning / physics-oriented generation cases [Demos].

Method

Lance ingests text, images, and videos as a single interleaved sequence. The same backbone is used for both directions of multimodal work, but understanding and generation paths inside the backbone are handled by dedicated experts — a Mixture-of-Experts-style separation analogous to Qwen3.5-Omni’s Thinker-Talker split, but for image+video understanding vs. image+video generation rather than for text-out vs. speech-out. The training recipe is multi-stage and explicitly multi-task: the paper’s headline framing is “multi-task synergy,” i.e. the staged co-training schedule across understanding, generation, and editing tasks is the contribution rather than a single new architectural primitive.

Inference-time displays for the project page use external 2× super-resolution (ByteDance-Seed/SeedVR2-3B) and 2× frame interpolation (ECCV2022-RIFE), so the raw model outputs are at lower resolution / frame rate than the showcased videos.

Results

Image generation (GenEval, Overall ↑): Lance 0.90 (3B) = TUNA-7B 0.90, > Mogao-7B 0.89, > BAGEL-7B 0.88, > Qwen-Image 20B 0.87, > FLUX.1-dev 12B 0.82 [GenEval].
Image generation (DPG-Bench, Overall ↑): Lance 84.67 (3B), behind Qwen-Image 20B (88.32), TUNA-7B (86.76), Show-o2-7B (86.14), but ahead of BAGEL-7B (85.07), OmniGen2-4B (83.57), Janus-Pro-7B (84.19) [DPG-Bench].
Image editing (GEdit-Bench Avg ↑): Lance 7.30 (3B), best among unified models — beats BAGEL-7B (6.52), InternVL-U-1.7B (6.66), Ovis-U1-1.2B (6.42), Lumina-DiMOO-8B (3.91); behind Qwen-Image-Edit 20B (8.01) and GPT Image 1 (7.49) [GEdit-Bench].
Video generation (VBench Total ↑): Lance 85.11 (3B), best in the unified-model group, ahead of TUNA-1.5B (84.06) and Show-o2-2B (81.34), and above generation-only Wan2.1-T2V 14B (83.69), Hunyuan Video (83.43), Gen-3 (82.32), CogVideoX-5B (81.61). Best Spatial Relation (93.61) and Color (92.61), and best Multiple Objects (93.86) in the table [VBench].
Video understanding (MVBench Avg ↑): Lance 62.0 (3B) — best among the listed unified models (Show-o2 7B 55.7, Show-o2 1.5B 50.6, TUNA-1.5B 54.4, UniVideo-7B 46.3), strong on Object Existence (96.0), Moving Attribute (97.5), Counterfactual Inference (77.0), Moving Direction (63.5). Still trails understanding-only Qwen2.5-VL 3B (67.0) and TimeMarker-8B (67.4) at comparable parameter counts [MVBench].

Why it’s interesting

Lance is the smallest model (3B) yet filed in the Unified Multimodal Models cluster to handle both image and video on both the understanding and generation sides — every other entry on that page is either text-image-only (Janus, BAGEL, DuoGen) or larger (Qwen3.5-Omni, OmniWeaving on Hunyuan-1.5). It also constrains the training budget to ≤128 GPUs, which is interesting as a compute-efficient counter to the “scale-everything-up” arm of the cluster (DeepSeek-V4 collection release (Flash + Pro, up to 1.6T), Mogao-7B). The most direct comparison is OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning (Tencent Hunyuan, video-output unified model on a much heavier MLLM+MMDiT stack) — Lance gets to a similar VBench tier with 1/10th the parameters, suggesting that for the unified setting the bottleneck is the staged training recipe more than the backbone size. On the understanding side, Lance’s MVBench 62.0 at 3B sharpens the UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? finding that unified training tends to underperform base VLMs: a 3B Qwen2.5-VL still reaches 67.0, so unified video understanding still pays an alignment tax even with the dedicated-expert split.