Skip to content
Bud's Wiki

Omni: Context Unrolling in Omni Models (ByteDance Seed)

TL;DR

Section titled “TL;DR”

ByteDance Seed’s Omni (informally “bagel2” — the BAGEL-line successor) is a unified multimodal model trained natively across text, images, videos, 3D geometry, and hidden representations, served as a standard MoE with only 3B activated parameters. Its central mechanism is Context Unrolling: the model explicitly rolls out intermediate textual chains-of-thought and/or visual context tokens (depth, novel views, camera poses) into its own context before producing the final prediction, and gains compound when text and visual unrolling are combined. The project page reports SOTA on T2I (GenEval2 54.12), image editing (G_O 7.75 on GEdit-Bench EN), and competitive video generation (VBench 1.0 Total 85.07), and brings camera-pose estimation, novel-view synthesis, and monocular depth in-house as atomic primitives.

Key claims

Section titled “Key claims”
  • A unified MoE model with only 3B activated parameters covers any-to-any multimodal modeling including understanding, T2I, image editing, T2V, video editing, 3D reconstruction, camera-pose estimation, novel-view synthesis, and monocular depth — all in one model [§Headline results across understanding, generation, and geometry benchmarks].
  • Context Unrolling: training natively across modalities enables the model to explicitly reason across heterogeneous modal representations before producing predictions, treating each modality as an atomic primitive that is composable, invocable, and writable back into a shared context [§intro].
  • Text-only “thinking” CoT on the understanding side gives consistent but small gains across BLINK / MMStar / MMBench-V11 / SimpleVQA / AI2D / Chartqa / Docvqa / HallusionBench / Erqa / MMSI: baseline → +text-thinking moves MMStar 59.4 → 66.5 and Erqa 41.5 → 44.5 [§Visual understanding table].
  • On compositional T2I (TIFAGM), visual context unrolling beats text unrolling, and combining them is strictly best: baseline TIFAGM 29.25 → +long-text 43.94 → +visual unrolling 48.02 → +long-text & visual unrolling 53.44 → +oracle & visual unrolling 57.21 [§Visual generation table].
  • Combining text thinking with visual token unrolling yields the most consistent gains on T2I; textual and visual contexts are complementary rather than redundant [§Visual generation].
  • Monocular depth: a depth-related caption (textual context describing scene structure) recovers key geometric features, and adding unrolled visual context further corrects residual ambiguities — δ₁ 83.21% → 84.01% and AbsRel 0.2028 → 0.1970 [§Monocular depth context table].
  • Spatial reasoning uses camera pose estimation and novel-view synthesis as atomic primitives — camera pose provides geometry-grounded textual context, synthesized views supply visual imagination — improving MMSI-Bench Overall from 27.14 baseline to 30.15 (text geometric) and 34.17 (visual geometric); Positional Relationship moves 19.63 → 30.25 → 33.95 → 35.80 [§Spatial understanding table].
  • Headline T2I numbers: GenEval2 54.12 vs Z-Image 41.83, Qwen-Image 30.67, Flux 34.59; DPG 88.55; LongText-EN 97.5 / LongText-CN 96.8; in-house benchmark 63.87 [§Text-to-Image table].
  • Headline image-editing numbers on GEdit-Bench EN (Full set): G_SC 8.42, G_PQ 7.85, G_O 7.75 — beating Qwen-Image-Edit (7.54), Z-Image-Edit (7.57), Emu-3.5 (7.59), Step1X-Edit v1.2 (7.24), Flux-Kontext-dev (6.51) [§Image editing table].
  • Headline T2V numbers on VBench 1.0: Total 85.07, Quality 84.29, Semantic 83.11 — competitive with Wan2.1 (Total 85.59) and ahead of Hunyuan Video (83.69) on Total, with the largest gap on Semantic Score (Wan2.1 76.11, Hunyuan 76.88, Omni 83.11) [§Text-to-video table].
  • Video editing: under the FiVE benchmark, Omni reaches Accuracy 78.03 vs prior best Wan-Edit 46.97 — a +31-point gap on the integrated structure/background/text-alignment/motion score, driven by motion-tracking (YN 69.67 vs 41.41) [§Video editing table].
  • Camera pose estimation on RealEstate10K: Omni reaches AUC@30 88.32 vs VGGT 88.23, RPE_trans 0.3766 vs 0.3886, RPE_rot 0.0289 vs 0.0386 — beating a dedicated estimator at every metric on this dataset [§Camera pose table].
  • Monocular depth: SOTA on KITTI (δ₁ 96.92%, AbsRel 0.0621), SINTEL (δ₁ 74.27, AbsRel 0.3340), ETH3D (δ₁ 98.91, AbsRel 0.0312), DIODE (δ₁ 83.83, AbsRel 0.2034) — beating VGGT and DA3 giant across most metrics [§Monocular depth table].
  • Multimodal understanding numbers are competitive but not uniformly best vs same-scale MoEs at 3B activated: Omni 91.5 vs Qwen3-VL-30B-A3B 85.0 on AI2D; Omni 70.1 vs Qwen3-VL 61.5 on HallusionBench; Omni 76.4 vs Qwen3-VL 67.5 on VlmsAreBlind; but Omni 63.8 vs Qwen3-VL 78.4 on MMStar and Omni 67.2 vs Qwen3-VL 74.5 on Video-MME [§Understanding benchmark table].
  • Camera-pose conditioning uses an in-band tag format: <campose>tx ty tz rx ry rz</campose> <fov>fovh</fov><fov>fovw</fov> — six values for the next-view translation/rotation relative to the current view, plus FOV [§campose tag].

Method

Section titled “Method”

Omni is a standard mixture-of-experts architecture with 3B activated parameters, trained natively over text, images, videos, 3D geometry, and (per the page) “hidden representations.” The training objective is not specified in the project-page material released so far. Two inference-time mechanisms are central:

  1. Text thinking — CoT-style textual rollout that enriches the latent workspace with finer semantic decompositions before producing the final answer. Applied to understanding and as one component of generation conditioning.
  2. Visual context unrolling — the model produces intermediate visual representations (depth maps, novel-view synthesis under specified camera poses, structural captions) and writes them back into context before the final generation/answer step. Cast as composable atomic primitives — camera pose estimation, novel-view synthesis, depth — that the model invokes on itself.

The combination (“text thinking + visual unrolling”) is reported as the most consistent regime across T2I, depth, and spatial reasoning. Camera-pose conditioning uses the <campose> tag with six translation/rotation values plus FOV, structurally analogous to how prior video-DiT controls expressed camera as in-band tokens. No training-recipe details, parameter counts beyond the 3B activated figure, code, or weights are disclosed on the project page.

Results

Section titled “Results”
  • T2I: GenEval2 54.12 (vs Z-Image 41.83, Qwen-Image 30.67, Flux 34.59); DPG 88.55; LongText-EN 97.5; in-house 63.87.
  • Image edit: G_O 7.75 on GEdit-Bench EN, beating Qwen-Image-Edit / Z-Image-Edit / Emu-3.5.
  • T2V: VBench 1.0 Total 85.07, with Semantic 83.11 — the Semantic gap (+7 over Wan2.1, +6 over Hunyuan) is the standout result.
  • Video edit: FiVE Accuracy 78.03 vs Wan-Edit 46.97 (+31).
  • Geometry: RealEstate10K camera pose AUC@30 88.32 beats VGGT 88.23; KITTI depth δ₁ 96.92 beats VGGT 94.29 and DA3 giant 93.96.
  • Understanding: leads on AI2D (91.5), HallusionBench (70.1), VlmsAreBlind (76.4), RealWorldQA (76.0); trails on MMStar, MVBench, Video-MME.
  • Compositional T2I (TIFAGM): +long-text & visual unrolling moves baseline 29.25 → 53.44, with the oracle ceiling at 57.21 — showing most of the gap is closeable with the right context.
  • Spatial reasoning (MMSI-Bench): Positional Relationship 19.63 → 35.80 under visual geometric context — nearly 2× the baseline.

No training-cost, inference-cost, parameter, or open-weight information is disclosed on the project page.

Why it’s interesting

Section titled “Why it’s interesting”

This is the cleanest filed example of Context Unrolling as a unifying interface across (i) the unified-multimodal stack and (ii) modality-as-CoT reasoning — it sits squarely at the intersection of the Unified Multimodal Models and Thinking with Modalities concepts. Two specific connections to filed work:

  • Relative to Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation (UniMRG), which uses training-time auxiliary generation of depth/segmentation/pixels to improve understanding, Omni applies the same intuition at inference time — emit depth, novel views, camera poses into context, then condition the answer on them. UniMRG’s open question (“does generation supervision generalize beyond depth + segmentation?”) is partly answered by Omni’s larger set of primitives (camera pose, novel-view synthesis, structural captions), though the project page does not isolate which signals do the work.
  • Relative to DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models and Demystifying Video Reasoning, Omni offers a third substrate position for modality-CoT: not a diffusion trajectory exploring multiple candidates in latent space, but an autoregressive-feeling loop that generates and re-ingests intermediate visual artifacts as context tokens. Closer in spirit to Google’s external-tool Agentic Vision (Introducing Agentic Vision in Gemini 3 Flash) but with no external Python sandbox — the “tools” are the model’s own atomic generation primitives.

Also worth noting: a 3B-activated MoE that competes with VGGT on RealEstate10K camera pose and beats DA3 giant on KITTI depth is a strong datapoint for the World Foundation Models “WFM as backbone” thesis — the unified model is not just generating, it is also exposing geometry/pose as queryable primitives.

See also

Section titled “See also”