Context Unrolling in Omni Models

TL;DR

The arxiv version of ByteDance Seed’s Omni — a unified multimodal MoE with 3B activated parameters trained natively across text, images, videos, 3D geometry, and hidden representations. The central mechanism is Context Unrolling: at inference, the model invokes atomic primitives (text-think, visual-token rollout, camera-pose estimation, novel-view synthesis, depth estimation) and writes their outputs back into its own context before decoding the final prediction. Reports SOTA on GenEval2 (54.12), GEdit-Bench EN (7.75), and competitive video generation, while matching or beating VGGT on RealEstate10K camera pose and DA3 giant on KITTI depth. The project-page version of this same work is filed at Omni: Context Unrolling in Omni Models (ByteDance Seed).

Key claims

A single MoE model with 3B activated parameters covers understanding, T2I, image editing, T2V, video editing, camera-pose estimation, novel-view synthesis, and monocular depth — packaging what were previously separate expert models into one architecture [§1, §2].
Context Unrolling reframes inference as iterative context construction followed by context-conditioned decoding: y = argmax p(y | Compose(x, φ_1(x), φ_2(x), ...)) where each φ_i is an atomic primitive (describe, predict pose, roll out visual tokens, synthesize novel view, estimate depth) [§2, Eq. 1].
Text-only CoT (“thinking”) on the understanding side gives small but consistent gains: MMStar 59.4 → 66.5, Erqa 41.5 → 44.5, AI2D 90.2 → 92.3, HallusionBench 69.6 → 71.3 [Table 1].
On compositional T2I (TIFAGM), visual context unrolling beats text unrolling, and combining them is strictly best: baseline 29.25 → +long-text 43.94 → +visual 48.02 → +long-text & visual 53.44 → +oracle-text-and-visual 57.21 [Table 2].
Textual and visual contexts are non-redundant: even when Gemini-3 Pro provides oracle textual rewrites, adding self-rollout visual tokens further improves results (52.20 → 57.21 on TIFAGM), indicating the two channels supply complementary constraints [Table 2, §2.2].
Spatial reasoning (MMSI-Bench, 200-question 3D subset): geometry-grounded textual context (predicted camera poses) improves Overall 27.14 → 30.15; generative “imagination” (novel-view synthesis) improves to 34.17, with Positional Relationship 19.63 → 35.80 [Table 3, §2.3].
Monocular depth: task-relevant textual context matters — a depth-focused caption improves δ₁ 83.21% → 83.88%, while a generic detailed caption gives no gain, showing that context-unrolling only pays when the intermediate is constraint-like rather than merely verbose [Table 4, §2.4].
Camera pose estimation on RealEstate10K: Omni AUC@30 88.32 vs VGGT 88.23, RPE_trans 0.3766 vs 0.3886, RPE_rot 0.0289 vs 0.0386 — a 3B-activated UMM matches or beats a dedicated feed-forward regressor at every metric on this dataset [Table 9].
Image generation headline: GenEval2 54.12 vs Z-Image 41.83, Qwen-Image 30.67, Flux 34.59; DPG 88.55; LongText-EN 97.5 / LongText-CN 96.8 [Table 6].
Image editing headline (GEdit-Bench EN Full): G_O 7.75 beats Qwen-Image-Edit 7.54, Z-Image-Edit 7.57, Emu-3.5 7.59, Step1X-Edit v1.2 7.24, Flux-Kontext-dev 6.51 [Table 6, editing sub-table].
Video editing (FiVE benchmark): Omni Accuracy 72.41 vs prior best Wan-Edit 46.97 — a +25-point gap driven mainly by motion-tracking (YN 62.83 vs 41.41), though structural preservation is lower than the specialized Wan-Edit baseline [Table 8].
Multimodal understanding is competitive but not uniformly best vs same-scale 30B-A3B MoEs: Omni leads on AI2D (91.5 vs 85.0 Qwen3-VL, 86.8 InternVL3.5), HallusionBench (70.1 vs 61.5 / 53.8), RealWorldQA (76.0 vs 73.7 / 72.3), but trails on MMStar (63.8 vs 78.4) and Video-MME (67.2 vs 74.5) [Table 5].

Method

Omni is built on the BAGEL design philosophy, expanding the training modality set from image-text pairs to text, images, videos, 3D geometry, and hidden visual representations. It uses a standard MoE architecture with 3B activated parameters. The paper positions Context Unrolling as an emergent capability of modality-scaled pretraining — not a special training objective, but a behavior that manifests when the model has been asked to jointly predict enough heterogeneous outputs that intermediate ones become invocable as reasoning primitives.

At inference the model can (i) emit a text-think trace before generating (short ~100 tokens, long ~250 tokens), (ii) roll out visual tokens carrying structural information (objectness, layout, coarse geometry) and re-condition on them, (iii) predict camera pose in-band via a <campose>tx ty tz rx ry rz</campose> <fov>...</fov> tag, (iv) synthesize novel views for “geometric imagination,” and (v) generate depth maps as an intermediate representation for downstream 3D reasoning. Training details (dataset composition, per-modality loss weighting, hidden-representation supervision recipe) are not disclosed beyond the modality list and the 3B-activated MoE topology.

Results

T2I: GenEval2 54.12 (best filed for a 3B-activated UMM), DPG 88.55, LongText-EN 97.5, in-house 63.87 — beats Z-Image, Qwen-Image, Flux across the board [Table 6].
Image editing: G_O 7.75 on GEdit-Bench EN Full — narrow but consistent lead over Qwen-Image-Edit, Z-Image-Edit, Emu-3.5 [Table 6].
T2V: reports competitive VBench numbers at the project-page level; the paper focuses on the video-editing FiVE result where Omni reaches Accuracy 72.41 vs Wan-Edit 46.97 [Table 8].
3D geometry: matches VGGT on RealEstate10K camera pose (AUC@30 88.32 vs 88.23), competitive on CO3Dv2 (AUC@30 75.21 vs 86.23); depth estimation strong on NYU (δ₁ 92.75 → competitive across published benchmarks) [Tables 9-10].
Understanding: mixed vs same-scale 30B-A3B MoEs — leads on AI2D, HallusionBench, VlmsAreBlind, RealWorldQA; trails on MMStar, MVBench, Video-MME [Table 5].
Context-unrolling ablations (Table 2 for T2I, Table 3 for MMSI, Table 4 for depth) collectively support the paper’s thesis that unification pays via context construction rather than parameter sharing.

Why it’s interesting

This is the arxiv counterpart to Omni: Context Unrolling in Omni Models (ByteDance Seed), which was previously filed under the project page. Same headline numbers, same “Context Unrolling” framing; the arxiv adds the formal problem statement (Eq. 1) and the ablation tables in inline form. Two cross-links worth carrying forward:

Against Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation (UniMRG), which uses training-time auxiliary generation of depth/segmentation to improve understanding, Omni is the inference-time analogue: emit depth, novel views, and camera poses into context, then condition the answer on them. The two together suggest the G2U effect is real but is mediated by whichever leg (training or inference) is given the intrinsic-representation targets to work with.
Against DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models (diffusion-native modality-CoT) and Introducing Agentic Vision in Gemini 3 Flash (external Python-tool modality-CoT), Omni’s context-unrolling is a fourth substrate for Thinking with Modalities: an autoregressive-feeling loop that generates and re-ingests intermediate visual artifacts as context tokens, with no external sandbox and no diffusion trajectory. It’s the “internalized tool” position on the substrate axis the concept page tracks.