Bernini: Latent Semantic Planning for Video Diffusion

TL;DR

Bernini (ByteDance) is a unified framework for video generation and editing that splits work between an MLLM-based semantic planner and a DiT-based renderer. The planner predicts a target semantic representation directly in the ViT embedding space, and the renderer synthesizes pixels conditioned on this plan plus text features and (for editing) source VAE features for detail preservation. Because the semantic embedding is the interface, planner and renderer can be trained largely separately with light co-training, preserving both pretrained backbones. It also introduces Segment-Aware 3D RoPE (SA-3D RoPE) to disambiguate tokens from different visual segments (multiple references, source, target) and adds chain-of-thought reasoning to the planner.

Key claims

A clean division of labor — MLLM does semantic planning, DiT renders pixels from that plan — is sufficient to unify video generation and editing without joint end-to-end training of both modules from scratch [Abstract, §Architecture overview].
Predicting the target ViT-embedding (not text, not tokens) as the planning interface lets the planner and renderer train mostly independently and only need light co-training, preserving the pretrained MLLM and DiT [Abstract].
For editing tasks, additionally conditioning the renderer on the source video’s VAE features preserves low-level detail that pure semantic planning would lose [Abstract].
Segment-Aware 3D RoPE (SA-3D RoPE) distinguishes tokens from different visual segments (text, source, multiple references, target placeholders) and is needed for handling multiple visual inputs cleanly [Abstract, project page Method section].
Chain-of-thought reasoning in the planner improves the transfer from understanding to generation versus a one-shot planner [Abstract].
Bernini reports state-of-the-art across “a wide range of video generation and editing benchmarks” with strong generalization on challenging editing tasks [Abstract]. (Specific benchmark numbers are not disclosed in the public abstract or project page; the project page shows qualitative results across prompt-driven editing, reference-image guidance, content insertion, and up-to-5 reference R2V.)

Method

Bernini has two modules connected by a semantic interface:

MLLM-based semantic planner. Reasons over text + source images + source videos + target placeholders. Instead of emitting text or discrete tokens for the generator, it predicts a target representation in the ViT embedding space. CoT reasoning is incorporated so the planner can “think before predicting.” Multiple visual inputs (source, references, target) are disambiguated via SA-3D RoPE, a positional encoding that adds a segment index alongside the standard 3D (temporal × height × width) positions so the model can tell which segment each token belongs to.
DiT-based renderer. Performs flow-matching denoising over VAE latent tokens. Conditioned on (a) the planner’s predicted ViT embedding (semantic plan), (b) text features, and — for editing — (c) source VAE features for detail preservation.

Training is decoupled: planner and renderer trained mostly separately, only lightly co-trained, preserving the pretrained MLLM and pretrained DiT backbones. The project page advertises the following editing/generation surfaces: prompt-driven video editing, reference-image-guided editing (object / material / weather / style), content insertion (image or video reference inserted into a video), and reference-to-video (R2V) supporting up to five reference images.

Results

The arxiv abstract claims state-of-the-art across a wide range of video generation and editing benchmarks; the public project page is qualitative (hover-to-play video grids). Specific benchmark scores are not disclosed in the surfaces fetched at filing time — they presumably live in the PDF body, which was not pulled in this ingest.

Why it’s interesting

Bernini is a strong data point in the MLLM-as-spatial-planner lineage and the closest video-domain counterpart to Exploring MLLM-Diffusion Information Transfer with MetaCanvas (MetaCanvas): both move the MLLM from “global text encoder” to “structured spatial/semantic planner” whose output is consumed by a frozen-ish DiT renderer through a small connector. MetaCanvas appends learnable canvas tokens to the MLLM input and reads them back as a spatial prior; Bernini goes one step further and predicts the entire target ViT embedding as the planning interface, with SA-3D RoPE to handle multi-segment visual inputs that MetaCanvas’s 2D/3D canvas tokens don’t natively address.

It also fits cleanly into the Decoupled (training-stage) corner of the Unified Multimodal Models taxonomy formalized by UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? (E2E / Decoupled / Agentic): planner and renderer are trained separately and only lightly co-trained, which preserves the pretrained backbones — the same recipe DuoGen: Towards General Purpose Interleaved Multimodal Generation argued for in interleaved image-text and that OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning partially adopts (Stages 1–2 frozen MLLM, Stage 3 unfrozen). And it sits alongside InstructX: Towards Unified Visual Editing with MLLM Guidance (Qwen2.5-VL + Wan2.1-14B via MLP + meta-queries) and UniVideo: Unified Understanding, Generation, and Editing for Videos (Qwen2.5-VL + MMDiT) as a ByteDance internal answer to the same question, with a more aggressive choice of interface (full ViT embedding rather than meta-queries or text conditioning).