BindWeave: Subject-Consistent Video Generation via Cross-Modal Integration

TL;DR

BindWeave (ByteDance × USTC, ICLR 2026) is a subject-to-video generation framework that replaces the shallow “encode-text + encode-image + cross-attention” fusion paradigm with a Qwen2.5-VL-7B MLLM that jointly parses a text prompt and 1–4 reference images into a sequence of hidden states, then conditions a Wan 2.1 14B DiT on those hidden states alongside CLIP semantic features and VAE appearance features. On OpenS2V-Eval (180 prompts, 7 categories), it sets a new state of the art at 57.61% Total Score — beating VACE-14B (57.55), Phantom-14B (56.77), Kling 1.6 (56.23), and the rest of the open-source + commercial field — with the gap driven by NexusScore (subject consistency, 46.84% vs 45.89 best baseline) and FaceSim (53.71% vs 55.09 leader). The released weights are now public on the project page.

Key claims

The “separate-then-fuse shallow paradigm” (independent text/image encoders + post-hoc concatenation or cross-attention) breaks down on prompts that specify complex spatial relationships, temporal logic, and multi-subject interactions — manifesting as identity confusion, action misplacement, and attribute blending [§1].
Replacing shallow fusion with an MLLM (Qwen2.5-VL-7B) that processes an interleaved sequence [prompt, <img>_1, <img>_2, ...] produces subject-aware hidden states that bind textual commands to specific visual entities before generation begins [§3.3, Eqns. 5–8].
The MLLM-derived condition is concatenated with a separately T5-encoded text embedding to form the final cross-attention key/value — both signals are needed; T5 supplies linguistic grounding that stabilizes optimization while the MLLM supplies relation-aware cues [§3.3 Eqn. 10, §4.4].
An MLLM-only setup (Qwen2.5-VL + DiT, no T5) was unstable and failed to converge within the training budget [§4.4].
Generation conditions on three streams in parallel: relational MLLM hidden states (high-level reasoning), CLIP features (semantic identity), and VAE features of references (low-level appearance) — channel-concatenated onto K padded temporal slots of the noisy latent rather than treated as actual video frames [§3.4 Eqn. 11].
Training is a two-stage curriculum on a 1M-pair subset distilled from OpenS2V-5M: 1k iterations on a small high-quality “core” subset for S2V adaptation, then 5k iterations on the full 1M dataset — 512 xPUs, global batch 512, constant LR 5e-6, AdamW [§3.5].
On OpenS2V-Eval Total Score, BindWeave reaches 57.61% — new SOTA, narrowly beating VACE-14B (57.55%) and clearly ahead of Phantom-14B (56.77%), Kling 1.6 (56.23%), SkyReels-A2-P14B (52.25%), Vidu 2.0 (51.95%), Pika 2.1 (51.88%) [Table 1].
NexusScore (the OpenS2V detect-then-compare subject-consistency metric, designed to be more perceptually aligned than global-frame CLIP/DINO) is 46.84% — the highest in the table, ahead of Kling’s 45.89% and Pika’s 45.40% [Table 1].
T5+Qwen2.5-VL beats T5-only across every reported axis: Total 57.61 vs 55.16, Aesthetics 45.55 vs 42.80, MotionAmplitude 13.91 vs 7.48, GmeScore 67.79 vs 62.26, NexusScore 46.84 vs 45.79 [Table 2].
Inference: 1–4 reference images + text prompt, 50 rectified-flow steps with CFG scale 5, prompt rephraser borrowed from Phantom to ensure the text matches the reference images [§3.5].
Qualitative comparisons single out commercial Vidu/Pika for “commonsense violations” (twisted limbs, objects passing through containers), Kling for strong aesthetics but poor subject consistency, and VACE/Phantom for weak subject consistency despite competitive numbers — the cross-modal MLLM reasoning is credited for preserving small prompt details like “hot oil” in multi-object scenes [§4.3, Fig. 5].

Method

BindWeave is fine-tuned from Wan 2.1 (14B DiT, spatio-temporal VAE, T5 encoder, rectified flow). The pipeline has two halves: an MLLM-driven Intelligent Instruction Planning stage and a Collectively Conditioned Diffusion stage.

Planning. A unified sequence is built by appending one image placeholder for each reference image after the text prompt, then fed to Qwen2.5-VL-7B alongside the corresponding image list. The MLLM emits a sequence of hidden states $h$ that encode entities, attributes, and inter-subject relations. A trainable two-layer MLP + GELU connector projects $h$ into the DiT conditioning space to yield $c_{mllm}$ . In parallel, the original prompt is encoded independently by T5 to yield $c_{t5}$ . The two are concatenated to form the final relational condition $c = [c_{mllm}; c_{t5}]$ .

Generation. The DiT conditions on $c$ at three points. First, low-level VAE features of the reference images are channel-concatenated onto K padded temporal slots of the noisy latent (zeros elsewhere), with a binary mask along the channel dimension to emphasize subject regions — this preserves the temporal integrity of the actual video while injecting reference appearance. Second, the cross-attention layer in each DiT block computes attention over both $c$ (relational/textual) and CLIP image features (semantic identity), summing the two attention outputs as in Equation 12. Training uses the standard Flow Matching MSE loss on the velocity field.

Data. The 1M-pair training corpus is filtered from the 5M OpenS2V-5M release, the same dataset behind Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset — i.e. BindWeave and Phantom-Data are the architecture and data contributions on the same underlying cross-pair S2V curation pipeline. The released model is distributed via the project page (linked in the original Slack pointer) and the bytedance/BindWeave repo, which requires the Wan 2.1 14B weights as a backbone dependency.

Results

OpenS2V-Eval Total Score, ordered by ranking [Table 1]:

Method	Total	Aesthetics	MotionSmooth	MotionAmp	FaceSim	GmeScore	NexusScore	NaturalScore
BindWeave (ours)	57.61	45.55	95.90	13.91	53.71	67.79	46.84	66.85
VACE-14B	57.55	47.21	94.97	15.02	55.09	67.27	44.08	67.04
Phantom-14B	56.77	46.39	96.31	33.42	51.46	70.65	37.43	69.35
Kling 1.6	56.23	44.59	86.93	41.60	40.10	66.20	45.89	74.59
MAGREF-480P	52.51	45.02	93.17	21.81	30.83	70.47	43.04	66.90
SkyReels-A2-P14B	52.25	39.41	87.93	25.60	45.95	64.54	43.75	60.32
Vidu 2.0	51.95	41.48	90.45	13.52	35.11	67.57	43.37	65.88
Pika 2.1	51.88	46.88	87.06	24.71	30.38	69.19	45.40	63.32

The headline number — Total 57.61 vs VACE-14B 57.55 — is a 0.06-point lead, basically a tie at the top. The interesting result is the composition: VACE wins Aesthetics and FaceSim, but BindWeave’s NexusScore is +2.76 over the best baseline (Kling). NexusScore is the metric specifically designed for OpenS2V to suppress background and pose confounds (detect → crop → multimodal-feature compare with text-image gating), so a NexusScore lead under a Total-Score tie is interpretable as “the relational MLLM helps the model actually attend to the right subjects rather than the right scene.” MotionAmplitude is low (13.91) compared to Phantom (33.42) and Kling (41.60) — the model produces less motion magnitude, which is a known trade-off for identity preservation but worth flagging. The ablation (T5 vs T5+MLLM, Table 2) is the cleanest controlled comparison and lifts every metric simultaneously — particularly +6.43 on MotionAmplitude and +5.53 on GmeScore.

Why it’s interesting

BindWeave is the architecture-side counterpart to Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset: both train on the OpenS2V-5M corpus and both target the OpenS2V-Eval leaderboard, but Phantom-Data’s contribution is the 1M cross-pair dataset construction while BindWeave’s contribution is the MLLM-as-instruction-parser front end. Comparing the two on the same Wan 2.1 backbone settles a useful question: is the subject-consistency × prompt-following Pareto frontier in S2V moved more by data (Phantom-Data: 1.3B Phantom-wan beats baselines on GPT-4o subject scoring at small scale) or by architecture (BindWeave: 14B model with MLLM front end beats Phantom-14B by +0.84 Total Score on OpenS2V-Eval)? The honest read from these two papers is “both, and they compose” — the MLLM front end is a real lift on top of the cross-pair data, with the explicit T5-only vs T5+Qwen2.5-VL ablation isolating the MLLM contribution. The recipe is also a concrete instance of the broader “MLLM as DiT front end” pattern tracked under Unified Multimodal Models (DuoGen’s MLLM-first / DiT-second, OmniWeaving’s three-stage Decoupled→E2E), with the wrinkle that BindWeave is not a unified model — it’s a one-way pipe from a frozen MLLM into a generation-only DiT, more like a heavyweight prompt encoder than a true UMM. Worth contrasting also with OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning, which co-trains an MLLM with an MMDiT for general video tasks: BindWeave is the narrow, S2V-specific, “frozen MLLM + trained connector” reduction of that recipe.