InstructX: Towards Unified Visual Editing with MLLM Guidance

TL;DR

InstructX (ByteDance) is a unified image+video instruction editing framework built by bolting Qwen2.5-VL-3B onto Wan2.1-14B via a two-layer MLP connector and learnable meta-queries, with the explicit design hypothesis that editing should happen inside the MLLM, not in the connector. The paper’s main empirical contribution is a comparative ablation across four integration architectures — direct T5, QFormer-compressed MLLM hidden states, MetaQuery-style large connector with frozen MLLM, and InstructX’s LoRA-tuned MLLM + small MLP — showing that LoRA-tuning the MLLM accelerates feature alignment and improves final editing quality. Separately, mixed image+video training transfers editing capabilities from data-rich image tasks to data-scarce video tasks zero-shot: style transfer and segmentation are absent from video training data but emerge in the video editing model. Modality-specific learnable queries (256 for images, 512 for video) keep the two streams distinct in a single weight set. Beats every reported open-source baseline on ImgEdit-Bench overall, competitive with Runway Aleph on the authors’ new VIE-Bench (140 instances × 8 categories).

Key claims

The central design hypothesis: MLLMs should not be frozen feature extractors in editing pipelines — editing should be primarily realized within the MLLM via LoRA fine-tuning, not delegated to a large downstream connector [§1, §3.2].
LoRA-tuning the MLLM converges significantly faster than freezing it during the feature-alignment stage, even when the connector or QFormer absorbs all training capacity [§3.2 Fig. 4 left].
After completing alignment and unfreezing the DiT, the LoRA-MLLM + small-MLP design also beats the MetaQuery (frozen MLLM + large connector) and QFormer (compressed hidden states) designs on ImgEdit-Bench [§3.2 Fig. 4 right].
Mixed image+video training transfers editing capabilities zero-shot: video style transfer and video segmentation emerge without explicit video supervision, because the image training data covers those tasks while the video data does not [§3.3, Fig. 6].
Modality-specific learnable queries (256 for image inputs, 512 for video) outperform shared queries on VIE-Bench by better separating modality-specific feature extraction [§4.4 Fig. 9a].
The MLLM must see multiple frames (the paper samples 13) to produce reliable editing guidance for video; single-frame conditioning collapses on edits where the target content appears mid-video [§4.4 Fig. 9b].
VAE encoding of the source image/video is added to the noisy latent in Stage 2 to maintain consistency with the source, separate from the MLLM conditioning path [§3.1, §3.3].
For reference-image tasks, the reference VAE features are concatenated to the noisy latent along the sequence dimension rather than via an additional cross-attention head [§3.1].
Three-stage training is needed: (1) freeze DiT, train query + MLLM LoRA + MLP for alignment; (2) full training on mixed image+video data with VAE concat for consistency; (3) quality fine-tuning on a small curated high-quality subset to remove “oily/plastic” textures from low-quality data [§3.3, Fig. 5].
On ImgEdit-Bench overall, InstructX scores 3.85 vs Step1X-Edit 3.39, BAGEL 3.52, UniWorld 3.13, OmniGen 2.64, InstructPix2Pix 2.01 — the best open-source score reported [Table 2].
On GEdit-Bench, InstructX scores Q_SC 7.47 / Q_PQ 7.22 / Q_O 6.68; ahead of Step1X-Edit (7.05 / 7.21 / 6.79) on instruction-following but behind closed-source GPT-4o (7.98 / 7.73 / 7.83) and SeedEdit 3.0 [Table 1].
VIE-Bench is introduced: 140 high-quality instruction-based video editing instances across 8 categories, with a GPT-4o judge over four axes (accuracy, preservation, quality, subject similarity for reference-based) on a 1-10 scale [§4.2, Appendix A.1].
On VIE-Bench, InstructX beats every open-source video editor and is competitive with Runway Aleph on most tasks — leads on Style/Tone Change (9.196 vs 9.133), Hybrid Edit (8.907 vs 8.510), and Reference-Based Swap (8.955 vs 8.301 Pika / 8.592 Kling); slightly behind Runway Aleph on Add (8.349 vs 8.447), Swap/Change (9.072 vs 9.161), and Remove (8.322 vs 8.504) [Table 3].
Training data leverages large-scale image editing corpora (NHR-Edit, X2Edit, GPT-Image-Edit) plus an internally-developed video-editing data synthesis pipeline; image sampling probability is set to 0.7 in mixed training, video 0.3 [§3.3, §4.1].

Method

InstructX is structurally a Wan2.1-14B video DiT conditioned by Qwen2.5-VL-3B through a learnable-query + LoRA + two-layer MLP bridge. Recall the integration design space: (a) T5 text encoder direct to DiT (no MLLM); (b) MLLM last-hidden-states → QFormer → fixed 256-token sequence → DiT; (c) MetaQuery-style — frozen MLLM, learnable queries appended to the input sequence, large transformer connector (~1.6B params); (d) InstructX — same learnable queries as MetaQuery but with MLLM LoRA fine-tuning + a small two-layer MLP connector. The InstructX bet is that variant (d) wins because the MLLM is doing the editing work, not the connector.

For image input the MLLM emits 256 meta-query features; for video input it emits 512 features and the MLLM sees 13 sampled frames. The meta-query features replace the text embeddings the DiT normally cross-attends to. To maintain pixel-level consistency with the source, the source-image/video VAE encoding is added to the noisy latent (introduced in Stage 2, not Stage 1, because Stage 1 is just establishing the MLLM↔DiT feature-space alignment). For reference-image conditioning the reference’s VAE features are concatenated to the noisy latent along the sequence dimension.

Training runs in three stages, each with global batch 256 image / 32 video at LR 1e-4: Stage 1 — DiT frozen, train query + MLLM LoRA + MLP on image instruction data only, 10K steps, to align feature spaces; Stage 2 — unfreeze DiT, train everything on mixed image+video data (0.7 image / 0.3 video sampling), 10K steps, with VAE-source concat for consistency; Stage 3 — quality fine-tuning on a small curated high-quality subset, 5K steps, to remove low-quality-data textures. All stages use flow matching as the objective.

The cross-modal transfer mechanism is the part that matters most. Image editing data covers tasks (segmentation, style transfer, complex local edits) that the in-house video editing pipeline cannot produce reliably; once both modalities are jointly trained in Stage 2 with modality-specific queries doing the separation, the video output head inherits the image data’s task coverage. This is the same observation that motivates EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning‘s “image data transfers to video” claim, but reached through MLLM-mediated conditioning rather than full-sequence in-context learning.

Results

ImgEdit-Bench (8 sub-tasks, GPT-4.1 judge, 1-5 scale): InstructX overall 3.85, ahead of all reported open-source baselines — Step1X-Edit 3.39, BAGEL 3.52, UniWorld 3.13, OmniGen 2.64, InstructPix2Pix 2.01 [Table 2]. Wins on Remove (3.92), Replace (4.03), Style (4.45), Action (4.24); behind closed-source GPT-4o (4.51) and SeedEdit 3.0 (4.06).
GEdit-Bench (Qwen2.5-VL-72B judge): Q_SC 7.47 / Q_PQ 7.22 / Q_O 6.68. Beats every open-source baseline on Q_SC including BAGEL (7.43) and Step1X-Edit (7.05); closed-source GPT-4o leads at 7.98 / 7.73 / 7.83 [Table 1].
VIE-Bench (140 instances × 8 categories, GPT-4o judge, 1-10 scale). Add — InstructX 8.349 vs Runway 8.447, Kling 6.602, Omni-Video 6.242, InsV2V 4.281, VACE 4.854. Swap/Change — InstructX 9.072 vs Runway 9.161, Kling 8.800, Omni-Video 4.748. Remove — InstructX 8.322 vs Runway 8.504, Kling 8.253, MiniMax-Remover 6.839, DiffuEraser 6.243. Style/Tone — InstructX 9.196 vs Runway 9.133 (leads). Hybrid Edit — InstructX 8.907 vs Runway 8.510 (leads). Reference-Based Swap — InstructX 8.955 vs Kling 8.592 / Pika 8.301 / VACE 7.987 (leads); Reference-Based Add — InstructX 9.157 vs Kling 9.321 / Pika 8.377 / VACE 3.973 [Table 3].
Ablation — separate vs shared queries: Separate (256 image / 512 video) wins on VIE-Bench, attributed to better feature separation per modality [§4.4 Fig. 9a].
Ablation — single-frame vs multi-frame MLLM input: Single-frame conditioning produces visible collapse on edits where the target content appears mid-video; 13-frame sampling is the chosen default [§4.4 Fig. 9b].

Why it’s interesting

InstructX and EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning (EditVerse, Adobe Research) were both filed in roughly the same window and target the same problem — unified image+video instruction editing — with completely opposite architectural bets. EditVerse collapses to a single dense transformer with no MLLM and uses interleaved token sequences + 4D RoPE + full self-attention. InstructX keeps a separate MLLM + DiT and bridges them with a small MLP + LoRA-tuned MLLM. Both converge on the same data observation — image training transfers to video editing — but arrive at it via different mechanisms (architectural in-context learning vs MLLM-mediated conditioning). The natural follow-up is a same-data A/B between the two recipes to isolate whether the MLLM’s reasoning capability is doing real work or whether full-sequence self-attention captures the same effect.

Within the Unified Multimodal Models taxonomy InstructX is a Decoupled design (MLLM and generator separable, stitched at inference with a learned connector) — direct contrast with EditVerse’s E2E design. The page’s open question “is the MLLM+DiT decoupled architecture strictly better than fully joint?” gets another data point here: InstructX explicitly argues that the MLLM matters and should be trained, where MetaQuery argued the opposite (freeze MLLM, scale the connector to 1.6B). The InstructX ablation is the cleanest filed direct comparison of those two recipes on the same backbone.

The most reusable artifact is probably VIE-Bench: 140 instances × 8 categories × four-axis GPT-4o judge is in the same shape as EditVerse’s EditVerseBench (200 instances × 20 categories) and provides a second independent benchmark with overlapping coverage — useful for cross-checking VLM-judge ranking stability since neither benchmark is yet established as a standard. The architectural finding (LoRA-tuned MLLM > frozen MLLM + large connector) is also a direct counter-claim to MetaQuery’s reported result and worth verifying on third-party setups.