VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation

TL;DR

VEGGIE (UNC × Adobe × Michigan) is an end-to-end framework that unifies instructional video editing — addition, removal, change, stylization — with video object grounding and reasoning segmentation in a single model trained with diffusion loss only. The trick is replacing the discrete-token MLLM-to-diffusion handoff (which blocks gradient flow) with per-frame continuous learnable task query embeddings: LLaVA-OneVision processes the input video and instruction, emits one query embedding per frame, and a MagicBrush-initialized SD1.5 UNet (inflated with temporal attention) renders those queries at pixel space. A two-stage curriculum first aligns the MLLM and diffusion model on 3.4M image-editing pairs, then fine-tunes end-to-end on 134K multitask video samples — including a novel image-to-video animation pipeline that lifts paired image-edit data into the video domain. Introduces VEG-Bench (132 video-instruction pairs across 8 editing skills) and demonstrates that grounding and editing tasks mutually reinforce each other in multitask training.

Key claims

The pipeline is end-to-end with only diffusion loss — no language loss, no mask loss, no intermediate text/caption/layout/mask guidance — by using continuous learnable task query embeddings per frame as the MLLM↔DiT bridge instead of discrete tokens [§1, §3.1].
Discrete text tokens used by prior work (e.g., InstructDiffusion-style or VPLM) disconnect the pipeline and block gradient propagation; continuous per-frame task queries keep the whole stack differentiable [§3.1].
Curriculum learning is necessary: directly training the whole pipeline on video data causes model collapse; Stage 1 aligns MLLM and diffusion on 3.4M image-edit pairs (frozen MLLM, train alignment MLP + task queries + 2D UNet), Stage 2 fine-tunes end-to-end on 134K video samples with MLLM LoRA + alignment + queries + inflated 3D UNet [§3.2].
A novel image-to-video data synthesis pipeline transforms paired image-edit data into instructional video-edit data: offline MLLM generates a motion prompt from the source image, an I2V model animates both source and edited images into videos using a first-frame-conditioned video editing model as a strong prior, and VBench-style metrics filter for motion smoothness / image quality / background consistency [§3.3, Fig. 2].
VEG-Bench is introduced: 132 video-instruction pairs balanced across 8 editing skills (concept addition / removal / changing, environment/background changing, visual feature changing, stylization, object grounding, reasoning segmentation), with skill-specific metrics including MLLM-as-Judge via GPT-4o and GroundingDINO-based detection for addition/removal verification [§4.1].
VEGGIE outperforms all 6 baselines on aggregate ranking — Avg. Ranking 1.78 vs TokenFlow 1.41, Flatten 1.96, InsV2V 2.00, VidToMe 2.61, InstructDiff 3.00, LGVI 3.21 — making it the strongest instructional video editor; non-instructional baselines (TokenFlow, Flatten) require additional conditions (depth maps, captions) and rank well on smoothness/quality but underperform on faithful semantic change [Table 2].
Concept addition leads: MLLM-Judge 7.44 (best), CLIP alignment 29.27, GroundingDINO detection 57.96% — beats InstructDiff (7.26 / 28.10 / 55.36) and InsV2V (5.69 / 28.27 / 48.01) [Table 2].
Stylization is strongest: MLLM-Judge 8.26 (second to Flatten 8.31 which uses depth-map conditions), CLIP alignment 29.38, ahead of all instructional baselines [Table 2].
Object grounding: VEGGIE achieves SSIM 70.90 / Jaccard 37.74 / F-measure 21.83, beating all video-editing baselines by a large margin (next best LGVI: SSIM 66.84 but Jaccard only 1.52) [Table 2].
Reasoning segmentation: VEGGIE 68.41 / 22.53 / 15.97 vs InsV2V 59.86 / 16.89 / 10.45 — generative-editing models substantially beat dedicated segmentation pipelines that lack reasoning [Table 3].
Multi-task training is mutually reinforcing on balanced data: training grounding and removal jointly on VPLM (5.5K samples each) yields FVD 987.80 vs removal-only 1098.52 and SSIM 55.21 vs grounding-only 52.34 — but excess grounding data introduces artifacts in editing [Table 4].
t-SNE of task queries shows distinct skill clusters: Reasoning and Grounding co-cluster (both require semantic reference); Color, Env, and Change co-cluster (visual-attribute edits); Style is adjacent; Removal and Addition are maximally distant (opposing operations) [§4.4 Fig. 5].
LGVI fails on global edits (stylization, color change) because it is removal-specialized; TokenFlow / VidToMe struggle with local edits (add/remove) because attention-control priors constrain object additions/removals — VEGGIE’s MLLM-grounded per-frame queries handle both regimes [§4.3, Tab. 2].
InsV2V achieves high quality/smoothness scores but underperforms on alignment and MLLM-Judge because it tends to make minimal changes; the quality/faithfulness trade-off VEGGIE strikes is the headline contribution [§4.3, Fig. 3].

Method

VEGGIE has four components: (1) LLaVA-OneVision-7B as the MLLM, (2) a set of learnable grounded task queries, (3) a single-layer MLP alignment network projecting MLLM hidden states into the diffusion conditioning space, and (4) a MagicBrush-initialized SD1.5 UNet inflated with AnimateDiff-style temporal attention layers for video. The data flow is: video frames + instruction → MLLM produces N frame-wise task query embeddings → MLP → cross-attention condition into the UNet, which concatenates the source video latent with the noise volume and denoises to produce the edited video. Classifier-free guidance is applied to both task queries and source video, with separate guidance scales.

The architectural bet — and the part that diverges from later contemporaries — is the continuous per-frame task query. InstructDiffusion-style methods serialize the MLLM output through discrete text tokens, which disconnects gradient flow and forces the MLLM and diffusion model to communicate through a quantized bottleneck. VEGGIE instead lets the MLLM emit N learnable embedding vectors directly (one per frame), which (a) keeps the pipeline differentiable end-to-end, (b) provides per-frame conditioning that the temporal-attention UNet can specialize on, and (c) lets curriculum learning gradually align the two representation spaces.

Stage 1 trains only the alignment MLP, task queries, and 2D UNet on 3.4M paired image-edit samples (Seed-Data-Edit 3M, MagicBrush 9.3K, EraseDraw 64.9K, SmartEdit 0.5K, PhraseCut 310K, gRefCoCo 13.6K, LISA 0.2K) — 862M trainable parameters, MLLM frozen. Stage 2 inflates the 2D UNet with temporal attention, adds a LoRA to the MLLM, and fine-tunes everything end-to-end on 134K video samples (InstructV2V 68.3K, VPLM 4.3K, ROVI 4.3K, GroundMore 1.3K, RVoS 1.9K, MeViS 1.5K, plus 4K newly synthesized VEG-Edit samples) — 1.3B trainable parameters.

The video data synthesis pipeline is the second non-trivial contribution. Given a paired image-edit sample (source image, edited image, instruction), an offline MLLM generates a motion prompt describing plausible dynamics within the scene. An I2V model animates the source image into a video. A first-frame-conditioned video editing model uses the edited image as a strong prior to propagate the edit across the animated video, producing the paired edited video. VBench-style automatic quality metrics then filter for motion smoothness, image quality, and background consistency. This is the key data-side innovation that makes the image-to-video transfer work without relying on text-driven I2V + prompt-to-prompt editing (which suffer from temporal inconsistency).

Results

VEG-Bench aggregate (8 skills, Avg. Ranking lower=better): VEGGIE 1.78 — best among instructional models, slightly behind TokenFlow 1.41 (non-instructional, uses depth-map conditioning) [Table 2].
Concept Addition (5 metrics, instructional models only): MLLM-Judge 7.44 (best), CLIP-T 29.27, CLIP-F 94.93, MUSIQ 61.31, GroundingDINO 57.96% — VEGGIE wins MLLM-Judge and detection; InstructDiff and InsV2V trail [Table 2].
Concept Removal: MLLM-Judge 5.07 (LGVI leads at 6.59 — it’s removal-specialized), detection 70.22% (second to LGVI 78.40%) [Table 2].
Object Changing: MLLM-Judge 6.63 vs InsV2V 6.60 and Flatten 7.37 (Flatten uses depth conditioning) [Table 2].
Environment/Background: MLLM-Judge 7.18 vs InsV2V 6.60, CLIP-T 29.15 (second to Flatten 30.04) [Table 2].
Visual Feature Changing: MLLM-Judge 7.33 vs InsV2V 7.53; close [Table 2].
Stylization: MLLM-Judge 8.26 — second only to Flatten 8.31, ahead of all instructional baselines [Table 2].
Object Grounding (vs dedicated segmentation models, Table 3): Jaccard/F-measure/Mean = 37.74 / 21.83 / 29.79. Beats InstructDiff (16.35) and InsV2V (15.63) decisively but trails purpose-built segmentation models HTR (47.35), VideoLISA (53.80), MoRA (55.68) — VEGGIE is a generative editor that does grounding as a side capability [Table 3].
Reasoning Segmentation (Table 3): 22.53 / 15.97 / 19.25 — beats all editing baselines; still trails dedicated MoRA (40.36) but within striking distance.
Multi-task ablation (VPLM, removal + grounding): Mixed FVD 987.80 (vs removal-only 1098.52); mixed SSIM 55.21 (vs grounding-only 52.34) — explicit evidence that grounding and removal mutually reinforce when data is balanced [Table 4].

Why it’s interesting

VEGGIE is the conceptual ancestor of the wave of unified instruction-based video editors that landed 5-7 months later: EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning (Adobe, Sep 2025), InstructX: Towards Unified Visual Editing with MLLM Guidance (ByteDance, Oct 2025), UniVideo: Unified Understanding, Generation, and Editing for Videos (Oct 2025), UNIC: Unified In-Context Video Editing (Kling, Jun 2025). All share VEGGIE’s core thesis — one model, many editing skills, MLLM-mediated grounding — but diverge sharply on the architectural bet. EditVerse collapses everything into a single dense transformer with no MLLM, betting on in-context learning. InstructX keeps a separate MLLM + DiT and bridges them with LoRA + meta-queries, directly echoing VEGGIE’s continuous-token bridge but with a more powerful diffusion backbone (Wan2.1-14B vs SD1.5). UNIC abandons the MLLM entirely and lets the DiT’s self-attention do the work via task-indicator biases. VEGGIE’s contribution in retrospect is the design pattern itself: continuous per-frame task queries + curriculum learning from image-edit data + diffusion-only loss.

The image-to-video data synthesis pipeline is the most reusable artifact. The trick of animating both sides of a paired image-edit sample with a first-frame-conditioned video editor as a propagator is now standard practice — visible in Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto) (Ditto/Editto) and the EditVerse data pipeline. VEGGIE was the first to make this work with a small (4K) curated dataset and prove that combining it with image-edit transfer is sufficient.

Within the Unified Multimodal Models taxonomy VEGGIE is a Decoupled design — MLLM and generator are stitched at training time but architecturally separable, contrast with EditVerse’s E2E design. The key open question that subsequent work answered differently: do you need the MLLM at all? InstructX says yes and LoRA-tunes it; EditVerse says no and replaces it with full self-attention; VEGGIE was the first to make the decoupled-but-fully-differentiable recipe work via continuous task queries. The fact that grounding + editing mutually reinforce (Table 4) is a finding that should generalize past VEGGIE’s specific stack.