EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning

TL;DR

EditVerse (Adobe Research × CUHK × Johns Hopkins) is a single LLaMA-3-style dense transformer that handles text-to-image, text-to-video, image editing, and instruction-based video editing in one weight set by representing all modalities — text (Flan-T5-XXL tokens), image, and video (convolutional VAE latents) — as one interleaved 1D token sequence processed by full self-attention. The bet is that in-context learning across this unified sequence transfers editing knowledge from the data-rich image domain to the data-scarce video domain. To make it work the authors design a four-dimensional RoPE (height / width / sequential / temporal), curate a 232K-sample video editing corpus by inverting/filtering task-specific pipelines (Grounded-SAM-2, DiffuEraser, VACE, ReCamMaster) plus filtered Señorita-2M, and introduce EditVerseBench as the first instruction-based video editing benchmark with mixed-resolution (horizontal + vertical) videos and 20 task categories. Beats prior open-source instruction-based methods on every metric and is competitive with Runway Aleph (commercial) on EditVerseBench.

Key claims

All modalities — text, image, video — are projected into a shared embedding space and concatenated into one interleaved 1D sequence; full self-attention does the cross-modal mixing, with no per-modality cross-attention or per-task adapter [§3.1].
A four-dimensional RoPE (height, width, sequential, temporal) is added on top, with text tokens having height/width/temporal all set to 0 and only the sequential index incrementing; NTK-aware interpolation is used for context-window extension [§3.2].
Training objective is Flow Matching velocity prediction on a randomly selected image-or-video target inside the interleaved sequence; the rest of the sequence is provided clean as context [§3.3].
KnapFormer packing is used to batch variable-length sequences during training [§5.1].
The model is pretrained at 360p text-to-image + text-to-video on a dense LLaMA-3-style transformer, then fine-tuned at variable area (256² to 512²) with global batch size of 256 for ~ training steps with AdamW (peak LR ) [§5.1].
Training data is a four-way mix totalling tens of billions of tokens across image-edit, image-gen, video-edit, video-gen; the video editing portion is the bottleneck and is solved by curating 232K samples via the EditVerse Editing Data pipeline [§4, Table 1].
The video-editing data pipeline is task-specialist composition: Grounded-SAM-2 for masks, DiffuEraser for removal, VLM-imagined target text + VACE for replacement, first-frame style-transfer + VACE depth-to-video for style, ReCamMaster for camera change; mask-detection and propagation datasets are constructed by reformulating the above pairs with templated instructions [§4 “Video Editing Data Pipeline”].
Object-addition data is constructed by inverting the removal pipeline (post-removal → original, “remove” → “insert”), since direct addition pipelines fail to produce physically consistent samples [§4].
A VLM (Qwen2-VL) scores each candidate on instruction adherence, context preservation, sharpness, temporal consistency, artifact presence, object integrity, aesthetics, physical plausibility; thresholds are calibrated by manual inspection. EditVerse’s pipeline yields a retention rate six times higher than Señorita-2M after filtering [§4 “Data Filtering”, Table 1].
EditVerseBench is the first instruction-based video editing benchmark with mixed-resolution videos (50% horizontal / 50% vertical), 20 task categories, and 200 editing pairs; metrics span VLM-judged editing quality, Pick Score, ViCLIP / CLIP text alignment, and CLIP / DINO temporal consistency [§5.2].
On EditVerseBench, EditVerse beats every reported open-source baseline on every metric — VLM editing quality 7.65 vs Señorita-2M 6.97 / Lucy Edit 5.89 / InsV2V 5.21 / TokenFlow 5.26 / STDF 4.41; Pick Score 20.07; CLIP temporal 98.56; DINO temporal 98.42 [Table in §5.3].
EditVerse exceeds Runway Aleph on VLM editing quality (7.65 vs 7.44) and matches it within ≤0.97 on the remaining metrics — the first filed open-source instruction-based video editing system to be competitive with Aleph on a third-party benchmark [Table in §5.3].
On TGVE+ (all-square videos, despite EditVerse seeing no square training data) EditVerse matches Movie Gen Edit on ViCLIPdir (0.225) and is the only model to tie or beat it on both ViCLIPdir and ViCLIPout (0.252 vs 0.248) [Table 3].
Emergent editing tasks not present in training data — material change, weather change, effect addition, and multi-task compositions (reference-image insertion + inpainting) — are reported to work [§5.4, Figure 1].
On in-distribution tasks (object removal, object replacement) EditVerse outputs are reported to surpass the ground-truth training samples in both quality and success rate, attributed to knowledge transfer from image gen/edit [§5.4, Figure 7].
Failure modes include positional errors (“add chest at man’s feet” placed elsewhere) and blurry artifacts in the edited region [§ Limitations, Figure 9].

Method

EditVerse uses a single LLaMA-3-style dense transformer with full self-attention. The input is constructed by tokenizing each modality separately — convolutional VAE for images and videos (with spatio-temporal downsample ratios, patchified into a 1×2×2 kernel), Flan-T5-XXL for text — then projecting each into the shared hidden dimension via single-layer linear heads. Tokens are concatenated in the original interleaved order from the instruction; learnable <sov> / <eov> tokens bracket every vision segment. Position is encoded via a four-dimensional RoPE: each of (height, width, sequential, temporal) gets its own RoPE block over a partition of the embedding dimension. Text tokens have height/width/temporal = 0 and only the sequential index incrementing; image tokens have height/width filled in and temporal = 0; video tokens have all four. NTK-aware interpolation extends context.

Training is Flow Matching: pick one image-or-video segment as the generation target, add noise per the schedule, and regress the velocity field with the rest of the sequence supplied as clean conditioning context. The model thus learns to fill in any one slot of an interleaved multimodal sequence given the others — text-to-image, text-to-video, image-to-image-edit, video-to-video-edit, image+text-to-video, and arbitrary novel mixes all reduce to the same objective.

The data work is more involved than the architecture. The video-editing corpus is constructed by composing task-specialist diffusion pipelines: Grounded-SAM-2 produces object masks; DiffuEraser inpaints to remove (giving an object-removal pair); VLM-imagined target prompts + VACE produce replacement; first-frame style transfer (image-side) + VACE’s depth-to-video synthesizes style-transferred videos; ReCamMaster produces camera-change pairs. Mask-detection and propagation samples are constructed by retemplating the above pairs with new instructions. Object-addition data is constructed by swapping the removal pair: post-removal video becomes the source, original becomes the target, and the instruction is rewritten — direct addition pipelines turn out to produce physically inconsistent samples. All candidates are then VLM-filtered along eight dimensions; the retention rate ends up 6× higher than Señorita-2M.

Inference uses classifier-free guidance with scale 6 applied only to text conditions, and a 50-step ODE solver. The same checkpoint switches between T2I, T2V, image-edit, video-edit, and mixed-modality inputs by varying which segment is masked at inference.

Results

EditVerseBench (200 instruction-based editing pairs, mixed horizontal/vertical, 20 tasks) — VLM Editing Quality 7.65 (best; next open-source Señorita-2M 6.97; commercial Runway Aleph 7.44); Pick Score 20.07; CLIP frame alignment 26.73; ViCLIP video alignment 23.93; CLIP temporal 98.56; DINO temporal 98.42 [§5.3 table]. EditVerse beats every reported open-source baseline on every metric and beats Runway Aleph on VLM editing quality.
TGVE+ (all-square videos, out-of-distribution since EditVerse trains on no square videos) — ViCLIPdir 0.225, ViCLIPout 0.252. Matches Movie Gen Edit on ViCLIPdir and beats it on ViCLIPout (0.248) [Table 3].
User study (pairwise preference, 20 participants, 3 criteria — instruction alignment, unedited-region preservation, overall quality) — preferred against InsV2V, Lucy Edit, Señorita-2M, TokenFlow, STDF [Figure 5]. The user study is reported to track the VLM editing-quality metric more closely than any of the CLIP/Pick/temporal metrics.
Emergent abilities — handles tasks outside the training set (material change, weather change, effect addition, multi-task compositions) and on in-distribution tasks (object removal/replacement) reportedly exceeds the ground-truth training samples on quality and success rate [§5.4].
Knowledge-transfer ablation — removing either the image-editing or the image-generation data degrades video editing performance (specific numbers in the truncated tail of §5.4) — direct evidence that the image domain is what’s carrying the video-editing capability.

Why it’s interesting

EditVerse is the closest published precedent for a “one weight set, edit and generate across image+video” stack — directly relevant to the Omni line of work alexm flagged in the Slack message. Three angles that matter:

Recipe contrast with OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning (Tencent OmniWeaving): both unify text/image/video editing+generation in one model, but with opposite architectural bets. OmniWeaving keeps a separate MLLM and MMDiT and bridges them via DeepStacked hidden states + a reasoning-trace loss; EditVerse collapses to a single full-self-attention transformer over interleaved tokens with no MLLM and no per-task adapters. EditVerse trains end-to-end Flow Matching from a single objective; OmniWeaving uses three training stages plus an NTP loss on reasoning traces. The Adobe team is betting on architectural simplicity + in-context learning + a 6× higher-retention data pipeline; the Tencent team is betting on a “comprehend-then-generate” two-component design.
In-context editing on the video side, before video-side adapters proliferated. EditVerse predates and is the conceptual parent of UNIC: Unified In-Context Video Editing (Kling UNIC, Jun 2025) — both abandon per-task adapters in favor of “concatenate everything and let the DiT’s self-attention do the work.” UNIC adds task-specific RoPE rearrangement + condition-bias as a task indicator; EditVerse argues that with sufficient data + a 4D RoPE + <sov>/<eov> brackets, even those primitives are unnecessary. Worth a direct A/B on the same backbone if Omni would adopt this recipe.
Data engineering as the real contribution. The single most reusable artifact here is probably the EditVerse Editing Data pipeline — invert removal to produce addition, VLM-filter on 8 dimensions, 6× retention vs Señorita-2M — and EditVerseBench, a mixed-resolution instruction-based video-editing benchmark with 20 task categories. Both are needed by any internal “Omni-style” effort and neither requires the full architecture to be reused.

The reported “competitive with Runway Aleph on VLM-judged quality” claim is a single benchmark with self-chosen judge, but it’s the first filed open-source instruction-based video editor to even claim that. The benchmark generalization (TGVE+ all-square, out-of-distribution at training time) and the emergent-abilities section are stronger evidence than the headline table.