EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing

TL;DR

EgoEdit (Snap Research × Rice × Oxford) targets instruction-guided egocentric video editing for interactive AR, addressing two gaps in the editing literature: existing editors trained on third-person footage suffer a large domain gap on egocentric views (rapid egomotion, frequent hand-object interactions), and offline editing pipelines are too slow for live interaction. The contribution is an end-to-end ecosystem: (1) EgoEditData, a 49.7k-video / 99.7k-pair manually curated dataset built by composing Wan 2.1 VACE 14B + Qwen-Image + GPT-5 Mini over Ego4D / EgoExo4D, with multi-stage human-in-the-loop filtering retaining only 0.4% of source videos; (2) EgoEdit, a streaming editor built by adapting a Wan-class T2V generator with channel-wise source conditioning, then distilled to autoregressive 4-NFE inference via bidirectional DMD followed by Self-Forcing — hitting 38.1 FPS / 855 ms first-frame latency on one H100; (3) EgoEditBench, a 15-task egocentric editing benchmark following EditVerseBench’s protocol. Beats Lucy Edit / InsV2V / Señorita-2M / StreamDiffusionV2 on egocentric tasks while matching EditVerse on the general-editing case.

Key claims

Egocentric editing is a distinct domain from exocentric video editing, with three challenges current editors don’t handle: rapid egomotion, frequent hand-object occlusion and manipulation, and the latency budget for live AR [§1].
The headline interactive numbers are 38.1 FPS sustained throughput and 855 ms first-frame latency on a single H100 GPU, at 512×384 / 16 FPS [§1, §4.3, Table 2].
EgoEditData curation is heavily lossy by design: only 1.8% of Ego4D/EgoExo4D videos survive video selection; 49.6% survive hand-mask filtering; 43.6% survive object-mask filtering; 37.8% of generated edits survive the final human QA — net retention 0.4% of source videos [§3.1, §3.2].
The data pipeline targets actively manipulated objects specifically, using a hand–object proximity criterion (edge distance between hand and object masks, plus hand-skeleton-keypoint distances from WiLoR) to filter out coincidental hand–object co-occurrence [§3.1, “Object mask segmentation”].
The generator stack for synthetic edits is GPT-5 Mini (proposes diverse target objects, ordinary + imaginary) → Qwen-Image (renders the target reference image) → Wan 2.1 VACE 14B (synthesizes the edited video conditioned on reference image, scene-level prompt, and object mask) [§3.1, “Object Editing”].
Object addition is not synthesized directly — instead, removal pairs are inverted (post-removal video → original; “remove” → “insert”), echoing EditVerse’s trick because direct addition pipelines fail to produce physically consistent samples [§3.1].
Channel-wise source conditioning is chosen over sequence-wise concatenation because the latter’s quadratic self-attention growth conflicts with real-time latency requirements; channel-wise keeps cost close to the base T2V model [§4.2].
Distillation is a two-stage pipeline: bidirectional DMD compresses the 40-step / CFG model to 4-step / distilled-guidance (80 → 4 NFEs), then Self-Forcing runs the causal model autoregressively at train time with a DMD loss against the bidirectional teacher’s score model, with chunks of three latent frames [§4.3].
The base T2V is trained on Wan 2.1 autoencoder latents, and the autoencoder natively supports autoregressive operation — i.e. the decoder side does not become a latency bottleneck during streaming [§4.2, §4.3].
Total editing-training corpus is EgoEditData (99.7k pairs) plus 1.31M additional video and 3.5M image editing pairs; 30k iterations at batch 96, lr 1e-5, AdamW + EMA [§5.1].
On EgoEditBench (15 tasks, 100 egocentric source videos derived from a held-out Ego4D split, K-means-clustered for diversity, GPT-5-generated instruction prompts), EgoEdit reaches VLM 7.76 / PS 19.21 / TA 16.89 / TC 96.70 — beating Lucy Edit (VLM 5.44), InsV2V (5.24), Señorita-2M (7.52), AnyV2V (6.72), TokenFlow (4.99), STDF (4.59) [Table 1].
On the exocentric EditVerseBench, EgoEdit scores VLM 8.00 vs EditVerse 8.26 / Lucy Edit 6.27 / InsV2V 5.71 — within 0.26 VLM of the EditVerse SOTA without specializing on the exocentric case [Table 1].
The egocentric→exocentric gap is much smaller for EgoEdit (-0.24 VLM) than for Lucy Edit (-0.83) or InsV2V (-0.47), a direct quantitative argument that EgoEditData closes the egocentric domain shift [§5.2].
EgoEdit-RT (the Self-Forcing-distilled streaming variant) outperforms the streaming baselines StreamDiffusion (VLM 4.32 / 4.33) and StreamDiffusionV2 (2.55 / 2.78) by large margins on both benchmarks, and the gap to its own bidirectional teacher is small across quantitative metrics [Table 1].
Streaming-mode resolution is below standard consumer video (480p+), with occasional temporal seams at chunk boundaries and degraded performance on highly OOD instructions or heavy occlusion — explicitly named as limitations [§ Limitations].

Method

EgoEdit starts from a Wan-2.1-autoencoder-latent text-to-video DiT (transformer backbone with cross-attention text conditioning per block, Rectified Flow training objective). To convert it into an editor, the source video is concatenated to the noised target video along the channel dimension before patchification (the Lucy-Edit-style choice, contra EditVerse / UNIC’s sequence-wise concatenation). This keeps the token sequence the same length as the base T2V model and avoids quadratic self-attention cost growth — a non-negotiable for the 855 ms first-frame budget.

The bidirectional editor is fine-tuned on EgoEditData (99.7k pairs) plus a 1.31M-video + 3.5M-image edit corpus for 30k steps at batch 96. The editor is then distilled in two passes:

Bidirectional DMD distillation (4.5k steps) — compresses 40-step + CFG inference (80 NFEs) into a 4-step student with distilled guidance (4 NFEs), using a separate critic at lr 4e-7.
Self-Forcing distillation (3.5k steps) — the now-4-step causal student is rolled out autoregressively at train time, with a DMD loss against the bidirectional teacher’s score model. Chunks are three latent frames; the Wan 2.1 autoencoder is used in its native AR mode.

EgoEditData itself is the larger contribution and is built as a six-stage filtering cascade on Ego4D + EgoExo4D: high-quality-camera filtering (1.8% retained) → hand mask annotation with WiLoR-prompted SAM 2 + human review (49.6%) → manipulated-object name extraction with Qwen2.5-VL-32B → Grounded-SAM + hand–object proximity + SAM-2 refinement + human review (43.6%) → synthetic edit generation via GPT-5 Mini target proposal + Qwen-Image reference + Wan 2.1 VACE 14B → human QA on the synthetic outputs (37.8%) → GPT-5-Mini instruction caption generation per (source, target) pair. The Wan-VACE generation step runs at 1/60 FPS on 8 H100s, which is why the human-curated rejection rate of 62.2% at that stage is acceptable cost — they only run VACE on videos that have already passed every cheaper filter.

EgoEditBench follows EditVerseBench’s evaluation protocol but is built from a held-out Ego4D split not used for EgoEditData. 100 source videos are selected by K-means (10 clusters × 10 samples) over BERT embeddings of (object name + scene description) to enforce diversity; GPT-5 then generates 15 task-specific instructions per source: Add Object, Add Effect, Remove Object, Change Object (4 variants per video, two replacement + two replacement-plus-effect), Change Background, Change Camera Pose, Stylization, Reasoning, Depth→Video, Sketch→Video, Pose→Video, and the four X→Video inversions (Video→Pose / Sketch / Depth), plus a Combined Task.

Results

EgoEditBench (egocentric, 15 tasks) — EgoEdit: VLM 7.76 / Pick Score 19.21 / Text Alignment 16.89 / Temporal Consistency 96.70. EgoEdit-RT (streaming): 7.71 / 19.13 / 16.34 / 96.41. Best filed open-source baseline: Señorita-2M at 7.52 / 18.85 / 16.25 / 95.86; Lucy Edit 5.44; InsV2V 5.24; StreamDiffusionV2 2.55. EgoEdit is the only model that achieves both >7.5 VLM and >96 TC in the egocentric setting [Table 1].
EditVerseBench (exocentric, propagation/inpainting/reference-image tasks removed) — EgoEdit: 8.00 / 19.61 / 24.40 / 98.54 vs EditVerse 8.26 / 19.69 / 25.29 / 98.68. EgoEdit-RT 8.18 / 19.59 / 17.61 / 98.55 (the TA dip on EgoEdit-RT is attributed to chunk-boundary effects in the truncated text). Lucy Edit 6.27; InsV2V 5.71; StreamDiffusionV2 2.78 [Table 1].
Domain-shift quantification — VLM drop egocentric→exocentric: EgoEdit -0.24, InsV2V -0.47, Lucy Edit -0.83. First-frame-propagation methods (Señorita-2M, AnyV2V) preserve their score because they receive EgoEdit’s edited first frame as input — i.e. they’re not solving the egocentric editing problem on their own [§5.2].
First-chunk latency — the non-distilled EgoEdit takes 80 NFEs and full-clip generation before any frame is shown; DMD-4-step reduces to 4 NFEs but still synthesizes the whole clip; Self-Forcing AR delivers 855 ms first-frame latency and 38.1 FPS sustained on one H100. Ablations in Table 2 confirm SF is the load-bearing piece — DMD-only without SF cannot deliver first-chunk latency [§4.3, Table 2].
Ablations — explicit ablations on EgoEditData’s contribution are reported (in the truncated tail of §5.3) and the qualitative picture is that EgoEditData itself is responsible for the egocentric-domain wins; the architecture is otherwise close to Lucy Edit.

Why it’s interesting

EgoEdit is the cleanest filed datapoint at the intersection of three lines the wiki has been tracking: video editing, real-time streaming AR video, and the Self-Forcing distillation lineage. Three things stand out:

It bolts the Autoregressive Video Generation streaming recipe onto an editor. The Self-Forcing / Causal Forcing / Causal Forcing++ thread on the wiki is mostly about unconditional or text-conditional AR video; EgoEdit shows the same recipe (bidirectional DMD pre-distill → Self-Forcing) ports to source-video-conditional editing without architectural changes, with the source supplied via channel-wise concatenation (Lucy Edit’s trick) rather than the sequence-wise concatenation EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning and UNIC: Unified In-Context Video Editing use. The cost contrast is the whole point — at 38.1 FPS and 855 ms TTFF, this is a different latency regime than EditVerse can reach. The same Causal Forcing analysis that says bidirectional→causal ODE distillation has an injectivity hole would apply to EgoEdit’s two-stage pipeline, so a Causal-Forcing-style fix is the obvious next refinement.
It’s a sharp counter-example to “scaling data fixes domain shift.” Lucy Edit and InsV2V both have access to large general-purpose editing corpora; both lose 0.47–0.83 VLM going egocentric. EgoEdit closes that to -0.24 with 99.7k task-specific pairs (against 232K for EditVerse and 2M for Señorita). The lesson is the same as EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning‘s 6× retention claim — quality and domain alignment of synthetic editing data dominate raw scale — but on a harder distribution (hand-object interaction + egomotion) where existing corpora simply don’t have coverage.
The synthetic-data pipeline is a maximalist instance of the Synthetic Training Data thread. GPT-5 Mini proposes targets, Qwen-Image renders references, Wan 2.1 VACE 14B generates the actual edited video, Qwen2.5-VL extracts object names, Grounded-SAM + SAM 2 + hand-keypoint geometry validate manipulation, and humans gate every intermediate stage. The 0.4% net retention is striking and explicit — this is the highest-effort curation pipeline filed on the wiki so far, and the editing community appears to be converging on “generator stack as curriculum + human filter at every stage” as the recipe.

The honest caveat is that 512×384 / 16 FPS / occasional chunk-boundary seams is genuinely below consumer-video quality, and the headline 38.1 FPS / 855 ms numbers are H100-specific. For an AR application this is the right tradeoff; for production-quality video editing it is not. But the recipe (channel-wise source conditioning + Self-Forcing distillation + a 0.4%-retention synthetic data pipeline) is reusable across both regimes.