Vidmento: Creating Video Stories Through Context-Aware Expansion With Generative Video

TL;DR

Vidmento is an Adobe Research / Harvard CHI 2026 system for authoring narrated video stories that interleave captured footage with generative-video clips. The contribution is the “generative expansion” framework — four design goals (Explore / Expand / Blend / Control) that position the generator as a context-aware collaborator rather than a from-scratch producer — and a linked canvas + script editor that operationalizes them. A 12-creator study found participants used Vidmento for travel vlogs, wedding reels, and personal essays they described as hard or impossible to make with existing workflows. For Luma the relevance is product surface, not modeling: it pins down a concrete UX recipe for “fill the gaps in my footage” that consumes whatever T2V/I2V backbone is plugged in.

Key claims

“Generative expansion” is articulated as four design goals — D1 Explore (surface latent story ideas from the user’s footage), D2 Expand (highlight narrative gaps where new material would improve continuity/pacing), D3 Blend (generate clips that match style and narrative context of surrounding shots), D4 Control (let the user steer sequence, shot composition, and generation parameters) [§4].
Formative interviews with 8 video creators identified three opportunities (story construction/exploration, telling previously-impossible stories from photos/artifacts, expressive reinterpretation) and three frictions (style/motion integration issues, authenticity concerns about generated content misrepresenting lived experience, preserving creative authorship) for blending generated and captured video [§3.2, §3.3].
Vidmento’s interface is two linked views: a semi-structured canvas (freeform spatial layout that doubles as a sequencing timeline via semantic zoom — story / scene / shot levels) and a script editor that segments per scene and offers Socratic-style suggestions rather than auto-drafting prose [§5.1, §5.2].
Context-aware video generation uses a two-stage pipeline: first a contextual keyframe is generated conditioned on the previous and next shots plus the scene script, then an I2V model animates the keyframe — so the join between captured and generated footage is grounded by neighboring frames rather than text alone [§5.3].
All AI features are framed as suggestive (accept / reject / adapt), and captured vs. generated clips are visually distinguished throughout the canvas — design choices made in direct response to formative-study concern C3 (preserving authorship) [§5.2].
In a 12-creator user study, participants produced rough cuts (travel vlogs, wedding reels, personal essays) that they described as difficult to achieve with existing workflows, and reported the system “amplified rather than diminished” their creative voice [§7, §1].

Method

Vidmento takes a story-construction-first stance: it outputs a rough cut, not a finished edit, deliberately leaving final post-production to Premiere/Resolve. Two surfaces drive interaction. The canvas lets users drop in their captured shots, auto-groups them into scenes (titled and colored), and lays them out into a visual storyline. As the user sequences scenes, Vidmento proposes new connecting scenes (gap detection) and proposes new shots within a scene (sequence-visuals feature) — both rendered in a distinguishable color so the user can see at a glance which material is real and which is suggested. Any image (captured or generated) becomes a video via an “animate” button that opens a generation panel with spatial annotations and structured cinematographic fields (camera movement, lighting, style, action) auto-suggested from the image. The script editor mirrors the scene structure of the canvas, syncs the user’s notes as a starting point, and provides Socratic-style suggestion prompts (e.g. “How did the calligraphy demonstration make you feel, and what specific details captured your attention?”) spanning imagery, structure, pacing, theme — explicitly not generating prose, on the basis that formative-study participants resisted ghost-written narration.

The generation pipeline is two-stage: contextual keyframe → image-to-video. The keyframe is conditioned on (a) the previous and next shots in the sequence, (b) the story context, and (c) the scene script — so the visual style and content of the synthetic shot inherits from its neighbors, addressing C1 (style/motion mismatch). The system then animates the keyframe with an I2V model. The paper is backbone-agnostic about which T2V/I2V model is plugged in.

Results

12-creator user study (separate from the 8-person formative study). Participants produced rough cuts across genres including travel vlogs, wedding reels, and personal essays [§7].
Qualitative finding: participants consistently reported the system supported narrative development and exploration — not just gap-filling — and “amplified rather than diminished” their creative voice. No quantitative comparison to a baseline editor is offered; the paper is a qualitative HCI study, not a horse-race [§7, §8].
The paper explicitly maps observed user behaviors back onto the D1–D4 design goals, and surfaces failure modes (where blending captured and generative footage requires more creator judgment to maintain coherence) as boundary conditions of the framework rather than as resolved [§7, §8].

Why it’s interesting

Most of the wiki’s video-generation papers are about the model (FastVideo, Helios, Wan-Animate, OmniWeaving) — Vidmento is about the seat the model sits in. Two-stage “context keyframe → I2V” framing complements OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning (unified composition + reasoning at model level) and EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing (effect-aware removal/insertion at clip level) by showing what the storyboard-level UX could look like once those primitives are available. The formative finding that creators reject ghost-written scripts but welcome Socratic prompts is a useful constraint to keep in mind for any Luma agentic-creator surface — compare the explicit-suggestion-only stance with the more autonomous direction in Introducing Runway Labs and Into the Dreamverse: Vibe Directing in FastVideo. The paper’s reference to Adobe’s existing “Generative Extend” feature as too-local (extending one clip without considering the larger narrative) is also a useful framing of the gap between per-clip generation and story-level authoring.