ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models

TL;DR

ShotAdapter is an Adobe Research / Georgia Tech / UIUC system (CVPR 2025) that fine-tunes a pretrained text-to-video diffusion model to generate multi-shot videos with discrete transitions — the same character performing distinct activities across same or different backgrounds — as a single full-attention sequence rather than as concatenated independent clips. The contribution is two-fold: (a) a small architectural extension — learnable “transition tokens” inserted between shot prompts, gated by a local attention mask that scopes each text prompt to its own shot — and (b) a synthetic data-collection pipeline that stitches existing single-shot videos into pseudo-multi-shot training data using both random splitting and identity-clustered concatenation, then post-processes with LLaVA-NeXT for shot-specific captions. A few thousand fine-tuning iterations suffice for shot-number, duration, and per-shot prompt control while preserving cross-shot character consistency.

Key claims

Multi-shot generation is reformulated as a single full-attention video generation problem with n-1 learnable transition tokens inserted into the text sequence, rather than stitching independently-generated clips [§ Method].
Transition tokens are gated by a local attention mask inside the DiT joint-attention blocks so that (i) each text token attends only to the visual tokens of its own shot, and (ii) transition tokens interact only with the visual frames where the corresponding cut occurs [§ Method, Fig. a–c].
The transition tokens are initialized as learnable parameters and trained end-to-end alongside the masked attention; no new attention heads or backbone changes are required [§ Method].
The training data pipeline has two synthesis methods: (1) sample single-shot videos with large motion, randomly split them into n shots of varied durations, and concatenate; (2) sample n clips from pre-clustered groups of same-identity videos and concatenate to preserve cross-shot identity [§ Data pipeline, Fig. b].
Post-processing applies an identity-consistency filter and uses LLaVA-NeXT to produce per-shot captions, yielding aligned (multi-shot-video, list-of-shot-prompts) training pairs [§ Data pipeline, Fig. c].
The user controls the number of shots, the duration of each shot, and per-shot textual prompts — multi-shot structure is exposed as conditioning, not hidden in the prompt [§ Method].
Fine-tuning a pre-trained T2V model “for a few thousand iterations” is sufficient to acquire multi-shot generation with shot-specific control, outperforming the cited baselines (MEVG, FreeNoise, Gen-L-Video, SEINE) on background and character consistency [§ Results].

Method

ShotAdapter fine-tunes an existing text-to-video DiT. Given a desired n-shot output, the input text sequence is constructed as [prompt_1, ⟨transition_1⟩, prompt_2, ⟨transition_2⟩, …, prompt_n], where the ⟨transition_i⟩ tokens are learnable embeddings initialized at fine-tune start. The video latent is a single contiguous sequence covering all shots; positions of the shot boundaries are fixed (the user specifies the per-shot durations), and the transition tokens are bound to the frame indices where those boundaries occur.

The crucial change is in the joint-attention layers of the DiT block. A local attention mask is applied so that the QKᵀ logits between text and visual tokens are nonzero only when:

A textual prompt token attends to visual tokens within its own shot.
A transition token attends to a small local window of visual tokens around its boundary frame.
Visual tokens attend to all visual tokens (full visual self-attention is preserved, so character/background continuity across shots is learned by the model rather than enforced by post-hoc stitching).

The model is fine-tuned on synthetic multi-shot videos produced by the data pipeline. Two complementary synthesis methods feed the dataset: (a) random-splitting, which picks single-shot clips with strong intra-clip motion, samples random cut points to form n sub-shots with varied durations, and concatenates them — yielding hard cuts within a coherent identity but no enforced cross-shot semantic link; and (b) identity-clustered concatenation, which pre-clusters a single-shot video pool into groups containing the same identity (face / character), then samples n videos from one cluster and concatenates them — yielding cross-shot identity consistency with arbitrary semantic transitions. Each composed video is post-filtered for identity drift and re-captioned per-shot by LLaVA-NeXT to produce the (multi-shot, list-of-prompts) pairs the model trains on.

Results

The project page reports qualitative comparisons on 2-, 3-, and 4-shot generation and references the baselines MEVG (Oh et al., ECCV 2024), FreeNoise (Qiu et al., ICLR 2024), Gen-L-Video (Wang et al. 2023), and SEINE (Chen et al., ICLR 2024). The fetched landing page emphasizes:

Generated 2-, 3-, and 4-shot results where ShotAdapter maintains identity across distinct activities and across both same-background and changing-background transitions (paired example: shot-1 “a young girl paints at an easel in her bedroom” → shot-2 “she then reads a comic book in her bed”).
The claim of outperforming baselines on background and character consistency is asserted but no headline numerical table is exposed on the landing page; for full quantitative comparison the CVPR 2025 paper is the source.

Why it’s interesting

ShotAdapter is the model-side complement to a cluster the wiki has been steadily filling out. Prompt Relay: Inference-Time Prompt Routing for Temporal Control in Multi-Event Video Generation solves the same per-shot-prompt problem entirely at inference time by biasing cross-attention with a Gaussian-tail distance penalty — no fine-tune, no architectural change, but no learned transition either. Co-Director: Agentic Generative Video Storytelling solves it pipeline-side by orchestrating multiple frozen black-box generators with a bandit over creative directions. OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning retrains a unified DiT for composition + reasoning over multi-event prompts. ShotAdapter sits between Prompt Relay (training-free, no transition primitive) and OmniWeaving (full retrain): a few-thousand-step fine-tune that adds an explicit, learned transition primitive plus a local attention mask — the cheapest model-side recipe filed so far. The synthetic data pipeline is also a clean instance of the wiki’s Synthetic Training Data thesis (“forward-generate the effect rather than reverse-annotate it”): existing single-shot clips can be programmatically composed into multi-shot training pairs without ever filming a real multi-shot video, mirroring the synthesis-over-annotation argument made by OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer for shot-boundary detection. Where ShotAdapter complements Vidmento (Vidmento: Creating Video Stories Through Context-Aware Expansion With Generative Video) is that Vidmento explicitly defers to “whatever T2V/I2V model is plugged in” for its animate button — ShotAdapter is a candidate for what gets plugged in once a UX expects per-shot prompts and discrete transitions as first-class.