TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation

TL;DR

TalkCuts is a NeurIPS 2025 dataset (UMass Amherst + Tencent + Sony AI + UCSD) of 164k clips / 500+ hours of 1080p human speech video drawn from talk shows, TED talks, and stand-up — over 10k unique speaker identities. Unlike prior talking-head corpora that fix the camera at a single close-up, TalkCuts annotates each clip with a camera-shot class (close-up, medium close-up, medium, medium-full, full, wide) along with 2D keypoints, 3D SMPL-X motion, and detailed text descriptions, giving the first large-scale supervision signal for multi-shot speech video. The paper also releases Orator, an LLM-as-director baseline that orchestrates per-shot camera/gesture/voice plans into a long-form generation pipeline, and shows that both pose-guided and audio-driven generators trained on TalkCuts beat their single-shot-trained counterparts on cinematographic coherence.

Key claims

164k clips, ~500 hours, 1080p, 10k+ identities sourced from talk shows / TED / stand-up comedy — the first large-scale human-speech corpus that varies camera framing instead of fixing it [§Abstract, §Dataset].
Six camera-shot classes derived from cinematographic conventions and 2D keypoint analysis: Close-Up (CU), Medium Close-Up (MCU), Medium Shot (MS), Medium Full Shot (MFS), Full Shot (FS), Wide Shot (WS) [§Dataset Construction].
Each clip carries per-frame 2D keypoints and 3D SMPL-X body parameters in addition to detailed textual descriptions, enabling pose-guided and audio-driven training in the same corpus [§Annotations].
Orator baseline: an LLM functions as a “multi-faceted director,” orchestrating camera-transition, gesture, and vocal-modulation specifications across shots, which a downstream multi-modal video module then synthesizes into coherent long-form output [§Method, §Orator].
Training on TalkCuts significantly improves cinematographic coherence and visual appeal of generated multi-shot speech videos in both pose-guided and audio-driven settings vs. single-shot-trained baselines [§Experiments].
The paper explicitly notes that even pose-guided generators that benefit from TalkCuts still struggle with temporal coherence, identity preservation, and high-fidelity motion across diverse camera perspectives — flagged as an open problem in motion-aware diffusion [§Discussion].

Method

The dataset construction pipeline starts from YouTube-sourced speech video (talk shows, TED talks, stand-up). Shot-boundary detection segments long videos into single-shot clips. A 2D-keypoint extractor estimates body framing per clip; clips are then automatically classified into the six camera-shot categories using a rule-based scheme grounded in cinematographic conventions (e.g. ratio of visible body relative to frame). 2D keypoints and 3D SMPL-X parameters are extracted per frame from an off-the-shelf body estimator; text descriptions are produced by a VLM caption pipeline. Quality filters apply identity-consistency checks and aesthetic gates before clips enter the released corpus.

The Orator baseline frames multi-shot generation as a planning + rendering pipeline. Given a target speech (audio + transcript), an LLM acts as director and emits a structured plan: a sequence of (camera-shot type, gesticulation description, vocal-modulation instruction) per shot, with explicit transition timing. A downstream multi-modal video generation module — operable in either pose-guided mode (driven by SMPL-X motion synthesized from the gesture spec) or audio-driven mode (driven by the speech) — renders each shot to video conditioned on the plan. The Orator + TalkCuts combination is positioned as a baseline for the community to beat, not a SOTA system.

Results

The paper reports comparisons on both pose-guided and audio-driven speech-video generation; concrete headline numbers from the project page and abstract:

Pose-guided generators trained on TalkCuts produce more cinematographically coherent and visually appealing multi-shot videos than single-shot-trained counterparts (qualitative + quantitative on the paper’s own benchmark; the abstract emphasizes “significantly enhances” without single-number summaries) [§Experiments].
Audio-driven generation likewise improves on the cinematographic coherence axis when trained on TalkCuts vs. single-shot speech corpora [§Experiments].
Even with TalkCuts training, the paper flags persistent failures on temporal coherence, identity preservation, and high-fidelity motion synthesis across diverse camera perspectives — these are framed as the motivation for the dataset’s release rather than as solved by it [§Discussion, ResearchGate summary].

The released artifacts: dataset, annotation tools, evaluation protocols (project page promises public release).

Why it’s interesting

TalkCuts plugs a hole the Audio-Driven Character Animation cluster has been quietly working around: every filed cluster entry — Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (MultiTalk) (MultiTalk), InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing (InfiniteTalk), Wan-S2V: Audio-Driven Cinematic Video Generation (Wan-S2V), TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models (TalkingMachines), HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning (HuMo) — trains on talking-head corpora (HDTF, CelebV-HQ, EMTD) that are single-shot close-ups, then has to either (a) restrict generation to talking-head outputs or (b) hope the Wan/Wan2.2 base contains enough cinematic prior to extrapolate. Wan-S2V explicitly introduces a custom benchmark to evaluate cinematic output because no shared multi-shot speech-video corpus existed. TalkCuts is that corpus. It also sharpens the open question on the concept page about “cinematic vs talking-head benchmark fragmentation” — TalkCuts is a candidate unification target. The closest method-side sibling is ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models (ShotAdapter), which fine-tunes a T2V model on synthetically composed multi-shot videos; TalkCuts is the real multi-shot corpus that ShotAdapter’s synthetic pipeline was meant to substitute for. The pose-guided + audio-driven dual-mode evaluation also makes TalkCuts a natural companion to InfinityHuman: Towards Long-Term Audio-Driven Human Animation, whose long-form recipe explicitly routes through a pose intermediate. On the dataset-pipeline side, TalkCuts sits in the same lineage as Action100M: A Large-scale Video Action Dataset and Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset — large-scale automated VLM/keypoint-pipeline annotation over in-the-wild video — and reinforces the Synthetic Training Data thesis that the labels (camera shot class, SMPL-X, captions), not the videos, are what’s synthesized at scale.