Co-Director: Agentic Generative Video Storytelling

TL;DR

Co-Director is a Google multi-agent framework that turns a six-point prompt (brand, product, gender, age, location, interest) into a 12-second advertising video, by formalizing storytelling as a global optimization over a discrete “creative direction” space steered by a Multi-Armed Bandit. The pipeline factors the work across Pre-Production (creative brief → storyline → asset → storyboard), Production (keyframe / video / audio agents), and Post-Production, with local self-refinement loops at the storyline and keyframe stages and a global UCB1 bandit selecting Creative Strategy × Narrative Mode × Aesthetic Archetype across iterations. It ships with GenAd-Bench, a 400-scenario fictional-brand benchmark, and reports 81.4 average MLLM-as-judge vs. 65.0 for Wan 2.6 and 63.6 for Veo 3.1. The interesting bit for Luma is the hierarchical-parameterization recipe: instead of tuning per-agent prompts, you sample a few categorical “axes of taste” and centrally inject them into every sub-agent’s system prompt.

Key claims

Co-Director formalizes generative video storytelling as a black-box global optimization, with UCB1 over a discrete 3×3×4 = 36-arm action space (Creative Strategy × Narrative Mode × Aesthetic Archetype) [§3.3].
Hierarchical parameterization centrally synthesizes the sampled creative direction into agent-specific instructions and injects them into every sub-agent’s system prompt, rather than letting each agent interpret the parameters independently [§3.3].
Local self-refinement is applied to the storyline (LLM scores hook / cohesion / product integration / engagement / adherence) and to the keyframe sequence (MLLM evaluates jointly for sequence-level character/product/environment consistency and selectively regenerates flagged frames) [§3.2].
The MAB uses a factored reward — strategic-efficacy scores per axis are decoupled from the holistic video score, so the bandit can do credit assignment per-arm rather than collapsing into a single scalar [§3.3, §5.3].
An LLM-driven warm start initializes arm expected values from the product domain and demographic before any pipeline execution, biasing early exploration toward theoretically sound configurations [§3.3].
GenAd-Bench: 400 paired scenarios across 50 fictional brands × 4 products × 2 contrasting demographics (stereotypical / unconventional); fictional entities are used explicitly to prevent memorization-based shortcuts [§4].
MLLM-as-judge rubric scores four dimensions on [0,100]: Visual Asset Fidelity, Demographic Alignment, Marketing Appeal, Visual Quality, anchored on the AIDA hierarchy [§4].
Co-Director (T=4) reaches average 81.4 vs. 75.7 for a Random-Search baseline at the same compute budget, 68.5 for a base agentic pipeline (T=1), 65.0 for Wan 2.6, 63.6 for Veo 3.1, and 62.2/65.3 for AniMaker/MovieAgent using the same underlying models [Table 1].
Human MOS (5 raters, 1–5 scale): Co-Director 3.96 vs. Veo 3.1 3.71, Kling 3.0 Omni 3.59, Wan 2.6 3.48; SOTA across all four individual dimensions [Table 3].
MLLM-as-judge metric validation: Pearson and Spearman correlation with human MOS approach the human–human ceiling on DA and MA, but VQ shows lower correlation because the MLLM penalizes brief structural artifacts more strictly than humans watching real-time playback [§5.2, Table 2].
Ablation: collapsing the factored MLLM reward into a single scalar (rather than per-arm strategic-efficacy scores) drops VAF by 5.0 points, validating the factored reward design [§5.3].
Ablation: removing the warm start increases early-iteration variance; warm-start + MAB beats Random Search in cumulative-best performance within the first 1–4 iterations [§5.3, Fig. 2].
Base agentic pipeline at T=1 (no MAB, no refinement) already beats AniMaker (62.2) and MovieAgent (65.3) at 68.5 average, attributed to the core hierarchical orchestration [Table 1].

Method

The system is a four-tier hierarchical multi-agent pipeline. The Orchestrator parses the six-point user prompt into structured constraints, ingests reference visuals (logos, products) with captions and semantic roles, and runs a global outer loop for T iterations. At iteration t, it asks a UCB1 bandit for a creative direction $a_t = (s, n, \alpha)$ — Creative Strategy ∈ {Informational, Transformational, Comparative}, Narrative Mode ∈ {Analytical, Vignette, Narrative Drama}, Aesthetic Archetype ∈ {Clarity/Energy, Cinematic Premium, Minimalist Focus, Kinetic Grit}. The orchestrator then synthesizes $a_t$ into agent-specific text and writes it into placeholders in every sub-agent’s system prompt — sub-agents never see the raw categorical labels, only their domain-relevant projection.

Pre-Production runs four sequential modules: Creative Brief (cultural/environmental enrichment), Storyline (scene-by-scene script), Visual Assets (synthesizing characters/props/wardrobes for any entity not in the user-supplied references), and Storyboard (per-scene semantic descriptors, camera kinematics, temporal constraints, entity-presence flags, plus global audio directives). A local LLM-based self-refinement loop scores and rewrites the storyline before it expands into the storyboard.

Production has three sub-agents: Keyframe (text-to-image conditioned on the creative direction; outputs are jointly evaluated by an MLLM for sequence-level character/product/environment consistency, and only the flagged frames are regenerated); Video (image-to-video with the keyframe as the first frame, parameterized per-scene by camera-kinematics descriptors); Audio (global voiceover + background music conditioned on $\alpha$ , with TTS + text-to-audio mixed to a single track).

Post-Production concatenates clips per storyboard timing and multiplexes the audio. The final video is then scored by an MLLM-as-judge along four rubric dimensions plus a factored set of per-axis “strategic efficacy” scores; the holistic score selects the best video across T executions, while the per-axis scores update the bandit’s arm expected values via UCB1. The reference implementation uses Gemini 3 Pro for LLM/MLLM roles, Nano Banana Pro for keyframes, Veo 3.1 for I2V, Gemini 2.5 Pro TTS for voiceover, and Lyria 2 for music; T=4 outer iterations; up to 3 local-refinement retries; 4 shots × 3s = 12s output.

Results

GenAd-Bench main table. Co-Director (T=4): VAF 82.1 / DA 91.4 / MA 82.0 / VQ 70.2 / Avg 81.4. Best monolithic: Wan 2.6 65.0; best prior agentic: MovieAgent 65.3; base Co-Director pipeline at T=1: 68.5; random-search Co-Director at T=4: 75.7 [Table 1].
Compute-matched comparison. The 5.7-point gap from Random Search (75.7) to Co-Director’s MAB (81.4) at the same T=4 budget isolates the value of the bandit + warm start over undirected sampling [Table 1, §5.1].
Human MOS. 5 raters × 5-point Likert, 50-scenario subset. Co-Director 3.96 > Veo 3.1 3.71 > Kling 3.0 Omni 3.59 > Wan 2.6 3.48 > MovieAgent 3.22 > AniMaker 3.07. Co-Director is SOTA on each of VAF/DA/MA/VQ individually [Table 3].
MLLM-as-judge validation. MLLM–human Pearson on DA: 0.647 vs. human–human ceiling 0.751; on MA: 0.570 vs. 0.737; on VAF: 0.601 vs. 0.733; VQ shows the largest gap (0.345 vs. 0.705) because the MLLM analytically penalizes brief temporal artifacts that humans don’t notice during playback [Table 2, §5.2].
Ablation — factored reward. Replacing the per-axis strategic-efficacy reward with a single scalar (the holistic video score) drops VAF by 5.0 points [§5.3].
Ablation — warm start. Removing the LLM-driven warm start increases early-iteration variance; warm-start MAB matches random-search’s eventual ceiling within fewer iterations [§5.3, Fig. 2].

Why it’s interesting

The closest sibling already in the wiki is Vidmento: Creating Video Stories Through Context-Aware Expansion With Generative Video — both papers attack the gap between per-clip generation and story-level authoring, but from opposite ends: Vidmento is a human-in-the-loop HCI system that deliberately stops short of writing prose for the user, while Co-Director is fully autonomous and treats “creative taste” as a discrete optimization variable to be searched over. Read together they bracket the design space. Co-Director’s hierarchical parameterization (centrally synthesize the global state into per-agent prompt placeholders) is a concrete answer to the orchestration question raised in OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning and LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model — those handle composition + reasoning at model level; Co-Director handles it at pipeline level on top of frozen black-box generators. The MAB-with-factored-reward trick is also useful regardless of video: it’s a clean recipe for credit assignment in non-differentiable agentic pipelines, applicable to any setting where you have a small set of categorical “knobs” and an MLLM-as-judge. For Luma’s storyboard / Dream Machine direction work, the most actionable takeaway is the per-axis creative-direction taxonomy (Strategy × Mode × Archetype) — it’s a deliberately small action space that compresses a lot of director intent into something a bandit (or a user-facing dropdown) can actually search.