ViMax: Agentic Video Generation (Director, Screenwriter, Producer, and Video Generator All-in-One)

TL;DR

ViMax is an open-source (MIT) multi-agent framework from HKUDS that orchestrates frozen black-box generators (Gemini chat model, Nano Banana image generator, Veo video generator by default; MiniMax M2.x is also configurable) into an end-to-end story-to-video pipeline. It exposes four entry points — Idea2Video, Novel2Video, Script2Video, AutoCameo — and internally factors the work into Script Understanding → Scene & Shot Planning → Visual Asset Planning → Asset Indexing + Consistency/Continuity → Visual Synthesis & Assembly, with a central orchestrator handling agent scheduling, stage transitions, and retry/fallback. The notable design choices are RAG-style long-script segmentation, intelligent reference-image selection that includes storyboards from the previous timeline (to keep characters/environments stable as videos get longer), parallel image generation with MLLM-based best-first-frame selection, and parallel shot generation within the same camera. It’s a code-level realization of the pipeline-on-top-of-frozen-generators paradigm rather than a model release.

Key claims

Pipeline factors into a 7-layer architecture: Input → Central Orchestration → Script Understanding + Scene/Shot Planning → Visual Asset Planning → Asset Indexing + Consistency/Continuity → Visual Synthesis & Assembly → Output [Architecture diagram in README].
Long-script handling uses RAG-based segmentation to split novel-length input into a multi-scene script while preserving plot beats and dialogue [README §Intelligent Long Script Generation].
Reference-image selection for the first frame of the current shot conditions on storyboards from the previous timeline, framed as the recipe for keeping multi-character / multi-environment scenes consistent as total length grows [README §Intelligent Reference Images Selection].
Image-generation prompts are auto-derived from the selected reference image plus the prior timeline’s spatial layout — character–environment positioning is arranged programmatically rather than re-asked of the LLM each shot [README §Automated Images Generation].
Consistency-check imitates a human creator workflow: generate multiple first-frame candidates in parallel, then use MLLM/VLM to pick the best consistent one [README §Automated Image Generation Consistency Check].
Sequential shots sharing the same camera are generated in parallel, framed as the production-efficiency primitive [README §High-efficiency Parallel Shot Generation].
Model providers are pluggable via OpenAI-compatible APIs; default config uses Gemini 2.5 Flash Lite + Nano Banana + Veo; MiniMax-M2.7 (1M ctx) and M2.5 (204K ctx) are first-class chat-model alternatives [README §Quick Start, §Using MiniMax as Chat Model Provider].
Four user-facing entry points: Idea2Video (concept → finished video), Novel2Video (book → episodic content), Script2Video (screenplay → video), AutoCameo (user photo → character in arbitrary scripts) [README §Key Features].

Method

The README describes a hierarchical multi-agent orchestration over frozen black-box generators rather than a trained model. The Central Orchestration layer handles agent scheduling, stage transitions, and retry/fallback for the four entry-point modes. Pre-production proceeds Script Understanding → Scene & Shot Planning → Visual Asset Planning, producing a storyboard with per-scene character/environment slots and a shot list. Asset Indexing maintains a catalog of frames and reference images (with embeddings for retrieval), feeding Consistency & Continuity, which does character/environment tracking, reference matching, and temporal-coherence checks. Visual Synthesis & Assembly then runs image generation (with parallel candidates), MLLM-based best-frame selection, first/last-frame-to-video generation, and timeline cut/assembly. The user supplies a single YAML config wiring a chat model, an image generator, and a video generator, plus a working directory; the same config shape supports OpenRouter, Google AI Studio, and MiniMax endpoints.

Results

No quantitative benchmark is reported in the README. The repo ships demo videos generated end-to-end from idea/script input, an MIT license, and a Python 3.12 + uv install path. Coming-soon items include a Dev-mode branch, AutoCameo integration, and explicit shot-planning module — the current release has the Idea2Video and Script2Video flows wired up.

Why it’s interesting

This is the open-source code-level counterpart of the multi-agent video pipelines that have been showing up as research papers — most directly Co-Director: Agentic Generative Video Storytelling (Google’s UCB1-driven autonomous advertising-video pipeline, also on top of Gemini + Nano Banana + Veo) and the storytelling-side authoring tool Vidmento: Creating Video Stories Through Context-Aware Expansion With Generative Video. Compared to Co-Director it is less ambitious as a research artifact (no bandit, no benchmark, no MLLM-as-judge rubric — just a heuristic pipeline) but more useful as a substrate: Co-Director’s recipes (factored MAB reward, hierarchical parameterization of creative direction, MLLM-jury keyframe consistency) would slot directly into ViMax’s orchestrator + consistency layers. Compared to Vidmento it gives up the HCI surface entirely and runs fully autonomous; its working-directory artifact log is the natural hook for adding the kind of intermediate-state editing Vidmento exposes. For Luma’s Dream Machine direction work, the most actionable bit is the timeline-aware reference-image retrieval — building the long-form character/environment cache as a retrieval problem keyed on previously generated frames, rather than re-prompting from scratch each shot.