CHAI: Building a Precise Video Language with Human–AI Oversight

TL;DR

CHAI is a CVPR’26 Highlight (top 3%) recipe for teaching video-language models to describe video like a professional cinematographer. The authors co-design — with 100+ content creators over a year — a structured 5-aspect specification (subject, scene, motion, spatial, camera) grounded in 200+ visual primitives, and a Critique-based Human–AI oversight (CHAI) pipeline that has an LLM draft fluent captions and humans critique the visual errors. Post-training an 8B Qwen3-VL on the resulting (pre-caption, critique, post-caption) triplets yields a model that surpasses GPT-5 and Gemini-3.1-Pro on captioning, reward modeling, and critique generation, and fine-tuning Wan2.2 to follow the resulting 400-word cinematic prompts unlocks precise control over dolly zooms, rack focus, speed ramps, Dutch angles, and POVs.

Key claims

A structured specification with 200+ co-designed visual primitives across subject, scene, motion, spatial framing, and camera dynamics gives video captioning a shared vocabulary that crowdsourced annotators do not have — even trained crowdworkers confuse dolly-in with zoom-in, mis-call full shots “close-ups,” and describe fisheye distortion as “circular buildings” [Crowdsourcing analysis].
The Critique-based Human–AI oversight loop produces (pre-caption, critique, post-caption) triplets that unlock three downstream signals at once — SFT/DPO captioning, reward modeling, and critique generation — from a single annotation pass [§CHAI].
Critique quality (precision, recall, constructiveness) is the bottleneck, not just critique presence: ablating any one quality dimension degrades downstream task performance, and prior critique datasets (OpenAI GDC, MM-RLHF) contain 50%+ non-constructive critiques (“this is wrong” with no fix) [Critique-quality Table].
An 8B Qwen3-VL post-trained on CHAI triplets surpasses Gemini-2.5 baselines on captioning (18.2 vs 12.7), reward modeling (89.8 vs 62.0), and critique generation (41.7 vs 26.2) [Critique-quality Table; quality-check row].
The same 8B model is reported to surpass GPT-5 and Gemini-3.1-Pro on professional video captioning, reward modeling, and critique generation [News, §Results].
Fine-tuning Wan2.2 on a re-captioned large-scale professional video corpus (films, commercials, music videos, games) lets the generator follow 400-word cinematic prompts and produce techniques current generators struggle with — dolly zoom, rack focus, speed ramps, Dutch angles, POVs [§Video generation].
Surveying 8 video–text datasets from 2016–2025 shows the same recurring failure modes — imprecise terminology, missing details, subjective filler descriptions, hallucinations — attributed to a lack of specification (no shared vocabulary) and a lack of oversight (no critique step) [§Dataset audit].

Method

CHAI is built in four pieces. (1) A structured specification in five aspects — subject, scene, motion, spatial, camera — grounded in 200+ visual primitives co-designed with professional cinematographers; this is what the caption is of. (2) A scalable human–AI oversight loop: an MLLM drafts a fluent caption (pre-caption), a human writes a free-text critique pointing out visual errors, and a model rewrites the caption (post-caption). The triplet captures the visual-error signal in machine-readable form. (3) Post-training recipes turn the triplet into three trainable objectives — SFT on the post-caption, DPO using the (pre, post) pair as the (loser, winner) ordering, reward-model training on the critique, and a critique-generation head. Training is done on Qwen3-VL-8B. (4) Downstream re-captioning: the post-trained model relabels a large-scale professional-video corpus, and Wan2.2 is fine-tuned to follow these dense, 400-word cinematic prompts.

Results

8B Qwen3-VL trained on CHAI triplets reaches 18.2 (caption) / 89.8 (reward) / 41.7 (critique) versus Gemini-2.5 at 12.7 / 62.0 / 26.2 on the paper’s evaluation — and is reported to beat GPT-5 and Gemini-3.1-Pro on the same three tasks [Critique-quality Table].
Removing critique precision drops the three metrics to 12.1 / 47.1 / 21.9; removing recall to 12.5 / 56.6 / 28.7; removing constructiveness to 13.4 / 67.2 / 32.9 — each quality dimension is independently load-bearing [Critique-quality Table].
Removing the quality-check filter from CHAI’s own pipeline drops the three metrics to 14.8 / 73.1 / 35.7 — the quality gate accounts for roughly the difference between “ours” and the all-quality-ablated baselines [Critique-quality Table].
Fine-tuned Wan2.2 follows cinematic prompts up to 400 words and produces dolly zooms, rack focus, speed ramps, Dutch angles, and POVs that the paper claims current generators fail to handle [§Video generation, project page demos].

Why it’s interesting

CHAI sits at the intersection of three threads the wiki is already tracking. As a VLM-as-Evaluator datapoint it is the first filed instance to train an 8B open VLM that explicitly beats a frontier closed judge (Gemini-3.1-Pro, GPT-5) on reward modeling — complementing Unified Personalized Reward Model for Vision Generation‘s context-adaptive Flex judge by showing the same architectural slot can be filled by a smaller, domain-specialized open model when paired with high-quality critique supervision. As a Camera-Controlled Video Diffusion datapoint it is the text-side complement to the Kling-MotionControl Technical Report / Wan-Animate: Unified Character Animation and Replacement with Holistic Replication / VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control adapter recipes — instead of adding a geometric trajectory channel, CHAI gives the Wan backbone a precise linguistic control surface and re-captions the training corpus to match. As a Synthetic Training Data datapoint it argues the verification gate (here, structured human critiques) is what separates working synthetic data from broken synthetic data, and offers a concrete head-to-head: ablating any single critique-quality dimension collapses downstream performance.