CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation

TL;DR

CineTechBench is an expert-annotated benchmark for evaluating both MLLM understanding and video-generator reproduction of cinematographic techniques across seven core dimensions: shot scale, angle, composition, camera movement, lighting, color, and focal length. It comprises 600+ curated stills from IMDb Top-250 films and 120+ short clips from the Movieclips YouTube channel, each manually labeled by trained annotators (with professional cinematographer review for ambiguous cases) — yielding 610 image QA pairs, 128 video QA pairs, and paired descriptive captions. Understanding is scored via accuracy and a CAPability precision/recall/F1 rubric; generation is scored via camera-trajectory error (RotError, TransError, CamMC via MonST3R) and CLIP-IS visual consistency across first-frame and first-last-frame conditioning. Across 15+ MLLMs and 7 I2V models, headline gaps are that fine-grained cinematography understanding remains far below film-craft standards (best overall ~70% on stills, ~57% on camera-movement videos) and that I2V models struggle severely on high-angular-velocity rotational camera movements.

Key claims

Taxonomy covers seven cinematography dimensions — shot scale, angle, composition, camera movement, lighting, color, focal length — and camera-movement follows CameraBench’s five-type primitive schema (translation, rotation, zoom, static, combined) [§3.1].
Dataset consists of 600+ manually annotated stills from IMDb Top-250 films plus 120+ Movieclips YouTube clips with clear intentional camera movement, expanded via GPT-4o synthesis + manual review to 610 image QAs, 128 video QAs, 100 image descriptions (avg 176 words), 128 video descriptions (avg 168 words) [§3.2].
GPT-4o leads image-QA overall at 70.16% and Gemini-2.5-Pro second at 69.67%; best open-source is InternVL3-8B at 55.25% — a ~15-point gap between commercial and open-source MLLMs on static cinematographic understanding [Table 1].
Focal length is the hardest static dimension: only Doubao-1.5-vision-pro (commercial) reaches 61.67% and every other model scores 20–43% [Table 1].
Camera-movement video-QA: Gemini-2.5-Pro leads at 56.69% overall; Qwen2.5-VL-7B and Qwen2.5-Omni-7B rank first/second among open-source; camera rotation is consistently the hardest sub-category across all models [Table 2].
Several open-source MLLMs (e.g., LLaVA-NeXT-Video) badly fail at recognizing fixed/static shots — an inverse failure where models over-predict motion [Table 2].
On description generation, Gemini-2.5-Pro leads on CAPability AP/AR/F1 and InternVL3-8B is the strongest open-source, surpassing some commercial models including Gemini-2.0-Flash [Table 3].
On camera-movement I2V generation, Kling v1.6 (first+last-frame condition) wins commercial RotError/TransError; Wan2.1-FLF-14B-720P and FramePack-FLF2V lead open-source when both endpoints are supplied — models conditioned on the first frame only (HunyuanVideo-I2V, SkyReels-V2, FramePack-I2V) perform substantially worse [Table 4].
I2V rotation error scales with the original clip’s angular velocity — models fail hardest on intense-rotation shots typical of fight scenes [Fig. 6].
Qualitative failures include reversed roll direction (Kling produces correct rotation sense but clockwise vs. ground-truth counter-clock; only Gen4turbo gets both magnitude and direction right in the featured example) [Fig. 5].

Method

CineTechBench is built by (1) constructing a hierarchical cinematography taxonomy from film-education keyword sources (Videomaker, StudioBinder, No Film School) and refining it with GPT-4o + professional cinematographer feedback into 7 dimensions; (2) manually sourcing 600+ high-resolution stills from IMDb Top-250 films and 120+ camera-movement clips from the Movieclips YouTube channel; (3) training annotators on 5 canonical examples per category from professional cinematography websites, then having them label each sample across all applicable dimensions with escalation to a professional cinematographer for ambiguous cases; (4) synthesizing QA pairs and long-form descriptions with GPT-4o under taxonomy + label constraints and reviewing every output by hand. Understanding evaluation reports accuracy (breakdown per dimension) plus n-gram metrics (BLEU-4, METEOR, ROUGE-L) and the CAPability rubric (hit rate, average precision, average recall, F1) for descriptions. Generation evaluation reconstructs the camera trajectory of the reference clip and each generated clip via MonST3R, then reports rotation error (relative and absolute), translation error, CamMC, and a CLIP-based frame-similarity score.

Results

MLLM understanding (image QA, Table 1): GPT-4o 70.16%, Gemini-2.5-Pro 69.67%, Gemini-2.0-Flash 59.34%, GLM-4V-Plus 60.00%, Qwen-VL-Plus 61.36%, Doubao-1.5-vision-pro 56.07%. Open-source best InternVL3-8B 55.25%; Qwen2.5-VL-7B 50.66%; Qwen2.5-Omni-7B 54.75%. Focal-length accuracy is the persistent trough (20–43%) except Doubao at 61.67%.
MLLM video-QA on camera movement (Table 2): Gemini-2.5-Pro 56.69% overall; open-source best Qwen2.5-VL-7B 50.78%. Rotation is the hardest sub-category across the board. Several open-source models score <30% on static shots — they over-predict motion.
Description generation (Table 3): Gemini-2.5-Pro leads CAPability F1 at 52.27; GPT-4o 49.08; InternVL3-8B open-source leader at 47.86, beating Gemini-2.0-Flash 44.43.
Camera-movement I2V generation (Table 4): commercial Kling v1.6 (FLT) achieves best RotError-relative 21.68 and TransError-relative 62.57; open-source best FramePack-FLF2V 23.88 RotError-rel / 71.98 TransError-rel. Pure I2V (first-frame-only) models trail substantially on translation error (HunyuanVideo-I2V 91.87, SkyReels-V2 100.96 in absolute translation error).
The paper shows via Fig. 6 stratification that translation error rises with clip translation speed and rotation error rises sharply with angular velocity, isolating high-motion cinematography as the concrete generation bottleneck.

Why it’s interesting

CineTechBench opens the dual understanding-and-generation axis of film-craft evaluation, sitting between two filed benchmarks in the Video Generation Benchmarks cluster: FilmBench: A Film-Grade Benchmark for Cinematic Video Generation (film-graded T2V/R2V generation, 3×12×35 sub-metrics co-designed with Beijing Film Academy) and Physion-Arc 1.0: Benchmarking Video Agents on Minute-Long Video Generation (agentic minute-long screenplay evaluation). CineTechBench is narrower and more diagnostic: rather than scoring narrative quality on multi-shot clips, it fixes seven canonical cinematography dimensions and asks whether MLLMs can recognize them and whether I2V models can reproduce the camera motion specifically. It complements CameraBench: Towards Understanding Camera Motions in Any Video (CameraBench, whose 5-primitive taxonomy CineTechBench reuses for the camera-movement axis) by adding six static dimensions (scale, angle, composition, color, lighting, focal length) that CameraBench does not touch, and by adding the generation side that CameraBench is not built for. It also complements CinemaCLIP: A hybrid CLIP model and taxonomy for the visual language of cinema — CinemaCLIP classifies existing films; CineTechBench tests both understanding and faithful generation. For Luma the concrete signal is on the generation side: even the best I2V systems fail on rotation-heavy shots and misidentify roll direction, which is exactly the kind of camera-craft failure that shows up in cinema-quality prompts.