CameraBench: Towards Understanding Camera Motions in Any Video

TL;DR

CameraBench is a NeurIPS 2025 Datasets-and-Benchmarks paper introducing a cinematographer-co-designed taxonomy of camera motion primitives and a ~3000-video, expert-annotated dataset with ~150K binary labels and free-form captions. It evaluates Structure-from-Motion (SfM) systems and Video-Language Models (VLMs) on camera-motion understanding, showing SfM struggles with semantic primitives that depend on scene content while VLMs struggle with geometric primitives that require trajectory estimation. The authors then fine-tune a generative VLM (Qwen-2.5-VL) on CameraBench and report the best-of-both-worlds, with applications to motion-augmented captioning, video QA, and video-text retrieval. The taxonomy and label-then-caption annotation protocol are the foundation that the later CHAI work generalizes from camera-only to a 5-aspect specification.

Key claims

A precise taxonomy of camera motion requires three reference frames (object-, ground-, and camera-centric) and primitive types covering translation, rotation, intrinsic changes (zoom), circular motion (arcing), steadiness, and tracking shots [Taxonomy section].
Domain expertise materially raises annotation accuracy: a large-scale human study with 100+ participants shows professional cinematographers consistently outperform non-experts; novices commonly confuse zoom-in (intrinsics change) with translating forward (extrinsics change), but can be trained with tutorials to differentiate [Human study].
A hybrid label-then-caption annotation framework — annotators label clear primitives directly, mark ambiguous aspects “I am not sure,” then write a free-text description of the motion (and why it moves) — yields ~150K binary labels plus captions across ~3000 videos covering diverse types, genres, POVs, capture devices, and post-production effects [Dataset section].
SfM systems (COLMAP, MegaSAM) fail on motions that require scene-content semantics (e.g. lead-tracking with unchanged subject framing and weak background texture) and on low-parallax scenes (e.g. roll-clockwise), where MegaSAM mis-detects translation and COLMAP either crashes or outputs random trajectories [Findings, Fig. left/right].
VLMs fail on geometric primitives that require precise trajectory estimation, while SfMs fail on semantic ones — establishing a complementary-failure pattern that motivates a hybrid system [Findings].
Fine-tuning a generative VLM (Qwen-2.5-VL) on CameraBench produces a model whose camera-motion captions are reported as more accurate than GPT-4o and Gemini-2.5-Pro [Captioning section].
Leaderboards are provided for both binary classification of motion primitives and video QA, with the authors inviting method submissions [Leaderboard sections].

Method

CameraBench is built around a taxonomy + annotation pipeline. The taxonomy is iteratively co-designed with vision researchers and professional cinematographers: it organizes camera motion into three reference frames and the primitive families above, including the explicit distinction between intrinsic (zoom) and extrinsic (translation) changes. Annotation uses a label-then-caption hybrid form. For each ~3000-video clip, annotators (i) decide whether camera motion is clear and consistent, (ii) classify confident primitives directly (e.g. pan-left, no-tilt), (iii) mark ambiguous aspects “I am not sure,” and (iv) write a free-text caption that may include reasoning about why the camera moves (follow a subject, enhance immersion, etc.). A multi-stage training program with detailed guidelines is run to improve accuracy of both novice and expert annotators, validated by a 100+-participant human study.

The dataset is then used as a benchmark for two classes of methods: SfM systems for trajectory geometry, and VLMs for semantic primitives. On top of evaluation, the authors fine-tune Qwen-2.5-VL on CameraBench triplets to produce a model that handles both axes.

Results

~150K binary motion-primitive labels and free-form captions over ~3000 videos spanning nature, films, games, 2D/3D, real/synthetic, GoPro, drone, and other capture modalities [Dataset section].
Human study quantifies a clear novice → expert gap, with cinematographer accuracy materially exceeding non-experts; tutorial-based training closes part of the gap on specific confusions (zoom vs translate) [Human study].
SfM failure cases include lead-tracking shots with unchanged framing/weak background texture (MegaSAM fails to detect translation; COLMAP crashes) and roll-clockwise shots in low-parallax scenes (both fail to converge) [Fig. left/right].
Binary classification and video QA leaderboards rank SfM, VLM, and hybrid systems separately; the fine-tuned Qwen-2.5-VL is the top entry on the captioning side, ahead of GPT-4o and Gemini-2.5-Pro [Leaderboard, Captioning section].

Why it’s interesting

CameraBench is the direct precursor to CHAI: Building a Precise Video Language with Human–AI Oversight: the same lab (Lin, Cen, Mitra, Du, Ramanan, +) takes the camera-only taxonomy here and generalizes it into the 5-aspect (subject, scene, motion, spatial, camera) cinematographer-grounded specification in CHAI, with the same expert-annotator / label-then-critique methodology. The two-paper arc is a worked example of how a careful benchmark + taxonomy turns into a post-training recipe one year later, and connects this work to Camera-Controlled Video Diffusion as the evaluator side: the cluster of Wan-adapter generators is currently evaluated mostly on VBench / trajectory error, but CameraBench’s semantic-primitive labels are the natural complement for testing whether ReCamMaster / VerseCrafter / Kling-MotionControl outputs would survive cinematographer scrutiny. It also fits VLM-as-Evaluator as a benchmark-time data point — here SfM is the alternative, and the paper’s complementary-failure finding (SfM weak on semantics, VLMs weak on geometry) is the clearest filed case for hybrid scoring on motion-bearing video.