CinemaCLIP: A hybrid CLIP model and taxonomy for the visual language of cinema

TL;DR

CinemaCLIP is a MobileCLIP-S1 fine-tune from OZU Technology that pairs CLIP-style zero-shot inference with 23 specialised classifier heads spanning a co-designed taxonomy of cinematographic concepts (color, shot, lighting, time of day). It is trained with multi-task contrastive supervision over 8 captioning tasks plus the 23 classification heads, with labels generated automatically by domain-specialist teacher classifiers OZU developed over the past several years (750k human-labelled samples + 750k teacher-labelled DataComp samples, kept in a 50:50 ratio to preserve generalist knowledge). On a 22-category cinema benchmark the classifier reaches 87.7% mean accuracy versus 55-58% for leading 4B VLMs (Qwen3.5-4B, Gemma4-4B, InternVL3.5-4B, Molmo2-4B) and 36-46% for larger CLIP models up to 28× CinemaCLIP’s parameter count. Ships in Torch / CoreML / ONNX for Apple Neural Engine and cross-platform deployment.

Key claims

Model scale alone does not close the cinematic-understanding gap: 4B VLMs and ViT-H/PE-bigG-scale CLIP models (up to 28× larger than MobileCLIP-S1) underperform CinemaCLIP across all 22 categories, with the gap attributed to dataset composition rather than architecture [§Motivation, §Per-category results].
Replacing one long caption with multiple parallel, mutually-exclusive single-task captions yields ~14% improvement over the single-caption formulation; the model learns multiple structured visual concepts concurrently without one collapsing into another [§Multi-task formulation, Ablations Table].
The 23 task-specific teacher classifiers (developed independently over several years for shot type, lighting, composition, etc.) provide auto-labels for 6/8 captioning tasks; a 0.85 confidence threshold gives the best balance (~+10% on cinematic 0-shot over a 0.0 threshold) [Ablations: confidence threshold].
Alpha-blending the fine-tuned weights with the pretrained MobileCLIP-S1 at α=0.75 (75% fine-tuned, 25% pretrained) retains 14% cinematic-accuracy gain at a ~1-2% cost on cinematic tasks but recovers ~+7% on ImageNet 0-shot, mitigating catastrophic forgetting [§Training, Ablations: alpha blending].
50:50 mixing of cinematic data with general-purpose DataComp data (both auto-labelled with cinematic tags by the teacher models) is critical for retaining generalist knowledge; no image-side augmentations were used because they destroy salient cinematographic information [§Training data, §Augmentations].
On the headline benchmark (22-category mean, classifier mode): CinemaCLIP 87.7%, Qwen3.5-4B 57.6%, Gemma4-4B 56.7%, InternVL3.5-4B 55.3%, Molmo2-4B 55.3%, DFN ViT-H-14 45.9%, MetaCLIP PE-bigG 45.2%, OpenAI ViT-L-14 44.8%, base MobileCLIP-S1 44.2%, SigLIP2 ViT-gopt 36.5% [Per-category results table].
Training was hardware-constrained to 3× RTX 3090s (24GB each) — effective batch size 1,152 images / 9,216 captions — but the recipe (multi-task contrastive + teacher-classifier auto-labels + alpha-blending + data mixing) is reported as generic and adaptable to other expert domains [§Conclusion, §Hyperparameters].

Method

The training recipe is built in five pieces. (1) An expert-designed taxonomy of 22 cinematographic concept families (color contrast/key/saturation/theory/tones; lighting cast/contrast/edge/silhouette; shot angle/composition/dutch-angle/focus/framing/height/lens-size/location/symmetry/time-of-day/type, plus binary OTS and Crowd heads). (2) Teacher-model auto-labelling: 23 in-house specialist classifiers (one per concept family) emit per-image predictions with a 0.85 confidence floor. (3) Multi-task contrastive training: instead of a single long caption per image, 8 parallel captions — 6 cinematic groups (e.g. “framing” caption “medium lens, medium shot, levelled camera angle”; “lighting” caption “soft lighting, neutral contrast lighting, sidelit lighting”) + 2 general tasks (synthetic generic caption + human tags) — are matched contrastively against the same image embedding. Each caption captures one disentangled aspect, eliminating the multi-concept-collapse problem that long captions cause. (4) CLIP fine-tuning on MobileCLIP-S1 with 6 vision-encoder layers and 3 text-encoder layers unfrozen, LR 2e-5 (encoders) / 2e-3 (classifier heads), 3 epochs over a 1.5M sample mix. (5) Alpha-blend with the pretrained checkpoint at α=0.75 to retain general knowledge, then bolt on the 23 classifier heads for downstream supervised inference (single-label softmax / multi-label sigmoid / binary). Ships as Torch (model.safetensors), CoreML (Apple Neural Engine), and ONNX (fp32 + fp16) artifacts.

Results

22-category mean accuracy: CinemaCLIP classifier 87.7% / 0-shot 83.2%, best non-CinemaCLIP score 57.6% (Qwen3.5-4B), best CLIP-family score 45.9% (DFN ViT-H-14) [Per-category results table].
CinemaCLIP either tops or ties every one of 22 categories against every other evaluated model (3 VLMs + 1 Molmo + 7 CLIP variants); only Shot Symmetry is close (CinemaCLIP 91.0 classifier / 87.8 0-shot vs Qwen3.5-4B 90.2 and SigLIP2 ViT-gopt 82.4) [Per-category table].
Largest absolute gaps over the strongest non-CinemaCLIP baseline appear on Shot Composition (97.0 vs 60.5 — Molmo2-4B), Shot Framing (82.3 vs 40.1 — InternVL3.5-4B), Lighting Edge (93.6 vs 40.4 — Molmo2-4B), Lighting Contrast (93.2 vs 46.1 — Molmo2-4B), and Color Contrast (87.4 vs 57.7 — SigLIP2 SO400M) [Per-category table].
Multi-task ablation: 2 captions → 72.4% / 3 → 72.8% / 8 → 82.2% / 22 → 82.7% cinematic 0-shot, with ImageNet 0-shot dropping monotonically from 65.6% (2 captions) to 60.5% (22 captions) [Ablations: no. of captions].
Alpha-blend ablation: α=1.0 (full fine-tune) gives 88.5% classifier on cinematic but 57% ImageNet top-1; α=0.75 gives 87.5% classifier / 82.2% 0-shot cinematic / 63.4% ImageNet — the ~+6.4% ImageNet recovery costs ~1% classifier accuracy [Ablations: alpha blending].
Confidence-threshold ablation: 0.0 → 74.9% / 0.5 → 79.5% / 0.7 → 80.2% / 0.85 → 82.2% cinematic 0-shot — the highest-confidence floor yields the cleanest captions and the strongest model [Ablations: confidence threshold].

Why it’s interesting

CinemaCLIP is the closest direct sibling to CHAI: Building a Precise Video Language with Human–AI Oversight on the wiki: both papers build an expert-designed cinematographic taxonomy and use it as supervision for a vision-language backbone, but they sit at opposite ends of the model-size / output-shape spectrum. CHAI fine-tunes Qwen3-VL-8B on long structured captions co-designed with 100+ creators and runs Critique-based Human–AI oversight as the data-quality gate; CinemaCLIP fine-tunes a tiny MobileCLIP-S1 with a teacher-classifier auto-labelling stack and uses confidence thresholding as the gate. The complementary recipes argue the same thesis from opposite directions — “internet-scale generalist captions don’t carry the visual grammar of cinema, so you have to inject domain structure on the supervision side, not the architecture side.” For Synthetic Training Data the relevant detail is the 23 pre-existing teacher classifiers acting as the verification-and-labelling stack: this is the same teacher-student pipeline shape as Action100M (Action100M: A Large-scale Video Action Dataset) and Video-Thinker (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning), but specialised for frame-level cinematic attributes rather than action segmentation or CoT traces, and notable for using a non-VLM teacher stack — the gates here are bespoke supervised classifiers, not Self-Refined LLM judges. The on-device-deployment angle (CoreML / Apple Neural Engine, faster than real-time on commodity hardware) is the production-ready half — CinemaCLIP can be the labelling stack for a Luma-scale video archive without the per-frame VLM cost.