OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer

TL;DR

OmniShotCut reformulates Shot Boundary Detection (SBD) as structured relational prediction: rather than emitting only temporal shot ranges, the model also predicts an intra-shot label (vanilla / dissolve / wipe / push / slide / zoom / fade / doorway) and an inter-shot label (transition / hard cut / sudden jump) per shot, so downstream curation can filter on type. The architecture is a DETR-style shot-query Transformer over 3D-positioned video tokens with a discrete frame-index classification head for boundaries (no GIoU regression, no Hungarian matching). Training is entirely synthetic — a programmatic pipeline composes ~9 transition families × 30 subtypes from a DINOv3-clustered video pool (300K synthetic videos) — and the released benchmark (OmniShotCutBench, 114 modern in-the-wild videos) reports a transition IoU of 0.632 vs. 0.18–0.25 for PySceneDetect / TransNetV2 / AutoShot.

Key claims

SBD should output structured relations alongside ranges: intra-shot type (8 classes incl. dissolve/wipe/push/slide/zoom/fade/doorway/vanilla) and inter-shot type (transition / hard cut / sudden jump) [§3.1].
“Sudden jump” — abrupt motion/texture discontinuities from cropping inside a single take — is a downstream-harmful class that prior SBD models systematically miss; OmniShotCut’s inter-shot head explicitly targets it [§1, §3.1].
Synthetic transition data dominates manual labeling because editing-software transitions are forward-generated by definition; reverse-annotating them on real footage cannot match the boundary precision of the forward synthesis [§1, §3.3].
Discrete classification over frame indices beats DETR-style + 1D GIoU regression for boundary localization; adding GIoU loss drops Range F1 from 0.811 to 0.633 [§3.4, Table 2].
DINOv3-embedding hierarchical-K-means clustering (27K clusters) of a 1M-clip pool, plus semantic deduplication at cosine 0.05, supplies semantically-coherent pre/post-transition source pairs that approximate real editing patterns [§3.2, §4.1].
Motion-strength filtering via CoTracker3 (medium-motion 25–60th percentile) is required for the sudden-jump synthesis class — fast motion is indistinguishable from hard cuts, slow motion is imperceptible to humans [§3.2, §3.3].
On OmniShotCutBench: Transition IoU 0.632 (vs. AutoShot 0.252, TransNetV2 0.192, PySceneDetect 0.183), Sudden-Jump Acc. 0.761 (vs. 0.455 / 0.261 / 0.416), Range F1 0.883 (vs. 0.814 / 0.814 / 0.754) [Table 1].
Intra-shot classification accuracy reaches 0.959 and inter-shot 0.836 — capabilities absent from all prior SBD baselines [Table 1].
Training is compact: ResNet18 backbone + 6 encoder / 6 decoder layers, 24 shot queries, 128×96 input, 8× A100 for ~2 days [§4.1].

Method

The model is a DETR-style Transformer adapted for video. A ResNet18 encodes each frame independently; per-frame features are flattened (T·H·W, C) with 3D sinusoidal positional embeddings (t, y, x) following VisTR. The Transformer decoder consumes 24 fixed learnable shot queries, each of which aggregates evidence for one shot slot via self-attention then cross-attention with the encoder tokens. A trainable termination token marks the end of the predicted shot sequence; queries past termination are discarded, so no Hungarian matching is needed.

Each surviving query feeds three heads: (1) a range head that predicts the last frame index of the shot as a discrete classification over T classes (the start is implicit — previous shot’s end + 1, with the first shot starting at frame 0), (2) an intra-relation head classifying the shot’s own type, and (3) an inter-relation head classifying the boundary with the previous shot. Loss is a weighted sum (5:1:1) of the three CE losses with auxiliary supervision at every decoder layer [§3.4, eq. 1–4].

Training data is fully synthetic. A 1M-clip pool is curated from OpenVid / VidGen / Sakuga / GamePhysics by (a) basic resolution/duration/aspect filtering, (b) DINO-cosine continuity check (>0.9 between consecutive sampled frames keeps the clip; <0.9 terminates it), (c) CoTracker3 motion-strength estimation to drop slow-motion clips, (d) DINOv3 ViT-L embedding + 27,000 hierarchical-K-means clusters + 0.05 semantic-dedup. The composition step then stitches 1–N clips per video (Poisson λ, durations ~ truncated Gaussian) with programmatic transitions covering 9 families × 30 subtypes, 80% of within-cluster pairs to preserve semantic continuity, sudden-jumps spliced from the medium-motion subset. 300K synthetic training videos, ≥240 frames each at 24 fps, with light augmentation (subtitles 5%, lighting 7.5%).

Results

OmniShotCutBench is a new 114-video, ~110-minute, 480p/30fps benchmark spanning vlog/anime/movie/concert/documentary/screen-recording/game/sports with multi-annotator labels and per-shot confidence. Headline numbers vs. the strongest prior open SBD models [Table 1]:

Transition IoU: 0.632 (Ours) vs. 0.252 (AutoShot) vs. 0.192 (TransNetV2) vs. 0.183 (PySceneDetect).
Sudden-Jump Accuracy (zero-tolerance): 0.761 vs. 0.455 / 0.261 / 0.416.
Range Precision / Recall / F1: 0.898 / 0.858 / 0.883 vs. AutoShot 0.849 / 0.782 / 0.814.
Intra Acc. 0.959; Inter Acc. 0.836 (no baseline reports these — capability is new).

Ablations [Table 2]: switching the range head from discrete classification to DETR-style 1D+GIoU regression drops Range F1 from 0.811 → 0.633 and Sudden-Jump Acc. from 0.568 → 0.319, confirming the discrete-classification reformulation as the dominant lift. Removing DINO-based source selection (random pairs across clusters) drops Inter Acc. by 3 points and Range F1 by ~2 points. A “short dense hard-cut” augmentation (28 short clips per video) lifts Sudden-Jump Acc. another ~7.5 points over base.

Why it’s interesting

For any video-generation pipeline at Luma scale, SBD is the gate that decides what counts as a clean training clip — and the failure modes OmniShotCut targets (subtle fades misaligned by a frame, sudden jumps from manual mid-shot cuts) are exactly the ones that poison temporal VAEs (the paper cites this as motivation, §2.1.1). The Shot-Query reformulation lets a single forward pass produce both the boundary and the transition-type tag, so the curation policy can be “keep only intra=vanilla, inter=hard-cut” without a second classifier pass. The discrete-frame classification trick (with auxiliary supervision and no Hungarian matching) is also a clean ablation point against the standard DETR template that should generalize to other dense-temporal localization tasks. This complements Summer-22B: A Systematic Approach to Dataset Engineering and Training at Scale for Video Foundation Model (which builds an end-to-end video-foundation-model dataset pipeline but defers shot-cut handling to off-the-shelf tools) and slots into the same lineage as CHAI: Building a Precise Video Language with Human–AI Oversight on video-language preprocessing — except OmniShotCut targets the structural shot graph instead of caption quality. It also extends Synthetic Training Data‘s “forward generation beats reverse annotation” thesis from text/CoT and 4D rendering into a concrete video-editing-effects domain.