MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs

TL;DR

MotionSight is a zero-shot visual-prompting recipe for boosting MLLM understanding of fine-grained motion in video, with two paired tricks: an object-centric visual spotlight (GroundingDINO + SAM2 trajectories darken everything outside a tracked, jitter-smoothed region) for object motion, and synthetic global motion blur (geometric-decay temporal aggregation) for camera motion. The same pipeline is then used to bootstrap MotionVid — a 40K-video / 87K-QA SFT+DPO dataset for fine-grained motion understanding, filtered with VQAScore and human preference. On MotionBench and FAVOR-Bench, applying MotionSight to InternVL3-78B beats prior open-source SOTA (TE Fusion) and is competitive with GLM-4V-Plus-0111; camera-motion accuracy in particular jumps +14.3% on Qwen2.5VL-7B [Tables 2, 3].

Key claims

Naively transferring static-image visual prompts to video hurts fine-grained motion: background blur (the best image-prompt baseline) is the worst video prompt because blurred object boundaries destroy contextual motion cues [§3, Fig. 3].
Decoupling object and camera motion is the key design choice: object motion benefits from a localized spotlight prompt; camera motion benefits from global motion blur (the opposite of localizing) [§3.1].
Object referring via GroundingDINO + SAM2 with periodic re-detection handles object entry/exit and is robust to initial detection errors at the action-inference level [§3.2].
A dynamic temporal aggregator switches between long-window union-of-boxes (low intra-trajectory variance) and short-window localized boxes (high variance), measured by Manhattan distance between bbox centers [§3.3].
Synthetic motion blur uses a geometric-decay temporal kernel (α^k over T preceding frames) and substantially improves camera-motion perception across MLLMs [§3.4, Table 4].
MotionVid is curated from ActivityNet, Kinetics-700, Charades(-Ego), Tarsier2-Recap-585K, OpenVid-1M, and MotionBench-train; VQAScore + human thresholds partition clips into a 35K-video SFT split and a 5K-video DPO split (7K chosen/reject QAs) [§4, Table 1].
MotionSight + Qwen2.5VL-7B gains +3.4% category-avg on MotionBench and +14.3% on camera motion specifically; InternVL3-78B + MotionSight reaches SOTA among open-source models on both MotionBench and FAVOR-Bench [Tables 2, 3].
Visual spotlight beats four alternatives for object motion enhancement on Qwen2.5VL-7B (53.0 OM AVG vs 52.5 crop / 50.6 pose / 50.2 object-motion-blur / 49.3 background-blur) [Table 4].

Method

MotionSight is a training-free pipeline composed of three modules selected by question type. (1) Object Referring: the MLLM first proposes a small set of motion-relevant object categories from sampled frames + query; GroundingDINO localizes them in a keyframe; SAM2 propagates trajectories with periodic re-detection. (2) Action Focusing: per object, a dynamic temporal aggregator merges/stabilizes the trajectory’s bounding boxes into refined spatial regions, with the aggregation window length governed by positional variance along the trajectory. A visual spotlight darkens the background outside these regions while preserving the original object pixels, producing the visually-prompted frame sequence the MLLM is finally asked. (3) Motion Blur: for camera-motion questions, the global frame is replaced by a temporally-weighted aggregation of preceding frames using a geometric-decay kernel, accentuating inter-frame trajectories. The output of the appropriate enhancement is fed back to the original MLLM (Qwen2.5VL-7B or InternVL3-78B in the paper) for the final answer.

Beyond inference, MotionSight is used as an annotation engine to build MotionVid: sourced clips are run through the pipeline; VQAScore + human thresholds gate the resulting (clip, motion-QA) pairs into a high-quality SFT subset (35K / 80K QAs) and a DPO preference subset (5K / 7K chosen-rejected QAs re-annotated with Tarsier2 as baseline and human preference signals).

Results

MotionBench (dev set, 8 categories): Qwen2.5VL-7B 53.0 overall / 48.8 avg → 55.6 / 52.2 with MotionSight; InternVL3-78B 61.5 / 57.6 → 63.0 / 59.3, exceeding TE Fusion (58 / 54, prior CVPR’25 SOTA) and approaching the proprietary GLM-4V-Plus-0111 at 62.8 / 60.3 [Table 2].
Camera Motion specifically: Qwen2.5VL-7B jumps 34.0 → 48.3 (+14.3); InternVL3-78B 55.8 → 58.7 [Table 2].
FAVOR-Bench (6 categories): Qwen2.5VL-7B 42.3 / 41.6 → 45.1 / 44.1 (+2.5 overall, +3.0 avg); InternVL3-78B 52.8 / 52.6 → 53.8 / 53.5 with category gains in Action Sequence (54.7 → 56.2), Holistic Action Classification (58.1 → 58.9), and Multi-Action Detail (57.2 → 58.4) [Table 3].
Ablation on object prompts (Qwen2.5VL-7B, OM AVG): visual spotlight 53.0 > object crop 52.5 > pose estimation 50.6 > object motion blur 50.2 > background blur 49.3 (49.3 is below the no-prompt baseline of 51.7) [Table 4].
MotionVid sits well above prior fine-grained motion corpora in scale: 40K videos / 87K texts with mixed QA + chosen/reject annotation types, versus MotionBench-train at 5K / 5K and FavorBench-train at 17K / 17K [Table 1].

Why it’s interesting

For Luma the dataset side is the bigger lever: MotionVid is one of the few open corpora with both decoupled object/camera motion annotations and DPO preference pairs, which is the right shape for post-training a generator on motion-specific prompt following. The visual-prompting side is a clean contribution but tied to MLLM understanding, not generation; the annotation engine is the part that travels. This complements CHAI: Building a Precise Video Language with Human–AI Oversight in a very direct way — CHAI uses critique-based human–AI oversight to make a 5-aspect cinematographer specification, MotionSight uses zero-shot visual prompts + VQAScore filtering to make a motion-decoupled action specification; both end up with SFT+DPO post-training data, both bet on a curation gate to make the synthetic labels load-bearing. It also fits the Synthetic Training Data thesis cleanly: the verification gate (VQAScore + human threshold + Tarsier2-as-baseline DPO) is the load-bearing piece, not the raw annotation volume. The camera-motion taxonomy here is coarser than the cinematographer taxonomy in CameraBench: Towards Understanding Camera Motions in Any Video but the +14.3% gain on MotionBench-CM via global motion blur is a striking datapoint that simple temporal aggregation matches a lot of what dedicated camera-motion supervision is trying to teach.