GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photography

TL;DR

GenDoP tackles camera-trajectory generation (the upstream of camera-controlled video diffusion) by treating camera poses as discrete tokens and training a decoder-only auto-regressive transformer conditioned on text and optionally RGBD from the first frame. It is paired with DataDoP, a 29K-shot, 11M-frame dataset of free-moving cinematic camera paths extracted from films with MonST3R, motion-tagged across 27 translation × 7 rotation combinations, and captioned by GPT-4o into both motion-only and “directorial-intent” descriptions. Against prior diffusion-based trajectory generators (CCD, E.T., Director3D), GenDoP wins on text-trajectory CLaTr-CLIP (36.2 vs 31.7), trajectory FID (22.7 vs 32.0), and a 27-expert user study on Alignment / Quality / Complexity. It’s a small AR model (12L OPT, 8 A100-hours, ~3s/trajectory at inference) — the contribution is the AR-for-trajectories framing, the free-moving-from-film dataset, and the directorial-intent caption channel.

Key claims

DataDoP is the first camera-trajectory dataset that targets free-moving (non-tracking, non-object/scene-centric) film shots — 29K samples, 11M frames, 113h, avg 14.4s/shot, 8698-word caption vocabulary, vs CCD (48), E.T. (1790), Director3D-style (no captions) [Table 1].
Captions come in two tiers: a Motion caption (camera movement only) and a Directorial caption (movement + scene interaction + directorial intent), both generated by GPT-4o from motion tags plus a 16-frame grid of the shot [§3.1, Fig. 1].
Motion tagging uses 27 translation combinations across (lateral, vertical, depth) × (left/static/right, up/static/down, forward/static/backward) plus 7 single-axis rotation states (pitch up/down, yaw left/right, roll left/right, static), with velocity-threshold tagging adopted from E.T. [§3.1, Fig. 2(a)].
Trajectory tokenization: canonical-normalize the first frame to identity, relativize subsequent poses, scale-normalize translations to a unit sphere via the largest-norm translation, convert rotations to quaternions, normalize all 10 per-frame parameters (4 quat + 3 trans + 2 focal + 1 scale) to [0, 1], multiply by bin size, integerize — 60 frames × 10 ints = a 600-int sequence with BOS/EOS/PAD [§4.2].
Architecture is a 12-layer / 12-head OPT decoder over a learnable codebook, conditioned by prepending a latent code formed from text (SD2.1 text encoder + MLP) and optionally RGB+depth (CLIP ViT + MLP, depth expanded to 3 channels), trained with cross-entropy + a regularization term [§4.3, Eq. 3].
On the DataDoP test set (3k held-out), GenDoP beats DataDoP-finetuned Director3D on F1 (0.400 vs 0.391), CLaTr-CLIP (36.18 vs 31.69), Coverage (0.872 vs 0.839), CLaTr-FID (22.71 vs 31.98) under Motion captions, and by larger margins under Directorial captions (CLaTr-CLIP 32.41 vs 23.51) [Table 3].
Pretrained CCD / E.T. / Director3D collapse on DataDoP — Director3D’s CLaTr-CLIP goes from 0.0 (pretrained) to 31.7 (after DataDoP finetuning), evidence that the dataset (free-moving + trajectory-aware captions), not just the model class, is the dominant axis [Table 3, §5.2].
Adding RGBD to the conditioning yields a third paradigm with metrics close to text-only Directorial (CLaTr-CLIP 30.23, FID 33.65) and qualitatively produces command-compliant trajectories that respect scene geometry [Table 3, Fig. 5].
27-expert AUR user study (1–5 ranking, 5 best) gives GenDoP 4.69 Alignment / 4.57 Quality / 4.71 Complexity under Motion captions vs Director3D’s 3.75 / 3.26 / 3.49 — the largest gap is on Complexity [Table 3].
Ablation: removing canonical normalization tanks CLaTr-CLIP from 36.18 to 14.92 and triples FID (22.7 → 68.6); replacing the SD2.1+CLIP encoders with simpler MLP-only encoders loses ~5 CLaTr-CLIP points [Table 4].
Inference is ~3 s/trajectory on a single A100 — orders of magnitude faster than diffusion-based trajectory generators, and the AR formulation is what unlocks variable-length trajectories at fixed-length training [§5.1].

Method

GenDoP reframes camera-trajectory generation as next-token prediction over a quantized pose stream. A trajectory is normalized to put the first frame at the world origin, every subsequent pose is expressed relatively, and the largest translation norm is used to rescale translations into a unit sphere. Each pose becomes a 10-dimensional vector (4-d quaternion, 3-d translation, 2-d focals, 1-d scale), each component is normalized to [0, 1] and multiplied by the bin size to integerize — a 60-frame trajectory therefore becomes a length-600 integer sequence, framed with BOS/EOS/PAD and embedded through a learnable codebook.

Conditioning is built as a single concatenated latent code prepended before BOS: text goes through the SD2.1 text encoder plus an MLP; the optional first-frame RGB and depth each go through a CLIP Vision encoder plus an MLP (depth is replicated to three channels first). A 12-layer 12-head OPT-style causal transformer decoder consumes the conditioning code + previous tokens and predicts the next pose token. Training is cross-entropy on the token stream plus a small regularization term on the latent code; 8 hours on a single A100.

Results

On a 3k held-out DataDoP test split, GenDoP beats DataDoP-finetuned Director3D on every alignment and quality metric across both Motion and Directorial caption conditions, and pretrained CCD / E.T. / Director3D collapse without DataDoP finetuning (Director3D’s CLaTr-CLIP goes 0.0 → 31.7 with finetuning, then GenDoP adds another +4.5 on top). On the 27-expert AUR user study, GenDoP scores 4.69 / 4.57 / 4.71 on Alignment / Quality / Complexity under Motion captions and 4.62 / 4.56 / 4.58 under Directorial captions — the largest margin is on Complexity, consistent with the dataset’s emphasis on long, multi-axis free-moving shots. The RGBD+Text conditional paradigm is reported as feasible (CLaTr-CLIP 30.23, FID 33.65) but with no baseline to compare against because no prior method takes that input combination. Ablations show canonical normalization is load-bearing (removing it triples FID) and that the pretrained-encoder choice (SD2.1 text + CLIP Vision) is worth ~5 CLaTr-CLIP points over simpler MLP encoders.

The qualitative claim in §5.3 (Fig. 4) is that DataDoP-finetuned Director3D captures coarse motion but jitters; pretrained Director3D produces smooth-but-orbit-only paths with no text correspondence; CCD and E.T. produce reactive tracking shots inappropriate for the free-moving regime; GenDoP produces coherent, complex, intentional camera moves. A downstream demo composes GenDoP trajectories with TrajectoryCrafter for camera-controlled video generation.

Why it’s interesting

This is the trajectory-generation half of the camera-controlled video-diffusion stack — none of the Camera-Controlled Video Diffusion papers filed so far (SpaceTimePilot, NeoVerse, VerseCrafter, OmniTransfer, Wan-Animate, Kling-MotionControl, ReTime) actually generate the camera trajectory; they consume one. GenDoP plugs in upstream and supplies the artistic-intent path that most of those papers handwave by demo’ing on hand-authored or extracted trajectories. Two design choices are worth flagging: (1) it picks AR-over-discrete-pose-tokens over diffusion-over-continuous-poses, on the explicit argument that diffusion-based trajectory generators (Director3D, CCD, E.T.) produce discontinuous and jittery paths — a rare case in 2025 where AR is claimed as the right answer in a continuous-control domain; (2) the Directorial caption tier — describing camera intent rather than just motion — is the lever that flips Director3D’s CLaTr-CLIP from 0 to 31.7 after finetuning, so the dataset’s caption-vocabulary delta (8698 words vs E.T.’s 1790, CCD’s 48) is doing as much work as the model class. Complements Cameras as Relative Positional Encoding (which is about how to condition a video model on cameras) by tackling the orthogonal question of where the camera trajectory comes from in the first place.