SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers

ListenRays as Coordinates: Teaching Video DiTs Where They're Looking11:46

Show notes

SCoPE (a.k.a. RayPE) retrofits camera-ray geometry into a pretrained video diffusion transformer by treating each token's 6D Plücker ray as a second positional coordinate, injected additively into attention's queries and keys. The key trick: a flipped Q/K arrangement makes the geometry-only cross-term equal the Plücker reciprocal product — the same bilinear invariant that tells you whether two lines of sight meet. On Wan2.2-14B the authors report a 29% drop in rotation error and a 43% drop in FVD versus the strongest baseline, at under 0.1% added parameters, plus "revisit consistency" that falls out of the coordinate system itself. **Sources** - SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

SCoPE (a.k.a. RayPE in the paper body) retrofits camera-ray geometry into a pretrained video DiT (Wan2.2) as a second positional coordinate, added inside self-attention rather than around it. Each token’s 6D Plücker ray is projected and added to queries and keys in a flipped Q/K arrangement so the geometry-only cross-term becomes the Plücker reciprocal product — the same bilinear invariant that measures whether two lines of sight meet. A Normalize-Gate-Inject (NGI) pipeline keeps the geometry channel stable across SfM / SLAM / metric pose sources, RoPE is preserved bit-exactly, and the total parameter cost is under 0.1%. On Wan2.2-14B it reports −29% rotation error, −43% FVD, and by-construction revisit consistency (loop back → return to the same room).

Key claims

The Plücker reciprocal product $r_1 \diamond r_2 = d_1{\cdot}m_2 + d_2{\cdot}m_1$ is bilinear in the two rays and vanishes exactly when they are coplanar (intersect / are parallel / coincide), giving an SE(3)-invariant algebraic form that matches the bilinear shape of the attention dot product [§3.3, Eq. 5].
Adding projected Plücker features to queries and keys in a flipped arrangement ( $q$ gets $[d;m]$ , $k$ gets $[m;d]$ ) makes the geometry-only cross-term of the expanded attention score coincide exactly with the reciprocal product at $W^Q_r = W^K_r$ [§4.1, Prop. 1].
The additive injection preserves RoPE bit-exactly and adds under 0.1% new parameters; the geometry projection is zero-initialized so training starts from the unchanged pretrained DiT [§1, §4.3].
Component ablation confirms all three attention terms are individually necessary: removing the two content×geometry cross-terms (B)+(C) raises RotErr from 0.085 to 0.135 and FVD from 543 to 695; removing the geometry×geometry term (D) raises RotErr to 0.159 and FVD to 733 [Table 3].
Direction/magnitude decoupling + per-token log-magnitude gate (NGI) is load-bearing for cross-dataset stability: replacing it with a raw 6D Plücker projection drops CamMC from 0.802 to 0.865 and FVD from 543 to 560 [Table 3].
Additive Plücker injection outperforms every multiplicative FreqSplit-RoPE variant (4×4 projective / camera-UV / Plücker-RoPE / 4-DoF Plücker) on every metric under the same Wan2.2-5B backbone and training budget [Table 2].
Against camera-conditioned baselines on Wan2.2-14B: RotErr 0.058 vs 0.082 (UCPE), TransErr 0.517 vs 0.693, FVD 280 vs 529, FID 41.0 vs 54.8; also beats ReRoPE and ReCamMaster on every reported metric [Table 1].
Revisit consistency (turn 90°, pass through a gateway, then return → previously seen regions remain anchored) is claimed to follow from the coordinate system itself, not from an added memory mechanism — outbound and return views reuse the same ray coordinates [project page §07, “Emergent property”].
Injecting geometry into the value stream $V$ (instead of Q/K only) consistently underperforms: adding a V transform raises FVD from 543 to 724 [Table 3], confirming ray geometry is most useful as an attention-bias positional signal rather than a value-channel feature.
The Plücker moment scales linearly with camera translation, so the reciprocal product scales as $\alpha \cdot r_1 \diamond r_2$ under a global rescaling — this is what forces NGI on real mixed-source data (RealEstate10K / DL3DV / PanShot / OmniWorld) with heterogeneous pose scales [§3.4, Eq. 6].

Method

Each video-DiT token has a camera ray, computed from per-frame extrinsics and intrinsics under the OpenCV convention. The ray is expressed as a 6D Plücker coordinate $r = (d, m)$ with $m = t \times d$ . SCoPE adds a per-layer projection of the ray to the pretrained content features:

$q_i = W^Q x_i + W^Q_r \tilde{r}_i, \quad k_i = W^K x_i + W^K_r \tilde{r}_i^{\text{flip}}$

where the flip swaps the direction and moment halves. Expanding the attention score gives four terms: content×content (the original attention), content×geometry (two cross-terms coupling appearance with ray geometry), and geometry×geometry (canonically equal to the Plücker reciprocal product under the flip). The 3D RoPE on $(x, y, t)$ indices is left untouched.

Normalize-Gate-Inject stabilizes the encoding across heterogeneous pose scales in three steps: (i) decouple the ray into a scale-invariant direction $\hat{d}$ , scale-invariant normalized moment $\hat{m}$ , and a scalar log-magnitude $\log\|m\|$ ; (ii) run the log-magnitude through a small 2-layer MLP to produce a per-channel sigmoid gate; (iii) project the 7D feature, apply a learnable RMSNorm symmetric to the content-branch QKNorm, and multiply by the gate. A scalar $\lambda$ (zero-initialized) sets the overall geometry/content balance so training starts from the pretrained DiT exactly. During training, a random per-clip offset perturbs $\log\|m\|$ into the gate only, simulating trajectory rescaling.

The public training recipe fine-tunes only the high-noise expert of Wan2.2-A14B (timesteps sampled from [0.9, 1.0)) on a concatenation of RealEstate10K + DL3DV + PanShot + OmniWorld. Per-clip near-depth is precomputed with a RAFT-based estimator so translations are comparable across pose sources; absolute scale is then handled internally by the learned gate.

Results

Wan2.2-14B, RE10K held-out, absolute-scale evaluation (no per-video rescaling) [Table 1]:

Camera controllability: RotErr 0.058 (vs 0.082 UCPE, 0.114 ReRoPE, 0.109 ReCamMaster), TransErr 0.517 (vs 0.693 / 0.820 / 0.976), CamMC 0.530, ATE 0.605.
Quality: CLIP 26.30, FVD 280.17 (vs 529 UCPE, 493 ReRoPE, 675 ReCamMaster), FID 41.01 (vs 54.75 / 49.18 / 59.21).
5B backbone: RotErr 0.085 (vs 0.113 UCPE / 0.137 ReRoPE / 0.152 CameraCtrl), FVD 543 (vs 703 / 685 / 824). The pattern of wins is consistent across the two backbone sizes.

Design-space ablation vs four multiplicative FreqSplit-RoPE variants on Wan2.2-5B [Table 2]: RayPE (additive Plücker) beats every variant on every metric; the closest FreqSplit-RoPE variant (4-DoF Plücker) reaches only 0.137 RotErr and 638 FVD.

Component ablation [Table 3]: the largest drops come from removing the content×geometry cross-terms (B+C) — RotErr 0.135, FVD 695 — and the geometry×geometry term (D) — RotErr 0.159, FVD 733 — confirming the full four-term attention decomposition is load-bearing.

Why it’s interesting

Fits directly into the Camera-Controlled Video Diffusion cluster and sharpens one of its live design questions: how should camera geometry enter a pretrained video DiT? The filed papers in that cluster occupy four paradigms — reconstruction-first (NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos), control-first via GeoAdapter (VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control / SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time), reference-video (OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer), and multiplicative RoPE replacement (Cameras as Relative Positional Encoding / UCPE / ReRoPE). SCoPE is the first filed instance of additive Plücker-into-attention on Wan, and it beats the multiplicative-RoPE variants (including the PRoPE-family baseline family) on the same backbone and budget — a concrete counter-datum to PRoPE’s design premise that camera geometry belongs inside a modified RoPE. Complements CamCloneMaster: Enabling Reference-based Camera Control for Video Generation (reference-video conditioning) as a diametrically opposite recipe: SCoPE trades the reference video for explicit trajectory + a 0.1% adapter, and reports better absolute-scale trajectory metrics. The claimed “revisit consistency by coordinate system” is a distinctive property that Mirage (Latent Spatial Memory for Video World Models) achieves via a latent spatial cache, and SPMem/NeoVerse via point-cloud rendering — SCoPE claims to get it without any memory or reconstruction module.