Cameras as Relative Positional Encoding

TL;DR

PRoPE (Projective Positional Encoding) is a relative positional encoding for multi-view transformers that injects the full camera frustum — both intrinsics and extrinsics — into self-attention via per-token block-diagonal projection matrices, in the same RoPE-style spirit as language-model rotary embeddings. It replaces both absolute Plücker/raymap conditioning (sensitive to world-frame choice) and relative-SE(3) schemes like CaPE/GTA (which only encode pose). Across feedforward novel view synthesis (RealEstate10K, Objaverse), stereo depth (UniMatch on RGBD/SUN3D/Scenes11), and a DL3DV spatial-cognition task, PRoPE matches or beats every alternative — and uniquely handles out-of-distribution intrinsics extrapolation (e.g. zoom 1×→5×). NeurIPS 2025; already adopted by Tencent Hunyuan World 1.5 (WorldPlay) for camera control.

Key claims

PRoPE encodes the relative projective relationship P_j P_i^{-1} between camera frustums via per-token block-diagonal matrices and a GTA-style attention block, preserving global-frame invariance, reducing to relative-SE(3) attention when intrinsics are identity, and reducing to RoPE for tokens within the same image [§3.4, Eq. 17–21].
Relative encodings (CaPE, GTA, PRoPE) consistently outperform absolute Plücker/naive raymaps for feedforward NVS; PRoPE wins on RealEstate10K (PSNR 22.80 vs 22.51 GTA vs 20.48 Plücker; LPIPS 0.146 vs 0.164 vs 0.209) and ties GTA on Objaverse where intrinsics are constant [Table 1].
On intrinsics-augmented data (RealEstate10K with random zoom; Objaverse with varied FoV) PRoPE is the only attention-level method that works at all — CaPE and GTA collapse to PSNR ~16 because they have no way to see intrinsics, while PRoPE reaches 21.42 / 22.98 PSNR [Table 2].
Hybrid token+attention conditioning is complementary: GTA+CamRay (a camera-frame raymap, intrinsics-encoding but origin-invariant) recovers most of PRoPE’s intrinsics performance, but PRoPE+CamRay beats both, while pure GTA+CamRay is needed because vanilla GTA has no intrinsic channel [§4.4, Table 3].
Out-of-distribution robustness: trained on 2 input views and tested on up to 16, and on test-time focal lengths 1×–5× training, PRoPE degrades least; CamRay added to PRoPE actually hurts intrinsics extrapolation, indicating PRoPE’s relative-projective signal generalizes more cleanly than absolute raymap intrinsics [§4.5, Fig. 4–6].
Benefits transfer beyond NVS: integrating PRoPE into UniMatch with a 50-line patch improves stereo depth on RGBD / SUN3D / Scenes11 across Abs Rel, Sq Rel, RMSE, RMSE-log; on a DL3DV discriminative spatial-cognition task PRoPE+CamRay reaches 86.1% / 93.0% / 94.3% at 5 / 9 / 17 views vs 69.1% / 76.9% / 74.6% for Plücker [Tables 4–5].
Gains hold when scaling to larger LVSM/CAT3D-class models — not just at the 25M parameter scale of the main study [§4.7].

Method

PRoPE is a drop-in replacement for the RoPE / SE(3)-pose terms inside GTA-style self-attention. Each camera i has a 4×4 lifted projection matrix P_i (intrinsics × extrinsics). For each token t in image i(t), PRoPE builds a block-diagonal D_t whose blocks are a projective block P_{i(t)} and a RoPE block on the patch’s 2D coordinates within image i(t). GTA-style attention then transforms Q, K, V by D_t before the dot product; the relative interaction between tokens t1, t2 collapses to (P_{i(t1)})^T (P_{i(t2)})^{-1} plus a RoPE term, i.e. the full projective relationship between the two camera frustums (Eq. 17–21).

The implementation is single-file (JAX and PyTorch versions in the repo), fused-attention compatible (because the per-token transformations are applied to Q/K/V before the softmax), and the only structural change to a transformer is the construction and application of D_t. For experiments, the authors reimplement LVSM (NVS) and patch UniMatch (stereo depth) to substitute PRoPE for whatever conditioning channel each method originally used. Padding tricks (fixed embedding instead of raymap channels) keep total input dimensionality matched across baselines.

Results

Feedforward NVS (RealEstate10K / Objaverse, constant intrinsics): PSNR 22.80 / 23.70, LPIPS 0.146 / 0.104, SSIM 0.725 / 0.879 — strictly best or tied-best across all six metrics [Table 1].
Intrinsics-augmented NVS: PRoPE 21.42 / 22.98 PSNR vs CaPE 15.94 / 16.78 and GTA 15.77 / 18.00 — a ~5–7 PSNR gap because attention-level methods without intrinsics encoding cannot reason about zoom [Table 2].
Hybrid: PRoPE+CamRay 21.78 / 22.98 PSNR, the best in the four-way comparison; CamRay improves PRoPE on in-distribution but hurts intrinsics extrapolation [Table 3, §4.5].
Stereo depth (UniMatch + PRoPE vs UniMatch): Abs Rel improves 0.123→0.105 on RGBD, 0.131→0.117 on SUN3D, 0.065→0.049 on Scenes11; RMSE improves on all three [Table 4].
Spatial cognition (DL3DV, multi-choice scene-question task): PRoPE+CamRay 86.1% / 93.0% / 94.3% accuracy at 5 / 9 / 17 views vs Plücker 69.1% / 76.9% / 74.6%; gap widens with more views [Table 5].
OOD robustness: PRoPE retains performance at 16× train sequence length and 5× train focal; Plücker, CaPE, and GTA all degrade much faster [§4.5, Fig. 4].

Why it’s interesting

This is the underlying primitive that several adapter-based camera-controlled video models in the wiki appear to converge toward. The Camera-Controlled Video Diffusion cluster’s filed papers (VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control, NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos, SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time) all bolt control onto a frozen Wan2.1 backbone via spatially-aligned camera/depth/Gaussian-trajectory channels rendered through cross-attention; PRoPE is the orthogonal proposal — don’t render rays into the token stream, change the attention bias instead. The README notes Tencent Hunyuan World 1.5 (WorldPlay) already adopted PRoPE for camera control, putting it on the production-side ledger of camera-conditioning techniques. The clearest contrast in the wiki is with the Plücker raymap recipe that LVSM-style NVS and most of the frozen-Wan adapter papers inherit: PRoPE is what the author is explicitly pitching as the “stop settling for Plücker” replacement.