Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning

TL;DR

Know Your Camera (KYC) is an ICRA 2026 study that adds explicit camera-geometry conditioning to standard behavior-cloning policies by concatenating a per-pixel 6D Plücker raymap — derived from calibrated intrinsics and cam2world extrinsics — with the RGB image before the visual encoder. Applied unchanged to ACT, Diffusion Policy, and SmolVLA, the raymap consistently improves generalization across novel viewpoints on six new manipulation tasks in RoboSuite and ManiSkill that pair fixed and randomized scene variants. The paired benchmarks isolate a specific shortcut — policies without extrinsics infer camera pose from static background cues — and quantify how appearance randomization removes it. Secondary ablations flag two decisions as load-bearing: delta end-effector action space beats absolute joint targets, and random cropping interacts non-trivially with camera-frame reasoning.

Key claims

Adding a per-pixel Plücker raymap r = (d_w, C × d_w) — unit ray direction in world frame plus its moment about the world origin — as an input channel improves viewpoint generalization for ACT, Diffusion Policy, and SmolVLA without any other architectural change [§III.A].
Fixed-scene simulated environments let policies without extrinsics use static background cues as a pose shortcut; the paper introduces paired fixed/randomized variants of six manipulation tasks (three in RoboSuite, three in ManiSkill) to decouple background cues from camera pose, and shows explicit conditioning matters most in the randomized variants where the shortcut is unavailable [Abstract, §I].
The Plücker raymap is the first filed instance of injecting explicit per-pixel camera geometry directly into RGB-based robot policies for control, imported from the multi-view-transformer / 3D-reconstruction literature where ray-map conditioning is standard [§II.C].
Action space matters independently of camera conditioning: delta end-effector pose is the best-performing action space in the ablation sweep [§IV, per sharer note; author list of factors §I].
Beyond action space, four additional factors shift performance in the empirical analysis: random cropping, early vs late conditioning, choice of pretrained visual encoder, and a scaling law over number of camera views [§I, §IV].

Method

Each observation carries known camera intrinsics K and cam2world extrinsics (R, t). For every pixel (u, v) the paper computes a normalized world-frame ray direction d_w = R^T K^{-1} [u,v,1]^T / ‖·‖ and a moment vector m = C × d_w about the world origin, where C = -R^T t is the camera center. The 6D Plücker coordinate r = (d_w, m) satisfies d_w · m = 0 and represents the oriented 3D line the pixel projects along in a form that is homogeneous, continuous, and origin-invariant along the ray. The per-pixel raymap has shape H × W × 6 and is concatenated with the RGB image before the visual encoder — a drop-in change with no other architectural modification. The same recipe applies unchanged across ACT (transformer with a chunked-action head), Diffusion Policy (denoising action head), and SmolVLA (VLM-backboned VLA), which are the three policy families the paper evaluates. The six manipulation tasks in RoboSuite and ManiSkill are released in paired fixed / randomized-scene variants so that background-cue leakage of camera pose can be measured and controlled separately from geometric conditioning.

Results

Author summary reports that conditioning on Plücker extrinsics significantly improves generalization across viewpoints for all three baseline policy families under standard behavior cloning [Abstract]. Diagnostic analysis on fixed-scene variants documents the shortcut mechanism: policies without extrinsics can implicitly infer camera pose from static background elements, which inflates performance on fixed backgrounds and collapses when the background is randomized. The paper isolates five additional factors that shape the size of the effect — action-space choice (delta end-effector best), random cropping, early vs late conditioning of the raymap, choice of pretrained visual encoder, and a scaling law with respect to the number of camera views [§I]. Concrete per-task numbers not extracted here — they live in the paper’s tables against the six new fixed/randomized RoboSuite + ManiSkill benchmarks, which are released with the demonstrations.

Why it’s interesting

This is the sharpest filed instance of moving the camera lever into a behavior-cloning policy on the same axis that See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models moves pointmaps into a full VLA — both papers diagnose the same underlying observation-to-action frame mismatch, and both fix it by pushing 3D-aware geometry directly into the input stream rather than into the backbone or a peer model. The See-like-a-Robot paper explicitly ran the head-to-head: on RoboCasa, robot-frame pointmaps beat “Plücker rays + depth” (RGB+Pointmap 34.7% vs RGB+Plücker+Depth 31.6% vs RGB 27.9%), so KYC’s Plücker channel captures a real but weaker version of the same signal, at lower hardware requirements (no depth sensor needed) and applicable to any standard BC policy — the trade is minimum viable geometry for maximum drop-in compatibility. Complementary to Cameras as Relative Positional Encoding (PRoPE) which stakes the opposite methodological position for multi-view transformers: PRoPE argues stop settling for Plücker raymaps and inject camera geometry into attention biases instead, and reports Plücker raymaps as the weakest baseline. KYC’s ICRA 2026 best-paper result therefore sits at a specific point: Plücker raymaps are worth adopting when you can’t change the architecture, while PRoPE is the follow-on when you can. The fixed/randomized paired-benchmark methodology is also a template — the pose-shortcut leak it names is the sim-side analog of the HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining ID/OOD split’s dataset-side analog, and worth checking on every sim benchmark on the VLA Models recipe-lever board.