Cambrian-S: Towards Spatial Supersensing in Video

TL;DR

A position-paper-plus-artifact-bundle from NYU+Stanford (Yang, Yang, Brown, Fergus, LeCun, Fei-Fei, Xie) that proposes spatial supersensing as a four-stage hierarchy beyond linguistic understanding (semantic perception → streaming event cognition → implicit 3D cognition → predictive world modeling), and ships four coupled deliverables: (1) VSI-Super, a two-task long-video benchmark (VSR for spatial recall, VSC for continual counting) deliberately resistant to brute-force context expansion; (2) VSI-590K, a 590k-sample spatial-instruction-tuning dataset assembled from 10 sources spanning real 3D-annotated scans, simulators, and web video; (3) the Cambrian-S MLLM family (0.5B–7B, Qwen2.5 + SigLIP2-SO400m) achieving >+30% absolute on VSI-Bench; and (4) a predictive sensing prototype that trains a latent-frame-prediction head on top of the MLLM and uses prediction-error “surprise” to drive memory compression and event segmentation, beating Gemini-2.5-Flash on VSI-Super despite being a small model. The paper’s central empirical claim is that data scaling alone gets Cambrian-S to SoTA on existing spatial benchmarks but cannot crack VSI-Super, motivating a paradigm shift toward predictive-world-model inference. The accompanying critique paper Solving Spatial Supersensing Without Spatial Supersensing has since shown that VSI-Super itself admits a shortcut (a frozen SigLIP2-only baseline reaches 95–98% on VSR) — so the contribution should be read primarily as the position and the Cambrian-S data + model artifacts, not as a solved benchmark.

Key claims

Existing video benchmarks predominantly probe linguistic understanding and semantic perception, not spatial cognition: Cambrian-1 with only 32 frame captions (no pixels) surpasses chance by >20 points on EgoSchema, VideoMME, LongVideoBench, VideoMMMU, Perception Test, and MVBench — implying the tasks are largely solvable from textual summaries [§2.1, Fig. 2 a–c, i].
VSI-Bench and Tomato are the only existing benchmarks where Cambrian-1’s caption-only condition falls below chance, isolating them as actually requiring visual sensing rather than language priors [§2.1, Fig. 2].
VSI-Super is constructed to be brute-force-resistant: VSR concatenates indoor walkthrough videos with diffusion-edited “needles” (out-of-context objects) inserted into 4 distinct frames, then asks the model to sequentially recall their locations across 10/30/60/120/240-minute streams; VSC concatenates room-tour clips and asks for cumulative unique-object counts across viewpoint shifts and scene transitions [§2.2, Fig. 4–5].
Gemini-2.5-Flash, despite a 1,048,576-token context, runs out of context on 2-hour VSI-Super videos and achieves only 41.5% on 60-min VSR and 10.9% on 60-min VSC — well below the model’s 80%+ on VideoMME / VideoMMMU [§2.2, Table 1].
Gemini’s predicted counts on VSC do not scale with video length or true object count; they saturate at a small constant, indicating reliance on training-distribution priors rather than learned counting [§2.2, Fig. 6].
The Cambrian-S base model upgrades Cambrian-1 with SigLIP2-SO400m + Qwen2.5 + a two-layer MLP projector; the rest of the Cambrian-1 recipe (hyperparameters, data mix) is held fixed to isolate the data-curation effect [§3.1].
VSI-590K is built from 10 sources balancing (i) annotated real scans (S3DIS, Aria Digital Twin, ScanNet, ScanNet++ V2, ARKitScenes), (ii) simulated trajectories (ProcTHOR, Hypersim), and (iii) unannotated YouTube room tours + Open X-Embodiment + AgiBot-World pseudo-labeled with vision tools (e.g. VGGT) — totaling 5,963 videos + 44,858 images + 590,667 QA pairs [§3.2, Table 2].
The diversity matters: a same-sized dataset drawn from a single source is “substantially less robust” than the 10-source mix; ablations supporting this are referenced as in-paper but the appendix is needed for numbers [§3.2].
Cambrian-S achieves >+30% absolute gain on VSI-Bench over its base MLLM “without sacrificing general capabilities” — strong evidence that spatial cognition is trainable from data within the current paradigm [§3, Fig. announced in §1].
But Cambrian-S still falls short on VSI-Super, isolating long-horizon predictive world modeling as a regime where the data-scaling lever runs out [§3.5].
Predictive sensing is a latent-frame-prediction (LFP) head trained on top of Cambrian-S; at inference time, the per-frame prediction error is used as a surprise scalar to (i) compress or skip non-surprising frames (memory management) and (ii) detect event/scene boundaries (event segmentation) [§4].
With LFP-driven memory + segmentation, the small Cambrian-S outperforms Gemini-2.5-Flash on VSI-Super despite having vastly fewer parameters — the paper’s headline evidence that mechanism, not scale, is the missing ingredient [Abstract, §4, Fig. 11 referenced].
The proposed framing is explicitly developmental: stages 0–2 are largely covered by current MLLMs; stage 3 (implicit 3D) is partially addressed by VSI-Bench-style benchmarks; stage 4 (predictive world modeling) is the open frontier and the paper’s primary target [§1].

Method

The paper’s structure mirrors its three-part argument. Part one (§2) is a benchmark audit: Cambrian-1 is run in four conditions (32 frames, single middle frame, 32 frame captions, blind text-only) across a suite of video-QA benchmarks; differences between conditions are read as evidence about what each benchmark actually tests. The audit motivates VSI-Super, constructed by (i) editing diffusion-inserted needles into room-tour videos with Gemini’s image-edit tool for VSR, and (ii) concatenating VSI-Bench room-tour clips for VSC. Part two (§3) is a data-and-training study: upgrade Cambrian-1’s encoder/LLM, curate VSI-590K across 10 sources with question templates instantiated from 3D annotations or pseudo-labels (e.g. VGGT for YouTube tours), and train Cambrian-S at 0.5B / 1.5B / 7B parameters. Part three (§4) swaps the inference algorithm: a self-supervised LFP head predicts the next latent frame from the MLLM’s tokenized stream; at test time, prediction-error surprise is thresholded for (i) memory-bank admission (surprising frames kept, others compressed/dropped) and (ii) event-boundary detection (surprise spikes trigger segment resets). Both mechanisms are inference-side — no policy gradients, no training-time exposure to the test tasks.

The structural commitment worth noting: the LFP head is trained self-supervisedly on raw video without VSI-Super supervision, so the predictive-sensing pipeline is positioned as architecture-only, not benchmark-fitted. (The critique in Solving Spatial Supersensing Without Spatial Supersensing disputes this, arguing the surprise-segmentation logic implicitly encodes the “rooms never repeat” assumption baked into VSC’s generative process.)

Results

VSI-Bench: Cambrian-S achieves >+30% absolute gain over the upgraded base MLLM, claimed as state-of-the-art on VSI-Bench at the time of release [§3, summarized in §1].
VSI-Super (Gemini-2.5-Flash baseline): VideoMME 81.5 / VideoMMMU 79.2 / VSI-Bench 45.7 / VSR-60-min 41.5 / VSR-120-min OOC / VSC-60-min 10.9 / VSC-120-min OOC [Table 1]. Despite a 1M-token context, Gemini exceeds context on 2-hour videos.
VSI-Super (Cambrian-S, no predictive sensing): falls short — exact numbers not in the fetched section but described as “limited” relative to Gemini, with VSI-Super performance gated by the lack of long-horizon mechanism rather than by short-video spatial cognition [§3.5].
VSI-Super (Cambrian-S + LFP predictive sensing): “substantially outperforms leading proprietary baselines” — the small Cambrian-S model beats Gemini-2.5-Flash on VSR and VSC [Abstract; §4 referenced as Fig. 11 in arXiv html].
VSC counting failure mode: Gemini’s predicted counts plateau at “a few dozen” regardless of how many objects are actually in the video — the model is counting from priors, not from the video [§2.2, Fig. 6].

The critique paper Solving Spatial Supersensing Without Spatial Supersensing later reports a different ceiling: a frozen SigLIP2-So400m-512 baseline (NoSense) — no LLM, no temporal model — hits 95–98% on VSR across 10-min to 4-hour splits, beating Cambrian-S by ~55 points; VSC-Repeat (concatenating each VSC clip with itself N times, leaving ground-truth count unchanged) collapses Cambrian-S’s mean relative accuracy from 42% → 0% at N=2. So the headline VSI-Super gains here should be read as relative to the long-context-LLM family of baselines specifically, not as evidence the benchmark is unsolvable without predictive-world-model machinery. The Cambrian-S authors’ rebuttal is appended to the critique paper as Appendix A.

Why it’s interesting

Two things make this paper structurally important to the wiki rather than just another video-MLLM release. First, it is the most concrete public articulation of the predictive-world-model thesis as an MLLM modification the wiki has filed — the four-stage hierarchy (semantic perception → streaming event cognition → 3D cognition → predictive world modeling) is essentially the JEPA-line thesis (Introducing the V-JEPA 2 world model and new benchmarks for physical reasoning, Inference-time Physics Alignment of Video Generative Models with Latent World Models, Action100M: A Large-scale Video Action Dataset) repurposed as a video-MLLM design schema rather than as a standalone representation-learning objective. LeCun and Fei-Fei as co-authors on a paper led by Saining Xie’s group is the public bridge between the World Foundation Models latent-predictive camp and the practical video-MLLM line that has historically been LLM-centric. The thesis Saining Xie’s tweet (Saining Xie joining AMI Labs (LeCun's new world-model lab) — announcement announcing AMI Labs) is now operationalized at company scale is here as a research paper.

Second, the rapid arrival of Solving Spatial Supersensing Without Spatial Supersensing — a critique by Prabhu’s Tübingen group that the headline VSI-Super gains rely on benchmark-specific shortcuts and that a SigLIP2-only baseline beats the full pipeline by 55 points — makes this pair the cleanest paper-and-its-rebuttal sequence on file. Read together they form a case study in VLM Perception Failures from the other direction: where prior wiki entries showed the LLM head ignoring intact visual features, NoSense shows the LLM head being unnecessary on a task marketed as requiring spatial cognition. The Cambrian-S authors’ included rebuttal makes this a constructive exchange rather than a flame war, but the methodological lesson stands: long-video spatial benchmarks need the kind of “simplest-shortcut baseline + invariance perturbation” probes that Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims argued for in the RL-reasoning literature. File Cambrian-S as the position paper; consult NoSense for the benchmark-design caveats; treat VSI-590K and the Cambrian-S model checkpoints as durable artifacts regardless of how the VSI-Super arms race resolves.