Solving Spatial Supersensing Without Spatial Supersensing

TL;DR

A meta-evaluation of Cambrian-S (Yang et al., NeurIPS 2025) — the recent “spatial supersensing” work that introduced two long-video benchmarks (VSI-Super-Recall / VSC) and bespoke predictive-sensing inference recipes for each. The authors build NoSense, a streaming bag-of-words baseline that uses only SigLIP2 (no LLM, no temporal model, no 3D state, no long-term memory) and near-perfectly solves VSR at ~95-98% accuracy on videos from 10 minutes up to 4 hours, beating Cambrian-S by ~55 percentage points. They then introduce VSC-Repeat (concatenate each video with itself 1-5 times, leaving unique-object counts unchanged) and show that Cambrian-S’s surprise-segmentation pipeline collapses from 42% to 0% mean relative accuracy. The conclusion: VSI-Super does not yet operationalize spatial supersensing — bespoke inference recipes exploit benchmark-construction shortcuts (rooms never repeat, exactly four object insertions) rather than building stable internal world maps.

Key claims

NoSense — SigLIP2-So400m-512 keeping only the top-4 frames most similar to the query object, then aggregating cosine similarities to auxiliary-object prompts to score answer permutations — achieves 98.3% / 98.3% / 96.7% on VSR 10/30/60-min splits, vs Cambrian-S’s 45.0% / 41.7% / 40.0% [Table 1, §2].
NoSense’s VSR performance stays above 95% even on the 2-hour and 4-hour splits despite discarding nearly all temporal information; this isolates VSR as solvable from a few semantically discriminative frames rather than requiring spatial cognition, world modelling, or long-horizon reasoning [§2, Fig. 1].
Ablations on NoSense are robust: using raw question text instead of an extracted object name drops 8-20% but still beats Cambrian-S by 35-40 points; removing prompt ensembling drops only a few points; varying the contrastive VLM (CLIP-L/14, SigLIP2-B/16, SigLIP2-So400m-384/512) all stay far above Cambrian-S [Table 1, §2].
Cambrian-S’s VSR inference pipeline implements the same functional pattern as NoSense — vision encoder + compact memory of salient frames + query-guided retrieval — so a substantial portion of Cambrian-S’s VSR gain plausibly comes from the same retrieval-style shortcut rather than from the additional MLLM machinery [§2 Conclusion].
VSC-Repeat is a symmetry-breaking sanity check: stitching each 10-min VSC clip to itself N=2-5 times leaves the ground-truth unique-object count unchanged. A model performing genuine spatial supersensing should give the same count; Cambrian-S’s mean relative accuracy instead drops from 42% → 0% at N=2 (30-min repeat) and stays near zero up to N=5 (1-hour repeat) [Fig. 5, §3].
The accuracy drop on VSC-Repeat is not explained by video length — the original VSC benchmark at matched lengths preserves Cambrian-S’s accuracy. Predicted counts on repeats grow with the number of revisits, consistent with the inference pipeline counting segments/rooms-visited (surprise boundaries) rather than unique objects [Fig. 5 right, §3].
The Cambrian-S VSC inference algorithm explicitly resets its event buffer at each surprise boundary, which only works if each spatial environment is visited at most once — a structural assumption baked into the VSC dataset’s generative process, not a property of natural video [§3, mechanism description].
Why was this missed in Cambrian-S’s own baseline analysis: single-frame baselines cannot apply to VSR (which requires ≥4 frames by design), and the caption-based baseline summarises only 32 frames (LLaVA-OneVision short-video setting), dramatically undersampling long videos — captioning at 1 FPS would have caught the shortcut [§2, “Why was this missed?”].
Gemini 2.5 Flash, used as a generic long-context video model with no supersensing-specific inference design, achieves ~41.5% on 60-min VSR — similar in magnitude to Cambrian-S — indicating “supersensing performance” on this benchmark is achievable from generic long-context video training alone [§4, citing Cambrian-S Fig. 11].
The proposed design principles for future spatial-supersensing benchmarks: (i) build invariance checks into the benchmark (repeats, revisits-after-delay, segment shuffles, playback-speed changes) that keep ground truth fixed; (ii) use natural long-form video (egocentric traversals, loops) rather than curated non-repeating episodes; (iii) move away from MCQ into open-ended formats; (iv) treat “the simplest streaming baseline that exploits obvious regularities” as a standard ablation [§4].
The Cambrian-S authors were included in the rebuttal loop and their response is appended as Appendix A — the paper is explicitly framed as constructive meta-evaluation rather than dismissal [Abstract, §1, Acknowledgments].

Method

NoSense (Algorithm 1) is intentionally minimal. Given a video sampled at 1 FPS, an object of interest o, four auxiliary objects, and four answer-permutation options: (i) encode each frame independently with SigLIP2-So400m-512 and CLIP-encode the prompt “a photo of a {o}”, (ii) keep only the top-4 frames in a buffer by cosine similarity to the object prompt, (iii) for each of the 4! permutations, score by summing cosine similarities of the top-4 frames against prompt-ensembled embeddings of the auxiliary objects in that ordering, and pick the highest. No language model, no temporal attention, no explicit 3D state, no long-term memory — only image–text retrieval against a frozen contrastive VLM.

VSC-Repeat is even simpler: for every video clip in VSI-Super-Counting, concatenate it with itself N ∈ {1,…,5} times and submit the resulting longer video to the Cambrian-S inference pipeline unchanged. Ground-truth unique-object counts are not altered. The expected behaviour of any genuine spatial-supersensing system is invariance; the observed behaviour of Cambrian-S is approximately linear count inflation with N.

The argument structure mirrors the kind of “model-card-style ablation, but for the benchmark” that the authors propose should become standard. NoSense functions as the shortcut-exploiter probe (what’s the simplest method that nearly saturates the benchmark?); VSC-Repeat functions as the invariance probe (what minimal-cost transformation breaks the method’s hidden assumption?). Both are diagnostic instruments, not new methods.

Results

VSR (Table 1, Fig. 1): NoSense beats Cambrian-S by +53.3 / +56.6 / +56.7 points on 10/30/60-min splits and maintains ≥95% on 2-hour and 4-hour splits.
VSC-Repeat (Fig. 5): mean relative accuracy on Cambrian-S drops from ~42% at N=1 → ~0% at N=2 → stays near zero through N=5. On the original (non-repeat) VSC at matched longer durations, accuracy is preserved, isolating the cause to the repeat-procedure rather than to length.
Robustness ablations (Table 1 right): NoSense’s lead over Cambrian-S holds across four contrastive VLM backbones spanning OpenAI CLIP-L/14 to SigLIP2-So400m-512 — i.e. the result is not a property of the strongest backbone.
Single-modality stress check (Table 1, “Raw”): even when the explicit object-extraction step is removed and the raw multiple-choice question text is used directly, NoSense retains a >35-point lead over Cambrian-S, indicating the result is largely independent of prompt engineering.

The paper does not propose a replacement benchmark — building one is named as the central open challenge. The contribution is methodological (meta-evaluation as ablation), diagnostic (these two specific failure modes), and prescriptive (the four design principles in §4).

Why it’s interesting

This is the closest filed analogue on the wiki to Incorrect Baseline Evaluations Call Into Question Recent LLM-RL Claims — both papers are authored by the Tübingen / Ameya Prabhu group, and both are doing the same methodological thing one layer down the stack: where Goel et al. expose underreported pre-RL baselines in the reasoning-RL literature, Udandarao et al. expose underreported shortcut baselines in the spatial-supersensing literature. Together the two papers crystallise an emerging genre: rigorous post-hoc benchmark autopsies whose findings shrink or invert headline claims in a recent hot subfield. Treat any future “video world model” benchmark as suspect until both probes (simplest-shortcut baseline + invariance perturbation) have been run.

Second, it sharpens the VLM Perception Failures story. That concept has accumulated three filed papers showing VLMs answer from LLM priors rather than from pixels (Vision Language Models are Biased, Hidden in plain sight: VLMs overlook their visual representations, Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs). The Udandarao finding goes the other direction: on VSR, the LLM is unnecessary — a frozen contrastive VLM with no language head beats the full MLLM pipeline by 55 points. The two failure modes (LLM-prior dominance, LLM-redundancy on retrieval-style tasks) together call into question what the language head is actually contributing on vision-centric benchmarks.

Third, it is direct evidence against the strongest reading of the World Foundation Models thesis. Cambrian-S markets itself as a step toward video world models with predictive sensing; the NoSense + VSC-Repeat probes show that strong VSI-Super performance does not in fact require any of the predictive-world-model machinery. This complements The flavor of the bitter lesson for computer vision‘s normative argument that explicit 3D / spatial intermediates may be the wrong substrate for embodied AI — but from the opposite direction: the benchmarks that are supposed to test spatial supersensing aren’t testing it either.