Planning with the Views (ViewSuite)

TL;DR

ViewSuite is a 6-DoF view-planning benchmark on 286 real ScanNet scenes that decomposes spatial reasoning into single-turn view-action tracking (P2V, V2P) and multi-turn target localization (Interactive View Planning, IVP). 13 frontier VLMs ace local tracking (~70% P2V short-horizon) but collapse on IVP, with the best closed model (Gemini 3.1 Pro) at 21.3% and most open-weight models below 5%. Direct PPO/GRPO can’t bootstrap from the ~2.5% base success rate, so the authors compress every rollout — successful or not — into a view graph whose paths are reformulated into supervised view-planning demonstrations; iterating RL → SFT on this graph lifts Qwen2.5-VL-7B from 2.5% → 47.8% IVP and beats every proprietary VLM evaluated.

Key claims

The Interactive View Planning gap is large and consistent: GPT-5.4 Pro 19.9%, Gemini 3.1 Pro 21.3%, Claude Opus 4.6 10.8%, every open-weight model under 5%, all sharply below their ~50-70% accuracy on the single-turn P2V/V2P probes built from the same view pairs [Results §Frontier VLM benchmark].
The gap is a cognitive failure, not view-encoding noise: ≥90% (up to 99.1% for Gemini 3.1 Pro) of successful IVP rollouts visit a view within the 0.5 m / 30° success threshold before answering — frontier successes are visual matching after arrival, not prospective pose inference [§1b, Table “IVP successes: visual encounter vs. genuine inference”].
P2V/V2P difficulty correlates with rotation distance (cumulative rotations are hard to mentally simulate), but IVP difficulty inverts the signal and collapses with position distance (ρ down to −0.42 for GPT-5.4 Pro on pos_dist) — the two task families are bottlenecked by different spatial-reasoning skills [§2, §2b, correlation panel].
IVP is bottlenecked by planning ability, not horizon length or rendering fidelity: doubling turn budget 10 → 20 helps modestly and 20 → 30 is essentially free; re-rendering with 3D Gaussian Splatting gives mixed and sometimes large negative swings (GPT-5.4 and Grok 4.20 Beta lose 14.4 and 13.0 V2P points) [§3, §4].
Direct RL fails to bootstrap from the 2.5% base success rate: PPO plateaus at 3.2%, GRPO with reward-variance filtering reaches 5.2%, success-only PPO+SFT bootstrapping gets 6.2% — all far below the 47.8% achieved by view-graph distillation [§5, training-recipe table].
The graph-distillation breakthrough: every trajectory, including failures, compresses into a shared view graph (nodes = viewpoints with rendered views, edges = actions); any path through the graph reformulates into a valid (initial, target, action chain) IVP demonstration regardless of whether the original episode succeeded [§Self-Exploration with View Graph Distillation, stages 2-3].
Iterating RL → SFT → RL → SFT on view-graph-derived demos lifts Qwen2.5-VL-7B from 2.5% → 47.8% IVP and Qwen3-VL-8B to 32.5%, both beating Gemini 3.1 Pro 21.3% and GPT-5.4 Pro 19.9% [§5, training-recipe table].
The 47.8% result is robust to evaluation-protocol choices: under No-Snap (raw rotation magnitudes, no on-grid rounding) the trained Qwen2.5-VL-7B drops to 19.6% but still leads Gemini 3.1 Pro’s 15.7%; under No-Submit (success the moment the pose enters threshold) it reaches 60.2% vs Gemini’s 31.5% [§5b].
Training reshapes attention as a two-phase strategy: image-attention fraction increases in early layers (L0-L4) and decreases in deep layers (L8+) compared to the base, and decreases monotonically across turns — consistent with front-loaded visual grounding followed by text-space planning [§7, layer × turn attention panels].
The learned priors transfer to external view-dependent tasks: under identical GRPO post-training, the IVP-trained Qwen2.5-VL-7B gains ~10 absolute points over the base on MindCube (66.2 vs 56.3) — a benchmark with no shared scenes, actions, or rendering pipeline [§9, spatial-prior transfer table].

Method

ViewSuite is a 6-DoF environment on 286 real ScanNet scenes with 12 actions (translation step = 0.5 m, rotation step = 30°), generating ~55K view pairs and ~165K task instances. Three probes share the same view pairs: P2V (predict resulting view from an action sequence), V2P (pick the action sequence that reaches a target view), and IVP (multi-turn closed-loop view-planning where the agent submits its estimated target pose, success at 0.5 m / 30°).

The training pipeline alternates two stages. In the RL stage, the agent runs PPO IVP rollouts with sparse +1 reward at the success threshold plus a small format reward; every trajectory streams into a background graph builder regardless of outcome. The graph stage incrementally compresses trajectories into a view graph (nodes = viewpoints with rendered views, edges = actions), deduplicated via viewpoint similarity. Sampled paths P = (v₀, a₁, v₁, …, aK, vK) yield (initial=v₀, target=vK, action chain) demonstrations for the SFT stage, with standard cross-entropy. The resulting model initializes the next RL stage; the loop runs for 3 iterations.

Results

IVP All-split: Qwen2.5-VL-7B base 2.5% → direct PPO 3.2% → GRPO with filter 5.2% → success-only bootstrapping 6.2% → random-graph ablation 13.0% → 1 iter 12.0% → 2 iter 27.9% → 3 iter 47.8%; Qwen3-VL-8B reaches 32.5% with 3 iters [§5 training-recipe table].
Frontier baselines on IVP All: GPT-5.4 Pro 19.9% (computed over 507 non-refused instances), Gemini 3.1 Pro 21.3%, GPT-5.4 16.6%, Grok 4.20 Beta 7.9%, GPT-5.1 6.2%, Claude Opus 4.6 10.8%, Gemini 3 Pro 9.2% [Results §Frontier VLM benchmark].
P2V/V2P single-turn: GPT-5.4 Pro 53.2 / 50.6 All; Gemini 3.1 Pro 48.9 / 49.4; Qwen2.5-VL-7B base 29.4 / 24.3 [Results §Frontier VLM benchmark].
Turn-budget ablation: Gemini 3.1 Pro gains only +1.9 points from B=10 → B=30 (21.3 → 23.2); Claude Opus 4.6 gains +8.8 but is still well below the trained 7B model [§3, IVP All-split accuracy table].
Gaussian-splat re-render at B=10: Gemini 3.1 Pro +2.7 overall vs point cloud, GPT-5.4 −5.6, Grok 4.20 Beta −10.3 — fidelity doesn’t unlock the gap and can hurt [§4].
Spatial-prior transfer (identical GRPO post-training): the IVP-trained checkpoint beats base Qwen2.5-VL-7B by +12.2 / +8.0 / +9.9 on P2V / V2P / MindCube post-GRPO accuracy [§9, transfer table].

Why it’s interesting

This is the cleanest filed example of “RL with near-zero reward can still work if you re-use the geometry the failures record” — direct PPO and reward-variance-filtered GRPO plateau under 6%, but compressing every rollout into a graph and reformulating arbitrary paths into supervised demonstrations breaks the bootstrap barrier and reaches 47.8%. It sharpens the open question in Reasoning RL about what the right credit-assignment primitive is when reward is too sparse for policy gradients to climb: graph-of-experience SFT is a different answer than Training-Free Group Relative Policy Optimization‘s natural-language experience library and On-Policy Distillation‘s on-policy distillation. It’s also a sharp new failure mode for VLM Perception Failures — frontier VLMs can track local view-action transitions but cannot plan toward an unseen target view, and ≥90% of their “successes” are post-hoc visual matching rather than prospective pose inference, which is mechanistically distinct from the LLM-prior-dominance and output-format-collapse failures already on that page. The MindCube transfer (+9.9 points) suggests the view-graph-distilled spatial prior is task-portable, not benchmark-specific.