GeneralVLA-2: Geometry-Aware Reconstruction and Governed Memory for Robot Planning

TL;DR

GeneralVLA-2 is an incremental upgrade to the GeneralVLA hierarchical VLA stack that targets two failure modes of the v1 system: monocular SAM3D-style object reconstruction hallucinating pose and unseen geometry when calibrated multi-view RGB-D is available, and the original semantic-similarity KnowledgeBank that appends new experience without governing quality, conflicts, confidence, or lifecycle. Two contributions address these: GeoFuse-MV3D, a geometry-prior-guided MV-SAM3D reconstruction branch that verifies external geometry cues against input-view masks via soft visual-hull support and axis-wise refinement and only fuses geometry (preserving appearance); and a re-engineered KnowledgeBank that adds explicit quality / confidence / lifecycle / verifier / conflict metadata plus precision-oriented retrieval. The reconstruction branch is evaluated on GSO-30 (small reconstruction-quality lifts vs MV-SAM3D baseline) and the memory module is evaluated, somewhat surprisingly for a robot-planning paper, on Terminal-Bench 2.0 and SWE-Bench Verified vs ReasoningBank.

Key claims

GeoFuse-MV3D verifies an external geometry prior against input-view masks, applies a soft visual-hull support term, performs axis-wise refinement, and fuses only geometry while leaving appearance untouched — separating the failure modes of monocular SAM3D-style reconstruction (pose hallucination, unseen-side geometry) from texture quality [Abstract].
On GSO-30, GeoFuse-MV3D improves over the MV-SAM3D baseline by reducing CD by 2.20% and LPIPS by 2.02% while increasing PSNR by 2.36% and SSIM by 1.03% [Abstract].
KnowledgeBank is upgraded from a semantic-similarity store with monotonic append into a governed long-term memory system with explicit per-entry quality, confidence, lifecycle, verifier, and conflict metadata, and a precision-oriented retrieval policy that uses those metadata fields [Abstract].
On Terminal-Bench 2.0, KnowledgeBank improves success rate over the ReasoningBank baseline by 4.53% while reducing the action-step count metric (AS) by 4.95% [Abstract].
On SWE-Bench Verified, KnowledgeBank improves resolve rate over ReasoningBank by 3.73% while reducing AS by 5.65% [Abstract].

Method

GeneralVLA-2 keeps the v1 hierarchical interface (language + RGB-D → 3D end-effector path) and slots in two replacements. For perception, GeoFuse-MV3D sits between calibrated multi-view RGB-D inputs and SAM3D-style reconstruction: it (1) generates an external geometry prior, (2) verifies that prior against per-input-view 2D masks rather than trusting it blindly, (3) adds a soft visual-hull support term so reconstructed geometry stays inside the union of view-cone constraints, and (4) refines pose axis-wise. Crucially the fusion step touches only geometric quantities — appearance fields from the monocular SAM3D branch are preserved, which is presented as the reason GeoFuse-MV3D improves CD/LPIPS and PSNR/SSIM rather than trading one for the other.

For long-term memory, the v1 KnowledgeBank stored manipulation snippets keyed by semantic similarity and appended new entries without arbitration. v2 attaches metadata to every entry — quality, confidence, lifecycle state, verifier identity, and conflict pointers to entries that contradict this one — and the retrieval policy weighs those fields against the query rather than returning the top-k semantic match. The Terminal-Bench 2.0 and SWE-Bench Verified evaluations exercise this memory module as a general agent-memory layer (head-to-head with ReasoningBank), distinct from the robot-manipulation framing the rest of the paper uses.

Results

GeoFuse-MV3D on GSO-30 vs MV-SAM3D baseline: CD −2.20%, LPIPS −2.02%, PSNR +2.36%, SSIM +1.03% [Abstract].
KnowledgeBank on Terminal-Bench 2.0 vs ReasoningBank: SR +4.53%, AS −4.95% [Abstract].
KnowledgeBank on SWE-Bench Verified vs ReasoningBank: resolve rate +3.73%, AS −5.65% [Abstract].

Only abstract-level numbers are retrievable at filing time. The paper does not, in its abstract, report end-to-end manipulation-success numbers tying both contributions together on a robot benchmark — the two contributions are evaluated on disjoint test suites (GSO-30 for perception, Terminal-Bench/SWE-Bench Verified for memory).

Why it’s interesting

The reconstruction half is a concrete attempt to address the perception-failure thread the wiki has been tracking via VLM Perception Failures and the sensorimotor counter-position in VLA Models — specifically the claim that monocular SAM3D-style geometry is unreliable enough to corrupt downstream planning when multi-view RGB-D is available — and it lines up with the perception-rich + heuristic-policy direction GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation takes (PointNet++ over a 3-keypoint articulated-axis abstraction) by similarly betting that an explicit geometric prior beats raw neural reconstruction. The memory half is more unusual: evaluating an agent-memory upgrade on Terminal-Bench 2.0 and SWE-Bench Verified rather than a robot benchmark positions KnowledgeBank as an Agentic Software Engineering / Tool-Use Agents contribution — sibling to the experiential-memory work surveyed in Memory in the Age of AI Agents (whose taxonomy explicitly names quality, conflict, and lifecycle as first-class memory-evolution operations) and to the iFlow-style external-memory pattern noted in Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem. The Terminal-Bench / SWE-Bench framing is a strong hint that the authors see governed experiential memory as a portable layer that should help any long-horizon agent, not just a robot planner.