Describe Anything Anywhere At Any Moment

TL;DR

DAAAM (Describe Anything, Anywhere, at Any Moment) is a real-time, large-scale spatio-temporal memory framework from MIT SPARK Lab (Gorlo, Schmid, Carlone, accepted to CVPR 2026) that constructs hierarchical 4D scene graphs from RGB-D streams and exposes them to an LLM tool-calling agent for embodied question answering. The system combines a Hydra-style metric-semantic mapping front-end with a novel optimization-based frame-selection step that batches localized captions from the Describe Anything Model (DAM), claiming an order-of-magnitude speedup over per-segment inference. Nodes (objects, places, regions) are continuously optimized via factor-graph backend, merged across time, and exposed through a tool API for downstream reasoning. Headline numbers: +53.6% question accuracy, −21.9% position error, −21.6% temporal error on the new OC-NaVQA benchmark, and +27.8% sequential task-grounding accuracy on SG3D, while running at 10 Hz sensor rate. Code is open-source (MIT-SPARK/DAAAM) with a ROS 2 interface; arXiv 2512.00565.

Key claims

The core tradeoff in embodied spatio-temporal memory — rich open-vocabulary descriptions vs. real-time 3D grounding — can be sidestepped by batch-processing localized captions over selected frames rather than per-segment inference [§Abstract, §3].
An optimization-based frame-selection front-end speeds up localized captioning by an order of magnitude, making real-time use of large VLM captioners viable for online robot memory [§Abstract, §3.A].
The system builds a hierarchical 4D scene graph with detailed, geometrically grounded descriptions while maintaining real-time performance on sensor input [§Abstract, §3].
Places are derived by convolving a robot bounding box with the local volumetric occupancy map (from Khronos’ active window), squashing along Z, and tessellating the resulting slice into “largest inscribed traversable rectangles” — yielding a region layer above the object layer [§3.D].
A factor-graph backend continuously optimizes the positions of all scene-graph nodes for spatial global consistency; for temporal consistency, object fragments and place nodes with similar geometry and descriptive features are merged in a reconciliation step, with description histories preserved with timestamps [§3.E].
The 4D SG interfaces with a tool-calling LLM agent for inference and reasoning, with tools designed to query the SG along spatial, semantic, and temporal axes [§Abstract, §4].
On OC-NaVQA (a new extended large-scale long-time evaluation benchmark curated by the authors), DAAAM achieves +53.6% question accuracy, −21.9% position error, and −21.6% temporal error over the most competitive baselines [§Abstract, §5].
On SG3D (sequential task grounding), DAAAM improves task-grounding accuracy by +27.8% over the most competitive baselines [§Abstract, §5].
DAAAM achieves real-time operation at the CODa sensor rate (10 Hz) via threaded batch-inference of large captioning models, with the authors noting that smaller captioners can be substituted for tighter hardware/latency budgets (e.g. aerial robots, VR) [§5, Table 6].
The system is positioned explicitly as building on prior MIT-SPARK Hydra / Khronos / 3D-dynamic-scene-graph line, with object fragments tracked over time using SAM segmentation + BoT-SORT-style tracking before semantic lifting [§3, GitHub README].

Method

DAAAM takes RGB-D video and per-frame poses as input and incrementally constructs a hierarchical 4D scene graph in three stages. Front-end metric-semantic mapping segments the scene into object fragments and tracks them across frames using a SAM-style segmenter plus a BoT-SORT-derived tracker, accumulating per-fragment observations into Khronos’ active window. Optimization-based frame selection is the paper’s main novelty: rather than caption every observation of every fragment, an optimization picks a small subset of frames per fragment that jointly maximize observation diversity and visibility quality, batching them for the Describe Anything Model (DAM) — a localized open-vocabulary captioner — to produce one detailed description per object. The descriptions are written back into the scene-graph nodes. Global optimization and region clustering then run a factor-graph backend that optimizes all node positions for global spatial consistency, reconciles temporally drifting duplicates (merging fragments whose geometry and descriptors match while preserving a timestamped description history), and clusters the resulting object/place layer into semantically informed regions. The output is a four-layer hierarchical SG (regions → places → objects → fragments) augmented with per-node textual descriptions and a temporal history.

At inference time a tool-calling LLM agent is given a natural-language question and can issue typed queries against the SG (spatial filters by region/place, semantic filters by description, temporal filters by timestamp/event). The authors evaluate on two existing benchmarks (NaVQA, SG3D) and curate a new extended benchmark (OC-NaVQA) for long-horizon and large-scale evaluation. The system runs at 10 Hz on CODa, with the slow path (large-VLM captioning) decoupled onto a separate higher-latency thread.

Results

OC-NaVQA (new extended large-scale long-time benchmark): +53.6% question accuracy, −21.9% position error, −21.6% temporal error over the most competitive baselines [§Abstract, §5].
SG3D (sequential task grounding): +27.8% task-grounding accuracy over the most competitive baselines [§Abstract, §5].
NaVQA (spatio-temporal question answering): top scores [§Abstract, §5].
Throughput: 10 Hz on CODa sensor input via threaded batch-inference of large models, with smaller captioners feasible for tighter budgets [§5, Table 6].
The batch-processing speedup over per-segment captioning is reported as an order of magnitude (claimed in abstract; the actual ratio is in the paper’s frame-selection ablation, not in the fetched excerpts).
Code is open-source at MIT-SPARK/DAAAM with a separate MIT-SPARK/DAAAM-ROS interface; data is released.

Why it’s interesting

DAAAM is the most concrete current example of the explicit-scene-graph answer to the spatial-supersensing problem Cambrian-S: Towards Spatial Supersensing in Video articulates from the MLLM side. Cambrian-S argues that long-horizon spatial QA needs predictive-world-model machinery grafted onto a video MLLM; DAAAM argues that the same capability emerges if you build a real-time 4D scene graph alongside the perception stack and expose it as a tool-callable database to a frozen LLM. The two papers don’t share benchmarks (Cambrian-S uses VSI-Super, DAAAM uses OC-NaVQA / SG3D), but they’re directly competing recipes for the same underlying capability — “answer questions about objects, places, and events in a 2-hour walkthrough” — and which one transfers better to deployed robots is an open empirical question the wiki will need to track.

The memory architecture is also a clean external-symbolic counterpoint to the in-state and in-weights memory recipes the Parametric memory cluster has been tracking: STEM puts factual storage in token-indexed embedding tables, Raven puts it in a slot-routed recurrent hidden state, the geometric-memory work puts it in emergent weight geometry — DAAAM puts it in an explicit, queryable, version-controlled hierarchical SG that an LLM accesses via tool calls. Under Memory in the Age of AI Agents‘s Forms × Functions × Dynamics taxonomy this is external-structural + factual/experiential + formation+evolution-heavy — orthogonal to the recipes already cataloged under Tool-Use Agents (Training-Free GRPO’s natural-language experience library; MiroThinker’s 256K context window). And the tool-calling-over-SG pattern is the same agent loop the rest of Tool-Use Agents uses, but with a geometrically grounded tool surface rather than an API/code/web surface.