ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory

TL;DR

ABot-AgentOS (Alibaba) is a deliberative “agent operating system” layer that sits above low-level VLM/VLA controllers on a robot, providing scene-conditioned planning, context-isolated skill execution, multi-stage verification, persistent multi-modal memory, and edge-cloud collaboration. The paper’s two load-bearing pieces are (i) a Universal Multi-modal Graph Memory that converts dialogue, visual observations, spatial context, temporal relations, and task traces into typed nodes and edges as a persistent source-grounded substrate, and (ii) a failure-driven self-evolution loop that turns diagnosed memory failures into gated “evo-assets” promoted only to later evaluation splits (a leakage-prevention design). It is evaluated on a new executable benchmark, EmbodiedWorldBench (16 scenes × 4 difficulty levels × 200+ tasks with navigation, object search, NPC dialogue, dynamic events, and trace-grounded scoring), plus off-the-shelf memory benchmarks (LoCoMo, OpenEQA, Mem-Gallery, NExT-QA) where ABot-AgentOS reports strong static numbers and modest self-evolution gains.

Key claims

ABot-AgentOS is positioned as a general Agent OS layer above low-level controllers, providing scene-conditioned planning, context-isolated skill execution, multi-stage verification, multi-modal memory, and edge-cloud collaboration — framed as the runtime that long-horizon embodied agents need beyond the perception/action bindings that VLM and VLA systems already provide [Abstract].
EmbodiedWorldBench is the paper’s companion executable benchmark: 16 indoor / outdoor / hybrid scenes, four difficulty levels, and over 200 tasks spanning navigation, object search, NPC dialogue, dynamic events, and trace-grounded scoring [Abstract].
Universal Multi-modal Graph Memory is a persistent, source-grounded substrate that converts dialogue, visual observations, spatial context, temporal relations, and task traces into typed nodes and edges — a graph rather than vector-index or text-record substrate [Abstract].
A failure-driven self-evolution loop converts diagnosed memory failures into gated runtime “evo-assets” that are only promoted to later evaluation splits, explicitly preventing current-split ground-truth leakage while allowing continual improvement [Abstract].
On an initial EmbodiedWorldBench subset, ABot-AgentOS improves over a single-controller baseline on both task success and goal completion [Abstract].
Static memory benchmarks: LoCoMo 87.5, OpenEQA EM-EQA 59.9, Mem-Gallery 88.6, NExT-QA Acc@All 76.5 [Abstract].
Self-evolution further improves LoCoMo to 88.7, OpenEQA to 60.4, and Mem-Gallery to 89.0 — small but consistent lifts over the static configuration [Abstract].

Method

ABot-AgentOS is presented as a runtime layer rather than a monolithic model: a deliberative agent sits above whatever perception/action controllers the robot already has, and provides the scaffolding that turns them into long-horizon behavior. The load-bearing services are (a) scene-conditioned planning (a plan is grounded in the currently-perceived scene, not a symbolic goal graph), (b) context-isolated skill execution (each skill runs in an isolated context so that inter-skill interference and prompt-pollution are bounded), (c) multi-stage verification (skill invocations are checked at multiple stages rather than only on final outcome), (d) multi-modal memory (the graph substrate below), and (e) edge-cloud collaboration (the deliberative loop can be split across a device-side controller and a cloud-side reasoner).

The Universal Multi-modal Graph Memory ingests dialogue turns, visual observations, spatial context, temporal relations, and task traces, and converts them into typed nodes and edges. The paper calls this a “persistent source-grounded substrate”, suggesting every node points back to the raw modality it came from (an image frame, a dialogue turn, a trace step) — this is what makes the memory auditable and what supports the multi-stage verifier’s ability to trace a claim back to evidence. The self-evolution loop closes over this substrate: when the verifier flags a memory failure, the diagnosis is materialized as a “runtime evo-asset” (a specific patch to how memory is written or retrieved). To prevent the loop from silently learning the current split’s ground truth, evo-assets are gated — they only take effect on later evaluation splits.

EmbodiedWorldBench is the companion benchmark: an executable simulator with 16 scenes across indoor/outdoor/hybrid categories, four difficulty levels, and 200+ tasks. Scoring is trace-grounded (i.e. scored over the whole execution trajectory, not just terminal state), which is what makes the multi-stage verification and dynamic-event components measurable.

Results

EmbodiedWorldBench (initial subset): ABot-AgentOS improves over a single-controller baseline on both task success and goal completion (magnitudes not stated in abstract) [Abstract].
LoCoMo (long-conversation memory): 87.5 (static) → 88.7 (self-evolution, +1.2) [Abstract].
OpenEQA EM-EQA (embodied question answering, exact-match subset): 59.9 (static) → 60.4 (self-evolution, +0.5) [Abstract].
Mem-Gallery: 88.6 (static) → 89.0 (self-evolution, +0.4) [Abstract].
NExT-QA Acc@All (video QA): 76.5 (static; no self-evolution number reported) [Abstract].

Only abstract-level numbers are retrievable at filing time; the EmbodiedWorldBench full-split table, per-difficulty breakdown, and per-skill ablations sit inside the PDF. The abstract explicitly notes the improvements are on an “initial” EmbodiedWorldBench subset, so headline embodied numbers should be read as preliminary.

Why it’s interesting

ABot-AgentOS is the paper-form counterpart to DimOS — The Agentive Operating System for Physical Space (v0.0.13) — both stake out the deployment-plane layer between VLA controllers and physical execution, but from opposite sides: DimOS is a Python SDK + MCP surface for developers to compose runtime blueprints, while ABot-AgentOS is a research-paper claim that the same layer itself is a research artifact worth benchmarking (via EmbodiedWorldBench). The Universal Multi-modal Graph Memory sits in the same design lane as GeneralVLA-2: Geometry-Aware Reconstruction and Governed Memory for Robot Planning‘s upgraded KnowledgeBank — both explicitly move agent memory from bare vector-similarity into a governed substrate with per-entry provenance and lifecycle, matching the Dynamics × Functions × Forms taxonomy Memory in the Age of AI Agents argues is what agent memory needs to become; the graph-typed substrate here is a stronger commitment on the Forms axis than KnowledgeBank’s metadata-annotated store. The self-evolution loop’s leakage-gated promotion mechanism is a concrete answer to the benchmark-hacking failure mode surfaced by RoboArena rolls back evaluations after benchmark hacking observed since April (Pranav Atreya announcement) (RoboArena rolling back evaluations after benchmark hacking was observed).