Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots

TL;DR

Embodied.cpp is a portable C++ inference runtime for embodied AI models (VLAs and World-Action Models) that runs on heterogeneous edge hardware (CPU / CUDA GPU / NPU) from GGUF weights, in the spirit of llama.cpp but specialized for the runtime contract embodied deployment actually needs: multi-rate execution inside closed-loop control, latency-first batch-1 inference, and I/O beyond fixed language tokens. The paper argues existing request-response inference servers (built for LLM serving) don’t fit this contract, and proposes a five-layer architecture — input adapters, sequence builders, backbone execution, head plugins, deployment adapters — factored from an analysis of representative VLAs and WAMs. Evaluated on two VLAs (HY-VLA, π0.5) and a preliminary WAM benchmark using a Causal World Modeling for Robot Control (LingBot-VA) Transformer block: 100.0% and 91.0% closed-loop task success and a WAM block-memory reduction from 312.2 MiB to 88.1 MiB. The release is the first VLA/WAM equivalent to llama.cpp on file, filling a serving-side gap between “trained VLA” and “policy running on a robot” that the recent DimOS — The Agentive Operating System for Physical Space (v0.0.13) deployment-runtime entry called out.

Key claims

• Existing inference runtimes (vLLM / SGLang / llama.cpp) target request-response LLM serving and therefore do not satisfy the runtime contract of embodied deployment: multi-rate execution inside closed-loop control, latency-first batch-1 inference on heterogeneous hardware, and extensible embodied interfaces beyond fixed token I/O [Abstract].
• Representative VLA models and World-Action Models share a common execution path that decomposes into five layers — input adapters, sequence builders, backbone execution, head plugins, and deployment adapters — which is what Embodied.cpp materializes as the runtime contract [Abstract].
• Closed-loop deployment on two VLA models achieves 100.0% task success rate for HY-VLA and 91.0% for π0.5 [Abstract].
• On a preliminary WAM benchmark using a LingBot-VA Transformer block, the runtime reduces block memory from 312.2 MiB to 88.1 MiB [Abstract] (~3.5× reduction).
• One backend abstraction covers CPU, CUDA GPU, and NPU deployment paths; the released build supports pi0.5, HY-VLA, and LingBot-VA today, with a documented roadmap for OpenVLA, Octo, Hi Robot, GeneralVLA, GR00T N1, Fast-in-Slow, UniPi, WorldVLA, DreamZero, FastWAM, Cosmos Policy, UWM, and Being-H0.7 [GitHub README §1.1].
• Serving surface is a ZeroMQ/Protobuf boundary rather than an HTTP/gRPC serving stack — one process per model (vla-pi05-server, vla-hy-vla-server, wam-lingbot-server), each fed GGUF weights [GitHub README §2.5].

Method

The paper’s central move is treating embodied deployment as a runtime-contract problem rather than a model-serving problem: rather than adapting a request-response LLM server (batch-N throughput-first, HTTP boundary, fixed text-token I/O) to VLA/WAM inference, it defines the contract from the robot side — closed-loop control ticks at a fixed rate, batch is always 1 at inference time, latency dominates throughput, and inputs/outputs include images, proprioception, action chunks, and (for WAMs) latent video streams. The authors survey the open-source VLA/WAM zoo (AR-token VLAs, VLM-backboned VLAs, hierarchical VLAs, asynchronous VLAs, predict-then-act WAMs, unified AR-modeling WAMs, shared-backbone WAMs, latent-space WAMs) and extract a shared five-layer decomposition: input adapters normalize sensor / simulator payloads, sequence builders assemble the model’s tokenized/vectorized context, backbone execution runs the shared transformer (mostly reusing llama.cpp / GGML kernels), head plugins compute the model-specific action or world-frame outputs, and deployment adapters translate outputs back to the simulator or robot. Backbone execution runs on a single backend abstraction that spans CPU (verified on Apple Silicon for π0.5), CUDA GPU (with custom kernels in kernels/), and NPU. Weights arrive as GGUF, produced either from the released HuggingFace collection or from the paper’s own conversion scripts (convert_pi05_to_gguf.py, convert_hy_vla_to_gguf.py, convert_lingbot_va_to_gguf.py), with quantization helpers for HY-VLA and LingBot-VA. Servers communicate with client-side evaluation harnesses (LIBERO, RoboTwin, real robots) via ZeroMQ + Protobuf, which lets the same policy binary drive a native C++ simulator or a Python-side robot stack without model-side changes. Evaluation is closed-loop task success on VLAs and block-level memory footprint on WAMs; the preliminary WAM benchmark is a single Transformer block from LingBot-VA rather than the full unified world+action model, i.e. a memory-side sanity check rather than a policy-side result.

Results

• VLA closed-loop deployment: HY-VLA reaches 100.0% task success and π0.5 reaches 91.0% task success on the reported simulator tasks (LIBERO for π0.5, RoboTwin for HY-VLA) [Abstract]. Success rates are relative to the same VLA architectures’ baseline PyTorch implementations, i.e. the runtime preserves policy quality — it is not a distillation or compression result.
• WAM memory footprint: The LingBot-VA Transformer block drops from 312.2 MiB to 88.1 MiB, ~3.5× reduction [Abstract]. This is a per-block figure on a preliminary benchmark, not full-model end-to-end memory.
• Deployment surface: single-binary servers per model, GGUF weight artifacts on HuggingFace, and simulator drivers for LIBERO (5 suites × 10–90 tasks) and RoboTwin (dual-arm), plus a real-robot path via ZeroMQ [README §2.6, §3].
• Portability: the release documents CPU-only builds on Linux and macOS Apple Silicon, and CUDA GPU builds; NPU support is claimed in the paper (backend abstraction) but not detailed in the README at filing time.

Why it’s interesting

Embodied.cpp is the first filed wiki artifact of a llama.cpp-shaped inference runtime for VLAs and WAMs — a serving-side stack that pairs GGUF weights, a single backend abstraction across CPU/CUDA/NPU, and multi-rate closed-loop execution. That directly fills the gap the recently-filed DimOS — The Agentive Operating System for Physical Space (v0.0.13) entry described between “trained VLA” and “policy running on a robot”: DimOS owns the message-passing plane (typed streams, transports, module lifecycle) while Embodied.cpp owns the model plane (kernels, weights, backbone execution, quantization) — the pair is a plausible open-source stack for the deployment surface that closed offerings (Inside NVIDIA Halos for Robotics: A Full-Stack Functional Safety System for Physical AI Halos, NVIDIA Isaac) target. It also lands a concrete data point for the LLM Inference Efficiency concept in a new regime — batch-1 latency-first serving with mixed image / action / language / world-latent I/O — where the LLM-serving optimizations logged so far (SpecBundle, TileRT, MegaScale-MoE) don’t directly translate. And the runtime’s willingness to treat both VLAs (per VLA Models) and world-action models (per World Foundation Models) as a single execution class is a small piece of evidence for the recent Reuss/µ₀/Cosmos Policy line that world-model peers and action policies should share deployment infrastructure — separate models, one runtime contract.

See also

• DimOS — The Agentive Operating System for Physical Space (v0.0.13) — sibling deployment-runtime entry that owns the message-passing/robot-connectivity plane; Embodied.cpp owns the model-inference plane below it.
• Causal World Modeling for Robot Control (LingBot-VA) — LingBot-VA is one of the three models Embodied.cpp ships with support for, and the WAM benchmark in this paper uses a LingBot-VA Transformer block.
• Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack — Hunyuan HY-VLA is one of the three supported models; the 100.0% RoboTwin closed-loop result reported here is a serving-side data point complementing the modeling-side report.
• TileRT: Tile-Based Runtime for Ultra-Low-Latency LLM Inference — closest LLM-serving analog on file: tile-based single-GPU runtime specialized for a specific model regime (large-MoE decode), whereas Embodied.cpp specializes for the VLA/WAM regime.
• Scale Robot Policy Evaluation with Ray (Distributed Sim-Eval on Anyscale) — complementary serving-side entry that measured cold-start latency, action-chunking amortization, and disaggregation constraints for VLA policies at the Ray + Anyscale level; Embodied.cpp is the C++/GGUF layer that would sit inside such a fleet.
• RAIDEN: A Toolkit for Policy Learning with YAM Bimanual Robot Arms — data-collection toolkit at the training end of the VLA pipeline; Embodied.cpp is the mirror-image entry at the deployment end.
• VLA Models — new candidate serving-runtime dimension on the concept page.
• World Foundation Models — WAMs share a runtime contract with VLAs under this paper’s five-layer factoring.
• LLM Inference Efficiency — batch-1 latency-first serving is a distinct regime from the throughput-first LLM serving logged elsewhere.