Isaac 0.5: An Open-Weight Embodied Foundation Model

TL;DR

Isaac 0.5 is Perceptron AI’s open-weight 36B-parameter embodied foundation model that unifies video perception, embodied reasoning, and robot control on a single sparse backbone. Its central bet is that inexpensive action-free video, scaled to one million hours alongside egocentric and UMI handheld-gripper video, can substitute for expensive teleoperation on a well-calibrated action-loss metric — by a factor of 210× in the headline within-grid measurement. Two structural contributions travel with the model: null-expert routing (each token uses 0–8 of 256 routed experts as needed) and a proprietary self-supervised “future-percept prediction” objective that predicts semantic content of future observations rather than reconstructing pixels. Weights, inference stack, evaluation settings, and deployment configurations for 35+ embodiments are released alongside the paper.

Key claims

Isaac 0.5 unifies four output surfaces on a single 36B sparse backbone: language, grounding, discrete FAST action tokens, and continuous actions through a dedicated Flow expert + DiT that cross-attends to backbone states [§1, §3.1, §3.3, Fig. 3].
Null-expert routing duplicates one learned null-router row into 256 tied null copies alongside the 256 real-expert scores; top-8 selection over the 512 candidates picks kreal = min(8, j) real experts where j is the count of real-expert scores strictly above the null score, so each token executes 0–8 routed MLPs while the full 256-expert library remains available [§3.2, Eq. 2, Fig. 3b].
The multiplicity of 256 null copies (rather than 8) matters through the softmax partition function — it contributes 256 · exp(u∅) and equivalently shifts the null-path initial probability mass by + log 256 without adding parameters [§3.2].
Scaling general action-free video from 1,000 to one million hours (at a fixed 80:30:30 general:egocentric:UMI ratio) moves the within-grid teleoperation crossing for a well-calibrated offline action-loss threshold from 5,884 hours to 28 hours — a 210.3× reduction; the adjacent measured teleoperation rungs bound the ratio at 83× to 300× [Abstract, §1, §4].
The video-vs-teleop exchange is conditional, not uniform: the empirical loss-vs-video slope is nearly flat at 1 teleoperation hour and stabilizes near −0.21 loss per 10× video increase from roughly 100 teleoperation hours onward [Abstract].
The 529-source data mixture splits scheduler mass ≈70/30 between perception+reasoning and robotics, but packed-token exposure reverses to ≈20/80 because robotics examples pack into more positions per draw — scheduler mass and packed exposure answer different questions and reporting one hides the other [§2.1, Table 1].
The self-supervised future-percept prediction objective predicts task-relevant visible state changes (object grasped, drawer open, contact about to occur) rather than future pixels, from observation history through time t against target zt+∆; the target-generator and loss implementation remain proprietary [§3.4, Eq. 3].
Egocentric annotations produced via Perceptron’s proprietary Mk1 model preserve a three-level hierarchy — recording / positive activity segment / per-hand frame-level instance — with left/right hands labeled independently for visible action, object, contact, and visibility; per-hand records preserve division of labor across bimanual tasks that collapsing to one frame-label would destroy [§2.2, Fig. 2].
Idle-span filtering compacts sustained idle runs only for robot sources with a validated idle signal, per-source rather than by a global motion threshold — held poses, mechanical settling, and contact-rich micro-motion can remain informative even when visual stillness is high; the data record keeps recorded and optimizer-consumed robot hours separate [§2.3].
Reported system metrics: end-to-end MFU is 24% in the released hyper-sparse setting (2.5B of 36B active per token at the 50% null reference where 4 of 8 routes are real), and 48% in an unreleased dense setting [Abstract, §1].
One checkpoint executes the same requested task on different embodiments (YAM and SO-101 cup stacking) by changing only the embodiment string in the configuration prompt [Fig. 4].

Method

Isaac 0.5 builds on an open Qwen-family vision-language recipe with a 2,048-wide 40-block sparse backbone (30 GDN linear-attention blocks + 10 full-attention blocks). Every block carries 256 real routed MLPs, one learned null-router row expanded into 256 null copies, and an always-on shared expert. The autoregressive route emits language, grounding, and 2,048-token FAST discrete actions from shared hidden states. The continuous-control route projects backbone states 2,048 → 768 as cross-attention K/V for a 36-block DiT that combines context with a noised action chunk and Flow time to predict a velocity field integrated into a continuous action chunk.

Training jointly optimizes the proprietary self-supervised future-percept objective (Eq. 3), autoregressive next-token loss (which subsumes FAST action tokens, Eq. 5), and continuous-action linear conditional Flow Matching (S=4 noise/flow-time samples per action chunk). The data mixture spans 529 streams across two accounting components: perception+embodied reasoning (3T native tokens, 69.7% scheduler mass) and robotics (100K source-hours across 35+ embodiment configurations, 30.3% scheduler mass but 79.6% of packed tokens). Continuous features are normalized per (dataset × scope × objective) using a 1st/99th-percentile linear map with a collapsed-quantile fallback for sparse contact channels and a |ã|>20 chunk-drop threshold.

The action-vs-teleop scaling experiment fixes the general:egocentric:UMI video ratio at 80:30:30 and sweeps all three streams against teleoperation, measuring an offline action-loss threshold τ=2.50 on a well-calibrated evaluation grid.

Results

Headline scaling result: at fixed 80:30:30 video composition, going from 1,000 to 1,000,000 hours of general video moves the teleop crossing at τ=2.50 from 5,884 h to 28 h — a 210.3× reduction, bounded by the adjacent measured teleop rungs at 83× to 300× [Abstract, §4].
Per-rung slopes stabilize near −0.21 loss per 10× video after ~100 teleop hours; below that the slope is nearly flat, so video and teleop are complements rather than substitutes at the low end [Abstract].
End-to-end MFU: 24% in the released hyper-sparse configuration (2.5B active of 36B), 48% in a dense configuration whose recipe and weights are not released [Abstract, §1].
Same checkpoint executes cup-stacking on YAM and SO-101 by changing only the embodiment string — three phases (initial reach, intermediate placement, completed stack) shown side-by-side [Fig. 4].
No headline benchmark comparison against other VLAs (e.g. π0.5, XR-1, Dyna-2, GEN-1.5, µ₀) is included in the released body text.

Why it’s interesting

Isaac 0.5 stakes out an action-free-video-scaling row on the VLA Models recipe-lever board — a sharper, quantitatively-calibrated cousin of Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models (Dyna-2’s million-hour scaling law over egocentric human video) and of HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining (HumanScale’s egocentric-beats-teleop at matched volume) — with the twist that general action-free video, not just egocentric, is the primary scaling axis and its exchange with teleop is calibrated on an offline loss surface rather than downstream success. That framing directly contests GEN-1.5: Embodied Foundation Models are One-Shot Learners GEN-1.5’s from-scratch-real-interaction-only position and Going Beyond World Models & VLAs Pete Florence’s “VLM/video pretraining is a crutch” thesis, while composing with Causal World Modeling for Robot Control (LingBot-VA) LingBot-VA’s unified-video+action pretraining and Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training XR-1’s UMI-hour scaling.

The null-expert routing contribution is the sharpest deployed instance yet of variable-per-token routed compute on the MoE Routing Design design-space board — closer to the token-adaptive spirit of DynaMoE (DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks) and Routing-Free MoE (Routing-Free Mixture-of-Experts) than to Path-Constrained MoE or DeepSeek-V3-style fixed-topK, and the multiplicity trick (256 null copies rather than 8, to shift the softmax partition function without extra parameters) is a router-design detail worth remembering.

Finally, the paper is the load-bearing counterpart to Perceptron AI’s earlier product-plane filings — Introducing Perceptron Mk1 (Mk1 as a VLM-tier data engine) and Introducing Perceptron Egocentric API (Egocentric API annotation pipeline) — which are cited internally as the annotation stack that produced Isaac 0.5’s egocentric supervision. Mk1 was chosen over Gemini on cost/throughput grounds for the production annotation pass; that pipeline choice, made downstream of Perceptron’s own model, is now embedded in the training data of a released open-weight embodied foundation model.