ARC: A Reasoning Recipe for Robot Foundation Models

ListenTeaching Robots to Think Between Actions10:13

Show notes

ARC is a reasoning-recipe wrapper that teaches existing robot foundation models like π0.5 and Cosmos3-Nano-Policy to reason about actions — no new robot demonstrations, no foundation-scale training. Reasoning traces are auto-recovered from the DROID corpus into ARC-Trace-DROID, and the policy is fine-tuned to act conditioned on those traces, with a counterfactual margin loss forcing it to actually use them. We dig into the headline numbers — including a 9.5% to 91.7% real-robot jump for π0.5 + ARC — the async deployment loop, and what this means for how we think about reasoning in generative models. **Sources** - ARC: A Reasoning Recipe for Robot Foundation Models (blog)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

ARC is a reasoning-recipe wrapper for existing robot foundation models (π0.5 and Cosmos3-Nano-Policy) that teaches them action-grounded reasoning without new robot demonstrations and without foundation-scale training. Reasoning traces are auto-recovered from the DROID corpus (~75K episodes / ~1.2M action-aligned frames) into a dataset called ARC-Trace-DROID. The pretrained model is then fine-tuned to predict demonstrated actions conditioned on causal traces, observations, and instructions; the vision-language backbone, trace encoder, and action expert are jointly fine-tuned, with factual examples trained via flow matching and contradictory traces receiving a counterfactual margin loss. Reported wins include up to +50 percentage points on RoboLab-Reasoning-50 and a jump from 9.5% → 91.7% real-robot success for π0.5 + ARC. A background VLM refreshes the reasoning trace asynchronously from current camera views while 15-action chunks execute at 15 Hz.

Key claims

ARC improves pretrained robot policies through action-grounded reasoning without new robot demonstrations and without foundation-scale training; the trace explains the next action and what it should accomplish, and the policy learns to use that explanation to choose actions [§Explain the next action].
RoboLab-120 default-instructions zero-shot: Cosmos3-Nano-Policy 36.8% → 48.8%, π0.5 28.0% → 45.3% [§RoboLab-120].
Gains of up to 50 percentage points on RoboLab-Reasoning-50 and new state-of-the-art on RoboLab-120 and MolmoSpaces for both π0.5 and Cosmos3-Nano-Policy [§headline].
Real-robot π0.5 + ARC reaches 91.7% success, up from 9.5% for the base π0.5 under matched conditions and with no additional hardware-specific fine-tuning [§Real-robot].
ARC policies are less sensitive to instruction detail: performance stays strong across vague / default / specific instructions, while base policies degrade as instructions become less explicit [§Instruction sensitivity].
Reasoning data is recovered automatically from DROID: ~75K episodes and ~1.2M action-aligned frames become ARC-Trace-DROID without new teleoperation [§Recover reasoning].
Fine-tuning jointly updates the VL backbone, trace encoder, and action expert; factual traces use flow matching, contradictory traces receive a counterfactual margin loss [§Fine-tune].
Compute efficiency on RoboLab-120 (specific): Cosmos3-Nano-Policy + ARC reaches the instruction-only baseline’s 10K-iteration performance with approximately 4.3× fewer training updates and continues improving [§Compute].
Sampling-step reduction: π0.5 + ARC comes within 0.4 percentage points of its ten-step base policy using one Euler step; Cosmos3-Nano-Policy + ARC exceeds its four-step base with two UniPC steps [§Solver steps]. The reduction is in action generation only — reasoning and context encoding remain separate costs.
Deployment loop runs two rates: a background VLM refreshes reasoning from the latest camera views (example rollout refreshes at ~1 Hz) while the policy executes 15-action chunks at 15 Hz against the latest validated trace [§Deployment].

Method

ARC is a three-stage pipeline on top of an existing robot foundation model. Stage 1 automatically recovers reasoning traces from an existing demonstration corpus — here DROID at ~75K episodes / ~1.2M action-aligned frames — producing ARC-Trace-DROID. Each trace connects current state to a cause, a possible consequence, and an intended effect, specifies the next action, what to avoid, and whether the task is complete. No new robot trajectories are collected.

Stage 2 fine-tunes the pretrained model to predict demonstrated actions conditioned on causal traces + observations + instructions. The fine-tuning objective jointly updates the vision-language backbone, a trace encoder, and the action expert. Factual examples (observed demonstrations paired with coherent recovered traces) are trained with flow matching; contradictory traces receive a counterfactual margin loss that penalizes the model for continuing to act consistently with a trace that was deliberately swapped out of context. The recipe is tailored per architecture — the same recipe is applied to π0.5 and Cosmos3-Nano-Policy without architectural changes.

Stage 3 is the deployment loop. A background VLM continually refreshes the reasoning trace from the latest camera views at approximately 1 Hz in the example rollout; the action policy consumes the most recent validated trace and executes 15-action chunks at 15 Hz. Reasoning refresh and robot control run at independent rates — the policy never waits for the reasoner, and the reasoner never blocks on the policy.

Results

RoboLab-120 (default instructions, zero-shot): Cosmos3-Nano-Policy 36.8% → 48.8%, π0.5 28.0% → 45.3% [§RoboLab-120].
RoboLab-Reasoning-50: gains of up to 50 percentage points over the respective base policies [§headline].
Real-robot π0.5: 9.5% → 91.7% with no additional hardware-specific fine-tuning [§Real-robot].
Compute efficiency: on RoboLab-120 (specific instructions), Cosmos3-Nano-Policy + ARC reaches the instruction-only baseline’s 10K-iteration score with approximately 4.3× fewer updates and continues improving beyond that [§Compute].
Few-step sampling: π0.5 + ARC is within 0.4 pp of its ten-step base at one Euler step; Cosmos3-Nano-Policy + ARC beats its four-step base with two UniPC steps [§Solver steps].
Instruction robustness: ARC policies maintain performance across vague / default / specific instructions; base policies degrade as instructions become less explicit [§Instruction sensitivity].
Ablations vary (a) the external reasoner, (b) the trace content, (c) the fine-tuning strategy — the report flags all three as load-bearing.

Why it’s interesting

ARC is the sharpest filed instance to date of a recipe-wrapper that extends existing robot foundation models with action-grounded reasoning without new robot data or foundation-scale pretraining. On the VLA Models recipe-lever board it sits in the “wrap frozen (or lightly adapted) base policy with a reasoning companion” cluster — comparable to FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models FailSafe (10-step polling companion VLM emitting executable deltas on failure) and to Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Harness VLA (LLM planner over frozen π0.5 primitives), but distinct in two ways: the companion reasoner runs every tick via async refresh rather than only on failure or at subtask boundaries, and the action policy itself is fine-tuned to consume the trace rather than left frozen. The 9.5% → 91.7% real-robot jump on π0.5 is the sharpest filed single-recipe gain over base π0.5 on this concept page.

For Thinking with Modalities, ARC extends the text-tokens-with-grounded- references lineage pioneered by Robotic Control via Embodied Chain-of-Thought Reasoning ECoT: reasoning stays as text (cause / consequence / intended effect / next action), but is auto-recovered rather than Gemini-synthesized, and the paired counterfactual margin loss is a new mechanism for enforcing that the policy actually uses the trace rather than treating it as a decorative prefix. The async deployment loop (reasoner at ~1 Hz, policy at 15 Hz) also directly addresses the ECoT inference-cost finding that reasoning blows up per-step latency 50× — ARC’s async refresh is the recipe-level answer that ECoT’s synchronous reuse / asynchronous execution tricks pointed toward.

For Synthetic Training Data, ARC-Trace-DROID is a labels-on-real-video pipeline specifically for reasoning-trace supervision, auto-recovering 75K episodes / 1.2M frames of causal traces from an existing demonstration corpus — sibling in framing to the Macrodata WGO-Bench / Action100M VLM-labeling line but with the labels being reasoning traces rather than captions or segmentation boundaries.