Dmytro Hrybov — OpenAI Decisions API (GPT-6 Luna) as a real-time robotics policy in MuJoCo from camera views

TL;DR

Dmytro Hrybov re-runs his earlier Jev-as-robotics-policy experiment on top of OpenAI’s newly released Decisions API (post-trained GPT-6 Luna), with the key change that the API accepts images, so this time the model gets actual MuJoCo camera views instead of simplified geometry + contacts as text. He factors each control step into two API calls — one for a high-level next-action decision, one for how to move the arm and gripper — and reports that one of the better configurations grasped the cube and carried it across a divider but could not place it correctly. Position-framing claim: pulling vision-conditioned typed decisions from a cloud frontier model is a far more practical interface than piping simulator state, even if raw robotics competence is weak.

Key claims

The Decisions API (post-trained GPT-6 Luna) can be driven as a real-time control policy for a tabletop manipulation task in MuJoCo, taking image observations and returning typed structured decisions each tick [Post].
Single-call-per-step configurations initially failed; splitting each update into two calls (decide next action, then decide arm + gripper motion) made the loop viable [Post].
The author’s best reported run grasped a cube and carried it across the divider but failed to place it at the correct spot — strong demo of grasp + transport, failure on precise placement [Post].
Dozens of configurations were ablated across views, prompts, output formats, and calls-per-step; the reported run uses one call per step despite earlier two-call fix [Post].
Image-input access is called out as the structural advantage over the author’s prior Jev experiment, which had to be fed simplified geometry and contacts as text because Jev is text-only [Post].
The author explicitly caveats that the model “probably” does not have strong robotics understanding; the useful property is interface practicality (camera views in, decisions out) rather than measured policy competence [Post].

Method

The experiment is a thin control-loop harness around the Decisions API: at each simulation tick, render MuJoCo camera views, send them with a prompt that defines the current question plus the admissible answer set, and execute the returned selection on a tabletop arm with gripper. Two concrete scaffolding tricks are called out:

Two-stage decision factoring: a single-call “what should I do and how” prompt struggled, so the author split control into a sequential pair — one call picks the next sub-action from a typed menu, a follow-up call picks concrete arm/gripper motion given that sub-action.
View / prompt / format sweep: the author reports testing “dozens of setups” varying camera placement, prompt wording, response schema, and the one-call-vs-two-calls factorization; the showcased run turned out to be a one-call configuration — the two-call fix is useful but not strictly necessary for the best run.

No training, no fine-tuning, no VLA; the model is called via the public Decisions API and the harness is pure prompt + schema engineering. The pre-image counterpart is the author’s prior Jev experiment where the same control loop had to encode geometry and contacts as text because Jev’s API is text-only — the structural change here is that the Luna-powered Decisions API accepts images as context.

Tweet is the artifact; no code, no video of the failure mode, no latency / cost / token numbers are posted.

Results

Qualitative only:

Best reported configuration: cube is grasped and carried across the divider; final placement at the target spot fails [Post].
No success rates, number of trials, latency per tick, API cost per rollout, or comparison to a trained VLA baseline on the same setup are given.
The author expects a future Luna 6.1+ distilled from Sol 6.1 to perform better on this harness [Post].

Why it’s interesting

Three threads cross here. First, this is a robotics data point for System One / Decision Models — the Decisions API (OpenAI Decisions API — real-time typed classification/routing decisions from GPT-6 Luna) was announced three weeks earlier as a routing / classification product, and the launch post explicitly flagged “selecting an agent’s next action” among its uses. This tweet is the first filed instance of a user actually running it as that agent-next-action loop on a physical-ish control problem, and it exercises exactly the Luna property Ishaan called out as the Jev-class model’s missing axis in that filing: multimodal input lets you feed camera views instead of simulator state. The author previously tried the same experiment with Introducing System One Models & Jev Jev and had to piping-encode geometry + contacts as text — the image-input delta is the whole point of this re-run.

Second, it adds a cell to the LLM-as-controller row on VLA Models’ recipe-lever board. Existing filings on that row (Astra via motion-planning tool harness in Wenli Xiao — GPT-6 Astra does physical in-context learning from a single human video via a motion-planning tool harness, Astra+Fable on YAM arms via IK in Jay Chooi (Robocurve) — Astra vs Fable 5.1 on real-robot manipulation via IK / Comparing Fable 5.1 to Fable 5 on robotic manipulation, Claude-as-supervisor over MolmoAct in How Claude Performs on Robotics Tasks) all use general-purpose frontier LLMs or VLAs as the controller. This is the first filed instance using the output-space-specialist pole — a typed-decision model rather than a free-form LLM — as the controller. Where general LLMs can emit full programs, this harness is forced into schema-constrained decoding each tick, which is why the two-stage factoring was needed and why the grasp + transport succeeded but placement failed (the typed-answer schema likely under-represents the continuous precision axis a VLA’s flow-matching head carries).

Third, the failure mode matches the Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level sensorimotor counter-position’s specific prediction for LLM / decision-model controllers run through a thin motion interface: grasping and bulk transport work, final-placement precision does not. Same shape as the Fable 5 → Fable 5.1 on YAM-arm bowl-stacking failure in Comparing Fable 5.1 to Fable 5 on robotic manipulation, and the clapboard failure in Gemini Robotics-ER 2 vs Claude Opus 5 clapboard eval — ER2 threw robot arms off table, hallucinated success (Robocurve / Jay Chooi).