Ornith-1.5: From Self-Scaffolding to Self-Improvement

TL;DR

Ornith-1.5 is an open-weights (MIT-licensed) family of LLMs at three scales — 9B dense, 35B MoE (3B active), and 397B MoE — trained with an end-to-end self-improvement loop that jointly optimizes task generation, harness (scaffold) construction, and solution rollouts under GRPO. The 397B flagship matches Claude Opus 4.8 on Terminal-Bench 2.1 (86.1 vs 85.0) and DeepSWE (56.0 vs 59.0) while outperforming similar-scale open peers GLM-5.2 and DeepSeek-V4-Flash-0731. The load-bearing recipe change over Ornith-1.0: the model now proposes progressively harder tasks (targeting a 20% empirical success frontier), generates task-specific scaffolds, and produces rollouts — with reward propagated through all three stages so the system learns to generate better training tasks and harnesses, not just better solutions.

Key claims

Ornith-1.5-397B matches Claude Opus 4.8 on Terminal-Bench 2.1 (86.1 vs 85.0) and DeepSWE (56.0 vs 59.0), and outperforms comparable open MoEs GLM-5.2 (82.7 / 46.2) and DeepSeek-V4-Flash-0731 (82.7 / 54.4) on the same benchmarks [§Full Table]
Ornith-1.5-35B-A3B (only 3B active parameters per token) outperforms the dense Gemma-4-31B and Meta Muse-Glimmer-30B by wide margins on agentic coding: Terminal-Bench 2.1 (Claude Code) 68.5 vs 43.4 / 51.7 and SWE-Bench Verified 79.0 vs 52.0 / 76.0 [§Full Table]
Ornith-1.5-9B in its Ornith-1.5-9B-Mobile quantized form is deployable on iPhone/Android while scoring 47.0 Terminal-Bench 2.1 (Claude Code) and 70.6 SWE-Bench Verified — matching or exceeding much larger Gemma-4-31B and Qwen3.6-35B-A3B on multiple coding tracks [§Full Table]
Each training cycle runs three coupled stages: given an environment/codebase + task-type instructions + the model’s own task-solving history, the proposer generates tasks progressively harder than solved ones; the model then generates or refines a task-specific scaffold; the policy produces solution rollouts. Reward flows back to all three stages so proposer, harness, and policy improve jointly [§Self-Improvement through Self-Generated Tasks, Harnesses, and Solutions]
Task reward is a multiplicative composition of three signals — validity/verifiability (V), frontier difficulty (D), novelty (N) — so the proposer must satisfy all three: coherent + reliably-evaluable tasks, empirical success rate near a target frontier p* = 0.2, and low similarity to a buffer of prior generated/trained tasks [§Task Reward]
Frontier difficulty is measured directly from N sampled rollouts of the current model (empirical success rate against target p* = 0.2), so the curriculum auto-evolves with model capability — once a task is solved reliably, its reward drops and the proposer is pushed toward harder ones [§Frontier Difficulty]
Harness reward decomposes into consistency with the task specification (C), fidelity of harness reward to solution quality (F), and resistance to reward hacking / evaluator failures (H); rollouts are then scored directly by the generated harness with binary or richer signals per task [§Harness and Rollout Rewards]
All three stages (question generation, harness generation, solution rollouts) are optimized with GRPO under their respective rewards, giving a single closed self-improvement loop rather than a fixed human-curated task and harness distribution [§Harness and Rollout Rewards]
Ornith-1.5 significantly closes the gap to Kimi K3 (2.8T) at 397B: Kimi K3 leads on Terminal-Bench (Terminus-2) 88.3 and DeepSWE 67.5, but Ornith-1.5-397B leads or ties on SWE-bench Verified (86 vs 86.2), HLE without tools (44.6 vs 43.5), and ClawEval (81.4 vs Kimi K3 does not report) [§Full Table]
All model sizes ship in FP8, GGUF, MLX, and NVFP4 quantized variants under the MIT license with unrestricted commercial + research use [§ (footnote / release framing)]

Method

The training-time loop generalizes Ornith-1.0’s self-scaffolding framework (which optimized only harness + rollout at fixed task set) into a three-stage joint optimization: task proposal → task-specific scaffold generation → solution rollouts. Given the current model’s task-solving history and high-level task-type instructions, the proposer emits tasks that exceed the model’s current frontier; conditioned on task + scaffold, the policy generates rollouts. The task reward is R_task = V(q, s) · D(q, s, {τ_i}) · N(q, B) where V measures validity/verifiability (does the scaffold execute; do high-confidence solutions pass; do incorrect solutions fail), D targets a p* = 0.2 empirical success rate estimated over N rollouts, and N penalizes similarity to prior tasks in buffer B. Validity is also usable as a hard gate. Harness reward R_h = f(C, F, H) similarly rewards specification-alignment, quality-tracking fidelity, and hack-resistance. All three losses are optimized under GRPO using their respective rewards. Because difficulty is measured against the current policy’s own rollouts, the curriculum evolves automatically with capability.

Results

Headline agentic-coding numbers at 397B: Terminal-Bench 2.1 (Terminus-2) 86.1, SWE-bench Verified 86, SWE-bench Pro 65.1, SWE-bench Multilingual 79.6, DeepSWE 56, ClawEval 81.4, Tool Decathlon (Toolathlon-Verified) 71.2. Reasoning: HLE (no tools) 44.6, HLE (with tools) 56.1, GPQA Diamond 92.8. Agentic search: WideSearch 80.8, BrowseComp 86.6, MCP-Atlas 80.0. All numbers averaged over 5 independent runs [§Full Table, §Footnote]. Versus Ornith-1.0-397B, the largest jumps come on DeepSWE (8 → 56, +48 pts), Frontier-Bench v0.1 (2.7 → 13.5), SWE Atlas – QnA (41.2 → 55.6), and Toolathlon-Verified (43.2 → 71.2). At 35B-A3B the same-recipe gains are equally sharp against Ornith-1.0-35B: Terminal-Bench (Claude Code) 62.8 → 68.5, DeepSWE 0 → 22, NL2Repo 34.6 → 46.2, SWE Atlas – QnA 37.1 → 39.8 [§Ornith-1.5-35B table]. Numbers on Kimi K3 (2.8T) show Kimi still leading on peak scores but Ornith-1.5-397B closer on reasoning + agentic axes at a fraction of parameters.

Why it’s interesting

The most direct comparison is AIDE²: First Evidence of Recursive Self-Improvement (AIDE²): both close a self-improvement loop but in complementary places. AIDE² does recursive optimization of the coding-agent itself (outer agent rewrites inner agent’s Python source over 8 days / 100 steps) and declines Level 2 RSI because the ignition test shows no asymptotic gain — so AIDE² concludes “optimize the optimizer” hits diminishing returns. Ornith-1.5 instead keeps the model fixed as the artifact and closes the loop earlier: it self-generates the training data (tasks + harnesses) under GRPO, with rollouts feeding back into the next round of task proposals. It contrasts with AREX: Towards a Recursively Self-Improving Agent for Deep Research (AREX), which also frames RSI but at inference-time (outer self-improvement loop reflecting on rollouts) rather than at training time. Compared to Qwen3-Coder-Next Technical Report‘s ~800K human-mined PR + bug-injection recipe, Ornith-1.5 replaces the static synthesizer with a self-evolving one — closer in spirit to Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (InfTool: closed-loop tool-use synthesis) but at foundation-model scale across coding, reasoning, and agentic tasks simultaneously. Also notable for Luma: this is one of the first open-weights MoE releases that reports treating harness generation as a first-class trainable object (with C/F/H reward decomposition), a design decision that overlaps with AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design‘s meta-harness optimization thesis.