Reef — Continual Learning Infra for Self-Improving Agents

TL;DR

Reef is an open-source (Apache-2.0) inference-and-training backend that turns live agent traffic into a continuous learning loop. It exposes OpenAI-/Anthropic-compatible chat completion endpoints so any agent can point at Reef instead of a provider; every interaction is stored as a versioned “record” that a pluggable learning recipe can consume to update either model weights (via an in-loop training backend adapted from THUDM/slime) or the agent harness (rules, skills, prompts, orchestration — via cordis). Updates are evaluated, versioned as Git-LFS-tracked artifacts, and hot-swapped into the serving runtime without restart. Positioned by its authors as “the trivial baseline before we know how to do continual learning properly” — online RL wrapped in production-grade infra rather than a new algorithm.

Key claims

Continual self-improvement needs infra where inference is not the end of the pipeline — the serving system must own experience collection, training, evaluation, and rollout end-to-end [Ao Qu thread §2, §3].
Reef’s serving layer stores every inference as a structured experience record keyed by an x-reef-agent-record-id receipt that later /reef/report calls attach feedback to (numeric score or textual/structured feedback) [README “Send an inference request and report feedback”].
The whole agent evolves, not just weights: recipes can update either model artifacts (LoRA adapters or full checkpoints, hot-synced via NCCL) or harness trees (rules, skills, config, prompts, extensions) [Ao Qu thread §3, README “Using Reef”].
Harness evolution is delivered as an installable update — users curl … | bash a Reef harness the same way they install codex/opencode, and interactive sessions offer “Update with …” / “Skip” prompts when a new evolved version is published [README “Install Reef harness that grows with you”].
Recipes are the extension point that unifies online RL, test-time training, skill evolution, and harness evolution under one abstraction — they differ along three axes: learning signal, experience acquisition (proactive vs reactive), and evolving target (model / harness / both) [Ao Qu thread §4].
Shipped recipes cover four workload patterns: SAO (test-verified rollouts → weights, arXiv:2607.07508), OpenClaw-RL (agent traffic with next-state signals, no explicit reports → weights), TTT-Discover / TTTD (repeated scored attempts at one problem → weights), SkillClaw (agent feedback → skill-pool harness edits, no GPU) [README “Cookbook recipes” table].
Each scenario has an append-only release chain backed by Git LFS with compare-and-swap on the release head, so a stale training publisher cannot overwrite a newer release [Ao Qu thread §3 “Own the updates”].

Method

Reef’s architecture is a four-step loop — Serve → Observe → Grow → Commit — implemented across reef/service (agent requests), reef/runtime (inference + artifact updates), reef/train/processors (feedback matching), reef/recipe + reef/train (update jobs), reef/train/evaluation (candidate gating), and reef/artifact + reef/surface (version history and delivery). Inference is native and provider-compatible: applications post to /v1/chat/completions or /v1/messages with a standard body plus an x-reef-scenario header on the first call; Reef returns the response plus a receipt header. A separate /reef/report endpoint attaches feedback (score, text, or structured object) referencing one or more receipts.

Training runs asynchronously against the record stream. Weight-updating recipes wrap slime for distributed training and produce checkpoint or LoRA candidates; harness-updating recipes propose edits to a Cordis-managed harness tree. Candidates flow through recipe-defined evaluation before promotion. Approved artifacts are published to the scenario’s Git-LFS-backed release chain, and the runtime hot-updates the serving engine (NCCL weight sync) with no restart. Harness deliveries are wrapped as installer scripts served from /reef/harness/install?adapter=…. Reef ships with slime, SGLang (inference), and cordis (harness evolution) as the underlying components.

Results

The launch materials are engineering-focused rather than benchmarked — no headline evals against a fixed suite. The two illustrated demos are qualitative: (a) OpenClaw-RL running inside Reef, where the served model gradually learns user preferences across sequential rounds without interrupting the session, and (b) TTT-Discover iteratively improving a “Packing 32” combinatorial-optimization solution over many attempts, retaining higher-scoring candidates. Package ships on PyPI as reef-infra under Apache-2.0; requires Python 3.10+ and Git LFS.

Why it’s interesting

Reef makes an explicit bet Luma should note: for practical continual improvement of deployed agents, the bottleneck is infrastructure that owns the whole loop (serving + record store + training + versioning + rollout), not another RL algorithm. That framing sharpens the argument in AIDE²: First Evidence of Recursive Self-Improvement (first RSI evidence) by generalizing RSI-the-goal to continual-self-improvement-the-baseline, and it directly competes with the “training framework wraps an inference engine” pattern of slime and veRL by inverting it — an inference engine that owns a training backend. The harness-as-installable-artifact idea is a novel deployment surface: existing agent-modification loops like AREX: Towards a Recursively Self-Improving Agent for Deep Research (AREX) and Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence (Zetta, closed-loop harness evolution over a frozen VLA) both evolve harnesses offline; Reef pushes evolved harness versions to end users the same way coding CLIs push updates. The bundled TTT-Discover recipe also gives a production-shaped datapoint for the Long context as weights cluster — test-time training as an ordinary recipe alongside RL rather than a bespoke architecture.