optimize_anything Goes omni — Composing Optimizers into Meta-Optimizer Pipelines

ListenNo Single Optimizer Wins: Composing LLM Optimizers with omni10:30

Show notes

This week we dig into `optimize_anything` going *omni* — a meta-optimizer that composes three different LLM-based text-artifact optimizers (GEPA, AutoResearch, and Meta-Harness) under one shared budget. The headline: on the Frontier-CS coding benchmark no single optimizer dominates, but the composed `omni` pipeline beats every standalone engine at a matched twenty-dollar-per-problem budget. We also talk about Terrarium, the head-to-head benchmark harness released alongside it, and what "switch scaffolds under plateau" might mean for image and video model work. Sources - optimize_anything Goes omni: Composing Optimizers into Meta-Optimizer Pipelines

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

The GEPA team makes optimize_anything engine-pluggable — a single engine= argument now dispatches the same (candidate, evaluator, objective) task to GEPA’s reflective proposer, Karpathy-style AutoResearch (long-horizon coding agent), or Meta-Harness (framework-owned loop with agent mutations) — and adds pipeline primitives (optimize_sequential, optimize_parallel, optimize_best_of, optimize_vote, optimize_adaptive_sequential) that compose engines. Using a new evaluation harness Terrarium with matched task / model / budget, they show on Frontier-CS that no standalone optimizer dominates (each wins ~1/3 of problems and the winner isn’t predictable from the problem) but every optimizer plateaus early, and swapping in a different optimizer past the plateau usually breaks through. The example meta-optimizer omni — run all three engines in parallel with 1/N budget each, keep the best candidate, then hand it to a fresh optimizer — beats every standalone optimizer at matched $20 budget on Frontier-CS (best variant 63.2 vs standalone best 55.4).

Key claims

Three LLM-driven text-optimization systems fit the same (candidate, score, loop) contract but differ in who proposes the next candidate and who owns the loop: GEPA (single reflective LLM call, Pareto-frontier parent selection), AutoResearch (autonomous coding agent owns the whole loop, Karpathy-inspired), Meta-Harness (framework owns the loop, agent mutates one candidate at a time) [§Different Optimizers].
Under matched Claude Sonnet 4.6 (medium thinking) / matched budget / fixed Frontier-CS eval server, all three optimizers far outscore zero-shot single-LLM-call (7.72 avg) at 43.8–55.4 — some optimizer is essential [§Why one optimizer is not enough].
No single optimizer dominates on Frontier-CS: AutoResearch has the best average (55.4) but per-problem each of the three wins on roughly a third of problems, and the authors report they couldn’t find a way to predict the winner from the problem itself [§Why one optimizer is not enough].
All three optimizers plateau: most gains happen early, then progress flattens. Handing the stalled candidate to a different optimizer usually breaks the plateau; which optimizer helps varies (e.g., on P0 AutoResearch breaks through while Meta-Harness stays put) [§Optimizers plateau, but a fresh optimizer may break through].
omni is a two-phase meta-optimizer: (Phase 1) run all engines in parallel at 1/N of budget each, keep the highest-scoring candidate; (Phase 2) hand that candidate to a fresh optimizer with the remaining budget. Encoded in ~10 lines using optimize_best_of then optimize_anything [§omni: composing optimizers into a meta-optimizer].
On Frontier-CS at matched $20 budget, every omni variant beats every standalone optimizer: GEPA 43.8 → omni-GEPA 61.8 (+18.0, +41%), AutoResearch 55.4 → omni-AutoResearch 63.2 (+7.8, +14%), Meta-Harness 50.9 → omni-Meta-Harness 59.3 (+8.4, +16%). The weakest omni variant still outscores the best single-engine standalone [Results table].
The new API exposes composition primitives beyond omni — optimize_sequential, optimize_parallel, optimize_best_of, optimize_vote, optimize_adaptive_sequential (the last watches for plateaus and automatically switches engines when progress slows) — with Terrarium as the shared task/eval-server/budget substrate for head-to-head comparison [§Getting started].

Method

optimize_anything was already a task-shaped API (candidate string + evaluator returning (score, info)), where info is used as Actionable Side Information for the proposer. The new release factors the proposer out into an engine interface: any class with a run(task, server) -> Result method that respects max_token_cost and stop_at_score can be registered via register_engine. Three engines ship — GEPA, AutoResearch, Meta-Harness — plus a best_of_n reference engine (~230 LOC). Composition helpers wrap engines: optimize_best_of runs several concurrently under a shared budget and keeps the highest-scoring result; optimize_sequential chains engines monotonically (each stage’s best seeds the next, so a regressing stage can’t poison later ones); optimize_adaptive_sequential schedules engine switches when a plateau is detected. Evaluation is done with Terrarium, which pins task, eval server, model, and dollar budget across engines to make head-to-head comparisons fair. The reported experiments use Claude Sonnet 4.6 (medium thinking) throughout and Frontier-CS (open-ended competitive-programming problems with hidden-test judges) as the task suite.

Results

Standalone Frontier-CS averages (matched budget, same model, same eval server): GEPA 43.8, Meta-Harness 50.9, AutoResearch 55.4, zero-shot single-call 7.72.
omni variants at matched $20 total budget: omni-GEPA 61.8, omni-Meta-Harness 59.3, omni-AutoResearch 63.2. Every omni variant > every standalone. Largest absolute gain: GEPA (+18.0, +41%). Best absolute score: omni-AutoResearch at 63.2, +7.8 over the strongest standalone.
Per-problem winner distribution: each of the three optimizers is best on approximately one third of Frontier-CS problems; the authors report no predictive signal from the problem to the winning optimizer.
Plateau-break diagnostic: after each standalone optimizer stalls, seeding a different optimizer from the stuck candidate usually keeps making progress on the same budget. Reported qualitatively per-problem (e.g., P0: AutoResearch continues, Meta-Harness does not).

Why it’s interesting

Two things line up cleanly with existing wiki concepts. First, this is a concrete composition of Weco’s AIDE² (recursive self-improvement of the inner agent, AIDE²: First Evidence of Recursive Self-Improvement) and Karpathy’s autoresearch loop (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster) — but the object being optimized is any text artifact, not a training script. The rejected-proposals story from AIDE² (“island GAs, MCTS, UCB-V, tournament LLM-judge, majority-vote were all -Δ or within noise”) and Karpathy’s emergent two-tier H100/H200 strategy both argued that simple parallel exploration + continuation from the best is what wins; omni’s Phase-1-parallel + Phase-2-fresh-optimizer is that same recipe made explicit as an API primitive. Second, this quantifies what GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning (GEPA) glossed over: reflective-mutation is one of several proposer families, and by itself is the weakest on Frontier-CS (43.8, below both agent-owned-loop baselines); the win comes from composition, and the largest absolute gain is precisely to GEPA (+18.0 when seeded from another engine’s plateau). The result is a small dose of humility for the “one optimizer to rule them all” framing common in prompt-optimization papers, and a concrete recipe (adaptive engine-switching at plateau) for the “which optimizer should I use” question that the concept page on AI-for-AI Research flagged as open.