Zoom AI sets new state-of-the-art benchmark on Humanity's Last Exam

TL;DR

Zoom’s CTO Xuedong Huang announces that a “federated AI” system from Zoom scored 48.1% on the Humanity’s Last Exam (HLE) full-set benchmark, 2.3 points above the previous SOTA of 45.8% held by Gemini 3 Pro with tool integration. The system is not a trained frontier model; it is an “explore-verify-federate” orchestration scaffold that dispatches queries across multiple proprietary and open-source models (GPT, Claude, Gemini, plus Zoom’s own small LMs) and aggregates outputs via a proprietary “Z-scorer.” The result drew a substantial backlash for taking SOTA credit on a benchmark via API stitching, with critics (Max Rumpf, Peter Wildeford, Clement Delangue) arguing this is engineering of a scaffold, not a frontier-model contribution. Useful as a datapoint for how far multi-model orchestration alone can move a hard reasoning benchmark.

Key claims

Federated-AI system from Zoom reaches 48.1% on the HLE full-set, vs. 45.8% for Gemini 3 Pro with tool integration — a 2.3-point absolute jump [blog body].
The architecture is described as “explore–verify–federate”: multiple reasoning paths are explored, each verified against constraints, then federated into a final answer; concrete model identities (GPT, Claude, Gemini, plus Zoom small LMs) and a proprietary “Z-scorer” pick or refine outputs [blog body].
Zoom did not train a new frontier model; the system is built around third-party APIs plus task-specific fine-tuning of Zoom’s own small LMs [blog body].
The same federated stack is presented as the underlying mechanism for Zoom AI Companion 3.0 features — meeting summaries, action-item extraction, multi-step workflow automation [blog body].

Method

The blog itself gives only a high-level description of “federated AI” — multiple models are orchestrated under an “explore–verify–federate” agentic workflow, with a proprietary scoring/selection module (“Z-scorer”) combining outputs. No paper, code, or per-component HLE numbers are released; the leaderboard validation status of the run is unclear (the official HLE leaderboard maintained by the Center for AI Safety distinguishes validated runs from marketing claims, and at the time of the announcement the Zoom run was not on it). Sup AI separately claimed 52.15% via a similarly orchestrated stack within the same week, also not on the official leaderboard.

Results

HLE full-set: Zoom federated AI 48.1% vs. Gemini 3 Pro w/ tools 45.8% (the blog’s only headline number).
No reported breakdown by HLE subject domain, no ablation of which constituent model contributes what, and no compute or latency cost reported.
No comparison against the natural baseline of a parallel-sampled ensemble of the same constituent models (best-of-N over Gemini/GPT/Claude), so the marginal value of the “federate” step over majority-vote is unmeasured from public information.

Why it’s interesting

This is the cleanest public datapoint that pure inference-time model orchestration — without training a new frontier model — can move a hard reasoning benchmark by ~2 absolute points over the SOTA single-model result. It belongs in the inference-time-scaling taxonomy (Inference-Time Scaling) as a scaffold-side axis adjacent to Recursive Language Models and Recursive Language Models: the paradigm of 2026, but with a critical difference: Zoom’s orchestrator routes across heterogeneous models (different providers / sizes / open vs. closed), where RLM dispatches homogeneous sub-LM calls of the same base model. The benchmark-credit controversy also pairs naturally with Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity and Sup AI’s 52.15% claim as evidence that frontier-benchmark leaderboards are now contested at the orchestration layer rather than the model layer — relevant context for how Luma frames its own benchmark reporting.