Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance

TL;DR

Soup Of Category Experts (SoCE) is a model-souping recipe that picks merge candidates and per-model weights using benchmark-category correlation structure rather than uniform averaging. The method identifies “expert” checkpoints for each weakly-correlated category cluster (e.g. multi-turn vs. live function calling on BFCL), then grid-searches non-uniform weights (0.1–0.9 in steps of 0.1) over the selected experts. On the Berkeley Function Calling Leaderboard, SoCE souping of three open-source LLaMA-3-derivative 70B tool-use checkpoints (xLAM-2-70b, CoALM-70B, watt-tool-70B) hits 80.68% — a new SOTA, +2.7% over the best individual model — and a souped 8B reaches 76.50%, +5.7% relative over xLAM-2-8b. Same recipe also lifts MGSM and ∞-Bench. The whole technique is post-hoc arithmetic on weights; no retraining.

Key claims

Benchmark categories exhibit highly heterogeneous cross-model correlation: BFCL multi-turn categories correlate 0.96–0.98 between models, but Multi-turn-base vs. Live-Accuracy correlates only 0.07 — i.e. tool-use skill decomposes into weakly-correlated competency clusters that no single model dominates [§3, Fig. 1].
The SoCE procedure has four steps: (1) compute Pearson-correlation heatmap over category × model performance, (2) pick the best-ranked “expert” model for each weakly-correlated category, (3) grid-search non-uniform weights ∈ {0.1, …, 0.9} (step 0.1) plus a uniform-weight comparison case, (4) average weights with the chosen scalars [§3, Algorithm 1].
On BFCL 70B, SoCE achieves 80.68% (xLAM-2-70b 0.5 + CoALM-70B 0.2 + watt-tool-70B 0.3), beating xLAM-2-70b (78.56%), CoALM-70B (54.49%), watt-tool-70B (73.57%), uniform souping of all 4 candidates (68.33%), and uniform souping of SoCE-selected candidates only (78.40%) [Table 1, 70B].
On BFCL 8B, SoCE achieves 76.50% (xLAM-2-8b 0.7 + ToolACE-2-8B 0.2 + watt-tool-8B 0.1), beating xLAM-2-8b (72.37%) and uniform-souping baselines (69.80% / 74.01%) [Table 1, 8B].
On MGSM (multilingual grade-school math, 4 LLaMA-7B-derivative MathOctopus/MAPO checkpoints), SoCE reaches 51.7% vs 50.9% best candidate vs 47.0% uniform-souping of all candidates — uniform souping over all candidates actually regresses below the best individual model [Table 2(a)].
On a 5-checkpoint LLaMA-3-70B ∞-Bench ablation where all candidates were trained on variants of one data mix, SoCE still lifts performance 27.44 → 28.0 (vs 27.85 for uniform-with-SoCE-selection) — both weight tuning and candidate selection contribute even when expert specialization is weak [Table 2(b)].
Ablation isolating the two SoCE components: candidate selection alone (uniform-souping → uniform-with-SoCE-selection) is the larger lever on BFCL 70B (68.33 → 78.40) and 8B (69.80 → 74.01); weight optimization on top contributes a further 2.28% (70B) / 3.44% (8B) relative [§4.2.1].
Souping increases cross-category Pearson correlation: across 35 of 37 LLaMA-70B souping experiments on BFCL/FLORES-36/∞-Bench, souped models exhibit a more linearly-correlated, more consistent performance profile across categories than their unsouped parents [§4.2.2, Fig. 2, Fig. 5].
Shapley-value analysis over all 2^4 candidate subsets on MGSM shows model contributions are not uniform — SoCE-selected subsets have significantly higher Shapley values than arbitrary subsets, and disproportionately weighting a weak parent (e.g. 0.9 on M4) collapses the average score to 37.0 [§4.2.3, Fig. 6(b)].
Gains shrink when expert specialization shrinks: on FLORES-36 (translation across 18 language pairs, all candidates equally good per category) and on ∞-Bench (same data-mix candidates), SoCE only marginally beats baselines — anti-correlation in candidate performance is the necessary precondition for the recipe to work [Table 2(b), Table 5, §4.2.3].

Method

The four candidate 70B BFCL models are domain-specialized post-trains of LLaMA-3 70B (xLAM, CoALM, watt-tool, functionary). The 8B set is similar. All experiments require same-architecture, same-pretrain checkpoints. The category-correlation heatmap is computed on the public BFCL leaderboard scores directly — i.e. the “development set” for picking the expert per category is the leaderboard itself, which the paper flags as a limitation (an oracle held-out dev set would be cleaner in production). The weight search is a discrete grid over the simplex with step 0.1 and per-model weights ∈ [0.1, 0.9]; the special uniform case is added for comparison. Souping is straight per-parameter weighted averaging with no fine-tuning step afterward. The Shapley analysis on MGSM uses 4 candidates so all 2^4 coalitions are enumerable and souping-as-coalition-aggregator is well-defined.

Results

Headline: new SOTA on BFCL: 80.68% at 70B and 76.50% at 8B, both via 3-model souped LLaMA-3 derivatives. The 8B SOTA at 76.50% is competitive with the previous 70B SOTA (78.56% xLAM-2-70b), suggesting category-aware souping at the 8B tier roughly closes a model-scale gap of ~10×. On MGSM, SoCE 51.7% > best individual 50.9% > uniform-souping 47.0%. On ∞-Bench, SoCE 28.0% > best individual 27.44% > uniform-souping 27.44%. Across 37 LLaMA-3-70B souping experiments on BFCL, FLORES-36, and ∞-Bench, souped candidates scored higher on more than 20 of 36 categories in 35 of the 37 runs [§4.2.2, Fig. 5]. The Shapley analysis confirms SoCE-selected coalitions are systematically higher-value than arbitrary subsets [§4.2.3]. Crucially, SoCE-derived checkpoints did not regress on unrelated benchmarks (HellaSwag, IFEval, BIG-bench Hard) — i.e. souping on a target benchmark’s category structure does not appear to overfit to that benchmark [Appendix A].

Why it’s interesting

SoCE is the first wiki paper to make a serious quantitative case for post-hoc weight arithmetic as a SOTA-producing recipe rather than a footnote — and it does so on BFCL, the same benchmark the Tool-Use Agents cluster’s parameter-space camp (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing, MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling) burns 10K+ GPU-hours on. The cost asymmetry is even sharper than Training-Free GRPO’s API-vs-GPU framing: SoCE needs only inference to evaluate candidates plus a small grid search of weighted averages, on already-released open-source checkpoints. The “increased cross-category correlation after souping” finding ([§4.2.2]) is also a non-trivial datapoint for Emergent Representation Geometry: weight-averaging linearly-mode-connected post-trains produces a more coherent skill profile across categories than any individual parent, which is at least suggestive that the post-training basin is shared and the specialization is local. The dependence on category anti-correlation as a necessary precondition (FLORES-36 and ∞-Bench show ceilings) maps cleanly onto the open question on the Tool-Use Agents page about whether self-generated trajectory data overfits its own synthesizer — SoCE is the dual recipe: when models don’t anti-correlate (everyone trained on the same mix), souping does nothing.