Learning to Replicate Expert Judgment in Financial Tasks

ListenBeating Frontier Models at Their Day Job10:53

Show notes

Thinking Machines Lab and Bridgewater's AIA Labs fine-tuned an open-weight Qwen3-235B on six financial document-triage tasks, beating the best frontier baseline — 84.7% vs 78.2% accuracy, 29.8% fewer mistakes — at 13.8× lower inference cost. We dig into the load-bearing pieces: an expert-labeling pipeline that routes only model-label disagreements to humans, plus a three-part RL recipe of interleaved batching, CISPO with asymmetric clipping, and on-policy distillation with a promoted teacher. The obvious read-across for image and video model teams: taste-driven judgment tasks may be a fine-tuning problem, not a prompting one. **Sources** - Learning to Replicate Expert Judgment in Financial Tasks — Sarah Su, Kevin Zhu, Emily Xiao, Rohan Alur, Daniel Kang

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Thinking Machines Lab + Bridgewater AIA Labs use Tinker to fine-tune Qwen3-235B on six financial information-triage tasks (filtering, classifying, and segmenting investment-relevant documents) where frontier models (Gemini, Claude, GPT-5.2/5.4 variants) plateau at ~78% accuracy even with heavy prompt engineering. Their trained model reaches 84.7% average accuracy — 29.8% fewer mistakes than the best frontier baseline — at 13.8× lower inference cost. The recipe: seed labels from non-expert vendors, use the trained model itself to route disagreements to expert investors for cleaning, then multi-task RL with three specific ingredients (interleaved per-task batching, CISPO with asymmetric clipping, and on-policy distillation where the teacher is promoted from the current student only when validation accuracy sets a new high).

Key claims

Frontier models (Gemini/Claude/GPT variants) average ~50% accuracy on the six information-filtering tasks with a naive multi-task prompt, and rise to the mid-70s with expert-written prompts and task reframing (e.g. adding a “relevant but uninteresting” third label for macroeconomic-relevance classification) — but none of them cross the 80% threshold Bridgewater’s investors require to trust the system [§Frontier model performance].
Per-dollar frontier progress on this task is flat over model generations: GPT-5.4 costs 43% more than GPT-5.2 but is only marginally more accurate on this benchmark [§Frontier model performance].
Label quality is the load-bearing input: models trained on the initial non-expert-vendor labels performed poorly, and reasoning-trace inspection revealed many of the labels themselves were wrong [§Training dataset construction].
A self-consistency routing cleaning procedure works: train a model on the noisy vendor labels, then flag examples where the trained model’s answer disagrees with the label and send only those to expert investors for re-labeling — cheaper than expert-labeling from scratch, and the model’s own gradient becomes the disagreement filter [§Training dataset construction].
Standard GRPO with importance-sampling loss on Qwen3-235B jumps average accuracy from 44.8% (base) to 73.5%, but stalls below the 80% target [§Training recipe, base-vs-GRPO table].
Interleaved per-task batching (one batch per task in round-robin order) beats both sequential per-task training and fully-mixed batches — the ablation shows +12.1% average accuracy over fully-mixed [§1. Interleaved batching, ablation table].
CISPO with asymmetric clipping replaces the importance-sampling loss and adds +10.1% average accuracy over the IS baseline; the paper cites Cui et al. arXiv:2510.13786 for the CISPO+asymmetric-clip recipe [§2. CISPO loss with asymmetric clipping, ablation table].
On-policy distillation with checkpoint-promoted teachers adds a further +3.1% over a frozen base-model teacher: every 20 steps, promote the current student to teacher — but only if validation accuracy set a new high (never distill toward a weaker model) [§3. On-policy distillation with strong teachers, ablation table].
Final trained model: 84.66% average accuracy / 92.99% Pos-F1 on the six tasks vs. 78.2% for the best frontier baseline — 29.8% fewer mistakes — at 13.8× lower per-task inference cost [§Results, final table].
The three-component recipe is not decomposable: ablations show that removing any single ingredient (batching / loss / OPD) drops accuracy 3–12 points, and the OPD-with-frozen-teacher variant is 3.1 points worse than the promoted-teacher version [§Results, “Training Method Ablations” table].
Position statement: high-quality proprietary datasets labeled by domain experts, used to fine-tune open-weight models, produce custom models that exceed frontier performance on organizational tasks — a “differentiated intelligence” thesis where task-specific fine-tunes rather than the frontier lead per-domain [§Conclusion].

Method

Base model is Qwen3-235B, chosen because its fine-tuning behavior is widely characterized in the literature. Training happens on Tinker (Thinking Machines’ fine-tuning platform).

The training loop is multi-task RL over six financial information-triage tasks. Three modifications to a vanilla GRPO baseline compound to the final recipe:

Interleaved batching: rather than train each task sequentially or fully mix task examples within a batch, run one batch per task in strict round-robin order. Beats the mixed-batch variant by +12.1 points.
CISPO with asymmetric clipping: replace GRPO’s importance-sampling loss with the CISPO variant + asymmetric clipping (Cui et al. arXiv:2510.13786). Beats vanilla IS by +10.1 points.
On-policy distillation with promoted-teacher schedule: the student is regularized against a teacher via reverse KL added to the RL advantage; every 20 steps, if validation accuracy has set a new high, promote the current student checkpoint to be the teacher. The teacher is thus a moving target that is always at least as good as any prior version. This adds +3.1 points over a frozen base-model teacher.

Data cleaning is a separate stage that runs before RL: start with a dataset from non-expert vendors, train a model on it, then evaluate the trained model on the training set. Any example where the model’s prediction differs from the vendor label is sent to an expert investor for re-labeling. The intuition: if the model can’t match the label after being trained on it, the label is likely wrong or the example is genuinely difficult — either way, expert review is the correct next step. The final eval is on a held-out test set that never sees this loop.

Results

Average accuracy across six tasks: base Qwen3-235B 44.8% → GRPO 73.5% → final recipe 84.7% [§Results, tables].
Best frontier baseline (GPT/Claude/Gemini variants with expert-written prompts and task reframing): 78.2% [§Results, headline].
Ablations (each row zeroes out one ingredient of the final recipe): Interleaved Batching 72.2% (−12.5), CISPO+Asymmetric Clips 74.6% (−10.1), OPD 72.4% (−12.3), OPD with best-val-accuracy teacher (i.e. promoted teacher held frozen instead of moving) 81.6% (−3.1) [§Results, “Training Method Ablations” table].
Inference cost: 13.8× lower per task vs. the best frontier baseline [§Results, headline].
No numerical comparison to a base-model-teacher OPD baseline is reported for the multi-task setting beyond the +3.1% delta for promoted vs. frozen teacher.

Why it’s interesting

Concrete post-training case study that inverts the “just prompt a frontier model” default: a 235B open-weight fine-tune beats GPT-5.2/5.4 / Claude / Gemini variants on a real production task at ~14× lower cost. Reads as a direct application of the recipes filed on Reasoning RL — GRPO as the RL primitive, CISPO+asymmetric-clip as the loss variant, and on-policy distillation as a dense-reward sibling — to a domain the wiki has not yet covered (financial document triage). The promoted-teacher OPD schedule is a notable variant of the recipe in On-Policy Distillation: that write-up used either a fixed larger teacher (Qwen3-32B) or a pre-mid-training checkpoint of the same model as a frozen reference; this work makes the teacher a monotonically-improving moving target tied to validation accuracy, which is a natural extension the OPD blog did not test.

The label-cleaning trick (“route disagreements between the trained model and the vendor labels to experts”) is an operational cousin of the diagnostic framing in Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — both papers use “the model can’t match its own training example” as a signal that something is wrong, but in that paper it’s a signal to inspect the training recipe whereas here it’s a signal to inspect the labels. For Luma’s internal fine-tuning efforts, the recipe transfers cleanly: the interleaved-batching, CISPO+asymmetric-clip, and promoted-teacher OPD ingredients are all open primitives and the “expert-labels + cheap-model-driven routing” data pipeline is the generic lever for domains where taste-driven judgment is the bottleneck.