Towards RL for Superhuman Text: Unslopping AI (RL-XAR)

TL;DR

Meta FAIR proposes RL-XAR (Reinforcement Learning from eXpert-Aligned Rubrics), a training recipe that turns readily-available expert human writing into a learned reward for RL post-training on non-verifiable text generation tasks. The key move is to meta-optimize the rubric-generating prompt itself against a training set of expert human-vs-model pairs until the rubric-scored judge assigns the human higher scores than a strong model writer — closing what the authors show is a very large gap in standard LLM-as-judge protocols (pairwise judges prefer the model 63.5–84.6% of the time, and unoptimized rubric judges prefer the model 100% of the time on academic-paper section writing). Applied to three domains — academic paper sections, Pulitzer/Nobel-grade story continuation, and Wikipedia section writing — a Qwen3.5-27B trained under RL-XAR reaches worst-of-three-rubric human-normalized scores that beat every frontier model tested, with 89% (papers) and 95% (stories) blind expert-preference win rates over the baseline 27B checkpoint.

Key claims

Standard LLM-as-judge protocols systematically prefer AI writing over expert humans: a rubric-free pairwise judge prefers GPT-5.6/Opus-4.8 model outputs over held-out human sections 63.5–84.6% of the time across writer/judge combinations, and a standard GPT-5.6 rubric-based judge picks the model 100% of the time [§A direct pairwise judge prefers the model].
Meta-optimizing the rubric-generating meta-prompt against a small train set of expert human-vs-model pairs closes and reverses the gap: 7 iterations on 8 training / 5 validation papers move the validation human–model gap from −4.2 to +2.76 (crossing zero at iteration 4), simultaneously raising the human’s judged score (3.4 → 5.0) and lowering the model’s (7.6 → 2.7) [§Meta-rubric optimization figure].
The learned rubrics differ qualitatively from the initial ones: initial rubrics act as “mini-paper” coverage checklists that reward fluent restatement; iterated rubrics reward sectional ownership, disciplined selection, and economy of scope, and are length-bounded so the criterion set stays small and stable rather than accumulating requirements [§Learned rubrics].
Iterating training makes RL-XAR work on the worst rubric: three outer rounds of (meta-optimize rubric → RL against accumulated rubrics) prevent overfitting to one rubric set. The RL-XAR-trained Qwen3.5-27B reaches worst-of-three human-normalized score 9.60 (human = 10) on paper-section writing, ahead of every external frontier model in the leaderboard [§Rubric Evaluation figure — paper sections].
Blind expert human evaluation confirms the automatic metric: on the authors’ own papers, RL-XAR beats baseline Qwen3.5-27B 16-to-2 (89% win rate) on paper sections and 19-to-1 (95%) on story continuations; the recurring baseline failure modes are “mini-paper” over-scope on intros/abstracts and clichéd similes on stories [§Expert Human Evaluation].
The judge model matters as much as the rubric: at fixed learned rubrics, only Qwen3.5-27B (the weakest judge tested) reverses the gap and ranks model above human on Story and Wikipedia — every stronger judge preserves the human-favoring gap [§Comparison of Judges figure].
The meta-optimizer model matters: Kimi-K2.6 and Opus-5 find a positive human–model gap over 3 seeds, while Muse Spark 1.1 struggles; a weaker model would likely fail entirely [§Different Rubric Meta-Optimizers figure].
Expert humans matter for rubric optimization: training on low-quality human writers gives a much smaller human–model gap (+0.37 vs +1.39 for high-quality writers under GPT-5.6), and the learned rubric correctly ranks high > medium > low human writing — a necessary sanity check that the rubric measures genuine quality [§Expert humans matter figure].
Rubrics generalize across strong judges: rubrics meta-optimized with a GPT-5.6 judge still separate humans from Muse Spark 1.1 model outputs when scored by Muse Spark 1.1, and vice-versa [§Cross-judge robustness figure].
RL-XAR generalizes across three domains with the same base recipe (GRPO on Qwen3.5-27B, Qwen3.8-2.4T-A95B rubric judge, Kimi-K2.6 meta-optimizer): paper sections 9.60 vs human 10; story continuation Qwen3.5-27B 2.8 → 8.2 (next best Claude 5 Opus 6.8); Wikipedia 2.7 → 4.0 (still below frontier baselines) [§Test performance figures].
Wikipedia section writing is the hardest domain of the three, and the paper attributes the smaller gain to the Qwen3.8-2.4T-A95B judge being too weak a grader for that task — a self-consistent failure that supports the “judge matters” claim above [§Rubric Evaluation — Wikipedia].

Method

RL-XAR is an outer-loop meta-optimization of the rubric generator, nested around standard GRPO. A meta-prompt takes a (paper minus one section) context and asks an LLM to generate a scoring rubric for the missing section; the meta-prompt is scored by the average expert-human-minus-model rubric-score gap on a small training set of (paper, human-written section) pairs against generations from the current writer. An optimizer LLM (Kimi-K2.6 in the main experiments) is shown the current meta-prompt, its train-set gap, and specific failure examples, and asked to propose a revision that widens the gap “for genuine-quality reasons, not superficial tells”. The meta-prompt is length-bounded, forcing consolidation rather than accumulation of criteria. The best-of-N candidate over 7 iterations is retained.

Once a rubric-generating meta-prompt exists, standard GRPO trains the writer against per-example rubrics generated on the fly, judged by a large LLM (Qwen3.8-2.4T-A95B for training, GPT-5.6 for evaluation to avoid same-judge bias). Length is controlled by supplying a target word count from the held-out human original within ±15%. Three outer iterations of (rubric meta-optimization against the current model → GRPO against all accumulated rubrics) prevent single-rubric overfitting; the third rubric set is used only for evaluation, not training. The authors also sketch a fully online co-training variant where rubrics are updated by GRPO simultaneously with the writer, with early positive results on story continuation.

Results

Headline numbers — worst-of-three-rubric human-normalized score under a GPT-5.6 judge (human = 10):

Academic paper sections — Qwen3.5-27B baseline underperforms every frontier model; RL-XAR-trained Qwen3.5-27B reaches 9.60, ahead of every frontier model tested. Blind expert evaluation on the authors’ own papers: RL-XAR beats baseline 16-to-2 (89% preference).
Story continuation (Pulitzer/Nobel corpus) — Qwen3.5-27B 2.8 → RL-XAR 8.2, next best Claude 5 Opus 6.8. Blind human: RL-XAR beats baseline 19-to-1 (95%).
Wikipedia sections — Qwen3.5-27B 2.7 → RL-XAR 4.0, still below GPT-5.6 (7.9) and Claude 5 Opus / Muse (7.2). Attributed to a too-weak Qwen3.8-2.4T-A95B training judge on this task.

Diagnostic ablations: judge capability strictly monotone (only the weakest judge, Qwen3.5-27B, ranks model above human on stories and Wikipedia); meta-optimizer capability necessary (Muse Spark 1.1 fails, Kimi-K2.6 and Opus-5 succeed); training on low-quality human writers shrinks the learnable gap (+0.37 vs +1.39). Cross-judge robustness holds: rubrics optimized with GPT-5.6 still separate humans on Muse Spark 1.1 scoring.

Why it’s interesting

RL-XAR is the crispest filed instance on the wiki of the “shape the judge, not the reward” position: given the Reasoning RL concept page’s consolidation that GRPO-with-shaped-rewards is the default, the interesting design surface has moved to what the reward actually measures, and RL-XAR shows that a small amount of expert human data is enough to move a rubric-generating meta-prompt from anti-expert to pro-expert. The result also sharpens the VLM-as-Evaluator cluster’s caveats: standard LLM-as-judge and standard LLM-generated rubrics not only fail to distinguish expert human from model writing, they invert it — a large calibration failure that any RLHF/RLAIF pipeline optimizing against unaudited LLM judges is currently inheriting.

The recipe complements Enhancing Rubric-based RL via Self-Distillation (CriPO), which repairs rubric-based GRPO’s credit-assignment layer at fixed rubrics: CriPO takes the rubrics as given and fixes what happens after they score a rollout, while RL-XAR fixes the rubrics themselves before RL starts. They should stack. RL-XAR also contrasts with the SFT-for-RL Readiness cluster (Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning, Behavior Injection: Preparing Language Models for Reinforcement Learning, TailSFT: Filtered Fine-Tuning Improves Post-Training Performance): those redesign the SFT stage to prepare the policy for RL; RL-XAR redesigns the reward stage, using expert human writing as ground-truth targets not for imitation but for rubric calibration. The “expert humans matter” ablation is the paper’s key epistemological claim — the whole recipe is only as good as the human writing quality bin the meta-optimizer sees.