BaseReward: A Strong Baseline for Multimodal Reward Model

TL;DR

A “recipe paper” for building multimodal reward models (MRMs): exhaustively ablate every choice — paradigm (Naive-RM vs Critic-RM vs Generative-RM), reward-head architecture, training strategy, data curation across ten-plus preference datasets, backbone choice and scale, ensemble methods — then crystallize the findings into BaseReward, a Qwen2.5-VL-backed scalar reward model with a two-layer reward head trained on a curated multimodal+text preference mixture. Reports new SOTA on MM-RLHF-Reward Bench, VL-Reward Bench (~18% headroom), and Multimodal Reward Bench, and shows the reward signal transfers to actual MLLM RLHF/GRPO training with consistent downstream gains across MMBench, MME-RealWorld, MMStar, MathVista, V*, Llava-Wild, and WildVision. The contribution is the systematic ablation matrix, not architectural novelty.

Key claims

A systematic guide for constructing state-of-the-art MRMs is currently absent in both academia and industry; this paper aims to provide a clear “recipe” through exhaustive experimental analysis covering reward modeling paradigms, reward head architecture, training strategies, data curation, backbone and scale, and ensemble methods [§1, Abstract].
BaseReward adopts a simple Qwen2.5-VL backbone enhanced with an optimized two-layer reward head, trained on a carefully curated mixture of high-quality multimodal and text-only preference data [§5.1].
BaseReward establishes new SOTA on MM-RLHF-Reward Bench, VL-Reward Bench, and Multimodal Reward Bench, with roughly 18% improvement on VL-Reward Bench over prior open and proprietary baselines [§5.3, Table 9, Table 10].
Generalization to pure-text reward benchmarks (RMBench, Reward Bench) is evaluated explicitly to test whether a multimodal reward model degrades on text-only inputs [§ Text-Only reward benchmarks].
Plugging BaseReward into a real RLHF post-training loop yields consistent gains across MMBench v1.1, MME-RealWorld-Lite, MMStar, MathVista, V*, LlavaWild, and WildVision compared to alternative reward schemes [§5.4].

Method

The paper is structured as an ablation matrix rather than a single architectural proposal. Each axis of the MRM design space is varied independently: (i) the reward-modeling paradigm — Naive-RM (scalar head on top of the VLM), Critic-based RM (LLM-judge style natural-language critique then scalar), and Generative RM (model emits a textual preference, scored from logits); (ii) the reward head — depth, width, normalization; (iii) the training strategy — Bradley-Terry pairwise loss vs pointwise regression, with or without SFT warmup; (iv) data curation — ten-plus multimodal and text-only preference datasets, including MM-RLHF, with category and quality balancing; (v) backbone — Qwen2.5-VL at multiple scales, plus alternative VLMs; (vi) ensemble — output averaging, model souping. The settled BaseReward design (Qwen2.5-VL + two-layer reward head + curated mixed mm+text data + pairwise BT loss) is the empirical winner of this matrix, not a new architecture. Section 5.4 then exercises the reward by running RLHF on an MLLM, scoring candidate responses with BaseReward and updating the policy.

Results

On the multimodal reward benchmarks the paper reports new SOTA across MM-RLHF-Reward Bench, VL-Reward Bench (~18% over the strongest prior baseline), and Multimodal Reward Bench, with comparisons to both open-source and proprietary MRMs [Tables 9, 10]. On the text-only side (RMBench, Reward Bench) BaseReward is shown to retain text-domain competence after multimodal training — i.e. multimodal training does not catastrophically degrade text-only reward accuracy [§ Text-Only reward benchmarks]. The §5.4 RLHF transfer experiment is the practical-utility claim: fine-tuning an MLLM with BaseReward as the reward signal beats alternative reward schemes consistently across a diverse evaluation suite (MMBench v1.1, MME-RealWorld-Lite, MMStar, MathVista, V*, LlavaWild, WildVision), arguing that benchmark-time reward accuracy and downstream policy quality move together.

Why it’s interesting

This is the empirical complement to Unified Reward Model for Multimodal Understanding and Generation (UnifiedReward). Where UnifiedReward argued one VLM judge can subsume task-specific reward models by training jointly across image-und, image-gen, video-und, video-gen, BaseReward goes the other way — fix one task family (MLLM-response preference), exhaustively ablate every other choice, and ship the empirical optimum. The two together are the “principled” and “empirical” poles of the same lineage; both feed VLM-as-Evaluator.

For the team, this is the paper to consult before standing up a new MRM. It also calibrates the Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image (Multimodal RewardBench 2) leaderboard discussion — BaseReward is one of the benchmark-defining baselines that newer omni-modal reward models are expected to clear. The methodological contrast with Unified Personalized Reward Model for Vision Generation (UnifiedReward-Flex) is sharp: Flex moves the rubric inside the model via per-prompt CoT, while BaseReward keeps the rubric implicit in the curated preference data and bets on scaling the data+head+backbone choice well. Both are SFT-then-RL-friendly; neither has been A/B’d head-to-head as a GRPO reward, which is the natural follow-up.