Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image

TL;DR

Multimodal RewardBench 2 (MMRB2) is a benchmark for reward models that must handle interleaved image+text inputs and outputs — the regime omni models (GPT-4o-style image generators, unified multimodal LLMs) actually operate in, which prior RM benchmarks don’t cover. It spans four tasks (text-to-image, image editing, interleaved generation, “thinking-with-images” multimodal reasoning) with 1,000 expert-annotated preference pairs per task, sourced from 23 models/agents over 21 source tasks. Pairs are curated via ensemble filtering for strong human-expert consensus. The authors evaluate existing judges — multimodal-LLM-as-a-judge and models trained on human preferences — for each subtask.

Key claims

Reward models are essential for training LLMs but remain underexplored for omni models that handle interleaved image and text sequences [Abstract].
MMRB2 is the first comprehensive benchmark for reward models on multimodal understanding and (interleaved) generation [Abstract].
The benchmark covers four tasks — text-to-image, image editing, interleaved generation, and multimodal reasoning (“thinking-with-images”) [Abstract].
1,000 expert-annotated preference pairs per task, drawn from 23 models and agents across 21 source tasks [Abstract].
Pairs are designed for (1) practical but challenging prompts, (2) responses from SOTA models and agents, (3) strong human-expert consensus via ensemble filtering [Abstract].
Existing judges studied per subtask include multimodal LLM-as-a-judge and models trained directly with human preferences [Abstract].

Method

MMRB2 sources candidate responses from 23 models/agents across 21 source tasks spanning four subtask categories: text-to-image (T2I), image editing, interleaved text-and-image generation, and “thinking-with-images” multimodal reasoning. For each (prompt, response-A, response-B) tuple, an ensemble filtering strategy retains only pairs where multiple human experts agree on the preferred response, yielding 1,000 preference pairs per task. The benchmark is then used to evaluate two families of judges: zero-shot multimodal-LLM-as-a-judge (using frontier MLLMs to pick between A and B), and reward models specifically trained on human preferences. Per-subtask judge accuracy is the headline metric.

Results

Headline numbers are not available without the paper body (fetch timed out at filing). The framing in the abstract is comparative — MMRB2 is positioned as the missing benchmark slot, and the paper’s contribution is the dataset construction methodology (23-model coverage, ensemble-filtered expert consensus) plus the per-subtask judge study, not a single SOTA number.

Why it’s interesting

This is the benchmark-time complement to the GRPO-time reward-model work already on the wiki — UnifiedReward-Flex (Unified Personalized Reward Model for Vision Generation) needed exactly this kind of held-out evaluation surface for omni judges, and prior MM reward benchmarks (Multimodal RewardBench v1) covered only single-modality VQA-style preferences. The “thinking-with-images” subtask aligns it with the broader interleaved-reasoning thread — Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm, DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models — and the interleaved-generation subtask gives a concrete RM-evaluation target for unified models like LightBagel: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation and EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture. The expert-ensemble-filtering recipe is methodologically close to ERNIE-Image-Aes’s Swiss-tournament labels (ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization), but at the level of preference pairs rather than tier labels.