Skip to content
Bud's Wiki

Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

TL;DR

Section titled “TL;DR”

SpectraReward is a training-free reward function that turns any pretrained MLLM into a zero-shot reward model for image-generation RL by measuring how well the original prompt can be recovered from the generated image via a single image-conditioned, teacher-forced forward pass — using the average image-conditioned prompt log-likelihood as the reward. The Self-SpectraReward variant specializes this for unified multimodal models: the policy’s own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework with no external reward models or preference labels. Across two diffusion policies, three RL algorithms, nine MLLM backbones (4B–235B, four families), and five OOD T2I benchmarks, both SpectraReward and Self-SpectraReward significantly outperform prior MLLM-derived reward training methods, and larger reward MLLMs are not always better — reward-policy alignment matters more than raw scale [Abstract].

Key claims

Section titled “Key claims”
  • SpectraReward reuses the MLLM’s pretrained image-text alignment without preference labels, reward-model fine-tuning, or a Bradley-Terry head — the reward signal is just the average teacher-forced prompt log-likelihood conditioned on the generated image [Abstract].
  • Self-SpectraReward is a special case for unified multimodal models: the policy’s understanding branch scores its own generation branch, forming a closed-loop self-improving framework with no external reward model or knowledge [Abstract].
  • Empirical study spans two diffusion models, three RL algorithms, nine reward MLLM backbones from four families (4B–235B parameters), and five out-of-distribution T2I benchmarks [Abstract].
  • Both SpectraReward and Self-SpectraReward significantly and consistently outperform prior MLLM-derived reward training methods across this sweep [Abstract].
  • Larger reward MLLMs are not always better — Self-SpectraReward can match or surpass much larger external reward models, suggesting reward-policy alignment is a key factor for effective image-generation RL [Abstract].

Method

Section titled “Method”

SpectraReward inverts the standard MLLM-as-judge pipeline. Instead of prompting the MLLM to judge a generated image (score it, answer decomposed verification questions, or CoT-then-scalar), it teacher-forces the MLLM to reproduce the original prompt conditioned on the generated image, and reads out the average token-level log-likelihood of the prompt as the reward. One forward pass, no sampling, no rubric, no preference data, no verifier fine-tuning. The signal is dense (per-token) and continuous, sidestepping the reward-variance-collapse failure that pointwise CoT scoring exhibits under GRPO (Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning). Self-SpectraReward specializes the pipeline for unified multimodal models — the same model is both policy (generation branch) and reward (understanding branch), yielding a closed-loop self-improving system without external annotators or reward-model training [Abstract].

Results

Section titled “Results”

Reported gains are consistent across two diffusion policies, three RL algorithms, nine MLLM reward backbones from four families spanning 4B to 235B parameters, and five OOD T2I benchmarks — both SpectraReward and Self-SpectraReward significantly outperform prior MLLM-derived reward training methods across the sweep. A key negative-scaling result is called out explicitly: larger reward MLLMs are not always better, and Self-SpectraReward (same-model reward) can match or exceed much larger external reward models, which the authors attribute to reward-policy alignment being more important than reward-model capacity [Abstract]. Per-benchmark numbers are not exposed in the arxiv abstract at filing time.

Why it’s interesting

Section titled “Why it’s interesting”

SpectraReward is the sixth+ output shape on the VLM-as-evaluator page’s enumeration — it’s neither a scalar tier (ERNIE-Image-Aes), nor a CoT-then-scalar (UnifiedReward-Flex, Qwen-Image-2.0-RL), nor a discrete pick (DiffThinker), nor benchmark-time QA (IGenBench, RISE-Video), nor retrospective synthesis (RoboArena §3.3), nor continuous logit-expectation over scoring tokens (LLM-as-a-Verifier: A General-Purpose Verification Framework). It’s the “invert the generation objective” shape: use the prompt-recovery log-likelihood as the reward, which composes cleanly with GRPO and requires zero reward-side training. The Self-SpectraReward variant is a direct generalization of ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation‘s cycle-consistent GRPO from the narrow count-faithfulness setting to general T2I RL, and pushes the same “one model is its own reward” thesis that Unified Reward Model for Multimodal Understanding and Generation and HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing have been building toward — but here the reward requires no fine-tuning of the understanding branch at all. If the negative-scaling claim holds up under scrutiny, it undercuts the recent trend of scaling reward MLLMs (BaseReward, RewardDance, UnifiedReward-Flex) and reframes the bottleneck as reward-policy alignment rather than reward-model size.

See also

Section titled “See also”