Skip to content
Bud's Wiki

VLM-as-Evaluator

TL;DR

Section titled “TL;DR”

Using a vision-language model itself as the evaluator for visual content — aesthetic scoring, reward signals for RLHF/GRPO, or verification of generated artifacts — rather than relying on fixed scalar predictors (CLIP, PickScore, HPSv3) or hand-designed Bradley-Terry preference models. Three filed instances split on what the VLM produces: a calibrated scalar tier (ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization — ERNIE-Image-Aes, 8B InternVL-3-based, fixed rubric trained on Swiss-tournament labels); a structured per-prompt assessment trace with self-generated criteria, read out as a reward (Unified Personalized Reward Model for Vision Generation — UnifiedReward-Flex, Qwen3-VL/Qwen35-based, plugged into GRPO post-training of FLUX.1-dev and FLUX.2-Klein-9B); and a discrete pick over candidate generations (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models — MLLM verifier picking among DiffThinker reconstructions on Jigsaw-level-4). All three argue that the next generation of evaluators must be VLMs, not bespoke scorers, but disagree on whether the rubric should be static, dynamically generated, or implicit in a multi-candidate ranking.

Key claims

Section titled “Key claims”

  • Fixed scalar scorers and Bradley-Terry preference models follow a “one-size-fits-all” paradigm that assumes a monolithic preference distribution and is insensitive to content-specific visual cues (Unified Personalized Reward Model for Vision Generation §1). The empirical failure mode is documented quantitatively by ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization: LAION-Aesthetic overscores AI-generated and anime content, ArtiMuse overscores black-and-white photography and casual snapshots, UniPercept strongly prefers monochrome and casual snapshots — a feedback-loop hazard when these scorers are used to curate diffusion training sets or as RLHF rewards.

    • Conflict: A Very Big Video Reasoning Suite argues the opposite direction at benchmark time — for deterministic visual-reasoning tasks where ground truth is procedurally available, rule-based task-specific scorers (not VLM judges) should be the default. VBVR validates its rule-based scores against a large-scale human pairwise-preference study (strong positive correlation in win ratios), explicitly listing reproducibility, determinism, granular verifiability, and freedom from LLM hallucination as advantages over LMM-as-judge (A Very Big Video Reasoning Suite §4.1, §4.2, Fig. 4). The disagreement is bounded by task type: ERNIE-Image-Aes’s domain (aesthetics) does not have a deterministic ground truth, so rule-based scoring is not available; VBVR’s domain (deterministic visual reasoning) does. Both can be right within their own settings.

  • The fix splits cleanly along one axis: whether the evaluation rubric is static (improved by better label collection) or dynamic (regenerated per-prompt). ERNIE-Image-Aes takes the static path — a pairwise Swiss-system tournament with tier labels 1-10, annotators drawn from professional fine-arts programs, category-balanced training, and roughly 1.6× SRCC over the strongest open baseline on the deliberately bias-exposed ERIA-1K benchmark (ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization Annotation Protocol, Results). UnifiedReward-Flex takes the dynamic path — for each (prompt, content) pair the VLM interprets semantic intent, grounds on visual evidence, then dynamically instantiates fine-grained criteria under both predefined and self-generated high-level dimensions before emitting a reward (Unified Personalized Reward Model for Vision Generation §1, Abstract). IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation sits between the two — the rubric is partially prompt-derived (atomic yes/no questions decomposed from input constraints) and partially expert-seeded (three fixed seed dimensions: Data Completeness, Ordering, Encoding) (IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation §4.1).

  • VLM-as-evaluator integrates into the RL post-training stack via GRPO: UnifiedReward-Flex is trained SFT-then-DPO and dropped into GRPO as the reward signal for image and video generators, with public downstream LoRAs on FLUX.1-dev and FLUX.2-Klein-Base-9B trained against the UniGenBench prompt set (Unified Personalized Reward Model for Vision Generation Abstract, project page). This is in the same architectural slot as HunyuanImage 3.0’s MixGRPO multi-reward stack (HunyuanImage 3.0 Technical Report §4.2), but argues that one context-adaptive VLM judge can subsume the bag of fixed reward heads.

  • The pixel-space-generator-proposes / token-space-VLM-verifies pattern is emerging as a default for vision-centric reasoning: DiffThinker × MLLM-verifier strictly beats either alone on Jigsaw-level-4 — DiffThinker generates multiple candidate reconstructions, the MLLM picks one against the original constraints (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2, Fig. 6). UnifiedReward-Flex’s interpret-ground-criteria pipeline is the GRPO-time analogue of the same pattern: the generator proposes images, the VLM verifier emits a scalar reward used to update the generator.

  • VLM-as-evaluator outputs span four shapes: scalar tier (ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization, one of 10 tiers from a pairwise tournament label set), scalar reward read out at the end of a structured CoT (Unified Personalized Reward Model for Vision Generation, with the trace itself carrying the per-prompt rubric), discrete pick over a candidate set (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2, Fig. 6), and per-sample question-answering at benchmark time (RISE-Video: Can Video Generators Decode Implicit World Rules? Scoreboard, IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation §4.1) — an MLLM ingests extracted frames or a single image plus a set of per-sample yes/no questions (human-written for RISE-Video, prompt-decomposed-plus-expert-seeded for IGenBench) and emits both a continuous Q-ACC-style score and a strict per-sample I-ACC / pass-fail. Only the second shape composes cleanly with GRPO; the first composes with DPO-style preference filtering and best-of-N reranking; the third is a verifier on top of multi-candidate decoding; the fourth is a benchmark-time signal that need not be differentiable.

    • Conflict: A Very Big Video Reasoning Suite argues none of the four shapes is appropriate for deterministic visual-reasoning tasks — for those, per-task hand-written rule-based scorers (spatial accuracy, trajectory correctness, temporal consistency, logical validity, weighted per task) should replace VLM-as-judge at benchmark time. Direct contrast with RISE-Video: Can Video Generators Decode Implicit World Rules?, which uses the LMM-judge shape on essentially the same model set in the same week.

  • VLM-as-evaluator can also score intermediate generated artifacts in a multi-step inference pipeline, not only final outputs: UniG2U-Bench introduces Reasoning-Alignment (RA, do the intermediate images align with the reasoning plan) and Answer-Alignment (AL, do they support the final answer) as two new VLM-judged metrics, used to attribute Generate-then-Answer failures to artifact quality rather than the protocol (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §1, §3.3). This is the first filed instance of using a VLM judge to score intermediate visuals in a chain of reasoning.

  • Judge-backbone choice matters and is empirically unstable across MLLMs: IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation §5.4 evaluates 12 candidate judges (9 open-source + 3 closed) by Pearson correlation with human ratings on infographic reliability, and finds Gemini-2.5-Pro is the only model exceeding the 0.8 strong-correlation threshold (r = 0.90), with GLM-4.5v at 0.75 and GPT-5-mini at 0.70 — most open-source MLLMs fall below 0.5. This is the first filed paper here to systematically pin down which judges are usable for image-side verification.

  • VLM-as-evaluator extends to video generators, but with looser absolute reliability: RISE-Video uses an OpenAI-API-compatible LMM as the scorer for Reasoning Alignment / Temporal Consistency / Physical Rationality / Visual Quality across 11 TI2V models, and the discriminative power is clearly real (Acc range 1.9% → 22.5%, Reasoning-Alignment range 30.7 → 76.6) but the judge’s own correlation with human raters is not measured in the released artifact (RISE-Video: Can Video Generators Decode Implicit World Rules? Scoreboard, README Evaluation). VBVR closes the human-alignment gap for the rule-based alternative — VBVR-Bench’s automatic scorers correlate strongly positively with human pairwise win ratios across the same 8+ video generators (A Very Big Video Reasoning Suite §4.2, Fig. 4) — but only for tasks where deterministic ground truth exists.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-19] Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation: SpectraReward introduces a seventh output shape on this page’s enumeration: prompt-recovery log-likelihood — teacher-force the MLLM to reproduce the original prompt conditioned on the generated image and read out the average token-level log-likelihood as the reward. One forward pass, no rubric, no preference data, no reward-model fine-tuning; dense per-token signal composes cleanly with GRPO and sidesteps the pointwise-CoT reward-variance-collapse failure mode Stable-Layers documented. Self-SpectraReward specializes this for UMMs — policy’s understanding branch scores its own generation branch. Sweep over two diffusion models, three RL algorithms, nine MLLM backbones (4B–235B, four families), five OOD T2I benchmarks. Headline negative-scaling claim: larger reward MLLMs are not always better, and Self-SpectraReward can match or surpass much larger external reward models, reframing the bottleneck as reward-policy alignment rather than reward-model size.
  • [2026-07-17] DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation: DenseReward is the robot-manipulation, dense-reward instance of the VLM-as-evaluator pattern — a VLM-based reward model trained on simulator-synthesized failure trajectories (collisions, missed grasps, drops, recoveries) that outputs per-frame scalar rewards conditioned on image + language, sitting between the productized success-detector shape (Instance Labs — Verifying Robot Learning Episode Success) and the training-free continuous logit-expectation verifier of LLM-as-a-Verifier: A General-Purpose Verification Framework; reports beating general-purpose VLMs and prior robotic reward models on dense-reward prediction in sim and real, and provides usable reward signal for downstream MPC and RL.
  • [2026-07-13] Instance Labs — Verifying Robot Learning Episode Success: Instance Labs is the first filed commercial productization of VLM-as-evaluator for robot learning — pitched as a per-episode success-verdict-plus-evidence service that ingests (task, camera footage) and outputs binary verdicts across arbitrary robots and camera angles. No calibration numbers, no benchmark, no disclosed backbone at filing time, so the human-alignment question this page tracks stays open for the productized-verifier shape.
  • [2026-07-09] RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies: RoboDojo publishes a “systematic analysis of current policy performance” over its 30-policy sweep alongside the leaderboard, positioned as the analog of RoboArena’s §3.3 VLM+LLM policy-report pipeline but grounded in structured sim + real success metrics rather than free-form evaluator feedback — a control point for how much of the analysis-of-policy-sweeps shape survives when the underlying data is scalar-metric-only.
  • [2026-07-08] LLM-as-a-Verifier: A General-Purpose Verification Framework: LLM-as-a-Verifier is a training-free general-purpose verification framework that replaces discrete LM-judge score prompting with a continuous score computed as the expectation over the scoring-token logit distribution — a sixth output shape on this page’s enumeration (scalar tier / CoT-then-scalar / discrete pick / benchmark-time QA / retrospective synthesis / continuous logit-expectation). Directly addresses the pointwise-CoT reward-variance-collapse failure mode flagged by Stable-Layers by avoiding ties in the readout. Three verification-scaling axes are shown to compose: score granularity, repeated evaluation, and criteria decomposition. Reports SOTA on Terminal-Bench V2 (86.5), SWE-Bench Verified (78.2), RoboRewardBench (87.4), MedAgentBench (73.3) without verifier training, and the same continuous scores serve as dense reward improving SAC and GRPO sample efficiency.
  • [2026-07-05] Implications of Large-Scale Test-Time Compute (Noam Brown): Noam Brown’s framing sharpens the evaluator side of the ledger: if performance is a curve over inference budget rather than a scalar, benchmark orgs (and any VLM-judge / LLM-judge harness they build) must either track compute consumed during evaluation or impose explicit token / cost / time budgets. Otherwise same-model, different-scaffold comparisons are incoherent. Directly relevant to how this wiki treats LMM-as-judge scoring at fixed vs unfixed sampling budgets.
  • [2026-07-04] RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation: RoboWorld’s VLM judge uses a 0–5 task-progress rubric scored from fixed external views while isolating world-model artifacts to the wrist view, avoiding binary success and fairly crediting policies under imperfect neural-simulator rollouts.
  • [2026-06-30] Segmenting Robot Video into Actionable Subtasks (WGO-Bench): Macrodata’s WGO-Bench uses Gemini-3.5-Flash as a Scale-style LLM-as-judge for subtask labels — strict JSON rubric on action / object / source-destination correctness; same pattern as EditReward and EdiVal-Agent but applied to robot/egocentric video subtask names rather than image-edit quality.
  • [2026-06-29] SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation: SimFoundry is the non-VLM counterpart to the VLM-as-judge / Veo-as-evaluator pattern: physics simulator rollouts of cousin scenes predict real policy success at r = 0.911, MMRV = 0.018, reportedly beating PolaRiS. A useful calibration point for how much of policy-evaluation accuracy is recoverable without a generative or judge VLM in the loop.
  • [2026-06-29] ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation: ABACUS uses the same model’s understanding branch as the reward model for its generation branch via cycle-consistent GRPO — pushing the VLM-as-judge pattern to a self-rewarding single-model loop for count-faithful image generation, sidestepping external annotators.
  • [2026-06-29] Qwen-Image-2.0-RL Technical Report: Qwen-Image-2.0-RL builds two task-specific composite reward models by fine-tuning VLMs in a pointwise CoT-then-scalar paradigm — same shape as UnifiedReward-Flex — covering alignment/aesthetics/portrait fidelity for T2I and instruction-following/face-ID preservation for editing. The intra-group reward-range filtering stabilizer is a direct response to the Stable-Layers reward-variance-collapse failure mode for pointwise VLM scoring under GRPO.
  • [2026-06-24] Krea 2 Technical Report: Krea 2 uses VLMs as evaluators inside the data pipeline rather than at training-loss time. Pattern A: large VLM crafts a task-specific system prompt for a filtering job, produces a pseudo-labeled set, then a small (<1B) DINOv3 / SigLIP-2 classifier is trained to run the filter at scale. Pattern B: SigLIP-2 sparse-autoencoder features are each annotated by a VLM from top-k activating samples, producing an unsupervised tag system used to filter visual artifacts without training an explicit per-artifact classifier. Pattern C: VLM-named hierarchical k-means clusters with human review for cluster-level drops at midtraining.
  • [2026-06-20] Spotlight: Identifying and Localizing Video Generation Errors Using VLMs: Spotlight measures the missing calibration number for VLM-as-judge in the T2V error-localization shape: zero-shot Gemini 2.5 Pro reaches S+P 0.250 / 12.6% match coverage, zero-shot Qwen3-VL-{8B,30B-A3B}-Instruct ~0.14, vs human 0.508 / 26.8%; a multi-agent inference-time strategy (one query per error category) doubles Qwen3-8B to match Gemini, evidence that decomposing the rubric per error type is the cheapest way to lift open VLM judges on localized video error tasks.
  • [2026-06-11] Do generative video models understand physical principles? (Physics-IQ): Physics-IQ uses Gemini-1.5-Pro in a 2AFC realism-discrimination protocol (real-vs-generated pair, MLLM picks the fake) as a visual realism metric rather than a correctness metric — chance accuracy means the model fooled the judge; Sora achieves the most-realistic 55.6% (near chance) while ranking worst on the protocol’s physics motion-mask metrics.
  • [2026-06-08] RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies: RoboArena’s §3.3 qualitative-analysis module adds a fifth shape to the VLM-as-evaluator enumeration: a GPT-4.5 + o3 pipeline that synthesizes a per-policy report (language-following ability, multi-step-task robustness, qualitative failure modes) from a database of evaluation rollouts, with every claim in the report grounded by citations to specific rollout videos as evidence. Distinct from the scalar-tier / CoT-reward / discrete-pick / per-sample-QA shapes already on this page: here the LLM is doing retrospective synthesis over a corpus of evaluation data, not scoring a single artifact — which is the natural shape when the underlying evaluation signal (human preference + free-form feedback + progress score) is already collected and the question is “what does it mean.”
  • [2026-06-06] AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects: AssemblyBench uses GPT-4.1 as a structured data generator rather than a scorer — two-stage prompting (consistent part naming across all parts → per-step instruction text with target-part highlighting and color-coded reference diagram) for VLM-driven instruction-manual synthesis from CAD assemblies.
  • [2026-06-05] EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks: EgoTL critiques VLM auto-labeling as the noisy supervision substrate (incorrect action labels, CoT, spatial annotations) compounding into long-horizon spatial-grounding errors, and offers human spoken CoT with word-level timestamps as the corrective.
  • [2026-06-03] VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation: VideoPhy-2 trains VideoPhy-2-Autoeval, a 7B multi-task evaluator distilled from ~50K human SA/PC/rule annotations that beats Gemini-2.0-Flash-Exp by +80.8%/+236.4% Pearson on SA/PC for unseen prompts and reaches 78.7% rule-classification accuracy vs Gemini’s 59.2% — strong evidence that small distilled-from-humans evaluators outperform frozen frontier VLM judges on physical commonsense.
  • [2026-06-03] Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X): RBench (DAGroup PKU) — automatic robot-oriented video-generation benchmark reporting Spearman 0.96 with human evaluators across 25 models, materially above PhyGenEval’s ~0.65 on a comparable physics-realism task. Sub-metrics (structural consistency, physical plausibility, action completeness) are reported per (task × embodiment) cell, providing the strongest filed automatic-vs-human alignment for video-generation evaluation.
  • [2026-06-02] Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning: Stable-Layers — concrete fix for the pointwise-VLM-score-collapse failure mode flagged across this page: pointwise CoT scoring on five edit-centric criteria gave near-zero within-group variance under GRPO, so the authors add a grid-based pass where the VLM re-scores all candidates side-by-side. The pairwise-grid head sits on top of the pointwise head and is what makes Flow-GRPO on Qwen-Image-Layered learn at all. Direct evidence that for GRPO post-training the loss shape question (pointwise vs pairwise vs CoT-then-scalar) is bottlenecked by reward-variance survival.
  • [2026-06-01] PAI-Bench: A Comprehensive Benchmark For Physical AI: PAI-Bench reports Pearson r ≈ 0.97 between its Qwen3-VL-235B-judged Domain Score (over 5,636 human-curated QA pairs anchored to the Cosmos-Reason1 ontology) and human ELO on pairwise physics-plausibility A/B — the strongest published human-correlation result for the benchmark-time MLLM-as-judge shape on video, complementing IGenBench’s 0.90 on infographics and partially answering RISE-Video’s open calibration question. Domain Score and Quality Score together separate Veo3 (Domain 86.8) from open-source leaders (Wan2.2-I2V-A14B 87.1, Cosmos-Predict2.5-2B 84.9), with all models trailing the source-video physics ceiling of 89.8.
  • [2026-06-01] Sophon — AI Research Catalog of Evals, Tools, and Labs: External catalog (sophon.at) indexing leaderboards, eval-lifting tools, and per-benchmark saturation — surfaces 8 frontier benchmarks at >93% top score, with MBPP already at 100%, useful as a dashboard for picking non-saturated evaluations.
  • [2026-05-31] ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMs: ArtifactLens — low-supervision endpoint of the VLM-as-judge axis: SOTA on five human-artifact benchmarks using a frozen pretrained VLM + ICL with only hundreds of labeled examples per category, vs the tens-of-thousands fine-tuning recipe used by ERNIE-Image-Aes and UnifiedReward-Flex. Scaffolding is in-context learning + text-instruction optimization; generalizes to object morphology, animal anatomy, entity interactions, and AIGC detection. First filed datapoint that pretrained VLMs already encode artifact-detection capability and only need prompt-side unlocking.
  • [2026-05-28] Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation: Qwen-Image-Bench (Q-Judger) — first open-weight, Apache-2.0 benchmark-time T2I judge: a fine-tuned Qwen3.6-27B that scores generations along a 5-L1 / 23-L2 / 56-L3 hierarchy with per-prompt expert-seeded facet metadata and deterministic CoT inference. Sits between the closed Gemini-2.5-Pro IGenBench protocol (the only judge clearing r=0.8 against humans on infographics) and the GRPO-time UnifiedReward-Flex — same benchmark-time MLLM-as-judge shape as IGenBench but with the judge itself shipped open.
  • [2026-05-27] Planning at Inference: MCTS Test-Time Scaling for Long Video Generation: Search-based test-time scaling for long video is verifier-bottlenecked: the authors flag explicitly that “results are limited by the quality of current generators and verifiers” — a direct datapoint that MCTS-over-continuations only gains as much as the reward/verifier model can score.
  • [2026-05-23] A Very Big Video Reasoning Suite: First filed paper to argue against VLM-as-judge as the default for video-reasoning evaluation. VBVR-Bench uses task-specific rule-based scorers (e.g. Key Door Matching scored on target ID 30% / path validity 30% / path efficiency 20% / animation quality 20%) over 100 deterministic visual-reasoning tasks, defended on reproducibility, determinism, granular verifiability, and freedom from LMM hallucination. Crucially, the rule-based scores are validated against human pairwise preferences (strong positive win-ratio correlation), answering the “is rule-based scoring human-aligned?” objection. Methodologically opposed to its sibling RISE-Video: Can Video Generators Decode Implicit World Rules?, which uses LMM-as-judge on essentially the same model set the same week.
  • [2026-05-23] UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?: Adds a new use case for VLM-as-judge — scoring intermediate generated images in a Generate-then-Answer pipeline. RA (Reasoning-Alignment) and AL (Answer-Alignment) metrics let the benchmark attribute GtA failure modes to inaccurate / implausible / misaligned intermediate artifacts rather than to the GtA protocol itself. First filed instance of the VLM-judges-the-chain-of-reasoning shape.
  • [2026-05-23] IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation: IGenBench — first benchmark for text-to-infographic reliability, with the same benchmark-time MLLM-as-judge shape as RISE-Video but on still images. Atomic yes/no questions per (prompt, image), half decomposed from the prompt and half instantiated from three expert-seeded dimensions (Data Completeness, Ordering, Encoding); Gemini-2.5-Pro produces both Q-ACC (per-question) and I-ACC (per-image all-or-nothing). Includes an explicit judge-selection study across 12 MLLMs: Gemini-2.5-Pro is the only judge to clear r = 0.8 against humans, answering one of the open questions on this page about judge-backbone sensitivity for the benchmark-time shape.
  • [2026-05-23] RISE-Video: Can Video Generators Decode Implicit World Rules?: RISE-Video — first filed video-scale instance of the VLM-as-evaluator pattern. An LMM (OpenAI-API-compatible) ingests human-written per-sample reasoning questions plus a manually-designed frame-sampling schedule, and emits two aggregates: a relaxed weighted score (W.Score) and a strict accuracy. The relative ranking discriminates 11 TI2V generators sharply (Acc 1.9–22.5%), but the LMM-judge’s absolute correlation with human raters on video reasoning is unvalidated. Partially answers the open question on the previous concept-page version about whether VLM-as-evaluator survives at video scale: yes for relative ranking, unknown for absolute calibration.
  • [2026-05-23] Unified Personalized Reward Model for Vision Generation: UnifiedReward-Flex — per-prompt context-adaptive VLM judge (Qwen3-VL / Qwen35), trained SFT-then-DPO on 90K distilled reasoning traces, dropped into GRPO post-training of FLUX.1-dev and FLUX.2-Klein-9B (LoRAs released). Positions itself as a single context-adaptive reward that replaces the bag of fixed reward models used in HunyuanImage 3.0’s MixGRPO stack. Adoption already broad — Meta, NVIDIA, Tencent Hunyuan, ByteDance, Apple papers cite UnifiedReward.
  • [2026-05-22] DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models: Establishes the discrete-pick endpoint — MLLM-as-verifier picks among DiffThinker’s pixel-space reasoning candidates on Jigsaw-level-4, beating either model alone. Not a reward model per se, but the same VLM-as-judge primitive in a multi-candidate setting.
  • [2026-05-22] ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization: ERNIE-Image-Aes establishes the static-rubric endpoint — 8B VLM (InternVL-3 → ArtiMuse → ERNIE-Image-Aes) trained on Swiss-tournament labels, category-balanced. Documents the bias modes of the prior generation of scorers (LAION-Aesthetic, ArtiMuse, UniPercept) and roughly doubles SRCC on the deliberately bias-exposed ERIA-1K benchmark.

Open questions

Section titled “Open questions”
  • When does rule-based scoring beat VLM-as-judge, and when is it the other way around? VBVR vs RISE-Video is the cleanest filed contrast — same model set (Sora 2, Veo 3.1, Kling, Wan, CogVideoX, Hunyuan, LTX, Runway Gen-4), same week, opposite scoring philosophy. VBVR shows rule-based is feasible whenever ground-truth videos are procedurally available; RISE-Video covers cases where they aren’t (commonsense/societal/experiential/temporal reasoning over real-world video). No filed paper directly A/Bs the two scoring approaches on the same tasks, which would isolate the cost of LMM-judge noise.
  • Does a context-adaptive single-VLM reward (UnifiedReward-Flex) actually beat a fixed multi-reward stack (HunyuanImage 3.0’s MixGRPO setup) on a controlled head-to-head? The Flex abstract claims yes; the paper body retrieved so far does not expose the comparison numbers, and no third party has run the A/B yet.
  • What’s the right loss shape for training the VLM judge — pairwise rank, pointwise score, or CoT-then-scalar? UnifiedReward-Flex uses CoT-then-scalar; UnifiedReward / UnifiedReward-Think variants support pair+point; ERNIE-Image-Aes is pointwise from Swiss-tournament labels. No paper isolates which loss best survives downstream GRPO use.
  • How robust is a context-adaptive judge to reward hacking by the generator? A static rubric is hackable in a predictable way; a per-prompt dynamic rubric may be more or less hackable, but no filed paper measures generator-judge co-adaptation under extended GRPO training. VBVR’s rule-based scorers are also gameable in principle (a model that overfits the exact path-validity check might still fail human inspection), but VBVR’s human-alignment study at least rules out the most obvious failure modes at fine-tuning scale.
  • Does the VLM-as-evaluator paradigm survive at video scale? Partially answered: RISE-Video shows the pattern discriminates TI2V models cleanly (Acc 1.9–22.5%) using a frozen LMM judge with per-sample human-written questions, but does not measure the judge’s calibration against human raters. A video-generator GRPO release with Flex (or a RISE-style benchmark-time judge) as reward would be the natural next data point.
  • Cost: a per-sample CoT-reward call from an 8-32B VLM is dramatically more expensive than a forward pass through CLIP or HPSv3. What’s the iso-quality cost crossover for GRPO post-training between a cheap fixed reward and an expensive context-adaptive one? VBVR’s rule-based scorers are essentially free per evaluation, so the cost question is also a usable axis along which to compare scoring philosophies.
  • ERNIE-Image-Aes and UnifiedReward-Flex disagree on the right training signal (curated tier labels vs distilled reasoning traces). A study mixing the two — Swiss-tournament-labeled pairs as the DPO half of a Flex-style two-stage pipeline — has not been filed.
  • For benchmark-time use (RISE-Video / IGenBench shape), the LMM-judge is consumed without retraining on the target domain. Partially answered for images by IGenBench’s 12-MLLM judge-selection study (Gemini-2.5-Pro is the only one above r = 0.8 against humans on infographics); RISE-Video does not vary the judge model in its public artifact, so the video side is still open.
  • Does the IGenBench-style “prompt-decomposed atomic yes/no” rubric transfer to settings where the prompt does not carry explicit constraints (open-ended T2I, aesthetic-only prompts)? Both RISE-Video and IGenBench rely on the prompt to seed the question set; the static-rubric path (ERNIE-Image-Aes) is the only filed approach that works without prompt-side structure.
  • For the new UniG2U “VLM scores intermediate visuals in a chain of reasoning” shape: how reliable are RA / AL when the intermediate images come from very different generator families (diffusion vs AR vs MAR)? UniG2U does not yet ablate the judge.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2025-03

Section titled “2025-03”

2024-12

Section titled “2024-12”

2024-11

Section titled “2024-11”

2024-10

Section titled “2024-10”

Unknown date

Section titled “Unknown date”