Skip to content
Bud's Wiki

VLM-as-Evaluator

TL;DR

Section titled “TL;DR”

Using a vision-language model itself as the evaluator for visual content — aesthetic scoring, reward signals for RLHF/GRPO, or verification of generated artifacts — rather than relying on fixed scalar predictors (CLIP, PickScore, HPSv3) or hand-designed Bradley-Terry preference models. Three filed instances split on what the VLM produces: a calibrated scalar tier (ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization — ERNIE-Image-Aes, 8B InternVL-3-based, fixed rubric trained on Swiss-tournament labels); a structured per-prompt assessment trace with self-generated criteria, read out as a reward (Unified Personalized Reward Model for Vision Generation — UnifiedReward-Flex, Qwen3-VL/Qwen35-based, plugged into GRPO post-training of FLUX.1-dev and FLUX.2-Klein-9B); and a discrete pick over candidate generations (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models — MLLM verifier picking among DiffThinker reconstructions on Jigsaw-level-4). All three argue that the next generation of evaluators must be VLMs, not bespoke scorers, but disagree on whether the rubric should be static, dynamically generated, or implicit in a multi-candidate ranking.

Key claims

Section titled “Key claims”

  • Fixed scalar scorers and Bradley-Terry preference models follow a “one-size-fits-all” paradigm that assumes a monolithic preference distribution and is insensitive to content-specific visual cues (Unified Personalized Reward Model for Vision Generation §1). The empirical failure mode is documented quantitatively by ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization: LAION-Aesthetic overscores AI-generated and anime content, ArtiMuse overscores black-and-white photography and casual snapshots, UniPercept strongly prefers monochrome and casual snapshots — a feedback-loop hazard when these scorers are used to curate diffusion training sets or as RLHF rewards.

    • Conflict: A Very Big Video Reasoning Suite argues the opposite direction at benchmark time — for deterministic visual-reasoning tasks where ground truth is procedurally available, rule-based task-specific scorers (not VLM judges) should be the default. VBVR validates its rule-based scores against a large-scale human pairwise-preference study (strong positive correlation in win ratios), explicitly listing reproducibility, determinism, granular verifiability, and freedom from LLM hallucination as advantages over LMM-as-judge (A Very Big Video Reasoning Suite §4.1, §4.2, Fig. 4). The disagreement is bounded by task type: ERNIE-Image-Aes’s domain (aesthetics) does not have a deterministic ground truth, so rule-based scoring is not available; VBVR’s domain (deterministic visual reasoning) does. Both can be right within their own settings.

  • The fix splits cleanly along one axis: whether the evaluation rubric is static (improved by better label collection) or dynamic (regenerated per-prompt). ERNIE-Image-Aes takes the static path — a pairwise Swiss-system tournament with tier labels 1-10, annotators drawn from professional fine-arts programs, category-balanced training, and roughly 1.6× SRCC over the strongest open baseline on the deliberately bias-exposed ERIA-1K benchmark (ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization Annotation Protocol, Results). UnifiedReward-Flex takes the dynamic path — for each (prompt, content) pair the VLM interprets semantic intent, grounds on visual evidence, then dynamically instantiates fine-grained criteria under both predefined and self-generated high-level dimensions before emitting a reward (Unified Personalized Reward Model for Vision Generation §1, Abstract). IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation sits between the two — the rubric is partially prompt-derived (atomic yes/no questions decomposed from input constraints) and partially expert-seeded (three fixed seed dimensions: Data Completeness, Ordering, Encoding) (IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation §4.1).

  • VLM-as-evaluator integrates into the RL post-training stack via GRPO: UnifiedReward-Flex is trained SFT-then-DPO and dropped into GRPO as the reward signal for image and video generators, with public downstream LoRAs on FLUX.1-dev and FLUX.2-Klein-Base-9B trained against the UniGenBench prompt set (Unified Personalized Reward Model for Vision Generation Abstract, project page). This is in the same architectural slot as HunyuanImage 3.0’s MixGRPO multi-reward stack (HunyuanImage 3.0 Technical Report §4.2), but argues that one context-adaptive VLM judge can subsume the bag of fixed reward heads.

  • The pixel-space-generator-proposes / token-space-VLM-verifies pattern is emerging as a default for vision-centric reasoning: DiffThinker × MLLM-verifier strictly beats either alone on Jigsaw-level-4 — DiffThinker generates multiple candidate reconstructions, the MLLM picks one against the original constraints (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2, Fig. 6). UnifiedReward-Flex’s interpret-ground-criteria pipeline is the GRPO-time analogue of the same pattern: the generator proposes images, the VLM verifier emits a scalar reward used to update the generator.

  • VLM-as-evaluator outputs span four shapes: scalar tier (ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization, one of 10 tiers from a pairwise tournament label set), scalar reward read out at the end of a structured CoT (Unified Personalized Reward Model for Vision Generation, with the trace itself carrying the per-prompt rubric), discrete pick over a candidate set (DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models §4.2, Fig. 6), and per-sample question-answering at benchmark time (RISE-Video: Can Video Generators Decode Implicit World Rules? Scoreboard, IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation §4.1) — an MLLM ingests extracted frames or a single image plus a set of per-sample yes/no questions (human-written for RISE-Video, prompt-decomposed-plus-expert-seeded for IGenBench) and emits both a continuous Q-ACC-style score and a strict per-sample I-ACC / pass-fail. Only the second shape composes cleanly with GRPO; the first composes with DPO-style preference filtering and best-of-N reranking; the third is a verifier on top of multi-candidate decoding; the fourth is a benchmark-time signal that need not be differentiable.

    • Conflict: A Very Big Video Reasoning Suite argues none of the four shapes is appropriate for deterministic visual-reasoning tasks — for those, per-task hand-written rule-based scorers (spatial accuracy, trajectory correctness, temporal consistency, logical validity, weighted per task) should replace VLM-as-judge at benchmark time. Direct contrast with RISE-Video: Can Video Generators Decode Implicit World Rules?, which uses the LMM-judge shape on essentially the same model set in the same week.

  • VLM-as-evaluator can also score intermediate generated artifacts in a multi-step inference pipeline, not only final outputs: UniG2U-Bench introduces Reasoning-Alignment (RA, do the intermediate images align with the reasoning plan) and Answer-Alignment (AL, do they support the final answer) as two new VLM-judged metrics, used to attribute Generate-then-Answer failures to artifact quality rather than the protocol (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? §1, §3.3). This is the first filed instance of using a VLM judge to score intermediate visuals in a chain of reasoning.

  • Judge-backbone choice matters and is empirically unstable across MLLMs: IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation §5.4 evaluates 12 candidate judges (9 open-source + 3 closed) by Pearson correlation with human ratings on infographic reliability, and finds Gemini-2.5-Pro is the only model exceeding the 0.8 strong-correlation threshold (r = 0.90), with GLM-4.5v at 0.75 and GPT-5-mini at 0.70 — most open-source MLLMs fall below 0.5. This is the first filed paper here to systematically pin down which judges are usable for image-side verification.

  • VLM-as-evaluator extends to video generators, but with looser absolute reliability: RISE-Video uses an OpenAI-API-compatible LMM as the scorer for Reasoning Alignment / Temporal Consistency / Physical Rationality / Visual Quality across 11 TI2V models, and the discriminative power is clearly real (Acc range 1.9% → 22.5%, Reasoning-Alignment range 30.7 → 76.6) but the judge’s own correlation with human raters is not measured in the released artifact (RISE-Video: Can Video Generators Decode Implicit World Rules? Scoreboard, README Evaluation). VBVR closes the human-alignment gap for the rule-based alternative — VBVR-Bench’s automatic scorers correlate strongly positively with human pairwise win ratios across the same 8+ video generators (A Very Big Video Reasoning Suite §4.2, Fig. 4) — but only for tasks where deterministic ground truth exists.

Recent contributions

Section titled “Recent contributions”
  • [2026-09-01] Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling: Lucida’s GizmoAct is a boundary case for VLM-as-evaluator: the VLM policy not only issues pose edits at each turn but decides itself when alignment is reached — the VLM is simultaneously the actor and the stopping-criterion judge on rendered multi-view observations. Distinct from filed cases where a VLM scores rollouts post-hoc (VLA success detection, dense-reward learning); here the same model closes the loop on its own actions in real time up to 12 refinement steps.
  • [2026-08-29] PAWBench: How Far Are We from Probabilistically Aligned World Modeling?: PAWEval extends the VLM-as-judge lineage from scalar-plausibility scoring to categorical outcome classification: each generated rollout is mapped to one of an enumerated set of physically valid outcomes (die face, pendulum direction, bounce angle bucket) so that repeated generations can be aggregated into an empirical histogram compared against a reference distribution. Distinct from the holistic-score judges (VideoPhy, PhyGenEval, RISE-Video LMM judge) and the evidence-tree agentic judges (HarnessEval-W) in that the evaluator’s output is a discrete label used for distribution matching rather than a quality score.
  • [2026-08-29] Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve): Inspect Robots’ scorer slot is designed to host arbitrary graders — scalar rubrics, unit-test-pass ratios, or VLM judges — with the choice of judge decoupled from the eval harness itself. All three prior Robocurve real-hardware evaluations on the wiki used human-rated 0-100 rubrics as the scorer rather than a VLM-as-judge, and the harness formalizes that choice as a plug-in point rather than an implementation detail — a concrete implementation of the “rubric substrate + swappable judge” pattern the concept page tracks.
  • [2026-08-26] On-Policy Self-Distillation in Diffusion Models: DiffusionOPSD evaluates across ten reward evaluators under reward-matched settings on two backbones (SD 3.5-M, Z-Image-Turbo), winning 19 of 20 — a broad-VLM-evaluator sweep that isolates the RL-objective contribution (self-distillation with bounded targets) from the reward-model contribution.
  • [2026-08-20] The Multimodal Intelligence of Muse Spark 1.2: WildArtifactBench (MSL, previewed) scores agents by pairwise win-rate / Elo against a baseline agent’s deliverable, judged by agentic or human judges — removing the need for ground-truth deliverables and objective rubrics so open-ended tasks with any deliverable format can be measured. Extends the VLM-as-judge substrate to agent-vs-agent pairwise scoring; 10-task preview subset released, full benchmark held internal.
  • [2026-08-18] HarnessEval-W: Agentifying the Evaluation of Visual Worlds: HarnessEval-W (MirroS + collaborators) extends the VLM-as-judge primitive into an agentic decomposition shape: a parent agent decomposes each evaluation case into measurable subproblems and spawns specialized sub-agents equipped with tailored context and diagnostic tools; the parent then validates gathered evidence and summarizes the final verdict, producing an inspectable evidence tree per case rather than a scalar. Applied to 18 world models over 330 cases with judgments reported as human-aligned. Same “decompose the rubric across sub-agents to lift open VLM-judge accuracy” strategy Spotlight already showed doubles Qwen3-VL-8B on video error localization, now scaled to full WM evaluation with tool use and released as a live open benchmark.
  • [2026-08-17] Training AI Scientists to Replicate Research: Per-task rubric-based LLM judge for paper-replication tasks, with a human validation study showing rubric decomposition achieves greater consistency and lower noise than a plain-LLM-judge baseline; the rubric judge is the training reward signal for Faraday’s long-horizon RL, and multi-sample aggregation + turn-level credit assignment are additional judge modifications needed for RL stability.
  • [2026-08-14] behavior-judge — compiling Agent Behavior specs into deterministic + semantic judges: behavior-judge (Khaitan) is the deterministic-core + semantic-residual endpoint of the LLM-as-judge design space, applied to text-agent trajectories rather than visual content: compiles a Braintrust Agent Behavior BEHAVIOR.md into a YAML judge with five typed predicates over trajectory events, invoking the LLM only for semantic triggers, one-question-per-clause semantic checks, and confirming deterministic-check failures. Reports byte-identical verdicts across 10 runs vs run-to-run inconsistencies for a one-call gpt-5-mini judge on the same inputs — a text-side echo of the A Very Big Video Reasoning Suite “rule-based scoring beats LMM-judge where ground truth is procedural” position, and the reliability-via-scope-reduction complement to LLM-as-a-Verifier: A General-Purpose Verification Framework‘s reliability-via-continuous-readout lever.
  • [2026-08-14] H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models: H2R-Bench uses source-grounded per-clip annotations (task goals, action events, functional contacts, object responses) as explicit evaluation targets across five dimensions rather than relying on holistic VLM judgment — a design choice that distinguishes it from PhyGenBench-style VLM-as-judge protocols while still targeting video-generation quality.
  • [2026-08-11] Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty: Proactive T2I agents (Hahn et al., Google DeepMind, Dec 2024) uses simulated-user self-play as its automatic evaluation protocol: a second LLM (Ag2-style) answers agent questions from a held-out ground-truth prompt + belief graph, and the agent’s final Imagen 3 image is scored via VQAScore against the ground-truth caption. Early T2I-side instance of the LLM-as-oracle-evaluator pattern (predates most filed VLM-judge work by ~a year), with the twist that the “judge” is not scoring an artifact directly but acting as a user whose latent intent is fixed by a held-out image. Also uses VQAScore itself — a benchmark-time VLM-judge shape on this page — as the terminal scalar. Reports ≥2× VQAScore over single-turn Imagen 3 within 5 simulated-user turns.
  • [2026-08-07] The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric: TPIPS (Wang, Nitzan, Hertzmann, Zhu, Shechtman, Efros, Zhang — Adobe/CMU/UC Berkeley) adds an eighth output shape to this page’s enumeration: text-conditioned pairwise perceptual similarity, where the user-supplied free-form aspect (color, pose, lighting, camera angle, cup handle position) selects the sense of similarity being measured. Trained on ~1M human triplet judgments over 25K FLUX-generated triplets × 260K triplet-aspect combos, plus a held-out 2AFC set from four real vision/graphics pipelines (image editing, composite, NVS, image-to-3D). Fine-tuning Qwen3-VL-Embedding closes the human-consensus gap from 22.7% (best baseline Gemini-3.1-Pro at ~53%) to 6.4% on odd-one-out and 12% on the OOD 2AFC set. Ablates three fusion architectures under the same backbone — late (embedding + cosine), mid (LPIPS-style per-layer activation distance with text-conditioned channel weights), early (joint-attention with register-token readout, symmetry + identity guaranteed by attention-mask design) — finding early ≥ mid ≈ late, with late fusion preferred in practice for cacheable retrieval. Contrasts sharply with LPIPS/DreamSim (near chance on aspect prompts, since aspect-agnostic) and with the dynamic-rubric UnifiedReward-Flex line (rubric is VLM-generated per prompt) by making the rubric a user input.
  • [2026-08-06] FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models: FailSafe-VLM extends the VLM-as-evaluator pattern past “detect and label” into “detect and act”: the fine-tuned LLaVA-OneVision-7B outputs failure type + an executable 7-DoF recovery delta rather than a scalar success flag (Instance Labs) or a dense per-frame reward (DenseReward). Beats frontier VLMs at failure reasoning by >4× on the specific failure-type accuracy metric — 0.8368 accuracy / 0.6522 cosine similarity to ground-truth recovery vs. GPT-4o 0.1960 / 0.0117 and Gemini-2.5-flash 0.1412 / -0.0121 across 1,712 held-out test entries. Sharpest existing evidence that domain-specific fine-tuning on verification-gated sim data buys substantially more failure-reasoning capability than raw frontier-VLM scale for robot manipulation.
  • [2026-08-06] CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation: CineTechBench evaluates whether MLLMs (15+ models) can judge cinematography before they are deployed as automatic judges: best commercial (GPT-4o, Gemini-2.5-Pro) hits ~70% on 7-dimension image QA, best open-source only 55%; focal-length is a persistent trough (<45% for most). Camera-rotation direction and static-shot recognition are consistent failure modes — a caution flag for any downstream benchmark that uses these VLMs as film-craft judges.
  • [2026-08-03] Robot-eval-bench: Benchmarking Task Success Evaluation for VLMs: Kite ML’s Robot-eval-bench is the first head-to-head benchmark of frontier VLM judges (Gemini 3.1 Pro, Gemini 3.6 Flash, Claude Opus 5, GPT-5.6 Sol, Kimi 3) on robot-policy episode success across 16 real-robot datasets, with manufactured failures (truncate/shuffle/reverse/task-swap) balancing the success-heavy source data. Two headline results: (1) family-inverts-by-input-form — Gemini strong on keyframes (Flash 0.94) but collapses on video (Flash 0.63), while reasoning models are steady or improve (GPT-5.6 Sol 0.90→0.92 keyframes→video); (2) cost also inverts — Gemini cheapest on native video (~$0.0005/episode), Claude/GPT ~10× more since they consume video as ~16 frames. Bigger isn’t better within a family: Flash beats Pro on both forms. On the keyframes efficient frontier only Kimi 3 (cheapest) and Gemini 3.6 Flash (most accurate) survive; Claude Opus 5 is strictly dominated. Sharpens the judge-backbone-sensitivity finding IGenBench raised on the image side, now for robot success detection.
  • [2026-07-29] FilmBench: A Film-Grade Benchmark for Cinematic Video Generation: FilmBench’s expert-grade automatic evaluation agent (with the open-sourced FilmOps Cinematic Language operator suite) achieves Spearman ρ = 0.95 (T2V) and 0.96 (R2V) with human model-level rankings — matching RBench’s best-in-class calibration but on a cinematic-taxonomy rubric (3 axes × 12 components × 35+3 sub-metrics) rather than physics-commonsense, showing the VLM-judge-with-domain-taxonomy pattern extends beyond physics.
  • [2026-07-29] Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies: Not a VLM-as-judge but occupies the same “scalable non-hardware policy eval” niche: R2S2R uses closed-loop policy rollouts inside an aligned generative world model to rank checkpoints, catch regressions, and predict failure regions before committing to costly hardware trials.
  • [2026-07-27] TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics: TOPReward (Chen, Harrison, Lee et al., UW/AI2/Meta) adds a robot-manipulation instance of the affirmative-token-log-likelihood readout — teacher-force “True” on a completion query and use the per-frame log-prob as dense reward, no fine-tuning. Sharpest evidence yet that open-source video-VLMs (Qwen3-VL-8B 0.857 mean VOC, Molmo-2-8B 0.417) can outperform GVL-style rank scoring (0.194 / -0.016) when the readout avoids numeric generation, though Gemini-2.5-Pro flips (0.433 vs 0.541 GVL) — backbone-sensitivity datapoint sibling to IGenBench’s judge-selection finding. Also improves success detection ROC-AUC and enables real-SO-100 AWR gains from 50 noisy demos.
  • [2026-07-22] Introducing Perceptron Mk1: Perceptron Mk1 is pitched into the same success/failure-labeling and reward-model-annotation slot as Instance Labs and DenseReward, but as a closed-source video-VLM API rather than a fine-tuned per-task model. No calibration numbers or human-alignment measurements in the launch post — the productized-verifier open question this page tracks stays open.
  • [2026-07-22] Physion-Arc 1.0: Benchmarking Video Agents on Minute-Long Video Generation: Physion-Arc 1.0 is a new benchmark for minute-long multi-shot agentic video generation that scores six systems (Runway, Luma, MiniMax, Kling, Utopai, TapNow) across narrative coherence, cinematic language, and production quality — a candidate testbed for how well VLM-judge pipelines transfer from single-clip physics/realism scoring to long-form narrative/directorial quality. Exact judge protocol (VLM-judge vs. human vs. hybrid) not readable at filing time since the blog URL 404s.
  • [2026-07-21] DriftWorld: Fast World Modeling through Drifting: DriftWorld (Yilun Du et al.) uses a fast one-step drifting WFM as an offline policy-evaluation simulator: rollout-based scores correlate with ground-truth real-world performance at r = 0.9916 (Robomimic Lift), 0.9250 (Can), 0.9515 (Push-T), outperforming Ctrl-World. Third datapoint in the WFM-as-policy-evaluator thread alongside RoboWorld (Step Forcing) and Veo-Robotics.
  • [2026-07-21] Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence: Apple-π’s evaluation suite is explicitly hybrid — MLLM-based subjective scoring combined with physics-law-grounded objective measures — a concrete response to the PhyGenBench-flagged problem that generic VLM/T2V judges systematically pass physically implausible smooth-motion videos. On 400 videos × 10 classical-mechanics tasks, the hybrid design enables stage-resolved diagnosis (Perception / Formulation / Deduction) that a VLM-only judge cannot deliver.
  • [2026-07-19] Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation: SpectraReward introduces a seventh output shape on this page’s enumeration: prompt-recovery log-likelihood — teacher-force the MLLM to reproduce the original prompt conditioned on the generated image and read out the average token-level log-likelihood as the reward. One forward pass, no rubric, no preference data, no reward-model fine-tuning; dense per-token signal composes cleanly with GRPO and sidesteps the pointwise-CoT reward-variance-collapse failure mode Stable-Layers documented. Self-SpectraReward specializes this for UMMs — policy’s understanding branch scores its own generation branch. Sweep over two diffusion models, three RL algorithms, nine MLLM backbones (4B–235B, four families), five OOD T2I benchmarks. Headline negative-scaling claim: larger reward MLLMs are not always better, and Self-SpectraReward can match or surpass much larger external reward models, reframing the bottleneck as reward-policy alignment rather than reward-model size.
  • [2026-07-17] DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation: DenseReward is the robot-manipulation, dense-reward instance of the VLM-as-evaluator pattern — a VLM-based reward model trained on simulator-synthesized failure trajectories (collisions, missed grasps, drops, recoveries) that outputs per-frame scalar rewards conditioned on image + language, sitting between the productized success-detector shape (Instance Labs — Verifying Robot Learning Episode Success) and the training-free continuous logit-expectation verifier of LLM-as-a-Verifier: A General-Purpose Verification Framework; reports beating general-purpose VLMs and prior robotic reward models on dense-reward prediction in sim and real, and provides usable reward signal for downstream MPC and RL.
  • [2026-07-13] Instance Labs — Verifying Robot Learning Episode Success: Instance Labs is the first filed commercial productization of VLM-as-evaluator for robot learning — pitched as a per-episode success-verdict-plus-evidence service that ingests (task, camera footage) and outputs binary verdicts across arbitrary robots and camera angles. No calibration numbers, no benchmark, no disclosed backbone at filing time, so the human-alignment question this page tracks stays open for the productized-verifier shape.
  • [2026-07-09] RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies: RoboDojo publishes a “systematic analysis of current policy performance” over its 30-policy sweep alongside the leaderboard, positioned as the analog of RoboArena’s §3.3 VLM+LLM policy-report pipeline but grounded in structured sim + real success metrics rather than free-form evaluator feedback — a control point for how much of the analysis-of-policy-sweeps shape survives when the underlying data is scalar-metric-only.
  • [2026-07-08] LLM-as-a-Verifier: A General-Purpose Verification Framework: LLM-as-a-Verifier is a training-free general-purpose verification framework that replaces discrete LM-judge score prompting with a continuous score computed as the expectation over the scoring-token logit distribution — a sixth output shape on this page’s enumeration (scalar tier / CoT-then-scalar / discrete pick / benchmark-time QA / retrospective synthesis / continuous logit-expectation). Directly addresses the pointwise-CoT reward-variance-collapse failure mode flagged by Stable-Layers by avoiding ties in the readout. Three verification-scaling axes are shown to compose: score granularity, repeated evaluation, and criteria decomposition. Reports SOTA on Terminal-Bench V2 (86.5), SWE-Bench Verified (78.2), RoboRewardBench (87.4), MedAgentBench (73.3) without verifier training, and the same continuous scores serve as dense reward improving SAC and GRPO sample efficiency.
  • [2026-07-05] Implications of Large-Scale Test-Time Compute (Noam Brown): Noam Brown’s framing sharpens the evaluator side of the ledger: if performance is a curve over inference budget rather than a scalar, benchmark orgs (and any VLM-judge / LLM-judge harness they build) must either track compute consumed during evaluation or impose explicit token / cost / time budgets. Otherwise same-model, different-scaffold comparisons are incoherent. Directly relevant to how this wiki treats LMM-as-judge scoring at fixed vs unfixed sampling budgets.
  • [2026-07-04] RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation: RoboWorld’s VLM judge uses a 0–5 task-progress rubric scored from fixed external views while isolating world-model artifacts to the wrist view, avoiding binary success and fairly crediting policies under imperfect neural-simulator rollouts.
  • [2026-06-30] Segmenting Robot Video into Actionable Subtasks (WGO-Bench): Macrodata’s WGO-Bench uses Gemini-3.5-Flash as a Scale-style LLM-as-judge for subtask labels — strict JSON rubric on action / object / source-destination correctness; same pattern as EditReward and EdiVal-Agent but applied to robot/egocentric video subtask names rather than image-edit quality.
  • [2026-06-29] SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation: SimFoundry is the non-VLM counterpart to the VLM-as-judge / Veo-as-evaluator pattern: physics simulator rollouts of cousin scenes predict real policy success at r = 0.911, MMRV = 0.018, reportedly beating PolaRiS. A useful calibration point for how much of policy-evaluation accuracy is recoverable without a generative or judge VLM in the loop.
  • [2026-06-29] ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation: ABACUS uses the same model’s understanding branch as the reward model for its generation branch via cycle-consistent GRPO — pushing the VLM-as-judge pattern to a self-rewarding single-model loop for count-faithful image generation, sidestepping external annotators.
  • [2026-06-29] Qwen-Image-2.0-RL Technical Report: Qwen-Image-2.0-RL builds two task-specific composite reward models by fine-tuning VLMs in a pointwise CoT-then-scalar paradigm — same shape as UnifiedReward-Flex — covering alignment/aesthetics/portrait fidelity for T2I and instruction-following/face-ID preservation for editing. The intra-group reward-range filtering stabilizer is a direct response to the Stable-Layers reward-variance-collapse failure mode for pointwise VLM scoring under GRPO.
  • [2026-06-24] Krea 2 Technical Report: Krea 2 uses VLMs as evaluators inside the data pipeline rather than at training-loss time. Pattern A: large VLM crafts a task-specific system prompt for a filtering job, produces a pseudo-labeled set, then a small (<1B) DINOv3 / SigLIP-2 classifier is trained to run the filter at scale. Pattern B: SigLIP-2 sparse-autoencoder features are each annotated by a VLM from top-k activating samples, producing an unsupervised tag system used to filter visual artifacts without training an explicit per-artifact classifier. Pattern C: VLM-named hierarchical k-means clusters with human review for cluster-level drops at midtraining.
  • [2026-06-20] Spotlight: Identifying and Localizing Video Generation Errors Using VLMs: Spotlight measures the missing calibration number for VLM-as-judge in the T2V error-localization shape: zero-shot Gemini 2.5 Pro reaches S+P 0.250 / 12.6% match coverage, zero-shot Qwen3-VL-{8B,30B-A3B}-Instruct ~0.14, vs human 0.508 / 26.8%; a multi-agent inference-time strategy (one query per error category) doubles Qwen3-8B to match Gemini, evidence that decomposing the rubric per error type is the cheapest way to lift open VLM judges on localized video error tasks.
  • [2026-06-11] Do generative video models understand physical principles? (Physics-IQ): Physics-IQ uses Gemini-1.5-Pro in a 2AFC realism-discrimination protocol (real-vs-generated pair, MLLM picks the fake) as a visual realism metric rather than a correctness metric — chance accuracy means the model fooled the judge; Sora achieves the most-realistic 55.6% (near chance) while ranking worst on the protocol’s physics motion-mask metrics.
  • [2026-06-08] RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies: RoboArena’s §3.3 qualitative-analysis module adds a fifth shape to the VLM-as-evaluator enumeration: a GPT-4.5 + o3 pipeline that synthesizes a per-policy report (language-following ability, multi-step-task robustness, qualitative failure modes) from a database of evaluation rollouts, with every claim in the report grounded by citations to specific rollout videos as evidence. Distinct from the scalar-tier / CoT-reward / discrete-pick / per-sample-QA shapes already on this page: here the LLM is doing retrospective synthesis over a corpus of evaluation data, not scoring a single artifact — which is the natural shape when the underlying evaluation signal (human preference + free-form feedback + progress score) is already collected and the question is “what does it mean.”
  • [2026-06-06] AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects: AssemblyBench uses GPT-4.1 as a structured data generator rather than a scorer — two-stage prompting (consistent part naming across all parts → per-step instruction text with target-part highlighting and color-coded reference diagram) for VLM-driven instruction-manual synthesis from CAD assemblies.
  • [2026-06-05] EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks: EgoTL critiques VLM auto-labeling as the noisy supervision substrate (incorrect action labels, CoT, spatial annotations) compounding into long-horizon spatial-grounding errors, and offers human spoken CoT with word-level timestamps as the corrective.
  • [2026-06-03] VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation: VideoPhy-2 trains VideoPhy-2-Autoeval, a 7B multi-task evaluator distilled from ~50K human SA/PC/rule annotations that beats Gemini-2.0-Flash-Exp by +80.8%/+236.4% Pearson on SA/PC for unseen prompts and reaches 78.7% rule-classification accuracy vs Gemini’s 59.2% — strong evidence that small distilled-from-humans evaluators outperform frozen frontier VLM judges on physical commonsense.
  • [2026-06-03] Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X): RBench (DAGroup PKU) — automatic robot-oriented video-generation benchmark reporting Spearman 0.96 with human evaluators across 25 models, materially above PhyGenEval’s ~0.65 on a comparable physics-realism task. Sub-metrics (structural consistency, physical plausibility, action completeness) are reported per (task × embodiment) cell, providing the strongest filed automatic-vs-human alignment for video-generation evaluation.
  • [2026-06-02] Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning: Stable-Layers — concrete fix for the pointwise-VLM-score-collapse failure mode flagged across this page: pointwise CoT scoring on five edit-centric criteria gave near-zero within-group variance under GRPO, so the authors add a grid-based pass where the VLM re-scores all candidates side-by-side. The pairwise-grid head sits on top of the pointwise head and is what makes Flow-GRPO on Qwen-Image-Layered learn at all. Direct evidence that for GRPO post-training the loss shape question (pointwise vs pairwise vs CoT-then-scalar) is bottlenecked by reward-variance survival.
  • [2026-06-01] PAI-Bench: A Comprehensive Benchmark For Physical AI: PAI-Bench reports Pearson r ≈ 0.97 between its Qwen3-VL-235B-judged Domain Score (over 5,636 human-curated QA pairs anchored to the Cosmos-Reason1 ontology) and human ELO on pairwise physics-plausibility A/B — the strongest published human-correlation result for the benchmark-time MLLM-as-judge shape on video, complementing IGenBench’s 0.90 on infographics and partially answering RISE-Video’s open calibration question. Domain Score and Quality Score together separate Veo3 (Domain 86.8) from open-source leaders (Wan2.2-I2V-A14B 87.1, Cosmos-Predict2.5-2B 84.9), with all models trailing the source-video physics ceiling of 89.8.
  • [2026-06-01] Sophon — AI Research Catalog of Evals, Tools, and Labs: External catalog (sophon.at) indexing leaderboards, eval-lifting tools, and per-benchmark saturation — surfaces 8 frontier benchmarks at >93% top score, with MBPP already at 100%, useful as a dashboard for picking non-saturated evaluations.
  • [2026-05-31] ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMs: ArtifactLens — low-supervision endpoint of the VLM-as-judge axis: SOTA on five human-artifact benchmarks using a frozen pretrained VLM + ICL with only hundreds of labeled examples per category, vs the tens-of-thousands fine-tuning recipe used by ERNIE-Image-Aes and UnifiedReward-Flex. Scaffolding is in-context learning + text-instruction optimization; generalizes to object morphology, animal anatomy, entity interactions, and AIGC detection. First filed datapoint that pretrained VLMs already encode artifact-detection capability and only need prompt-side unlocking.
  • [2026-05-28] Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation: Qwen-Image-Bench (Q-Judger) — first open-weight, Apache-2.0 benchmark-time T2I judge: a fine-tuned Qwen3.6-27B that scores generations along a 5-L1 / 23-L2 / 56-L3 hierarchy with per-prompt expert-seeded facet metadata and deterministic CoT inference. Sits between the closed Gemini-2.5-Pro IGenBench protocol (the only judge clearing r=0.8 against humans on infographics) and the GRPO-time UnifiedReward-Flex — same benchmark-time MLLM-as-judge shape as IGenBench but with the judge itself shipped open.
  • [2026-05-27] Planning at Inference: MCTS Test-Time Scaling for Long Video Generation: Search-based test-time scaling for long video is verifier-bottlenecked: the authors flag explicitly that “results are limited by the quality of current generators and verifiers” — a direct datapoint that MCTS-over-continuations only gains as much as the reward/verifier model can score.
  • [2026-05-23] A Very Big Video Reasoning Suite: First filed paper to argue against VLM-as-judge as the default for video-reasoning evaluation. VBVR-Bench uses task-specific rule-based scorers (e.g. Key Door Matching scored on target ID 30% / path validity 30% / path efficiency 20% / animation quality 20%) over 100 deterministic visual-reasoning tasks, defended on reproducibility, determinism, granular verifiability, and freedom from LMM hallucination. Crucially, the rule-based scores are validated against human pairwise preferences (strong positive win-ratio correlation), answering the “is rule-based scoring human-aligned?” objection. Methodologically opposed to its sibling RISE-Video: Can Video Generators Decode Implicit World Rules?, which uses LMM-as-judge on essentially the same model set the same week.
  • [2026-05-23] UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?: Adds a new use case for VLM-as-judge — scoring intermediate generated images in a Generate-then-Answer pipeline. RA (Reasoning-Alignment) and AL (Answer-Alignment) metrics let the benchmark attribute GtA failure modes to inaccurate / implausible / misaligned intermediate artifacts rather than to the GtA protocol itself. First filed instance of the VLM-judges-the-chain-of-reasoning shape.
  • [2026-05-23] IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation: IGenBench — first benchmark for text-to-infographic reliability, with the same benchmark-time MLLM-as-judge shape as RISE-Video but on still images. Atomic yes/no questions per (prompt, image), half decomposed from the prompt and half instantiated from three expert-seeded dimensions (Data Completeness, Ordering, Encoding); Gemini-2.5-Pro produces both Q-ACC (per-question) and I-ACC (per-image all-or-nothing). Includes an explicit judge-selection study across 12 MLLMs: Gemini-2.5-Pro is the only judge to clear r = 0.8 against humans, answering one of the open questions on this page about judge-backbone sensitivity for the benchmark-time shape.
  • [2026-05-23] RISE-Video: Can Video Generators Decode Implicit World Rules?: RISE-Video — first filed video-scale instance of the VLM-as-evaluator pattern. An LMM (OpenAI-API-compatible) ingests human-written per-sample reasoning questions plus a manually-designed frame-sampling schedule, and emits two aggregates: a relaxed weighted score (W.Score) and a strict accuracy. The relative ranking discriminates 11 TI2V generators sharply (Acc 1.9–22.5%), but the LMM-judge’s absolute correlation with human raters on video reasoning is unvalidated. Partially answers the open question on the previous concept-page version about whether VLM-as-evaluator survives at video scale: yes for relative ranking, unknown for absolute calibration.
  • [2026-05-23] Unified Personalized Reward Model for Vision Generation: UnifiedReward-Flex — per-prompt context-adaptive VLM judge (Qwen3-VL / Qwen35), trained SFT-then-DPO on 90K distilled reasoning traces, dropped into GRPO post-training of FLUX.1-dev and FLUX.2-Klein-9B (LoRAs released). Positions itself as a single context-adaptive reward that replaces the bag of fixed reward models used in HunyuanImage 3.0’s MixGRPO stack. Adoption already broad — Meta, NVIDIA, Tencent Hunyuan, ByteDance, Apple papers cite UnifiedReward.
  • [2026-05-22] DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models: Establishes the discrete-pick endpoint — MLLM-as-verifier picks among DiffThinker’s pixel-space reasoning candidates on Jigsaw-level-4, beating either model alone. Not a reward model per se, but the same VLM-as-judge primitive in a multi-candidate setting.
  • [2026-05-22] ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization: ERNIE-Image-Aes establishes the static-rubric endpoint — 8B VLM (InternVL-3 → ArtiMuse → ERNIE-Image-Aes) trained on Swiss-tournament labels, category-balanced. Documents the bias modes of the prior generation of scorers (LAION-Aesthetic, ArtiMuse, UniPercept) and roughly doubles SRCC on the deliberately bias-exposed ERIA-1K benchmark.

Open questions

Section titled “Open questions”
  • When does rule-based scoring beat VLM-as-judge, and when is it the other way around? VBVR vs RISE-Video is the cleanest filed contrast — same model set (Sora 2, Veo 3.1, Kling, Wan, CogVideoX, Hunyuan, LTX, Runway Gen-4), same week, opposite scoring philosophy. VBVR shows rule-based is feasible whenever ground-truth videos are procedurally available; RISE-Video covers cases where they aren’t (commonsense/societal/experiential/temporal reasoning over real-world video). No filed paper directly A/Bs the two scoring approaches on the same tasks, which would isolate the cost of LMM-judge noise.
  • Does a context-adaptive single-VLM reward (UnifiedReward-Flex) actually beat a fixed multi-reward stack (HunyuanImage 3.0’s MixGRPO setup) on a controlled head-to-head? The Flex abstract claims yes; the paper body retrieved so far does not expose the comparison numbers, and no third party has run the A/B yet.
  • What’s the right loss shape for training the VLM judge — pairwise rank, pointwise score, or CoT-then-scalar? UnifiedReward-Flex uses CoT-then-scalar; UnifiedReward / UnifiedReward-Think variants support pair+point; ERNIE-Image-Aes is pointwise from Swiss-tournament labels. No paper isolates which loss best survives downstream GRPO use.
  • How robust is a context-adaptive judge to reward hacking by the generator? A static rubric is hackable in a predictable way; a per-prompt dynamic rubric may be more or less hackable, but no filed paper measures generator-judge co-adaptation under extended GRPO training. VBVR’s rule-based scorers are also gameable in principle (a model that overfits the exact path-validity check might still fail human inspection), but VBVR’s human-alignment study at least rules out the most obvious failure modes at fine-tuning scale.
  • Does the VLM-as-evaluator paradigm survive at video scale? Partially answered: RISE-Video shows the pattern discriminates TI2V models cleanly (Acc 1.9–22.5%) using a frozen LMM judge with per-sample human-written questions, but does not measure the judge’s calibration against human raters. A video-generator GRPO release with Flex (or a RISE-style benchmark-time judge) as reward would be the natural next data point.
  • Cost: a per-sample CoT-reward call from an 8-32B VLM is dramatically more expensive than a forward pass through CLIP or HPSv3. What’s the iso-quality cost crossover for GRPO post-training between a cheap fixed reward and an expensive context-adaptive one? VBVR’s rule-based scorers are essentially free per evaluation, so the cost question is also a usable axis along which to compare scoring philosophies.
  • ERNIE-Image-Aes and UnifiedReward-Flex disagree on the right training signal (curated tier labels vs distilled reasoning traces). A study mixing the two — Swiss-tournament-labeled pairs as the DPO half of a Flex-style two-stage pipeline — has not been filed.
  • For benchmark-time use (RISE-Video / IGenBench shape), the LMM-judge is consumed without retraining on the target domain. Partially answered for images by IGenBench’s 12-MLLM judge-selection study (Gemini-2.5-Pro is the only one above r = 0.8 against humans on infographics); RISE-Video does not vary the judge model in its public artifact, so the video side is still open.
  • Does the IGenBench-style “prompt-decomposed atomic yes/no” rubric transfer to settings where the prompt does not carry explicit constraints (open-ended T2I, aesthetic-only prompts)? Both RISE-Video and IGenBench rely on the prompt to seed the question set; the static-rubric path (ERNIE-Image-Aes) is the only filed approach that works without prompt-side structure.
  • For the new UniG2U “VLM scores intermediate visuals in a chain of reasoning” shape: how reliable are RA / AL when the intermediate images come from very different generator families (diffusion vs AR vs MAR)? UniG2U does not yet ablate the judge.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2025-03

Section titled “2025-03”

2024-12

Section titled “2024-12”

2024-11

Section titled “2024-11”

2024-10

Section titled “2024-10”

Unknown date

Section titled “Unknown date”