Skip to content
Bud's Wiki

HarnessEval-W: Agentifying the Evaluation of Visual Worlds

TL;DR

Section titled “TL;DR”

HarnessEval-W is an agentified evaluation pipeline for visual world models: rather than applying a fixed rubric or a single VLM-judge call, a parent agent decomposes each evaluation question into measurable subproblems and spawns specialized sub-agents equipped with tailored context and diagnostic tools; the parent then validates gathered evidence and summarizes a final verdict, producing a transparent evidence tree per case rather than a scalar. Applied to 18 representative world models over 330 evaluation cases, the pipeline’s judgments align with human preferences while providing verifiable, fine-grained diagnoses. The pipeline, skill libraries, cases, and results are open-sourced as a live benchmark inviting community contribution of new skills and cases.

Key claims

Section titled “Key claims”
  • The load-bearing failure of existing world-model benchmarks is that scores are computed brute-force with no reasoning chain to examine or verify [Abstract].
  • HarnessEval-W’s parent–sub-agent hierarchy turns each evaluation case into an evidence tree whose complete reasoning chain justifies the result [Abstract].
  • Sub-agents receive tailored context and diagnostic tools per subproblem rather than a single monolithic rubric [Abstract].
  • Applied to 18 representative world models over 330 evaluation cases, HarnessEval-W’s judgments closely align with human preferences [Abstract].
  • The pipeline is released as a live benchmark — harness, skill libraries, cases, and results all open-sourced with an explicit community-contribution charter [Abstract].

Method

Section titled “Method”

HarnessEval-W ports the “harness” paradigm from the LLM-eval ecosystem into world-model benchmarking. Each evaluation case is handled by a parent agent that first interprets the case’s context, decomposes the target evaluation question into a set of measurable subproblems, and then spawns specialized sub-agents — each with its own tailored context and diagnostic-tool set — to reason over its assigned subproblem. Sub-agent outputs feed back to the parent, which validates the gathered evidence and summarizes it into the final verdict for that case. The output is not a scalar but an evidence tree whose branches (subproblem → sub-agent tool calls → gathered evidence → parent verdict) can be inspected by a human. The paper positions this against “static rubric” evaluators whose scoring path is opaque.

Results

Section titled “Results”
  • Coverage: 18 representative world models over 330 evaluation cases [Abstract].
  • Alignment: HarnessEval-W’s judgments closely align with human preferences [Abstract] — no numeric human-correlation figure is quoted in the abstract; the launch tweet advertises the leaderboard and full technical report for details.
  • Deliverables: the full pipeline (harness + skill libraries + evaluation cases + results) is publicly released; the tweet lists code (github.com/mirros-lab/harnesseval-w), leaderboard (mirros-lab.github.io/HarnessEval-W), a blog post, and the arxiv technical report.
  • Evaluated labs per the launch tweet include ByteDance, Alibaba, Kuaishou, MiniMax, NVIDIA, Tencent, Ant Group, and Lightricks — a coverage sweep across the closed-flagship and open-weights sides of the WFM landscape.

Why it’s interesting

Section titled “Why it’s interesting”

HarnessEval-W is the first filed entry in the video-WM benchmark cluster to move the evaluator itself from a single-call VLM judge (RISE-Video, PhyGenBench, Physion-Arc) or a fixed automatic pipeline (RBench, VideoPhy-2-Autoeval) to an agentic decomposition that emits an evidence tree per case. That directly targets two of Video Generation Benchmarks‘s open questions — whether one-shot LMM-judge scoring is calibrated at absolute (not just rank) accuracy, and whether inference-time decomposition beats holistic prompting — with the same decomposition-per-error-category strategy that Spotlight: Identifying and Localizing Video Generation Errors Using VLMs already showed doubles open VLM-judge accuracy on video error localization. The design is also the video-world-model analog of the behavior-judge — compiling Agent Behavior specs into deterministic + semantic judges “compile the rubric into typed predicates + semantic residual” pattern from text-agent evaluation and of the LLM-as-a-Verifier: A General-Purpose Verification Framework “criteria decomposition composes with verification-scaling” recipe for LLM verifiers. Being released as a live benchmark places it alongside WorldModelGym: a decision-based fidelity benchmark for world models and GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation as filed community-extensible WFM evaluation platforms.

See also

Section titled “See also”