Physion-Arc 1.0: Benchmarking Video Agents on Minute-Long Video Generation

TL;DR

Physion-Arc 1.0 is a new benchmark from Physion Labs that evaluates video agents — not raw single-clip generators — on minute-long, multi-shot video production. The suite tests six systems (Runway, Luma, MiniMax, Kling AI, Utopai Studios, TapNow) on 100 screenplays yielding 600 generated videos, scored across narrative coherence, cinematic language, and production quality. Runway Agent 2.0 ranks #1 overall and leads every evaluation dimension; the announced motivation is that individual scene-generation quality no longer captures what matters once agents produce complete multi-scene experiences with pacing and direction.

Note: the blog URL that appeared in the Slack pointer (physionlabs.ai/blog/physion-arc1.0) returned 404 at filing time and could not be fetched. Details here come from the announcement text circulated on LinkedIn by Anastasis Germanidis (Runway) and third-party coverage. The Physion-Eval dataset and 20260322-dated metadata JSON that accompanies the benchmark are hosted at huggingface.co/datasets/PhysionLabs/Physion-Eval.

Key claims

Scope: video agents rather than single-clip T2V models — the unit under test is a system that produces a complete, multi-scene, minute-long video from a screenplay input [announcement].
Corpus: 100 screenplays × 6 systems = 600 videos evaluated [announcement].
Evaluation dimensions span narrative coherence, visual continuity, pacing, direction, and emotional impact — explicitly aimed at “whether individually generated scenes come together as a complete experience” [announcement].
Systems evaluated: Runway (Agent 2.0), Luma, MiniMax, Kling AI, Utopai Studios, TapNow [announcement].
Reported leaderboard: Runway Agent 2.0 ranks first overall and leads every dimension, with subjective / cinematic-taste dimensions showing the widest margin [announcement].
Companion dataset release: PhysionLabs/Physion-Eval on Hugging Face (videos.zip + Physion_Eval_20260322.json + all_metadata_with_caption.json) — the released metadata JSON does not currently include the text/image prompts used to generate the videos, per an open community discussion on the dataset repo.

Method

Physion-Arc feeds a screenplay to each video-agent system and collects the resulting minute-long, multi-shot video. Six systems are evaluated on 100 screenplays. The evaluation appears to combine subjective / cinematic dimensions (narrative coherence, visual continuity, pacing, direction, emotional impact) with production-quality axes — the announcement highlights eight dimensions on which Runway Agent 2.0 ranks first. Exact judge protocol (VLM-judge vs. human rater vs. hybrid) and per-dimension rubric definitions are not readable from the announcement text and could not be pulled from the blog page at filing time.

The companion Physion-Eval dataset on Hugging Face ships the 600 generated videos plus JSON metadata (rated dimensions, per-system leaderboard positions), which suggests the scores are released alongside the benchmark rather than kept behind a submission portal — a design choice that makes reproduction on new models straightforward but complicates leaderboard integrity over time.

Results

Runway Agent 2.0 places #1 overall and #1 on all eight announced evaluation dimensions [announcement].
Runway’s stated advantage is largest on the subjective / cinematic-taste dimensions [announcement].
The comparison lineup — Runway, Luma, MiniMax, Kling AI, Utopai Studios, TapNow — is unusual in explicitly including “agentic” / director-style systems (Runway Agent 2.0, TapNow) alongside single-shot commercial generators (Luma, Kling), which likely dominates the leaderboard shape (see Why it’s interesting) [announcement].

Detailed numeric scores per dimension, per model, are not available from the announcement text alone and would need the blog page or dataset JSON to reproduce.

Why it’s interesting

Physion-Arc adds a genuinely new axis to the Video Generation Benchmarks cluster. All previously filed entries — Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench) (PhyGenBench), VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation (VideoPhy-2), Do generative video models understand physical principles? (Physics-IQ) (Physics-IQ), Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X) (RBench), RISE-Video: Can Video Generators Decode Implicit World Rules? (RISE-Video), Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence (Apple-π), Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation? (Dream.exe) — probe physical realism, implicit world rules, or robot-task executability over short (5-10s) clips. Physion-Arc instead probes narrative direction over multi-shot, minute-scale output — the axis that becomes load-bearing once systems like Runway Agent 2.0, Runway GWM-1 launch broadcast — world model, avatars, audio, multi-shot editing GWM-1, and Luma’s Modify-with-Instructions cross the “one shot at a time” barrier.

Two follow-on questions that make this benchmark worth watching: (a) does the ranking survive re-scoring with a different VLM-judge or human panel — Spotlight: Identifying and Localizing Video Generation Errors Using VLMs found that different VLM judges disagree on which generator is worst on short-clip error localization, so the same effect at minute-scale would be diagnostic; (b) the finding that closed models don’t dominate open ones on physical-commonsense benchmarks (VideoPhy-2, Physics-IQ) inverts here — Runway Agent 2.0 is a closed agent product beating a mixed lineup — which suggests narrative/cinematic-quality is a very different capability axis from physics-commonsense and may reward system-level agent scaffolding more than raw model scale. This matters to Luma directly since Luma is one of the six evaluated systems.