PAWBench: How Far Are We from Probabilistically Aligned World Modeling?

TL;DR

PAWBench formalizes probabilistic alignment as a distribution-level requirement for video-based world models — a good world model should reproduce not just one plausible trajectory but the correct distribution of possible behaviors under the same initial observation and action. Existing benchmarks (RISE-Video, PhyGenBench, VideoPhy-2, Physics-IQ, Physion-Eval) score individual-video plausibility; PAWBench scores repeated generations against a reference distribution. Across 50 scenarios and 11 current systems, no model consistently matches reference probabilities while covering the full range of valid behaviors, and the paper further probes whether language prompts, initial-noise sampling, or model training reshape the predictive distribution.

Key claims

Probabilistic alignment — reproducing the distribution of physically valid behaviors under a fixed initial observation + action — is a distinct evaluation axis from single-sample plausibility, and existing benchmarks do not test it [Abstract, §1].
The benchmark contains 50 scenarios covering physical processes with multiple valid outcomes under identical initial conditions [Abstract].
Across 11 evaluated video-generation systems, no model consistently matches the reference probabilities while also recovering the full range of valid behaviors — surface plausibility and distributional coverage are not simultaneously achieved by any current model [Abstract].
PAWEval is an outcome-level protocol that converts repeated video rollouts into empirical distributions over possible physical behaviors, so the evaluator scores the sampler rather than individual samples [Abstract, §3].
The paper further tests three interventions — language-prompt variation, initial-noise sampling variation, and training-time modifications — as candidate levers for reshaping the model’s predictive distribution [Abstract, §4].

Method

PAWBench frames a video generator as a stochastic sampler over world dynamics rather than as a deterministic map from prompt to video. Each of the 50 scenarios pins the initial observation and action, then defines a reference distribution over discrete physical outcomes (a die landing on face N, a pendulum falling left vs right, an object bouncing off a surface at some angle bucket, etc.). To evaluate a model, the same conditioning is passed many times with independent seeds; the resulting videos are converted to categorical outcomes via PAWEval — an outcome-level classifier that maps each rollout to one of the enumerated buckets — and the model’s empirical outcome histogram is compared to the reference.

The evaluation reports two distinct failure modes explicitly: (i) mode collapse — the model concentrates on one valid outcome and skips others, and (ii) miscalibration — the model covers the outcome set but assigns wrong relative probabilities. Both are captured by the same protocol because both show up as divergence between empirical and reference histograms rather than as low visual quality on any individual clip. The intervention study then repeats the protocol under paraphrased language prompts, held-fixed vs varied initial noise, and small training perturbations to isolate which of the three levers a downstream user could pull to shift the sampler’s distribution.

Results

Across 11 systems on 50 scenarios, no model simultaneously (a) covers the full support of valid outcomes and (b) matches the reference probability weights [Abstract].
Language prompts, initial noise, and training changes each reshape the predictive distribution but none of the three interventions closes the gap on their own [Abstract].
Individual videos remain visually plausible under the same models that fail distributional alignment — reproducing the RISE-Video / Physion-Eval / VideoPhy-2 pattern that per-frame surface quality decouples from the deeper capability being probed.

Why it’s interesting

PAWBench opens a distribution-level axis on the Video Generation Benchmarks cluster that every prior filed entry misses. RISE-Video (RISE-Video: Can Video Generators Decode Implicit World Rules?) probes strict per-video reasoning accuracy and finds a 22.5% ceiling; Physion-Eval (Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning) finds 83–94% of generated videos contain at least one human-identifiable physical glitch; VideoPhy-2 (VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation) reports 32.6% joint SA∧PC as the top score. All three score one video at a time. PAWBench is the first filed benchmark on the wiki that says “run the model N times on the same prompt and check the histogram”. This is the missing evaluation counterpart to FlowWM (Flow Matching in Feature Space for Stochastic World Modeling), which argues on the training side that world models should be trained as stochastic samplers with flow matching in feature space rather than deterministic predictors — and shows F1-error 4.31 vs 17.1 on Bouncing-Shapes where all outcome modes can be enumerated. PAWBench formalizes the evaluation shape those training-side arguments have been implicitly asking for.

The Kyle-note question (“why do WMs struggle to generate plausible probability distributions?”) lines up with two structural failure modes the wiki has been circling: (i) most video generators are trained on a next-frame or flow-matching objective that has no explicit distributional loss over outcomes, only over pixels — a sampler trained to match p(pixels | conditioning) can produce plausible pixels while placing all its mass on one mode of the outcome distribution; and (ii) initial-noise, prompt, and CFG all act as low-dimensional handles on a very high-dimensional latent space, so the intervention study finding that no single lever closes the gap matches what one would expect if the model has learned a low-mode joint over outcomes rather than a rich one. That connects PAWBench to the noise-conditioning-free-diffusion candidate cluster (Sun et al., Geometry-of-Noise) where the conditioning distribution over noise is being actively questioned. The natural follow-up for Luma is whether OPAL’s mode coverage can be measured on the PAWBench protocol without any Luma-specific changes to the eval side.