Skip to content
Bud's Wiki

Training Object Permanence in World Models

TL;DR

Section titled “TL;DR”

Object permanence — the expectation that objects continue to exist when occluded and cannot pass through solid barriers — is a cognitive-science-defined prior that current video-generation world models are not systematically evaluated on. WROP is a cognitive-science-inspired data + evaluation package: 150 hand-designed Blender tasks across six categories, procedurally randomized over speed / lighting / camera / other nuisance parameters to yield >10,000 samples each. The authors release a 1.5M-sample training corpus, a 300-question exam, and PWM-WROP — a 16B world model trained on it. In a blind pairwise Elo study across 14 video models (3 reference-to-video, 7 edit, 4 continuation), PWM-WROP ranks first among continuation models and third overall, statistically tied behind two reference-to-video systems [Abstract].

Key claims

Section titled “Key claims”
  • Video-generation world models lack a systematic evaluation for object permanence and solidity as core-cognition priors, motivating a purpose-built benchmark [Abstract].
  • WROP consists of 150 hand-designed cognitive-science-inspired tasks organized into six cognitive categories, with Blender generators that randomize speed, lighting, camera angle, and other nuisance parameters while preserving each task’s cognitive structure [Abstract].
  • The released package bundles a 1.5M-sample training corpus, a 300-question exam, and PWM-WROP, a 16B model trained on WROP data [Abstract].
  • 14 video models are evaluated across three interaction shapes (3 reference-to-video, 7 edit, 4 continuation), including the authors’ PWM-WROP [Abstract].
  • In blind pairwise Elo, PWM-WROP ranks 1st among continuation models and 3rd overall, behind only a statistical tie between two reference-to-video models [Abstract].
  • The release includes data, exam, model answers, scores, model weights, and PWM — a native-PyTorch training stack running on AWS Trainium2 [Abstract].

Method

Section titled “Method”

WROP starts from a cognitive-science taxonomy of object-permanence tasks — 150 hand-designed scenarios grouped into six categories — and implements each as a Blender scene generator. Each generator randomizes surface parameters (speed, lighting, camera angle) while preserving the underlying cognitive-structural check (does the model correctly maintain / recover an occluded or solid object), producing 10,000+ samples per task. The training corpus aggregates to 1.5M samples; the exam is a 300-question held-out set. Video models are evaluated in three interaction shapes — reference-to-video (a target scene → generation), edit (modification of an input clip), and continuation (rollout from a prefix). PWM-WROP is a 16B continuation model trained on the corpus via the accompanying PWM native-PyTorch stack on AWS Trainium2. Model comparison uses blind pairwise Elo across all 14 systems on the exam.

Results

Section titled “Results”

Headline result from the blind pairwise Elo study: PWM-WROP ranks first among the four continuation models and third overall, with the top two positions occupied by a statistical tie between two reference-to-video models [Abstract]. The abstract does not disclose per-category breakdowns, exact Elo scores, or the identity of the reference-to-video ties. Absolute performance numbers, per-cognitive-category scores, and the exact 14-model roster await a full-text read.

Why it’s interesting

Section titled “Why it’s interesting”

Object permanence sits inside this wiki’s World Foundation Models and Video Generation Benchmarks clusters as a specific cognitive-science-defined probe, distinct from the physics-commonsense (PhyGenBench, VideoPhy-2, Physics-IQ), reasoning (RISE-Video), or law-grounded mechanics (Apple-π) axes already tracked. It complements Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench) by pushing the probe upstream from Newtonian laws into developmental core cognition, and complements RISE-Video: Can Video Generators Decode Implicit World Rules? by pairing benchmark with matched training corpus and a purpose-trained model rather than only a leaderboard. The fact that a 16B continuation model reaches near-parity with two reference-to-video models on this task is a data point for the “does WROP-style curated training data close the physics/cognition gap frontier video models fail on?” question, though generalization outside the six categories is untested at filing.

See also

Section titled “See also”