Skip to content
Bud's Wiki

World Foundation Models

TL;DR

Section titled “TL;DR”

World foundation models (WFMs) are large pretrained models marketed as priors over the physical world — generating, predicting, or reasoning about video, geometry, and dynamics at scale. The filed papers cover the two ways the wiki currently sees them: (i) as first-party releases of WFM stacks (NVIDIA Cosmos Reason / Predict / Transfer, Meta VL-JEPA via Action100M), and (ii) as off-the-shelf backbones reused inside downstream systems — Cosmos Reason 2 plugged into Isaac GR00T N1.6 as the reasoning backbone for humanoid VLA, and VJEPA-2 plugged into WMReward as a frozen physics-plausibility reward for video-generation inference-time alignment. The unifying claim is that physics/dynamics understanding lives in predictive latent representations, not in general-purpose VLMs.

Key claims

Section titled “Key claims”
  • WFMs are positioned as the backbone layer of physical-AI stacks: NVIDIA bundles Cosmos Reason 2 (reasoning VLM), Cosmos Predict 2.5 and Cosmos Transfer 2.5 (synthetic-video generators) as the open layer beneath embodied policies (NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry).
  • The same WFM can be reused as the perception/reasoning prior for a downstream VLA without retraining — Isaac GR00T N1.6 uses Cosmos Reason 2 for full-body humanoid control (NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry).
  • Latent-predictive WFMs (the V-JEPA / VL-JEPA line) outperform image–text-pretrained encoders on motion-focused tasks at far fewer training samples — VL-JEPA ViT-L at 3.3B samples beats PE-Core ViT-G at 86B samples on average action-recognition top-1 (52.5 vs 44.7) and especially on SSv2 / EPIC-KITCHENS-100 / EgoExo4D (Action100M: A Large-scale Video Action Dataset Tab. 3).
  • Latent-predictive WFMs work as off-the-shelf physics-plausibility reward models: VJEPA-2’s predictive surprise drives MAGI-1 24B from PhysicsIQ 55.22 → 62.00 on V2V (+6.78, 1st place at ICCV 2025) without any fine-tuning (Inference-time Physics Alignment of Video Generative Models with Latent World Models §3.1, Table 1).
  • VLM-based judges are not a substitute for latent-WFM signals at this task — Qwen2.5-VL / Qwen3-VL scored as reward models perform near chance on PhysicsIQ and can regress baselines (Qwen2.5-VL(BoN) on MAGI-1 I2V: −4.78 vs +6.79 for VJEPA-based WMReward) (Inference-time Physics Alignment of Video Generative Models with Latent World Models §3.1, Table 1).
  • The same predictive-latent prior that helps zero-shot action recognition (VL-JEPA on Action100M) is what makes VJEPA-2 a useful reward model — a single architectural family pays off in both directions (Action100M: A Large-scale Video Action Dataset Tab. 3; Inference-time Physics Alignment of Video Generative Models with Latent World Models §2.3).
  • WFMs are typically released open: NVIDIA on HF + GitHub + build.nvidia.com; Meta’s VL-JEPA accompanied by Action100M (~205 GB, 147M segments) on HF (NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry; Action100M: A Large-scale Video Action Dataset §4).
  • …but the flagship interactive-rollout WFMs ship closed: Google DeepMind’s Genie 3 is gated behind a Google AI Ultra subscription via a Google Labs web prototype (Project Genie), with no model weights, no architecture details, and no benchmarks disclosed (Project Genie: Experimenting with infinite, interactive worlds).
  • The interactive-rollout WFM product surface centres on three primitives — text+image World Sketching (image-grounded preview pre-commit), World Exploration (real-time path generation as the user moves, 20–24 fps / 720p), and World Remixing (gallery + prompt-remix layer) (Project Genie: Experimenting with infinite, interactive worlds).
  • Closed-flagship WFMs are specializable into vertical foundation models via post-training rather than from-scratch retraining: Waymo’s autonomous-driving simulator is built by post-training Genie 3 onto Waymo’s multi-sensor driving data, inheriting Genie’s broad world knowledge while extending the output modality from camera-only to camera + lidar (4D point clouds) — a modality the upstream model never trained on (The Waymo World Model: A New Frontier For Autonomous Driving Simulation).
  • The post-trained vertical WFM exposes the same three control primitives the upstream flagship does (driving-action / counterfactual control, scene-layout control, language control), plus a domain-specific dashcam-to-simulation mode and an “efficient variant” for long-horizon rollouts — though no quantitative results, ablations, or benchmarks are disclosed (The Waymo World Model: A New Frontier For Autonomous Driving Simulation).
  • Frontier video generators (Hailuo 2.3, Veo 3.1, Sora-2, Wan 2.6) fail the strict-reasoning check on most samples even while their per-frame Visual Quality stays >90% — RISE-Video reports a top-line strict accuracy ceiling of 22.5% across 11 representative TI2V models, with open-weight models like Wan 2.2-TI2V-5B and CogVideoX 1.5-5B dropping to 5.4% and 1.9% (RISE-Video: Can Video Generators Decode Implicit World Rules? Scoreboard). This separates “video generator as WFM” claims into a fidelity axis (largely solved) and a reasoning axis (largely open).
  • The community treats “world model” as a cross-domain umbrella spanning game simulation, autonomous driving, embodied AI, science, NLP, and agents — with the term’s definition tracing back to Ha & Schmidhuber 2018 and LeCun’s “Path Towards Autonomous Machine Intelligence” position paper (Awesome World Models Definition of World Models).
  • Foundation-layer normative position: generative-rollout world models should replace, not coexist with, explicit 3D representations (point clouds, radiance fields, SDFs, voxel grids) and SE(3) camera poses as the perception-side pre-training substrate for embodied AI; explicit 3D is a hand-crafted intermediate that the Bitter Lesson should be expected to dissolve. The same logic applies to camera-pose estimation (ego-motion is just the simplest action). 3D meshes are still reasonable at the human–machine interface layer (CAD, manufacturing output formats), but should not be baked into training architectures (The flavor of the bitter lesson for computer vision).
  • Open problem named by the same source: the chicken-and-egg of action-conditional world models — training interactive WFMs requires paired action–observation data, which is exactly the resource embodied-AI lacks (excepting self-driving). Video generative pre-training is offered as a candidate scalable objective whose latents implicitly encode physics + human-skill structure, but it is not claimed to be sufficient (The flavor of the bitter lesson for computer vision).
  • Organizational framing: at least one closed-flagship lab is publicly restructuring around the GWM thesis rather than around video generation per se — Runway announces an internal incubator (Runway Labs) explicitly chartered to prototype products on top of its GWM stack across film, healthcare, education, gaming, advertising, sports, retail, real estate and training (Introducing Runway Labs). This is a portfolio framing in contrast to DeepMind’s single-product Project Genie launch and Waymo’s single-vertical Genie-3 specialization.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Happy Oyster — Real-Time World Model for Interactive Creation: Happy Oyster (Alibaba Token Hub) — product-surface entry for a closed real-time interactive WFM with two operating modes (Directing: text/voice/image steering up to 3 min at 480p/720p with audio+video output; Wandering: first-person WASD exploration up to 1 min at 480p). No architecture, weights, or benchmarks disclosed. Distinguishing features vs Genie 3 / GWM-1 on file: baked-in joint audio-video generation and an explicit voice input channel; joins the closed-flagship real-time WFM product race alongside DeepMind’s Project Genie and Runway GWM-1.
  • [2026-07-20] Introducing Cosmos 3 Edge: Cosmos 3 Edge is the first filed on-device variant of an omnimodal WFM that keeps the flagship’s full dual-tower AR + DM architecture — a 4B open-weights model with a 2B Nemotron reasoner, targeted at Jetson (incl. new T2000/T3000), DGX Spark, and RTX PRO/GeForce RTX. As a post-trained WAM it operates at 640×360, generates 32 actions per inference at 15 Hz on Jetson Thor, and is #1 on VANTAGE-Bench among ~4B open models. Adds an edge-scale WFM-as-backbone row alongside the existing flagship (Cosmos 3 Super/Nano) and closed on-device gaps — the same understanding + generation + action interface, just resized.
  • [2026-07-20] Flow Matching in Feature Space for Stochastic World Modeling: FlowWM (Porcher/Carion/Alahari/Chen, FAIR + Inria) makes DINO-world stochastic: flow matching directly in a frozen DINOv3 encoder’s high-dimensional feature space, with a shallow-wide projection head (RAE-style) and a differentiable one-step projection that lets temporal-consistency and task-driven losses supervise the endpoint without ODE-through-time backprop. Introduces the FuturePerception Waymo benchmark that scores predicted features via linear-probe detection + depth heads rather than pixel FVD. Concrete counterpoint to VAE-latent stochastic WMs on real driving (AP(3) 21.7 vs 18.2 for WAN 2.2 VAE-WM, depth RMSE 0.078 vs 0.10), and to the deterministic DINO-world / DINO-WM line (F1 error 4.31 vs 17.1 on the synthetic Bouncing-Shapes benchmark where all modes are enumerable). First filed paper porting the RAE recipe to stochastic world modeling.
  • [2026-07-19] RynnBrain 1.1 — Open Embodied Foundation Models (Alibaba DAMO Academy): RynnBrain 1.1 is the perception/planning tier of Alibaba DAMO’s Rynn* stack that pairs with the RynnWorld-4D world-model tier; extends the “unified embodied brain” position with metric-3D grounding + contact-point prediction, and pushes the model scale to 122B-A10B sparse MoE.
  • [2026-07-18] Haiyu Wu — critique of reachability-augmented CEM planning in JEPA world models (with VISReg counter-example): Haiyu Wu (@HaiyuWu1) shares an unnamed JEPA WFM paper that adds a reachability score + K-step training rollouts to CEM planning over LeWM latents; Wu counters with a working example that swapping LeWM’s SIGReg for VISReg (arxiv 2606.02572) recovers the same or better planning success on TwoRoom and OGBench Cube — a datapoint that the reported “train-test objective misalignment” may be a latent-regularization artifact rather than a planning-objective problem, and a concrete instance of the swappable-JEPA-components design space bundled by x-jepa — Explorations into JEPA approaches (lucidrains).
  • [2026-07-17] WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time: WAM-TTT (Feng, Lyu, Wang et al.) makes the frozen WAM itself the substrate for user-preference / task-variant steering: rather than fine-tuning or adding a separate steering module, a lightweight adaptive key–value memory inside the WAM absorbs raw human-play videos via self-supervised video prediction at test time, aligned by a paired-human-robot meta-training stage with a KV memory reconstruction objective. Concrete evidence that a pretrained WAM can be steered efficiently without touching its weights — complements the “WAM-as-policy-peer” recipe of World Pilot: Steering Vision-Language-Action Models with World-Action Priors and the “frozen 3D-trace WM + small action expert” recipe of μ₀: A Scalable 3D Interaction-Trace World Model with a “frozen WAM + TTT-adapted memory” position.
  • [2026-07-17] HDR: Hierarchical Denoising for Multi-Step Visual Reasoning: HDR reports that its hierarchical video reasoner transfers to real robot maze interaction after fine-tuning on only 50 real-world robot maze videos — a data point that coarse-to-fine hierarchical video priors are compact enough to serve as world models for embodied control with minimal domain data.
  • [2026-07-16] VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders: VideoRAE closes the loop on the “frozen video foundation model as latent space” thesis (V-JEPA 2, VideoMAEv2) by showing those latents can be decoded back to pixels via a lightweight projector + representation-aligned decoder — providing a generative video-model counterpart to the latent-predictive world-model line rather than a competitor to it.
  • [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain performs 4-frame future world-state prediction in the FLUX VAE latent space via parallel flow matching, with an internal mask preserving temporal causality across generated frames.
  • [2026-07-16] FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space: FlowDAgger’s noise-space-steering adaptation recipe transfers across the VLA/WAM boundary: applied to Cosmos-Policy (a video-conditioned world-action model), the same action-inversion primitive lifts mean success 0.53 → 0.74 on 7 MetaWorld tasks — matched-magnitude to the π0.5 (VLA) result (0.53 → 0.79) on the same task set. Evidence that the interface between “generative policy” and “downstream adapter” is architecture-agnostic across action-head VLAs and video-conditioned WAMs, provided the base is trained as a flow-matching sampler.
  • [2026-07-15] Pantograph introduces Pan-1 — Minecraft model with RL-based pretraining, arguing video games are the right testbed for robotics: Pantograph launches Pan-1 (Minecraft, RL-based pretraining) and explicitly argues video games are the right robotics testbed — reproducible, safe, open-ended, long-horizon — stakes out an RL-pretraining-on-games corner adjacent to NitroGen’s BC-on-controller-overlay-video corner, and a distinct bet from Sitzmann’s video-generative-rollout framing. Product-tweet marker; no benchmarks disclosed.
  • [2026-07-14] Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model: Xiaomi-Robotics-U0 is a 38B AR multimodal WFM continually trained from EMU3.5 that jointly optimizes T2I, editing, multi-view embodied scene generation, controllable transfer, and video rollout — ranks 1st on WorldArena and lifts π₀.₅ OOD success from 36.9% → 63.2% when used as a data engine, arguing the pretraining prior is what carries embodied generalization.
  • [2026-07-14] Towards Generalizable Robotic Manipulation in Dynamic Environments (DOMINO / PUMA): PUMA (Fang et al., ECCV 2026) adds a compact-internal-WFM design point to the VLA-with-world-model recipe board: no separate video generator, no denoising loop — historical optical-flow perception (Grounded-SAM-2 masks) as input plus learnable world queries as an implicit object-centric-future-state bottleneck inside the VLA forward pass. Sibling to RynnWorld-4D’s tri-branch DiT + inverse dynamics, µ₀’s frozen-trace WM + action expert, and World Pilot’s dual-path steering, but with the prediction machinery collapsed into a fixed-size query set inside the policy; reports +6.3% absolute SR over baselines on the new DOMINO dynamic-manipulation benchmark.
  • [2026-07-13] x-jepa — Explorations into JEPA approaches (lucidrains): lucidrains ships an exploratory PyTorch codebase (x-jepa) that bundles the recent latent-predictive-WFM pole — LeWorldModel, LeJEPA, VISReg, Next-Latent Prediction Transformers, Compositional Planning with Jumpy World Models, and Kimi Attention Residuals — into a single reimplementation, treating them as one composable design space and giving the wiki a portable minimal reference for the JEPA-end-to-end-from-pixels recipe.
  • [2026-07-13] Video Generation Models are General-Purpose Vision Learners: GenCeption (Google DeepMind, ECCV 2026 — Kaiming He, Saining Xie, Zisserman, Sminchisescu et al.) is the sharpest filed empirical vindication of Sitzmann’s “generative-rollout video pretraining is the right substrate” position: a pre-trained text-to-video diffusion backbone, turned into a feed-forward text-steered vision model, matches or beats DepthAnything3, SAM3, D4RT, VGGT-Ω, Sapiens, and Lotus-2 on depth / normals / camera pose / expression-referring segmentation / 3D keypoints with 7×–500× less training data — and, critically, the paper reports that video-generation pretraining outperforms V-JEPA and VideoMAE under comparable settings. First filed head-to-head between generative-rollout and latent-predictive video pretraining on downstream perception with the generative side winning, updating the cluster’s central open question with a data point that leans toward the generation side.
  • [2026-07-10] Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio: Temporal Ratio adds an eighth WFM deployment pattern to the ledger — WFM-as-source-of-inference-time-guidance-signal — distinct from the seven already tracked (backbone, predictive-surprise reward, interruptible tool, policy-internal substrate, edit-prior via KV cache, frozen trace-WM + small action expert, value-model). Keeps the pretrained Cosmos-Predict 2.5-2B backbone in the VAM stack but reads out its internal attention statistics (Temporal Ratio = action-attention(future)/action-attention(current)) and feeds them back into the same policy’s guidance schedule at inference. Also a concrete data point on Sitzmann’s action-conditional-WFM chicken-and-egg: the video prior’s compositional generalization does survive robot-domain finetuning (autoregressive rollouts imagine plausible novel object–goal combinations) — the gap is entirely in how the action head consumes it, not in the WFM’s world knowledge.
  • [2026-07-10] LingBot-VA 2.0: Native Video-Action Pretraining for Generalizable Robot Control: LingBot-VA 2.0 sharpens the WFM-as-policy-internal-substrate pattern (established by v1 Causal World Modeling for Robot Control (LingBot-VA)) with a natively-pretrained variant: no bidirectional-to-causal retrofit, no off-the-shelf video-generator base — the ViT visual tokenizer + latent-action tokenizer + causal MoE DiT are all trained from scratch under a joint objective aligning to a frozen Perception Encoder. Concrete partial answer to Sitzmann’s action-conditional-WFM chicken-and-egg: the latent-action tokenizer bootstraps action signal from unlabeled video via an inverse-dynamics+forward-dynamics bottleneck, so the world-model half never needs paired action-observation data. Foresight Reasoning + learned-forward-dynamics re-grounding is a concrete recipe for keeping an autoregressive WFM coupled to reality under async execution — an open sub-question Next Forcing: Causal World Modeling with Multi-Chunk Prediction and v1 both raise.
  • [2026-07-09] From Foundation to Application: Improving VLA Models in Practice (LingBot-VLA 2.0): LingBot-VLA 2.0 adds a WFM-as-auxiliary-proxy datapoint to the recipe board: rather than a full generative rollout engine or a frozen trace-WFM feeding an action expert, it formulates future frame prediction as a proxy task supervised by two pretrained models — a video representation model for semantic priors and a depth estimator for geometric cues. Structurally lighter than PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation‘s physics-reinforced rollout or Causal World Modeling for Robot Control (LingBot-VA)‘s unified AR flow-matching, and closer in spirit to representation-alignment losses than to world-model-in-the-loop planning.
  • [2026-07-09] ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation: ProxyPose is a clean stress-test of the “video diffusion as universal perception backbone” claim: after fine-tuning only on synthetic (input clip, proxy-video) pairs, the same model transfers zero-shot across sensing modalities (event cameras, SPAD arrays) and object classes (rigid objects → non-rigid faces → camera pose on textureless scenes) — all downstream of a single video prior.
  • [2026-07-09] LingBot-World 2.0 / LingBot-World-Infinity — Infinite Worlds with Versatile Interactions: LingBot-World 2.0 (Robbyant / Ant Group) — second-generation open interactive WFM, headlines causal pretraining (not just distillation) as the base objective, 720p / 60 fps with sub-second latency (higher fps than Genie 3’s public spec), unbounded interaction horizon, and a pilot+director agentic harness wrapping the rollout. License drops from v1’s Apache-2.0 to CC BY-NC-SA 4.0 non-commercial; tech report URL not yet resolvable at filing. Direct sequel to Advancing Open-source World Models (LingBot-World).
  • [2026-07-08] LingBot-Video: Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence: LingBot-Video is the video-foundation-model layer of the Robbyant LingBot stack (paired with LingBot-World, LingBot-VA, LingBot-VLA) — argues WFMs for embodied AI need physical-rationality + task-completion reward signals and robot-augmented pretraining corpora, not just aesthetic optimization on internet video; claims to be the first large-scale open-source MoE video foundation model built for embodiment.
  • [2026-07-08] RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation: RynnWorld-4D (Alibaba DAMO, Apache-2.0) co-generates synchronized RGB + depth + optical flow via a tri-branch DiT with cross-modal attention and 3D RoPE, then reads its internal 4D representations with a single-forward-pass inverse-dynamics head — a WFM deployment pattern that skips multi-step denoising to run as a real-time policy.
  • [2026-07-08] Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models: VLA-MBPO adds a sixth WFM deployment pattern alongside backbone / reward / interruptible-tool / policy-internal-substrate / edit-prior: UMM-as-joint-dynamics-and-reward-model. A single finetuned Bagel emits multi-view pixel next-frames + a discrete reward token in one forward pass, matching a dedicated video world model (Ctrl-World) on prediction fidelity at ~2× the speed and matching a dedicated VLM reward model (Qwen3-VL-8B, 98.4% vs 97.0% accuracy). Concrete evidence the “world model + separate reward model” two-stack pattern is redundant when the backbone is a pretrained UMM.
  • [2026-07-06] Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots: Embodied.cpp treats VLAs and World-Action Models as a single execution class under one five-layer runtime contract (input adapters / sequence builders / backbone / head plugins / deployment adapters) and one backend abstraction across CPU/CUDA/NPU — small structural-evidence entry for the “world-model peer + action policy share deployment infrastructure” thread; preliminary WAM memory result reduces a LingBot-VA Transformer block from 312.2 MiB to 88.1 MiB.
  • [2026-07-04] RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation: RoboWorld (KAIST) uses a DROID-trained AR video world model + VLM judge as a neural simulator for RoboArena policy eval; rankings across 8 policies × 4,186 rollouts correlate with the real leaderboard at Pearson r = 0.989.
  • [2026-07-02] Markov AI gaming dataset — 500+ hours of Valorant / Minecraft / GTA gameplay screen recordings + keystrokes/mouse: Markov AI ships a 500+ hour open gaming dataset with paired keystroke/mouse action labels across three distributionally distinct genres (tactical FPS / sandbox / open-world) — a small but genre-diverse fine-tuning + evaluation substrate for action-conditioned world models over game screens, complementing NitroGen’s 40,000h gamepad-overlay corpus with native OS-level input labels.
  • [2026-07-01] AdaJEPA: An Adaptive Latent World Model: AdaJEPA (Wang, Bounou, LeCun, Ren) proposes test-time adaptation of the JEPA world model inside the MPC loop — after executing an action chunk, the observed next-state transition is used as a self-supervised update signal on the predictor, and replanning uses the freshly-adapted model. Concrete recipe for the “action-conditional WFM under distribution shift” open sub-problem: pushes the shift-correction signal into the parameters at inference (contrast: ICWM pushes it into the context, D-MPC keeps the model frozen). Reports substantial planning-success gains with as few as one gradient step per replan. Complements the LeCun-lab lineage from LeWorldModel and V-JEPA 2.1.
  • [2026-06-30] TrueLabel — Physical AI Dataset Directory: Robotics, Humanoid & Egocentric Data: TrueLabel Physical AI Dataset Directory — 658-entry third-party catalog of robotics / humanoid / egocentric / teleop / simulation datasets with license and modality metadata; sibling to datasets.bot, useful for sourcing the perception substrate WFMs are trained on.
  • [2026-06-30] EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: EgoScale (NVIDIA GEAR) treats 20,854 h of egocentric human video as a scalable embodied pretraining substrate and shows the resulting representation transfers across both task (one-shot to unseen Fold Shirt / Unscrew Bottles) and embodiment (Sharpa 22-DoF → Unitree G1 7-DoF tri-finger). The framing — humans as another embodiment whose data is treated like any other pretraining source — is the strongest filed argument that “world-foundation prior over physical interaction” can be learned from egocentric video alone without a generative video objective.
  • [2026-06-30] datasets.bot — Curated catalog of robotics and embodied-AI training datasets: datasets.bot — third-party catalog of robotics/embodied datasets tagged by modality and license; relevant when scoping WFM training corpora (egocentric video, simulator dumps, X-embodiment data) against the human-vs-robot-data substrate debate.
  • [2026-06-30] ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining: ACE-Ego-0 supplies another datapoint on egocentric human video as a complementary pretraining substrate for embodied VLAs — 1.48K hours of pseudo-action-labeled egocentric data jointly trained with 4.53K hours of robot + sim under a reliability-aware loss; reports gains in both joint pretraining and SFT regimes on RoboCasa GR1 TableTop and RoboTwin 2.0.
  • [2026-06-30] World Pilot: Steering Vision-Language-Action Models with World-Action Priors: World Pilot adds a refinement to the WFM-as-policy-peer pattern alongside µ₀ (frozen trace WM + action expert) and ImageWAM (KV-cache injection from a pretrained editor): the WAM steers the policy through two complementary paths simultaneously — a scene-evolution latent into the VLA’s perception layer (Latent Steering) and an anticipated trajectory into the action generator (Action Steering). Concrete partial answer to Sitzmann’s action-conditional-WFM chicken-and-egg: the perception-side latent transfers effectively from a video-pretrained world model that was not action-post-trained, so the dynamics prior does not require paired action–observation data. Reports 84.7% on LIBERO-Plus zero-shot OOD with largest margins on viewpoint / geometry / deformable / pose shifts.
  • [2026-06-29] Flexion Reflect v1.0: The Path Towards Long-Horizon Autonomous Humanoid Work: Flexion Reflect v1.0 opts against a generative WFM as the spatial prior — uses a precomputed semantic map from a one-time building scan plus VLM-mediated natural-language queries and global path planning over it — a structural counter-recipe to the WFM-as-substrate framings filed elsewhere on the page.
  • [2026-06-29] SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation: SimFoundry takes the modular foundation-model pipeline position against monolithic generative WFMs: rather than fine-tuning Veo or Cosmos into a robotics simulator, it composes segmentation + depth + 2D-to-3D + physics validation into a sim-ready digital twin, with the physics engine as the gate. Reports r = 0.911 sim-to-real Pearson correlation and MMRV 0.018 over 7 tasks × 5 policy architectures — the strongest filed signal that explicit physics simulators are competitive with generative WFMs as policy-evaluation substrates.
  • [2026-06-29] World Value Models for Robotic Manipulation: WVM (ByteDance Seed + PKU + Tsinghua) adds a new WFM deployment pattern — WFM-as-value-model — distinct from the prior six on this page (backbone, predictive-surprise reward, interruptible tool, policy-internal substrate, edit-prior via KV cache, frozen trace-WM + small action expert): the world model is the substrate of a learned generalist value head that produces task-progression scores for downstream policy learning across embodiments. Reports SOTA Value-Order Correlation (0.95 avg expert, 0.78 avg suboptimal) and introduces Suboptimal-Value-Bench (800 human-frame-annotated multi-embodiment trajectories) as the first filed value-model evaluation on suboptimal data; baselines split sharply on suboptimal data (VLAC collapses to -0.37 VOC vs WVM’s 0.78).
  • [2026-06-29] In-Context World Modeling for Robotic Control: ICWM (Fudan NLP) — the arxiv paper behind the HuggingPapers tweet 2070794187063099893 — In-Context World Modeling for Robotic Control (ICWM) stub, now resolvable. Candidate answer to Sitzmann’s action-conditional WFM chicken-and-egg (training interactive WFMs needs paired action–observation data, which embodied AI lacks): the robot generates its own short task-agnostic interaction prefix on the fly, and the model treats that prefix as in-context system identification rather than relying on demonstration prompts or pretraining-scale paired data. Same family as DAWN’s coupled world-action inference (The DAWN of World-Action Interactive Models) but pushed into the test-time-context regime.
  • [2026-06-29] PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation: PhysisForcing adds two physics-informed auxiliary losses (pixel-trajectory + relational semantic alignment) during finetuning of video DiTs and lifts robotics-oriented R-Bench scores by 22.3% on Wan2.2-I2V-A14B and 9.2% on Cosmos3-Nano, with WorldArena closed-loop success going 16% → 24%.
  • [2026-06-28] HuggingPapers tweet 2070794187063099893 — In-Context World Modeling for Robotic Control (ICWM): HuggingPapers tweet stub flagging “In-Context World Modeling for Robotic Control” (ICWM) — claims a world model that learns dynamics from a few seconds of self-generated interaction and zero-shot generalizes to unseen cameras / new morphologies without fine-tuning; underlying paper not linked from the tweet and not locatable at filing — kept as a re-resolve target. If the claim holds it is a candidate answer to Sitzmann’s action-conditional-WFM chicken-and-egg.
  • [2026-06-27] Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond: First filed capability-centric survey for the WFM cluster — proposes a three-level hierarchy (L1 Predictor / L2 Simulator / L3 Evolver) crossed with four governing-law regimes (physical / digital / social / scientific), with testable boundary conditions for each level (L2 = long-horizon coherence + intervention sensitivity + constraint consistency; L3 = deployment-evidence-driven stack revision, distinct from closed-loop use). Synthesizes 400+ works / 100+ systems across model-based RL, video generation, web/GUI agents, multi-agent simulation, and AI-for-science. Orthogonal to Fei-Fei Li’s renderer/simulator/planner output-channel split — together they give the cluster a 2D vocabulary it previously lacked. Argues L3 model revision requires a symbolic substrate (sharp counter to Sitzmann’s “explicit 3D should dissolve” position); names “meta-world modeling” (learnable governing laws) as the open stage beyond L3.
  • [2026-06-26] Survey notes on World Action Models / VLA for robotics (index): Luma-internal survey-notes index (Dong Guo + co-author) covering WAM / VLA directions with per-direction “what does this actually solve vs. only appear to solve” commentary — sits at the WFM↔policy boundary alongside Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models and A Functional Taxonomy of World Models.
  • [2026-06-26] Remediating Alfred Hitchcock: From Parametric to Recursive Montage: Boutet de Monvel (Bibliotheca Hertziana – MPI for Art History) reframes “world models” as a contemporary phase in a 70-year history of cinematic montage — classical cut → parametric (Whitney’s analog-computer animation for Vertigo) → recursive (Chatonsky’s Readonlymemories VI, a ComfyUI chain of LLaVA-Phi + FLUX.1 Kontext + Wan 2.6 producing an in-principle infinite Vertigo). The conclusion delivers the first filed humanities-side critique of the sim-to-real bet: gaming/entertainment uses are legitimate, but deploying models trained on mediated representations as substrates for robotics and embodied agents is a category error — “driving a car is not like racing in Forza Horizon, cleaning a house is not like playing The Sims.” Stands as a critical counterweight to Sitzmann’s “explicit 3D should dissolve into generative-rollout WFMs” position, and complements VideoPhy-2 / RISE-Video / Dream.exe (which produce quantitative versions of the same structural gap).
  • [2026-06-24] Qwen-AgentWorld: Language World Models for General Agents: Qwen-AgentWorld opens a third pole in the WFM cluster alongside pixel-rollout (Cosmos 3, Genie 3, Qwen-RobotWorld) and latent-predictive (V-JEPA 2.1, LeWorldModel): language world models that simulate the token-space dynamics of agentic environments (browsers, code interpreters, terminals) via long CoT reasoning. Trained on 10M+ real-environment trajectories across 7 domains through CPT → SFT → RL with hybrid rubric+rule rewards; 35B-A3B and 397B-A17B MoE variants. Reports gains over real-environment training when used as a decoupled simulator for agentic RL, plus warm-up benefit across 7 agentic benchmarks when used as a unified agent foundation model. Sibling to Qwen-RobotWorld (pixel-side); language-side analog to Cosmos 3’s omnimodal “world model that emits the environment response”.
  • [2026-06-24] Next Forcing: Causal World Modeling with Multi-Chunk Prediction: Next Forcing adapts LLM-style multi-token prediction to causal video WAMs: auxiliary MCP heads denoise future chunks (next¹/²/³) during training, giving 2.3× faster convergence vs. LingBot-VA at 50 fps and new SOTA on RoboTwin (94.1/93.5 Clean/Random); also reports significant gains on PhyWorld and 2× inference speedup when heads are kept for parallel next-chunk prediction.
  • [2026-06-23] ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?: ImageWAM adds a fifth WFM deployment pattern alongside WFM-as-backbone (Cosmos+GR00T), WFM-as-reward (VJEPA-2/WMReward), WFM-as-interruptible-tool (VoLo), and WFM-as-policy-internal-substrate (LingBot-VA): WFM-as-edit-prior consumed via KV cache — the world model is a pretrained image editor whose denoising KV cache feeds a flow-matching action expert, and the target frame is never actually decoded at inference, reportedly cutting FLOPs to 1/6 and latency to 1/4 of video-based WAMs.
  • [2026-06-23] Jun Gao thread (4/5) on Cosmos2.5-2B finetuned with cleaned data + kinematic skeleton conditioning: Jun Gao reports Cosmos2.5-2B finetuned with cleaned data + kinematic skeleton conditioning beats baselines on action following, appearance, and motion consistency, trained on a single GH200.
  • [2026-06-23] Causal World Modeling for Robot Control (LingBot-VA): LingBot-VA fuses the world-model and action policy into one autoregressive flow-matching network — adding a fourth WFM deployment pattern alongside WFM-as-backbone (NVIDIA Cosmos+GR00T), WFM-as-reward (VJEPA-2/WMReward), and WFM-as-interruptible-tool (VoLo): WFM-as-policy-internal-substrate, where the world model is part of the action-decoding network rather than a separate peer. Concrete answer to µ₀’s “frozen WFM + small action expert” factorization with the orthogonal “end-to-end joint training” recipe on Wan2.2-5B.
  • [2026-06-22] Inside NVIDIA Halos for Robotics: A Full-Stack Functional Safety System for Physical AI: NVIDIA Halos for Robotics uses Isaac Sim as the SIL closed-loop testing harness for safety agents — a deployment-side role for world simulators that complements their pretraining-side use in policy-evaluation work (e.g. Evaluating Gemini Robotics Policies in a Veo World Simulator)
  • [2026-06-21] HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos: HumanEgo’s encoder-shared auxiliary heads (3D object-motion + 2D trace + latent consistency) construct a lightweight in-policy world model of hand-object interaction inside a flow-matching manipulation policy, rather than as a separate WFM peer — a structurally distinct WFM-inside-VLA recipe relative to the NVIDIA GR00T+Cosmos two-model split or μ₀’s frozen-trace world model + small action expert.
  • [2026-06-19] EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning: EgoInfinity is a perception-stack composition (segmentation + reconstruction + interaction-aware refinement + retargeting under cross-module metric calibration) that consumes any-view RGB and emits agent-agnostic 4D hand-object trajectories — a concrete instance of “world-perception modules composed into a metric-calibrated pipeline” feeding the WFM-for-robotics line.
  • [2026-06-19] UMA: Unified Motion-Action Modeling for Heterogeneous Robot Learning: UMA is a non-pixel, non-action world-model interface that uses 3D object motion as the common ground across human video, real-robot, and simulation data, and toggles between policy / dynamics-predictor / adapter-target via mask pattern in a single masked generative model. Reports MSE 0.042 on motion-based dynamics modeling vs PointWorld 0.054, with the human-video and sim sources each providing measurable lift.
  • [2026-06-19] HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining: HumanScale reports egocentric human video can invert (not just match) the teleop-pretraining advantage for embodied foundation models — 24% lower validation loss and 52.5% / 90% higher in-distribution / OOD real-robot success rates at matched pretraining volume, with a “carefully designed filtering and labeling pipeline” as the lever.
  • [2026-06-19] MolmoMotion: Language-guided 3D motion forecasting: MolmoMotion is a sharp counter-datapoint to the pixel-rollout WFM frame: a Molmo-2-based 3D point-trajectory forecaster reportedly beats Wan2.2-5B and Cosmos Predict on PointMotionBench 3D ADE by 2–6× across HOT3D / WorldTrack / DAVIS, extending the “tracks beat pixels for motion” position of Forecasting Motion in the Wild to language-conditioned, object-grounded 3D forecasting with explicit robotics + video-gen downstream transfer.
  • [2026-06-19] Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation: DKT (BAAI/USC/Tsinghua et al.) extends the DVD: Deterministic Video Depth Estimation with Generative Priors “video diffusion as the prior for video depth regression” recipe to transparent and highly reflective objects — a regime where stereo/ToF/discriminative-depth all fail by construction. Same LoRA-on-Wan family, different design choices (channel-concat RGB+depth latents inside the DiT, standard flow-matching loss, 5-step fixed inference), plus a new 11k-video / 1.32M-frame synthetic dataset (TransPhy3D) filtered by Qwen2.5-VL aesthetic scoring; reported zero-shot SOTA on ClearPose / DREDS / TransPhy3D-Test depth and ClearPose normals, with grasp success jumping from 0.46–0.48 to 0.73 on reflective/translucent tabletops. Strongest filed evidence that VDM-internalized “physics” extends past rigid-body motion into refraction/reflection, and a sharp cash-out of the Sitzmann thesis in a vertical where explicit-3D pipelines are structurally blocked.
  • [2026-06-18] M*: A Modular, Extensible, Serving System for Multimodal Models: Stanford’s M* flags world-model rollout as a serving-optimization frontier. Reports up to 12.5× faster V-JEPA 2 rollout vs Meta’s native implementation on 1×H100, purely from treating the variable-horizon rollout as a first-class Loop primitive with a persistent KV cache across steps rather than recomputing the cache each iteration. No kernel or quantization changes — the speedup is structural. This is the largest single multiplier reported in the M* benchmark suite (vs ≈2.7× / ≈4× on Qwen3-Omni TTS, ≈1.3–2.6× on BAGEL), suggesting that world-model rollout is currently the most under-served loop type relative to AR text decode, and that the same continuous-batching + CUDA-graph machinery the wiki has tracked for LLM and diffusion serving has not yet been ported into the published world-model rollout recipes (Genie 3, Cosmos 3, V-JEPA 2). Implication for this page: serving-side recipes are now a missing axis in the world-model evaluation toolkit.
  • [2026-06-18] Robot Learning 2026 Main Lectures — ETH Zürich (Oier Mees): ETH Zürich Robot Learning 2026 main-lectures playlist (Oier Mees) — the course’s foundation-models-for-robotics segment is a structured walkthrough of the world-foundation-model literature the concept tracks.
  • [2026-06-18] ActiveMimic: Egocentric Video Pretraining with Active Perception: ActiveMimic adds a signal-design axis to the egocentric-pretraining substrate debate started by EgoVerse and HumanNet: treating the demonstrator’s camera motion as a first-class action (not nuisance noise) during pretraining is reported to close the long-standing gap to robot-data pretraining on real-world tasks.
  • [2026-06-15] μ₀: A Scalable 3D Interaction-Trace World Model: µ₀ (Seungjae Lee, Jusuk Lee, Jia-Bin Huang, Furong Huang et al.) — third design path between pixel-rollout WFMs and direct-action VLAs: a world model whose interface is 3D trajectories of salient interaction points (objects, tools, hands, contacts), parameterized as B-spline control points and pretrained action-free from web video via the TraceExtract auto-supervision pipeline. Frozen + paired with embodiment-specific action experts, reported competitive with π₀ on downstream manipulation despite no action pretraining. Direct successor to DynaFLIP from the same group — dynamics pushed from the encoder layer into the world-model interface layer.
  • [2026-06-15] VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation: VoLo treats a world-action model (WAM) as an interruptible tool inside a VLM orchestrator loop — a third deployment pattern alongside WFM-as-backbone (NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry) and WFM-as-reward (Inference-time Physics Alignment of Video Generative Models with Latent World Models); the orchestrator can monitor the WAM rollout and intervene before it completes.
  • [2026-06-14] LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World: LEGS uses 3D Gaussian Splatting as the photorealistic background substrate for a humanoid simulator, decoupling foreground motion from scene appearance so the same motion can re-render against new scenes/objects at ~15× lower cost than teleop re-collection — a concrete instantiation of the “simulator/world model as data engine for policies” lineage.
  • [2026-06-14] RoboArena rolls back evaluations after benchmark hacking observed since April (Pranav Atreya announcement): RoboArena’s lead author publicly retracts the post-April leaderboard ordering after benchmark hacking was detected, with NVIDIA’s Cosmos among the entries removed (per the Slack pointer) — directly puncturing the “Cosmos 3 Nano leads RoboArena” headline claim in NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI and How Cosmos 3 Helps Physical AI Think Before It Acts, and casting retroactive doubt on the Spirit-v1.6-vs-Cosmos3-Nano scoreboard reported in Spirit AI beats Nvidia on RoboArena robotics benchmark.
  • [2026-06-11] Yann LeCun: World Models — Enabling the next AI revolution (ETH Frontiers of Embodied AI): Yann LeCun ETH “Frontiers of Embodied AI” lecture (29 May 2026), “World Models: Enabling the next AI revolution” — anchors the latent-predictive JEPA pole of the cluster’s central debate. Recording filed as a stub (no transcript available at filing time); pending /bud refresh once a transcript or write-up appears.
  • [2026-06-10] AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization: AnchorWorld (Tsinghua + Kuaishou Kling) introduces anchor-view conditioning for egocentric interactive world simulation on Wan2.2 TI2V-5B: each anchor view bundles an RGB image, a 3D pose in a unified world coordinate, and a text evolution prompt, injected via in-context concat / pose-embedding addition / masked cross-attention; uses hybrid TPV+FPV training with projection-based action representation so the model learns full-body motion despite first-person view truncation; benchmarks against PlayerOne and Context-as-Memory (Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval) baselines on action control, static-scene consistency, dynamic evolution, and UE/real OOD generalization.
  • [2026-06-09] The DAWN of World-Action Interactive Models: DAWN (COWARobot + SJTU + Hohai) formalizes World-Action Interactive Models (WAIMs): future world and future action are inferred as a coupled, recursively-refined pair rather than via parallel heads or a one-way predict-then-plan pipeline. Instantiated for autonomous driving with V-JEPA 2 Large + a token-space resampler, a causal-Transformer world predictor, and a DiT action denoiser that shares weights between proposal and refinement roles. Hits perception-free PDMS 89.1 on NAVSIM v1 and the best L2 (0.33 m avg) and collision rate (0.11%) on nuScenes; ablations isolate the gain to interactive coupling rather than latent compression or rollout alone, and show 2–3 s latent rollout captures most of the benefit — concrete instance of an action-conditional WFM that ducks Sitzmann’s chicken-and-egg by leveraging AV’s paired action-observation data.
  • [2026-06-09] Robots Need More than VLA and World Models: Bou-Ammar et al. position paper arguing scaling VLA + WMs is insufficient for generalist robotics — proposes four missing “interfaces” (data autolabelling, embodiment retargeting, physics-grounded 3D world-model reasoning, and reward inference from video+language) as the actual bottleneck. Frames the WFM cluster as one of four required components rather than the central one; adds a vocabulary aid (the four-way interface decomposition) to the robotics-adjacent WFM literature.
  • [2026-06-07] Scaling Robotic Manipulation via Structured World Models and Tactile Sensing — Yunzhu Li (Montreal Robotics): Yunzhu Li (Columbia) Montreal Robotics talk arguing scaling manipulation requires both structured world models (action-conditioned predictive models with explicit physical structure) and tactile/contact sensing — vision alone is the ceiling. Adds a fourth dimension to the wiki’s WFM framing — sensing modality — orthogonal to the renderer/simulator/planner split from Fei-Fei Li’s taxonomy. Position lecture, no quantitative results at filing time; transcript pending.
  • [2026-06-06] Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners: Rhoda AI introduces Direct Video-Action Models (DVA) — a from-scratch causal video model pre-trained on web video is used as the policy at every closed-loop control step, with a separate inverse-dynamics model translating predicted futures into end-effector motion; deployments report autonomous bimanual decanting (1.5 h uncut, 11 h of robot data) and Contico container breakdown (160 min uncut, 17 h of data). Strongest filed instance of the “video model as the policy” framing — concrete instantiation of the Sitzmann thesis, and complementary to the WFM-as-evaluator framing of 1XWM and Veo-as-Gemini-Robotics-eval.
  • [2026-06-06] Learning Causal World Models: A Closed-Loop Recipe for Exploration, Representation, and Decision Making: Aether Labs position essay proposing a “causal WFM” recipe — closed-loop causality-guided exploration (probing for causal contrast), unified latent action (inverse-forward dynamics → cross-embodiment shared intervention language), and causal verifier (predictive dynamics energy used at inference to score video-generator rollouts for physical plausibility). Generalizes the Inference-time Physics Alignment of Video Generative Models with Latent World Models VJEPA-2-as-reward pattern from a single-signal reward to a structured dynamics-consistency critic, and answers one of Sitzmann’s open chicken-and-egg sub-problems with a concrete probing-as-data-collection prescription. Position essay, no benchmarks.
  • [2026-06-06] Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?: Dream.exe operationalizes WFM-as-rollout as a benchmark — generated manipulation videos are extracted into trajectories and executed in a physics simulator; on 101 tasks × 8 models, several frontier video generators achieve measurable execution success and visual quality fails to predict the executability ranking.
  • [2026-06-05] EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks: EgoTL evaluates world models on 60-second CoT-conditioned egocentric rollouts (CLIP-Score + VBench) and documents that locally photorealistic frames mask systematic long-horizon drift: object size/position change, occlusions go inconsistent, and cameras teleport across rooms — symptomatic of a missing explicit egocentric spatial backbone.
  • [2026-06-05] Kairos-HomeWorld — whole-home 3D scene generation with object-level interactivity (ACE Robotics + CUHK MMLab + Shenzhen Loop Area Institute): ACE Robotics + CUHK MMLab + Shenzhen Loop Area Institute announce Kairos-HomeWorld — “first” unified framework for whole-home 3D scene generation with object-level interactivity from a single text prompt, plus an open-source 300K Chinese residential floor plans + 5K interactive home dataset purpose-built for embodied-AI robot training. Press coverage attributes a four-stage hierarchical pipeline (floorplan → 2D-to-3D lifting → recursive refinement → manipulable-object placement). Sibling to SAGE and WorldGen but with a whole-home + localized-corpus framing; product/press marker, no paper or code at filing time.
  • [2026-06-03] VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation: VideoPhy-2 stress-tests the “video models as world simulators” thesis on 197 real-world actions: Wan2.1-14B tops at 32.6% joint SA∧PC on full / 21.9% on hard, closed models (Sora, Ray2) trail top open models, and PC scores are nearly uncorrelated with aesthetics (r=0.09) or optical-flow motion quality (r=0.002), so surface-metric optimization doesn’t help.
  • [2026-06-03] A Functional Taxonomy of World Models: Fei-Fei Li / World Labs position essay proposing a three-way functional split of “world model” grounded in the POMDP loop — renderer (action → observation pixels, optimizes visual fidelity), simulator (→ explicit state: geometry + physics + dynamics, serves humans and downstream programs), planner (observation → action). Argues simulation is the linchpin (most consequential, least mature) and that the three categories are converging into a unified world model; Marble (splats + collision meshes from one model) named as World Labs’ first move. Complements 3D as code (3D-as-durable-interface predecessor) and contrasts sharply with The flavor of the bitter lesson for computer vision (which argues explicit 3D / state should dissolve into generative-rollout WFMs, where this essay preserves state as a first-class output channel). First filed taxonomic vocabulary for the WFM cluster’s three-way split.
  • [2026-06-03] NeuROK: Generative 4D Neural Object Kinematics: Object-level WFM datapoint on the explicit-physics-state branch — learns a low-dimensional kinematic latent space per object and rolls out dynamics via the Euler–Lagrange equation in that space, decoding back to geometry frame-by-frame. Composites generated 4D objects into a static 3D kitchen for an interactive world. Sits between Genesis World (classical multi-physics engine) and the neural-rollout WFMs (PAN, Genie 3) — explicit physics integrator, but the coordinates the integrator operates on are learned from a large 4D shape dataset.
  • [2026-06-03] Ego1 — egocentric capture headset for Physical AI (General Intelligence Labs): Ego1 (General Intelligence Labs) — hardware + perception co-designed first-person headset targeting everyday manipulation as pretraining substrate for embodied/robotic models; data not yet released.
  • [2026-06-03] Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X): RBench + RoVid-X (DAGroup PKU) — first filed robotics-oriented video-generation benchmark spanning 5 task domains × 4 embodiments, validated at Spearman 0.96 with human raters. Headline: robotics-specific models (Cosmos 2.5, DreamGen, Vidar, UnifoLM-WMA-0) trail general commercial video models (Wan 2.6 leads at 0.607); Sora v2 Pro ranks 17. Paired with RoVid-X, a 4M-clip 720p robotic video dataset with optical flow + diverse forms + diverse captions, framed as the data-side response to RBench’s diagnostic failures.
  • [2026-06-03] HumanNet: Scaling Human-centric Video Learning to One Million Hours: HumanNet (PKU DAGroup / Ant) — ~1M-hour human-centric video corpus (first- and third-person) framed as embodied-pretraining substrate; validated by VLA ablation where 1,000 h egocentric video ≥ 100 h real-robot data and substantially closes the gap to 20,000 h real-robot baseline on the LingBot-VLA stack.
  • [2026-06-03] Project Chrono — Open-Source Multi-Physics Simulation Engine: Project Chrono — classical BSD-3 multi-physics simulation library (rigid + FEM + granular DVI/DAE + fluid-solid + ODE) with a ROS2 sensor suite. Anchors the analytic-physics-engine baseline that recent neural WFMs are implicitly competing with on fidelity and generality.
  • [2026-06-02] DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation: DynaFLIP argues the vision encoder layer (not the world-model backbone, not the VLA head) is where robotics dynamics should be encoded, and trains a DINOv2/SigLIP-replacement encoder via image-transition × language × 3D-flow triplets — a third “WFM-adjacent” pattern alongside world-model-as-backbone and world-model-as-on-policy-advisor.
  • [2026-06-02] Project Eden — persistent multiplayer world model that decouples world state from visual rendering (Tripo / VAST AI Research): Tripo / VAST AI Research announces Project Eden — a persistent multiplayer world model that decouples world state from rendering (research preview, demo video only, no architecture/benchmarks). Third independent 2026 datapoint for the “hoist shared state out of the network + condition learned renderer on it” pattern, alongside Odyssey’s Agora-1 (closed flagship) and MultiGen (open research). First filed multiplayer-WFM entry from a 3D-asset-focused lab.
  • [2026-06-01] LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels: LeWorldModel (FAIR + Mila, LeCun & Balestriero) — first JEPA WFM trained stably end-to-end from raw pixels with only two losses (next-embedding prediction + SIGReg isotropic-Gaussian regularizer). 15M params, single GPU, 48× faster planning than DINOv2-backed WMs, beats PLDM on PushT by +18% and matches or beats DINO-WM even without DINO-WM’s proprioceptive input. Concrete instantiation of the Sitzmann “no explicit 3D, pixel-to-latent-prediction” thesis at research-budget scale; complements V-JEPA 2.1 (end-to-end + dense-token loss) and contrasts with DINO-world (frozen DINOv2 + learned dynamics only).
  • [2026-06-01] Diffusion Model Predictive Control: D-MPC (DeepMind, NeurIPS 2024) — the cleanest filed factorized state-space WFM: separately-trained multi-step diffusion dynamics and action-proposal models with a sample-score-rank planner. Matches CQL/IQL on D4RL while enabling independent dynamics fine-tuning under hardware drift (Walker2d defective-actuator: 22.74 → 30.65 vs Diffuser 25.85 → 6.8). Counterweight to monolithic pixel-rollout WFMs (Genie 3, Cosmos Predict) and to joint state+action models (Diffuser); the inference-cost gap is closed by distilling the planner into a reactive MLP at 65.08 vs 65.98 normalized score.
  • [2026-06-01] RenderFlow: Single-Step Neural Rendering via Flow Matching: RenderFlow (Disney Research, CVPR 2026) trains one flow-matching backbone for both directions of the rendering map: forward G-buffer → photoreal RGB in a single deterministic step, and inverse RGB → intrinsics via a lightweight adapter on the same weights. Reinforces the Generative World Renderer bidirectional-rendering framing — explicit G-buffer intermediates as a useful interface — against the Sitzmann no-explicit-3D position.
  • [2026-06-01] EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World: EgoVerse — first filed consortium-style human-data ecosystem (Georgia Tech / Stanford / UCSD / ETH / MIT / Meta RL / Mecka / Scale): 1,362 h / 80k episodes / 1,965 tasks / 240 scenes / 2,087 demonstrators, split into EgoVerse-A (controlled Aria captures, six shared flagship tasks, mirrored protocols) and EgoVerse-I (industry in-the-wild). Living dataset (EgoDB) with continuous ingestion plus a phone-capture path. Cross-lab co-training study reports human-data scaling helps when aligned with the robot objective — first filed datapoint of a consortium-coordinated human-data initiative as a counterweight to single-company scaling plays like Build AI’s Egocentric-1M.
  • [2026-06-01] Nyx — GPU path-traced renderer plugin for Genesis World: Nyx renderer release — path-traced GPU renderer purpose-built for robotics evaluation with first-class 3D Gaussian splat support and the IBL+splat-under-path-tracing reconciliation problem framed as the hard one. Sharp explicit-rendering counterpoint to neural-rendering WFMs (Mirage LSD, Runway Gen-4.5) targeting the same “millions of frames that match a real camera” requirement.
  • [2026-06-01] Genesis World — Simulation platform for general-purpose robotics & embodied AI learning: Open-source release (Apache 2.0) of the Genesis World multi-physics simulator — rigid + FEM + MPM + SPH + PBD + IPC in one scene/state with three swappable couplers, multi-platform (NVIDIA + AMD + macOS + CPU). The explicit-physics-engine counterpart to neural WFMs like HY-World 1.5 and PAN; physical-robotics analog to the terminal/coding env-as-package pattern in RL Environment Platforms.
  • [2026-06-01] The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward: Genesis World 1.0 — Genesis AI’s open-source robotics simulation stack reports Pearson 0.8996 sim-to-real correlation and 45% smaller FID reality gap vs next-best simulator, explicitly arguing simulation’s first job is evaluation (not data generation) and that explicit-physics + path-traced rendering remains the trustworthy substrate for robotics iteration. The strongest filed counter-bet to the Sitzmann “explicit 3D dissolves into neural pre-training” thesis.
  • [2026-06-01] PAI-Bench: A Comprehensive Benchmark For Physical AI: PAI-Bench provides a 2,808-case Physical AI benchmark covering video generation, conditional video generation, and video understanding across AV / robotics / industry / ego-view / physical common sense. Headline result: 15 evaluated VGMs (including Veo3, Wan2.2, Cosmos-Predict, MAGI-1, HunyuanVideo-I2V, CogVideoX, LTX) match real-source visual quality (~78.0) but no model reaches the source Domain Score of 89.8, with the Robotics sub-domain the hardest (Veo3 72.1 vs source 96.6). Quantifies what world-foundation-model proponents have argued must be the next push: physics, not visuals.
  • [2026-06-01] NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI: NVIDIA launches Cosmos 3 (Super 32B + Nano 8B, OpenMDW 1.1) — a mixture-of-transformers omnimodel pairing a reasoning transformer with a generation transformer that emits text, video, audio, and numerical action (joint angles, gripper positions, trajectory points). Collapses the prior Cosmos Reason 2 + Predict 2.5 + Transfer 2.5 + GR00T-VLA stack (NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry) into a single open model. Reported #1 on Artificial Analysis open-weights, Physics-IQ, R-Bench, PAI-Bench, VANTAGE-Bench, RoboLab, RoboArena — no closed-flagship comparison disclosed.
  • [2026-06-01] Cosmos 3: Omnimodal World Models for Physical AI: NVIDIA Cosmos 3 — first filed open WFM that unifies VLM + image/video/audio generation + VLA/world-action policy under one Mixture-of-Transformers backbone (dual-tower AR + DM layers with joint attention), released under OpenMDW-1.1 in Super and Nano variants with five SDG datasets and the Cosmos-HUE benchmark; reported as best open-source T2I and I2V on Artificial Analysis and best policy on RoboArena at submission.
  • [2026-06-01] How Cosmos 3 Helps Physical AI Think Before It Acts: NVIDIA Cosmos 3 launch (GTC Taipei, June 1 2026) — a single mixture-of-transformers omnimodel that pairs a reasoning block with an expert generation block, emitting text, video, ambient sound, AND native action trajectories (joint angles, gripper positions). Collapses the prior Cosmos Reason 2 + Predict 2.5 + Transfer 2.5 three-model stack into one model, and adds the NVIDIA Cosmos Coalition (Agile Robots, Black Forest Labs, Generalist, LTX, Runway, Skild AI) as an open-WFM alignment against closed flagships like Genie 3 and Runway GWM-1. Reports leaderboard-first claims on RoboLab, RoboArena, VANTAGE-Bench, TAR, Physics-IQ, R-Bench, PAI-Bench — no numbers disclosed, but a 7-leaderboard sweep at a single open release is itself notable.
  • [2026-05-28] Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners: LILA trains a DPT decoder on top of a frozen DINOv2 encoder via linear-in-context consistency over depth/flow cue maps, producing pixel-resolution feature grids that outperform FlowFeat / LoftUp on DAVIS-VOS, NYUv2 normals, and COCO-Stuff — extending the latent self-supervised video-representation line (V-JEPA 2.1, DINO-world) to dense pixel outputs.
  • [2026-05-27] stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation: stable-worldmodel (Meta FAIR / Brown / Mila incl. LeCun & Balestriero) — an open-source platform unifying Lance-based data loading (MP4 / HDF5 / LeRobot), reference WFM baselines and planning solvers, and environments with controllable visual/geometric/physical factors of variation for systematic OOD evaluation. First filed infrastructure paper aimed at the WFM cluster — proposes to pay the “fair-comparison tax” Sitzmann named.
  • [2026-05-27] Planning at Inference: MCTS Test-Time Scaling for Long Video Generation: Demonstrates Cosmos-Predict2 as a substrate for inference-time MCTS scaling: applies modular search over continuations on top of NVIDIA’s WFM without retraining the base, generating coherent >20s rollouts.
  • [2026-05-25] WildPose: A Unified Framework for Robust Pose Estimation in the Wild: Uses a frozen MASt3R backbone as a 3D-aware perception frontend for monocular camera pose estimation, then layers a learned update operator + differentiable bundle adjustment on top — a “freeze the backbone, add explicit optimization” recipe that contrasts with VGGT-Ω’s “scale end-to-end” bet.
  • [2026-05-23] Introducing Runway Labs: Runway announces Runway Labs, an internal incubator for prototyping products on top of its GWM-1 stack, led by co-founder Alejandro Matamala Ortiz. Strategic/hiring announcement only — no technical disclosure. First filed datapoint of a closed-flagship lab publicly committing to a portfolio productization strategy for WFMs, in contrast to DeepMind’s single-product Project Genie launch and Waymo’s single-vertical Genie-3 specialization. Useful primarily as a market-signal marker.
  • [2026-05-23] The flavor of the bitter lesson for computer vision: Vincent Sitzmann (MIT, Scene Representation Group) position essay arguing that conventional computer-vision intermediate representations — 3D reconstructions (NeRF, point clouds, SDFs), SE(3) camera poses, segmentation, optical flow — are historical artifacts of scoping tractable sub-problems, and that the Bitter Lesson predicts they will dissolve into end-to-end perception–action loops with video-generative world models as the most promising current pre-training objective. First filed foundation-layer normative argument for the WFM cluster — sharp contrast with the still-3D-heavy 4D Scene Generation design pattern. Notable that the author’s own group built much of the modern 3D-representation stack (SIREN, scene representation networks, light-field networks), so the argument against 3D carries unusual weight.
  • [2026-05-23] Awesome World Models: A curated awesome-list (CC0, 1k+ GitHub stars in 30 days) cataloging world-model papers across 10 sections — definitions, surveys, game simulation (pixel & 3D-mesh), autonomous driving, embodied AI, science, positions, theory, general approaches, and evaluation. First filed navigational hub for the WFM cluster; useful as a map when surveying a sub-area, but does not synthesize. Confirms the cross-domain umbrella framing and pins canonical definitional sources (Ha & Schmidhuber 2018; LeCun 2022). The list catalogs surveys the wiki has not filed individually — including “A Comprehensive Survey on World Models for Embodied AI” (arXiv:2510.16732) and several driving-specific surveys.
  • [2026-05-23] RISE-Video: Can Video Generators Decode Implicit World Rules?: RISE-Video — a reasoning-oriented TI2V benchmark (467 human-annotated samples × 8 knowledge categories × 4 metrics) that probes whether video generators “decode implicit world rules”. 11-model leaderboard separates a relaxed weighted score (cluster 49.5–79.4) from a strict accuracy (cluster 1.9–22.5), making this the first filed benchmark-as-probe of the WFM thesis on closed and open video generators. Frontier closed models (Hailuo 2.3 / Veo 3.1 / Sora-2) lead but are tightly bunched and never exceed 22.5% strict accuracy.
  • [2026-05-23] The Waymo World Model: A New Frontier For Autonomous Driving Simulation: Waymo announces a Genie-3-derived driving simulator that emits synchronized camera + lidar (4D point clouds), exposes driving-action / scene-layout / language control, supports dashcam-to-sim conversion, and includes an efficient long-rollout variant. First filed datapoint of a closed-flagship WFM being post-trained into a vertical foundation model with a new output modality. No architecture, parameter counts, or benchmarks disclosed; pure product/research framing.
  • [2026-05-22] Project Genie: Experimenting with infinite, interactive worlds: Google DeepMind productizes Genie 3 as Project Genie — a Google Labs prototype for US Google AI Ultra subscribers (Jan 29, 2026). Real-time path-ahead generation at 20–24 fps / 720p, 60-second cap, World Sketching / Exploration / Remixing UI. Product framing only; no architecture, weights, or benchmarks disclosed. The closed-source flagship counterpart to LingBot-World in the same time window.
  • [2026-05-22] Inference-time Physics Alignment of Video Generative Models with Latent World Models: Demonstrates VJEPA-2 functioning as a frozen, off-the-shelf reward model for physics-plausibility — the world model’s predictive surprise outperforms VideoMAE pixel-reconstruction and VLM-judge rewards as a steering signal for video generation. Wins ICCV 2025 PhysicsIQ Challenge with MAGI-1 + ∇+BoN at 62.0% V2V.
  • [2026-05-22] Action100M: A Large-scale Video Action Dataset: Releases the data backing Meta’s VL-JEPA — 147M annotated segments from 1.2M HowTo100M videos. VL-JEPA outperforms CLIP-style image–text encoders on motion-centric benchmarks at far fewer samples-seen, pinning the WFM win to predictive-latent training.
  • [2026-05-22] NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry: NVIDIA’s CES 2026 omnibus release of Cosmos Reason 2 / Predict 2.5 / Transfer 2.5 (open WFM stack), Isaac GR00T N1.6 (humanoid VLA built on Cosmos Reason), and Alpamayo 1 (AV VLA). Establishes the “WFM as backbone, VLA as task-head” architectural pattern.

Open questions

Section titled “Open questions”
  • Which axis of WFM-ness actually matters for downstream use — generation quality (Cosmos Predict / Transfer for synthetic data, video models themselves) or prediction quality in latent space (VJEPA-2 as a reward, VL-JEPA as a representation)? The two filed Meta entries lean predictive; NVIDIA’s stack offers both; Waymo’s Genie-3-derived simulator leans generative-rollout; RISE-Video probes the generative side and finds it reasoning-bound at 22.5% strict accuracy. Sitzmann argues the generative-rollout side is the right pre-training objective and that latent-predictive WFMs are complementary but not sufficient — adding a normative tilt to a previously empirical question.
  • What’s the right reward-model architecture for video generation? VJEPA-2 surprise beats VLM-judge by a large margin on PhysicsIQ but is text-agnostic — semantic-adherence drops slightly under WMReward on VideoPhy T2V. A composable physics + semantics reward is the obvious next step. RISE-Video uses an LMM-as-judge for scoring, but its own absolute reliability against human raters is not validated in the released artifact.
  • Does the “WFM as backbone” pattern transfer beyond robotics/AV — e.g. can Cosmos Reason 2 or VJEPA-2 plug in as the perception/reasoning prior for generation-time systems (video editing, controllable T2V) the way they already do for policy-time systems (VLA)?
  • How tightly coupled are the released WFMs to their training corpora? Action100M is HowTo100M-derived (instructional/English), so VL-JEPA’s motion-benchmark wins may not generalize to social/sports/ego non-procedural domains. Cosmos’s training data is not detailed publicly.
  • What does the closed flagship (Genie 3 / Project Genie) actually do that the open Wan-based recipe (LingBot-World, Context Forcing, PFP, FlowAct-R1) doesn’t? Genie 3 claims “breakthrough consistency” with no numbers; LingBot-World reports specific VBench gains over Yume-1.5 / HY-World-1.5 but cannot be directly compared. RISE-Video does not include Genie 3 in its leaderboard, so the closed-flagship vs frontier-TI2V gap on reasoning is also unmeasured.
  • How is camera→lidar modality transfer actually trained? Waymo’s post-training claim — that a Genie-3-style 2D-video prior can be adapted to emit structurally different modalities (4D point clouds) — is asserted but undocumented. If it works, this opens the door to specializing closed flagships into modalities they never trained on (audio, depth, segmentation, force) by analogous post-training procedures.
  • Is the RISE-Video Reasoning-Alignment metric (LMM-judge over human-designed per-sample questions) actually a faithful operationalization of “decoded implicit world rules” — i.e., does it correlate with human strict-accuracy judgment? No filed paper measures this; if the metric is noisy, the 22.5% ceiling could be an artifact rather than a real capability boundary.
  • The Awesome-World-Models repo’s surveys section catalogs ≥10 WFM-related survey papers the wiki has not filed — including dedicated surveys on world models for embodied AI (arXiv:2510.16732), 3D/4D world modeling (arXiv:2509.07996), and “From Masks to Worlds” (arXiv:2510.20668). Should any of these be filed individually as concept-anchoring synthesis sources, or treated as second-order references? Open until a reader needs synthesis the filed papers don’t cover.
  • The chicken-and-egg of action-conditional WFMs (named by Sitzmann): training interactive world models requires paired action–observation data, which is precisely the resource embodied AI lacks except in self-driving. Video pre-training is one candidate bridge but not a proven one. Open sub-problems Sitzmann explicitly names: intrinsic motivation, exploration, long-horizon memory, continual learning, real-time control with large models.
  • Does the wiki’s 4D-generation cluster — explicit 3D / 4D Gaussian state plus a frozen diffusion backbone (NeoVerse, VerseCrafter, SpaceTimePilot) — survive the Bitter-Lesson critique? Sitzmann’s essay is the strongest articulation on file of why this whole design family may be a local optimum. No filed paper directly compares an explicit-3D-state video generator against a no-3D generative-rollout WFM on the same benchmark; until that exists, the question is open.
  • Does the portfolio-incubator strategy (Runway Labs) actually produce more durable products than the single-product (Project Genie) or single-vertical (Waymo) approaches? Filed strategy announcements are too early to tell — open until Runway Labs ships, fails to ship, or a competitor announces a comparable internal-incubator unit and the wiki can compare.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2024-10

Section titled “2024-10”

2023-12

Section titled “2023-12”

Unknown date

Section titled “Unknown date”