NeuROK: Generative 4D Neural Object Kinematics

TL;DR

NeuROK is a category-agnostic framework for turning a static 3D object into a simulative 4D asset — generating realistic temporal deformations under physical conditions (forces, actions, velocities) without any predefined physical model or per-category annotations. The method learns a data-driven kinematic state parameterization: a transformer-based encoder–decoder maps 4D shape sequences into a low-dimensional latent space, and an Euler–Lagrange equation of motion is solved within that latent space rather than in the original geometric coordinates. Trained on a large-scale 4D shape dataset spanning elastic bodies, cloth, continuum bodies, and multi-body objects, the same model handles all of them with one shared inductive bias — Lagrangian mechanics with the right learned coordinates. Relevant to Luma because it offers a route to physically-controllable 4D assets that side-steps both rigid-body simulators and per-category physics-informed networks.

Key claims

A learned kinematic state parameterization (an encoder–decoder over deformed shapes) is sufficient to express the configuration space of diverse dynamic objects without category-specific assumptions [§Abstract, Fig. 2].
Posing dynamics in this latent space lets a single Euler–Lagrange equation handle every supported object type — elastic bodies, cloth, continuum bodies, and multi-body objects — replacing the per-category physical-model assumptions that constrain prior inverse-simulation work [§Abstract].
Symbolic parameterizations (concise but inaccessible to inverse problems) and geometry-derived parameterizations (over-parameterized, requiring dense physical constraints) both fail the inverse-modeling setting; the learned latent parameterization is positioned as a third option that avoids both failure modes [Fig. 2].
The transformer-based encoder–decoder is trained on a curated large-scale 4D dataset without explicit physical annotations — the supervision is shape sequences, not forces or material parameters [§Abstract].
The decoder maps any sampled latent to a plausibly deformed shape — interpolating a 2D slice of the latent space for an eyeglass yields valid deformations frame-by-frame, demonstrating the latent space encodes a continuous manifold of physically-reasonable states [Project page].
Dropping an object then amounts to solving for a trajectory of the latent vector over time and decoding it frame-by-frame — the same recipe yields the full 3D motion regardless of object type [Project page].
Generated 4D objects can be composited into a static 3D environment (the project shows a 3D kitchen) to form an interactive 3D world responsive to user perturbation [Project page].

Method

NeuROK observes that classical Lagrangian mechanics becomes tractable in the right generalized coordinates, and treats the choice of coordinates as a learning problem. A transformer-based encoder–decoder is trained on a large 4D shape dataset to map deformation sequences into a low-dimensional latent space such that (a) every latent decodes to a plausible deformed shape and (b) the latent dynamics are simple enough for the Euler–Lagrange equation to govern them. At inference, an input static 3D shape is encoded; given physical conditions (forces, actions, initial velocity) the system integrates the latent equation of motion forward in time and decodes the latent trajectory frame-by-frame into a 4D shape sequence. Critically, the same network and the same Euler–Lagrange formulation handle elastic bodies, cloth, continuum bodies, and multi-body objects — distinctions that typically force prior work to use category-specific simulators or physics-informed losses.

Results

The paper claims effectiveness and generality across diverse dynamic object types with “clear advantages over prior works” [§Abstract]. Specific quantitative comparisons against prior simulators or learned physics models were not retrievable in the project page / abstract material available at filing time; the arXiv full PDF (CVPR 2026, pp. 39239–39251) is the authoritative source. Qualitative results in the project page include eyeglass-frame deformation interpolated on a 2D latent slice, drop dynamics with frame-by-frame decoded geometry, and insertion of generated 4D objects into a 3D kitchen to form an interactive scene.

Why it’s interesting

NeuROK is the cleanest filed datapoint for an object-centric, category-agnostic, physics-aware 4D pipeline whose inductive bias is “learn the right coordinates, then classical mechanics is easy.” This contrasts with ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion, which also outputs object-level 4D but as a topology-consistent animated mesh produced by a temporal-3D-diffusion model with no explicit physical conditioning — NeuROK takes forces/actions/velocities as input, ActionMesh takes a text/image prompt. It also contrasts with the scene-level explicit-state methods in 4D Scene Generation (VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control, NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos, SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time), which encode the world state as point clouds + per-object Gaussian trajectories and rely on a frozen video diffusion backbone for rendering; NeuROK instead encodes per-object configuration state and decodes geometry directly. Compared to the neural-rollout WFM family in World Foundation Models (PAN, Genie 3, LingBot-World), NeuROK trades pixel-space rollout for an object-state-space rollout with an explicit physics integrator. The Bitter-Lesson critique on file (The flavor of the bitter lesson for computer vision) is worth holding against this paper: NeuROK’s explicit latent-state parameterization + Euler–Lagrange integrator is exactly the hand-crafted intermediate Sitzmann argues will dissolve into end-to-end perception–action loops — but unlike most explicit-3D representations, the coordinates here are learned, so the disagreement is narrower than it looks (Sitzmann objects to fixed hand-crafted intermediates; NeuROK fixes the equation of motion but learns the coordinates the equation operates on).