LaGSplat: Inferring Physics-Governed Interactive Simulation from Monocular Video Using Latent Lagrangian Gaussian Splatting

TL;DR

LaGSplat reconstructs a physics-governed, interactive dynamical model of a filmed object from a few seconds of monocular video, then lets a user apply unseen forces to it at inference and renders the response in real time. A single low-dimensional latent state q ∈ ℝ^d plays two roles at once: it is the generalized coordinate of a learned dissipative Euler–Lagrange equation, and the conditioning variable of a 3D Gaussian Splatting decoder whose primitives are explicit points μ_i(q). Because the Gaussians move with q, an image-space click can be pulled back through the Jacobian into a latent generalized force that enters the equations of motion — something CNN or NeRF decoders cannot support. The learned dissipative Lagrangian trades generality for a bounded, plausible response to unseen forces.

Key claims

• A shared latent state q serving both as the Lagrangian’s generalized coordinate and as the Gaussian-Splatting conditioning variable is what makes interactive force application from an image click well-defined [§Abstract].
• The explicit-point primitive μ_i(q) of Gaussian Splatting is essential: it provides a Jacobian J(q) that pulls back a pixel-space force f into a latent generalized force J(q)ᵀ f, closing the loop from user click to equations of motion; CNN and neural-field (NeRF) decoders cannot support this [§Abstract].
• A dissipative Euler–Lagrange equation over a few generalized coordinates delivers bounded, plausible responses to arbitrary unseen forces, where an unconstrained predictor diverges [§Abstract].
• The method validates from rigid to deformable and from autonomous to externally-forced real systems, using combined monocular video and sensor measurements [§Abstract].
• User-applied forces of arbitrary magnitude and direction can be injected at inference and the response rendered in real time in 2D or 3D [§Abstract].

Method

Given one or a few monocular RGB videos of an object (rigid or deformable, autonomous or forced), LaGSplat jointly learns two things sharing a latent state q ∈ ℝ^d:

1. A dissipative Lagrangian in q. The system’s dynamics obey a learned Euler–Lagrange equation with dissipation, so q(t) evolves under L(q, q̇) plus a dissipation term and any external generalized force. This is the “physics” head.
2. A q-conditioned Gaussian-Splatting decoder. Each Gaussian primitive’s center μ_i(q) is an explicit function of q. Rendering is standard 3DGS; because μ_i is a differentiable function of q, the map from latent state to image is fully known.

At inference, a user click at pixel p with force f becomes a latent generalized force J(q)ᵀ f, where J(q) = ∂μ(q)/∂q restricted to the visible primitives — this term enters the Euler–Lagrange equation directly. The trained model is used as a real-time interactive simulator: it responds to forces that were never measured, annotated, or seen during training.

Training combines monocular video reconstruction losses (through the Gaussian Splatting decoder) with sensor-measurement supervision on forced cases where available.

Results

The paper reports validation across a progression of test cases:

• Rigid autonomous → rigid forced → deformable autonomous → deformable forced real systems.
• A soft continuum robot arm demo (flagged by community discussion as the strongest datapoint) with sliders parameterized in kPa matching the real actuator’s input.
• Interactive use: arbitrary forces applied at any time, response rendered in real time in 2D or 3D.

The central positive result is qualitative but structural: the dissipative-Lagrangian inductive bias produces bounded, plausible extrapolations to unseen forces, where “unconstrained predictors diverge” [§Abstract]. No quantitative benchmark numbers are surfaced in the abstract.

Why it’s interesting

LaGSplat is the second filed paper this year to put a physics parameterization directly into the latent of a 4D generation model — the first being NeuROK: Generative 4D Neural Object Kinematics (NeurOK, CVPR 2026), which also derives a category-agnostic Euler–Lagrange dynamics in a learned latent state for elastic / cloth / continuum / multi-body objects. NeurOK operates on static 3D shape input and generates a physics-conditioned 4D response; LaGSplat operates on monocular video input and infers the physics from observation. Together they define an explicit “physics-in-the-latent” sub-family within 4D Scene Generation, distinct from the Wan-conditioned explicit-4D-state methods (VerseCrafter, NeoVerse, SpaceTimePilot) and from the mesh-output methods (ActionMesh, Shape-for-Motion). Both survive the Bitter-Lesson critique from The flavor of the bitter lesson for computer vision differently: NeurOK because the latent is fully the representation and geometry is just decoded; LaGSplat because the explicit Gaussians are load-bearing for the Jacobian pullback that closes the interaction loop.

The interactivity claim is the sharper differentiator against the rest of the World Foundation Models cluster: most world models predict rollouts under scripted conditions; LaGSplat accepts a per-frame external force from the user and answers with a physically-plausible response in real time. This is closer to a differentiable simulator than to a video generator.

See also

• NeuROK: Generative 4D Neural Object Kinematics — closest sibling on file: also learns an Euler–Lagrange dynamics in a latent state for category-agnostic physical objects, but takes static 3D shapes as input rather than monocular video.
• Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy — the closest object-centric monocular-video-to-4D method already on file (deformable-3DGS canonical mesh + SVD ControlNet for edits), but no explicit physics parameterization or interactive force application.
• ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion — the mesh-output sibling for object-level 4D generation; different output representation (topology-consistent animated mesh vs q-conditioned Gaussians) and no interactive dynamics.
• NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos — feed-forward monocular-video-to-4DGS reconstructor; comparable input regime but reconstructs appearance/geometry only, no physics head.
• The flavor of the bitter lesson for computer vision — the position essay LaGSplat’s explicit-Gaussian design bumps up against; LaGSplat’s counter is that explicit primitives are what makes the Jacobian pullback (and thus the interactive loop) well-defined.
• 4D Scene Generation — parent concept.
• World Foundation Models — LaGSplat is a distinctly interactive WFM: user-force conditioning at inference sets it apart from most generative-rollout world models.