NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics

TL;DR

NewtonGen is a two-stage T2V framework that injects explicit Newtonian dynamics into a frozen, motion-controlled video generator (Go-with-the-Flow on top of a CogVideoX-style base). The core component, Neural Newtonian Dynamics (NND), is a second-order autonomous ODE — linear physics-informed terms plus a residual MLP — operating on a 9-dimensional latent physical state (position, velocity, rotation, dimensions, area) extracted via SAM2 masks and morphological analysis. From a parsed initial state and time stamps, NND predicts future physical states; these are projected to pixel-level optical flow and fed as structured noise to the video generator alongside a scene prompt. Across 12 motion types, NewtonGen outperforms Sora, Veo 3, CogVideoX-5B, Wan 2.2, and PhysT2V on a Physical Invariance Score, often by 0.2–0.4 absolute, with the gap widest on rotation, deformation, and damped oscillation where appearance-only models collapse.

Key claims

Current state-of-the-art T2V models (Sora, Veo 3, CogVideoX-5B, Wan 2.2) learn motion only from visual-appearance distributions, lack explicit dynamics, and routinely violate basic physical laws on out-of-distribution prompts — scaling alone does not close this gap [§1, §2.1].
A single 9-dimensional latent physical state vector (position xyz, velocity xyz, rotation, shortest/longest dimension, projected area) is expressive enough to capture translation, rotation, deformation, and 3D motion under a unified ODE framework [§3.2].
Combining linear physics-informed ODEs with a residual three-layer MLP lets one architecture model both clean linear motions (free-fall, uniform motion) and nonlinear/unknown dynamics (damped pendulum, deformation) — ablation shows the MLP is load-bearing for nonlinear and noisy cases (e.g. damped oscillation NAE drops from 0.2275 to 0.1045) [§5.3, Table 2].
NND is trained encoder-only (no decoder back to pixels) on physics-clean simulator-rendered video; a Video Physical Encoder uses SAM2 masks + morphological analysis + OpenCV to extract physical-state labels per frame [§4.2].
100 simulator-rendered videos per motion type and ~2 hours on a single A100-80GB are enough to train NND for a given dynamical class; scaling from 100 to 500 samples gives little improvement, suggesting the physics-informed inductive bias dominates over data scale [§5.3, Table 2].
Go-with-the-Flow is chosen as the video backbone because structured-noise warping of independently initialized Gaussian noise (per Chang et al.) handles deformation/rotation/complex motion better than ControlNet- or trajectory-encoder-based motion conditioners [§4.3].
At inference, the physical-state sequence is converted to an approximate pixel-level optical flow using world-setting metadata (scene dimensions, object size, closest geometric primitive) parsed from the user’s physical prompt, then spatiotemporally downsampled to the video generator’s latent resolution [§4.3].
The Physical Invariance Score (PIS) — relative standard deviation of an expected invariant (e.g. horizontal velocity in parabolic motion) over time — is the evaluation metric, with simulator-rendered reference videos providing a per-motion-type upper bound [§5.1, Eq. 7].
NewtonGen achieves the highest PIS on all 12 motion types vs. five baselines; representative gaps: Uniform Motion 0.9830 vs Veo 3’s 0.9784 vs Wan 2.2’s 0.6395; Parabolic Motion 0.9803 vs Veo 3 0.9042; Circular Motion 0.9788 vs Veo 3 0.8932; Rotation 0.8838 vs PhysT2V 0.7842; Damped Oscillation 0.5240 vs PhysT2V 0.2841 [Table 1].
The framework is limited to continuous dynamics and is less effective for multi-object collisions or coalescence, which the authors flag as motivating future work on event-based or discrete dynamics modules [§7].

Method

NewtonGen splits T2V into two decoupled stages. Stage 1 — Neural Newtonian Dynamics (NND): a 9-D latent physical-state vector $\mathbf{s} = [x, y, z, \dot x, \dot y, \dot z, \theta, l_\min, l_\max, a]$ (note: paper specifies 9 dims but lists 10 entries; the area $a$ is jointly used with the dimensions to encode shape). The dynamics are modeled as $\dot s_i = a_i s_i + b_i + \mathrm{MLP}(s)_i$ , a second-order autonomous ODE with learnable linear coefficients $\{a_i, b_i\}$ plus a residual MLP capturing nonlinear/unknown terms (Eq. 4–5). Training is encoder-only: a Video Physical Encoder (SAM2 masks + morphological centroids + OpenCV geometry) reads physical states from physics-clean simulator videos; an L2 loss between the ODE-predicted state trajectory (integrated by odeint from Chen et al. 2018) and the encoder-extracted ground-truth trajectory is minimized in latent space only (Eq. 6, Fig. 2b). Training uses 100 videos per motion type, AdamW with cosine schedule, batch size 64, 20k epochs, ~2h on one A100.

Stage 2 — Inference: the user provides a “physical prompt” specifying initial physical state + world settings + scene description. NND rolls out the physical-state sequence given the parsed initial conditions. From this sequence and the world metadata, NewtonGen computes an approximate pixel-level optical flow per frame (using the closest geometric primitive for shape), downsamples to the video latent resolution, and feeds it as a structured-noise warp to Go-with-the-Flow on top of a Wan-style or CogVideoX-style base. Scene prompts go through the standard text path. Twelve motion types are covered: uniform velocity / acceleration / deceleration / parabolic / 3D / slope sliding / circular / rotation about an axis / parabolic-with-rotation / damped oscillation / size changing / deformation (Fig. 1, §5.1).

Results

Headline (PIS, higher is better, simulator reference ≈ 0.85–1.00): NewtonGen leads on all 12 motion types vs Sora, Veo 3, CogVideoX-5B, Wan 2.2, PhysT2V [Table 1]. Selected wins: Uniform Motion 0.9830 (Veo 3: 0.9784, Wan 2.2: 0.6395); Parabolic 0.9803 (Veo 3: 0.9042); Circular 0.9788 (Veo 3: 0.8932); Rotation 0.8838 (next-best PhysT2V: 0.7842); Parabolic-with-Rotation 0.9446 (next-best PhysT2V: 0.7689); Damped Oscillation 0.5240 (next-best Sora: 0.4418); Deformation 0.5492 (next-best PhysT2V: 0.3601).
Per-baseline patterns: Veo 3 is the strongest appearance-only baseline on simple kinematic motions but degrades on rotation (0.5285), deformation (0.3466), and size-changing (0.4167). PhysT2V — which uses LLM/VLM self-refinement loops over a base T2V — is competitive on rotation but trails NewtonGen by 0.07–0.24 PIS across the harder dynamics categories. Wan 2.2 and CogVideoX-5B lag everywhere.
Ablations [Table 2]: removing the residual MLP from NND cripples nonlinear motions — Circular NAE 0.0255 → 0.5388, Parabolic-with-Rotation 0.0064 → 0.7451, Damped Oscillation 0.0425 → 0.2275. Scaling from 100 → 500 training videos barely moves NAE on most motions (e.g. Parabolic 0.0042 → 0.0040), confirming that physics-informed inductive bias dominates over training-set size in this regime.
Controllability (qualitative, Fig. 4): NewtonGen faithfully reflects parsed world settings, object properties, and initial conditions in the rendered trajectory; baselines do not vary trajectory/velocity in response to numerical changes in the initial-condition portion of the prompt.

Why it’s interesting

NewtonGen is a sharp counterpoint to the World Foundation Models thesis that physics-understanding will emerge from scale — the paper explicitly cites this debate and shows that a tiny 100-sample-per-motion-type ODE module bolted onto a frozen video generator beats Sora/Veo 3 on physical invariance by margins that no amount of further pretraining has closed. It is the structural opposite of Inference-time Physics Alignment of Video Generative Models with Latent World Models, which uses a frozen latent-predictive world model (VJEPA-2) as a reward signal over the same base T2V; NewtonGen instead uses a learned explicit-symbolic world model (neural ODE in physical-state space) as a conditioning signal via optical flow. Together they map two ends of the design space: latent-predictive reward (data-driven, weakly typed, post-hoc) vs symbolic-dynamics conditioning (physics-informed, strongly typed, in-the-loop). Methodologically it slots into Camera-Controlled Video Diffusion as another “structured control signal → frozen video generator” recipe — but the control signal is predicted by a physics module from a parsed initial condition, not extracted from a reference video or a user-drawn trajectory; this is the first filed paper where the control signal is itself the output of a trained dynamics simulator. The headline limitation — continuous single-object dynamics, no collisions — is exactly where the wiki’s other physics-evaluation papers (Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench) PhyGenBench mechanics category) report the largest model gaps, so the obvious follow-up is an event/contact-aware NND variant.