Envisioning the Future, One Step at a Time (Myriad)

TL;DR

Myriad casts open-set future-motion prediction as autoregressive diffusion over sparse point trajectories rather than dense pixels or latents — paying no “visual tax” for rendering appearance and unrolling motion step by step instead of leaping to a final state. A 665M-parameter transformer with parallel “fast reasoning” blocks, axial-RoPE shared across image and motion tokens, and a flow-matching head over scale-cascaded velocity increments matches or beats 1.3B–14B video models (MAGI-1, Wan2.2, CogVideo-X 1.5, SkyReels V2, SVD 1.1) on a new OWM benchmark while running at ~2,200 samples/min vs 0.05–0.7 for the video baselines. The same model is reused zero-shot for billiards shot planning by sampling many counterfactual rollouts and picking the action whose predicted trajectory best matches a goal. Authors are CompVis @ LMU Munich + Netflix; the paper is accepted at CVPR 2026 and is the sibling of Forecasting Motion in the Wild in the larger “trajectories as the right representation” research line.

Key claims

Frames motion prediction along four axes — visual tax (does the model render pixels?), granularity (dense vs sparse), domain (closed vs open-set), and paradigm (single-shot vs step-wise) — and argues prior work falls short on at least one; Myriad avoids the visual tax (no pixels), is sparse (user-defined query points), open-set (in-the-wild training), and step-wise (autoregressive over time) [§2].
Motion is modeled as an autoregressive transformer that factorizes the joint over time and over trajectories within each step, enabling KV-caching during rollout; each step is a short, locally predictable transition trained with teacher forcing [§3, Eq. 2].
Each motion token aggregates three signals: appearance at the trajectory’s origin (bilinear-sampled DINOv3-L/16 features), local context at the current position, a Fourier embedding of observed motion, and a per-trajectory identity vector drawn fresh from the unit sphere $\mathcal{S}^{d-1}$ each iteration (random near-orthogonal IDs that scale to arbitrary trajectory counts without a finite codebook) [§3, Fig. 2].
Shared spatiotemporal positional encoding via axial RoPE applied to both image and motion tokens — image tokens get $(p, p, 0)$ for $(p_\text{current}, p_\text{origin}, t)$ ; motion tokens get $(p_t, p_0, t)$ — with a reserved un-encoded channel slice for global semantic attention [§3, Fig. 3].
“Fast reasoning blocks” combine parallel transformer blocks (one residual, fused QKV+FFN-up and attn-out+FFN-down projections) with a prefix-layout self+cross attention where image tokens attend to nothing (emulating cross-attention without modifying image features over depth) and motion tokens attend causally to both streams — collapsing the per-layer kernel-launch count [§3, Eq. 3-4, Fig. 4].
Posterior is parametrized via a flow-matching head ([Lipman 2022], [Li 2024 AR-FM]) over stepwise motion $\Delta z$ rather than via a GMM; the head has separate branches for noise and conditioning so the conditioning branch can be cached across ODE sampling steps; flow matching converges faster and predicts more accurately than the GMM head used in the predecessor flow-poke-transformer [§3].
“Scale cascade”: motion has heavy-tailed statistics (excess kurtosis in the hundreds, vs ~0 for images), so the head receives a logarithmic cascade of scale coefficients $z/c_i$ — small $c_i$ preserve fine motion detail, large $c_i$ saturate and bound outlier influence — and a high-variance noise prior $\sigma_\text{noise}=2$ [§3, Fig. 6].
Introduces OWM (Open-World Motion), a 95-video static-camera benchmark with ground-truth tracker-extracted trajectories over 2.5–6.5s for diverse in-the-wild scenes, plus supplementary subsets from PhysicsIQ (solid-mechanics) and Physion [§4.1].
Evaluation protocol uses two budgets — Best-of-5 (sample 5 trajectory sets, evaluate closest to ground truth) and Best-within-5min (any number of hypotheses within a 5-min H200 wall-clock budget, also Best-of-N over those samples) — and reports a one-sided Wasserstein-like mean distance to the single observed ground-truth trajectory [§4.2, Eq. 7].
On OWM Best-of-5, Myriad (665M) achieves error 0.029 vs MAGI-1 4.5B 0.037, SkyReels V2 1.3B 0.058, SVD 1.1 1.5B 0.054, Wan2.2 14B 0.039, CogVideo-X 1.5 5B 0.051; on Best-within-5min the gap widens to 0.013 vs 0.066–0.119 (video baselines) [§5.2, Tab. 1a].
Throughput on a reference H200: 2,200 samples/min vs 0.051–0.714 for the video baselines — ~3,000–43,000× faster sampling for comparable or better accuracy [§5.2, Tab. 1].
On PhysicsIQ solid-mechanics and Physion subsets Myriad similarly leads in Best-of-5 (PhysicsIQ 0.115 vs 0.100–0.138; Physion 0.045 vs 0.061–0.070) and widens the gap in Best-within-5min, with some video baselines (Wan2.2, CogVideo-X 1.5) failing to complete the time budget (“DNF”) [§5.2, Tab. 1b-c].
Action selection by rollout exploration: in a billiard planning task (predict the shot direction+magnitude that drives a target ball to a goal), Myriad outperforms image-to-video baselines (SVD, CogVideoX, MAGI, SkyReels), full-sequence trajectory diffusion in the style of Track2Act, and the single-leap “flow poke transformer” [Baumann 2025] — sparse-trajectory step-wise unrolling dominates on the accuracy × throughput envelope [§5.3, Tab. 2].
3D-tracks variant: a model trained on V-DPM-derived 3D tracks projected back to camera view (to disentangle scene motion from camera motion) trades a small accuracy drop for static-camera-agnostic motion forecasting — 1.5M-clip subset vs 10M for the 2D model [§5.1, Tab. 1].

Method

A reference image is encoded by a DINOv3-L/16 ViT into spatial features; the user provides a sparse set of query points (visible 2D locations) optionally accompanied by a short “poke” hint of initial motion. The model unrolls future trajectories as an autoregressive transformer that emits, at each timestep $t$ and for each trajectory $k$ , an incremental motion vector $\Delta z_{t,k}$ . Motion tokens carry (i) the DINOv3 feature at the trajectory’s origin (the “what”), (ii) the feature at the current position (the “where”), (iii) a Fourier-embedded motion observation (the “how”), and (iv) a random unit-sphere identity vector (the “who”). Image and motion tokens share an axial-RoPE positional encoding scheme keyed on $(p_\text{current}, p_\text{origin}, t)$ , with a reserved un-encoded channel slice that lets global semantic attention coexist with strict positional reasoning.

Each transformer layer is a “fast reasoning block” — a parallel (single-residual) GPT-J-style block with fused QKV + FFN-up and attn-out + FFN-down projections, combined with a prefix-layout self+cross attention where image tokens attend to nothing (so their features stay constant through depth and the layer is effectively cross-attention) and motion tokens attend causally to both streams. This collapses the per-layer launch count and is the main lever that gets the model from “fast” to “3,000× faster than video diffusion.”

The posterior $p(\Delta z_{t,k} | \cdot)$ is parameterized by a flow-matching head that regresses the velocity field of a noisy-to-clean ODE; the head receives a logarithmic scale cascade of $\Delta z$ to handle motion’s heavy-tailed statistics, and has separate branches for noise and AR-conditioning so the conditioning branch is computed once per step and reused across ODE solver iterations. Training is end-to-end (image encoder + AR transformer + FM head) with teacher forcing and the standard FM loss, supervised by (pseudo-)ground-truth tracks from TAPNext (2D) or V-DPM (3D). 400k steps with batch size 128, ~20 hours on 16 H200s for the 665M model.

Results

OWM Best-of-5: Myriad 0.029 vs the best video baseline 0.037 (MAGI-1 4.5B). Under the Best-within-5min wall-clock budget the gap widens to 0.013 vs 0.066 (1.3B SkyReels V2) — three video baselines (Wan2.2, CogVideo-X 1.5, SVD 1.1) “DNF” within the budget. Physics-diagnostic subsets show the same pattern: PhysicsIQ Best-5 0.115 vs 0.100–0.138, Best-5min 0.045 vs 0.137–0.241; Physion Best-5 0.045 vs 0.061–0.070, Best-5min 0.020 vs 0.081–0.147. Throughput: ~2,200 samples/min on H200 vs 0.051–0.714 for video baselines [Tab. 1].

Billiards planning (zero-shot action selection by sampling rollouts and picking the action with the best goal-distance prediction): Myriad-style sparse-trajectory step-wise rollouts beat all image-to-video baselines, full-sequence trajectory diffusion (Track2Act-style), and the single-leap flow-poke transformer [Baumann 2025] on the accuracy × throughput frontier — directly supporting the claim that step-wise unrolling beats single-shot “leap to final state” even when the leap model is itself trajectory-based [Tab. 2].

Qualitative samples show physically-consistent motion respecting kinematic constraints (e.g., articulated motion of multiple linked points), and the project page advertises billiards-rollout videos at 3,000× the throughput of video-model baselines.

Why it’s interesting

This is the second “trajectories are the right representation for world models” paper Riddhish has flagged in two weeks — direct sibling of Forecasting Motion in the Wild (Thakkar et al., DeepMind+Berkeley), but pushing the design further on three axes: (a) sparse rather than dense tracks (user-specified query points, not a quasi-dense grid), (b) step-wise autoregressive rather than full-sequence diffusion (Forecasting Motion denoises the whole trajectory in one DiT pass; Myriad unrolls Δz step by step with KV caching), and (c) explicit benchmarking against frontier video generators (MAGI-1, Wan2.2, CogVideo-X 1.5, SkyReels V2) with a wall-clock-bounded Best-of-N protocol — making the “thousands of futures per minute” claim quantitative rather than rhetorical. The billiards-planning experiment is the right ablation to settle the “step-wise vs single-leap” debate the related-work section flags: when the same authors’ prior flow-poke-transformer ([Baumann 2025] WHN, also on the wiki via 2509.21592 references) loses to its step-wise descendant, that’s strong evidence the step-wise paradigm matters. For Luma, this lands directly in the World Foundation Models open question of “generative-rollout vs latent-predictive vs trajectory-based” representation — Myriad is the strongest data point on file for the third option being competitive on accuracy and dominant on throughput. It also dovetails with Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels (perception-side sibling: feed-forward dense 3D tracking) and contrasts with explicit-3D-state generative-rollout systems like NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos and SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time — Myriad’s bet is that you can do future-motion reasoning without ever materializing the scene’s geometry, which is the same Bitter-Lesson-style argument The flavor of the bitter lesson for computer vision makes, but for tracks rather than for generative rollout.