MotionStream: Real-Time Video Generation with Interactive Motion Controls

TL;DR

MotionStream is a real-time, streaming, motion-conditioned video generator that hits 17 FPS at 480p / 10 FPS at 720p (29 FPS with a Tiny VAE) on a single H100 GPU with sub-second latency, while supporting infinite-length generation. The recipe: equip a Wan2.1-1.3B / Wan2.2-5B I2V teacher with lightweight sinusoidal track conditioning + joint text-motion CFG, then distill into a causal student via Self Forcing-style DMD with an attention-sink + rolling-KV-cache mechanism explicitly simulated during training. Beats ATI, Diffusion-As-Shader, Go-With-The-Flow and Image Conductor on DAVIS / Sora motion-transfer benchmarks while being two orders of magnitude faster; zero-shot beats DepthSplat / ViewCrafter / SEVA on LLFF novel-view synthesis without being designed for it.

Key claims

Sinusoidal positional embeddings with a learnable track head outperform RGB-encoded tracks fed through a frozen VAE on both efficiency and quality: PE-Head 16.29/0.452/0.456/6.54 vs RGB-VAE 16.03/0.433/0.463/8.57 on DAVIS, and ~42× faster track encoding [Table 3, §3.1].
Joint text + motion CFG (ω_txt=2.0, ω_trk=3.0) balances trajectory adherence with natural secondary dynamics; pure motion CFG yields rigid 2D-planar drags, pure text CFG drifts off trajectory [§3.1, Fig. 5].
Direct channel-wise concatenation of track embeddings with video latents (only the patchify layer’s input channels grow) replaces ControlNet-style block duplication and avoids the FLOP doubling that approach incurs [§3.1].
Stochastic mid-frame track masking during fine-tuning lets the model distinguish “occluded track” from “no track specified”, removing the appear/disappear artifact when a user releases drag controls mid-generation [§3.1].
A minimal context — 1 sink chunk + 1 rolling window chunk + chunk size 3 (c3s1w1) — beats both larger windows and Self Forcing’s c3s0w6 sliding window on long-rollout LPIPS/EPE (0.464/25.34 vs 0.480/28.09) while preserving throughput [Table 4, Fig. 6].
Adding the attention sink during training (via self-rollout with KV-cache rolling) — not just at inference — is what closes the train-test gap for indefinite-length extrapolation; this distinguishes MotionStream from TalkingMachines, which uses synchronized causal-mask denoising and therefore breaks teacher continuity at the sink/window boundary [§3.2].
Causal distillation uses DMD on the full self-rolled-out sequence with joint-CFG real score but no CFG on the fake score (critic), keeping the score difference informative [§3.2, Eq. 3].
RoPE positions for tokens in the rolling window are assigned by cache position rather than absolute temporal index, so positional encodings stay bounded regardless of generated length [§3.2].
On DAVIS motion-transfer: the distilled causal student (Wan2.1-1.3B, 480p) reaches PSNR 16.20 / EPE 7.80 at 16.7 FPS vs ATI’s PSNR 15.33 / EPE 17.41 at 0.23 FPS — comparable or better quality at ~70× the throughput [Table 1].
Zero-shot camera-control evaluation on LLFF: PSNR 15.7 / SSIM 0.38 / LPIPS 0.23 at 16.7 FPS beats DepthSplat (13.9/0.28/0.30 @ 1.4 FPS), ViewCrafter, and SEVA, even though the model was never trained on multi-view-consistent 3D supervision [Table 2].
Chunk size 1 (causal frame-by-frame) catastrophically degrades quality (LPIPS 0.597, EPE 76.21 on long Sora) vs chunk size 3 (0.464 / 25.34) — small but non-unit chunks are needed for stable AR rollouts [Table 4].

Method

The pipeline has two stages. Stage 1 (teacher): A Wan2.1-1.3B (480p) or Wan2.2-5B (720p) I2V flow-matching model is conditioned on 2D point tracks. Each track ID gets a sinusoidal embedding; visible track embeddings are placed at their spatially downsampled locations to form a track-conditioning tensor, which is processed by a lightweight track head (temporal compression + 3D conv) and then channel-wise concatenated with the video latents — no ControlNet branch. Training is two-phase: first pure flow-matching to learn track-following, then a fine-tune with stochastic mid-frame masking to resolve the occlusion/release ambiguity. At inference, joint text-motion CFG (ω_txt, ω_trk) = (2.0, 3.0) gives the best motion/dynamics trade-off.

Stage 2 (causal distillation): Initialize student from the teacher; replace bidirectional attention with a chunk-wise causal mask augmented by a fixed-size attention sink (first chunk’s KV) plus a rolling local window of the most recent w chunks. Train via Self Forcing-style DMD: the student autoregressively rolls out the full clip, then the frozen bidirectional teacher (with joint CFG) and a learnable critic produce real and fake scores; the generator is updated on the score difference. Crucially, the KV-cache rolling and sink configuration is simulated during training with positions assigned by cache index, so train-time and inference-time activation distributions are identical even past the teacher’s 81-frame horizon. The generator-to-critic update ratio is 1:5, and gradient truncation (random t ∈ [0, T-1]) keeps memory bounded.

At inference, a single chunk of latents is denoised in a small number of steps, appended to the rolling KV cache (sink chunk preserved with frozen post-RoPE activations; window chunks store pre-RoPE activations and receive positional indices by cache slot), and the next chunk begins. Throughput is constant in generated length.

Results

Motion transfer (DAVIS / Sora, vs ATI / DAS / GWTF / Image Conductor): Wan2.1-1.3B causal student reaches DAVIS PSNR 16.20 / SSIM 0.447 / LPIPS 0.443 / EPE 7.80 at 16.7 FPS; ATI is 15.33 / 0.374 / 0.473 / 17.41 at 0.23 FPS. Wan2.2-5B causal: 16.30 / 0.456 / 0.438 / 11.18 at 10.4 FPS [Table 1].
Camera control zero-shot on LLFF: Wan2.1-1.3B causal student PSNR 15.7 / SSIM 0.38 / LPIPS 0.23 at 16.7 FPS; SEVA (purpose-built) PSNR 14.1 / SSIM 0.30 / LPIPS 0.29 at 0.2 FPS. The track-conditioned model gets 3D control “for free” by projecting depth + camera through a single COLMAP-aligned scale [Table 2, §4.2].
Streaming-config ablation on Sora extended (~194 frames): c3s1w1 (1-chunk sink + 1-chunk window) wins at LPIPS 0.464 / EPE 25.34 / latency 0.70s / 16.92 FPS. Removing the sink (c3s0w1) degrades to 0.501 / 26.64. Self Forcing-style c3s0w6 sliding window gives 0.480 / 28.09 at 14.96 FPS [Table 4].
Track-representation ablation: PE-Head beats RGB-VAE on DAVIS PSNR (16.29 vs 16.03) and EPE (6.54 vs 8.57) while being ~42× faster (24.8 ms vs 1053 ms) [Table 3].
Resolutions and FPS: 17 FPS @ 480p, 10 FPS @ 720p, up to 29 FPS with a custom adversarially-trained Tiny VAE [Abstract, §4.4].
Real-world latency: sub-second time-to-first-frame; “rendered” video extends indefinitely at constant cost — the headline qualitative claim is that this turns video generation from a render-and-wait loop into a live, paint-and-watch interaction.

Why it’s interesting

MotionStream is a clean fusion of three threads the wiki has been tracking. It sits in the Autoregressive Video Generation lineage — same Self Forcing / DMD scaffold as Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion and the Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation / Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation follow-ups — but adds a load-bearing attention-sink + rolling-KV-cache mechanism that the AR-video field had been picking up piecemeal (TalkingMachines, StreamingLLM-style observations in StreamingLLM and the Discovery of Attention Sinks and The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks). The MotionStream contribution over TalkingMachines is sharp: simulate the sink at training time via self-rollout so the teacher never sees a discontinuous sink/window boundary, which the paper credits as the reason long-rollout stability holds. In contrast, Context Forcing: Consistent Autoregressive Video Generation with Long Context reaches a similar conclusion about minimal context (slow + fast memory) but does so via cache management, not via in-training rollout simulation.

On the control axis, it complements the frozen-Wan + adapter recipes in Camera-Controlled Video Diffusion (VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control, SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time, OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer) by collapsing camera control into the same 2D-track channel as object drags, then beating the dedicated novel-view-synthesis baselines zero-shot — a strong argument that trajectory (2D tracks with depth-scaled lift) is a sufficient unified control signal for camera + object motion at this scale, when paired with depth-based COLMAP rescaling at inference.

On the distillation axis, the result lines up with Diffusion Distillation‘s emerging consensus that closing the asymmetric train-test gap is more important than picking a particular objective. Where Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation fixes the frame-level injectivity gap and Streaming Autoregressive Video Generation via Diagonal Distillation fixes the next-chunk noise-level gap, MotionStream fixes the positional-extrapolation gap by simulating sink + rolling-window position assignments during training. These three are orthogonal and likely compose.