MotionBricks: Scalable Real-Time Motions with Modular Latent Generative Model and Smart Primitives

TL;DR

MotionBricks (NVIDIA, TOG 2026) is a real-time generative framework for interactive character motion that combines a VQ-VAE-tokenized in-betweening backbone with a high-level “smart primitives” authoring interface — smart locomotion for navigation and smart object for object interaction — that both compile down to keyframe constraints on a single pretrained model. The backbone is a structured multi-head tokenizer (root and pose disentangled; codebook split into multiple heads along feature dim rather than by body part) plus a two-stage coarse-to-fine generator (root module → pose module) that runs at 15,000 FPS with 2 ms latency while modeling a 350k-clip corpus. All downstream applications — full UE5 game-scene demos and Unitree G1 humanoid deployment — reuse the same frozen backbone with zero fine-tuning or task-specific tagging.

Key claims

A structured multi-head tokenizer with root-pose disentanglement outperforms both single-codebook and manually body-part-partitioned tokenizers, and yields graceful degradation when individual token heads are mispredicted [§4.1, §4.2].
The two-stage backbone (root module predicts timing + initial trajectory; pose module predicts pose tokens conditioned on that trajectory) supports arbitrary combinations of missing/present in-betweening constraints via learnable mask embeddings, so a single trained model serves both sparse (1–2 keyframes, navigation) and dense (consecutive hand keyframes, object interaction) supervision at inference [§3, Fig. 3].
The system meets industry real-time budgets: 2 ms latency, 15,000 FPS throughput, while modeling 350,000+ motion clips with a single model [§Abstract, §6].
On open-source and proprietary datasets, MotionBricks reports state-of-the-art motion quality vs six in-betweening baselines [§7 experiments referenced in §Abstract].
Smart primitives (smart locomotion + smart object) compose zero-shot on a frozen backbone into a “fully connected motion graph where transitions are generated with the neural backbone” — no per-application fine-tuning or tagging required, replacing the labor-intensive animation-graph construction (Assassin’s Creed cited: 15,000 animations, 5,000 states, 12 levels deep) that currently gates high-quality runtime behavior systems to well-resourced studios [§1, §Abstract].
The same trained backbone deploys onto a Unitree G1 humanoid for real-time robotic control, framing the framework as a bridge between virtual character animation and physical robot motion synthesis [§Abstract, §6].

Method

Motions are represented per-frame as (global root pose + local root velocities + joint positions/rotations/velocities + contact labels) in global coordinates without heading canonicalization, so the model handles crawling, flipping, and non-canonical orientations. The tokenizer encodes only local pose (not root) through 1D-conv or transformer downsampling by 4×, quantizes the resulting continuous latent into discrete codes using a multi-head codebook split along the feature dimension (learned decomposition, not manual body-part split), and decodes back to continuous motion using progressive upsampling with root-trajectory skip connections and up to 10 randomly-positioned keyframe constraints injected via skip connections. Standard VQ-VAE loss with EMA codebook updates (or FSQ) trains this stage.

The generative backbone has two modules. The root module is a transformer over learnable frame-slot embeddings that first predicts in-between frame count (Step 1, from keyframe constraints), then predicts initial global root trajectory conditioned on the timing and keyframe constraints (Step 2). The pose module is the largest transformer: it consumes root trajectory + local pose keyframes + masked pose token embeddings and predicts the pose-token distribution using iterative masked-token modeling à la MaskGIT — one forward pass typically suffices at inference. At runtime, smart primitives (locomotion: velocity/heading/style → proxy keyframes; smart object: authored keyframes for approach/contact/follow-through) emit keyframe commands into this shared interface; the framework runs autoregressively with replanning triggered by control-signal changes or buffer exhaustion.

Results

15,000 FPS throughput / 2 ms latency on real-time in-betweening — orders of magnitude above the ~real-time floor that autoregressive diffusion and DDIM motion models struggle to reach [§Abstract, §6].
SOTA motion quality against six in-betweening baselines on both open-source datasets and NVIDIA’s proprietary 350k-clip corpus (specific FID/foot-slide numbers deferred to §7 tables not fully in the abstract-front page fetch) [§Abstract].
Complete production-level UE5 demo covering navigation + object-scene interaction at industry AAA quality with no foot-locking, no animation blending, no collision detection, and no hand-authored transitions — every motion generated by the neural backbone [project page uncut 2:40 demo].
Zero-shot deployment onto Unitree G1 humanoid for real-time physical robot control from the same character-animation checkpoint [§Abstract, §6].

Why it’s interesting

MotionBricks is the second text-conditioned character-motion generator filed after HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generation and takes a very different bet on the architecture → interface trade. HY-Motion 1.0 scales a Dual→Single MMDiT with flow matching and reads text prompts directly as the primary control channel (Qwen3-8B + CLIP-L conditioning), aiming at “text-to-motion at 1B parameters” as the scaling question. MotionBricks instead compresses motion into VQ tokens with iterative masked-token prediction (MaskGIT-style) and pushes text/user intent up into a smart-primitives layer above the backbone — the neural model only ever sees keyframe constraints, and control diversity is absorbed by the primitives rather than the network. The two papers therefore probe opposite sides of the same design space: HY-Motion asks whether motion admits image/video-style scaling of a monolithic conditional generator, while MotionBricks argues that industrial-quality real-time control needs a factored (backbone + authoring-layer) split to hit 15,000 FPS while retaining fine-grained authoring. The Unitree G1 deployment is the load-bearing bridge from the Human-to-Robot Retargeting cluster’s human video → robot recipes: MotionBricks flips the substrate direction — a character-animation model trained on mocap/animation data transfers to humanoid control without a paired-embodiment training stage — which is a distinct axis from ScaleBFM’s (Scaling Behavior Foundation Model for Humanoid Robots) retargeted-mocap PPO recipe or ReActor’s (ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting) physics-in-loop kinematic retargeting. It also touches Pose Estimation and Motion Capture on the generation side (like HY-Motion), extending the flow-matching-vs-token-based comparison already visible there between FMPose3D (estimation, flow matching) and HY-Motion (generation, flow matching).