HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generation

TL;DR

HY-Motion 1.0 is the first published attempt to scale a Diffusion Transformer + flow-matching text-to-motion generator to the billion-parameter range, paired with a full Pretrain → SFT → DPO → Flow-GRPO training stack. The architecture is a hybrid dual-stream / single-stream MMDiT (a la HunyuanVideo) operating on a SMPL-H skeleton representation, conditioned by Qwen3-8B token embeddings (via a bidirectional refiner) plus a CLIP-L global vector through AdaLN, with a separate Qwen3-30B-A3B head fine-tuned to predict motion duration and rewrite the user prompt. Trained on 3000+ hours of motion data — assembled by running GVHMR on 12M HunyuanVideo clips plus 500 hours of mocap and 3D animation assets — the model reports a 3.24/5 instruction-following rating on a 2000-prompt human eval versus 2.31 for MoMask and 2.17 for LoM, and 78.6% SSAE vs the 42–58% of prior open-source baselines.

Key claims

A DiT + flow-matching text-to-motion model can be scaled to 1B+ parameters and the resulting instruction-following / quality scores beat MoMask, DART, LoM, and GoToZero by ≥0.9 points on a 1–5 human-eval scale across all 6 coarse motion categories [§5.1, Table 1].
A three-stage Pretrain (3000h) → SFT (400h) → RL training paradigm is the recipe: the SFT stage uses the same flow-matching objective with a 10× learning-rate decay to sharpen kinematic precision without erasing the broad semantic priors learned in pretraining [§4.1, §4.2].
The RL stage is split into DPO over human-labeled (winner, loser) pairs followed by Flow-GRPO on a composite reward = semantic reward (custom TMR retrieval score) + physical reward (foot-sliding and root-drift penalties) [§4.3, Eqs. 2–4].
The MMDiT uses an asymmetric cross-modal attention mask: motion tokens attend globally to text, but text tokens are masked from motion, “preventing diffusion noise inherent in the motion states from propagating back to the text embeddings” [§3.2].
A narrow-band sliding-window attention mask (≈k frames at 30 fps) is used inside the motion stream, decomposing long sequences into local-continuity substructures and giving linear complexity in sequence length [§3.2].
Both text and motion tokens are concatenated into a single sequence prior to applying full RoPE, so that positional encoding establishes “a continuous relative coordinate system” across the text↔motion boundary [§3.2].
The motion representation drops temporal derivatives (velocities) and foot-contact labels that HumanML3D uses, and reports faster training convergence with the simpler representation [§3.1].
Duration prediction and prompt rewriting live in a separate fine-tuned LLM (Qwen3-30B-A3B), trained with SFT then GRPO against a Qwen3-235B-A22B-Instruct-2507 reward judge that scores semantic consistency and temporal plausibility [§3.3].
Data acquisition fans out from a 12M-clip HunyuanVideo corpus through shot-boundary detection, human-presence filtering, and GVHMR-based 3D SMPL-X reconstruction — i.e. a video → mocap pipeline rather than direct mocap collection [§2.1].
The taxonomy spans 200+ leaf-level motion classes under 6 coarse categories (Locomotion, Sports & Athletics, Fitness & Outdoor Activities, Daily Activities, Social Interactions & Leisure, Game Character Actions), which the paper claims is the broadest coverage in the open-source motion-generation literature [§2.3, Fig. 3].

Method

HY-Motion DiT processes a fixed-length motion latent (30 fps, ≤12 s) over the SMPL-H 22-joint skeleton, where each frame is a 6D-rotation + position + global-translation vector. The network is a hybrid MMDiT: early dual-stream blocks run independent QKV/MLPs for motion and text with a joint-attention bridge, then later single-stream blocks concatenate the two and process them with shared parameters (this is the same Dual→Single layout previously filed in Waver: Wave Your Way to Lifelike Video Generation §2.1 and SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model §3.1). Text conditioning is dual-track: Qwen3-8B token embeddings flow through a Bidirectional Token Refiner (lifted from HunyuanVideo’s recipe) into the dual-stream blocks, and a global CLIP-L text vector concatenated with the timestep embedding drives a separated AdaLN.

Training runs three stages. (1) Pretraining on the full 3000-hour mixed-quality corpus with a constant LR of 1e-4 and the standard flow-matching MSE-on-velocity loss. (2) High-quality fine-tuning on the 400-hour curated subset at LR 1e-5, intended to concentrate density on optimal motion modes without collapsing semantic coverage. (3) RL alignment in two phases: DPO against human-curated (winner, loser) pairs to internalize perceptual preference, then Flow-GRPO (Liu et al. 2025) with composite TMR-similarity + physics-penalty reward to enforce explicit semantic and kinematic constraints. Independently, a Qwen3-30B-A3B is fine-tuned (SFT + GRPO with a larger Qwen3 judge) on synthetic (user-prompt, optimized-prompt, duration) triplets to handle informal, ambiguous, or bilingual user input before the diffusion stage runs.

Results

On a 2000-prompt human evaluation covering all 6 coarse categories, HY-Motion 1.0 reaches 3.24/5 average instruction-following (3.76 / 3.18 / 3.15 / 3.06 / 3.25 / 3.01 across categories) versus 2.31 for MoMask, 2.19 for GoToZero, 2.31 for DART, and 2.17 for LoM — a ≥0.9-point absolute lead across all categories [§5.1, Table 1]. Motion quality on the same protocol is 3.43 vs 2.79 / 2.86 / 3.11 / 3.06 — narrower but consistent margin [Table 2]. The automated SSAE metric (VLM-judged Structured Semantic Alignment Evaluation, from HunyuanImage) gives 78.6% vs 42.7%–58.0% for the four baselines [§5.1, Table 1].

The pretraining stage rapidly converges to “semantically correct but jittery” motions with high-frequency foot sliding; the SFT stage closes the jitter/foot-slide gap while preserving diversity; the DPO stage adds a clean Pass-Rate lift; and Flow-GRPO further tightens physical constraints — the paper claims each stage contributes a distinct, identifiable quality improvement [§4.1–§4.3].

Why it’s interesting

This is the first text-to-motion paper filed on the wiki, and it ports a stack that the wiki has now seen converge across three other domains in 2025–2026: (1) a hybrid Dual→Single-stream MMDiT layout previously reported in video (Waver: Wave Your Way to Lifelike Video Generation §2.1) and joint A+V (SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model §3.1) — Waver and SkyReels-V4 both showed Hybrid converges faster than pure Dual or pure Single, and HY-Motion is the first datapoint in 3D motion with the same recipe; (2) a Flow-GRPO + composite-reward RL stack that closely mirrors the HuDA recipe (Human detectors are surprisingly powerful reward models) of using a zero-/cheap-supervision perception-stack reward to GRPO a diffusion policy, and shares with MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling the SFT → DPO → online GRPO sequencing; (3) a video-to-mocap data pipeline (12M HunyuanVideo clips → GVHMR → SMPL-H) that uses generated-video corpora as a source of real-motion priors, complementing the motion estimation direction filed under Pose Estimation and Motion Capture (FMPose3D, MoCapAnything). On the open-question side, this is the first paper to actually scale a text-to-motion DiT to 1B and report a scaling-law-style result for the modality, where prior work (MoMask, DART, LoM) sat well below 1B; whether the motion domain shows the same clean log-linear scaling that image/video DiTs do is left unanswered by the filed evaluation.