Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition

TL;DR

Hunyuan-GameCraft fine-tunes HunyuanVideo (MM-DiT, text-to-video) into a high-dynamic interactive game-video generator. Keyboard/mouse inputs (WASD, arrows, Space, mouse-look) are unified into a continuous camera-trajectory space (Plücker embeddings + velocity scalars) so the model interpolates smoothly between actions. Long-horizon coherence is handled by chunkwise autoregressive generation conditioned on a “head latent” (one image, the last latent of the previous clip, or a longer clip segment) plus a binary mask that flags which tokens are clean — i.e. no causal attention mask, just chunked clean/noisy conditioning. A Phased Consistency Model + CFG distillation pass cuts inference to 8 steps for 10–20× speedup, hitting 6.6 FPS while keeping RPE-trans/rot at 0.08 / 0.20 (the best in their table). Trained on >1M 6-second clips mined from 100+ AAA games (Cyberpunk, RDR2, Assassin’s Creed, etc.) plus ~3k rendered synthetic motion sequences.

Key claims

Discrete keyboard/mouse actions can be unified into a continuous camera-trajectory representation — translation unit vector + rotation unit vector on S² + scalar speed/angular-velocity terms — supporting fine-grained interactive control and smooth interpolation between actions [§4.1, Eq. 1].
Token-addition injection of Plücker-embedding-derived action tokens into MM-DiT outperforms token concatenation and channel-wise concatenation (FVD 1554.2 vs 2236.4 / 1725.5), with the dual-branch learnable tokenizer scheme adding negligible overhead [Table 4(c,d,g)].
The action encoder is intentionally minimal — convolutional spatial downsampling + temporal pooling + a learnable scaling coefficient — replacing the cascaded-residual / transformer encoder used in prior camera-conditioning work, while reaching SOTA control on their test set [§4.1].
Hybrid history conditioning mixes three head-conditioning regimes during training — (i) single image frame latent, (ii) final latent of previous clip, (iii) longer multi-clip latent segment — at fixed ratios (12% single-clip, 25% multi-clip, 63% single-frame) [§5.1, §4.2].
Conditioning is implemented by concatenating clean head latents with noisy chunk latents at both condition and noise levels and using a binary mask (1 = head/clean, 0 = chunk/noisy) — the flow-matching schedule then denoises only the noisy positions, without any causal attention mask [§4.2].
The hybrid mix resolves a fidelity vs interactivity tradeoff observed across the three ablation variants alone: image-only conditioning gives high control accuracy but quality collapse over long sequences; clip-only conditioning gives temporal coherence but degraded control responsiveness; the hybrid recovers both [Table 4(e,f,g), Fig. 6].
A Phased Consistency Model distillation (8-step PCM) plus a Classifier-Free Guidance Distillation objective trains the student to emit guided outputs without external CFG — combined 10–20× speedup to 6.6 FPS at real-time [§4.3, Eq. 2].
Distillation costs little: PCM-distilled variant (“Ours + PCM”) retains the best RPE-trans/rot (0.08 / 0.20) and ties on Temporal Consistency while dropping FVD from 1554.2 → 1883.3 and Dynamic Average from 67.2 → 43.8 vs the undistilled model [Table 2].
Hunyuan-GameCraft beats Matrix-Game, CameraCtrl, MotionCtrl, and WanX-Cam on FVD, RPE-trans, RPE-rot, Temporal Consistency, and Dynamic Average; Dynamic Average is 67.2 vs Matrix-Game’s 31.7, with control-error reduction “by 55%” in cross-domain tests [Table 2, §5.2].
Data balancing matters: synthetic-only training has FVD 2550.7 / Dynamic 34.6 (precise control, no dynamics); live-only training is 1937.7 / 77.2 (lots of dynamics, weak control); the 1:5 render:live mix hits 1554.2 / 67.2 [Table 4(a,b,g)].
A two-pronged distribution-balancing strategy on camera trajectories — stratified sampling of start–end vectors in 3D + temporal-inversion augmentation to double backward-motion coverage — stabilizes training on a naturally forward-biased gameplay corpus [§3.3].
Despite training only on game scenes, the HunyuanVideo prior carries through: the model generates plausible interactive videos from real-world reference images with conditioned camera motion [§6].
192 NVIDIA H20 GPUs, batch 48, two-phase training (30k iters at 5e-5 + 20k iters at 1e-5 with augmentation); 720p, 25 FPS, 33-frame chunks [§5.1].

Method

Hunyuan-GameCraft is built on HunyuanVideo (MM-DiT text-to-video) and adds three components: a continuous-action encoder, a hybrid history-condition mechanism, and a PCM+CFG distillation stage.

Action representation. Keyboard/mouse inputs are mapped into a subset of the camera-parameter space — unit vectors n_t, n_r on S² for translation/rotation direction plus scalars v, ω for speed/angular velocity — eliminating the roll DoF and adding velocity control. The same representation is convertible to standard extrinsics and Plücker embeddings. A lightweight encoder (conv spatial down + pooling temporal down + learnable scaling) aligns Plücker embeddings with video latents; action tokens are then added to video tokens inside MM-DiT.

Hybrid history conditioning. Each autoregressive step denoises a chunk latent conditioned on (a) a head latent and (b) action signals. The head can be a single image latent, the final latent of the previous clip, or a longer multi-clip latent segment. Head and chunk are concatenated at both condition and noise levels; a binary mask (1 on head, 0 on chunk) tells the model which positions are clean — the flow-matching schedule denoises only the noisy positions. There is no causal attention mask: the architectural ingredient that enforces autoregressive structure is the clean/noisy concatenation pattern plus the mask indicator, not attention masking. Training mixes the three head regimes at fixed ratios (12% / 25% / 63%) so the same model serves both initial-frame generation and long-video extension.

Distillation. A Phased Consistency Model distills HunyuanVideo’s diffusion process and CFG into an 8-step consistency student. A CFG-Distillation objective additionally trains the student to produce guided outputs without external guidance computation. Combined speedup is 10–20×, taking the system to 6.6 FPS at 720p / 25 FPS chunk output.

Data. ~1M+ 6-second 1080p clips from 100+ AAA games, segmented by PySceneDetect (scene-level) and RAFT optical-flow gradients (action-level); 6-DoF camera trajectories from Monst3R; structured 30-char and 100+-char captions from a game-specific VLM. ~3k synthetic motion sequences rendered from curated 3D assets cover diverse trajectories at multiple speeds.

Results

On their 150-image / 12-action test set:

Model	FVD ↓	Dynamic Avg ↑	RPE trans ↓	RPE rot ↓	Infer FPS ↑
CameraCtrl	1580.9	7.2	0.13	0.25	1.75
MotionCtrl	1902.0	7.8	0.17	0.32	0.67
WanX-Cam	1677.6	17.8	0.16	0.36	0.13
Matrix-Game	2260.7	31.7	0.18	0.35	0.06
Ours	1554.2	67.2	0.08	0.20	0.25
Ours + PCM	1883.3	43.8	0.08	0.20	6.6

[Table 2]. The 30-evaluator user study has Hunyuan-GameCraft ahead of all four baselines on Video Quality, Temporal Consistency, Motion Smoothness, Action Accuracy, and Dynamic, with Action Accuracy at 4.61 vs Matrix-Game’s 1.63 [Table 3]. Cross-domain control-error reduction is reported at 55% vs Matrix-Game [§5.2]. Ablations confirm token addition > concat (FVD 1554.2 vs 2236.4 / 1725.5) [Table 4(c,d,g)] and the hybrid mix > image-only or clip-only conditioning [Table 4(e,f,g)]. Generalization beyond games is demonstrated qualitatively on real-world reference frames [§6].

Why it’s interesting

This is a productized version of the “interactive video model as a game engine” thesis with three design choices that distinguish it from the wiki’s existing AR-video lineage. First, there is no causal attention mask — long-horizon AR structure is enforced via clean/noisy chunk concatenation plus a binary mask indicator, in contrast to Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation‘s teacher-forced AR diffusion fine-tune and the asymmetric causal mask in FlowAct-R1: Towards Interactive Humanoid Video Generation; the hybrid head-condition mix occupies the same design-space slot as Context Forcing: Consistent Autoregressive Video Generation with Long Context‘s functional-cache partitioning and MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation‘s recurrent memory vector, but with a training-distribution fix rather than an architectural one. Second, the unified continuous keyboard/mouse → camera-trajectory space is conceptually adjacent to the “frozen Wan + camera adapter” recipe surveyed in Camera-Controlled Video Diffusion (cf. SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time, VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control) but on HunyuanVideo rather than Wan, and with discrete-key actions folded into the continuous representation rather than treated as a separate channel. Third, the PCM+CFG distillation route to 6.6 FPS is a concrete real-time data point for the Diffusion Distillation cluster that runs alongside Streaming Autoregressive Video Generation via Diagonal Distillation‘s DiagDistill (31 FPS via diagonal scheduling) and Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation‘s causal consistency distillation — three different acceleration philosophies hitting comparable real-time-or-near-real-time budgets on different base models. Filed today because it’s the open-source production stack a serious world-foundation-model effort would need to beat.