Neodragon: Mobile Video Generation using Diffusion Transformer

TL;DR

Neodragon is a text-to-video system that runs end-to-end (text encoder + first-frame image generator + denoiser + decoder + super-resolution) directly on a Qualcomm Hexagon NPU, generating 49-frame 640×1024 24-fps clips in 6.7s with a 3.5GB peak RAM budget. It starts from Pyramidal-Flow’s spatio-temporally pyramidal causal video DiT and stacks four targeted compressions: a prompt-only Text-Encoder Distillation (4.726B FLAN-T5-XXL → 0.13B DistilT5 + 0.13B ContextAdapter), an Asymmetric Decoder Distillation that swaps the original codec-latent-VAE decoder for a mobile-friendly one without touching the encoder/latent space, MMDiT Block Pruning with two-stage recovery distillation, and Pyramidal-DMD step distillation. Final VBench Total is 81.61, competitive with Pyramidal-Flow’s 720p 81.72 score, while being the first DiT-based T2V system optimized for mobile silicon.

Key claims

A 35× compression of the FLAN-T5-XXL text encoder (4.726B → 0.13B DistilT5 + 0.13B ContextAdapter, “Replace-Mode”) loses only 0.67 VBench Total points (80.31 → 79.64) and requires no image or video supervision — distillation is done with MSE + cosine loss against the original ContextEmbedder’s tokens on 1.4M generative text prompts [§3.1, Table 2].
Cosine-distance loss is essential for stability of text-encoder distillation; disabling it (α=1, β=0) causes training divergence, while disabling MSE (α=0, β=1) is the second-best option behind the chosen α=0.5, β=0.5 [§3.1, Fig. 5(a)].
A LoRA-only adaptation of the smaller DistilT5 (no MLP ContextAdapter) fails badly — VBench 64.47 vs. 79.64 for Replace-Mode — even at high rank and large α, indicating that text-encoder substitution for video generation needs high-rank approximation, not low-rank [§3.1, Fig. 5(b), Table 2].
The native codec-latent-VAE decoder is the second mobile bottleneck despite being only 226M parameters: its conv3D-heavy graph requires 3500ms on Snapdragon X Elite and can’t fit large feature-map buffers on-device. Asymmetric Decoder Distillation replaces the decoder while keeping the original encoder and latent space — by minimally adapting an alternative decoder’s compression factor to match the encoder’s [8,8,8] and fine-tuning [§3.2].
MMDiT Block Pruning based on relative block importance reduces the denoiser from 2.009B → 1.499B (>25%) with minimal quality loss; recovery uses two stages — data-based fine-tuning then full-teacher distillation [§3.3, Abstract].
Pyramidal-DMD step distillation (DMD adapted for pyramidal flow-matching) preserves motion dynamics better than Pyramidal-MeanFlow alternatives, though with some color saturation and semantic artifacts; pairing it with an SSD-1B first-frame generator addresses first-frame degradation [§3.4].
Pyramidal-Flow’s spatio-temporally pyramidal causal attention gives an asymptotic 32× attention-compute reduction vs. a fully bidirectional baseline (a 2× temporal-pyramid saving × a 16× spatial-pyramid saving under 3-stage uniform allocation), motivating its choice as the base model over CogVideoX/LTXVideo/Wan/HunyuanVideo/OpenSORA [§2, Eqs. 1–9].
End-to-end system (DistilT5 + ContextAdapter + asymmetric-decoded VAE + pruned MMDiT + Pyramidal-DMD + SSD-1B first-frame + QuickSRNet 2× super-res) totals 4.945B parameters, 3.5GB peak RAM, 6.7s E2E latency on a Hexagon NPU, VBench Total 81.61 [Abstract, §9].

Method

The pipeline starts from Pyramidal-Flow [Yang et al. 2024] — a causal frame-by-frame video DiT that denoises each frame in a coarse-to-fine spatial pyramid, giving a combined ~32× attention-compute saving over fully bidirectional video DiTs. Neodragon layers four orthogonal compressions on top:

Text-Encoder Distillation (§3.1): replace FLAN-T5-XXL with DistilT5 (0.13B) plus a learnable 4-layer MLP ContextAdapter (0.13B) with skip connections. Train with α·MSE + β·CosineDistance (α=β=0.5) between the adapted DistilT5 tokens and the original ContextEmbedder’s MMDiT conditioning tokens, on 1.4M text-only prompts (CommonText, DiffusionDB, LAION-high-aes, T2ICompbench). Ablates four modes — Replace, Extend, LoRA, Trainable — and selects Replace.
Asymmetric Decoder Distillation (§3.2): keep the Pyramidal-Flow encoder and latent space; swap in a different model’s decoder, minimally adapted to match the [8,8,8] compression factor; fine-tune on 350K stock+Panda-70M videos.
MMDiT Block Pruning (§3.3): score block importance, prune to 18 blocks (2.009B → 1.499B), then two-stage recovery — Stage 1 data fine-tune, Stage 2 full-teacher distillation. The Stage-1→Stage-2 qualitative jump is visible in the project page videos.
Pyramidal-DMD step distillation (§3.4): standard DMD adapted to Pyramidal-Flow’s pyramidal flow-matching. Compared against Pyramidal-MeanFlow, Pyramidal-DMD wins on motion preservation at the cost of some saturation and semantic artifacts; first-frame artifacts are absorbed by the SSD-1B image generator.

The deployed system bolts on QuickSRNet for 2× super-resolution (output 640×1024). Native pre-super-res target is 320×512.

Results

	Pyramidal-Flow 480p (base)	Neodragon
VBench Total	80.31	81.61
E2E parameters	~7B (incl. T5-XXL)	4.945B
Peak RAM	—	3.5 GB
E2E latency	cloud GPU	6.7s on Hexagon NPU

Text-encoder distillation Replace-Mode: 79.64 VBench (-0.67 vs base) at 35× param reduction; on-device latency for the text encoder alone is 3ms on Snapdragon X Elite [§3.1].
The system is competitive with cloud-based offline models per Table 1’s VBench column (Pyramidal-Flow 81.56, LTXVideo 82.30, Wan 84.26, HunyuanVideo 85.09, OpenSORA 77.70, CogVideoX 81.55) while being the only entry that fits on a mobile NPU.
The Adobe Premiere Pro plugin demo (project page) runs on a Snapdragon X Elite laptop with the same Hexagon NPU as Snapdragon 8 Gen 4 mobile SoCs — entire pipeline on-device, no cloud calls, integrated into a creative tool.

Why it’s interesting

Neodragon is the first wiki entry framed around mobile-NPU video generation, and it operationalizes a stack of techniques the wiki already tracks at cloud scale. The text-encoder-distillation result connects to HunyuanImage 2.1 — Efficient 2K Text-to-Image with 32× VAE + DINOv2-Aligned REPA and MeanFlow Distillation‘s recipe of swapping heavy text encoders for lighter conditioning paths in image diffusion, and to the broader “is full T5 capacity actually needed for diffusion conditioning?” question that the field has been answering empirically. The Pyramidal-DMD step distillation is a concrete instance of the cluster on Diffusion Distillation — specifically a cascaded-trajectory variant adjacent to RMD (Cross-Resolution Distribution Matching for Diffusion Distillation), with the cascade running over the pyramid stages of Pyramidal-Flow’s denoiser rather than over arbitrary resolution segments. The choice of Pyramidal-Flow as base — for its causal frame-by-frame attention plus spatial pyramid — sits squarely inside the Autoregressive Video Generation thread; this is the first filed paper that frames AR causal attention as a mobile-deployment lever rather than just a long-horizon-rollout one. Combined with FastWan: Generating a 5-Second Video in 5 Seconds via Sparse Distillation (sparse-distillation on Wan for 5s in 5s on a single H100), Neodragon (6.7s on a phone NPU), and the cloud-side Diffusion serving optimization cluster (FastVideo, Modal-FLUX), the wiki now has three distinct points on the “what is the smallest box that can run modern T2V?” curve.