ContentV: Efficient Training of Video Generation Models with Limited Compute

TL;DR

ContentV is an 8B text-to-video DiT built by ByteDance via minimal adaptation of Stable Diffusion 3.5 Large: swap the 2D VAE for the Wan2.1 3D-VAE, add time position embeddings, keep QK-Norm, then train with flow matching on 256×64 GB NPUs (Ascend, not GPUs) for four weeks. The headline is efficiency: the report claims VBench 85.14, beating CogVideoX-5B and HunyuanVideo-13B and matching Wan2.1-14B in human evals despite using ~33% fewer parameters than the larger open competitors and a much smaller training cluster. The other notable contribution is a memory-friendly RLHF recipe that backpropagates reward gradients through a randomly sampled subset of flow-matching sampling steps rather than every step.

Key claims

VBench score 85.14 reached after one month of training on 256×64GB NPUs, outperforming CogVideoX-5B and HunyuanVideo-13B and roughly matching Wan2.1-14B in human evals [§1, §5].
Replacing SD3.5L’s 2D-VAE with the Wan2.1 3D-VAE (4× spatial, 4× temporal compression, 16-channel latent) is sufficient to convert an image LDM into a video LDM with minimal DiT structural change [§4.2].
Direct VAE substitution destroys generation (FID jumps from 12.8 to 294.3 on SD3.5L at 512px) but the DiT recovers fast — FID 27.8 after 200 steps and 13.05 after 1,600 steps, i.e. essentially full restoration of source-model quality in <2k steps [§5.2].
Replacing SD3.5L’s APE with RoPE on video input gives no significant VBench improvement after longer training despite faster early adaptation; ContentV keeps APE [§4.3].
A three-stage progressive curriculum — low-res/short, low-res/long, high-res/long — concentrates compute in stage 2 (60k steps), which is where the model learns dynamics; stage 3 only refines resolution [§5.3, Table 2].
Flow shift of 1 at training and 17 at sampling: aggressive sampling-time shift improves visual quality, while keeping training shift low maintains general generation ability [§5.1, Fig. 5–6].
The RLHF recipe (motivated by Wu et al. 2024) enables gradients on randomly sampled timesteps rather than at equal intervals, and drops the KL constraint entirely, citing NPU memory limits for long-sequence video [§6.2].
SFT uses 1M-clip subset filtered with stricter aesthetic and motion-dynamics thresholds (top 10% on both), with LR at 10% of pretraining LR and KL regularization against the pretrained model — ~5% of pretrain compute for substantial quality gains [§6.1].
Data pipeline retains 100M images and 10M video clips (2–8s) after filtering for scene segmentation, deduplication, watermark/subtitle removal, blur, motion dynamics, and aesthetic score [§3.1].
Captioning uses Qwen 2.5 VL: 7B for pretraining data (cost reasons), 72B for the smaller post-training subset where caption precision matters more [§3.3].

Method

ContentV’s design philosophy is to do as little new modeling work as possible and instead extract video capability from a strong pretrained image base. The base is Stable Diffusion 3.5 Large (8B, 38 layers, 38 attention heads, head dim 64, FFN dim 9728, patch size 1×2×2). The VAE swap (SD3.5L’s 2D VAE → Wan2.1’s 3D causal VAE) is the only material architectural change: videos of shape $T \times H \times W$ are compressed to $T/4 \times H/8 \times W/8 \times 16$ latents, then 3D-patchified and flattened into a 1D sequence of ~116k tokens for the DiT. Time position embeddings extend SD3.5L’s absolute positional embeddings; QK-Norm (RMSNorm on Q and K pre-attention) is retained as a stability primitive because half-precision training of long sequences otherwise causes gradient explosion.

Training proceeds in four phases: a brief VAE-adaptation phase (5k steps on images to re-establish image generation in the new latent space — this is what restores FID from 294.3 to ~13), then three progressive video-pretraining stages that increase duration before resolution (40k + 60k + 30k steps), then a 5k-step SFT, then RLHF. Bucketization across seven aspect ratios and discrete duration buckets handles variable-length data; dynamic batch sizing keeps memory bounded. Joint image-video training is staged — video-only first to establish temporal coherence, then mixed — to avoid spatial bias regressing temporal quality.

The infrastructure is distinctive: 256×64 GB NPUs (Ascend), with feature extraction and DiT training decoupled across separate clusters connected by asynchronous data pipelines, layered with 3D parallelism. The RLHF stage uses VideoAlign (a VLM-based reward model) and backpropagates through differentiable sampling, choosing which steps carry gradient randomly rather than at fixed intervals; KL regularization to the reference model is dropped to fit in NPU memory.

Results

VBench total score 85.14 after 4 weeks on 256 NPUs — claimed to outperform CogVideoX-5B and HunyuanVideo-13B, with human-eval parity against Wan2.1-14B despite ContentV being 8B (40% smaller than HunyuanVideo, ~57% smaller than Wan2.1) [§1, §5]. On VAE-adaptation FID, the model recovers from 294.3 to 27.8 in 200 steps and to 13.05 (vs the 12.8 SD3.5L baseline) in 1,600 steps, i.e. effectively full image-generation parity after the swap [§5.2]. RoPE vs APE on video gives no VBench advantage after longer training despite faster initial loss curves [§4.3]. SFT yields significant prompt-following and aesthetic gains at ~5% of pretraining compute [§6.1].

Why it’s interesting

The headline ratio — VBench-SOTA in 4 weeks on 32 nodes — is what makes this matter to a team building video models. If reproducible, it sets a new floor for what “limited compute” means in T2V. The recipe is deliberately boring: take SD3.5L, swap the VAE, train. This complements Exploring Diffusion Transformer Designs via Grafting which argues an analogous point at the image-DiT level — that grafting operations onto a pretrained DiT is cheaper than retraining — and contrasts with Seedance 2.0: Advancing Video Generation for World Complexity which targets the same SOTA-T2V territory via a much larger purpose-built stack. The RLHF mechanism — random-timestep gradient sampling with KL dropped for memory — is a pragmatic variant of the differentiable-flow-RL recipe shared by Finite Difference Flow Optimization (FDFO): RL Post-Training for Flow-Based Image Generators and DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models, and worth benchmarking against them. The NPU-only training run is also one of the few openly documented frontier-scale text-to-video runs on Ascend hardware.