Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization

TL;DR

Quant VideoGen (QVG) is a training-free KV-cache quantization framework for autoregressive video diffusion models. It exploits the empirical observation that video KV-caches are highly spatiotemporally redundant but have highly heterogeneous numeric ranges across tokens and channels, which makes off-the-shelf LLM KV-cache quantization (KIVI, KVQuant, QuaRot) collapse. QVG combines Semantic-Aware Smoothing (k-means cluster tokens, subtract per-group centroid → quantization-friendly residuals) with Progressive Residual Quantization (coarse-to-fine multi-stage residual encoding). Up to ~7× KV memory reduction at <4% latency overhead, enabling HY-WorldPlay-8B on a single RTX 4090 (previously infeasible) and longer effective context on Self-Forcing that beats the BF16 reference at the original cache budget.

Key claims

Video KV-caches dominate GPU memory long before compute saturates — 5s 480p generation on LongCat-Video-13B requires ~75k latent tokens / ~74 GB of KV-cache vs. only ~26 GB for model parameters; this already exceeds a single RTX 5090 [§1, §3.1].
KV-cache memory is not just an efficiency bottleneck but a capability bottleneck: existing AR systems force fixed-length context windows (Self-Forcing 21 latent frames, HY-WorldPlay 20 frames) primarily because of GPU memory, and that truncation directly causes long-horizon drift in identity / scene / motion [§3.1, Fig. 2(a)].
Naive port of LLM KV-cache quantization recipes (KIVI, KVQuant, QuaRot) to video diffusion collapses because video activations are far more heterogeneous than LLM activations — empirically max(K) / mean(|K|) ratios are extreme across both tokens and channels, with channels that are outliers for some tokens but not others [§3.2, Fig. 3(a-b)].
Video KV-caches exhibit strong spatiotemporal redundancy: tokens at the same spatial location across adjacent frames are highly similar, and spatially adjacent patches map to highly similar latent tokens [§3.3, Fig. 2(b-c)].
Semantic-Aware Smoothing — k-means cluster KV tokens along the sequence axis by hidden-representation similarity, then subtract each group’s centroid — produces low-magnitude residuals that quantize cleanly: 4× Key-cache and 2× Value-cache quantization-error reduction across all precision choices [§4.1, Fig. 6].
Progressive Residual Quantization re-applies semantic-aware smoothing to the residual tensor across multiple stages, capturing information from coarse semantic structure to fine-grained variation; the first stage gives ~5× MSE reduction over naive quantization, subsequent stages add at least 2× each with diminishing returns [§5.4, Fig. 5(b-c)].
On LongCat-Video-13B at INT2, QVG-Pro reaches PSNR 30.376 / SSIM 0.935 vs. QuaRot’s 21.573 / 0.759 — the best LLM baseline trails by ~9 PSNR points and is below image-quality usability [Table 1].
On HY-WorldPlay-8B at INT2 with ~7× compression, QVG reaches PSNR 29.174 vs. QuaRot’s 25.207, while VBench Background / Image Quality / Subject Consistency / Aesthetic scores are essentially indistinguishable from the BF16 reference [Table 1].
End-to-end latency overhead is 2.1% on LongCat, 1.5% on HY-WorldPlay, 4.3% on Self-Forcing — i.e. KV-cache quantization is essentially free in the streaming-AR setting [§5.3].
On Self-Forcing, image-quality stays near-lossless out to 700 frames under QVG/QVG-Pro, while RTN/KIVI/QuaRot collapse after ~100 frames — the BF16 reference itself degrades moderately, so QVG-with-longer-cache can exceed baseline BF16 quality at the same hardware budget [§5.2, Fig. 5(a)].
HY-WorldPlay-8B becomes deployable on a single RTX 4090 for the first time at INT4/INT2 QVG, achieving PSNR 29 relative to the BF16 reference [§1].
Algorithm–system co-design matters: a centroid-caching scheme that initializes the next chunk’s k-means from the previous chunk’s assignment reduces k-means overhead by ~7×; a fused dequant+centroid-add kernel keeps intermediates in registers [§4.3].
Best operating point in their sensitivity study: block size 64 for best compression/quality trade-off, block size 16 for best quality; QVG uses M=2 stages with group size G=64, QVG-Pro uses M=4 stages with G=16 [§5.1, §5.4].

Method

QVG quantizes the KV-cache of an autoregressive video diffusion model chunk-by-chunk. For each new chunk’s KV tensor [NHW × d], it runs k-means clustering along the sequence-length axis to partition tokens into M groups by hidden-representation similarity, then subtracts each group’s centroid c_m from its members to produce a residual tensor R = X − assign(c). Because k-means groups numerically similar tokens, the residuals are low-magnitude and homogeneous (Fig. 3) — quantizing them at low bit-width incurs proportionally lower error (Eq. 4). The centroids are stored in FP8 (E4M3 per-group scale), the assignment vector in uint8 (16 centroids), and the residuals in INT2/INT4.

Progressive Residual Quantization then recursively re-applies semantic-aware smoothing to the residual tensor itself across stages: X → R_1 → R_2 → … → R_T, storing each stage’s centroids + assignments and quantizing only the final residual. This mirrors progressive video codecs (SVC) — coarse semantic structure first, fine-grained details added as residuals — and gives a smooth quality–memory knob via stage count.

Two system optimizations make it practical for streaming inference: (1) a centroid-caching trick that initializes each new chunk’s k-means using the previous chunk’s assignment, cutting k-means overhead ~7×; (2) a fused CUDA/Triton dequant+centroid-add kernel that keeps intermediate sums in registers. Pre-RoPE key caching is used for quantization-friendlier key distributions, mirroring KVQuant practice.

Results

Evaluated on LongCat-Video-13B (73-frame fixed context, generates 20-frame continuations), HY-WorldPlay-8B (12-frame chunks over full history), and Self-Forcing-Wan-1.3B (16-frame chunks), at 480p, using MovieGenBench prompts and VBench attributes plus BF16-referenced PSNR/SSIM/LPIPS.

At INT2 on LongCat-Video-13B: QVG-Pro 30.376 PSNR / 4.97× compression vs. QuaRot 21.573 / 6.40×; QVG 28.716 / 6.94× — i.e. QVG matches QuaRot’s compression ratio and beats it by ~7 PSNR points [Table 1]. At INT4: QVG-Pro 37.095 / 3.05×, QVG 37.141 / 3.72× vs. QuaRot 33.997 / 3.55×.

At INT2 on HY-WorldPlay-8B with ~7× compression, QVG reaches PSNR 29.174 / VBench Aesthetic 69.80 vs. the BF16 reference’s 69.85 — i.e. perceptual quality essentially preserved while compressing the cache 7× [Table 1].

Long-horizon Self-Forcing: image quality stays near-lossless to 700 frames; baselines collapse after ~100 frames [Fig. 5(a)]. End-to-end overhead 1.5–4.3% across the three models.

Why it’s interesting

QVG is the first filed paper to attack the KV-cache memory wall of Autoregressive Video Generation from the quantization side, complementing the training-time approaches the wiki already tracks: Context Forcing: Consistent Autoregressive Video Generation with Long Context‘s functional cache partitioning (Sink + Slow + Fast Memory), PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference‘s learned mid-buffer compression with top-k retrieval, and Pretraining Frame Preservation in Autoregressive Video Memory Compression‘s pretrained frame-preservation encoder. Where those papers select or compress the cache architecturally and require fine-tuning, QVG is training-free and orthogonal — it changes the per-token storage representation rather than the cache structure. The diagnosis the paper sharpens is also new on the wiki: LLM-KV-cache quantization recipes (KIVI/KVQuant/QuaRot, all tracked under LLM Inference Efficiency) silently fail on video models because video activations are more heterogeneous, not less, than language. This is structurally similar to the lesson the WeDLM / RaBitQ-vs-TurboQuant thread surfaced for KV-cached LLM serving — that the right primitive is workload-aware, and that benchmark numbers from the LLM literature don’t port cleanly. QVG also produces an empirically usable artifact: making HY-WorldPlay-8B run on a single RTX 4090 unlocks the same “world-model on consumer hardware” deployment regime that Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory and similar interactive world-model systems are aiming at.