Skip to content
Bud's Wiki

Video Super-Resolution / Restoration via Diffusion Priors

TL;DR

Section titled “TL;DR”

Modern video super-resolution and restoration is converging on “reuse a frozen T2V foundation model (CogVideoX1.5-5B, Wan2.1, etc.) as a generative prior, with a thin task-specific adapter that injects the low-quality video as a condition.” Within that frame, the 2025–26 wiki cluster pulls four orthogonal levers: (1) HR-keyframe conditioning (SparkVSR: feed a sparse set of off-the-shelf-ISR’d keyframes into the DiT alongside the LR video, exposed as a CFG-style Reference-Free Guidance knob); (2) one-step distillation procedure (DUO-VSR: dual-stream DMD + RFS-GAN with score models reused as discriminator backbones + DPO refinement, fixing DMD’s “frozen real-score is a degraded supervisor” failure mode); (3) kernel + decoder systems co-design (FlashVSR: three-stage distillation + Locality-Constrained Sparse Attention + Tiny Conditional Decoder for ~17 FPS at 768×1408 on a single A100); (4) training-data alignment via T2V-synthesized pairs (Vivid-VR: use the frozen T2V to denoise a caption-perturbed real video and treat the output as a perfectly aligned training sample, killing the distribution drift that VLM captioner imperfection induces during fine-tuning). The four are largely composable in principle. A fifth, image-only lever has now arrived: collapse the decode + upsample stages of an LDM pipeline into one pixel-diffusion model (PiD), eliminating the cascaded VSR step entirely for image LDM outputs.

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-05-25] PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion (PiD, NVIDIA): the image-side challenger. PiD is a pixel-diffusion decoder that decodes latents and upsamples in one model, replacing the standard LDM “VAE decoder → external super-resolution pipeline” cascade. A lightweight sigma-aware adapter injects noise-corrupted latents (so it can decode partially-denoised latents from the LDM, letting the LDM exit early), and DMD2 4-step distillation gets the decode+4× upscale down to ~210 ms on GB200 and under 1 s at 13 GB peak on a consumer RTX 5090. Project-page Quantitative Results report 5.9× faster than SeedVR2 (211.2 ms vs 1237.5 ms) at the same 512²→2048² task with better visual fidelity in side-by-side judging. Works on both VAE and SigLIP/DINOv2 (RAE-style) latents. This is the image-only frontier; PiD does not address VSR’s temporal axis (no Wan/CogVideoX adaptation in the paper). For image upscaling-from-latent the cluster’s “LDM → external VSR” framing is contested.
  • [2026-05-25] RTX Video SDK — RTX Video Super Resolution and RTX Video HDR (NVIDIA RTX Video SDK product page): productized non-diffusion AI video pipeline shipping two effects — RTX Video Super Resolution (upscale + sharpen + compression-artifact removal) and RTX Video HDR (SDR→HDR10 tone mapping). Tensor-Core-optimized for real-time playback or fast offline processing on consumer RTX GPUs. No architecture, training, or benchmark detail published. Internal test: 720p→4K Ray3 upscale in ~5s. Sits at the opposite end of the latency/quality axis from the diffusion VSR cluster and is the consumer-deployed baseline they have to clear.
  • [2026-05-25] Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration (Vivid-VR, ICLR 2026): ControlNet-on-CogVideoX1.5-5B with concept distillation (T2V synthesizes aligned training data by denoising caption-perturbed real videos) + lightweight LQ artifact projector + dual-branch (MLP + cross-attention) ControlNet connector. Beats SeedVR-7B, SeedVR2-7B, STAR, DOVE on no-reference perceptual metrics across 6 testsets (incl. new UGC50 and AIGC50). Quality-first, multi-step inference; one-step distillation flagged as future work. Memory: ~43GB for 121-frame clip on a single GPU. Inherits SparkVSR’s CogVideoX1.5 base but doesn’t use HR keyframes.
  • [2026-05-25] FlashVSR v1.1 — Real-Time Diffusion-Based Streaming Video Super-Resolution (HF model release) (FlashVSR v1.1, model card): drop-in weights update on the Oct 2025 v1 release. v1 architecture: three-stage distillation pipeline + Locality-Constrained Sparse Attention (LCSA) + Tiny Conditional Decoder. Wan2.1 VAE base. ~17 FPS at 768×1408 on single A100, ~12× faster than prior one-step diffusion VSR. v1.1 advertises “enhanced stability + fidelity” qualitatively (no quantitative deltas on the card). Block-Sparse Attention backend required for LCSA; third-party ComfyUI ports fall back to dense attention and lose quality.
  • [2026-05-23] DUO-VSR: Dual-Stream Distillation for One-Step Video Super-Resolution (DUO-VSR, CVPR 2026 submission, HKU + Kuaishou/Kling): three-stage distillation to one-step student. Stage I: Progressive Guided Distillation Initialization (CFG-distill, then progressively halve denoising steps to 1). Stage II: dual-stream Distribution Matching + Real–Fake Score Feature GAN (real-score and fake-score models as discriminator backbones). Stage III: Preference-Guided Refinement via DPO on a synthetic VQA-ranked preference dataset. Best/near-best NIQE+MUSIQ on YouHQ40, VideoLQ, AIGC60. Fidelity metrics “comparable” — not best. The first wiki datapoint combining DMD with GAN through a shared score-model backbone.
  • [2026-05-23] SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation (SparkVSR, Tu @ Texas A&M): one-step VSR on CogVideoX1.5-5B I2V with sparse HR-keyframe conditioning (channel-concatenated 16+16 → 32-channel input; zero-filled at non-keyframe indices). Two-stage training (latent MSE → joint image+video pixel-space MSE + DISTS + temporal consistency); reference-dropout (p=0.5) for reference-free fallback. Reference-Free Guidance (RFG): CFG in reference space, w<1 = blind SR, w>1 = amplified external prior, w=1 = standard. Keyframe modes: manual, codec I-frames, random. Beats FlashVSR-Full, SeedVR2-7B, STAR on perceptual metrics across 5 benchmarks including MovieLQ vintage footage.

Open questions

Section titled “Open questions”
  • Are the four levers composable? SparkVSR’s keyframe-conditioning, DUO-VSR’s distillation procedure, FlashVSR’s LCSA kernel, and Vivid-VR’s concept distillation each attack independent failure modes. The clean test would be SparkVSR’s reference-keyframe channel + DUO-VSR’s three-stage distillation + FlashVSR’s LCSA + Vivid-VR’s synthetic training data. Nobody has tried more than two of these together.
  • Does PiD’s “decode + upsample as one diffusion model” idea lift to video? The image case caps at 4×/8× spatial upscale; the video case has to add temporal consistency across the joint decode+upsample step. PiD does not address this. The natural test is to take CogVideoX1.5 / Wan2.1’s latent and try a sigma-aware pixel-space DiT decoder trained for joint decode+upsample on video — but PiD has not been tested on video and the cluster’s incumbents (SparkVSR, FlashVSR, Vivid-VR, DUO-VSR) all keep the temporal axis through a frozen T2V’s prior, not through a from-scratch pixel-space video DiT.
  • Does the “4K keyframe + 720p I2V + 4K upsample” pipeline that Terrance flagged actually work? SparkVSR’s RFG channel was built exactly for this, but the paper does not test at ×6 effective upscale with high-quality external anchors. This is the obvious validation experiment for Luma’s I2V pipeline. PiD also doesn’t test this regime — its evaluations are 512²→2048² (4×) and 512²→4096² (8×) decode-from-latent, not “keyframe + LR-video → HR-video” conditioning.
  • Is one-step inference necessary for production? Vivid-VR keeps 50-step multi-step inference and stays close to native CogVideoX1.5 quality. FlashVSR and DUO-VSR cut to one step at some perceptual cost. The right operating point on the quality × latency × memory surface depends on the deployment regime — research evaluations can afford the multi-step model, but a real-time interactive workflow can’t. RTX Video SDK pushes the latency floor to “consumer playback on a single RTX card”, so the diffusion line has to defend the quality gap, not just close latency. PiD’s 4-step DMD2 distillation lands between Vivid-VR and FlashVSR/DUO-VSR on this axis, for the image case.
  • Which T2V base is the right substrate? CogVideoX1.5-5B (SparkVSR, Vivid-VR) vs Wan2.1 (FlashVSR, DUO-VSR’s likely base). No paper compares the same recipe across two T2V foundations to isolate the base-model effect from the adapter design.
  • Are no-reference perceptual metrics gameable? MUSIQ, CLIP-IQA, DOVER, MD-VQA dominate this cluster’s evaluation. The perception-distortion tradeoff is real, but the methodology — all four papers picking the metrics they win on — leaves room for metric-specific overfitting. A clean human-MOS study across the four papers would settle it. RTX Video SDK ships with no published metrics at all, complicating any head-to-head comparison. PiD reports human-judged preference rates per project page, but not standardized perceptual metrics — same caveat applies.
  • Restoration vs super-resolution vs editing as one problem? Vivid-VR includes the AIGC50 testset, explicitly targeting restoration of generative-video outputs. SparkVSR demonstrates out-of-the-box video style transfer with no retraining. The “video restoration” problem is collapsing into a generic “guided video-to-video at higher quality” problem; if so, the cluster will eventually merge with Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting, Lucy Edit 1.1 Dev: Open-Weight Instruction-Guided Video Editing on Wan2.2 5B, and the broader in-context video editing line. PiD pushes this further at the image scale: it explicitly subsumes the “LDM-decode then external super-res” cascade into one model, suggesting that for images the right framing is no longer “super-resolution” at all but “high-resolution latent decoding”.
  • How does the LCSA kernel pattern relate to other structured sparse-attention designs? Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation (energy decay), VideoNSA, and FlashVSR’s LCSA all bet on structured sparsity for video. The right primitive — which locality structure — is open.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-08

Section titled “2025-08”

2025-06

Section titled “2025-06”

Unknown date

Section titled “Unknown date”