Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration

TL;DR

Vivid-VR is a ControlNet-conditioned video restoration model (ICLR 2026) built on the CogVideoX1.5-5B T2V foundation, from the Alibaba Taobao/Tmall group. The headline idea is concept distillation: instead of trying to fix the imperfect text-video alignment in the captioned training corpus, use the frozen T2V model itself to synthesize aligned training pairs — perturb a real video with noise, denoise with the T2V conditioned on its VLM-generated caption, and use the resulting video (semantically aligned by construction, in the T2V’s own latent space) alongside real videos during fine-tuning. The other two contributions are architectural: a lightweight 3-block CNN “control feature projector” that strips degradation artifacts from LQ video latents before ControlNet sees them, and a dual-branch ControlNet connector that fuses MLP-mapped control tokens with cross-attention-retrieved control tokens. Reported wins on no-reference perceptual metrics (NIQE, MUSIQ, CLIP-IQA, DOVER, MD-VQA) across SPMCS, UDM10, YouHQ40, VideoLQ, plus the authors’ new UGC50 and AIGC50 testsets [§4.2].

Key claims

Current T2V-based VSR/VR methods underperform native T2V models on texture realism and temporal coherence due to distribution drift induced by imperfect VLM-captioner alignment during fine-tuning — pretrained T2V models don’t suffer this because their training corpus is large/diverse enough to absorb misalignment, but restoration fine-tuning amplifies it [§1].
The concept distillation training strategy: perturb a source video with noise (std σ corresponding to timestep t), denoise with frozen CogVideoX1.5-5B conditioned on the VLM caption, yielding a synthetic video that retains source content but has concepts shifted to align with the text in the T2V’s own latent space [§3.2, Fig. 3].
The control feature projector — three cascaded spatiotemporal residual blocks appended to the VAE encoder — filters degradation artifacts from LQ video latents before they enter ControlNet, avoiding the cost of jointly fine-tuning the VAE encoder (as SUPIR does) or doing FaithDiff-style joint optimization [§3.1, §5.1].
Decoupled VAE-encoder fine-tuning (SUPIR’s approach) creates feature incompatibility with downstream DiT and ControlNet, producing suboptimal results [Table 2(b) vs Table 2(f)].
The dual-branch ControlNet connector fuses control tokens via both an MLP feature mapping and a cross-attention dynamic retrieval — disabling the cross-attention branch produces detail-poor outputs (MUSIQ 59.78 vs 67.61) and disabling the MLP branch causes the model to fail to converge entirely [Table 2(c)/(d), §5.2, Fig. 6].
The ZeroSFT connector used in SUPIR underperforms the dual-branch connector and its internal normalization causes residual artifacts from adjacent frames to bleed into outputs; removing the normalization causes gradient explosion [§5.2, Fig. 6(d)].
On no-reference perceptual metrics, Vivid-VR is best or near-best across SPMCS, UDM10, YouHQ40, VideoLQ, UGC50, AIGC50 vs Real-ESRGAN, SUPIR, MGLD, UAV, STAR, DOVE, SeedVR-7B, SeedVR2-7B — e.g. MUSIQ 70.03 on SPMCS vs 65.23 (SUPIR) and 66.11 (UAV); MUSIQ 67.61 on UGC50 vs 61.42 (MGLD next-best) [Table 1].
Full-reference metrics (PSNR/SSIM/LPIPS) are less favorable — DOVE and SeedVR2-7B beat Vivid-VR on PSNR/SSIM — which the authors argue reflects the known perception-distortion tradeoff and limitations of pixel-fidelity metrics on severely-degraded inputs where many plausible HQ outputs exist [§4.2].
Concept distillation is necessary, not optional: ablating the synthetic data drops DOVER from 14.46 to 12.99 on UGC50 (~10% lower) and the qualitative results show oversharpened textures and temporal degradation — both perceptual quality and temporal coherence fail without it [Table 2(e), Fig. 7(f), Fig. 5(f)].
Training data: ~1M screened high-quality videos (>720p, >24fps, >10s) captioned by CogVLM2-Video, plus ~50k synthetic samples generated via the concept distillation procedure [§3.2, §4.1].
Training cost: NVIDIA H20-96G GPUs (×64 implied), 30k iterations, batch size 1/GPU, ~5000 GPU-hours; only the control feature projector, ControlNet, and connectors are trained — DiT and VAE remain frozen [§4.1].
Inference uses 50 denoising steps with DPM solver at 1024-pixel short-side; higher resolutions handled via aggregation sampling with direct block concatenation (no Gaussian weighting to avoid overlap artifacts) [§4.1].
GPU memory: ~43GB for a 121-frame clip with enable_model_cpu_offload; sequential CPU offload reduces this to ~25GB at higher latency [README].
Restoration-guided sampling (added 2025-09-20 in the repo) provides a fidelity-vs-realism knob via --restoration_guidance_scale; analogous to but distinct from SparkVSR’s RFG [README, post-paper update].
The model only does multi-step inference; the authors flag one-step fine-tuning as future work — currently inherits CogVideoX1.5-5B’s full inference cost [§6].

Method

Vivid-VR follows the standard “ControlNet on a frozen T2V” pattern but plugs degradation handling and alignment supervision in three specific places. Input pipeline. The LQ video is encoded by the CogVideoX1.5-5B VAE encoder into a latent containing both content and artifacts. The three-block spatiotemporal residual control feature projector sits between the VAE output and the patchifier, filtering artifacts before the latent becomes a sequence of visual tokens for ControlNet. CogVLM2-Video produces an aligned caption from the LQ input; T5 encodes it into text tokens — keeping the captioner consistent with CogVideoX1.5-5B’s pretraining configuration.

ControlNet and connector. ControlNet blocks are initialized from the DiT’s first blocks. The dual-branch connector between each ControlNet block and the corresponding DiT block fuses control tokens two ways in parallel: an MLP layer maps the ControlNet token directly, and a cross-attention module retrieves it conditioned on the corresponding DiT token. The sum is the residual injected into the DiT — the MLP branch carries direct control signal, the cross-attention branch retrieves dynamically based on DiT context.

Concept distillation procedure. Given a (video, VLM caption) pair, add Gaussian noise to the video at timestep t (the paper doesn’t specify t precisely, but it lies in the high-noise range where T2V semantics dominate), then denoise from t→0 with frozen CogVideoX1.5-5B conditioned on the caption. The output is a video that (a) retains most of the source’s content and structure, (b) has its concepts shifted to align with the caption in the T2V’s latent space, by construction. Mixing these synthetic samples with real samples during fine-tuning gives the restoration model perfectly-aligned training pairs — the model never sees a misaligned (video, text) pair from the synthetic stream, and the real stream supplies the residual distributional grounding the synthetic stream lacks.

Loss. v-prediction (matching CogVideoX1.5-5B’s pretraining), supervising the DiT+ControlNet on the noised latent of the high-quality video given a synthesized LQ counterpart from the Real-ESRGAN-style degradation pipeline. Only the projector, ControlNet, and connectors update; the DiT and VAE stay frozen [§3.2, Eq. 2].

Results

Five real / synthetic benchmarks (SPMCS, UDM10, YouHQ40, VideoLQ, plus the authors’ new UGC50 and AIGC50). On no-reference perceptual metrics Vivid-VR is consistently best or near-best:

SPMCS: MUSIQ 70.03 (next: SUPIR 65.23), CLIP-IQA 0.483 (next: SUPIR 0.469), DOVER 11.35 (next: SUPIR 10.07), MD-VQA 86.55 (next: DOVE 83.07).
UDM10: MUSIQ 64.71 (next: SUPIR 60.84), CLIP-IQA 0.426 (next: SUPIR 0.418), DOVER 11.97 (next: SUPIR 10.49), MD-VQA 90.05 (next: SUPIR 85.21).
YouHQ40: MUSIQ 70.55 (next: SUPIR 68.46), CLIP-IQA 0.447 (next: SUPIR 0.485 — only metric where SUPIR ties/beats), DOVER 14.61 (next: SeedVR-7B 14.00).
VideoLQ (real-world): MUSIQ 62.47 (next: MGLD 57.70), CLIP-IQA 0.338 (next: STAR 0.265), DOVER 9.743 (next: MGLD 8.830), NIQE 4.371 (next: MGLD 4.565).
UGC50: MUSIQ 67.61 (next: MGLD 61.42), CLIP-IQA 0.450 (next: MGLD 0.396), DOVER 14.46 (next: DOVE 14.49 — ties), MD-VQA 88.89 (next: MGLD 84.81).
AIGC50: MUSIQ 67.18 (next: DOVE 62.07), CLIP-IQA 0.445 (next: MGLD 0.391), MD-VQA 89.69 (next: STAR 86.97).

Full-reference fidelity numbers (PSNR/SSIM/LPIPS) are worse — DOVE and SeedVR2-7B beat Vivid-VR by 2-5 PSNR points across SPMCS / UDM10 / YouHQ40. The authors frame this as the perception-distortion tradeoff bites hardest in generative-restoration regimes where multiple plausible HQ outputs exist [§4.2].

Ablations [Table 2]: removing concept distillation alone drops DOVER from 14.46 → 12.99 on UGC50 (~10%) and increases NIQE from 4.361 → 5.364 — the largest single hit of any ablation. Removing the cross-attention branch of the connector drops MUSIQ from 67.61 → 59.78. Removing the MLP branch fails to converge. ZeroSFT scores below the dual-branch connector and produces visible adjacent-frame artifacts. Replacing the projector with VAE-encoder fine-tuning (SUPIR-style) modestly hurts [(a) vs (b) vs (f)].

Why it’s interesting

Direct addition to the growing video-super-resolution / video-restoration cluster Paul has been seeding the wiki with — joining SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation (keyframe-conditioning lever), DUO-VSR: Dual-Stream Distillation for One-Step Video Super-Resolution (distillation-procedure lever), and FlashVSR v1.1 — Real-Time Diffusion-Based Streaming Video Super-Resolution (HF model release) (systems / sparse-attention kernel lever). Vivid-VR adds a fourth orthogonal lever: training-data alignment via T2V-generated synthetic pairs. The four bets are now: (a) keep the multi-step model, condition on HR keyframes (SparkVSR); (b) cut to one step via better distillation procedure (DUO-VSR); (c) cut to one step via sparse-attention kernel + tiny decoder (FlashVSR); (d) keep the multi-step model, but fix the supervision data (Vivid-VR). Vivid-VR is the only one that doesn’t try to be faster — it explicitly inherits CogVideoX1.5-5B’s full inference cost and flags one-step distillation as future work, making it the “quality-first, latency-later” reference point in this cluster. The 43GB-for-121-frames memory footprint is the practical knock against deploying it in the I2V evaluation pipeline.

The concept-distillation idea itself is interesting beyond VSR. It’s an instance of the broader “synthesize training data with the model you’re trying to fine-tune from” pattern that shows up in NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale (visual self-correction), DataRater: Meta-Learned Dataset Curation (meta-learned curation), and Sparsity is Cool: Reverse-Engineering MoBA and NSA. The specific contribution here is using the T2V to de-noise a caption-perturbed real video — not to generate from scratch, but to gently shift content into the T2V’s own latent-space alignment with the caption. The fact that this fixes both texture realism and temporal coherence (Fig. 5(f)) is the empirically interesting bit: the distribution drift from misalignment hurts time consistency, not just appearance. This is a useful argument for any team relying on VLM captioning of real videos as a training signal.

The architectural pieces (control feature projector, dual-branch connector) are smaller contributions but worth noting in the ControlNet-on-DiT design space — both directly attack failure modes of SUPIR/FaithDiff carried over from the T2I-restoration era. The “MLP-only fails to converge” ablation is a nice negative result for the naive baseline.