SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation

TL;DR

SparkVSR is a diffusion-based video super-resolution model built on CogVideoX1.5-5B I2V that accepts a sparse set of user-provided high-resolution keyframes as an additional control signal alongside the low-resolution video. The HR keyframes are encoded into the latent space at their temporal indices (zero-filled elsewhere) and channel-concatenated with the LR video latents (16 + 16 → 32 channels) to drive a one-step denoising pass. Training is two-stage (latent MSE → joint image/video pixel-space MSE + DISTS + temporal-consistency loss), with reference-dropout and zero-latent training enabling reference-free guidance — a CFG-style knob that trades external prior strength against the model’s blind-SR prior. Reported wins over SeedVR2-7B, FlashVSR-Full, DOVE, and STAR on perceptual and VQA metrics (CLIP-IQA, DOVER, MUSIQ, FasterVQA) across five benchmarks, with the largest gains on real-world MovieLQ vintage footage.

Key claims

VSR is inherently ill-posed: the same LR input maps to many plausible HR reconstructions differing in texture and sharpness, which is fundamentally a user-intent problem rather than a learning problem — motivating an interactive control interface [§1].
Editing a small number of keyframe anchors is a more practical interactive interface than per-frame supervision; if the model propagates anchors reliably, the anchors strongly constrain the full sequence [§1].
Per-frame ISR independently applied to each video frame causes severe temporal flickering, because per-frame generation ignores cross-frame dynamics — motivating decomposition into “ISR for keyframe anchors + dedicated VSR for propagation” [§1, §2.2].
The model is built on CogVideoX1.5-5B I2V; the LR video is encoded by the 3D causal VAE (×4 temporal downsampling) into a 16-channel latent, and HR keyframes are encoded by a separate spatial encoder into a 16-channel sparse-reference latent (zero-filled at non-keyframe indices). These are channel-concatenated into a 32-channel joint condition fed to the DiT [§3.1, Fig. 1, Eq. 1].
One-step diffusion (timestep fixed at $τ$ ) inspired by DOVE — preserves global structure from the LR latent while letting the DiT focus on high-frequency detail conditioned on the keyframe reference [§3.1].
Two-stage training: Stage 1 trains only in latent space with MSE for efficiency; Stage 2 transitions to pixel space and adds a joint image+video branch with pixel-MSE + DISTS perceptual + a frame-consistency loss; the image branch uses a zero-latent in place of reference latents to maintain channel-count and reinforce the model’s reference-free priors [§3.2, Eq. 2].
Reference-dropout during training (probability $p$ =0.5) — keyframe latents replaced with zero tensors — enables the same model to operate either with or without keyframe references [§3.2, §4.1].
Keyframe sampling constraint: the temporal gap between any two selected keyframes must exceed the 3D-VAE temporal downsampling rate (=4), so that distinct keyframes do not collide in the same latent frame index [§3.2].
Reference-Free Guidance (RFG): inference combines a referenced prediction and a zero-latent prediction with a scale $w$ , mirroring CFG; $w$ =1 is standard guided generation, $w$ >1 amplifies external priors, $w$ <1 or $w$ =0 falls back to blind SR — letting the user tune adherence to potentially imperfect keyframes [§3.3, Eq. 3].
Keyframe selection supports three modes: manual (artistic intent), codec I-frames (free, high-quality anchors from the video stream itself), and random (automated batch processing) [§3.3].
Stage-1-only training preserves fidelity (PSNR 26.73 on UDM10 zero-reference) but underperforms perceptually; adding Stage 2 trades ~0.1 PSNR for large jumps in MUSIQ / CLIP-IQA / DOVER across all reference modes [Table 2].
On UDM10, the reference-free variant (Ours*) gets the best PSNR (26.62) and SSIM (0.7756) of any compared method including DOVE (26.52 / 0.7697); reference-guided variants then dominate on MUSIQ / CLIP-IQA / FasterVQA / DOVER [Table 1].
On the new MovieLQ dataset (10 vintage film clips, 8s × 24fps × 360p), the PiSA-SR-referenced variant (Ours‡) tops every baseline on MUSIQ (68.88), CLIP-IQA (0.6361), FasterVQA (0.8028), DOVER (0.6212) — beating FlashVSR-Full and STAR [Table 1, §4.2].
Reference number ablation: scaling from 0 to 4 evenly-spaced reference frames monotonically improves all perceptual metrics (e.g. MUSIQ 56.34 → 65.76, CLIP-IQA 0.462 → 0.610); codec I-frames are competitive with manually-placed 4-frame anchors [Table 3].
The framework generalizes out-of-the-box to old-film restoration (validated on MovieLQ) and video style transfer with no task-specific retraining [§1, §4.2].

Method

SparkVSR fine-tunes CogVideoX1.5-5B I2V to perform single-step VSR with sparse keyframe references. Given a degraded LR video $V$ , the 3D causal VAE produces a 16-channel latent $z_{LR}$ . Separately, the user selects sparse keyframe indices $K$ , super-resolves them with any off-the-shelf ISR model (Nano-Banana-Pro for interactive, PiSA-SR for blind), and a spatial encoder maps them into a 16-channel sparse reference latent $z_{ref}$ — non-keyframe positions are zero tensors. The two latents are channel-concatenated to form a 32-channel conditioning input. Denoising initializes from $z_{LR}$ as the noisy latent and runs a single step at a fixed intermediate timestep $τ$ , after which the VAE decoder reconstructs $\hat{V}$ .

Training is two-stage (Fig. 2). Stage 1 keeps the VAE frozen and trains only the DiT in latent space with MSE between predicted and ground-truth latents; keyframes are randomly sampled from the GT video, augmented with ColorJitter / GaussianBlur / noise to simulate ISR outputs, and dropped to zero with probability 0.5 to teach reference-free behavior. Stage 2 moves to pixel space with a joint training scheme: the video branch uses the same sparse-keyframe protocol and is supervised by $\mathcal{L}_{MSE} + \mathcal{L}_{DISTS} + \mathcal{L}_{cons}$ with weights (0.5, 0.5, 1.0); a parallel image branch processes single LR images concatenated with a zero latent and is supervised by $\mathcal{L}_{MSE} + \mathcal{L}_{DISTS}$ , hardening the model’s reference-free priors. At inference, the user picks a keyframe-selection strategy and an RFG scale $w$ that linearly mixes the conditional and zero-latent denoising directions.

Results

Trained on HQ-VSR (2055 clips) + DIV2K (900 images) on 4× A100-80G for 10000 latent-stage iters + 500 pixel-stage iters. Evaluated on UDM10, SPMCS, YouHQ40, RealVSR, and MovieLQ against STAR, DOVE, SeedVR2-3B/7B, FlashVSR-Tiny/Full [Table 1]. Reference-free SparkVSR (Ours*) wins on PSNR/SSIM on UDM10 (26.62 / 0.7756); reference-guided variants win on MUSIQ / CLIP-IQA / FasterVQA / DOVER across all datasets. On MovieLQ — the most challenging real-world test — Ours‡ beats FlashVSR-Full on MUSIQ (68.88 vs 66.38) and CLIP-IQA (0.6361 vs 0.5754) and crushes SeedVR2-7B (CLIP-IQA 0.2894). Headline gains over the best baseline are “up to 24.6%, 21.8%, and 5.6% on CLIP-IQA, DOVER, and MUSIQ” per the abstract. Ablations [Tables 2-3]: Stage 2 is necessary for perceptual quality; reference count scales perceptual metrics monotonically; codec I-frames match hand-picked 4-frame anchors.

Why it’s interesting

Direct relevance to Luma’s I2V pipeline: the recipe Terrance flagged (generate one anchor at high res → downsample → run I2V at low res → upsample with the anchor as a reference) is exactly the setup SparkVSR’s reference-keyframe channel was built to support, and this is the first filed paper that explicitly adds a CFG-style adherence knob (RFG) to that workflow. The paper does not test the 4K-anchor / 720p-I2V / 4K-upsample regime, so its empirical claims are bounded by standard ×4 SR benchmarks — that gap is the thing to validate before adopting. Architecturally it complements ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video Generation (same lead institution — Tu @ Texas A&M — also using auxiliary-keyframe-as-conditioning on a video DiT, but on the generation side via MVGM dual-stream attention rather than SparkVSR’s channel-concat for restoration). It contrasts with OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models‘s training-free strategy: SparkVSR fully fine-tunes with explicit keyframe supervision, while OmnimatteZero re-uses a base video diffusion model via self-attention readout and latent arithmetic — two opposite bets on whether high-frequency video tasks need keyframe-specific fine-tuning. The RFG mechanism is also a natural Classifier-Free Guidance variants datapoint where the “unconditional” branch is “no reference (blind SR)” rather than “no text”.