DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models

TL;DR

DiffHDR converts 8-bit LDR videos into HDR videos by formulating LDR→HDR as a generative radiance-inpainting problem in the latent space of a pretrained video diffusion model (Wan-2.1-VACE-14B), with no VAE finetuning. The central trick is a Log-Gamma color mapping that compresses HDR radiance into a bounded range compatible with the LDR-trained VAE’s pretrained domain, so the existing VAE and DiT priors can be reused. Training uses rank-32 LoRA adapters on a curated 5,400-sequence HDR dataset rendered in Blender from Polyhaven 16K HDRIs, plus an exposure-mask-guided context-focused cross-attention that lets text prompts or reference images locally steer the hallucination in saturated and underexposed regions. The method beats SingleHDR and LEDiff across image and video benchmarks (SI-HDR, Cinematic, in-the-wild Pexels, Veo2-generated), with the largest gains on temporal stability and on real-world generalization despite training only on rendered HDRIs.

Key claims

• A Log-Gamma color mapping (logarithmic compression of linear HDR radiance scaled by max radiance, with a gamma factor) aligns HDR content with the LDR-trained VAE’s operational range, enabling HDR encode/decode through the pretrained VAE without any VAE finetuning [§3.2, Fig. 5, Eq. 3].
• Wan-2.1-VACE-14B is adapted with rank-32 LoRA adapters inserted into DiT attention and feed-forward layers; the backbone is frozen, preserving the pretrained generative prior [§3.3.3, §4.1].
• A curated HDR video dataset of ~5,400 81-frame sequences is synthesized in Blender from ~800 Polyhaven 16K HDRIs using three camera motion patterns (highlight-zoom, shadow-zoom, 360° rotation), with LDR formation simulated via exposure shift, AR(1)-correlated heteroscedastic camera noise, sRGB quantization, and clipping [§3.1, §3.1.2, Eq. 1–2].
• Context-focused prompting uses a structured [overexposed: ...]; [underexposed: ...] format and is applied at inference only through a context-focused cross-attention (CFA) module that mask-routes the over/under embeddings against a base unconditional embedding inside every DiT cross-attention layer [§3.4.2, Eq. 7–8].
• Despite training only on synthetic videos derived from static HDRIs, DiffHDR generalizes to real-world Pexels videos and to Veo2-generated content, outperforming SingleHDR and LEDiff on DOVER / MUSIQ / CLIPIQA across both [Table 3].
• DiffHDR achieves the best PU21-PIQE (19.37) and FID (18.68) on SI-HDR images and ranks second on HDR-VDP3 (6.98); the slightly lower HDR-VDP3 is attributed to generatively inpainted plausible details not matching ground-truth pixels in clipped regions [§4.2.1, Table 1].
• On Cinematic and Polyhaven video benchmarks, DiffHDR wins all four metrics (FOVVDP, DOVER, MUSIQ, CLIPIQA) against SingleHDR and LEDiff [Table 2].
• Training requires mixed precision with FP32 for the VAE and BF16 only for the DiT — BF16 in the VAE introduces banding artifacts in HDR decoding due to limited precision [§4.1].

Method

The pipeline is a single-stage video-to-video latent diffusion finetune on top of VACE [jiang2025vace]. An input LDR video is linearized via inverse sRGB, mapped to Log-Gamma color space, and encoded by the pretrained Wan-2.1 VAE into a latent. That latent feeds the VACE context branch; a luminance-based exposure mask (computed in linear space with Rec.709 luminance, thresholded at 0.95 / 0.05, and EMA-smoothed across frames with α=0.6) is concatenated as conditioning. Starting from a Gaussian-noise latent, the main DiT branch denoises under rectified flow matching toward an HDR latent, which is decoded by the same VAE and inverse Log-Gamma mapped back to linear HDR.

Two control surfaces are exposed at inference. (1) Context-focused prompting splits the caption into per-region descriptions formatted as [overexposed: ...]; [underexposed: ...]; a CFA module re-evaluates each DiT cross-attention layer three times (unconditional, overexposed, underexposed) and routes the additional over/under residuals to pixels whose exposure-mask says “this region is clipped.” (2) An optional reference image is VAE-encoded and concatenated along the temporal dimension to inject appearance cues — exactly the channel VACE already uses for its native reference-image conditioning.

Training runs 10k steps on 8×A100 at 480p×81-frame, AdamW with constant LR 1e-4, mixed precision with FP32 VAE.

Results

• SI-HDR image benchmark: PU21-PIQE 19.37 (best), FID 18.68 (best), HDR-VDP3 6.98 (2nd, behind SingleHDR’s 7.37). HDR-VDP3 penalizes generative inpainting that doesn’t match GT pixel-wise in clipped regions [Table 1].
• Cinematic Video dataset: FOVVDP 6.89 vs LEDiff 3.75 / SingleHDR 6.56; DOVER 0.81 vs 0.63 / 0.77; MUSIQ 58.38 vs 47.90 / 51.79; CLIPIQA 0.41 vs 0.28 / 0.33 [Table 2].
• Polyhaven Synthetic Video (held-out from training): FOVVDP 7.65, DOVER 0.68, MUSIQ 60.02, CLIPIQA 0.50 — best across all four [Table 2].
• In-the-wild Pexels: DOVER 0.74 / MUSIQ 55.79 / CLIPIQA 0.48 — best across all three; Veo2-generated videos: DOVER 0.61 / MUSIQ 46.06 / CLIPIQA 0.34 — best across all three. Demonstrates that the rendered-HDRI-only training generalizes [Table 3].
• Qualitatively, DiffHDR recovers sun radiance with wider dynamic range and stable temporal behavior on Pexels footage; LEDiff and SingleHDR flicker in saturated regions [Fig. 3, 4].
• Ablations on the Polyhaven held-out set validate Log-Gamma (vs Linear or pure-Log encoding before the VAE) as necessary for accurate HDR encode/decode [§4.2.3].

Why it’s interesting

DiffHDR is a Luma-led collaboration (authors include Mingming He, Pablo Salamanca, Dmitriy Smirnov, Pablo Delgado, Ning Yu, Julien Philip, Paul Debevec) and is the rare HDR paper that demonstrates that a pretrained LDR-only video VAE can be reused for an out-of-distribution radiance domain without VAE retraining, simply by reshaping the input distribution into the VAE’s operational range. That structural choice — preserve the pretrained prior, finetune only a small LoRA on top — puts it in direct conversation with Parameter-Efficient Finetuning, specifically the µA finding that LoRA’s optimum is rank-invariant and that the backbone’s generative prior is fragile to full-rank finetuning (Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning). The Log-Gamma compatibility recipe also rhymes with HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing‘s message that frozen-backbone diffusion editors get most of their lift from carefully shaped conditioning rather than from weight surgery; HDR is an unusually clean stress test of that hypothesis because the domain gap is information-theoretic (8-bit vs floating-point radiance), not semantic. Contrast with MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines and other Wan/VACE-finetuned downstream systems: DiffHDR keeps the entire VACE conditioning interface intact (Video Condition Unit + reference image branch) and reuses it as-is for the HDR setting.

See also

• Parameter-Efficient Finetuning — DiffHDR is a rank-32 LoRA finetune of a 14B video DiT with a frozen backbone, the canonical Init[B] use case for the µA workflow.
• Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning — µA’s LR-transfer prescription is directly applicable to this kind of LoRA recipe; DiffHDR uses LR=1e-4 across all ranks, a candidate to revisit with µA-style sweeps.
• HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing — same pattern of shaping the conditioning interface (functional neural memory there; Log-Gamma + CFA here) so the frozen diffusion backbone can be repurposed for an unfamiliar task.
• MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines — also a VACE-based downstream finetune; useful contrast on what to freeze vs adapt.