HDR Video Generation via Latent Alignment with Logarithmic Encoding (LumiVid)

TL;DR

LumiVid generates HDR video from a single SDR input by LoRA-adapting a pretrained SDR-only video diffusion transformer (LTX-Video) without retraining its VAE or modifying its architecture. The central observation is that the cinematic LogC3 transfer curve, applied as a fixed differentiable pre-encoder, maps scene-linear HDR radiance into the VAE’s pretrained input distribution well enough (KL ≈ 0.215 in pixel space, 0.007 in latent space against SDR) that the frozen VAE can round-trip HDR with PU21-PSNR 34.7 / SSIM 0.97. A camera-mimicking augmentation suite (MP4 compression, contrast scaling, selective highlight/shadow blur) deliberately destroys extreme-luminance detail in the SDR conditioning so the model is forced to infer — not copy — highlight and shadow content from its learned priors. The LoRA adds <1% parameters, trains for 10k steps on 300 clips (8h on one GPU), and beats X2HDR, HDRTVNet, and LEDiff by 3.7 dB PU21-PSNR on out-of-distribution ARRI cinema footage, with the largest gap on temporal flicker.

Key claims

A fixed LogC3 transfer curve, applied to scene-linear HDR before the frozen VAE, achieves the lowest pixel-space KL (0.215) and latent-space KL (0.007) against the SDR distribution among the candidates tested (LogC3, PQ, HLG, ACES), enabling HDR round-trip through the SDR-trained VAE without VAE retraining [§3.1, Table in §3].
A two-stage KL-divergence selection — first $\mathrm{KL}_{\text{px}}$ against normalized SDR pixel stats, then $\mathrm{KL}_{\text{lat}}$ against the VAE-encoded SDR latent distribution — is the criterion used to pick LogC3 over PQ/HLG/ACES; ACES catastrophically fails PU21-PSNR (16.42), HLG fails by clamping highlights [§3.1, Fig. 4].
Three SDR-only augmentations (MP4 compression, contrast scaling, selective highlight/shadow blur) are required to prevent the model from pixel-copying and to force generative reconstruction from priors; ablating them (“Blur only”) drops JOD from 7.86 to 6.90 [§3.2, Table 5].
The full method runs on top of LTX-Video with the AVControl conditioning interface ([ben2026avcontrol]), a LoRA adapter adding less than 1% of parameters, trained for only 10,000 steps on 300 clips (8 GPU-hours) [§5].
On out-of-distribution ARRI cinema 12-bit footage, LumiVid reaches PU21-PSNR 36.20 / LPIPS 0.020 / JOD 7.86, vs HDRTVNet 26.48 / 0.089 / 6.94 and X2HDR 20.68 / 0.250 / 3.54 [Table 2].
On UPIQ static-image HDR (16-bit, calibrated to cd/m²), LumiVid generalizes despite being a video model: PU21-PSNR 30.05, JOD 8.22, vs HDRTVNet 22.59 / 4.48 and LEDiff 14.94 / 0.40 [Table 2].
Native joint denoising of all 49 frames gives F2F-PSNR 45.63 and flicker 0.0245, beating per-frame X2HDR (36.36 / 0.163) by a wide margin on temporal stability [Table 3].
Encoding-ablation: PQ matches LogC3 closely (PU21-PSNR 36.72 vs 36.97, JOD 7.62 vs 7.86) while ACES gets higher PU21-PSNR (39.30) but lower JOD (7.40) — the pure-pixel metric and the perceptual metric disagree, and the authors prefer LogC3 on perceptual grounds [Table 4].
Training data is curated from PolyHaven HDRIs rendered with synthetic camera rotations plus the open-source HDR film Tears of Steel; this is the first time the latter is reported being used as an HDR generative-model training source [§3.2].

Method

LumiVid is a video-to-video LoRA finetune on a frozen LTX-Video DiT + frozen LTX VAE. An SDR input frame is mapped through the LogC3 cinematic transfer curve (a fixed, differentiable monotone function chosen because it minimizes a two-stage KL divergence against the VAE’s SDR pretraining distribution), encoded by the frozen VAE, and routed through AVControl as the conditioning latent for the DiT. The DiT, starting from Gaussian noise, denoises into a target HDR latent; the same VAE decodes it back to LogC3-space pixels, and the inverse LogC3 recovers scene-linear float16 EXR.

Three classes of augmentation are applied to the SDR conditioning only — MP4 compression artifacts, contrast scaling that clips dynamic range, and selective Gaussian blur targeted at highlights and shadows — to destroy exactly the regions the model is supposed to hallucinate. A fourth augmentation, exposure shift, is applied jointly to SDR conditioning and HDR target to teach exposure-invariance while preserving SDR↔HDR pixel correspondence. Training: 10k steps, 300 clips (PolyHaven HDRIs + Tears of Steel), single GPU, 8 hours. Inference: 11 denoising steps; two-stage pipeline (the paper notes future work on collapsing to a single text-to-HDR-video pass).

Results

ARRI 12-bit OOD video: PU21-PSNR 36.20 / LPIPS 0.020 / JOD 7.86; beats next-best (HDRTVNet) by 9.7 dB PU21-PSNR and 0.92 JOD [Table 2].
UPIQ 16-bit OOD image: PU21-PSNR 30.05 / JOD 8.22; the model is trained on video but processes images as 9-frame sequences and still leads HDRTVNet by 7.5 dB and LEDiff by 15.1 dB [Table 2].
Temporal stability on ARRI: F2F-PSNR 45.63 dB, flicker 0.0245, JOD 7.86 — vs per-frame X2HDR’s catastrophic flicker 0.163 and JOD 3.54 [Table 3].
Encoding ablation: LogC3 wins JOD (7.86) over PQ (7.62) and ACES (7.40); ACES wins PU21-PSNR (39.30) but loses perceptually, illustrating the PSNR/perceptual gap on HDR [Table 4].
Augmentation ablation: full pipeline reaches JOD 7.86; removing all augmentations drops to 7.43; using only blur drops to 6.90 — i.e. the augmentations are roughly worth ~1.0 JOD point [Table 5].
Compute envelope: <1% adapter parameters, 10k steps, 300 clips, 8 hours on one GPU, 11 inference steps [§5].

Why it’s interesting

LumiVid is a near-simultaneous, near-identical recipe to DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models arrived at independently: both file in April 2026, both adapt a frozen pretrained video DiT + frozen VAE, both apply a logarithmic input transform to bring HDR radiance into the VAE’s pretrained domain, both use rank-tiny LoRA, and both rely on a curated PolyHaven HDRI render pipeline. The differences pin down which design choices actually matter. DiffHDR uses Log-Gamma on Wan-2.1-VACE-14B with a context-focused cross-attention for region-targeted prompting at inference; LumiVid uses LogC3 on LTX-Video with AVControl and shifts the regional-reconstruction lift onto training-time augmentation (MP4/contrast/highlight-blur) instead of inference-time control. Both achieve dB-scale gains over X2HDR/HDRTVNet/LEDiff with strong temporal stability. Together they form a striking convergence story for Parameter-Efficient Finetuning: when the domain gap is information-theoretic (8-bit → float16 radiance) rather than semantic, the right primitive is “find a fixed transform that aligns the data with the pretrained input distribution + small LoRA,” and the backbone choice (LTX vs Wan) appears secondary. The PSNR-vs-JOD disagreement on the ACES ablation also rhymes with the broader pixel-metric-isn’t-perceptual-metric story in Making Reconstruction FID Predictive of Diffusion Generation FID.