Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation

TL;DR

DKT (BAAI + USC + Tsinghua + others) repurposes a pretrained video diffusion model (Wan) into a video-to-video depth/normal estimator specialized for transparent and highly reflective objects, where stereo, ToF, and discriminative monocular depth all fail because of refraction and reflection. Two contributions support the claim: (i) TransPhy3D, a synthetic Blender/Cycles dataset of 11k videos × 120 frames = 1.32M frames covering glass/plastic/metal materials over category-rich and shape-rich asset banks; and (ii) a LoRA adaptation of the Wan DiT that concatenates RGB and noisy depth latents along the channel dimension, co-trained on TransPhy3D plus existing single-frame synthetic image datasets via a heuristic sampling strategy. Reported zero-shot SOTA on ClearPose, DREDS (CatKnown/CatNovel), and TransPhy3D-Test for depth, plus best video normals on ClearPose, with a 1.3B variant running at ~0.17 s/frame at 832×480 and lifting grasp success on translucent/reflective/diffuse tabletops from 0.46–0.48 (DAv2/DepthCrafter baselines) to 0.73 mean.

Key claims

Generative video diffusion models appear to have internalized the optical rules of refraction/reflection through transparent materials, motivating their use as the prior for transparent-object depth/normal estimation rather than training discriminative models from scratch [§I].
TransPhy3D is the first synthetic video dataset specifically built for transparent and highly reflective objects: 11k scenes × 120 frames = 1.32M frames, rendered in Blender/Cycles with PBR + OptiX denoising and circular camera trajectories around object centroids [§III-A].
The asset bank combines 574 BlenderKit static assets (filtered for transparency/reflectivity by Qwen2.5-VL-7B aesthetic scoring) with a procedurally parameterized asset family (after T2SQNet) to cover both category richness and shape richness [§III-A].
The depth estimator is constructed by concatenating RGB and noisy depth latents along the channel dimension of the Wan DiT and training LoRA adapters with a flow-matching velocity loss; all base-model weights stay frozen [§III-C, Eq. 5].
A heuristic sampling strategy co-trains on TransPhy3D videos and existing single-frame synthetic depth datasets (HISS, DREDS, ClearGrasp) inside one pipeline, treating images as length-1 videos to amortize rendering cost [§III-C, Eq. 4].
LoRA fine-tuning substantially outperforms naive full fine-tuning at the same model size, and scaling to 14B compounds the LoRA gain [Tab. III].
Inference uses a fixed 5 denoising steps; more steps lose detail and fewer steps lose accuracy [Fig. 7, §III-D].
Zero-shot depth on ClearPose: δ₁.₀₅ = 38.17 / δ₁.₁ = 65.50 / δ₁.₂₅ = 93.04 (vs DepthCrafter at 31.92/55.46/88.59 and DAv2 at 32.21/56.37/89.94) [Tab. I].
Zero-shot depth on TransPhy3D-Test: δ₁.₀₅ = 87.17 / δ₁.₁ = 97.09 / δ₁.₂₅ = 98.56 vs the next-best DepthCrafter at 31.92/55.46/88.59 — the paper attributes the large gap to the circular-trajectory rendering protocol that magnifies temporal-consistency errors after global alignment [Tab. I, §IV-C].
Zero-shot depth on DREDS-STD-CatKnown/CatNovel: DKT ranks 1.0 / 1.2 mean across REL, RMSE, δ thresholds, beating VGGT, MoGe, Marigold-E2E-FT, DepthCrafter, DAv2, Depth4ToM [Tab. II].
Zero-shot video normal estimation on ClearPose: DKT-Normal-14B beats both NormalCrafter (ICCV25) and Marigold-E2E-FT (WACV25) on mean angular error (26.03°) and the 11.25°/22.5°/30° accuracy thresholds [Tab. V].
Compute: DKT-1.3B runs at 167.48 ms/frame total (46.53 encode + 52.88 denoise + 68.07 decode) on Nvidia L20, beating DAv2-Large (277.75 ms) and DepthCrafter (565.55 ms), with peak GPU memory of 11.19 GB [Tab. IV].
Real-world grasping with AnyGrasp + CuRobo on a PiPER ARM: DKT-1.3B raises mean grasp success from 0.46 (DAv2) / 0.48 (DepthCrafter) to 0.73, with the largest gain on reflective surfaces (0.27 → 0.59) [Tab. VI].

Method

The system adapts Wan (a video diffusion transformer trained with flow matching) for video-to-video depth translation. RGB and depth videos are each encoded by the Wan VAE to latents z_x and z_d. During training, a flow-matching intermediate z_t = (1−t) z_d + t ε is constructed, and the DiT input is the channel-wise concatenation [z_x, z_t]. The loss is MSE between the DiT velocity prediction and the ground-truth velocity v = ε − z_d. All weights of the base Wan model — VAE, text encoder, DiT — remain frozen; only LoRA adapters inside the DiT are trained.

The data side compensates for the scarcity of supervised transparent-object video by rendering TransPhy3D. The pipeline curates static and procedurally-parameterized 3D assets, pairs them with a glass/plastic/metal/ceramic material library, drops them into a tabletop or container scene under Blender’s rigid-body physics until they settle, and renders 120-frame videos with circular camera trajectories (plus sinusoidal radius/elevation perturbations) using Cycles ray tracing and the OptiX denoiser. Each video carries pixel-aligned RGB + metric depth + normals.

To make rendering tractable, the training loop also draws from existing image datasets (HISS, DREDS, ClearGrasp) treated as 1-frame videos. A heuristic samples a per-batch frame count n: when n=1, it mixes images and videos; when n>1, it draws only videos. Depth is converted to disparity and normalized to [−1,1] to match the VAE training space. Inference runs 5 denoising steps and stitches arbitrary-length inputs by following the DepthCrafter overlapping-segment recipe.

A normal-estimation variant, DKT-Normal-14B, is trained identically, with depth replaced by surface normals as the target latent.

Results

ClearPose depth (Tab. I): DKT REL 9.72 (best), RMSE 14.58 (second-best behind DAv2’s 12.21), and δ thresholds 38.17/65.50/93.04 — ahead of every image and video baseline (Depth4ToM, DAv2, Marigold-E2E-FT, MoGe, VGGT, DepthCrafter). The rank-1.8 mean matches DAv2.
TransPhy3D-Test depth (Tab. I): REL 2.96 vs the next-best DepthCrafter at 11.32, with δ₁.₀₅ jumping from 31.92 to 87.17. The paper attributes the large margin to the circular camera trajectories used at render time, which amplify any temporal-consistency error after global affine alignment.
DREDS-STD-CatKnown / CatNovel depth (Tab. II): DKT REL 5.30 / 5.71, RMSE 4.96 / 4.66, rank 1.0 / 1.2 mean across all metrics, ahead of VGGT (rank 2.0 / 2.8) and Marigold-E2E-FT (rank 3.2 / 4.4).
ClearPose video normals (Tab. V): DKT-Normal-14B mean angular error 26.03° vs NormalCrafter 27.08°, and 30.06 / 59.63 / 70.98% under the 11.25°/22.5°/30° thresholds (best across the board).
Inference latency (Tab. IV): DKT-1.3B 167.48 ms/frame total on Nvidia L20, vs 277.75 ms for DAv2-Large and 565.55 ms for DepthCrafter; peak GPU memory 11.19 GB.
Grasping (Tab. VI): DKT-1.3B mean success 0.73 across translucent/reflective/diffusive surfaces, vs RAW 0.384, DAv2 0.46, DepthCrafter 0.48. Translucent 0.80, reflective 0.59, diffusive 0.81.
Ablations (Tab. III, Fig. 7): LoRA at 1.3B (REL 11.17) beats full fine-tuning at 1.3B (REL 11.86); scaling LoRA to 14B brings REL to 9.72. Inference-step ablation supports 5 steps as the cost/quality sweet spot.

Why it’s interesting

DKT extends the pattern that DVD: Deterministic Video Depth Estimation with Generative Priors established earlier this year — video diffusion as the prior for depth regression, not as the predictor itself — to a vertical (transparency/reflection) where the discriminative-depth field has been stuck for years because the ground-truth supervision is fundamentally broken. DVD’s headline framing was “video diffusion models are overqualified depth estimators”; DKT is the strongest filed evidence so far that the kind of physics a video diffusion model has implicitly internalized goes beyond rigid-body motion into refraction and reflection, which are exactly the failure modes of stereo and ToF. Both papers also share the LoRA-adapt-a-frozen-Wan recipe, but the design choices diverge: DVD fixes the diffusion timestep as a deterministic structural anchor and adds a latent-manifold-rectification loss, while DKT keeps the standard flow-matching training and instead concatenates RGB and depth latents channel-wise inside the DiT and trades stochasticity for a fixed 5-step denoising schedule. The pair makes a clean A/B on how aggressively to “deterministize” a generative depth predictor.

It also lands as a concrete instantiation of the foundation-layer-normative argument in The flavor of the bitter lesson for computer vision: explicit-3D and ground-truth-depth pipelines are giving way to perception heads that ride generative video pre-training. DKT supplies the data side (TransPhy3D) and the model side (LoRA on Wan) of that bet in a regime — transparent-object grasping — where the explicit-3D pipeline has no good answer at all. As a Synthetic Training Data case study it is also notable for using a VLM (Qwen2.5-VL-7B) to filter the asset bank for transparency/reflectivity before any rendering happens, which converts a curation problem into a captioning problem.