HDR: Hierarchical Denoising for Multi-Step Visual Reasoning

TL;DR

HDR (Hierarchical Denoising for Visual Reasoning) is a unified video-generation framework that organizes latents into a tree-structured temporal hierarchy so a causal video DiT reasons coarse-to-fine before emitting a streaming output. Upper layers of the tree hold uncertain global hypotheses; lower layers refine them into concrete visual states; a sparse hierarchical attention pattern (SHAP) restricts each token to local, parent-level, and first-frame context, keeping temporal attention cheap while letting global plans flow into per-frame generation via a shared KV cache. Evaluated on a level-stratified benchmark of six long-horizon reasoning tasks (maze, Tower of Hanoi, one-line drawing, sliding puzzle, Sokoban, water pouring), HDR raises overall success from 34.22 to 60.29 and average progress from 76.00 to 89.56 vs. a streaming autoregressive diffusion baseline while retaining 0.70s-per-latent low-latency streaming.

Key claims

Existing streaming autoregressive video diffusion is efficient but lacks reasoning ability; bidirectional diffusion allows global revision but incurs high inference cost due to dense frames in fixed-sequence denoising — a gap HDR closes by injecting a hierarchical latent tree into the causal generation process [§Overview].
Video latents are organized as a tree-structured hierarchy across temporal resolutions; during inference tokens are generated in coarse-to-fine autoregressive order (upper layers = global hypotheses, lower layers = concrete visual states, final layer = streamed output) [§Method, hierarchy figure].
The Sparse Hierarchical Attention Pattern (SHAP) restricts each token’s temporal attention to fixed local, parent-level, and first-frame contexts, keeping attention sparse while allowing coarse-planning-layer information to flow into fine visual generation through a shared KV cache [§SHAP].
A level-stratified multi-step visual-reasoning benchmark covering six long-horizon tasks (maze navigation, Tower of Hanoi, one-line drawing, sliding puzzle, Sokoban, water pouring) is introduced, explicitly including out-of-distribution cases and requiring logical consistency across generated trajectories rather than local visual plausibility [§Benchmark].
On this benchmark, HDR raises overall success 34.22 → 60.29 and average progress 76.00 → 89.56 vs. a streaming AR-diffusion baseline while keeping 0.70s-per-latent low-latency streaming [§Results].
The hierarchy improves robustness under reduced denoising budgets and limited training data; HDR also transfers to real-robot maze interaction after fine-tuning on only 50 real-world robot maze videos [§Results, physical-world maze].

Method

HDR takes a causal video diffusion transformer and reorganizes its latent factorization: instead of a flat sequence of per-frame latents denoised uniformly, latents are arranged as a temporal-resolution tree. Each internal node summarizes a longer time span at lower resolution; each leaf corresponds to one output-latent frame. Generation proceeds top-down: the model first samples the coarsest layer (a small number of long-horizon “planning” latents), then conditions each finer layer on its parent, and finally streams out the leaf layer as the visible video. Because the same forward pass emits leaves in causal order once the upper layers are fixed, streaming latency stays close to the underlying AR baseline.

The attention structure is the second half of the recipe. SHAP restricts each token’s temporal attention to (a) a fixed local window at its own level, (b) its parent(s) at the level above, and (c) a first-frame anchor. This gives a sparsity pattern that grows only logarithmically in effective context — coarse-plan information can reach any leaf via at most one hop up the tree — and permits a shared KV cache across levels so that upper-layer keys are computed once and reused by every finer descendant. The benchmark deliberately covers six tasks whose ground-truth trajectories require global logical consistency (a maze path must actually reach the goal; a Hanoi solution must respect the stacking constraint; a one-line drawing must not cross itself), and includes OOD splits so a baseline cannot pass by memorizing training layouts. Reported physical-world results come from fine-tuning HDR on 50 robot-maze videos, i.e. the same architecture transfers to real interaction with very little in-domain data.

Results

Overall success rate across the six-task benchmark: HDR 60.29 vs. streaming AR-diffusion baseline 34.22 (+26.07 absolute) [§Results, overall table].
Average progress: HDR 89.56 vs. baseline 76.00 (+13.56 absolute) [§Results].
Streaming latency: 0.70s per latent (retained from the AR baseline) [§Results].
Robustness: hierarchy improves accuracy under reduced denoising-step budgets and under limited training data (specific numbers not surfaced in the project-page copy).
Physical-world transfer: fine-tuning on 50 real-world robot-maze videos transfers HDR to real robot interaction [§Physical-world maze].
Qualitative demos across all six task families (Maze 020-054, Hanoi 066-113, One-line 164-189, Sliding 195-242, Sokoban 255-301, Water 315-344) are provided on the project page.

Why it’s interesting

HDR sits exactly on the collision course between two lines already tracked on the wiki. The first is Demystifying Video Reasoning‘s Chain-of-Steps finding — that pretrained video diffusion already reasons along the denoising trajectory, holding multiple candidate solutions in early steps and consolidating late — evaluated on the same maze/Hanoi/puzzle families HDR benchmarks. HDR’s tree-structured latents and coarse-to-fine sampling schedule can be read as an architectural way to spend more compute on exactly this CoS mechanism: upper-layer latents are the “early denoising step” where multiple plans coexist, lower-layer latents are the late-step consolidation, and SHAP makes the plan → detail information flow explicit rather than emergent. It also complements Thinking with Modalities‘s substrate (b) — diffusion-native parallel exploration — by proposing a concrete way to route that native exploration at inference time without sacrificing streaming.

The second collision is with the Autoregressive Video Generation streaming lineage — Self Forcing, Rolling Forcing, Context Forcing, Causal Forcing, PFP, LingBot-VA — which trades global revision for real-time throughput and struggles past ~10s of coherent generation. HDR argues the missing ingredient is not a bigger memory or a better distillation but a hierarchical plan fixed at generation time; SHAP’s local + parent + first-frame recipe overlaps directly with the attention-sink + local-window + bounded-RoPE machinery in Rolling Forcing: Autoregressive Long Video Diffusion in Real Time but adds an explicit vertical planning axis. Composing HDR’s hierarchy with e.g. Context Forcing’s Slow-Fast Memory KV cache is the obvious follow-up. Finally, the 50-video robot transfer connects to World Foundation Models and the VLA line — coarse-to-fine plans are cheaper to condition on real observations than dense per-frame latents.