Skip to content
Bud's Wiki

Level-of-Token Diffusion

TL;DR

Section titled “TL;DR”

Level-of-Token (LoT) Diffusion generalizes mixed-resolution token layouts for diffusion transformers from the specific case of foveation to arbitrary multiresolution token layouts derived from any spatial detail cue. Tokens are rectangular patches of varying sizes and shapes; finer tokens are placed where detail is needed and coarser tokens elsewhere, specified at inference as a Level-of-Token layout. Pretrained diffusion transformers are adapted to such layouts via a patch-wise asymmetric flow parametrization plus multiresolution token embeddings that preserve full-resolution flow prediction at every denoising step while only processing the reduced token sequence. The paper demonstrates layouts derived from semantic masks, bounding boxes, texture variance, depth-of-field cues, and agentic plans across image and video generation, with speedups determined by the per-image token budget.

Key claims

Section titled “Key claims”
  • A single trained diffusion model can serve arbitrary multiresolution token layouts at inference — layouts derived from semantic masks, bounding boxes, texture variance, depth-of-field, or agentic plans all work with the same checkpoint [§Abstract].
  • A patch-wise asymmetric flow parametrization keeps full-resolution flow prediction at every denoising step while the network only processes the reduced token sequence [§Abstract].
  • Multiresolution token embeddings are the vehicle by which rectangular patches of varying sizes and shapes are consumed by a transformer whose attention and FFN were pretrained on a uniform grid [§Abstract].
  • Image and video generation both benefit from LoT layouts, with inference speedup determined by the layout’s token budget rather than by a fixed compression ratio [§Abstract].
  • Adaptation preserves the pretrained generative prior — the method is a fine-tune of an existing DiT rather than retraining from scratch [§Abstract].

(The abstract is the extent of the fetched content; headline numbers — token-budget-to-speedup curves, image/video FID or VBench, user-study results — are not captured here and will need to be read off the PDF for a quantitative comparison against the sibling papers in ## Why it's interesting.)

Method

Section titled “Method”

LoT Diffusion takes a pretrained diffusion transformer and three ingredients:

  1. Level-of-Token layout. A pre-generation specification of per-region patch granularity, derived from any upstream cue — a semantic segmentation mask, a bounding box, a texture-variance heuristic, a depth-of-field map, or an agentic plan. The layout is a multiresolution token grid: fine (small) tokens where detail is anticipated, coarse (large rectangular) tokens where it isn’t.
  2. Multiresolution token embeddings. Tokens of different sizes/shapes are embedded into the pretrained transformer’s token space without changing attention or FFN shapes — the network sees a reduced sequence of mixed-granularity tokens.
  3. Patch-wise asymmetric flow parametrization. The flow-matching velocity target is reparametrized per-patch so that even though the network processes a reduced token sequence, the prediction at each denoising step is equivalent to a full-resolution flow prediction. This is what preserves the pretrained generative prior under the layout change.

The adaptation is a fine-tune of the pretrained DiT; the layout is swapped in at inference time per generation.

Results

Section titled “Results”

The abstract claims favorable quality-efficiency tradeoffs across image and video generation, with significant speedups determined by the layout’s token budget, but the fetched content does not include specific numbers. The project page is referenced in the abstract and would be the next place to check for FID / VBench / user-study results against uniform-grid baselines and against the sibling papers below.

Why it’s interesting

Section titled “Why it’s interesting”

LoT Diffusion is the direct generalization of Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation by two of its authors (Chao and Yariv, with Wetzstein again senior). Foveated Diffusion handled one specific layout family — binary HR/LR masks around a gaze point — and already showed 2× image / 4× video speedup at perceptual parity. LoT drops the “foveal mask” constraint entirely: any spatial cue that can be rendered as a multiresolution token layout works, including non-perceptual ones (semantic masks, bounding boxes, texture variance, agentic plans). That matters because the two main criticisms of Foveated Diffusion — it needed a gaze signal, and the only non-interactive deployment path was saliency-guided — are directly addressed here.

The paper also contrasts cleanly with the two other spatial-token-reduction datapoints on the wiki. Elastic Token Compression for Pixel-Space Diffusion Transformers derives its layout internally from middle-block feature-change signal and uses Hilbert-curve cuts to guarantee connected regions; LoT accepts the layout from any external source and lets the user pre-plan it. Accelerating Vision Transformers with Adaptive Patch Sizes (APT) does the same “variable patch sizes” trick on the encoder side of a ViT using pre-network entropy; LoT is its DiT-generation-side counterpart, with the structural difference that generation requires the asymmetric flow reparametrization to keep the prior intact (an encoder-side ViT needs no such step). Reading all four together — Foveated, ETC, APT, and now LoT — the spatial-token-reduction axis the wiki has been tracking is now mature enough that “cue-agnostic layout + flow reparametrization” is becoming the generic recipe.

For Luma specifically, the “agentic plans” layout is the one worth watching: a planner that pre-emits a per-shot detail map (characters finer, sky coarser) and a DiT that honors it is a cleaner interface for interactive video generation than either gaze-driven foveation or purely learned saliency, and preserves per-user control over where compute lands.

See also

Section titled “See also”