Elastic Token Compression for Pixel-Space Diffusion Transformers

TL;DR

ETC (Region Token Interface, “RTI”) is a token-count reduction recipe for pretrained pixel-space diffusion transformers. Pixel-native DiTs spend a full token on every patch at every layer and timestep, but their middle-block features are redundant wherever the image is flat, and that redundancy sits in connected, content-shaped regions. ETC orders patches along a Hilbert curve — where any contiguous run is guaranteed to be a connected 2D region — cuts the curve where features change most, and pools each run into a single “region token.” A short LoRA-style fine-tune on MiniT2I with the region count drawn at random per step gives one checkpoint that serves every token budget: matches dense quality at 2.0× the speed and stays close at 2.6×. Positions the recipe as reintroducing, inside the diffusion model, the content-adaptive compression that pixel-space diffusion gave up by dropping the VAE.

Key claims

• Middle-block tokens of a pretrained pixel-space DiT are redundant wherever the image is flat, and the redundancy occupies connected, content-shaped regions — so a compression scheme that exploits it needs tokens with the same geometry [§Abstract; §Method probing analysis].
• Cutting a Hilbert ordering of the patches gives connected region tokens by construction: consecutive Hilbert positions are always image neighbours, so any contiguous run on the curve is a connected 2D region whose size and shape follow content [§Method].
• Existing token reductions each lose part of the required structure: similarity merging scatters its groups across the image, latent-array bottlenecks discard position, and skipping deletes what it should summarize [§Abstract; §Related work].
• Region-count randomization during fine-tuning yields a single checkpoint that serves every inference budget elastically — no separate model per compression ratio [§Method].
• ETC leads prior reduction methods at matched token budgets, matches dense-token quality at 2.0× speed and stays close at 2.6× speed on the MiniT2I pixel-space backbone [§Abstract; §Results].
• The compression pattern is timestep-dependent: at high noise, regions are meaningless and stay small; as the image forms, regions grow large over flat areas and stay fine over detailed ones — tokens spread where the detail emerges [tweet thread, matching §Method visualization].

Method

The base model is MiniT2I (MiniT2I: Pixel-Space Text-to-Image with MM-JiT at 258M Parameters) — a 258M pixel-native MM-JiT — chosen because it is the smallest filed instance of the JiT-family pixel DiT and has no VAE to absorb low-level redundancy for it. The recipe has three parts:

1. Hilbert reordering. The 2D patch grid is walked along a Hilbert space-filling curve. This turns any 2D grouping problem into a 1D cut problem: contiguous runs on the curve are guaranteed to be connected regions in the image, and cut points are 1D positions.
2. Feature-change cuts. Cuts are placed at positions in the reordered sequence where the network’s own features change most between adjacent patches. Flat areas produce long runs (one region token summarizing many patches); textured areas produce short runs (regions close to single patches).
3. Region Token Interface fine-tune. A lightweight adapter converts each run to one region token and back, and the base model is briefly fine-tuned end-to-end with the target region count drawn uniformly at random per step. One checkpoint therefore serves the whole 1×–2.6× speedup range.

At high noise the reduction is nearly a no-op (regions are meaningless before content emerges); at low noise the reduction concentrates in flat sky/background and preserves resolution over faces / high-frequency detail.

Results

• On MiniT2I (258M pixel-space DiT), ETC matches dense-token quality at 2.0× inference speed and remains close to dense at 2.6× [§Abstract; §Results].
• Beats prior token-reduction baselines (similarity merging, latent bottleneck, token skipping) at matched token budgets [§Abstract; §Results].
• Elastic behavior: a single checkpoint covers the full compression range without retraining [§Method].
• Code and weights open-sourced [§Abstract].

Precise FID / GenEval numbers, exact epoch/GPU budgets for the fine-tune, and the ImageNet-vs-T2I decomposition were not captured in the fetched abstract or thread — they will need to be read off the PDF for a proper claim comparison against Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation.

Why it’s interesting

ETC sits at the intersection of two lively strands on the wiki and adds a data point neither had.

Relative to Pixel-space diffusion, it accepts the community’s recent conclusion — PixelGen: Improving Pixel Diffusion with Perceptual Supervision, PixelDiT: Pixel Diffusion Transformers for Image Generation, Asymmetric Flow Models — that pixel-native training is competitive with latent diffusion once the loss/parameterization/architecture are right. Its contribution is orthogonal: even after those fixes, pixel-space DiTs are inference-inefficient in a way latent DiTs aren’t, because they lack the VAE’s content-independent compression step. ETC reintroduces compression inside the model, and unlike a VAE the compression adapts per-image and per-timestep.

Relative to Diffusion training efficiency — and specifically Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation, which is the closest precedent for spatial-token-count reduction as an efficiency axis — ETC differs on three points: (a) the token layout is content-derived (Hilbert cuts on feature-change signal) rather than gaze/saliency-mask-derived; (b) regions are guaranteed connected by construction rather than by mask design; (c) the elastic-budget fine-tune serves every compression ratio from one checkpoint rather than post-training a fixed ratio. Foveated Diffusion also required post-training to avoid duplicated-object artifacts from naive mixed-resolution denoising; ETC needs a fine-tune too, so both agree that this efficiency lever is not training-free.

The Hilbert-curve trick — reduce a 2D geometry-preserving grouping problem to a 1D cut problem — is unusually clean and may transfer to other content-adaptive spatial operations (attention windowing, KV-cache layout, patch dropout).

See also

• Pixel-space diffusion — this concept; ETC is a new (inference-efficiency) angle on it.
• Diffusion training efficiency — this concept; ETC contributes a spatial-token-count-reduction datapoint distinct from Foveated Diffusion.
• Diffusion serving optimization — this concept; the 2.0×–2.6× headline is an inference-time win, not a training-time one.
• Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation — closest sibling: spatial token reduction via post-training on a pretrained DiT, saliency-mask-based rather than Hilbert-cut.
• MiniT2I: Pixel-Space Text-to-Image with MM-JiT at 258M Parameters — the MiniT2I backbone ETC fine-tunes.
• PixelDiT: Pixel Diffusion Transformers for Image Generation — PixelDiT also does per-patch pixel-token compaction, but as an architectural choice during pretraining rather than a post-hoc content-adaptive scheme.
• Asymmetric Flow Models — parameterization-side pixel-space efficiency recipe compatible in principle with ETC’s token-reduction recipe.
• Don't Look Twice: Faster Video Transformers with Run-Length Tokenization — Run-Length Tokenization for video, another example of content-adaptive contiguous-token pooling.