Accelerating Vision Transformers with Adaptive Patch Sizes

TL;DR

Adaptive Patch Transformers (APT) replace the uniform-patch grid of a ViT with a content-aware quadtree: low-entropy image regions get large patches (e.g. 64×64), high-entropy regions get small ones (16×16), and all patches are projected into the same token-embedding dimension via a zero-initialized MLP combined with downsampled sub-patch embeddings. The patchification is computed pre-network from pixel-space entropy, so it adds essentially no overhead and converges in as little as 1 epoch from any pretrained ViT checkpoint. Reported wall-clock speedups: 40% on ViT-L at 336², 50% on ViT-H, ~30% on COCO object detection and ADE20K semantic segmentation, ~22% on LLaVA-1.5 VQA, all at matched downstream accuracy.

Key claims

Pre-network entropy-based patchification beats layer-level token merging (ToMe, EViT, DTEM) on both throughput and accuracy, in part because layer-level mergers cannot use FlashAttention while APT can [§4.1, §4.2, Fig. 4].
A zero-initialized MLP that combines a downsampled large patch with embeddings from its constituent sub-patches lets a pretrained ViT match its original accuracy with ≤1 epoch of fine-tuning, vs. ≥50 epochs for DynamicViT/MS-ViT-style learned-gating approaches [§3.2, §4.2, Table 2].
Speedup grows with both resolution and model size: ViT-L at 336² gains ~2× the speedup of ViT-L at 224² because attention dominates compute more at higher token counts [§4.2, Table 1].
APT applied to LLaVA-1.5-7B / 13B with only the new components trained for 5% of the original fine-tuning budget exceeds the original LLaVA’s VQA scores on GQA, POPE, MMB, MMVet while running 22–23% faster [§4.3, Table 4].
COCO object detection on EVA-02-L at 1536² gets 30% wall-clock speedup with mAP 62.07 vs. baseline 62.28 (-0.21 mAP) using a ViTDet-style head with window attention [§4.3, Table 5].
ADE20K semantic segmentation on EVA-02-L gets 11% speedup with mIoU 60.01 vs. baseline 60.05 at 640² — APT is content-aware enough to preserve pixel-level boundary accuracy despite assigning large patches to homogeneous regions [§4.3, Table 6].
Random masking at the matched token-reduction fraction underperforms APT by 2–3 mAP / mIoU points on dense tasks and by 1–4 points on VQA, showing that content-awareness has measurable value beyond raw token-count reduction [Table 4, Table 5].
Sequence packing with block-diagonal attention masks (NaViT-style) is what makes variable per-example token counts compatible with FlashAttention — without it, throughput gains evaporate [§3.3].
Zero-initialization of the residual MLP beats non-zero and plain-residual initializations both with and without training; matches the ControlNet finding that zero-init lets new capabilities be added without injecting harmful noise into a pretrained model [Table 7].
The entropy threshold τ is the main tunable; speed-accuracy trade-off curves are reported in Fig. 4 and additional thresholds in the appendix [§4.4, Fig. 4].

Method

APT modifies only the patchification stage of a standard ViT. First, the image is recursively partitioned into a quadtree of patch sizes — the largest scale P₀ (e.g. 64×64) is evaluated first, and any patch whose pixel-intensity Shannon entropy exceeds a per-scale threshold τₖ is recursively split into 4 sub-patches at the next scale Pₖ₊₁ = Pₖ/2. The recursion stops at the smallest scale P (e.g. 16×16). Entropy is computed from a binned histogram of pixel intensities and serves as a proxy for compressibility — homogeneous regions have low entropy and are safely represented by a single token.

Second, every retained patch must produce an embedding of the standard token dimension. Sub-patches of size P are embedded with the original ViT’s linear projection f. For a larger patch of size Pₖ that subsumes a set S of sub-patches, APT computes f(resize(patch, P)) + ZeroMLP(downsample_k(Σᵢ∈S f(subᵢ))) — the first term gives an immediately-usable resized-patch token, the second term is a zero-initialized residual that lets the network gradually incorporate high-resolution detail from the larger patch’s constituents. ZeroMLP is a single linear layer initialized to zero, following ControlNet.

Third, because different images now produce different token counts, APT uses NaViT-style sequence packing: per-batch tokens are concatenated into one sequence and a block-diagonal attention mask isolates examples. This is natively supported by FlashAttention / xFormers so adds no inner-loop overhead. For dense prediction tasks that need a rectangular feature map, larger-patch tokens are replicated 2^(2k) times to fill the grid they came from, yielding a fully-differentiable upsampleable feature map.

Results

ViT-L / 336², ImageNet: 40% throughput speedup at matched 86.0 → 86.0 top-1 accuracy after full fine-tuning from MAE [Table 1].
ViT-H / 336², ImageNet: 50% throughput speedup [Table 1].
Short fine-tune (1 epoch from a fine-tuned ViT-L/14): APT reaches 87.98 vs. baseline 88.15 top-1 with no training; closes to 88.13 with 1 epoch; beats Residual/NonZero alternatives at both checkpoints [Table 2, Table 7].
LLaVA-1.5-7B VQA: 22% throughput speedup; GQA 77.9 vs. 78.5 baseline (Δ-0.6), POPE 61.4 vs. 61.0 (Δ+0.4), MMVet 30.8 vs. 30.7 (Δ+0.1) [Table 4]. LLaVA-1.5-13B sees similar 23% speedup with most metrics within ±1 point.
COCO detection, EVA-02-L 1536²: 30% speedup; mAP 62.07 vs 62.28 (Δ-0.21), AP50 80.64 vs 80.80 [Table 5].
ADE20K segmentation, EVA-02-L 640²: 11% speedup; mIoU 60.01 vs 60.05 (Δ-0.04) [Table 6].
Overhead: APT’s input rearrangement adds ~5–7% inference-time overhead relative to baseline ViT before the token-reduction speedup is accounted for; net throughput gain still dominates at high resolution [Table 7].
Qualitative patchification (Fig. 5): dark backgrounds and blurry backdrops uniformly get the largest patch, faces and in-focus objects get the smallest, with a cityscape exceeding a cartoon’s total patch count by ~3× — confirming the heuristic produces semantically reasonable allocations even though it’s purely entropy-driven.

Why it’s interesting

APT is the image-encoder sibling of the same first author’s earlier video work Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (RLT): both exploit pixel-space content redundancy as a pre-network drop, both use NaViT-style packing to preserve FlashAttention, both add length / size encodings to communicate the reduction to the transformer, and both achieve their wall-clock speedups without the in-network gating that hampered DynamicViT/A-ViT. Read together, RLT + APT make a clean argument that “content-aware variable-length tokenization” should be the default pre-processing step for ViTs in both spatial and temporal axes, with the merge primitive being pixel-space entropy/L1 rather than a learned scorer.

APT also contrasts cleanly with Single-pass Adaptive Image Tokenization for Minimum Program Search (KARL), which is the generative-tokenizer counterpart: KARL learns to predict per-image token count under a halting head trained with Upside-Down RL and is intended as a tokenizer for downstream diffusion; APT is a forward-pass-only heuristic for ViT encoders. The two papers agree that fixed-rate patchification is wasteful and disagree on whether the budget-allocation should be learned (KARL) or computed (APT). For Luma’s ViT-based perception stacks (VLMs, world-model encoders, perception backbones in VLAs), APT’s “1 epoch on any pretrained checkpoint” property makes it nearly free to try, while KARL is the right framing if the tokenizer is being trained from scratch.

Finally, the spatial-token-reduction lever now has three filed datapoints — APT (encoder-side, pre-network), Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation (DiT-side, post-training LoRA), and KARL (tokenizer-side, learned halting) — that together suggest “fewer tokens per forward pass” is becoming a first-class efficiency axis distinct from step-distillation, latent-redesign, and external-feature alignment.