FastVLM: Efficient Vision Encoding for Vision Language Models

TL;DR

FastVLM (CVPR 2025, Apple) is a vision-language model designed for on-device deployment that attacks VLM latency at the vision-encoder side rather than via post-hoc token pruning. The authors introduce FastViTHD, a 5-stage hybrid convolution+transformer vision backbone (125M params) that produces ~4× fewer tokens than ViT-L/14 at matched accuracy and runs the self-attention layers only on heavily downsampled tensors. They study the (resolution, LLM-size) Pareto frontier directly on hardware time-to-first-token (TTFT), arguing that the optimal accuracy/latency knob is input resolution itself, not tile counts or learned token reducers. The 0.5B variant matches LLaVA-OneVision-0.5B with 85× faster TTFT and a 3.4× smaller vision encoder; the 7B variant outperforms Cambrian-1-8B at 7.9× faster TTFT.

Key claims

VLM latency optimization should consider both vision-encoder time and LLM prefilling time as a single TTFT budget, since visual tokens enter the LLM context and quadratically affect prefilling cost [§1, §3.2.1].
Hybrid convolution+transformer encoders (FastViT, FastViTHD) Pareto-dominate ViT-based encoders for VLMs: at matched visual-token count (576) and matched LLM, FastViT at 768² beats ViT-L/14 at 336² on text-rich benchmarks (TextVQA 62.3 vs 58.2, DocVQA 34.4 vs 28.1) while being faster to encode (34.5ms vs 127.4ms) [Table 1].
Multi-scale features pooled via depthwise convolutions across stages of a hierarchical backbone give a small but consistent VLM accuracy lift over single-stage features [§3.1.1, Table 2].
FastViTHD adds a 5th stage with self-attention layers operating on tensors downsampled by 32× per side (vs 16× in ViTamin) and 64× for the widest MLPs, producing 4× fewer tokens than typical hybrids while staying smaller than ViT-L/14 (125M vs 304M) [§3.2, Fig. 2].
Under CLIP pretraining on DataCompDR-1B, FastViTHD matches ViT-L/14 in average performance across 38 zero-shot tasks while being 2.4× smaller and 6.9× faster [Table 3].
For VLM tasks, FastViTHD matches ConvNeXT-XXL at 6.8× smaller size and 3.3× faster encoding [Table 4].
Native input-resolution scaling beats AnyRes-style tiling for the accuracy/latency tradeoff up to 1024² — tiling only helps above 1536² where memory bandwidth dominates [§3.2.2, Fig. 6].
FastViTHD with 16 visual tokens at 256² outperforms ViT-L/14 + state-of-the-art token-pruning methods (PruMerge+, VisionZip, FastV, SparseVLM, DynamicLLaVA) at matched or higher token budgets across GQA, TextVQA, POPE, SeedBench [Table 5].
FastVLM-0.5B at 1024² (256 visual tokens) reaches comparable accuracy to LLaVA-OneVision-0.5B at 1152² (7290 tokens) with 85× faster TTFT (166ms vs 14124ms) and a 3.4× smaller vision encoder [Table 6, R2 vs R3].
FastVLM-7B/Qwen2 at 1024² (256 tokens) hits 75.1 SeedBench, 73.3 DocVQA, 46.2 MMMU with 641ms TTFT — outperforming ConvLLaVA, LLaVA-NeXT, MM1, VILA, ShareGPT4V, and LLaVA-1.5 7B baselines while being faster [Table 6, R28].
The Pareto-optimal (Resolution, LLM) curve consists of varying LLM sizes — pairing very high resolution with a small LLM is suboptimal because TTFT becomes vision-encoder-bound and the small LLM cannot effectively consume the tokens [§3.2.1, Fig. 4–5].

Method

The FastVLM stack is a standard VLM topology — vision backbone → linear projector → decoder-only LLM (Qwen2 or Vicuna). The contribution is the backbone. FastViTHD takes a 5-stage hierarchical design: stages 1–3 use RepMixer (convolutional) blocks at decreasing spatial resolution; stages 4–5 use multi-head self-attention blocks operating on tensors downsampled by 32× and 64× respectively. Depths are [2, 12, 24, 4, 2] with embedding dims [96, 192, 384, 768, 1536] and MLP expansion 4.0 (125.1M params). A downsampling layer precedes stage 4 specifically to keep the attention layers off the larger feature maps that dominate FLOPs in 4-stage hybrids like ViTamin. The encoder is CLIP-pretrained on DataCompDR-1B following the MobileCLIP recipe. Multi-scale features are pooled from stages 3, 4, and 5 via 2D depthwise convolutions and concatenated before the projector. Visual instruction tuning follows the LLaVA-1.5 recipe (pretraining on a 15M-image dataset, instruction tuning on 1.1–11.9M samples). Input resolution is set natively per (model size, latency budget) rather than via AnyRes tiling. Evaluation measures end-to-end TTFT on actual hardware (M1-class GPUs) including both vision-encoder forward and LLM prefilling.

Results

FastVLM-0.5B/Qwen2 at 1024² with 15M+1.1M training data: 166ms TTFT, 61.6 GQA, 87.4 POPE, 65.6 SeedBench, 77.0 VQAv2, vs LLaVA-OneVision-0.5B at 1152²: 14124ms TTFT, 65.5 SeedBench, 70.0 VQAv2 [Table 6, R2–R3] → 85× TTFT speedup at parity.
FastVLM-1.5B/Qwen2 at 1024² with 15M+11.9M training: 233ms TTFT, 90.5 SQA, 87.6 POPE, 72.9 LLaVA-Bench-W, 80.8 VQAv2, 72.5 DocVQA [Table 6, R11].
FastVLM-7B/Qwen2 at 1024² with 15M+11.9M training: 641ms TTFT, 96.0 SQA, 74.8 TextVQA, 87.2 POPE, 73.5 LLaVA-Bench-W, 45.4 MM-Vet [Table 6, R28–R39].
Vs ConvLLaVA-7B at 1024²: FastVLM uses fewer visual tokens (256 vs 256), runs 2× faster, and beats it on every reported benchmark (TextVQA, DocVQA, MMMU, SeedBench) [Table 6, R24–R27].
Vs ViTamin-L (a competing hybrid): FastViTHD is 2.7× smaller, 5.6× faster, with better average retrieval performance [§3.2, Table 3].

Why it’s interesting

This is one of the clean datapoints for the “VLM latency = vision-encoder + LLM-prefill” co-design framing that connects to the more LLM-side work in LLM Inference Efficiency — FastVLM attacks the same TTFT budget from the front of the pipeline that speculative-decoding and KV-cache compression attack from the back. It complements Don't Look Twice: Faster Video Transformers with Run-Length Tokenization, which reduces video-ViT tokens via run-length pre-network encoding on the discriminative side; FastVLM’s claim is the more aggressive one — token pruning on top of ViT is the wrong abstraction and a hierarchical encoder native-scaled to the target resolution dominates at every operating point. The same Apple/Pouransari group also released the companion MobileCLIP2: Improving Multi-Modal Reinforced Training eight months later, which is the CLIP-pretraining recipe FastViTHD is built on. Worth tracking against Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation, which argues the entire vision encoder is the wrong abstraction and pixel embeddings + a unified transformer work better — FastVLM is the strongest “no, the encoder is fine, just make it hierarchical” counter-position.