Autoregressive-to-Diffusion Vision Language Models (A2D-VL)

TL;DR

A2D-VL adapts a pretrained autoregressive VLM (Qwen2.5-VL 7B) into a parallel-decoding diffusion VLM by finetuning on a block-masked-diffusion objective with only 400K visual-QA pairs — ~30× less than LLaDA-V 8B’s ≥12M. Two adaptation tricks are load-bearing: block size annealing (gradually grow the diffusion prediction window from 1 token up to the target 8) and noise level annealing (position-dependent masking inside each block, easy-left/hard-right early in training, uniform later). The result preserves the base AR VLM’s capabilities while unlocking a tunable speed-quality trade-off via the confidence threshold; A2D-VL outperforms LLaDA-V 7B on detailed image captioning (BERTScore vs GPT-4o/4V/Gemini-1.5-Pro references) and on MMMU-Pro chain-of-thought, where LLaDA-V degrades past 512 tokens but A2D-VL holds up to 16K. Because A2D-VL trains with block-causal attention, it gets exact KV caching for free, unlike LLaDA-V’s approximate-caching-with-periodic-recompute.

Key claims

Adapting an existing AR VLM (Qwen2.5-VL 7B) to a parallel diffusion VLM requires ~30× less training data than training a diffusion VLM from scratch — 400K VQA pairs for A2D-VL 7B vs ≥12M for LLaDA-V 8B [Limitations of prior diffusion VLMs / Efficient training].
Block size annealing — starting with single-token (AR-equivalent) blocks and gradually growing the diffusion prediction window to 8 tokens — is critical for retaining the base AR model’s capabilities during AR→diffusion conversion [Fig. 3 ablation].
Noise level annealing — masking left-most (easier) tokens more frequently early in finetuning, then transitioning to uniform within-block masking — is independently load-bearing in the same ablation [Fig. 3].
Block-causal attention (attend to previous blocks + within-current-block) supports exact KV caching of completed blocks, vs LLaDA-V’s bidirectional attention which needs approximate caching with periodic recomputation every 32 steps [Limitations / KV caching support].
Decoding in blocks of 8 tokens (rather than full-sequence diffusion as in LLaDA-V / LaViDa / MMaDA) improves both response quality and length generalization, particularly for long-form outputs like chain-of-thought reasoning and detailed captions [Limitations / Quality degradation in long-form responses].
A2D-VL beats LLaDA-V on detailed image captioning, scored by BERTScore against GPT-4o / GPT-4V / Gemini-1.5-Pro references at ≤512-token outputs [Speed-quality results / Detailed image captioning].
A2D-VL maintains MMMU-Pro accuracy with chain-of-thought up to 16K tokens; LLaDA-V’s accuracy degrades past 512 tokens so its evaluation is capped there [Speed-quality results / Chain-of-thought reasoning].
The training data mix is 100K reasoning traces distilled from Qwen2.5-VL 72B plus 50K MAmmoTH-VL samples plus the remaining VQA, framed as a small high-quality reasoning corpus rather than mass instruction-tuning data [Limitations / Improved quality in long-form responses].
The confidence threshold is the single user-facing knob trading parallelism (tokens-per-step) for quality; throughputs are reported per-device on 8× H100 [Fig. 4].

Method

A2D-VL takes Qwen2.5-VL 7B as the base AR VLM and finetunes it under the block diffusion framework (Arriola et al. 2025 / ref [8]), which interpolates between AR (block size 1) and full diffusion (block size = sequence length). Mask tokens noise the input; the model predicts originals. The target block size is 8 tokens. Attention is block-causal: each block sees prior blocks and all positions within itself, but no future blocks — this is what unlocks exact KV cache reuse for finalized prefix blocks.

Two adaptation curricula run simultaneously during finetuning. Block size annealing starts with block size 1 (recovering Qwen2.5-VL’s original AR behavior on day one of finetuning) and grows the block to 8 over the schedule, so the loss landscape moves gradually rather than jumping straight to a fully bidirectional in-block target. Noise level annealing uses position-dependent within-block mask probabilities: early in training the left positions (closer to context, easier to predict) are masked more often than right positions; the distribution flattens to uniform over training, ending in standard any-order block diffusion. The Fig. 3 ablation removes each independently and shows both are needed to preserve base-model benchmark performance on the 50K-pair finetune.

At inference, the model fills one 8-token block at a time. Within a block, the masked-diffusion sampler iteratively unmasks tokens; the confidence threshold (a la Fast-dLLM / LLaDA-V) determines how many tokens can be unmasked per step. A high threshold (~90%) sequentializes the decode and matches AR quality; a lower threshold (~30%) commits multiple tokens per step at some quality cost. Once a block is finalized, its KV is cached exactly and never recomputed.

Results

Training data: 400K visual-QA pairs total, vs LLaDA-V 8B’s ≥12M [Limitations / Efficient training]. Composition includes 100K Qwen2.5-VL-72B-distilled reasoning traces + 50K MAmmoTH-VL.
Throughput: measured on 8× H100 as per-device averages, with confidence-threshold sweep producing the speed-quality curve in Fig. 4. Parallelism factor reported as average concurrent tokens per generation step.
Detailed image captioning: A2D-VL captions score higher BERTScore against GPT-4o / GPT-4V / Gemini-1.5-Pro references than LLaDA-V captions [Speed-quality results].
MMMU-Pro CoT: A2D-VL consistently beats LLaDA-V; A2D-VL evaluated up to 16K-token CoT, LLaDA-V capped at 512 due to long-form degradation [Speed-quality results].
Multiple-choice VQA (no CoT): A2D-VL competitive with the AR Qwen2.5-VL baseline; baseline numbers from published reports [Fig. 5].
Ablation evidence: Fig. 3 shows removing either block-size annealing or noise-level annealing meaningfully drops benchmark performance on a 50K-pair finetune of Qwen2.5-VL 7B.

Why it’s interesting

This is the first filed diffusion-LM result on the wiki that targets the vision-language setting head-on. It directly answers one of the Diffusion Language Models open questions (“Native multi-modality — none of the filed dLLMs handle vision/audio natively at frontier quality”) with a concrete recipe: adapt a frontier AR VLM rather than train a diffusion VLM from scratch, and use block-causal attention so KV caching works without approximation. It complements Introspective Diffusion Language Models (I-DLM converts Qwen3-8B to diffusion for text-only) and WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference (WeDLM does the same with topological reordering) — A2D-VL is the multimodal sibling, with block-causal attention as a third route to making a dLLM serve under existing AR-serving infra. It also contrasts with The Design Space of Tri-Modal Masked Diffusion Models‘s from-scratch tri-modal masked-diffusion approach, which establishes that masked-diffusion UMMs want more tokens-per-parameter than AR — Runway’s pragmatic answer is “skip the from-scratch budget by adapting an AR VLM instead.”

Worth noting per @amit’s posting note: this is an intern project (Marianne Arriola, who is also first author on the block diffusion paper [ref 8]) and the residual accuracy question for dLLMs vs AR remains — A2D-VL is competitive with its Qwen2.5-VL base rather than strictly dominating it, and Fig. 5 reports VQA gains over LLaDA-V (the prior diffusion-VLM SOTA) rather than over the AR base.