DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

TL;DR

DiffuCoder is a 7B masked diffusion LLM for code, trained by Apple on 130B tokens of code with continual adaptation from Qwen2.5-Coder, plus an “annealing” mid-training stage, instruction tuning, and a diffusion-native RL post-training step called coupled-GRPO. Two contributions matter: (1) the paper introduces local/global autoregressive-ness (AR-ness) metrics and uses them to characterize how dLLMs actually decode — revealing an “entropy sink” toward the left of the completion and showing that higher sampling temperature in dLLMs diversifies both token choice and generation order; (2) coupled-GRPO replaces the standard single-mask Monte Carlo log-prob estimator with two complementary masks per training example whose union covers every token, an antithetic-variates construction that gives every token at least one accurate log-prob estimate. Coupled-GRPO lifts DiffuCoder-Instruct by +4.4 absolute on EvalPlus (+7.3 on the average of EvalPlus tasks) over the instruct baseline, with a stronger non-AR generation pattern post-RL.

Key claims

DiffuCoder-Instruct on Stage 1+2+3 training (continual pretraining → annealing → SFT) reaches 63.6 average on EvalPlus (HumanEval+ avg with MBPP+), competitive with Qwen2.5-Coder+SFT (70.9) and well above LLaDA-Instruct (30.2) and Dream-Instruct (54.9) at similar scale [Table 1].
Existing dLLM RL recipes (d1, LLaDA1.5+DPO, MMaDA+GRPO) either show marginal gains or rely on semi-AR / block decoding (block-wise generation), which “reintroduces causal bias” and deviates from diffusion’s global-planning nature [§1, §3].
Local AR-ness@k (fraction of length-k spans that are strict next-token continuations) and Global AR-ness@k (fraction of steps that unmask within the leftmost k masks) jointly measure how AR a dLLM’s actual decoding order is; both are below 1.0 for current dLLMs but well above zero, with adapted dLLMs (DiffuCoder, Dream) more AR than from-scratch dLLMs (LLaDA) because they inherit left-to-right token dependencies from their AR base [§4.1, §4.2, Fig. 3].
Code decoding has lower mean and higher variance in global AR-ness than math decoding, consistent with code having intrinsic non-sequential structure (“the model jumps back and forth like a programmer refining an implementation”) while math is sequential by nature [§4.2].
The “entropy sink” phenomenon: at the first denoising step (fully masked completion given prefix), recovered-token confidences form an L-shape — tokens immediately to the right of the prefix receive disproportionately high confidence, producing a causal bias under low-confidence remasking [§4.2, Fig. 3a].
Unlike AR models where temperature only diversifies token choice, in dLLMs raising temperature from 0.2 to 1.2 also diversifies generation order (lowers global AR-ness), and lifts pass@10 substantially — indicating latent capacity recoverable by RL [§4.3, Figs. 1a, 5, 6].
Coupled-GRPO samples timestep pairs (t, T−t) with complementary masks whose union covers every token, guaranteeing each token gets at least one log-prob estimate under a realistic partial-masking context; the construction is shown to be an antithetic-variates variance-reduction scheme [§5, Eq. 4, Appx. A.3].
Coupled-GRPO on DiffuCoder-Instruct gives +4.4 EvalPlus (+3.1 HumanEval+, +5.6 MBPP+), +4.7 BigCodeBench-Full, and +1.2/+3.5 HumanEval/MBPP over the instruct baseline; +3.9 average across all eight code benchmarks [Table 1, Table 2].
Ablations: a fully-masked completion baseline (d1-style, ε=0) regresses on HumanEval (−5.5) and a decoupled-sampling baseline (same number of samples but no complementary constraint) lags coupled-GRPO on every task — the complementary masking, not just extra samples, drives the gain [Table 2].
After coupled-GRPO, the optimal sampling temperature shifts upward (e.g. from 0.2 to 1.0–1.2), global AR-ness drops further, and the performance drop when halving the diffusion step count shrinks — i.e. the post-RL model is more parallel-friendly [§5 Experiment Results, Fig. 1c, Fig. 4 right, Appx. 11].
Masking condition tokens (d1’s recipe to inject diversity) does not yield stable reward improvement on code, because code demands higher token-level accuracy than math; the authors revert to keeping the prompt fully unmasked [§5 Baseline methods, Fig. 7].

Method

DiffuCoder is trained in four stages, all initialized from Qwen2.5-Coder via the Gong-et-al. AR→MDM adaptation recipe. Stage 1 (continual pretraining) trains on 65B of a 400B-token code corpus (RefineCode + Stackv2) — early stopping at 65B is required because going to 700B degrades the downstream signal, attributed to pretraining-data quality. Stage 2 (mid-training / annealing) trains 4 epochs on 16B high-quality annealing tokens (OpenCoder). Stage 3 (instruction tuning) uses 436K SFT samples (OpenCoder). Stage 4 (RL post-training) is coupled-GRPO on 21K hard, test-case-verified samples from Acecoder-87K; one epoch.

The coupled-GRPO objective replaces the standard log-prob ratio log π/π_old per token by an estimate from a paired forward pass: for each example, sample λ pairs of timesteps (t, T−t), then construct two complementary completion masks such that every token is unmasked in exactly one of the two passes. Each token therefore receives at least one log-prob evaluation under a realistic partial-masking context (rather than the d1 default where every token is fully masked at all times). The full GRPO surrogate is summed over the per-pair log-prob ratios, with PPO-style clipping and a KL penalty to a reference policy. The pairing is an antithetic-variates construction — pairs of correlated estimates whose covariance is provably non-positive — giving variance reduction relative to two independent samples. For advantage estimation, the standard group-mean baseline is compared against a leave-one-out variant; the plain group-mean version yields slightly stronger overall numbers on the filed benchmark mix. Rollouts use temperature 1.2, matching the temperature at which pass@10 peaks for DiffuCoder-Instruct.

Reward shaping uses a verified code-format reward (parseable function definition) plus an execution-pass-rate reward over the example’s test cases.

Results

DiffuCoder-Instruct + coupled-GRPO: 73.2 HumanEval, 68.3 HumanEval+, 78.6 MBPP, 67.5 MBPP+, 40.4 BigCodeBench-Full, 10.8 BigCodeBench-Hard, 37.5 BigCodeBench-Instruct-Full, 10.8 BigCodeBench-Instruct-Hard [Table 1, Table 2].
Average gain over DiffuCoder-Instruct across eight code tasks: +3.9 absolute; EvalPlus average +4.4 (63.6 → 67.9) [Table 1].
Coupled-GRPO matches or exceeds the AR baseline Qwen2.5-Coder+SFT+GRPO on MBPP/MBPP+/BigCodeBench-Full/BigCodeBench-Instruct-Hard, despite the dLLM being adapted from the same AR base [Table 2].
Reward learning curves: coupled-GRPO trains stably with monotone reward; d1, full-mask-completion, and decoupled-sampling all show unstable / non-monotone reward trajectories [Fig. 7].
After coupled-GRPO, halving the diffusion steps (e.g. 256→128) produces a smaller quality drop than for the pre-RL model, consistent with reduced AR-ness and more parallel decoding [Fig. 1c, Appx. 11].
Coupled-GRPO is not strictly Pareto-optimal: BigCodeBench-Hard drops 12.2 → 10.8 even as the other seven tasks improve [Table 1].
The cpGRPO checkpoint on Hugging Face is the released variant: apple/DiffuCoder-7B-cpGRPO, with apple/DiffuCoder-7B-Instruct and a Base checkpoint also released.

Why it’s interesting

Two angles. First, coupled-GRPO is a constructive fix for the exact failure mode that Stabilizing Reinforcement Learning for Diffusion Language Models later formalizes: ELBO/mean-field log-prob estimates in dLLMs are noisy, and naively dropping them into GRPO leads to unstable training. DiffuCoder’s antithetic-variates construction attacks the variance of the estimator; StableDRL attacks the clipping + normalization that amplifies the noise. They’re complementary — coupled-GRPO is the upstream fix at the estimator, StableDRL is the downstream fix at the surrogate — and the team should read them together when designing a diffusion-RL pipeline. Second, the AR-ness measurements and the entropy-sink finding are useful diagnostics independent of the RL story: they give a quantitative handle on “how diffusion is this dLLM actually behaving?” that the Reasoning RL concept page and any future dLLM work can reuse. The fact that adapted-from-AR dLLMs (DiffuCoder, Dream, and indirectly WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference‘s Qwen-adapted recipe) retain measurable AR bias is the right framing for thinking about how much “diffusion-ness” you actually get from the conversion.