Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation

TL;DR

Foveated Diffusion is a perceptually motivated efficiency lever for DiT-based image and video generation: instead of denoising a uniform token grid, it denoises a mixed-resolution token sequence with high-resolution tokens near a user-specified gaze location and low-resolution tokens (1 token per k×k high-res block) in the periphery. A LoRA-based post-training procedure on FLUX.2 Klein 4B (image) and Wan2.1 1.3B (video) — together with a modified RoPE that aligns key/query phases across resolutions — yields 2× speedup for images and 4× for video at perceptual parity with full-resolution generation, validated by a pseudo-eye-tracked 2AFC user study. The training-free baseline (“naïve mixed-resolution denoising”) fails with duplicated objects and scale mismatches; the contribution is the post-training recipe that makes the mixed-resolution token sequence coherent by construction.

Key claims

Naïve mixed-resolution denoising — running a pretrained DiT directly on a foveated token grid with RoPE-adapted attention — produces strong structural artifacts (duplicated objects, fused entities, scale mismatches), so a training-free foveation recipe does not work [§3.2.2, Fig. 4, Fig. 5].
Foveated training fixes this by constructing the supervision target as a single coherent mixed-resolution token sequence: the same image is encoded twice (full-res via the VAE, and bicubically downsampled before encoding), and the foveation mask merges the two streams into one target before the flow-matching loss is computed [§3.2.3, Eqs. 8–10].
At 26% token ratio the model retains HPSv2.1 0.280 (vs. 0.280 for full-res) and FID 12.62 (vs. 11.38 for full-res) with 2.08× wall-clock speedup, while the naïve baseline drops to HPSv2.1 0.275 with FID 12.83 at the same token ratio [Table 1].
On video (Wan2.1 1.3B, VBench), foveated diffusion at the same token-ratio operating point matches full-resolution VBench Subject Consistency (0.9446 vs 0.9407), Background Consistency (0.9393 vs 0.9363), and Motion Smoothness (0.9946 vs 0.9943) with a 4× speedup, while the naïve mixed-resolution baseline degrades on every consistency metric [Table 2].
Standard generative metrics (FID, Precision) are misaligned with the foveated-perception goal — FID even rewards the naïve baseline despite visible structural artifacts — so a 2AFC user study under a pseudo-eye-tracked protocol (1-second display, red-dot fixation) is necessary; under that protocol foveated samples reach near perceptual parity with full-resolution generation and are strongly preferred over the naïve baseline [§4.2.2, Fig. 7].
Mixed-resolution RoPE — subsampling key-token phases to match query-token resolution, per Wu et al. (CRPA) — is required for the attention math to work across resolutions, but is not sufficient on its own; foveated post-training is what closes the structural-coherence gap [§3.2.2].
Foveation masks are arbitrary binary masks fed in at training time; randomized circular masks produce a gaze-agnostic generator, saliency-guided masks bias the foveal region toward salient content, and bounding-box masks also work — no architecture change required, only the mask sampling distribution changes [§3.2.4].
Speedup scales with token ratio non-trivially: at 26% tokens the image speedup is 2.08× (sub-linear), and the video speedup is larger than image speedup at the same token ratio because spatiotemporal 3D attention is more aggressively quadratic [§4.2.1, Fig. 6].
LoRA rank 32 on FLUX.2 Klein 4B and Wan2.1 1.3B is sufficient — the recipe does not require full fine-tuning [§4.1].

Method

The pipeline (Fig. 2) starts with a binary foveation mask M defining the high-resolution (HR) and low-resolution (LR) spatial regions, with downsampling factor s between them (the paper uses s=2 or s=4 in the periphery). At inference, Gaussian noise is sampled directly in the foveated token space of reduced length N_HR + N_LR / s² rather than in the full uniform token grid. The DiT iteratively denoises this reduced sequence using flow-matching velocity prediction; positional embeddings use a Mixed-Resolution RoPE that aligns query/key phases across the two resolutions by subsampling the higher-resolution side when attending across scales. The clean foveated token sequence is then split into HR and LR subsets, decoded separately by the VAE, the LR decode is spatially upsampled, and the two are blended via the upsampled latent foveation mask.

The non-obvious contribution is the training recipe. Given a high-res training image x, two latent token sequences are constructed — z_HR by VAE-encoding x at full resolution, z_LR by VAE-encoding the bicubically downsampled x — and merged via the foveation mask into a single target z̃_0. Both halves of z̃_0 are derived from the same image content, so the target is mixed-resolution-coherent by construction. Standard flow-matching loss is then applied: noise is added per Eq. 2, the model predicts the velocity field, and the L2 loss is computed across the variable-length foveated sequence. All compute is on the reduced sequence at all times. During training, masks are sampled randomly per step (circular for image, foveation-path for video) so the trained generator is foveation-agnostic. Adaptation is done with LoRA rank 32 on FLUX.2 Klein 4B (image, Aesthetic-Train-V2, 90k images) and Wan2.1 1.3B (video, Vchitect-T2V-Dataverse).

Results

Image generation, FLUX.2 Klein 4B base, centered rectangular mask, FLUX.2-Klein evaluation prompts. At 43% token ratio: Foveated HPSv2.1 0.279 / FID 11.38 / Precision 0.777 / runtime 1.61× speedup; naïve mixed-res 0.268 / 10.99 / 0.769. At 26% token ratio: Foveated 0.280 / 12.62 / 0.792 / 2.08× speedup; naïve 0.275 / 12.83 / 0.775. Full high-res baseline: 0.280 / 11.38 / 0.792 at 10.45s [Table 1].

Video generation, Wan2.1 1.3B base, centered circular mask at ~26% token ratio, VBench evaluation. Foveated matches or beats full-resolution on Subject Consistency (0.9446 vs 0.9407), Background Consistency (0.9393 vs 0.9363), Motion Smoothness (0.9946 vs 0.9943), Aesthetic Quality (0.5432 vs 0.5434); naïve baseline degrades on every metric (e.g. Subject Consistency 0.9072). Reported speedup: 4× at this operating point [§4.2.3, Table 2].

User study: 2AFC pairwise comparisons with 1-second display + red-dot fixation. Foveated samples reach near perceptual parity with full-resolution generation and are strongly preferred over the naïve baseline [Fig. 7]. The user study is the headline evidence — the authors explicitly argue FID/Precision are misaligned with foveated perception and that the user study is the appropriate metric.

Why it’s interesting

Foveated Diffusion adds a spatial axis to the efficiency-via-token-reduction story that the wiki has so far accumulated almost entirely on the temporal axis. TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generation prunes drifting AR tokens at inference, Don't Look Twice: Faster Video Transformers with Run-Length Tokenization removes temporally redundant patches before the encoder, and Pretraining Frame Preservation in Autoregressive Video Memory Compression compresses history through a learned encoder — all of these reduce tokens by exploiting redundancy across time. Foveated Diffusion reduces tokens by exploiting human perceptual eccentricity, which is a content-agnostic redundancy that none of those methods touch. The two axes appear stackable.

Two pieces are notable beyond the speedup numbers. First, the training-free version fails, which is a useful negative result given how many recent efficient-generation papers (Run FLUX.1-dev three times faster, FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU) are training-free — it places foveation closer to Diffusion Distillation in terms of “needs post-training to actually work” than to those caching/serving recipes. Second, the FID-vs-user-study divergence is sharp and consistent with the broader theme that VLM-as-Evaluator-style holistic metrics and human-preference protocols are diverging from FID/Precision at the high end of generative quality — here even a naïvely broken generator scores well on FID, which is a useful caution for anyone evaluating efficient-generation tradeoffs with FID alone.

For Luma specifically: the recipe is gaze-dependent at inference but the training can be gaze-agnostic (randomized masks), meaning a foveated checkpoint can be deployed in real-time interactive video settings where eye-tracking is available (VR/AR), while a saliency-guided variant could automatically allocate compute to salient regions even without eye-tracking. The 4× video speedup on Wan2.1 1.3B is at the same scale as Luma’s current real-time video work.