Paris: A Decentralized Trained Open-Weight Diffusion Model

TL;DR

Paris is a text-to-image diffusion model trained with zero inter-worker communication — no gradient sync, no parameter sharing, no activation exchange. The recipe partitions 11M LAION-Aesthetic images into 8 DINOv2-clustered subsets, trains one 605M-parameter DiT-XL/2 expert in isolation on each subset across heterogeneous AWS/GCP/Runpod hardware, and post-hoc trains a lightweight ~129M DiT-B router that selects among experts during the reverse diffusion process. The author claim is 14× less data and 16× less compute than the prior DDM baseline (McAllister et al. 2025) at a 1.27× FID cost (12.45 vs 9.84 on LAION-Aesthetic subsets), and the open MIT-licensed release is positioned as the first decentralized-trained diffusion model with public weights.

Key claims

8 DiT-XL/2 experts (605M each, 4.84B total) are trained in complete isolation on disjoint DINOv2-clustered LAION-Aesthetic partitions with no gradient, parameter, or activation exchange between them during training [§2.1, §2.5].
The router is a smaller DiT-B (~129M params, ~1/3 the size of each expert) trained post-hoc on the full dataset via cross-entropy against ground-truth cluster assignments, conditioned on the noisy latent and timestep so it can route at any point in the reverse process [§2.5.1, §2.5.3].
Three inference strategies are supported: Top-1 (one expert per step, constant cost), Top-K (weighted combination of K experts), and Full Ensemble (all 8 weighted by router probability) — Top-1 is the default, Top-K=2 is reported as best quality at DiT-B/2 scale [§2.7].
At DiT-B/2 scale, Top-2 routing reaches FID-50K = 22.60 vs 29.64 for a monolithic single-model baseline trained on the full data — a 7.04 FID improvement; the Full Ensemble degrades to 47.89, suggesting selective routing beats naive averaging [Table 2].
At DiT-XL/2 scale, Paris (Top-1) reaches FID 12.45 on LAION-Aesthetic with 11M training images and 120 A40 GPU-days (~72 A100-equivalent); DDM (McAllister et al. 2025) reports FID 9.84 with 158M images and 1176 A100 GPU-days — a 14.4× data and 16.3× compute reduction at 1.27× the FID [Table 3].
The training framework extends decentralized flow matching (McAllister et al. 2025) and the DiT architecture (Peebles & Xie 2022) with AdaLN-Single (PixArt-α) conditioning, which the paper cites as a 23% per-block parameter reduction [§2.4.3].
The zero-communication design is positioned against Data Parallel (periodic all-reduce, slowest-worker blocking), Model Parallel (sequential layer transfers), and Pipeline Parallel (bubble overhead) — Paris reports no synchronization, no straggler blocking, and arbitrary topology as the explicit tradeoff [§2.8, Table 1].
Released under MIT license (weights + code) with diffusers integration; the HF model card documents Top-1/Top-2/Full-Ensemble usage and architecture details, and is the form that brought the paper into the wiki [HF model card].

Method

The training procedure has four offline stages: (1) pre-encode all 11M LAION-Aesthetic images into 32×32×4 sd-vae-ft-mse latents (8× spatial downsampling from 256×256); (2) extract 1024-dim DINOv2-ViT-L/14 features per image and run hierarchical k-means to produce 8 semantically coherent clusters; (3) train 8 DiT-XL/2 experts (1152 hidden / 28 layers / 16 heads / patch 2 / cross-attention to frozen CLIP-ViT-L/14 text, AdaLN-Single conditioning) in complete isolation across heterogeneous infrastructure (AWS, GCP, local clusters, Runpod) using flow matching with velocity prediction — each expert sees only its cluster, optimizes its own objective, and never communicates with any other expert; (4) train a DiT-B router on the full dataset, sampling timesteps uniformly and applying the corresponding noise schedule so the router learns to classify noisy latents into the correct cluster across all noise levels.

At inference, the router scores the current noisy latent $x_t$ at each denoising step and one of three combination rules is applied: Top-1 picks $\operatorname{argmax}_i p(c_i | x_t, t)$ ; Top-K renormalizes the top-K router probabilities and forms a weighted sum of K expert velocity predictions; Full Ensemble forms the weighted sum over all 8 experts. The flow-matching velocity prediction decomposes naturally across the data partitions, so the weighted ensemble approximates the marginal flow without any additional training [§2.3, Eq. 3-4].

Results

DiT-B/2 (129M/expert, 8 experts, 1.03B total) on LAION-Art: Monolithic 29.64 FID-50K, Top-1 30.60, Top-2 22.60 (best), Full Ensemble 47.89. Top-2 beats the monolithic baseline by 7.04 FID points [Table 2].
DiT-XL/2 (605M/expert, 8 experts, 4.84B total) on a LAION-Aesthetic subset: Paris Top-1 reaches FID 12.45 with 11M images and 120 A40 GPU-days; DDM (McAllister et al. 2025) reaches 9.84 FID with 158M images and 1176 A100 GPU-days. Ratios: 14.4× less data, 16.3× less compute, 1.27× higher FID [Table 3, Fig. 3].
The Full-Ensemble strategy underperforms both Top-1 and Top-2 at DiT-B/2 scale (47.89 vs 30.60/22.60), which the paper attributes to interference from less-relevant experts adding noise to the predicted velocity [§2.7.3, Table 2].
The paper notes the FID comparison against DDM is not on the identical evaluation subset (DDM’s subset is not publicly disclosed); the headline efficiency-vs-FID-tradeoff claim should be read with that caveat [Table 3 caption].

Why it’s interesting

This is the first filed paper that proposes zero-communication training as a primary design point rather than as a fault-tolerance fallback — every system filed under Distributed training parallelism so far (veScale-FSDP: Flexible and High-Performance FSDP at Scale, SpectraX — True MPMD Pipeline Parallelism for JAX, Scaling and Optimizing Frontier Model Training (Fireworks AI), KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training, VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo) reduces communication overhead but still assumes periodic synchronization; Paris assumes none. The closest sibling on the algorithmic side is DiLoCo (Douillard et al. 2023, cited as the inspiration for local-SGD-style infrequent sync), but Paris pushes that further to no sync.

The router-over-isolated-experts design is also a different point in the MoE Routing Design space than every other filed paper: standard MoE routes inside a single jointly-trained model with auxiliary load-balancing losses to keep experts comparable; Paris’s experts are independently trained on disjoint data and the router is trained post-hoc on the union. This sidesteps load balancing entirely (each expert sees its cluster’s data distribution) but introduces a new question — whether the router can reliably classify noisy latents into the correct cluster, which is what the noise-aware training in §2.5.3 is designed to address. As an open-weights MIT-licensed release with public training code, this also fits Open foundation-model releases alongside other 2026 decentralized-friendly releases.