Product of Experts for Visual Generation

TL;DR

Product of Experts (PoE) is a training-free inference-time framework that samples from a product of heterogeneous knowledge sources — pretrained generative diffusion/flow models, VLM-based discriminative reward functions, and non-neural physics/graphics simulators (Genesis) — by running Annealed Importance Sampling with parallel Sequential Monte Carlo. The product distribution is high-likelihood only where every expert agrees; per-step Langevin MCMC on a composed score plus particle resampling on discriminative weights makes the sampling tractable. Demonstrated on three tasks: graphics-engine-driven image editing, physics-simulator-driven video generation (Wan 2.1 + Genesis), and bbox-region T2I composition with VQAScore as a discriminative expert. The headline relevance for Luma is the second task — the framework lets a frozen video diffusion model take object-level rigid-body simulator outputs (poses + dynamics) as a hard control signal at inference, no fine-tuning.

Key claims

Visual generation can be formulated as sampling from a product distribution $p(x) \propto \prod_k p_k(x_k|\cdot) \cdot \prod_j \exp(R_j(x))$ over generative experts $p_k$ (diffusion/flow) and discriminative experts $R_j$ (VLM rewards), enabling inference-time fusion of heterogeneous knowledge with no joint training [§3, Eq. 1].
Naïve rejection sampling on the product is computationally prohibitive in high-dim visual synthesis; the paper proposes per-step Langevin MCMC on the composed score plus particle resampling on discriminator weights as a tractable Annealed Importance Sampling / SMC implementation [§3.2, §3.3, Algorithm 1].
Conditional generative experts — each expert’s velocity prediction is modified to depend on parent regions of the partial sample (Eq. 7) — give materially better samples than unconditional product sampling; ablation “No Cond” worsens both controllability and image quality on the editing benchmark [§3.1, §4.2, Table 1].
On graphics-engine-instructed image editing (Genesis 3D-asset insertion + FLUX.1 Depth + FLUX.1 Fill), PoE beats RF-Solver inversion baseline on MSE-bg, LPIPS-fg, aesthetic, ImageReward, and GPT-4o controllability/quality/text-alignment scores; RF-Solver fails because text prompts cannot specify object pose [§4.2, Table 1, Fig. 2].
On object-centric physics-instructed video generation (Genesis-simulated rigid-body trajectories + Wan 2.1 Image-to-Video + Wan 2.1 Depth-to-Video), PoE beats Traj2V, Depth2V, and Image2V monolithic baselines on LPIPS, GPT-4o controllability, smoothness, aesthetic, MUSIQ imaging, ViCLIP semantic alignment, and GPT-4o semantic [§4.3, Table 2 top].
Monolithic depth-/trajectory-to-video baselines compensate for foreground object motion with spurious upward camera motion — visualized by tracking trajectories — because they lack a foreground-only prior; PoE’s region-restricted simulator expert avoids this failure [§4.3, Fig. 4].
On full-scene physics simulation (PhysGen3D dataset, 9 scenes), initializing SMC particles from flow-ODE inversion rather than random noise improves both fidelity and quality over Depth2V, Image2V, and a flow-inversion baseline [§4.3, Table 2 bottom, Fig. 5].
On text-to-image generation with regional bbox composition (LLM-parsed local prompts + FLUX.1 [dev] per region as generative experts + VQAScore as discriminative expert), the discriminative expert provides counting/relational corrections that pure generative composition (1 particle) misses; matching compute via best-of-N on vanilla FLUX is also outperformed [§4.4, Fig. 6].
The discriminator side requires only black-box access — likelihood values, not gradients — so non-differentiable rewards (VQAScore, GPT-4o, simulator rules) can be plugged in directly [§3.2].
Region-restricted experts are implemented via Gaussian-blurred foreground masks applied to the velocity prediction, allowing an expert defined over the full scene to influence only a sub-region (foreground object) while a separate “background” expert covers the rest [§4.1].

Method

The framework targets sampling from $p(x) \propto \prod_k p_k(x_{x_k}|x_{pa(k)}) \cdot \prod_j \exp(R_j(x))$ , where each generative expert $p_k$ has prior over a (possibly local) region $x_k$ conditioned on a “parent set” of other expert outputs, and each discriminative expert $R_j$ returns a scalar score (VLM logit, simulator rule check, VQAScore). For diffusion/flow generative experts, the implementation uses the intermediate score functions to define an annealed sequence of intermediate product distributions $\pi_t(x) \propto \prod_k p_{k,t}(x_{x_k}) \cdot \prod_j \exp(\gamma_t \cdot R_j(\hat{x}_0))$ , parameterized by the noise level $t$ .

Sampling proceeds with $N$ particles. At each annealing step: (1) take one Euler step in the flow ODE using the composed velocity $\sum_k v_k$ (zero-padded to full scene from any region-restricted expert); (2) run several Langevin MCMC steps on the composed score for local refinement at $\pi_t$ ; (3) compute discriminator weights $w_i = \exp(R(\hat{x}_0^{(i)}))$ using the Tweedie-estimated clean sample, and resample particles proportionally to $w_i$ (SMC). The conditional-expert modification (Eq. 7) replaces $v_k$ with $v_k + \eta (v_{pa(k)} - v_k)|_{x_{pa(k)}}$ , pulling each expert’s velocity prediction toward agreement with parent regions and reducing multi-modality in the product.

Two concrete instantiations: (a) graphics-engine editing — Genesis renders a posed 3D asset to depth + RGB; FLUX.1 Depth [dev] uses the depth as a controllability expert over the foreground mask, FLUX.1 Fill [dev] inpaints the rest as a background-prior expert; (b) physics-simulator video — Genesis simulates rigid-body trajectory + depth; Wan 2.1 Image-to-Video provides the full-scene natural-video prior, Wan 2.1 Depth-to-Video (VideoX-Fun checkpoint) provides simulator-conditional foreground motion. The (c) regional T2I task uses FLUX.1 [dev] conditioned on bbox-local prompts as multiple regional generative experts plus VQAScore as a global discriminative expert.

Results

Image editing (30 scenes, Genesis + FLUX): PoE beats RF-Solver on every reported metric — MSE-bg, LPIPS-fg, GPT-4o controllability, aesthetic, ImageReward, GPT-4o text-alignment — and the “No Cond” ablation (without the parent-conditioning update Eq. 7) underperforms PoE on aesthetic and visual harmony [Table 1].

Object-centric video (12 scenes, Genesis + Wan 2.1): at fixed compute, PoE matches or beats Traj2V / Depth2V / Image2V Wan 2.1 baselines on controllability (LPIPS, GPT-4o), motion smoothness, imaging quality (MUSIQ), and semantic alignment (ViCLIP, GPT-4o). The qualitative diagnostic — monolithic depth-/trajectory-conditioned models invent unwanted upward camera pans to compensate for foreground object falling — is the clearest signal that region-restricted PoE captures something the alternatives don’t [Table 2 top, Fig. 4].

Full-scene video (PhysGen3D 9 scenes): flow-ODE inversion initialization beats random-noise initialization; PoE beats PhysGen3D, raw inversion, Depth2V, Image2V on most quality and controllability metrics. Visible qualitative win: cleaning up dark/missing-region artifacts in simulator outputs (e.g. apple’s black regions) while preserving simulated motion [Table 2 bottom, Fig. 5].

Regional T2I (16 particles): PoE with discriminative VQAScore expert produces correct counting/relational compositions where vanilla FLUX, 1-particle generative-only composition, and compute-matched best-of-N on vanilla FLUX all fail [§4.4, Fig. 6].

No FLOPs/wall-clock numbers in the released body; particle count is the main compute knob.

Why it’s interesting

This is the inference-time counterpart to two existing wiki strands. First, Inference-time Physics Alignment of Video Generative Models with Latent World Models (WMReward, the ICCV 2025 PhysicsIQ winner) also adds physics-plausibility at inference time over a frozen video model, but via a single neural reward (VJEPA-2 surprise) — PoE generalizes this to a product of arbitrarily many heterogeneous experts including non-neural simulators that provide hard constraints rather than soft rewards. The two recipes are complementary: WMReward gives “videos that look physical to a learned world model”; PoE gives “videos that obey a specific named rigid-body simulation”. Second, Classifier-Free Guidance variants reframes CFG as a special case of a two-distribution mix; PoE is the natural N-expert generalization, with SMC replacing the simple velocity averaging that CFG uses. Beyond those, PoE connects to World Foundation Models from a different angle than the existing entries: rather than treating a pretrained WFM (VJEPA, Cosmos Reason) as a backbone, it treats non-neural physics engines as just another expert in the product — a Bitter-Lesson-violating but pragmatically powerful move that depends on the simulator being cheap and the SMC weights being reliable. For Luma, the immediate read is that this is a way to give video models hard pose/dynamics control without retraining — a knob that’s complementary to LoRA-style fine-tuning and could compose with existing controlled-video stacks (Kling-MotionControl, MoRight).