PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation

TL;DR

PrismAudio is the first Video-to-Audio (V2A) framework that integrates reinforcement learning with specialized Chain-of-Thought planning, decomposing what prior V2A models like ThinkSound treated as a monolithic reasoning step into four parallel CoT modules (Semantic, Temporal, Aesthetic, Spatial), each paired with a targeted reward (MS-CLAP / Synchformer / Audiobox-Aesthetics / StereoCRW). To make multi-dimensional RL on a flow-matching audio DiT computationally practical, the paper introduces Fast-GRPO — a hybrid ODE-SDE GRPO variant that applies SDE sampling only inside a small random window of denoising steps and uses deterministic ODE elsewhere, dropping policy-model NFE from O(T²) to O(T) per sample. The 518M-parameter model beats ThinkSound (1.3B) and HunyuanVideo-Foley (5.31B) on every perceptual axis (CLAP 0.47, DeSync 0.41, CRW MSE 7.72 vs 13.47, MOS-Q 4.21) on VGGSound and on the paper’s new AudioCanvas benchmark while being ~1.7× faster at inference. Accepted to ICLR 2026.

Key claims

V2A generation requires balancing four interdependent and competing perceptual axes (semantic consistency, audio-visual temporal synchrony, aesthetic quality, spatial accuracy), and prior methods that optimize a single reconstruction loss suffer from “objective entanglement” that conflates these goals [§1, Abstract].
Decomposing the monolithic reasoning of ThinkSound into four specialized CoT modules (Semantic, Temporal, Aesthetic, Spatial) — each paired with a targeted reward function — enables multidimensional RL optimization without losing interpretability [§1, §3.2].
Two CoT-aware architectural upgrades over ThinkSound: VideoPrism replaces CLIP for video understanding (frame-wise CLIP misses temporal context), and T5-Gemma replaces standard T5 to handle the structured reasoning text emitted by the four CoT modules [§3.1].
The four reward functions are off-the-shelf perception models — MS-CLAP for semantic alignment, Synchformer for audio-visual synchrony, Meta Audiobox Aesthetics (MOS predictor) for aesthetic quality, and StereoCRW for stereo spatial positioning — no learned reward network is trained [§3.3.1].
Fast-GRPO confines stochasticity and policy-gradient computation to a small randomly-placed window of denoising steps and runs deterministic ODE elsewhere, reducing per-sample policy-model NFE from O(T²) to O(T) and yielding near-linear GRPO training complexity [§3.3.2, Eq. 7].
The hybrid ODE-SDE sampling preserves the terminal data distribution required for correct reward computation, with a closed-form Gaussian per-step policy and tractable GRPO ratio over the SDE-window steps [§3.3.2, App. B.1–B.3, Eqs. 3–4].
Group-relative advantage normalization is applied to a weighted sum of the four rewards (Eqs. 5–6), preserving GRPO stability while enabling principled multi-objective trade-offs via per-dimension weights wₖ [§3.3.2].
On VGGSound, PrismAudio (518M params) reaches CLAP 0.47 vs ThinkSound’s 0.43, DeSync 0.41 vs 0.55, CRW MSE 7.72 vs 13.47, MOS-Q 4.21 vs 4.05, and MOS-C 4.22 vs 4.18 — new SOTA across all four perceptual axes [Table 1].
The ablation “PrismAudio w/o CoT-RL” already beats prior SOTA on DeSync and CRW (i.e. the architectural upgrades alone are competitive), with CoT-RL then providing a further +4.7% MOS-Q / +2.7% MOS-C boost on top [Table 1, §4.2].
On the paper’s new AudioCanvas out-of-domain benchmark (3,177 expert-filtered real-world videos, 501 multi-event), ThinkSound’s DeSync collapses from 0.55 to 0.80 and CRW from 13.47 to 22.82, while PrismAudio remains stable and even exceeds ground-truth on CLAP / DeSync (which the paper attributes to its rewards being explicit optimization targets) [§4.2, Table 2].
The CoT-RL contribution grows with task complexity: on AudioCanvas the gap between PrismAudio and the no-CoT-RL ablation widens — e.g. CLAP 0.42 → 0.52 — indicating that multi-dimensional optimization is more critical when pattern matching fails [§4.2].
Single-dimensional rewards exhibit suboptimal trade-offs (improving one axis at others’ expense), while multi-dimensional CoT-RL balances all axes without compromising individual performance [§1, Table 4].
PrismAudio is more efficient than baselines despite the additional CoT machinery: 518M params (vs ThinkSound 1.3B, HunyuanVideo-Foley 5.31B, AudioX 1.1B), inference at 0.63 s per 9-second clip (vs ThinkSound 1.07 s, MMAudio 1.30 s) [Table 1].
AudioCanvas itself is a contribution: 300 single-event classes + 501 multi-event samples with expert-filtered modality alignment and Gemini-2.5-Pro CoT annotations validated to >94% human-verified accuracy [§4.1, App. C].

Method

PrismAudio sits on top of a ThinkSound-derived flow-matching MMDiT audio foundation model (the same lineage as MMAudio), with two CoT-enabling backbone upgrades: VideoPrism replaces CLIP for video understanding (so the encoder can capture actions and scene context across frames rather than treating frames as independent images), and T5-Gemma replaces the standard T5 text encoder (so the conditioning channel can actually parse the structured reasoning text emitted by the four CoT modules). Training proceeds in three phases. Phase 1 pretrains the audio MMDiT on large-scale video-audio pairs with standard flow-matching. Phase 2 trains four specialized CoTs by first using Gemini 2.5 Pro to construct CoT training data, then fine-tuning VideoLLaMA2 to emit four separate reasoning passages — Semantic CoT (audio events and their characteristics), Temporal CoT (event ordering), Aesthetic CoT (naturalness/fidelity), Spatial CoT (directional positioning) — which are concatenated and used as structured conditioning to fine-tune the audio MMDiT.

Phase 3 is the multi-dimensional RL post-training. Each candidate audio sample drawn from the policy is scored by four off-the-shelf rewards: MS-CLAP for text-audio semantic alignment, Synchformer for video-audio synchrony, Audiobox-Aesthetics (a no-reference MOS predictor) for aesthetic quality, and StereoCRW for stereo directional accuracy. The four rewards are linearly combined with weights wₖ to produce a scalar reward per candidate, and GRPO group-relative advantage normalization is applied within each prompt’s candidate group. The Fast-GRPO contribution is the sampler: rather than the “pure SDE” path used by Flow-GRPO and DanceGRPO (SDE at every denoising step so the policy can be evaluated everywhere), Fast-GRPO randomly picks a starting position k and an optimization window of width ω, runs SDE sampling only inside [k, k+ω], and runs deterministic ODE everywhere else. This preserves the terminal data distribution (App. B.2) while reducing policy-model NFE from O(T²) to O(T) per sample. The GRPO objective is restricted to the SDE-window steps (Eq. 7), giving a windowed importance ratio that has a closed-form per-step Gaussian policy (Eqs. 3–4).

Results

On VGGSound test (in-domain), PrismAudio at 518M parameters sets SOTA on every axis: CLAP 0.47 (vs ThinkSound 0.43, AudioX 0.41), DeSync 0.41 s (vs ThinkSound 0.55, HunyuanVideo-Foley 0.55), Audiobox CE 4.29 / CU 5.68 (vs ThinkSound 3.95 / 5.48), GCC MSE 3.77 (vs ThinkSound 4.65), CRW MSE 7.72 (vs ThinkSound 13.47 — a ~43% reduction), FD 1.08 (vs 1.17), KL 1.23 (vs 1.32), MOS-Q 4.21±0.35 (vs 4.05±0.55), MOS-C 4.22±0.29 (vs 4.18±0.51) [Table 1]. Inference time is 0.63 s per 9-second sample vs ThinkSound’s 1.07 s and MMAudio’s 1.30 s [Table 1]. The “w/o CoT-RL” ablation already beats prior SOTA on DeSync (0.51 vs 0.55) and CRW (10.29 vs 13.47), so the CoT-RL post-training is contributing the remaining gap, not the full headline gain.

On AudioCanvas (out-of-domain, expert-filtered, harder), the gap widens. ThinkSound’s DeSync degrades from 0.55 → 0.80 and CRW from 13.47 → 22.82, while PrismAudio remains stable. The CoT-RL contribution also widens — semantics CLAP improves 0.42 → 0.52 with CoT-RL (vs 0.47 in-domain), aesthetics CE 3.81 → 4.26 [§4.2, Table 2]. PrismAudio’s MOS scores (4.12 ± 0.28 / 4.01 ± 0.25) dominate baselines on this harder split. The paper notes PrismAudio sometimes exceeds ground truth on CLAP/DeSync because its rewards are direct optimization targets while the ground-truth audio has natural variations that the proxy metrics penalize — high MOS confirms this corresponds to better perceptual quality, not just metric gaming.

Why it’s interesting

PrismAudio is the first wiki entry that fully ports the GRPO-with-shaped-rewards recipe tracked under Reasoning RL into a flow-matching audio diffusion policy with a multi-dimensional reward composition. The closest existing wiki points are Human detectors are surprisingly powerful reward models (HuDA — GRPO on Wan 2.1 video diffusion with a single perception-stack reward) and Training-Free Group Relative Policy Optimization (training-free GRPO over context). PrismAudio’s contribution beyond HuDA is two-fold: (1) it stacks four perception-model rewards on disjoint perceptual axes and ties each reward to a matching CoT module (semantic/temporal/aesthetic/spatial), and (2) Fast-GRPO’s hybrid ODE-SDE sampler cuts policy NFE from O(T²) to O(T), making multi-reward diffusion RL tractable where Flow-GRPO and DanceGRPO would have been too expensive. The “single-dim rewards trade off; multi-dim rewards don’t” finding (Table 4) is the direct evidence that the reward decomposition is doing work beyond just summing scalars.

On the Joint audio-video generation axis, PrismAudio is the V2A-specialist counterpart to the joint-T2AV systems already tracked there (LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model) — those generate audio and video jointly from text, while PrismAudio generates audio from a silent video, a different problem with different evaluation conventions. Two observations: (a) PrismAudio’s spatial-accuracy (stereo / GCC / CRW) reward is the first time the wiki sees stereo directionality treated as a first-class objective in audio generation — most joint T2AV systems generate mono audio (LTX-2 is the open exception with 44k stereo); (b) PrismAudio’s Fast-GRPO is in principle stackable onto joint T2AV systems for multi-dimensional preference alignment of joint generators, which is currently an open question on this wiki. The architectural choice to replace CLIP with VideoPrism also echoes the broader “frame-independent image encoders are insufficient for video tasks” thread that runs through V-JEPA 2.1 (V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning) and the video-language-encoder lineage.