UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation

TL;DR

UniForm is a unified diffusion transformer for text-to-audio-video (T2AV), audio-to-video (A2V), and video-to-audio (V2A) generation, using a single shared-weight DiT trunk over concatenated video and audio latent tokens — no per-modality towers, no cross-attention bridges. Task tokens select which of the three tasks the model performs; one model handles all three without per-task fine-tuning. The recipe is structurally different from the dominant dual-stream DiT pattern (LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model) and predates that consensus; it argues that a single diffusion process over concatenated tokens implicitly learns cross-modal correlations without explicit coupling layers.

Key claims

A single shared-weight DiT operating on concatenated vision and audio latent tokens can jointly handle T2AV, A2V, and V2A in one model, with a learned task-token embedding selecting the task at inference [§3.2, §3.2.2].
Concatenating modalities into a unified latent space, rather than running two coupled streams, is the central architectural choice — modalities undergo a single diffusion process together and cross-modal correlations are learned implicitly [§1, §3.2].
The trunk is STDiT3 (spatial + temporal blocks from Open-Sora), with a Multimodal Patch Embedder and 2D positional encoding over the unified token sequence; text conditioning enters through cross-attention in both spatial and temporal blocks [§3.2.2].
Loss masking lets one architecture train on three tasks: V2A masks all but audio tokens for the loss, A2V masks all but video tokens, T2AV masks only the task tokens — Eqs. (1)–(3) [§3.2.4].
Classifier-free guidance is applied by randomly dropping the text condition 50% of the time during training; the same model can therefore generate with or without text guidance for any of the three tasks [§3.2.4, §3.1.3].
On VGGSound V2A, UniForm reaches FAD 1.3 / FD 6.21, outperforming FoleyCrafter (FAD 2.51) and VATT (FAD 2.77) on those metrics despite being multi-task rather than V2A-specific [Table 1, §4.2].
On Landscape A2V, UniForm reports FVD 319 and IS 4.61 — leading the listed baselines on both metrics; AV-Align 0.51 is just below TempoToken’s 0.54 [Table 2, §4.3].
On joint Landscape T2AV, UniForm leads FVD (326), KVD (22.8), and AV-Align (0.30), and is within 0.1 FAD of the Discriminator-Guided Diffusion baseline (5.61 vs 5.52) [Table 3, §4.4].
Text prompts strongly improve both V2A and A2V on every reported metric — ablation in Table 4 shows e.g. V2A IS rises from 7.06 to 14.68 and FAD drops from 2.39 to 1.3 when text is provided [Table 4, §4.5].
Pre-trained encoders/decoders are reused without retraining: Open-Sora’s 3D-VAE for video, AudioLDM’s VAE for audio mel-spectrograms, HiFi-GAN for waveform reconstruction; DiT weights are initialized from PixArt-α [§4.1].

Method

UniForm encodes each input video with Open-Sora’s 3D-VAE and each audio (after STFT + mel-filter) with AudioLDM’s VAE, producing two sets of latent tokens. A task-ID embedding (T2AV / A2V / V2A) is concatenated with the video and audio latents; the joint sequence is patch-embedded with 2D positional encoding and passed through a stack of spatial and temporal STDiT3 blocks, with text features injected via cross-attention. Crucially the trunk is one network with shared weights — there is no separate “audio tower” and “video tower,” and no inter-modality cross-attention; both modalities live in a single attention pool throughout depth.

Training is multi-task: each sample is assigned one of the three tasks, the corresponding loss mask is applied to the noise-prediction objective (mask everything except audio tokens for V2A, except video tokens for A2V, only mask task tokens for T2AV), and 50% of the time text guidance is dropped to support classifier-free guidance. At inference the same network generates each task by setting the task-token embedding and (for A2V/V2A) replacing the unknown modality’s tokens with Gaussian noise. Decoding uses the same VAE decoders and HiFi-GAN as encoding. Training data are AudioSet-balanced, AudioSet-strong, VGGSound, and Landscape; videos are 4 s × 17 fps × 256² and audio is 16 kHz.

Results

V2A on VGGSound: FAD 1.3 (vs FoleyCrafter 2.51, VATT 2.77), FD 6.21 (vs VATT 10.63, FoleyCrafter 16.24), IS 14.68 (vs FoleyCrafter 15.68 first), KL 2.66 [Table 1].
A2V on Landscape: FVD 319 (lowest among listed baselines), IS 4.61 (highest), AV-Align 0.51 (vs TempoToken 0.54) [Table 2].
Joint T2AV on Landscape: FVD 326, KVD 22.8, AV-Align 0.30 — all leading; FAD 5.61 vs Discriminator-Guided Diffusion’s 5.52 [Table 3].
Text-conditioning ablation: removing text drops V2A IS from 14.68 → 7.06 and raises FAD from 1.3 → 2.39; A2V without text gets FVD 319 / IS 2.91 / AV-Align 0.32 vs 521 / 4.61 / 0.51 with text. The numbers suggest text helps audio sharpness for V2A and content diversity / sync for A2V, with a small FVD penalty when text is added on A2V [Table 4, §4.5].
The competing single-task baselines (FoleyCrafter, VATT, TempoToken, Discriminator-Guided Diffusion) are each restricted to one of the three tasks, so the comparison is single-task SOTA vs multi-task UniForm; UniForm is not fine-tuned per task [§4.2].

Why it’s interesting

This is the first-cluster recipe for joint A/V — single DiT trunk over concatenated tokens with task tokens — and it sits as a third architectural family next to the dual-stream consensus (Dual-stream diffusion transformer, used by LTX-2: Efficient Joint Audio-Visual Foundation Model / MOVA: Towards Scalable and Synchronized Video-Audio Generation / SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model) and the discrete tri-modal MDM of The Design Space of Tri-Modal Masked Diffusion Models. UniForm’s premise is the opposite of dual-stream’s: rather than carefully coupling two specialized towers, share one trunk over concatenated tokens and let the diffusion objective learn the cross-modal correlations implicitly. That the recipe predates and is structurally simpler than dual-stream is what makes it a useful contrast point for Joint audio-video generation — it raises the question of what dual-stream’s per-modality towers buy you that a single concatenated-token trunk cannot, particularly at small scale (UniForm’s DiT is initialized from PixArt-α image weights, not video).