Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model

TL;DR

ByteDance Seed’s next-generation video model, repositioned as a native joint audio-visual foundation model rather than a T2V model with bolted-on audio. The recipe is a dual-branch (MMDiT) diffusion transformer with a cross-modal joint module, trained via a multi-stage data pipeline that scales audio-video coherence captions, then post-trained with SFT and multi-dimensional RLHF. Distinguishing capabilities are precise multilingual / dialect lip-sync, autonomous cinematic camera control (long takes, dolly zoom), and >10× inference acceleration via multi-stage distillation plus quantization and parallelism. Released on Volcano Engine alongside the technical report.

Key claims

Seedance 1.5 pro uses a unified MMDiT-based framework for joint audio-visual generation with a cross-modal joint module enabling deep visual↔audio interaction with precise temporal synchronization [§Architecture].
The model supports both text→audio-video and image→audio-video as different conditioning configurations of the same underlying architecture [Blog: model homepage].
A multi-stage data pipeline prioritizes audio-video coherence, motion expressiveness, and curriculum-based scheduling; captions are extended with audio descriptions on top of dense video captions [§Data].
Post-training is two-stage: SFT on high-quality audio-video data, followed by RLHF with multi-dimensional reward models targeting motion quality, visual aesthetics, and audio fidelity for both T2V and I2V tasks [§Post-training].
An acceleration framework based on a multi-stage distillation pipeline reduces NFE substantially; combined with quantization and parallelism it delivers end-to-end inference speedup exceeding 10× while preserving generation quality [§Inference].
The model achieves precise multilingual and dialect lip-sync, with native support for Chinese (including Sichuanese, Cantonese), English, Japanese, Korean, Spanish, Indonesian [Abstract; Blog §3].
Authors introduce SeedVideoBench 1.5, an evaluation benchmark co-developed with film directors covering visual axes (prompt following, motion stability/vividness, aesthetic quality) and audio axes (audio prompt following, A/V sync, audio quality, audio expressiveness) [Blog: Evaluations].
The release explicitly enumerates remaining failure modes — physical stability in complex motion, multi-character dialogue, and singing performance — rather than framing the model as solved [Blog: Summary].

Method

Seedance 1.5 pro is built on a unified MMDiT architecture with a dual-branch design: one branch processes video latents, the other processes audio latents, and a cross-modal joint module lets the two streams attend to each other so that temporal synchronization and semantic consistency are enforced inside the network rather than as a post-hoc alignment step. The model is pretrained on large-scale mixed-modality data using multi-task objectives so a single set of weights handles T2V, I2V, and the corresponding audio-conditioned variants.

The training pipeline has three notable stages. (1) A multi-stage data pipeline produces dense audio-visual captions and applies curriculum-based scheduling that prioritizes audio-video coherence and motion expressiveness. (2) SFT on high-quality curated audio-video pairs. (3) RLHF with a multi-dimensional reward model that scores motion, aesthetics, and audio fidelity separately for T2V and I2V. Inference uses a multi-stage distillation framework that compresses NFE alongside system-level quantization and parallelism, yielding >10× end-to-end speedup over the undistilled teacher.

Results

Authors report top-tier benchmark performance on their own SeedVideoBench 1.5, which they argue is more representative than off-the-shelf audio-video benchmarks because it was co-designed with film directors. On video metrics, the model shows strong prompt following for complex action and camera instructions and matches reference-image style under image-to-video conditioning, though motion stability remains an area for improvement. On audio metrics, the model ranks among the strongest industry models in audio prompt following, A/V sync, audio quality, and audio expressiveness, with particular strength in Chinese / dialect dialogue. The model is deployed on Volcano Engine. Concrete numerical comparisons against open baselines (LTX-2, MOVA, SkyReels-V4) are not included in the abstract or blog release notes available at filing time.

Why it’s interesting

This is the first closed-source frontier joint audio-video model to publish a technical report at the same time as the product release, and it lands in the middle of a tight cluster of open dual-stream MMDiT recipes — LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation, SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model — that have converged on bidirectional cross-attention between modality-specific streams; Seedance 1.5 pro independently describes the same dual-branch MMDiT + cross-modal joint module recipe, adding ByteDance’s vote to that consensus. It is also the first joint A+V paper on the wiki to disclose RLHF as its primary post-training lever (in contrast to LTX-2 which relies on data scale alone), and the >10× inference acceleration via “multi-stage distillation + quantization + parallelism” sits alongside FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU‘s purely systems-level FastVideo path and OmniForcing: Unleashing Real-time Joint Audio-Visual Generation‘s streaming-distillation path as a third deployment recipe — combining the algorithmic step reduction of Diffusion Distillation with system-level NVFP4-style quantization in one model.