SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model

TL;DR

SkyReels-V4 is SkyReels’ third entry in the joint video–audio foundation-model race — a dual-stream MMDiT that pairs a video branch (initialized from a pretrained T2V model) with an audio branch (trained from scratch) sharing a single frozen MLLM text encoder. Beyond joint T2AV, the model unifies image-to-video, video extension, start-end interpolation, and video editing under a single channel-concatenation inpainting interface, and accepts text, images, video clips, masks, and audio references as conditioning. It targets cinematic settings — up to 1080p, 32 FPS, 15-second multi-shot clips with synchronized audio — using a two-pass strategy that generates low-resolution full sequences plus high-resolution keyframes, then runs dedicated super-resolution and frame-interpolation models. Reported as #1 on the Artificial Analysis Arena and beating proprietary systems on the authors’ SkyReels-VABench on instruction following, motion quality, and multi-shot narratives.

Key claims

A dual-stream MMDiT with one video branch and one audio branch, sharing a single MLLM text encoder, jointly generates synchronized video and audio under multi-modal instructions [§3.1, §Abstract].
The video branch is initialized from a pretrained T2V model while the audio branch is trained from scratch at matched architectural specifications — i.e. asymmetric training over symmetric architecture [§3.1].
Each transformer block transitions from a Dual-Stream layout (separate AdaLN/QKV/MLP per modality, joint self-attention) in early layers to a Single-Stream layout (shared parameters over concatenated tokens) in later layers, claimed to converge faster than either pure approach [§3.1, Eqs. 1–3].
A bidirectional audio-video cross-attention is inserted in every block so the audio stream queries video features and vice versa, exchanging synchronization cues throughout network depth [§3.1, Eq. 5].
Temporal scale mismatch between modalities (21 video frame latents vs 218 audio latent tokens for 5 s of 44.1 kHz audio) is handled by scaling the audio RoPE frequencies by 218/21 ≈ 10.4 so the two streams attend on a common temporal index [§3.1].
A unified video-side input formulation concatenates the noisy latent, VAE-encoded conditional frames (black-latent for non-conditional positions), and a binary spatiotemporal mask along the channel dimension, expressing T2V / I2V / video extension / start-end interpolation / video editing as different mask configurations within one model [§3.2, Eq. 7].
The inpainting mechanism is video-only; during inpainting/editing the audio branch always generates audio from scratch conditioned on the (partially conditioned) video, ensuring acoustic consistency with edits while keeping temporal sync via bidirectional cross-attention [§3.2].
Reference images and video clips are VAE-encoded and prepended to the noisy video latents before self-attention as in-context conditions, with negative temporal RoPE indices used to disambiguate condition tokens from generated tokens [§3.3, Eqs. 8–9].
The MLLM-based text encoder enables compositional multi-modal prompts that interleave image/video reference tokens with free text (e.g. @image_1 and <dialogue>...</dialogue> markers) and is consumed by both video and audio branches [§3.3].
To make 1080p × 32 FPS × 15 s computationally feasible, the model first jointly produces a low-resolution full sequence and high-resolution keyframes, then specialized super-resolution and frame-interpolation modules reconstruct the final video [§Abstract, §1].
Training is progressive over five video-pretrain stages: T2I at 256px → T2V at 256px → add I2V/V2V/inpaint/edit tasks (5% each) → mixed 256/480px → higher resolutions, with a separate audio pretrain and joint video–audio training phase before SFT [§4.1].
Audio data is uniformly captioned by Qwen3-Omni after category classification (sound effects / music / speech / singing), quality filtering (SNR, MOS, clipping, bandwidth), VAD-based silence filtering (<0.2 silence ratio), and duration normalization to 15 s chunks [§3.4.2].
Structured captions with explicit tags <text>, <sfx>, <dialogue>, <singing>, <bgm> are used in the final training stages, and a prompt enhancer reformats user free-form input into this structured representation at inference [§3.4.3].
Audio-video sync filtering uses SyncNet-derived confidence/offset over millions of clips, retaining only clips above a confidence threshold and ≥ −60 dB mean volume [§3.4.2].
Reported #1 on the Artificial Analysis Arena, with the authors’ SkyReels-VABench human-evaluation suite showing significant wins over proprietary commercial systems on instruction following, motion quality, and multi-shot narratives [§1, §Abstract].

Method

SkyReels-V4 is built on a symmetric twin-MMDiT backbone: parallel video and audio transformer towers with identical architectural specifications, the video one warm-started from a pretrained T2V checkpoint and the audio one trained from scratch. Each block runs (i) self-attention within its modality, (ii) text cross-attention against MLLM embeddings (video side only — added to combat semantic dilution of text in the later single-stream layers), (iii) bidirectional audio↔video cross-attention so each modality continuously perceives the other’s hidden state, and (iv) FFN. Early blocks keep per-modality parameters in a dual-stream layout; later blocks share parameters over concatenated tokens for parameter efficiency. The two modalities are kept on a common temporal coordinate by applying RoPE on both and rescaling the audio frequencies by the video/audio latent rate ratio (~10.4×) so attention is temporally consistent without intermediate projections.

The video-side input is a channel-concatenation of [noisy_latent, VAE(conditional_frames), mask], where the binary mask m ∈ {0,1} selects which spatiotemporal positions are conditions and which are to be generated. Different mask configurations recover T2V (all zeros), I2V (first frame ones), extension (first k frames ones), start-end interpolation (first and last ones), and arbitrary-region editing — all handled by the same forward pass. For vision-referenced generation, additional reference images/clips are VAE-encoded, padded to a uniform spatial size, and prepended to the noisy video latents before self-attention; condition tokens receive negative temporal RoPE indices so the model treats them as context rather than as part of the generated sequence. Audio references for the audio branch are handled by an analogous in-context conditioning step.

The model is trained with flow matching: given paired (video, audio) latents, sample timestep t, build noisy xt = (1-t) ε + t x0, and regress the velocity field. A frozen MLLM text encoder produces multi-modal embeddings from a single combined prompt (with image/video references inline), which both branches consume via self- and cross-attention. Training proceeds through T2I → low-res T2V → mixed inpainting/editing tasks → mixed-resolution scaling on the video side, plus a separate audio pretrain and a final joint video–audio phase followed by SFT. At inference, instead of generating native 1080p directly, the model jointly produces a low-resolution full sequence plus high-resolution keyframes; specialized super-resolution and frame-interpolation modules then reconstruct a temporally consistent 1080p / 32 FPS / 15 s output with synchronized audio.

Results

#1 on the Artificial Analysis Arena (T2AV leaderboard cited by the authors) [§1].
On the authors’ SkyReels-VABench human evaluation, SkyReels-V4 significantly outperforms proprietary commercial systems with reported strengths in instruction following, motion quality, and complex multi-shot narratives [§1, §Abstract].
Headline supported settings: up to 1080p resolution, 32 FPS, 15-second duration, multi-shot sequences with synchronized audio [§1, §Abstract].
Robust performance across diverse multimodal conditioning tasks — reference-to-video, motion-to-video, and video editing — validates the unified inpainting interface as a single foundation rather than a specialized pipeline [§1].
Comparison set explicitly named by the authors: Veo-3.1, Sora-2, Kling-2.6 / Kling-3.0 / Kling-Omni, Gen-4.5, Seedance-1.5 / Seedance-2.0, Wan-2.6, Vidu / Vidu-Q3 (proprietary T2AV/multimodal systems) and Omnihuman-1/1.5, SkyReels-A3, KlingAvatar, Multitalk (audio-driven talking-head systems) [§1, §2].
Numerical FID / FVD / lip-sync / Elo numbers were not present in the fetched portion of the paper — the available text emphasizes qualitative and arena results.

Why it’s interesting

This is the third joint-T2AV foundation model the team has filed this cycle, after LTX-2: Efficient Joint Audio-Visual Foundation Model and MOVA: Towards Scalable and Synchronized Video-Audio Generation. All three independently converge on a dual-stream DiT with bidirectional cross-attention at every block; SkyReels-V4 is the data point that pushes this from “emerging pattern” to “established recipe” and crosses the 3-paper threshold for both Joint audio-video generation and Dual-stream diffusion transformer concepts. The interesting axis of disagreement: LTX-2 splits capacity asymmetrically (14B+5B dense), MOVA goes asymmetric+MoE (32B/18B), and SkyReels-V4 goes symmetric in architecture but asymmetric in initialization (pretrained video tower + from-scratch audio tower). All three use bidirectional cross-attention; all three rescale audio positional encoding to match video time; all three claim joint training does not cost video quality.

SkyReels-V4’s distinctive contribution is the unified inpainting interface: channel-concatenation of noisy latent + condition latents + mask gives a single model that handles T2V, I2V, extension, interpolation, and editing as different mask configurations. This is conceptually cleaner than LTX-2’s pipeline-per-task approach and lines up with the in-context-conditioning philosophy used in Wan-Animate: Unified Character Animation and Replacement with Holistic Replication (where reference identity and motion control are also injected by latent prepend rather than as separate adapters). Worth watching whether this unified interface scales to longer-than-15-s videos without per-task degradation.

The two-pass cinematic-resolution recipe (low-res full sequence + high-res keyframes + dedicated SR + interpolation) is an explicit efficiency tradeoff worth flagging — it’s structurally similar to the latent-upscale + tile-refine pipeline LTX-2 uses for 1080p, but factored differently (keyframes vs full-sequence latent upscaling). The choice of dedicated SR/interpolation models as separate modules, rather than letting the base model handle all resolutions, is the kind of decision that may or may not survive contact with longer rollouts.