Skip to content
Bud's Wiki

Dual-stream diffusion transformer

TL;DR

Section titled “TL;DR”

DiT architectures with two parallel transformer streams — one per modality, or one per text-vs-content — coupled via cross-attention (and sometimes cross-modality AdaLN) at every block. Variants in the wiki are all instances of joint-audio-video DiTs: LTX-2 (asymmetric dense 14B video + 5B audio), MOVA (asymmetric MoE, 14B video + 1.3B audio backbone, 32B total / 18B active), and SkyReels-V4 (symmetric in architecture but asymmetric in initialization — pretrained video + from-scratch audio). The key architectural primitive is bidirectional cross-attention at every block; the open variables are capacity allocation, whether early/late blocks split or share parameters across modalities, and how the two modalities’ positional encodings are aligned.

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Introducing Cosmos 3 Edge: Cosmos 3 Edge shows the dual-tower AR + DM design (per-tower norm/MLP, shared multimodal attention) scales down to 4B parameters while keeping the same understanding+generation+action interface as its 32B flagship — an on-device datapoint for the pattern.
  • [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain uses a modality-aware MoT with language and vision as two attention branches communicating through global self-attention, plus hybrid causal-across-sequence + bidirectional-within-image attention on visual tokens.
  • [2026-07-10] Emerging Properties in Unified Multimodal Pretraining: BAGEL — Mixture-of-Transformer-Experts with shared self-attention every layer: understanding expert (text + SigLIP2 ViT) and generation expert (FLUX-VAE latents, rectified flow) hard-routed by token type. A 1.5B controlled study shows MoT (attn + FFN duplicated) outperforms both dense and MoE (FFN-only duplicated) at matched FLOPs, most notably on generation MSE loss.
  • [2026-07-08] LingBot-Video: Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence: Counter-datapoint: LingBot-Video explicitly rejects dual-stream for its base generator on systems grounds — argues single-stream forms larger unified GEMM computations, avoids per-block concat/split of condition and latent tokens, and simplifies sequence-parallel communication under distributed training, while relying on 3D MM-RoPE with non-overlapping temporal coordinate ranges to keep condition and visual tokens separable inside a shared parameter path.
  • [2026-07-08] RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation: RynnWorld-4D extends the dual-stream MMDiT pattern to a tri-branch DiT — one stream each for RGB, depth, and optical flow — coupled by cross-modal attention and unified by frame-wise 3D RoPE to keep appearance, geometry, and motion evolving consistently.
  • [2026-06-24] Krea 2 Technical Report: Krea 2 ablates single- vs dual- vs hybrid-stream (dual for first third of blocks, single for last two-thirds) and reports the hybrid design slightly outperforms the other two — but ships single-stream for simplicity. Concurs with the daVinci-MagiHuman / Z-Image “single-stream is sufficient at production scale” verdict while confirming the hybrid pattern’s small edge.
  • [2026-06-23] i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models: i1 (Princeton, Zhuang Liu lab) runs a controlled head-to-head between cross-attention, single-stream, and dual-stream backbones at matched config and picks dual-stream MMDiT with long skip connections as the best performance-parameter trade-off — the cleanest filed ablation behind the dual-stream choice. Sharpens the conflict with Z-Image / Ideogram 4.0 single-stream wins by isolating that the dual-stream advantage is specific to the recipe (T5Gemma-2B + 2-block transformer adapter + no AdaLN + long skip).
  • [2026-06-23] Causal World Modeling for Robot Control (LingBot-VA): LingBot-VA gives the cluster its first VLA datapoint: asymmetric video↔action dual-stream MoT with the video stream initialized from Wan2.2-5B (d=3072) and a narrower action stream at d_a, fused by projecting action Q/K/V up to the video width for joint self-attention and projecting back via residual. Action-stream weights are initialized by interpolating pretrained video weights and rescaled by √(d/d_a) to preserve output variance — fixes the training instability that occurs when the action stream is initialized from scratch. Extends the pattern from joint A+V (LTX-2, MOVA, SkyReels-V4) and text+video (Waver) into video+action.
  • [2026-06-18] MiniT2I: Pixel-Space Text-to-Image with MM-JiT at 258M Parameters: MM-JiT extends the dual-stream / MM-DiT text-conditioning pattern to a pixel-space JiT backbone, with text and image streams fused for T2I generation at 258M params.
  • [2026-06-16] Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments: Single-stream DiT flow-matching action expert (~1.15B params, 16 DiT blocks of 70.8M each) concatenates VLM hidden states with a noisy action chunk into one self-attention sequence, with AdaLN timestep conditioning and multi-section RoPE aligned to the Qwen3.5-4B backbone — the largest filed instantiation of the DiT-as-action-expert pattern, and an explicit single-stream choice for the action side of a VLM+action stack.
  • [2026-06-05] Ideogram 4.0 Technical Details: Open model at the forefront of design: Ideogram 4.0 ships as a 34-layer single-stream DiT where text and image tokens share projections at every layer — counter-evidence to the dual-stream design choice, alongside Z-Image’s S3-DiT. Adds a new wrinkle on the encoder side: hidden states from 13 intermediate layers of Qwen3-VL-8B-Instruct are concatenated along the feature dimension rather than reading a single hidden state.
  • [2026-06-01] Cosmos 3: Omnimodal World Models for Physical AI: Cosmos 3 generalizes dual-stream DiT from audio+video to a dual tower layer structure: each layer has a separate AR parameter set (for language + ViT-encoded understanding) and a DM parameter set (for VAE-encoded image/video, audio, and action), coupled by dual-stream joint attention — extending the pattern from modality-pair fusion to understanding-vs-generation routing.
  • [2026-05-30] Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE: Mamoda2.5’s AR-front-end + DiT-MoE backbone is an AR-Diffusion variant where the AR module supplies conditioning to a Top-8 / 128-expert diffusion transformer; one data point on how to wire an AR stream into a sparse DiT for unified understanding + image/video generation.
  • [2026-05-25] Mixture of States: Routing Token-Level Dynamics for Multimodal Generation: MoS (Mixture of States) — does not add a second stream, but is a direct contrast data point on the same problem (text-conditioning in a DiT). Single image-side denoiser with a per-token, per-timestep, top-k learnable router over the text encoder’s hidden-state layers; ε-greedy training. Claims SOTA on T2I and image editing at 3B–5B while matching 4×-larger counterparts. Shares LTX-2’s “one frozen text encoder layer is not enough” observation but resolves it with adaptive layer-wise routing rather than per-stream multi-layer connectors.
  • [2026-05-24] Waver: Wave Your Way to Lifelike Video Generation: Waver — ByteDance Foundation Vision’s 12B video-only (text+video) datapoint for the Hybrid Dual→Single-Stream layout. 16 Dual + 40 Single blocks over Wan-VAE latents, dual flan-t5-xxl + Qwen2.5-32B-Instruct text encoders, 3D RoPE + factorized learnable absolute position embeddings. Unifies T2I/T2V/I2V via 36-channel input concatenation (no head-level changes). Reports Hybrid Stream converges faster than either pure Dual or pure Single at 12B scale. Second independent paper to confirm the Hybrid layout (after SkyReels-V4) and the first one outside the joint-A+V setting — promotes the page from “joint-A+V technique” to “general DiT primitive.”
  • [2026-05-23] SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model: SkyReels-V4 uses a symmetric twin-MMDiT design (pretrained video + from-scratch audio backbones with matching architectural specs), with a hybrid Dual→Single stream transition through depth, bidirectional A↔V cross-attention at every block, and a reinforced text cross-attention layer in the video block after self-attention. Audio RoPE is rescaled by 218/21 ≈ 10.4× to match video temporal index. Third independent convergence on the dual-stream-with-bidirectional-cross-attention recipe.
  • [2026-05-22] MOVA: Towards Scalable and Synchronized Video-Audio Generation: MOVA pairs a 14B video DiT with a 1.3B audio DiT (operating on mel-spectrogram latents) in a 32B / 18B-active MoE configuration. Bidirectional cross-attention bridges the two backbones at every block; Aligned RoPE puts both modalities on a unified physical-time coordinate. Adds an MoE data point to the dual-stream pattern.
  • [2026-05-22] LTX-2: Efficient Joint Audio-Visual Foundation Model: LTX-2 — asymmetric dense 14B video + 5B audio dual-stream DiT. Each block runs self-attention, text cross-attention, audio-visual cross-attention (with only the temporal RoPE component on Q/K), then FFN. Adds cross-modality AdaLN as a second coupling mechanism on top of cross-attention. Baseline data point for the family.

Open questions

Section titled “Open questions”
  • Symmetric vs asymmetric width: is the 14B+5B split (LTX-2) information-density-correct, or is the 14B+1.3B split (MOVA) sufficient? SkyReels-V4 picks symmetric and lets pretraining-vs-from-scratch differentiate the two streams instead.
  • Hybrid Dual→Single stream transition (SkyReels-V4, Waver) vs strictly dual stream throughout (LTX-2, MOVA): two filed papers now report the hybrid converges faster (Waver §2.1 Fig. 5, SkyReels-V4 Eqs. 1–3), but neither runs a controlled pure-Dual vs pure-Single vs hybrid ablation at matched compute — open whether the speedup is the layout or co-located confounds.
  • The Waver datapoint extends the pattern to text+video (no audio): is the Hybrid layout doing the same work when there is no second sequence to align with, or is it a general convergence-acceleration trick for MMDiT regardless of the second-stream modality?
  • Per-stream RoPE with temporal-only cross-attention (LTX-2) vs unified-time rescaled RoPE (MOVA Aligned RoPE; SkyReels-V4 audio rescaling): two equivalent framings of the same alignment problem, or do they generalize differently when adding more modalities or longer sequences? Waver sidesteps this by being video-only and using 3D RoPE + factorized absolute embeddings.
  • Cross-modality AdaLN (LTX-2) as a coupling primitive beyond cross-attention — is it doing real work, or is it redundant with the cross-attention pathway? No ablations have been filed.
  • The dual-stream pattern is currently filed only as a joint-A+V technique. Does it generalize to other modality pairs (video+depth, video+IMU, video+control), or is the recipe specific to the video/audio sampling-rate mismatch? Partially answered: Waver shows the Hybrid Dual→Single layout transfers to a text+video setting at 12B scale. Open whether the pattern extends to denser second-modality streams like depth or IMU.
  • All three joint-A+V filed papers report joint training does not cost video quality — but the comparison baselines are different (Wan 2.2-14B for LTX-2, Ovi for MOVA, Veo/Sora-tier proprietary for SkyReels-V4). Open whether a matched-compute video-only comparison reveals a small joint-training tax that the released numbers don’t surface. Waver provides an indirect data point: it ranks Top-3 on Artificial Analysis as a video-only 12B Hybrid-Stream model, comparable to LTX-2 at 14B+5B and SkyReels-V4 at the joint-A+V scale.
  • New: dual-stream cross-attention vs single-stream layer-wise routing. MoS argues a 3B–5B single-stream model with a router over a frozen text encoder’s hidden-state layers matches 4×-larger counterparts on T2I/editing (Mixture of States: Routing Token-Level Dynamics for Multimodal Generation). The dual-stream cluster has not run a head-to-head against a per-token, per-timestep, per-layer router. Open: is the parameter cost of the second stream worth it at frontier scale, and does MoS-style adaptive layer reading work as the text-conditioning module inside a dual-stream A+V DiT (i.e. compose the two ideas)?

Papers

Section titled “Papers”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-05

Section titled “2025-05”