Skip to content
Bud's Wiki

Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

TL;DR

Section titled “TL;DR”

Chimera is Adobe’s hybrid visual diffusion backbone unifying text, image, and video tokens in a single raster-ordered stream with no positional embeddings — the modality routing lives entirely in the architecture rather than in per-stream parameters. Attention is a KDA (Kimi Delta Attention) + interleaved MLA (Multi-head Latent Attention) hybrid, with modality-aware short convolutions for local spatiotemporal context and sparse MoE for capacity expansion at controlled active-compute. To tune the heterogeneous stack, the paper introduces HeteroP, a module-wise µP-style hyperparameter-transfer scheme keyed on each tensor’s functional fan-in and depth position. A HeteroP-tuned family fits Chinchilla-style compute-optimal laws jointly over activated size, training tokens, and image/video data ratio, guiding an 11B / 2B-active production run that reports 1.7× (dense-only) and 7.3× (full system) pretraining-loss compute-efficiency over a Wan-2.1 2B full-attention baseline and zero-shot temporal extrapolation from 5s training clips to 30s videos.

Key claims

Section titled “Key claims”
  • Chimera runs text, image, and video tokens as one raster-ordered stream with no positional embeddings, combining KDA (linear, O(N) state tracking), interleaved MLA (softmax global anchor), modality-aware short convolutions for local spatiotemporal context, and sparse MoE for parameter expansion at controlled activated compute [§Abstract, §Method].
  • HeteroP is a module-wise hyperparameter-transfer scheme that scales each tensor’s learning rate / init by its functional fan-in and depth position, rather than applying a single µP scaling factor across the whole network — designed specifically to handle the heterogeneous KDA/MLA/conv/MoE stack [§Abstract, §HeteroP].
  • HeteroP-tuned reference runs fit a Chinchilla-style joint scaling law over three axes — activated model size, training-token count, and image-to-video data ratio — enabling compute-optimal allocation decisions rather than the standard two-axis (N, D) fit [§Abstract, §Scaling Laws].
  • On matched compute against a Wan-2.1 2B full-attention baseline, Chimera reports 1.7× compute-efficiency for the dense backbone alone and 7.3× for the complete system (dense + MoE + hybrid attention) as measured by pretraining diffusion loss [§Abstract, §Results].
  • The 11B / 2B-active production model extrapolates zero-shot from 5-second training clips to 30-second video generation, with FID degrading only 6.5% in the last five seconds — no length-specific fine-tuning required [§Abstract, §Results].
  • The fitted scaling laws show compute-optimal image pretraining splits compute nearly evenly between activated model size and training tokens, whereas video pretraining modestly favors model size at higher budgets — a first filed image-vs-video allocation asymmetry within one law [§Abstract, §Scaling Laws].

Method

Section titled “Method”

Chimera’s core architectural claim is that the standard dual-stream (per-modality parameters coupled by cross-attention) or MoT (modality-of-Transformer routing) recipes are unnecessary when the attention primitive itself is heterogeneous enough. All modalities flow through one shared parameter path, raster-ordered, with no positional embeddings. Modality separation is carried by a combination of KDA’s fixed-state linear-attention (long-context state tracking with O(N) complexity), interleaved MLA (Multi-head Latent Attention, softmax global anchor for direct cross-token interaction), and modality-aware short convolutions that handle local spatiotemporal structure within each modality’s raster region. Capacity is expanded via sparse Mixture-of-Experts layers.

The scaling side introduces HeteroP: rather than apply µP’s single width-transfer factor, each tensor gets a learning-rate scaling determined by its module type’s functional fan-in and its depth position in the stack. This yields a consistently tuned family of small-to-large reference runs that can be used to fit standard Chinchilla isoFLOP curves. The paper fits a three-axis law over activated parameters, training tokens, and the image/video mixture ratio, then uses the fit to allocate compute for the 11B / 2B-active flagship run.

Results

Section titled “Results”
  • Compute efficiency vs full-attention baseline (pretraining diffusion loss): 1.7× dense-only, 7.3× complete system, against Wan-2.1 2B full-attention at matched compute [§Abstract].
  • Zero-shot temporal extrapolation: trained on 5-second clips, generates 30-second video with 6.5% FID degradation in the last five seconds — no length-specific fine-tuning [§Abstract].
  • Modality-specific compute-optimal allocation: image pretraining wants roughly equal split between activated model size and training tokens; video pretraining favors model size more strongly at higher compute budgets [§Abstract].
  • Flagship configuration: 11B total / 2B activated parameters, trained under the fitted compute-optimal image + video mixture [§Abstract].

Why it’s interesting

Section titled “Why it’s interesting”

Chimera is the first filed paper on the wiki that composes the LLM-side hybrid-linear-attention consensus (KDA + MLA at a fixed ratio, as in Kimi Linear: An Expressive, Efficient Attention Architecture) with a diffusion-side video training objective, into a single production visual-generation stack — sharpening the trend already visible in SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation (which independently picked a 3:1 linear:softmax ratio for video DiT) that KDA/MLA hybrids are not LLM-specific and are now the default attention primitive for compute-optimal long-context video generation. The HeteroP scheme adds a third position to the wiki’s Fitting/Transfer/Enforce fork (see Hyperparameter scaling laws): module-wise µP variants that adapt the transfer factor per tensor rather than per network, an idea structurally similar to Beyond MuP 3: Special Cases, Special Treatment (Embedding, LM Head, RMS Norm)‘s per-module treatment of embedding / LM head / RMS-norm. The single-stream-with-heterogeneous-attention design is also a direct counterpoint to the dual-stream cluster (LTX-2: Efficient Joint Audio-Visual Foundation Model, MOVA: Towards Scalable and Synchronized Video-Audio Generation) — Chimera argues you don’t need per-modality streams if the attention primitive can carry the modality separation, complementing single-stream video-only precedents like Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model (daVinci-MagiHuman) but extending them to text+image+video jointly. The video/image-mixture axis in the scaling law is a rare filed instance of a modality-mixture dimension being fit jointly with (N, D) — most existing entries on Hyperparameter scaling laws hold the data mixture fixed.

See also

Section titled “See also”