MonarchRT: Efficient Attention for Real-Time Video Generation

TL;DR

MonarchRT parameterizes 3D video-attention maps as Monarch matrices (a structured factorization that strictly generalizes sparse + low-rank), rather than approximating them with top-k sparsity. The key empirical claim is that 3D video attention is not reliably sparse — it mixes pronounced spatiotemporal-periodic structure, sparse semantic links, and dense mixing — so even oracle top-k attention fails at the high-sparsity regimes needed for real-time, few-step, autoregressive video DiTs. By aligning Monarch block sizes with the (f, h, w) video latent shape, introducing a tiled Monarch parameterization that gives monotonic refinement with compute, and finetuning to collapse iterative refinement to one step, MonarchRT reaches 95% effective attention sparsity on Self-Forcing with no VBench loss, and 16 FPS true-real-time generation on a single RTX 5090 (vs. 11 FPS with FA-2).

Key claims

Video attention combines (i) separable spatiotemporal periodic structure $\mathbf{D}_{(f_0,h_0,w_0),(f_1,h_1,w_1)} = d_f(f_0,f_1)\,d_h(h_0,h_1)\,d_w(w_0,w_1)$ , (ii) sparse semantic correspondences $\mathbf{S}$ , and (iii) dense mixing — and is not well-modeled by pure sparsity [§3.2].
Oracle top-k attention requires >50% of full-attention FLOPs to retain quality on real-time AR video; even with that budget it causes visible geometric distortion (e.g. car-front collapse in motivating example) [§3.1, Fig. motivating_example_generations].
For block sizes $(b_1,b_2)$ with $N=fhw=b_1 b_2$ , a Monarch parameterization can exactly represent the separable positional component only if each spatiotemporal axis lies entirely inside one of the two block sizes — yielding exactly six valid aligned configurations: $(fh,w),(w,fh),(f,hw),(hw,f),(fw,h),(h,fw)$ [§4.1].
Misaligned block choices (e.g. flattened-index splits) destroy approximation quality even at higher parameter count — empirically shown to collapse video quality despite identical FLOPs as aligned choices [§3.4.1, Fig. 7].
Tiled Monarch parameterization subdivides each Monarch block into $c_1 \times c_2$ sub-tiles, each independently rank-1; this set strictly contains untiled Monarch ( $\mathcal{M}(b_1,b_2)\subsetneq\mathcal{M}_{\mathrm{tile}}(b_1,b_2;c_1,c_2)$ for $c_1>1$ or $c_2>1$ ) and gives monotonic accuracy improvement with compute [Theorem 4.1, §4.2].
The MonarchAttention iterative-refinement runtime is collapsed by finetuning the model with 1 refinement iteration, matching the quality of much longer refinement chains [§4.3].
On Self-Forcing with MonarchRT injected at the DMD distillation stage at 95% effective attention sparsity: VBench Quality 0.846 vs. dense 0.844, Semantic 0.805 vs. 0.804, Total 0.838 vs. 0.836 [Table — VBench / Self-Forcing].
Kernel-level wall-clock speedups (480p, 95% sparsity) over FlashAttention: 3.9× on RTX 5090 (4.97 → 1.27 ms), 1.5× on H100 (1.58 → 1.04 ms) [Table — kernel timings].
End-to-end with torch.compile at 95% sparsity on RTX 5090 480p: FA-2 11 FPS → MonarchRT 16 FPS — first true real-time 16 FPS Self-Forcing generation on a single consumer 5090 [§5, summary table].
At 720p bidirectional attention on RTX 5090 with 98% effective sparsity, MonarchRT reports an 11.8× attention speedup over FA-2 [§5 extrapolation].

Method

The method has three parts. (1) Structural insight. The paper models the post-softmax 3D attention matrix as $\mathbf{A} = \mathbf{D} + \mathbf{S} + \varepsilon$ , where $\mathbf{D}$ is the separable positional term and $\mathbf{S}$ is a sparse semantic term. Under an appropriate permutation $\mathbf{P}$ that groups tokens by spatiotemporal structure, the positional component becomes block-wise rank-1, which Monarch matrices ( $\mathbf{M} = \mathbf{P}\mathbf{L}\mathbf{P}^\top\mathbf{R}$ , with $\mathbf{L},\mathbf{R}$ block-diagonal) can represent exactly.

(2) Tiled Monarch + alignment. The Monarch block sizes $(b_1, b_2)$ with $b_1 b_2 = fhw$ must keep each video axis ( $f$ , $h$ , $w$ ) inside exactly one of the two factors — only six valid configurations exist. To enable monotonic refinement beyond these six points, each Monarch block is further subdivided into $c_1 \times c_2$ independently rank-1 sub-tiles; this set strictly contains untiled Monarch and can represent attention blocks of rank > 1 (e.g. a 2-spike semantic block) that standard Monarch cannot.

(3) Finetuning + Triton kernels. MonarchAttention’s iterative alternating-maximization for the factors $\mathbf{L},\mathbf{R}$ is collapsed to a single iteration by finetuning the underlying model end-to-end with the 1-iteration approximation in the loop. Custom Triton kernels (forward + backward) implement the tiled-Monarch attention efficiently and replace FA-2/3/4 in the hot path. The kernel exploits the block structure to avoid ever materializing the dense $N \times N$ attention matrix.

Results

Quality on Self-Forcing at 95% effective sparsity matches or slightly exceeds dense attention across VBench Quality / Semantic / Total [Table — VBench]. Kernel speedups over FlashAttention at 480p / 95% sparsity: 3.9× on RTX 5090, 1.5× on H100 [Table — kernel timings]. End-to-end with torch.compile: 11 FPS (FA-2) → 16 FPS (MonarchRT) on 5090 480p Self-Forcing — first true real-time on consumer hardware. Bidirectional 720p on 5090 at 98% sparsity reports up to 11.8× kernel speedup over FA-2. The method is also evaluated on the bidirectional Wan 2.1-1.3B and matches sparse baselines at comparable quality.

Why it’s interesting

MonarchRT is the first attention-approximation paper filed here that targets the specific brittle regime created by the Self-Forcing / Causal Forcing / Context Forcing lineage — few-step + autoregressive video DiTs where prior sparse approximations break down. The structural insight (3D attention = separable positional + sparse semantic + dense mixing) is sharper than the “video attention is sparse” assumption underlying Radial Attention, SparseVideoGen, SLA, etc., and explains why those approaches plateau around 50% FLOPs. The six-aligned-configuration result is a clean, falsifiable design rule that constrains the search space for any future structured-attention work on video. Pairs naturally with Context Forcing: Consistent Autoregressive Video Generation with Long Context‘s Slow-Fast Memory cache (both exploit the same surprisal/locality intuition at different levels) and could drop into the FlowAct-R1: Towards Interactive Humanoid Video Generation 3-NFE distillation pipeline as the attention primitive.