IO-Aware Kernel Design

TL;DR

A recurring GPU-kernel design template, originating with FlashAttention, that treats HBM round-trips of intermediate tensors as the binding bottleneck — not arithmetic intensity. The recipe: identify an N × K intermediate that’s materialized in HBM only to be reduced along K; fuse the produce/consume pair into a streaming kernel with on-chip state; exploit an algebraic decomposition (softmax over blocks, argmax over partitions, segmented sums) that lets the reduction commute with tiling. The result is mathematically exact, not an approximation. Recent filed work generalizes the template off attention onto Lloyd’s k-means and LM-head categorical sampling.

Key claims

• HBM bandwidth, not tensor-core throughput, is the binding constraint for many GPU ops once tiles fit on chip; on Blackwell B200 the doubled BF16 throughput shifts the attention bottleneck onto SMEM traffic and the MUFU exponential unit, not the matmul (FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling §1, §2.2).
• The “fuse a producer kernel with its single-consumer reduction” pattern generalizes off attention: Flash-KMeans applies the same dataflow restructuring to Lloyd’s k-means, fusing distance computation with online argmin and the centroid update with sorted-segment reductions (Flash-KMeans: Fast and Memory-Efficient Exact K-Means §4.1, §4.2). FlashSampling applies it to LM-head decoding, fusing the matmul with Gumbel-max sampling so the B × V logits tensor is never materialized (FlashSampling: Fast and Memory-Efficient Exact Sampling §3).
• A sibling lever — exploiting algebraic structure of the intermediate matrices (symmetry, triangularity) rather than fusing produce/consume — gives a second route to IO-aware speedup. Gram Newton-Schulz rewrites the Muon Newton-Schulz iteration to operate mostly on the small symmetric Gram matrix $\mathbf X \mathbf X^\top$ instead of the large rectangular $\mathbf X$, and uses custom CuTeDSL symmetric-GEMM kernels (Hopper SM90 + Blackwell SM100) that write only the lower triangle (Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon §“Symmetric GEMM kernels in CuTeDSL”). Same hardware-co-design philosophy, distinct algebraic trick.
• The reductions that allow this fusion are all partition-decomposable: softmax-over-blocks for attention (FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling), online argmin + segmented sum for k-means (Flash-KMeans: Fast and Memory-Efficient Exact K-Means), and argmax-over-partitions plus hierarchical factorization of the categorical distribution for sampling (FlashSampling: Fast and Memory-Efficient Exact Sampling §2.3, §3).
• Outputs are mathematically exact — these are dataflow rewrites, not algorithmic approximations. Flash-KMeans is bit-identical Lloyd (Flash-KMeans: Fast and Memory-Efficient Exact K-Means §1, §4); FlashSampling is exact in distribution under the Gumbel-Max equivalence (FlashSampling: Fast and Memory-Efficient Exact Sampling §3); FA4 is exact softmax-attention with deterministic-mode support (FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling Abstract); Gram Newton-Schulz is mathematically identical to standard Newton-Schulz “up to floating-point error” and matches Muon training quality within 0.01 validation perplexity (Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon §“Stabilized Gram Newton-Schulz”).
• Tensor-parallel variants exploit the same algebraic property to replace expensive all-gathers of the full intermediate with cheap exchanges of partial-reduction state (FlashSampling: Fast and Memory-Efficient Exact Sampling §3, §4 — per-rank (max, argmax) scalars replacing logits all-gather; FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling §3.2 — 2-CTA MMA halves SMEM operand staging).
• End-to-end speedups land in the 1.3–200× range depending on what was being replaced and how much HBM traffic the baseline was wasting: FA4 ~1.3× over cuDNN, 2.7× over Triton on B200 (FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling Abstract); Flash-KMeans up to 17.9× over PyTorch, 33× over cuML, >200× over FAISS at the iteration level (Flash-KMeans: Fast and Memory-Efficient Exact K-Means §5.2); FlashSampling up to 19% E2E reduction in vLLM time-per-output-token (FlashSampling: Fast and Memory-Efficient Exact Sampling Abstract); Gram Newton-Schulz cuts the Muon orthogonalization step by 40–50% wall-clock and ~55–68% in FLOPs at typical aspect ratios (Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon §“GramMuon”, §“Gram Newton-Schulz” FLOP analysis).
• Implementation language matters for adoption: FA4 reports 20–30× faster compile times by moving from C++ templates to CuTe-DSL in Python (FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling §1); Flash-KMeans is in Triton with a cache-aware compile heuristic that picks tile sizes analytically instead of via 5-minute autotune sweeps (Flash-KMeans: Fast and Memory-Efficient Exact K-Means §4.3, §5.3); FlashSampling ships in Triton (FlashSampling: Fast and Memory-Efficient Exact Sampling §5); Gram Newton-Schulz is implemented in CuTeDSL for both Hopper (SM90) and Blackwell (SM100), ships as a drop-in for Muon’s Newton-Schulz routine (Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon §“Symmetric GEMM kernels in CuTeDSL”); Attn-QAT’s NVFP4 FA4 kernel is also CuTeDSL on B200 (Attn-QAT: 4-Bit Attention With Quantization-Aware Training Blog Part 2). The Dao-AILab compute-kernel stack (FA4 + Gram-NS + quack’s symmetric GEMMs + Attn-QAT’s FP4 FA4) is converging on CuTeDSL-in-Python as the new lingua franca.
• Half-precision arithmetic introduces failure modes that don’t appear in the math, and characterizing them is part of the design: FA4 uses an emulated $2^x$ on FMA units and conditional rescaling to keep softmax numerically tight (FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling §2.2); Gram Newton-Schulz exhibits exponential blowup in bfloat16 from spurious negative Gram-matrix eigenvalues and eigenvector drift, fixed by a periodic “restart” that re-projects onto the manifold of legitimate iterates (Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon §“Stability analysis”, Figs. 4–10). The IO-aware-kernel template now includes “numerical-stability investigation in the target precision” as a sub-task.
• Training-side co-design with the fused backward. Attn-QAT shows that the algebraic identity FlashAttention uses to keep the backward $O(N)$ in memory — $\mathbf{P}_i^\top \mathbf{dP}_i = \mathbf{dO}_i^\top \mathbf{O}_i$ — quietly assumes the forward pass computed $\mathbf{O}$ in the same precision as the gradient computation. Under fake-quantized FP4 forward ($\mathbf{P}^F, \mathbf{V}^F$) the identity silently breaks; the fix is to materialize a small auxiliary high-precision output $\mathbf{O}' = \mathbf{P}\mathbf{V}^F$ during the forward (25% extra storage) and substitute it into the backward, restoring the identity exactly (Attn-QAT: 4-Bit Attention With Quantization-Aware Training Blog §1). Combined with fake-quantizing the recomputed $\mathbf{P}$ in the backward to match the forward’s precision, this eliminates the need for inference-time outlier-mitigation heuristics (SageAttention3’s Q/K smoothing, two-level P quantization) and recovers BF16-attention quality on Wan-2.1-14B video diffusion and Qwen3-14B / Llama-3.1-70B continued pretraining and SFT. This is the first filed instance of the template being co-designed with a training recipe rather than only an inference primitive.
• “FP4 only where it’s free” as the corollary to the softmax bottleneck. On B200 the marketed 9 PFLOPS FP4 doesn’t translate to attention speedups uniformly: quantizing $\mathbf{PV}$ in addition to $\mathbf{QK}$ actually slows the kernel because the existing softmax bottleneck (MUFU exp + CUDA-core scale-factor computation) absorbs the extra group-quantization overhead. FA4-FP4 runs block-scaled NVFP4 on $\mathbf{QK}$ + BF16 on $\mathbf{PV}$ and reaches 1.39× over FA4 BF16 at long context (Attn-QAT: 4-Bit Attention With Quantization-Aware Training Blog §“TMEM overlap schedule”, Table — kernel benchmarks). The hardware roadmap (B300 doubling exp throughput, Rubin quadrupling) is the gating factor for moving more of the attention MMAs to FP4.
• The kernel bottleneck in learnable sparse attention is the compression branch. When NSA’s three-branch attention is run end-to-end on a video MLLM at 8K–128K context, wall-clock latency is dominated by the compression (CMP) branch — it grows roughly linearly with context length, while the selection and sliding-window branches stay cheap; this isolates CMP as the right target for the next round of IO-aware kernel work in the sparse-attention family (VideoNSA: Native Sparse Attention Scales Video Understanding §4 Finding 5).

Recent contributions

• [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 contributes a KDA-compatible prefix-cache implementation to vLLM — a serving-side kernel dependency for the hybrid-linear-attention backbone — and recommends supernode deployment (≥64 accelerators) so expert-parallel communication stays inside a single high-bandwidth domain.
• [2026-07-14] Inside TPU and GPU Clusters: The Anatomy of Collective Communication: Documents the multi-GPU-side hardware constraints that the compute-communication kernel work on this page (ThunderKittens ParallelKittens, FA4, SonicMoE, NVSHMEM) is designed to exploit: TPU ICI torus with per-hop latency and bandwidth hierarchy vs GPU fat-tree with NVSwitch SHARP in-network reduction and multicast. Concretely notes SHARP’s ~2× ideal All-Reduce speedup collapses to ~1.3× in practice on NCCL 2.29.7 / H100 IB — the specific gap that motivates kernel-level comm/compute overlap in Loads and Loads of Fluffy Kittens and NVSHMEM device-initiated communication.
• [2026-07-10] The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL: Concrete Blackwell/PTX kernel work for end-to-end NVFP4 RL: (a) per-token FP32 activation scaling fused into the quant kernel to avoid batch-shared scales; (b) a Four-Over-Six fast path using cvt.rn.f16x2.e2m1x2, cvt.rn.f16x2.e4m3x2, and mul.rn.f16x2 to keep FP4·FP8 multiplies lossless in FP16 registers (BF16 lacks mantissa bits), ~2.8× faster than the FP32-scalar reference at >99.97% scale-choice agreement; (c) TransformerEngine PR eliminating duplicate quantized tensor storage, cutting NVFP4 linear peak memory 150 MB → 45 MB (–70%) on a 2k×2k→8k proxy.
• [2026-06-30] waterloo_intern (ali / Baseten) — Worklog: retrofitting sparsity for the world's fastest video generation kernel (54×): Baseten engineer ali (@waterloo_intern) teases a worklog on retrofitting sparsity into a frontier OSS video generation model at inference time, claiming the world’s fastest video generation kernel and 54× cumulative speedup. Worklog details (sparsity pattern, kernel implementation, baseline) not retrievable at filing time — captured as a tweet-as-pointer until the full writeup is accessible.
• [2026-06-29] Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication: Consolidates the NVSHMEM design — symmetric memory + device-initiated put/get + IBGDA — that the compute-communication kernel work on this page (ThunderKittens, FA4, SonicMoE) sits one layer above; identifies where current runtime defaults (completion semantics, fence/quiet cost) leak into upstream kernel performance, complementing Hazy Research’s earlier observation that off-the-shelf NVSHMEM defaults are suboptimal.
• [2026-05-26] Laguna M.1/XS.2 Technical Report: Laguna fuses MoE all-to-all dispatch/combine into the CUTLASS grouped-GEMM kernel: 8 of 132 SMs per H200 copy expert tokens over NVLink in expert order and set HBM ready-flags, the tile scheduler waits on flags before processing, and the GEMM epilogue stores results directly to remote HBM via shared-memory vectorized stores — concrete production instance of the compute-communication-fusion thesis.
• [2026-05-24] VideoNSA: Native Sparse Attention Scales Video Understanding: Locates the wall-clock bottleneck inside NSA’s three-branch sparse attention — at 128K video-text context, the compression branch dominates runtime (linear-in-context wall clock), while selection and sliding-window remain cheap. Concrete kernel target for the IO-aware template applied to learnable sparse attention: fuse NSA’s per-block average-pooling + MLP projection with downstream gated combination so the block-level KV tensor is never materialized to HBM. Companion observation (Finding 6) that learnable sparse attention produces branch-specific attention sinks — compression manufactures them, selection eliminates them — adds a numerical-stability dimension worth tracking when designing the fused kernel.
• [2026-05-24] Attn-QAT: 4-Bit Attention With Quantization-Aware Training: extends the template to training-side co-design with low-precision fused attention. Identifies that FlashAttention’s memory-efficient backward identity ($\mathbf{P}_i^\top \mathbf{dP}_i = \mathbf{dO}_i^\top \mathbf{O}_i$) implicitly assumes forward and backward use matching precision; under fake-quantized FP4 forward the identity breaks and gradients become inconsistent. Two fixes: (1) materialize an auxiliary $\mathbf{O}' = \mathbf{P}\mathbf{V}^F$ during the forward (25% extra storage) and use it in the backward; (2) fake-quantize the recomputed $\mathbf{P}$ in the backward to match the forward precision. Result: FP4 attention matches BF16 on Wan-2.1-14B video diffusion (human 2AFC) and Qwen3-14B / Llama-3.1-70B continued pretraining + SFT, without any outlier-mitigation heuristics. Paired CuTeDSL kernel (FlashAttention-4 FP4 on B200) runs block-scaled NVFP4 on $\mathbf{QK}$ + BF16 on $\mathbf{PV}$ — the “FP4 only where it’s free” design point dictated by Blackwell’s softmax bottleneck — reaching 1801 TFLOPs/s and 1.39× over FA4 BF16 at long context.
• [2026-05-23] Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon: extends the template to optimizer-state computation. Gram Newton-Schulz rewrites Muon’s Newton-Schulz iteration so the dominant computation runs on the small symmetric Gram matrix $\mathbf X \mathbf X^\top$ instead of the rectangular momentum matrix, then exploits custom CuTeDSL symmetric-GEMM kernels (Hopper + Blackwell) that write only the lower triangle. 40–50% wall-clock reduction in the Muon orthogonalization step on trillion-parameter MoEs (Kimi K2-scale), with training quality preserved within 0.01 validation perplexity. Adds “structured-matmul exploitation” as a distinct mechanism alongside HBM-traffic elimination, and introduces a half-precision stability investigation (spurious negative eigenvalues amplified by $(15/8)^2$ per step, plus eigenvector drift) fixed by periodic restart.
• [2026-05-23] FlashSampling: Fast and Memory-Efficient Exact Sampling: extends the template to LM-head categorical sampling. Argmax-over-partitions plays the same role for sampling that softmax-over-blocks plays for attention; the categorical distribution’s hierarchical factorization enables a tensor-parallel variant that replaces all_gather(logits) with streaming per-rank (max, argmax) exchanges.
• [2026-05-23] Flash-KMeans: Fast and Memory-Efficient Exact K-Means: applies the template to Lloyd’s k-means. FlashAssign fuses streaming distance computation with online argmin (no N×K distance matrix in HBM); Sort-Inverse Update converts scatter-atomic centroid updates into argsort + segmented partial sums. Up to 200× over FAISS at iteration latency.
• [2026-05-23] FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling: re-tunes the original FlashAttention design for Blackwell. New roofline accounts for SMEM traffic and MUFU exp throughput as co-dominant with matmul; introduces software-emulated 2^x on FMA units, conditional softmax rescaling, and a backward pass that uses 2-CTA MMA to halve operand-B SMEM staging.

Open questions

• Where else does the template apply on the wiki? Any kernel that materializes an N × K intermediate solely to reduce along one axis is a candidate: top-k logits, MoE router gating, perceptual losses, RoPE applied before attention, contrastive logits. None of these have been filed yet as IO-aware rewrites. Partial answer: Gram Newton-Schulz is an adjacent variant — instead of fusing a produce/reduce pair, it rewrites the iteration so the dominant matmul is structured, and ships a kernel that exploits the structure. This suggests an enlarged template: “find an intermediate whose algebraic structure (symmetry, low rank, banded) is implicit but not exploited by standard libraries, restructure the algorithm to make the structure dominant, write the structured kernel.” VideoNSA adds a learnable sparse candidate: NSA’s compression branch is N × K (per-block mean-pool + MLP) → reduced by the per-head gate; fusing the pool + projection + gate combination is the obvious next target.
• Backward-pass coverage. FA4 has a backward pass; Flash-KMeans, FlashSampling, and Gram Newton-Schulz are inference / online primitives only. Attn-QAT is the first filed paper that targets the backward pass head-on, showing that the FlashAttention backward identity implicitly assumes forward/backward precision match — and that under low-precision training this assumption must be made explicit by carrying a small high-precision auxiliary output. Whether the template applies to backward passes that don’t have an obvious partition decomposition (e.g. CE loss + LM-head backward, which is exactly the locus that Lost in Backpropagation: The LM Head is a Gradient Bottleneck flags as a bottleneck) is still unaddressed.
• Compile-time as the bottleneck. All filed papers either rebuild compile infrastructure (FA4’s, Gram-NS’s, and Attn-QAT’s CuTeDSL moves) or replace autotune sweeps with analytical heuristics (Flash-KMeans’s cache-aware config selector). The implicit claim is that iteration speed on the kernel-design loop itself is now load-bearing for architecture-specific tuning at the rate Blackwell/Hopper/B300 ship; whether this means Python-DSL becomes the dominant paradigm or whether C++-template + cache modeling wins is open. Dao-AILab’s stack (FA4 + Gram-NS + quack/gemm_symmetric + Attn-QAT FA4-FP4) is the strongest existing endorsement of the CuTeDSL-in-Python direction.
• Approximation-as-extension. Three of the original four papers emphasize exactness. The natural next axis is controlled approximation — block-sparse attention, approximate k-means with bounded error, top-k sampling — built on the same kernel skeleton. The wiki has MonarchRT: Efficient Attention for Real-Time Video Generation as one such datapoint on the attention side; Attn-QAT is the precision-side version of approximation — quantization is a controlled approximation, and the training-time fix lets it match BF16 in practice. VideoNSA / NSA is the learnable sparse version of the same axis: sparsity pattern + gating are trained, not statically chosen.
• Numerical stability in half precision. Gram Newton-Schulz’s restart trick is the explicit instance of this: bfloat16 spurious negatives blow up exponentially via $r_t < (15/8)^2 r_{t-1}$. FA4’s conditional softmax rescaling addresses a milder version of the same problem. Attn-QAT adds a third instance: the FlashAttention backward identity silently breaks under precision mismatch, and only the auxiliary $\mathbf{O}'$ trick + matched backward recomputation restores it. Whether other applications of the template (Flash-KMeans, FlashSampling) have undiagnosed half-precision pathologies is open.
• Where does the FP4 attention story go next? Attn-QAT’s design point on B200 — NVFP4 $\mathbf{QK}$ + BF16 $\mathbf{PV}$ — is dictated by the softmax bottleneck; B300 doubles MUFU exp throughput and Rubin quadruples it, which should make NVFP4/MXFP8 $\mathbf{QK}$ + FP8 $\mathbf{PV}$ a net win. The open meta-question is whether the QAT recipe needs to be re-derived per hardware generation, or whether the precision-consistency fixes (auxiliary output + matched backward recomputation) transfer.

Papers

2026-05

• TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times — TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times (published 2026-05-25)
• FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling — FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling (published 2026-05-23)
• LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation — LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation (published 2026-05-18)
• Interaction Models: A Scalable Approach to Human-AI Collaboration — Interaction Models: A Scalable Approach to Human-AI Collaboration (published 2026-05-11)
• How to Make LLM Training Faster with Unsloth and NVIDIA — How to Make LLM Training Faster with Unsloth and NVIDIA (published 2026-05-06)

2026-04

• Attn-QAT: 4-Bit Attention With Quantization-Aware Training — Attn-QAT: 4-Bit Attention With Quantization-Aware Training (published 2026-04-08)
• Better MoE model inference with warp decode — Better MoE model inference with warp decode (published 2026-04-06)
• Scaling and Optimizing Frontier Model Training (Fireworks AI) — Scaling and Optimizing Frontier Model Training (Fireworks AI) (published 2026-04-03)

2026-03

• Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon — Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon (published 2026-03-30)
• Composer 2 Technical Report — Composer 2 Technical Report (published 2026-03-25)
• Flash-KMeans: Fast and Memory-Efficient Exact K-Means — Flash-KMeans: Fast and Memory-Efficient Exact K-Means (published 2026-03-10)
• Lost in Backpropagation: The LM Head is a Gradient Bottleneck — Lost in Backpropagation: The LM Head is a Gradient Bottleneck (published 2026-03-10)

2026-02

• FlashSampling: Fast and Memory-Efficient Exact Sampling — FlashSampling: Fast and Memory-Efficient Exact Sampling (published 2026-02-28)
• MonarchRT: Efficient Attention for Real-Time Video Generation — MonarchRT: Efficient Attention for Real-Time Video Generation (published 2026-02-12)

2026-01

• 3x Faster LLM Training with Unsloth Kernels + Packing — 3x Faster LLM Training with Unsloth Kernels + Packing (published 2026-01-03)

2025-12

• SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations — SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations (published 2025-12-16)
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling — Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling (published 2025-12-01)

2025-11

• Loads and Loads of Fluffy Kittens: Compute-Communication Kernels with Multi-GPU ThunderKittens — Loads and Loads of Fluffy Kittens: Compute-Communication Kernels with Multi-GPU ThunderKittens (published 2025-11-17)

2025-10

• Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention — Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention (published 2025-10-05)
• VideoNSA: Native Sparse Attention Scales Video Understanding — VideoNSA: Native Sparse Attention Scales Video Understanding (published 2025-10-02)
• lmms-engine — A simple, unified multimodal models training engine — lmms-engine — A simple, unified multimodal models training engine (published 2025-10-01)

2025-09

• BackendBench: Ship Correct and Fast LLM Kernels to PyTorch — BackendBench: Ship Correct and Fast LLM Kernels to PyTorch (published 2025-09-10)

2025-07

• MirageLSD: The First Live-Stream Diffusion AI Video Model — MirageLSD: The First Live-Stream Diffusion AI Video Model (published 2025-07-17)

2025-06

• Sparsity is Cool: Reverse-Engineering MoBA and NSA — Sparsity is Cool: Reverse-Engineering MoBA and NSA (published 2025-06-25)
• Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation — Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation (published 2025-06-24)

2025-05

• Test-Time Training Done Right (LaCT) — Test-Time Training Done Right (LaCT) (published 2025-05-29)
• Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation — Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation (published 2025-05-24)

2025-04

• microsoft/dion — distributed Muon, Dion2, Dion, and NorMuon orthonormal optimizers — microsoft/dion — distributed Muon, Dion2, Dion, and NorMuon orthonormal optimizers (published 2025-04-08)

Unknown date

• HeavyBall — optimizer library and diagnostic benchmark for silent optimizer failures — HeavyBall — optimizer library and diagnostic benchmark for silent optimizer failures (published unknown)
• Kittens virtual machine fuses entire training runs into a single GPU kernel (Ben Spector / Hazy Research) — Kittens virtual machine fuses entire training runs into a single GPU kernel (Ben Spector / Hazy Research) (published unknown)