Mamba-3: Improved Sequence Modeling using State Space Principles

TL;DR

Mamba-3 redesigns the Mamba SSM block from an inference-first perspective, targeting the fact that prior linear models (Mamba-2, Gated DeltaNet) simplified the recurrence so aggressively for training speed that decoding became memory-bound. Three SSM-centric upgrades — an exponential-trapezoidal discretization (more expressive recurrence), a complex-valued state update (richer state tracking, implemented via RoPE on B/C), and a multi-input multi-output (MIMO) projection (more parallel work per step at near-free decode cost) — close most of the remaining quality gap to Transformers while improving prefill+decode latency over Mamba-2, Gated DeltaNet, and vLLM Llama-3.2-1B at the 1.5B scale. The short causal convolution that every previous Mamba/GDN block needed is dropped entirely; the architecture is otherwise modernized with QKNorm-equivalent BCNorm and interleaved MLPs.

Key claims

Mamba-3 outperforms Mamba-2 and Gated DeltaNet on language-modeling benchmarks across pretrained scales; the MIMO variant adds a further >1pp downstream-accuracy lift at 1B over plain Mamba-3 SISO [Empirical Results, Language Modeling].
The exponential-trapezoidal discretization is a second-order-accurate analog of Mamba-2’s first-order Euler step and induces a structured mask that is the product of a decay and a two-band convolutional mask — eliminating the need for the explicit short causal convolution kept by Mamba-1/2 and Gated DeltaNet [§3.1, Fig. 1].
A complex-valued SSM transition (interpreted as rotations) is realized without rewriting kernels by applying RoPE to the B and C projections; this gives richer state-tracking dynamics at the same throughput [Architecture, “New Components”].
MIMO formulation processes vector-sized inputs $x(t) \in \mathbb{R}^R$ per recurrence step instead of scalar inputs, raising per-step FLOPs but leaving decode latency essentially unchanged because decode is memory-bound and the extra FLOPs fill otherwise-idle tensor cores [Empirical Results, “How can training costs go up but not inference?”].
At 1.5B / H100-SXM / batch 128, Mamba-3 SISO achieves the lowest prefill+decode latency across 512–16384 token sequences, beating Mamba-2, Gated DeltaNet, and vLLM Llama-3.2-1B; Mamba-3 MIMO (r=4) matches Mamba-2 latency at strictly higher quality [Benchmarking Latencies, prefill+decode table].
Mamba-3 matches strong LLM baseline perplexities at roughly half the decoding cost of comparable Transformers [Tl;dr].
BCNorm (RMSNorm on projected Q/K i.e. B/C activations) stabilizes Mamba-3 training and brings the block in line with QKNorm-equipped Transformers and Gated DeltaNet; with BCNorm in place, Mamba-2’s outer RMSNorm becomes optional but still helps length extrapolation in hybrid stacks [Architecture, “Norms”].
Linear models remain underperforming on retrieval relative to Transformers even at Mamba-3 quality, so the recommended deployment is hybrid — Mamba-3 layers interleaved with a small number of global self-attention layers, which beats both pure-Transformer and pure-linear configurations [Empirical Results, Retrieval Tasks].
The kernel stack uses three GPU DSLs at different abstraction levels — Triton, TileLang, and CuTe DSL — chosen per-component so the simpler Mamba-3 primitives can hit hardware peak; Mamba-3 SISO Triton prefill matches Mamba-2’s, Mamba-3 MIMO decode beats it via CuTe DSL [Kernels Here, There, and Everywhere].

Method

The Mamba-2 SSM recurrence is rebuilt around three independent SSM-level upgrades, all motivated by the observation that decode is memory-bound and any extra per-step computation is roughly free until the tensor cores saturate. (1) Exponential-trapezoidal discretization: instead of converting the continuous-time SSM ODE via a first-order Euler step (which is what Mamba-2’s scalar-times-identity transition implements), the discretization uses an exponential-trapezoidal rule that gives a second-order-accurate update. The induced attention-equivalent mask factors as decay × two-band convolution, so the short causal conv that every previous Mamba/GDN block stacked outside the recurrence is now subsumed inside the recurrence and can be removed. (2) Complex-valued state via RoPE on B/C: a complex-valued transition matrix corresponds to a rotation; this is implemented by applying RoPE-style rotary projections to the data-dependent B and C matrices, so the kernels never see complex numbers and only the projection layers change. (3) MIMO: the per-dimension SISO recurrence is generalized to operate on vector-sized inputs of width R, with B and C projections expanded accordingly. This raises the per-step matmul size (training cost grows) but does not raise decode latency until the tensor cores stop being idle.

The full block stack adds BCNorm (RMSNorm on B and C, the SSM equivalent of QKNorm), interleaves MLPs in the standard Transformer pattern, and is co-designed with CuTe DSL kernels for decode and TileLang for the MIMO prefill path. Hybrid deployment with a small number of global self-attention layers is recommended to close the retrieval gap. The release includes the paper, open-source kernels in state-spaces/mamba, and the GoombaLab cross-post; the model is positioned as the default replacement for Mamba-2 in hybrid stacks.

Results

Language modeling: Mamba-3 (both SISO and MIMO) ahead of Mamba-2 and Gated DeltaNet across pretrained scales; MIMO adds >1pp downstream accuracy over SISO at 1B [Language Modeling].
Prefill+decode latency, 1.5B / H100-SXM 80GB / batch 128 / wall-clock seconds, three reps (Table from blog): at n=16384, Mamba-2 = 149.02s, Gated DeltaNet = 145.87s, Mamba-3 MIMO r=4 = 151.81s, vLLM (Llama-3.2-1B) = 976.50s; Mamba-3 SISO row not numerically shown in the blog excerpt but text reports it as the fastest across all lengths [Benchmarking Latencies].
Prefill-only latency, n=16384: Mamba-2 = 16.22s, Mamba-3 SISO = 16.22s (parity), Mamba-3 MIMO r=4 = 19.44s (~20% prefill slowdown for the >1pp accuracy gain) [Benchmarking Latencies].
Retrieval: pure Transformer still best on retrieval; Mamba-3 strongest among sub-quadratic alternatives; hybrid Mamba-3 + self-attention beats pure Transformer at the same compute on retrieval [Retrieval Tasks].
Quality vs decode cost: matches strong LLM baseline perplexities at ~½ decoding cost [Tl;dr].
Kernel parity: Mamba-3 SISO Triton kernel speed is on par with Mamba-2’s, and the new discretization + RoPE on B/C do not add overhead; Mamba-3 MIMO is faster than Mamba-2 at decode via the CuTe DSL implementation [Kernels Here, There, and Everywhere].

Why it’s interesting

Mamba-3 is the first time the SSM-track linear-model story has been pitched explicitly around the inference-bound regime the rest of the wiki is converging on. It directly complements Inference-Time Hyper-Scaling with KV Cache Compression (which makes Transformers cheaper at decode via KV-cache compression): Mamba-3 attacks the same bottleneck from the other side, with a fixed-size hidden state and an inference-first architecture. The hybrid-deployment recipe also sharpens the framing in Long context as weights, which lists Mamba 2 / Gated DeltaNet as fixed-state-size baselines that are distinct from KV-cache compression and from test-time weight updates — Mamba-3 raises the quality floor of that baseline class enough that the “linear core + sparse global attention” pattern becomes a serious alternative to attention+compressed-cache. It is also the natural follow-on to Mamba-2 dt_bias initialization bug in HuggingFace transformers and flash-linear-attention (which documented a Mamba-2 init bug that silently degrades 7B-MoE training in HF/FLA): Mamba-3 removes the short causal conv and adds BCNorm, so several of the Mamba-2 sensitivity surfaces — including dt_bias-style targeted inits guarded only by wrapper-class scope — get reshaped, though the new RoPE-on-B/C path may introduce its own framework-level footguns worth watching.