Tilde Research — Wall Attention: length generalization via diagonal forget gates in softmax attention

TL;DR

Tilde Research announces Wall Attention, a softmax-attention variant that replaces RoPE with per-channel diagonal forget gates derived from the same induced-action framework used by KDA / RWKV-7 / GLA’s linear-RNN gates. Wall trains at 4k sequence length and generalizes without further training to 200k+ tokens, beating both RoPE and data-dependent Forgetting Attention (FoX). The mechanism keeps vanilla attention’s parallel structure, is compatible with GQA and MLA, and ships with open-source Triton kernels including a WallDecode kernel claimed to hit SOTA-level decode throughput. The accompanying blog post on blog.tilderesearch.com was not retrievable at filing time; the tweet itself is the available primary record.

Key claims

• Wall replaces RoPE with diagonal forget gates inside softmax attention, generalizing the per-channel selective-forgetting mechanism from linear RNNs (KDA, RWKV-7, GLA) to softmax attention via a principled “induced action” framework [tweet body].
• Wall outperforms both RoPE and Forgetting Attention (FoX) on length generalization; models trained at 4k sequence length generalize to 200k+ tokens with no further training [tweet body].
• Wall preserves the parallel structure of vanilla softmax attention and is compatible with GQA and MLA, i.e. it is positioned as a drop-in replacement for RoPE rather than a new architecture [tweet body].
• A custom decode kernel (WallDecode) is released alongside reference training Triton kernels; Tilde claims it reaches “SOTA-level” decode throughput [tweet body].
• The framing is that long-context generalization is fundamentally about selective forgetting, and Wall makes that mechanism per-channel and data-dependent without giving up softmax’s expressivity [tweet body].

Method

The tweet thread (without the underlying blog post, which was unreachable) characterizes Wall as a “RoPE-free” softmax attention where the only positional signal is a diagonal forget-gate tensor applied per channel within each attention head. The conceptual lineage is linear-RNN forget gates as instantiated in KDA, RWKV-7, and GLA: each channel learns its own data-dependent decay, so different channels can remember at different timescales. Wall lifts that mechanism out of the linear-RNN regime and into softmax attention via an “induced action” construction (not detailed in the tweet). Crucially the construction retains the standard parallel attention computation, which is why GQA/MLA compatibility and IO-aware kernels remain straightforward.

Results

• Length generalization 4k → 200k+ tokens with no continued pretraining on long sequences [tweet body]. No specific benchmark numbers are quoted in the tweet.
• Outperforms RoPE and Forgetting Attention (FoX) [tweet body]; the comparison protocol is not specified.
• WallDecode kernel reaches “SOTA-level” decode throughput [tweet body]; no numbers given.

(All quantitative claims here are tweet-level summaries; the underlying blog post was 500-erroring at filing and would be the authoritative source.)

Why it’s interesting

Wall slots directly into the Hybrid Linear Attention cluster but inverts the usual move: instead of replacing softmax layers with linear-RNN primitives that already have per-channel diagonal gates (KDA, GDN, RWKV-7), it imports those gates into softmax attention. If the 4k → 200k+ generalization holds up under independent eval, this is a different point on the long-context Pareto than the current hybrid prescriptions (3:1 KDA:MLA in Kimi Linear, 7:1 lightning:softmax in MiniMax-M1) — pure softmax with diagonal gating, no linear-attention branch needed. It also contrasts with Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free, where the Qwen team showed a single output-side sigmoid gate eliminates attention sinks and helps length extension; Wall pushes the gating mechanism per-channel and inside the attention score, more in the spirit of FoX but with a richer (diagonal, data-dependent) parameterization. The same Tilde Research team previously reverse-engineered NSA / MoBA gating in Sparsity is Cool: Reverse-Engineering MoBA and NSA, so this is consistent with their broader thesis that selective forgetting / gating is the right design surface for long context.

See also

• Hybrid Linear Attention — Wall imports the diagonal-gate primitive from this cluster’s linear-RNN primitives (KDA, RWKV-7, GLA) into softmax attention
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free — Qwen’s sigmoid-gated softmax attention; complementary mechanism (single output gate vs per-channel diagonal in-score gate)
• Sparsity is Cool: Reverse-Engineering MoBA and NSA — prior Tilde work on gating in sparse softmax attention (NSA / MoBA)
• Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings — alternative line on what’s wrong with RoPE (content/position confound in polar form)
• RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably — independent critique of RoPE’s positional discrimination in long contexts
• Context Length / Quality Trade-off in Video Generation — Wall’s 4k → 200k+ claim is the strongest positive datapoint on this concept’s open question