RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

TL;DR

Proves that Rotary Position Embeddings (RoPE) intrinsically fail in two ways as context length grows: (1) attention loses its locality bias — nearer positions are no more likely to score higher than far ones, with failure probability approaching 0.5; (2) attention loses consistency in token relevance — a key that beats an alternative at one position can lose at another. The analysis abstracts away from content and depends only on length. Adjusting the RoPE base trades distinguishing positions for distinguishing tokens; the common practice of increasing the base helps tokens but sacrifices positions. Empirically, multi-head, multi-layer architectures do not rescue these limitations, and LLMs tuned for needle-in-a-haystack retrieval still fail a “return the k-th item in a list” task.

Key claims

As context length increases, RoPE-based attention’s preference for nearer over substantially farther positions has failure probability approaching 0.5 — i.e. locality bias degrades to random guessing [Abstract].
The attention score can remain unchanged when a key token is moved to a different position, or replaced by a different token — RoPE attention thus fails to distinguish both where and what [Abstract].
Adjusting the RoPE base hyperparameter trades distinguishing positions against distinguishing tokens; both cannot be preserved simultaneously [Abstract].
Increasing the RoPE base (a common long-context recipe) helps distinguish different tokens but inevitably sacrifices the ability to distinguish positions [Abstract].
The theoretical analysis abstracts over content and depends only on context length, then is verified empirically — multi-head, multi-layer architectures are insufficient to overcome the limitations [Abstract].
LLMs optimized for needle-in-a-haystack-style retrieval still fail a very simple “return the k-th item in a list” task, as predicted by the theory [author thread].

Method

The paper performs a theoretical analysis of RoPE-based attention in the long-context regime, abstracting over the specific content of the context and depending only on its length. Two properties of attention are studied: (1) locality bias — the probability that RoPE assigns higher attention to a nearer key than a farther one; (2) token-relevance consistency — whether the relative ordering of attention scores between two candidate keys is preserved as those keys move through positions. Both properties are shown to degrade toward random (failure probability → 0.5) as length grows. A separate result analyses the effect of varying the RoPE base $\theta$ and proves a position-vs-token tradeoff on $\theta$ . The theoretical results are then verified on real multi-layer, multi-head LLMs via a needle-in-a-haystack-style retrieval setup contrasted with a k-th-list-item retrieval task that the theory predicts should fail even in long-context-tuned models.

Results

The empirical headline is that LLMs optimised for needle-in-a-haystack retrieval — the canonical long-context benchmark — still fail the k-th-list-item retrieval task that the theory flags as a clean test of position-token distinguishability. The theoretical results give the failure probability of locality bias and of token-relevance consistency both approaching 0.5 as length grows [Abstract]. The RoPE-base sweep verifies the position-vs-token tradeoff: higher base improves token-distinguishing but degrades position-distinguishing, consistent with the proof [Abstract]. Specific numbers (failure-probability curves, k-th-list-item accuracy as a function of list length and RoPE base) are reported in the paper body.

Why it’s interesting

This is the first filed paper that pins the “long-context degradation” pathology on RoPE itself with a formal, content-independent proof, rather than on training data or attention-implementation issues. It sharpens an open question in Context Length / Quality Trade-off in Video Generation: Bounded Positional Encoding in Context Forcing: Consistent Autoregressive Video Generation with Long Context, incremental temporal-only RoPE adjustment in PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference, and relative-PE-inside-memory-attention in MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation are all RoPE-side workarounds that this paper would predict are necessary — none can fully fix the underlying limit, only restrict the regime in which it bites. The author’s own takeaway — that agentic frameworks which break long contexts into shorter ones may be more effective than scaling raw context — connects directly to the recursive-LM / context-folding line in the wiki’s _candidates.md. Contrasts with The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks, which locates a different long-context pathology (massive activations / attention sinks) and frames it as outlier-driven rescaling rather than as a positional-encoding failure.