Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

TL;DR

Re-expresses RoPE in polar form to show that its attention score confounds key/query content (the “what”, via component magnitudes) with position (the “where”, via phase) — the dot-product alignment angle depends on both. PoPE is a minimal re-parameterisation: take softplus of each key/query element as a non-negative magnitude, use a position-only phase $p\,\omega_i$ per element, and score with the standard complex inner product. Result: a diagnostic indirect-indexing task that RoPE solves at 11% jumps to 95% with PoPE; perplexity improves on MIDI, human-genome, and OpenWebText at 124M–774M parameters; and PoPE extrapolates to 10× the training context (1024 → 10240 tokens) without fine-tuning or YaRN-style interpolation while RoPE diverges.

Key claims

Re-expressing RoPE in polar coordinates shows the attention score for a key/query component depends on both component magnitudes and an interaction term that shifts the position-of-maximum-response — proving content and position are entangled in RoPE [§2, Eq. 2].
PoPE replaces RoPE’s 2D component rotation with a per-element complex representation whose magnitude is softplus of the real-valued key/query element and whose phase is $p\,\omega_i$ — the position-only phase removes the what-where interaction term [§3, Eq. 3-5].
A learnable per-head, per-frequency phase bias $\delta_i \in [-\pi, \pi]$ replaces the RoPE interaction term; zero initialization helps length generalization, uniform initialization gives slightly better in-distribution performance [§3, Eq. 6].
On the Indirect Indexing diagnostic task (find a target character at a specified offset from a specified source character in a random string), RoPE reaches 11.16% ± 2.45 final-token accuracy; PoPE reaches 94.82% ± 2.91 [Table 1].
On symbolic music, PoPE lowers NLL from 0.5081 → 0.4889 on JSB Bach Chorales and 1.501 → 1.486 on MAESTRO [Table 2].
On the Human Reference Genome with Nucleotide-Transformer tokenization, PoPE lowers NLL from 4.217 → 4.152 [Table 3].
On OpenWebText, PoPE lowers perplexity at 124M / 253M / 774M parameters from 21.55 / 18.88 / 15.85 (RoPE) to 21.33 / 18.55 / 15.45 (PoPE); the gap holds steady or grows with model size [Table 4].
On six zero-shot downstream tasks (LAMBADA, BLiMP, CBT, HellaSwag, PIQA, ARC-E), PoPE has higher mean accuracy than RoPE at all three model sizes [Table 5].
Trained on 1024-token sequences and evaluated on PG-19 up to 10240 tokens, PoPE’s perplexity remains stable while RoPE’s degrades significantly without fine-tuning or position interpolation; RoPE’s extrapolation degrades with model size while PoPE’s stays largely stable [Fig. 2].
Frequency-norm analysis shows RoPE concentrates large norms on a sparse subset of low frequencies, while PoPE assigns large norms to high-frequency features in all layers except the first and distributes usage across the full frequency range [Fig. 3, Fig. 4].
A Triton/Flash-Attention-2 implementation adds only one multiplication over standard RoPE attention; the memory overhead from carrying complex-valued K/Q in Cartesian form can be avoided by re-deriving them inside the kernel from magnitudes [§3, Eq. 7-10].

Method

RoPE partitions a $d$ -dim key/query into $d/2$ two-element components and rotates each by an angle $p\,\omega_i$ proportional to the token’s position $p$ , with $\omega_i$ a component-specific frequency. Re-expressing in polar form ( $r_i^{(q)} e^{i\theta_i^{(q)}}$ , similarly for the key) makes the attention score $\sum_i r_i^{(q)} r_i^{(k)} \cos\bigl((p_q - p_k)\omega_i + (\theta_i^{(q)} - \theta_i^{(k)})\bigr)$ — the term $(\theta_i^{(q)} - \theta_i^{(k)})$ is the what-where confound, because the angle that maximises the cosine now depends on what’s in the key and query, not just their relative positions.

PoPE removes the confound. Each scalar element of the projected key/query is passed through softplus to give a non-negative magnitude $r_i$ ; the phase is set to $p\,\omega_i$ exactly (no content contribution). The attention score is then the real part of the standard complex inner product $\hat{q}^\dagger \hat{k}$ , with an optional learnable bias $\delta_i$ replacing the removed interaction term (per-head, per-frequency, bounded to $[-\pi, \pi]$ ). Two changes versus RoPE: there are now $d$ frequencies rather than $d/2$ (one per element, not per pair); and the phase depends only on position. The efficient kernel materialises the Cartesian-form complex K/Q outside the attention block, then needs exactly one extra multiplication inside the FlashAttention-2 inner loop to compute the real part of the dot product.

Results

PoPE solves the Indirect Indexing diagnostic — 94.82% vs RoPE’s 11.16% — establishing the qualitative claim that disentangling what from where makes pointer-arithmetic-style position lookup learnable [Table 1]. On three real autoregressive sequence modeling settings (Bach Chorales, MAESTRO MIDI, Human Reference Genome) PoPE lowers NLL across the board [Table 2, Table 3]. On OpenWebText, PoPE wins at 124M, 253M, and 774M parameters with the gap stable or widening with model size [Table 4]; the perplexity advantage carries through to higher mean zero-shot accuracy on the six standard downstream tasks [Table 5]. The biggest headline is length extrapolation: trained at 1024 tokens, tested up to 10240 on PG-19, PoPE holds while RoPE diverges — and unlike RoPE the gap is not closed by going bigger, since RoPE extrapolation gets worse with model size where PoPE stays flat [Fig. 2]. Frequency-norm heatmaps suggest the mechanism: RoPE concentrates on a sparse subset of low frequencies (with high-frequency channels behaving like noise the model suppresses); PoPE makes broad use of the full frequency range, including the high end [Fig. 3, Fig. 4].

Why it’s interesting

This is the constructive counterpart to RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably, which proved a content-independent lower bound on RoPE’s long-context discriminability and predicted that any patch on RoPE was working around a fixed point. PoPE removes the interaction term that both papers identify as the root cause — it is what the lower-bound paper would suggest is necessary, not just helpful. The 10× zero-shot length extrapolation on PG-19 is the practical payoff and is the property the Slack note flags as relevant for image/video: every RoPE-side workaround in Context Length / Quality Trade-off in Video Generation (Bounded Positional Encoding from Context Forcing: Consistent Autoregressive Video Generation with Long Context, incremental temporal RoPE adjustment from PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference, relative-PE-inside-memory-attention from MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation) is restricting the regime in which RoPE’s failure bites; PoPE proposes to replace the encoding entirely. No vision experiments are reported, leaving open whether the axial / 2D extensions used in image and video DiTs (e.g. as taught in Rotary Positional Embeddings Explained | Transformer) admit the same drop-in re-parameterisation.