Rotary Positional Embeddings Explained | Transformer

TL;DR

A walk-through explainer of Rotary Positional Embeddings (RoPE) — the positional-encoding scheme used in modern Transformer models across text, image, and video modalities — covering both the underlying math and a from-scratch PyTorch implementation. The video frames RoPE as injecting positional information by rotating query/key feature pairs by an angle proportional to position, with frequency controlled by a base hyperparameter θ. Nearby tokens receive smaller rotational differences than far ones, and the inner product after rotation depends only on the relative offset between positions. By @domo, posted to the “Outlier” YouTube channel.

Key claims

RoPE encodes positional information by applying 2D rotations to pairs of feature dimensions in the query and key vectors, rather than by adding a position vector to token embeddings [video].
The rotation angle for the i-th feature pair at position m is θ_i = base^(-2i/d) · m, so each pair rotates at its own frequency, with the per-dimension frequency controlled by the base hyperparameter [video].
Because rotation is norm-preserving, the dot product between rotated query at position m and rotated key at position n depends only on the relative offset (m − n) — RoPE encodes relative position inside dot-product attention without separate relative-PE machinery [video].
The same construction generalizes across modalities — 1D sequence index for text, 2D (height, width) for images via axial RoPE, 3D (frame, height, width) for video — by assigning different feature pairs to different positional axes [video].
Implementing RoPE in PyTorch reduces to precomputing cos/sin tables for the rotation angles and applying them as element-wise products on paired feature dimensions inside the attention QK projection, rather than as a full matrix multiplication [video].

Method

The video first motivates positional encoding from the standpoint that bare self-attention is permutation-equivariant and therefore cannot distinguish token order on its own. It contrasts additive sinusoidal / learned absolute PEs (which are added to embeddings before attention) with RoPE, which is applied inside attention by rotating Q and K. The mathematical construction pairs even/odd feature dimensions and rotates each pair by an angle that scales linearly with position index and inversely with feature index — high-frequency rotations for the first dimensions, low-frequency for the last. The PyTorch implementation section builds up the cos/sin caches, then shows the standard fused element-wise application: q_rot = q · cos + rotate_half(q) · sin. The video notes that this fused form avoids constructing the explicit block-diagonal rotation matrix and is what gets used in production attention kernels.

Results

This is an explainer, not a research paper — no benchmark numbers. The implementation walkthrough shows that the rotated-Q / rotated-K attention scores depend only on relative position, demonstrated on a small example: moving a key from position 3 to position 7 while moving the query from 5 to 9 leaves the score unchanged, confirming the relative-position property.

Why it’s interesting

This is the first explainer-format reference the wiki has on RoPE itself — useful as background for the growing cluster of filed papers that analyze or work around RoPE’s failure modes. It complements RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably, which proves that RoPE’s locality bias and token-relevance consistency degrade toward random as context length grows; the video covers the construction the proof is operating on. It also complements Cameras as Relative Positional Encoding, which extends RoPE-style relative encoding from sequence positions to 6-DoF camera poses for multi-view transformers — a generalization that only makes sense once the relative-offset property emphasized in this explainer is internalized. For Luma’s video-DiT work, Context Length / Quality Trade-off in Video Generation catalogues RoPE-side workarounds (Bounded Positional Encoding, incremental temporal-only RoPE adjustment, relative-PE-inside-memory-attention) that all assume familiarity with the rotation/base-frequency mechanics walked through here.