Positional Encoding — Sinusoidal, Learned, RoPE

Why this session exists

A token's identity does not tell us whether it came before or after another token. Position-free, unmasked self-attention is permutation-equivariant, not invariant: shuffle the inputs and the outputs shuffle correspondingly. A permutation-invariant pooling operation would then lose the order altogether. A causal mask already supplies an asymmetric visibility pattern, so the unmasked proof must not be applied unchanged to a decoder.

This session develops the actual symmetry, computes sinusoidal positions, derives rotary attention and tests offset-aware code. Position representation is one part of long-context design alongside training distribution, attention compute, memory and evaluation—not an explanation of an undisclosed commercial model's architecture.

You will be able to: prove the unmasked symmetry; compute frequencies and rotations; distinguish learned absolute tables, fixed sinusoids, relative bias and RoPE; implement fixed sinusoids, learned absolute tables and offset-aware RoPE, while treating relative bias conceptually; and separate a computable long position from demonstrated length generalization.

Prerequisites

Use the tensor conventions of sessions 108–109: input (B,T,d), per-head Q/K (B,T,h,dh), and softmax over keys. Session 106 supplies tokenization; session 110 is not a replacement for a linear-algebra course. A 2×2 rotation and matrix transpose are the only extra algebra required here.

(a) Intuition · 5 min

Imagine a shuffled book. Page numbers let a reader reconstruct order, but giving every page a number does not teach the reader how to interpret a much longer book. Additive positional embeddings write a mark beside the content; RoPE rotates the query/key coordinate system before matching. Neither creates more memory or guarantees accurate reasoning over a long document.

Why attention is permutation-equivariant

Let P be a permutation matrix and let Q=XW_Q, K=XW_K, V=XW_V. With no position signal, fixed mask or stochastic dropout, replacing X by PX gives Q'=PQ, K'=PK, V'=PV. If S=QKᵀ/√d_h, then S'=PSPᵀ. Row-wise softmax commutes with this simultaneous row/column permutation, so

Thus outputs remain attached to their input tokens, but no computation identifies an absolute slot. “Dog bites man” and “man bites dog” do not generally produce identical ordered tensors; they produce a permutation of the same token-associated outputs under these assumptions. Positionwise MLPs and LayerNorm preserve this equivariance. A fixed triangular causal mask does not commute with an arbitrary permutation. A symmetric mean over final token states is invariant, which explains the bag-of-tokens limitation for that particular readout.

Sinusoidal positional encoding

The 2017 Transformer adds fixed positional vectors to scaled token embeddings [1]. For even width d and pair index i:

Small i means high angular frequency, not low frequency. Pair zero advances one radian per token and has wavelength 2π tokens. Larger i rotates more slowly. The last wavelength is $2\pi\,10000^{1-2/d}$, not exactly 10,000 tokens. The base is an architectural choice, not an integer position at which the last clock completes a cycle.

The position vector's squared norm is d/2. Its dot product with another position is a sum of cosines:

This is relative-offset dependent but oscillatory, not a monotone decay. Adding p before learned projections creates content-content, content-position, position-content and position-position terms. A pure PE dot-product identity does not imply that the full projected attention score is only a function of distance.

Sinusoidal PE at 5 positions, d=8

The four frequencies are 1, 0.1, 0.01, 0.001. At m=0 the vector is [0,1,0,1,0,1,0,1]. At m=1 its first two pairs are approximately [0.841471,0.540302,0.099833,0.995004]; the second angle is 0.1, not 0.01. At m=2 those pairs are [0.909297,-0.416147,0.198669,0.980067]. At m=100 the last pair's angle is 0.1; at m=10000 it is 10 radians, about 1.59155 revolutions. The program prints all five positions so these are reproducible values rather than an illustrative heatmap.

For the earlier frequency-band exercise d=128 and training length 2048, compute wavelengths and periods seen at pair indices 0,20,40,62. Slowly moving bands may cover only part of a cycle. This identifies an unfamiliar-input risk, not proof that a particular band alone causes every extrapolation failure.

Position schemes compared

BERT and GPT-2 use learned absolute positions. T5 uses bucketed relative bias, not the original sinusoidal input scheme. DeBERTa's disentangled attention is not identical to T5's scalar bias. An allocated but unused learned row may be untrained; an unallocated row is not a random vector waiting beyond the table. Extending or interpolating a learned table and adapting the model is possible—it changes the model and needs evaluation.

RoPE — the geometric idea

For each adjacent coordinate pair use $R(\theta)=\begin{bmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{bmatrix}$. Project content first, then rotate q at position m and k at position n by mω and nω. Orthogonality and angle composition give

For q=(a,b), k=(c,d), and Δ=(n−m)ω, the contribution is

The sine term matters: q=(1,0), k=(0,1) gives −sinΔ, not a cosine-only score. The explicit positional factor depends on relative offset; q and k still depend on content, possibly already contextualized by earlier layers. Equal offsets with different content need not give equal scores. Simultaneously shifting both positions leaves the score unchanged when the underlying q/k are held fixed.

Clock hands at many speeds

RoPE keeps each pair's norm unchanged while changing its angle. Fast clocks resolve nearby offsets; slow clocks supply larger-scale variation. This mental picture explains why a base or scaling change can affect local as well as distant attention. RoPE in the usual decoder recipe rotates Q and K, not V; this is the chosen scoring construction, not a theorem that positional values can never be useful. Adjacent-pair and split-half implementations require the corresponding checkpoint layout; they are not interchangeable without a coordinate permutation.

Computable positions versus length generalization

A position formula extending beyond training is not a guarantee of better length generalization than a learned table. Ordinary RoPE can fail beyond training even though the formula remains defined. Position Interpolation scales positions into a shorter range and fine-tunes the model in the cited study [4]. At 4× extension, m becomes m/4, so 8000 becomes 2000, but a one-token offset also becomes 0.25. That changes local geometry. Frequency-dependent scaling and YaRN address different aspects of this tradeoff; no universal 4–8× safe multiplier follows from the rotation identity [5].

ALiBi adds a head-dependent linear distance penalty to legal causal attention scores [6]. It biases toward nearer keys; it does not ban a distant key or guarantee a specific retrieval score. Compare it, a learned table, fixed sinusoid and RoPE using the task's visibility pattern and training regime. “Use RoPE for every task” is not a selection argument. Two-dimensional patch position and diffusion-step embeddings also encode indices, but an image location and a noise timestep have different semantics.

(c) Hands-on · 25 min

This complete CPU program computes fixed/learned positions and a multi-head RoPE layer. It validates dimensions and table ranges, supports a nonzero offset, and tests geometry independently of training. It is not a length-generalization experiment on a pretrained model.

import math
import torch
from torch import nn
from torch.nn import functional as F
torch.set_num_threads(1)
torch.manual_seed(110)
 
def sinusoidal_pe(length, width):
    if length < 0 or width <= 0 or width % 2:
        raise ValueError('nonnegative length and positive even width required')
    angle = torch.arange(length, dtype=torch.float64)[:, None] * 10000.0 ** (
        -torch.arange(0, width, 2, dtype=torch.float64) / width)
    return torch.stack((angle.sin(), angle.cos()), -1).flatten(-2)
 
class LearnedPE(nn.Module):
    def __init__(self, length, width):
        super().__init__()
        self.pe = nn.Embedding(length, width)
    def forward(self, x, offset=0):
        if offset < 0 or offset + x.shape[1] > self.pe.num_embeddings:
            raise ValueError('position outside allocated table')
        return x + self.pe(torch.arange(offset, offset+x.shape[1]))[None]
 
def precompute_rope_freqs(width, length, base=10000.0):
    if width <= 0 or width % 2 or length < 0 or not math.isfinite(base) or base <= 1:
        raise ValueError('invalid rotary table')
    angle = torch.arange(length, dtype=torch.float64)[:, None] * base ** (
        -torch.arange(0, width, 2, dtype=torch.float64) / width)
    return angle.cos(), angle.sin()
 
def apply_rope(x, cos, sin, offset=0):
    if x.ndim != 4 or x.shape[-1] % 2 or offset < 0:
        raise ValueError('expected B,T,H,even-D and nonnegative offset')
    if cos.shape != sin.shape or cos.ndim != 2 or cos.shape[1]*2 != x.shape[-1]:
        raise ValueError('table width mismatch')
    if offset+x.shape[1] > len(cos):
        raise ValueError('table too short')
    c = cos[offset:offset+x.shape[1]][None, :, None].to(x)
    s = sin[offset:offset+x.shape[1]][None, :, None].to(x)
    even, odd = x[..., 0::2], x[..., 1::2]
    return torch.stack((even*c-odd*s, even*s+odd*c), -1).flatten(-2)
 
class AttentionWithRoPE(nn.Module):
    def __init__(self, width=16, heads=2, max_len=64):
        super().__init__()
        if heads <= 0 or width % heads or (width//heads) % 2:
            raise ValueError('even per-head width required')
        self.heads, self.dh = heads, width//heads
        self.qkv = nn.Linear(width, 3*width, bias=False)
        self.out = nn.Linear(width, width, bias=False)
        c, s = precompute_rope_freqs(self.dh, max_len)
        self.register_buffer('cos', c); self.register_buffer('sin', s)
    def forward(self, x, offset=0):
        b,t,d = x.shape
        q,k,v = [y.reshape(b,t,self.heads,self.dh) for y in self.qkv(x).chunk(3,-1)]
        q=apply_rope(q,self.cos,self.sin,offset).transpose(1,2)
        k=apply_rope(k,self.cos,self.sin,offset).transpose(1,2)
        v=v.transpose(1,2)
        y=F.scaled_dot_product_attention(q,k,v,dropout_p=0.0)
        return self.out(y.transpose(1,2).reshape(b,t,d))
 
pe=sinusoidal_pe(10001,8)
for p in (0,1,2,100,10000): print('PE',p,pe[p].tolist())
assert torch.allclose(pe[0],torch.tensor([0,1]*4,dtype=torch.float64))
assert abs(pe[1,2].item()-math.sin(.1)) < 1e-14
wavelength=2*math.pi*10000.0**(torch.arange(0,128,2,dtype=torch.float64)/128)
for k in (0,20,40,62): print('band',2*k,'wavelength',float(wavelength[k]),'periods',float(2048/wavelength[k]))
c,s=precompute_rope_freqs(8,100)
q,k=torch.randn(2,1,3,8,dtype=torch.float64),torch.randn(2,1,3,8,dtype=torch.float64)
a=(apply_rope(q,c,s,2)*apply_rope(k,c,s,7)).sum(-1)
b=(apply_rope(q,c,s,40)*apply_rope(k,c,s,45)).sum(-1)
assert torch.allclose(a,b,atol=1e-12)
assert torch.allclose(apply_rope(q,c,s,23).square().sum(-1),q.square().sum(-1),atol=1e-12)
# Independent complex-number oracle uses Python complex multiplication.
expected=[]
for x,y in zip(q[0,0,0,0::2],q[0,0,0,1::2]):
    i=len(expected)//2; angle=23*10000**(-2*i/8)
    z=complex(float(x),float(y))*complex(math.cos(angle),math.sin(angle))
    expected.extend([z.real,z.imag])
assert torch.allclose(apply_rope(q,c,s,23)[0,0,0],torch.tensor(expected,dtype=torch.float64))
# Wrong offset is shape-valid but changes the score.
assert not torch.allclose(a,(apply_rope(q,c,s,2)*apply_rope(k,c,s,0)).sum(-1))
model=AttentionWithRoPE().double().eval();x=torch.randn(2,6,16,dtype=torch.float64)
assert torch.allclose(model(x,0),model(x,20),atol=1e-12)
assert model(x).shape==x.shape
learned=LearnedPE(512,8);assert learned(torch.zeros(1,2,8),100).shape==(1,2,8)
for bad in (lambda:learned(torch.zeros(1,1,8),1000),lambda:sinusoidal_pe(2,3),
            lambda:apply_rope(q,c,s,-1),lambda:apply_rope(q,c,s,100)):
    try:bad()
    except ValueError:pass
    else:raise AssertionError('negative control not rejected')
# Explicit permutation proof for position-free unmasked attention.
x=torch.randn(1,5,8,dtype=torch.float64);p=torch.tensor([3,1,4,0,2])
f=lambda y:torch.softmax(y@y.transpose(-1,-2)/math.sqrt(8),-1)@y
assert torch.allclose(f(x[:,p]),f(x)[:,p],atol=1e-12)
print('position geometry, offsets, equivariance and negative controls passed')

Anatomy of the script

sinusoidal_pe explicitly requires even width; a production odd-width implementation could slice the final cosine but must say so. LearnedPE checks allocated range. Buffers are non-parameters and move with a module; the default persistent buffers also enter its state dictionary. apply_rope uses adjacent even/odd coordinates and offset slices; reusing the start of the table for every incremental token is a subtle cache bug. Cache already rotated keys at their original absolute positions, and do not rotate them again when appending a token.

The attention layer uses (B,T,h,dh) before transposing to (B,h,T,dh). It is intentionally unmasked to isolate positional behavior; a causal language model must add the causal/padding policy. The test holds content and weights fixed when comparing common shifts. Changing the random model between tests would invalidate that comparison.

Practice: extrapolation without pretending to measure quality

Query a learned table of 512 rows at position 1000: the expected result is an exception. Compute the sinusoid/rotation at 1000: finite values are expected. Then try d=3, insufficient table length and a negative offset: reject them. These are interface tests. To assess actual quality, hold checkpoint/tokenizer/data fixed; evaluate several lengths and locations, including short contexts after scaling, and compare evidence retrieval plus answer correctness. A smooth position-similarity plot alone cannot certify any of those outcomes.

(d) Production reality · 15 min

LLaMA's published recipe includes RoPE [7]; the Position Interpolation paper is a separate context-extension result. Do not attribute YaRN to LLaMA-2's launch or infer Claude's private attention implementation from a context-window announcement. T5's bucketed bias deliberately maps sufficiently large distances into shared bins, limiting positional resolution rather than creating an unlimited precise distance ruler [3].

For serving, retain checkpoint revision, rotary layout, base, scaling algorithm, maximum trained/evaluated length, cache dtype and offset policy. Prefill and incremental decode should match within tolerance for the same prefix. A scaling change that computes finite tensors can still alter logits materially. Monitor short and long task performance, truncation, padding position IDs and cache reuse across unrelated requests. FlashAttention can reduce intermediate memory without teaching the model to use longer contexts.

Where this shows up in the rest of the plan

Session 111 assembles the block; 115 separates attention arithmetic from memory and cache layout; 117 handles evidence selection rather than treating a bigger context as a replacement for retrieval. Vision patches can use 2D locations; diffusion models can encode noise steps with a sinusoidal basis. Fine-tuning at a different length requires inspecting the existing position contract, not automatically applying a scaling trick.

(e) Recall + stretch · 10 min

1. Bag of words? Unmasked position-free attention is equivariant. Mean pooling makes it invariant; a fixed causal mask changes the symmetry. Ordered outputs are not identical under a shuffle.
2. Formula? Use sin(mω_i), cos(mω_i), with ω_i=10000^(−2i/d), and pair sine/cosine at the same frequency.
3. Why several frequencies? They supply variation at several offset scales. Small i is fast, large i slow. Distinguish a useful multi-scale representation from a finite-precision uniqueness guarantee.
4. Learned table weakness? Its allocated range is finite; unused rows and nonexistent rows differ. Extension is an adaptation problem, not impossible in principle.
5. RoPE geometry? Rotate Q/K in matched two-dimensional subspaces; the dot product's explicit positional factor becomes R(n−m), while content still matters.
6. Why not V here? The selected construction changes where to retrieve through QK scores and leaves value coordinates intact. It is not a prohibition on every alternative positional-value design.
7. Position interpolation? Rescale positions and adapt/evaluate; 4× context maps 8000 to 2000 but also compresses local offsets. It is not merely increasing an allocation.
8. Frequency table? Outer-product absolute positions with pair frequencies, then sine/cosine; validate shape, dtype and offsets. Different checkpoint pair conventions need different reshapes.
9. Why can short quality fall after extension? Rescaling changes local angles and fine-tuning changes weights/data distribution. Training to a longer length alone does not mathematically rescale old positions. Measure the regression rather than assume it.
10. How do long-context systems differ from early models? Position handling, long-input training, kernels, cache layout and evaluation can all change. A public context limit does not disclose which private techniques a vendor used.

Explain-out-loud test

Prove the permutation equation, expand the sine term of one rotary dot product, and explain why finite outputs beyond a table's training range are not a quality result. Replace “LLaMA can extend but GPT-2 cannot” with the concrete difference between their position interfaces and adaptation requirements.

What comes next

Full Transformer architecture combines position, attention, residual paths and per-token computation. Learning-plan hub.

Primary references

1. Vaswani et al., Attention Is All You Need, §3.5, v7.
2. Su et al., RoFormer, rotation derivation.
3. Raffel et al., T5, JMLR 2020, §2.1.
4. Chen et al., Position Interpolation, 2023.
5. Peng et al., YaRN, 2023 preprint.
6. Press et al., ALiBi, 2021 preprint.
7. Touvron et al., LLaMA, 2023.