Positional Encodings — Sinusoids, Learned Tables and RoPE

Order, equivariance and what the mask already tells us

Unmasked self-attention with shared tokenwise projections and no position signal is permutation-equivariant: permuting input rows permutes output rows. It is not invariant unless an additional permutation-invariant readout removes row order. Thus the statement “attention cannot distinguish shuffled sentences” needs a readout/conditioning qualification: an output list still has row indices, and an external position-sensitive readout can use them. A fixed causal mask itself introduces ordered accessibility and breaks arbitrary permutation symmetry.

Let Q=XW_Q, K=XW_K and V=XW_V and let P be a permutation matrix. The permuted score matrix is P QK transpose P transpose. Row softmax respects that simultaneous permutation, so:

This holds for fixed deterministic inference, not necessarily two independently sampled dropout masks. It also holds with a mask that is permuted consistently; holding a triangular mask fixed changes the problem. “Dog bites man” and “man bites dog” motivate explicit position-aware representations, but predicting chance on every no-position copy/reversal task would ignore causal structure and readout design.

Prerequisites are self-attention, MHA and sine/cosine angle addition. Outcomes: derive a linear shift, implement and test sinusoidal/learned/RoPE variants, diagnose shape/range failures, and separate a formula being defined from demonstrated length generalization.

Derive the sinusoidal shift, including its limits

For one frequency omega, represent position p by [sin(omega p),cos(omega p)]. Angle addition gives:

In this sin,cos coordinate ordering the matrix is a rotation by negative omega k under the conventional x,y orientation. It depends only on offset k. Pairing many frequencies creates a bank of clocks with different periods; this supplies a clean linear-shift property, not the unique possible representation or an exact collision-free odometer under finite precision.

The original choice for even width d is omega_i=10000 to the power (−2i/d), for i=0 through d/2−1. Therefore:

The fastest wavelength is 2pi. The slowest included wavelength is 2pi×10000^(1−2/d), not 20000pi exactly: i=d/2 is outside the index range. The base 10000 is a design choice giving geometric scales, not a constant forced by the shift identity. In the actual exponent range, directly computing these frequencies does not underflow FP32; the exponential/log form is a vectorization convenience.

For full paired encodings the unprojected dot product is sum_i cos(omega_i k), independent of starting position. It oscillates with offset and need not decrease monotonically. After adding content and applying arbitrary learned Q/K projections, cross terms and the learned bilinear form mean scores need not depend only on offset. An absolute encoding with a useful relative identity is not a pure-relative attention theorem.

Complete positional functions and mathematical contracts

Save these fences as positions_lab.py. The sinusoidal function deliberately requires a positive even width so every sine has a cosine partner; this avoids the old odd-width assignment error and keeps the paired identities exact. It supports explicit offsets, dtype/device and a declared base. Computing a formula beyond training length is allowed here but is not evidence the model will work there.

import math, os
from pathlib import Path
import torch
from torch import nn
 
def sinusoidal_pe(length,width,offset=0,base=10000.,device=None,dtype=torch.float32):
    if length<1 or width<2 or width%2 or offset<0 or base<=1:
        raise ValueError('positive length, even width, nonnegative offset, base>1 required')
    pos=torch.arange(offset,offset+length,device=device,dtype=torch.float64)[:,None]
    freq=torch.exp(torch.arange(0,width,2,device=device,dtype=torch.float64)*(-math.log(base)/width))
    angle=pos*freq
    return torch.stack([angle.sin(),angle.cos()],-1).flatten(-2).to(dtype)
 
class LearnedPE(nn.Module):
    def __init__(self,max_len,width):
        super().__init__();self.table=nn.Embedding(max_len,width)
    def forward(self,x,offset=0):
        length=x.size(1)
        if offset<0 or offset+length>self.table.num_embeddings:raise ValueError('position table exceeded')
        return x+self.table(torch.arange(offset,offset+length,device=x.device))
 
def rope_angles(positions,width,base=10000.):
    if width<2 or width%2 or base<=1:raise ValueError('even positive width and base>1')
    freq=torch.exp(torch.arange(0,width,2,device=positions.device,dtype=torch.float64)*(-math.log(base)/width))
    angles=positions.to(torch.float64)[...,None]*freq
    return angles.cos(),angles.sin()
 
def apply_rope(q,cos,sin):
    if q.size(-1)%2:raise ValueError('RoPE pairs need an even width')
    q1,q2=q[...,0::2],q[...,1::2]
    cos=cos.to(q);sin=sin.to(q)
    return torch.stack([q1*cos-q2*sin,q1*sin+q2*cos],-1).flatten(-2)
 
def plain_attention(x,causal=False):
    score=x@x.transpose(-1,-2)/math.sqrt(x.size(-1))
    if causal:score=score.masked_fill(~torch.ones(x.size(-2),x.size(-2),dtype=torch.bool).tril(),-torch.inf)
    return score.softmax(-1)@x
 
torch.manual_seed(38)
x=torch.randn(1,5,6,dtype=torch.float64);perm=torch.tensor([3,0,4,1,2])
assert torch.allclose(plain_attention(x[:,perm]),plain_attention(x)[:,perm],atol=1e-12)
assert not torch.allclose(plain_attention(x[:,perm],True),plain_attention(x,True)[:,perm])
pe=sinusoidal_pe(200,8,dtype=torch.float64)
freq=10000.**(-torch.arange(0,8,2,dtype=torch.float64)/8)
for p in (0,10,50,100):
    k=5;pair=pe[p].reshape(-1,2);a=freq*k
    shifted=torch.stack([a.cos()*pair[:,0]+a.sin()*pair[:,1],-a.sin()*pair[:,0]+a.cos()*pair[:,1]],-1).flatten()
    assert torch.allclose(shifted,pe[p+k],atol=1e-12)
    assert torch.allclose(pe[p]@pe[p+k],(freq*k).cos().sum(),atol=1e-12)
dots=pe@pe[0]
assert (dots[1:]-dots[:-1]>0).any()  # rejects monotone distance-decay claim
for invalid in (1,3):
    try:sinusoidal_pe(5,invalid)
    except ValueError:pass
    else:raise AssertionError('odd width')
learned=LearnedPE(4,8)
assert sum(p.numel() for p in learned.parameters())==32
try:learned(torch.zeros(1,5,8))
except ValueError:pass
else:raise AssertionError('learned table overflow')
q=torch.randn(8,dtype=torch.float64);k=torch.randn(8,dtype=torch.float64)
cm,sm=rope_angles(torch.tensor(7),8);cn,sn=rope_angles(torch.tensor(12),8)
cd,sd=rope_angles(torch.tensor(5),8)
assert torch.allclose(apply_rope(q,cm,sm)@apply_rope(k,cn,sn),q@apply_rope(k,cd,sd),atol=1e-12)
assert torch.allclose(apply_rope(q,cm,sm).norm(),q.norm(),atol=1e-12)
print('permutation, shift, oscillation, table and RoPE contracts: pass')

Additive usage is x=token_embedding(ids)+PE[:T], broadcasting a (T,d) table over batch. GPT-style learned positions use a table lookup instead. Either is added once at the input for that architecture. Reinjection at every block defines a different architecture rather than a universally forbidden operation. RoPE is different: apply rotations inside every attention layer to Q/K, not add them to token embeddings. Keep positions aligned with the cache and specify whether pairs are interleaved or split-half; checkpoint conventions are not interchangeable without a permutation.

Inspect the clock bank rather than imagining a smooth decay

The following optional local SVG contains the actual computed table, with fast oscillation at low feature indices and slow change at high indices. No plotting package or browser server is required to create it. It is a report/teaching figure, not an application viewport test.

def export_pe(path,length=40,width=16):
    table=sinusoidal_pe(length,width,dtype=torch.float64)
    parts=['<svg xmlns="http://www.w3.org/2000/svg" width="430" height="470">',
           '<rect width="100%" height="100%" fill="white"/>',
           '<text x="80" y="22">Position rows / feature columns</text>']
    for p in range(length):
        for j in range(width):
            value=float(table[p,j]);v=int(255*(1-abs(value)))
            color=f'rgb(255,{v},{v})' if value>=0 else f'rgb({v},{v},255)'
            parts.append(f'<rect x="{60+j*21}" y="{40+p*10}" width="20" height="9" fill="{color}"/>')
    parts.append('<text x="10" y="55">0</text><text x="8" y="438">39</text></svg>')
    Path(path).write_text(''.join(parts))
if __name__=='__main__' and os.environ.get('PE_FIGURE_PATH'):
    export_pe(os.environ['PE_FIGURE_PATH'])

The finite table can still run out even though the sinusoidal formula exists for every position. A registered nonpersistent buffer follows device moves without being a learned parameter; saving the generating configuration is necessary even if table bytes are regenerated. Large angles and finite precision also require attention to calculation dtype. An apparent smooth envelope is not a monotonic dot-product guarantee.

Learned embeddings, RoPE, T5 and ALiBi

Learned absolute positions add max_length×d parameters and can adapt to task statistics. The allocation limit differs from the largest position actually trained: a row can exist and be undertrained. Extending a table needs a deliberate initialization/interpolation and continued-training policy. Taking positions modulo the old length creates aliases, not meaningful extrapolation. A fixed pretrained GPT-2-style checkpoint should retain its learned-position contract unless an explicitly evaluated conversion is intended.

Sinusoidal positions add no learned parameters and have the derived shift structure, but are not guaranteed better or worse in-distribution. The original Transformer compared learned and sinusoidal embeddings with very similar results. Learned content projections can exploit the fixed signal, so “fixed cannot specialize at all” was also too strong. Addition keeps model width fixed; concatenation followed by a projection is a valid alternative. Exact content/position separation is not guaranteed simply because the space is high-dimensional.

RoPE uses the conventional two-dimensional rotation R(theta). For fixed content q,k:

The score depends on content and relative offset, not on offset alone. At deeper layers content may already depend on position, masks and context, so whole-model absolute invariance does not follow. Rotation preserves vector norms; it adds no trainable position table but does require computation and position/frequency configuration. Its formula has no hard mathematical length ceiling, while actual buffers, cache memory, trained distribution and numerical precision do.

T5 uses learned bucketed relative score bias, not the old table's contradictory “T5 sinusoidal” description. Nearby offsets get finer buckets; distant ones share coarser categories. ALiBi adds head-specific linear distance penalties directly to scores instead of adding embedding vectors. Its authors demonstrate length extrapolation in particular setups. Thus RoPE is not the only common scheme with post-training extension possibilities, nor does every frontier model expose or use the same positional implementation.

No universal inference-speed ordering follows from these rows: cache precomputation, fused kernels, dimensions and hardware matter.

Context extension without magical length promises

Position Interpolation (PI) maps positions from a target range back into the training range by a linear scale. It compresses neighbor phase differences as well as far offsets; its paper uses fine-tuning and reports experiments up to 32,768, not a proof any scaling factor works.

NTK-aware base scaling changes frequency scales nonuniformly through the RoPE base, preserving higher-frequency local detail differently from uniform position scaling. It is a heuristic, not a theorem of model context validity. YaRN distinguishes frequency bands: high frequencies with many rotations retain more original behavior, low frequencies are interpolated more, intermediate bands transition, and attention temperature/scaling is adjusted. The paper supplies a fine-tuning/evaluation recipe; it is not a universal 400-step guarantee for every checkpoint.

LongRoPE searches nonuniform interpolation using frequency and token-position effects, progressively extends context, and readjusts for short-context performance. Its 2048k result is a particular reported experiment with continued training, not evidence that Gemini privately uses it or that all long contexts work without retraining. Evaluate retrieval, perplexity and downstream tasks at both short and long lengths, because a long-context adaptation can regress local behavior.

Changing Llama's base alone does not make 128k “just work”; long-context model reports include continued pretraining/data/evaluation decisions. Llama 4's official design uses its own interleaved positional/local/global choices; do not infer “YaRN + no RoPE in chunked layers” from a generic extension survey. Likewise the old 1B-token universal recipe is unsupported. Pin model configuration, tokenizer, original window, extension factor, frequency implementation, training budget and evaluation before deciding a method.

Shape and causality debugging

Check (T,d) versus (d,T), positive width, even pairing, dtype/device, max position and offsets. A square T=d test can hide a transposition mistake; use unequal dimensions. An odd-width sinusoidal helper may choose a truncated final sine, but then the full paired norm/dot identities no longer hold unchanged. The explicit rejection above is intentional.

A causal model without additive positions is not identical to a set model because its accessible prefix changes by row. A padding mask must travel with any permutation in an equivariance test. A cache requires absolute query/key positions when applying RoPE; resetting new queries to zero breaks relative offsets even if output shapes match. For a learned-position sliding window, recomputing all retained tokens from positions zero…window−1 is a declared truncation policy; it is not automatically equivalent to retaining their old cached states.

Practice and worked answers

1. Permutation proof. Shared projections commute with P, row softmax respects simultaneous row/column permutation, and P transpose P cancels in AV. State the no-fixed-non-equivariant-mask/no-independent-dropout qualification.
2. Formula and base. Even dimensions get sin(p×10000^(−2i/d)), odd dimensions cosine. Base 10000 is a geometric-scale design choice. The final included i is d/2−1, so the maximum wavelength is below 20000pi.
3. Two failures. Learned table overflow must be rejected or explicitly adapted; repeated additive PE inside every residual block changes the architecture and scale. It is not how the original input-additive model is wired.
4. Trade table. Learned is a lookup with trainable rows, sinusoidal is a fixed computable table, RoPE rotates Q/K per layer. All can be implemented efficiently; none guarantees extrapolation quality merely because it can emit numbers beyond training length.
5. When not RoPE. Keep the learned-position specification when evaluating/fine-tuning a pretrained GPT-2-compatible checkpoint. For unordered set data, an imposed order may be unnecessary. For a controlled paper reproduction, use the paper's actual scheme.
6. YaRN's bands. Uniform PI squashes all phase distances equally. Band-dependent interpolation preserves some high-frequency resolution while extending low-frequency range, with adjusted attention scale. Report the paper's measured regime and local validation instead of declaring an 8× win inevitable.

A further worked test is the oscillation negative control above: the existence of an increasing adjacent dot product disproves “similarity must decay at every offset.” Revisit the shift matrix tomorrow and repeat the cache-offset thought experiment next week.

Sources and optional reading

Primary: Transformer v7, RoFormer, T5, ALiBi, PI, YaRN, LongRoPE. Optional original Hedu, RoPE video, Kazemnejad derivation, and trigonometry refresher retain their resource role.

Executed: CPU equivariance/counterexample, sinusoidal shift/dot identities, norm-preserving RoPE, table overflow and figure export. Not executed: pretrained checkpoint conversion, YaRN/LongRoPE fine-tuning, million-token retrieval, GPU timing or application UI. Previous · Next.