S110 · Positional Encoding — Sinusoidal, Learned, RoPE

Module M13: NLP & Transformers · Session 110 of 130 · Track: LLM 📍

What you'll be able to do after this session

• Explain Positional Encoding to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S108 — Q/K/V Math — Scaled Dot-Product Attention Derived

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Positional Encodings in Transformers — CodeEmporium — Short intro to why transformers need positional info.
• 🎥 Deep dive (30–60 min) · Rotary Position Embedding (RoPE) — Efficient NLP — The intuition and math behind RoPE step by step.
• 🎥 Hands-on demo (10–20 min) · Positional Encoding from scratch — Umar Jamil — Codes sinusoidal PE inside a full transformer.
• 📖 Canonical article · Transformer Architecture: The Positional Encoding — Kazemnejad — The clearest single explanation of sinusoidal PE anywhere.
• 📖 Book chapter / tutorial · RoFormer: Enhanced Transformer with Rotary Position Embedding (Su 2021) — The RoPE paper — now used in Llama, Mistral, Qwen, DeepSeek.
• 📖 Engineering blog · EleutherAI — Rotary Embeddings: A Relative Revolution — Best engineering-oriented deep dive on why RoPE won.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: How transformers know word order.

Attention has one giant blind spot: it's permutation-equivariant. Feed a transformer the tokens [The, cat, sat] or [sat, cat, The] and — without positional information — it gives you the exact same attention weights, because dot products of embeddings don't depend on order. This is a disaster for language, where "dog bites man" and "man bites dog" are wildly different sentences.

Positional encoding is the fix: at the input, we add or mix a position-dependent signal into each token's embedding so that the model knows "this token is at position 5" separately from "this token means 'cat'". Three approaches have won at different points:

1. Sinusoidal (Vaswani 2017) — a fixed, non-learned function of position using sines and cosines at multiple frequencies.
2. Learned — treat position IDs like tokens; look up a learned position embedding and add it to the token embedding (BERT, GPT-2).
3. Rotary (RoPE, 2021) — the modern winner. Instead of adding a position vector to the embedding, it rotates the Q and K vectors by an angle proportional to position, so that dot products naturally depend on relative position.

Gotcha to carry forever: Learned positional embeddings only work up to the max sequence length they were trained on — try to feed a longer sequence and the extra positions have never been seen. This is why every LLM claim of "unlimited context" involves clever extrapolation tricks (position interpolation, YaRN, RoPE base scaling). Sinusoidal and RoPE, being functions, at least have values for any position — but their behaviour beyond training length is still fragile.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Sinusoidal PE formula. For token at position pos and embedding dimension index i:

• PE(pos, 2i) = sin(pos / 10000^(2i/d))
• PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Each dimension of the position vector is a sinusoid of a different frequency. Low i (leading dims) → high-frequency sines that change quickly with position — good for fine-grained order. High i (trailing dims) → very slow sinusoids — encode coarse position across the whole sequence.

Worked example — sinusoidal values. For d = 4, pos = 3:

• PE[0] = sin(3 / 10000^(0/4)) = sin(3) ≈ 0.14
• PE[1] = cos(3 / 10000^(0/4)) = cos(3) ≈ -0.99
• PE[2] = sin(3 / 10000^(2/4)) = sin(3 / 100) ≈ 0.03
• PE[3] = cos(3 / 10000^(2/4)) = cos(3 / 100) ≈ 1.00

Note the leading pair changes fast with pos; the trailing pair barely moves. That's the whole point — it's a positional "clock" running at many speeds.

RoPE intuition. Instead of adding PE to the embedding, RoPE splits each Q and K vector into 2D pairs and rotates each pair by an angle θ_i * pos. Because rotation preserves norms, the dot product q · k naturally becomes a function only of the relative position pos_q − pos_k — which is what attention should care about. That's why RoPE generalises to longer sequences more gracefully.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# positional_encoding.py — three schemes side by side
import torch, torch.nn as nn, math
 
# ---- 1. Sinusoidal PE (Vaswani 2017) ------------------------------------
def sinusoidal_pe(seq_len, d):
    pos = torch.arange(seq_len).unsqueeze(1)                     # [T, 1]
    i   = torch.arange(0, d, 2)                                  # [d/2]
    div = torch.exp(-math.log(10000.0) * i / d)                  # [d/2]
    pe  = torch.zeros(seq_len, d)
    pe[:, 0::2] = torch.sin(pos * div)
    pe[:, 1::2] = torch.cos(pos * div)
    return pe                                                    # [T, d]
 
pe = sinusoidal_pe(16, 8)
print("sinusoidal PE shape:", pe.shape)
print("first 3 positions:\n", pe[:3].round(decimals=2))
 
# ---- 2. Learned PE (BERT, GPT-2) ----------------------------------------
class LearnedPE(nn.Module):
    def __init__(self, max_len, d):
        super().__init__()
        self.pe = nn.Embedding(max_len, d)
    def forward(self, x):                    # x: [B, T, d]
        T = x.size(1)
        return x + self.pe(torch.arange(T, device=x.device))
 
lpe = LearnedPE(max_len=128, d=8)
out = lpe(torch.randn(1, 16, 8))
print("learned PE output shape:", out.shape)
 
# ---- 3. Rotary PE (RoPE) -------------------------------------------------
def apply_rope(x, base=10000.0):
    # x: [B, T, d] where d is even
    B, T, d = x.shape
    half = d // 2
    pos = torch.arange(T, device=x.device).float().unsqueeze(1)      # [T, 1]
    freqs = base ** (-torch.arange(0, half, device=x.device).float() * 2 / d)
    angles = pos * freqs                                             # [T, half]
    cos, sin = angles.cos(), angles.sin()
    x1, x2 = x[..., :half], x[..., half:]
    return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)
 
q = torch.randn(1, 4, 8)
q_rot = apply_rope(q)
print("RoPE input :", q[0, 0].round(decimals=2))
print("RoPE output:", q_rot[0, 0].round(decimals=2))
# Same-content tokens at different positions get DIFFERENT rotated vectors
print("norm preserved:", torch.allclose(q.norm(dim=-1), q_rot.norm(dim=-1), atol=1e-5))

Checklist while it runs:

1. Sinusoidal PE: earlier dimensions oscillate fast, later ones slow — plot with matplotlib and see the bands.
2. Learned PE: only works up to max_len=128. Feed T=200 and you'll get an index-out-of-range crash.
3. RoPE norm is preserved — that's the geometry of a rotation.
4. Feed the same token twice, once at pos=0 and once at pos=100, and observe RoPE gives different vectors.
5. Two RoPE-rotated Q and K vectors: their dot product depends only on |pos_q − pos_k|, not on absolute positions — verify.

Try this modification: Compute cosine similarity between PE(pos=5) and PE(pos=5+k) for k = 1..50. You'll see a smooth decay with distance — that's the property that lets attention use PE as a distance signal.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Positional encoding sounds like a solved detail until you try to build a 128K-context LLM. Then it becomes half of your problem.

Why RoPE won. Every recent open-weights LLM (Llama 2/3/4, Mistral, Mixtral, Qwen, DeepSeek, Yi) uses RoPE. The reason is a happy combination: (1) it's relative, so attention behaves consistently across positions; (2) it's a rotation, so norms are preserved and it doesn't destabilise activations; (3) it needs zero extra parameters; (4) with adjustments (base scaling, NTK, YaRN) it can be extended to much longer contexts than it was trained on. GPT-2's learned PE, by contrast, has a hard ceiling at its training length.

Context extension tricks. The base frequency 10000 in RoPE was tuned for ~2K context. For longer contexts, you can (a) linearly interpolate positions (Meta's Position Interpolation, 2023), (b) scale the base frequency (dynamic NTK), or (c) use YaRN (Peng et al., 2023) which combines both smartly. Every 100K+ context model uses some variant of this.

Bugs and gotchas:

• Off-by-one in absolute positions. BERT uses [CLS] at position 0 and shifts real tokens to position 1. Get this wrong during fine-tuning and every prediction is off by one position.
• Sinusoidal PE added after dropout — a subtle bug that reduces PE strength. The original Transformer applies dropout to the sum token_embedding + positional_encoding.
• RoPE base mismatch on inference. If you train with base = 10000 and load into an inference engine that assumes base = 500000, positions are wildly wrong. Ship the base with the model.
• RoPE applied to values. RoPE rotates Q and K only — not V. Applying it to V is a common bug in from-scratch implementations that silently degrades quality.

ALiBi (Attention with Linear Biases) is the main competitor: instead of encoding position at all, it just adds a distance-based bias to the raw attention scores (score += -m · |i - j|). BLOOM and MPT use it. It extrapolates well but was less quality-preserving than RoPE in careful comparisons, which is why RoPE dominated.

(e) Quiz + exercise · 10 min

1. Why must transformers have some form of positional encoding at all?
2. What's the fundamental difference between absolute and relative positional encodings?
3. In sinusoidal PE, why do we use sines/cosines at different frequencies across the embedding dimensions?
4. What is the single biggest advantage of RoPE over learned positional embeddings?
5. Name one production technique used to extend a model's context length beyond its trained maximum.

Stretch (connects to S108 — Q/K/V Math): RoPE rotates Q and K but not V. Explain why applying the same rotation to V would break the mathematical elegance of the relative-position property.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Positional Encoding? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S111): Full Transformer Architecture — Encoder + Decoder
• Previous (S109): Multi-Head Attention — Parallel Views
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.