DL S038 · Positional Encodings — Sinusoidal, Learned, and a Sneak Peek at RoPE

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 38 / 80 · Module M07 · ~2 hours

The story

Here's an experiment you can run in three minutes: take the S036 attention module, feed it a sentence, then feed it the same sentence with the words shuffled. You'll get the same set of output vectors, just permuted. Attention doesn't know about word order. Permute the input, the output permutes identically. Permutation-equivariance is a beautiful property for set data (deep sets, point clouds) but a disaster for language, where "dog bites man" and "man bites dog" are very different sentences.

Something has to inject position information. The obvious ideas are (1) concatenate a position index onto every embedding, (2) add a learned position embedding at each slot, or (3) use a fixed function of position that has convenient math properties. The original transformer paper picked (3): sinusoidal encodings. GPT-2 picked (2): learned. Llama and every 2023+ frontier model picked a fourth option: rotary position embeddings (RoPE), which cleverly encodes relative position directly inside the Q/K dot product without ever adding to the embeddings.

This session we derive sinusoidal PE from a specific wish — "I want a fixed encoding where relative offsets are computable from any starting position" — and out falls sines and cosines of geometric-frequency waves. Then we contrast with learned embeddings, and give RoPE just enough of a preview that when S043 asks you to implement it, you won't be starting from zero.

Prerequisites

• Session 036 and Session 037 — where attention comes from.
• Basic trig (sine, cosine, angle-addition identities). We'll re-derive angle-addition; if you're rusty, 3B1B's Chapter 4 of Essence of Trigonometry is a 15-minute refresher.

1 · The permutation problem — proved in three lines

Claim: self-attention is permutation-equivariant.

Let P be a permutation matrix (rows of the identity, shuffled). Then PX is X with rows permuted. And:

because (PX)(PX)ᵀ = P (XXᵀ) Pᵀ and soft-max is row-wise, and pre-multiplying by P and post-multiplying by Pᵀ on V = PX cancels out to leave P · Attention(X).

Consequence: the model's output for shuffled input is just the shuffled output for unshuffled input. It literally cannot distinguish "dog bites man" from "man bites dog" from information in attention alone.

We must inject position info somewhere. The two clean places are:

• Additive on the input: X ← X + PE, where PE is a (T, d_model) matrix of position codes.
• Inside the Q/K dot product: rotate Q and K by a position-dependent angle so their inner product depends on relative position. This is RoPE.

Original transformer, GPT-2, BERT all use additive. Modern LMs use RoPE. We derive additive first.

2 · Deriving sinusoidal PE from a wish

Wish: "I want a position encoding PE(pos) ∈ ℝ^d such that for any offset k, PE(pos + k) is a linear function of PE(pos)."

Why this wish? Because attention scores are computed via dot products, and if PE(pos + k) = M_k · PE(pos) for some matrix M_k that depends only on k (not on pos), then the model can potentially learn to be sensitive to relative positions — the crucial thing for language.

The elegant solution: pair up dimensions and rotate each pair by an angle proportional to pos. For a single 2-D pair at frequency ω:

Then by the angle-addition formulas:

That 2×2 matrix on the right is a rotation by angle ω k. It depends only on k. Wish granted.

Now to cover a large range of positions with different sensitivities, we use d/2 different frequencies. The original paper chose a geometric progression:

• At i = 0: ω_0 = 1 → wavelength 2π.
• At i = d/2: ω = 10000^{-1} = 0.0001 → wavelength 20000π ≈ 62800.

So low-index dimensions fluctuate rapidly with position (fine-grained locality), high-index dimensions barely change (broad position bucket). Together they give the model a multi-scale positional "clock".

Final formula:

Read as: "even indices get sines, odd indices get cosines, of a position scaled by a geometric-frequency wavelength".

3 · Implementing sinusoidal PE in PyTorch

import torch, math
 
def sinusoidal_pe(max_len, d_model):
    pe = torch.zeros(max_len, d_model)
    pos = torch.arange(0, max_len).unsqueeze(1).float()        # (T, 1)
    div = torch.exp(torch.arange(0, d_model, 2).float() *
                    -(math.log(10000.0) / d_model))            # (d/2,)
    pe[:, 0::2] = torch.sin(pos * div)
    pe[:, 1::2] = torch.cos(pos * div)
    return pe                                                  # (T, d_model)

Six lines. The exp(arange * -log(10000) / d) trick is just 10000^{-2i/d} in a numerically stable form (computing 10000^{-large} directly underflows in float32).

Add it to your input embeddings:

x = token_embed(input_ids)              # (B, T, d_model)
x = x + sinusoidal_pe(T, d_model)[:T]   # broadcast over batch

That's it. PE is a constant, not a parameter. It's the same for every sequence, every batch, forever.

3.1 · Sanity check — the "PE heatmap"

Plot PE as a (max_len, d_model) heatmap. You'll see the classic barcode pattern: rapid oscillation in the leftmost columns, slow oscillation on the right. This visualisation is in every transformer tutorial ever; it's worth generating yourself once so you internalise that low-index dims move fast, high-index dims move slow.

import matplotlib.pyplot as plt
pe = sinusoidal_pe(100, 128).numpy()
plt.imshow(pe, aspect='auto', cmap='RdBu')
plt.xlabel('embedding dim'); plt.ylabel('position')
plt.colorbar(); plt.show()

pe = sinusoidal_pe(200, 128)
 
k = 5
for pos in [0, 10, 50, 100, 150]:
    dot = (pe[pos] @ pe[pos + k]).item()
    print(f"pos={pos:3d}, pos+{k}={pos+k:3d}   dot = {dot:8.4f}")

4 · Learned positional embeddings

Simpler alternative: just make position an nn.Embedding.

class LearnedPE(nn.Module):
    def __init__(self, max_len, d_model):
        super().__init__()
        self.pe = nn.Embedding(max_len, d_model)
 
    def forward(self, x):
        T = x.size(1)
        pos = torch.arange(T, device=x.device)
        return x + self.pe(pos)

Two lines effectively. Trained end-to-end with the rest of the model. GPT-2, GPT-3, BERT all use this.

The reason nobody uses either in 2025 is that neither one gets relative position quite right, and RoPE does.

5 · RoPE preview — position by rotation, not addition

Instead of adding a PE to the embedding and then computing q · k, RoPE rotates Q and K in 2-D pairs by a position-dependent angle before the dot product. The magic property: the dot product ⟨R_{pos_q} q, R_{pos_k} k⟩ depends only on the difference pos_q - pos_k, not on the absolute positions. Pure relative encoding, no extra parameters, no length ceiling.

Sketch of the derivation for a single 2-D pair (q_1, q_2):

So the score depends only on n - m. Extended to d-dimensional Q and K by pairing consecutive dims and using the same geometric-frequency scheme as sinusoidal PE.

That's the whole idea. We build it in S043 with three lines of tensor manipulation. If you want to preview the code:

def apply_rope(q, cos, sin):
    q1, q2 = q[..., ::2], q[..., 1::2]
    return torch.stack([q1 * cos - q2 * sin,
                        q1 * sin + q2 * cos], dim=-1).flatten(-2)

Don't fully unpack this yet. Just note: RoPE is applied to Q and K inside attention, not added to embeddings.

6 · When each scheme is used (in the real world)

If you're implementing anything from scratch in 2025, use RoPE. If you're studying the original paper for educational clarity, sinusoidal is fine. If you're forking GPT-2, keep the learned embeddings so you don't invalidate the pretrained weights.

7 · Pitfalls

8 · Mermaid — where PE plugs in

One addition. Before block 1. Never inside a block.

9 · Modern-2025 twist — YaRN, LongRoPE, and the million-token context race

When everyone switched to RoPE around 2021–2022, we inherited a new problem. RoPE encodes position via wave frequencies chosen at training time; if you train with 4k context and try to run at 128k, the high-frequency rotations wrap around into positions the model has never seen, and quality collapses hard — what the community calls the train–test length mismatch. The 2023–2025 arc solved this with a small family of clever tricks:

• Position Interpolation (PI) — Chen et al., Meta AI, June 2023. Scale the position indices down by L_train / L_target before feeding into RoPE. Trivial change, works OK up to 4× extension after a few hundred fine-tune steps.
• NTK-aware scaling — bloc97 on r/LocalLLaMA, July 2023. Adjust the RoPE base (default 10000) instead of interpolating positions. Better preservation of high-frequency detail.
• YaRN (Yet another RoPE extensioN) — Peng, Quesnelle, Fan, Shippole, Nov 2023. Combines NTK-by-parts + attention-scale temperature. Shipped in Qwen 2, DeepSeek-V3, Mistral Large, Command-R+, and most open 32k–128k models today. ~400 steps of fine-tuning extends a 4k model to 128k with minimal quality loss.
• LongRoPE — Ding et al., Microsoft, Feb 2024. Learns per-dimension frequency rescaling via evolutionary search. First method to hit 2M-token context on a 4k-pretrained model. Powers Phi-3 Long and parts of Gemini 1.5's context extension.
• RoPE base = 500,000 — what Llama 3 did instead: retrain from scratch with a much larger base so 128k "just works." Simple, brute-force, only available to labs with pretraining budgets.
• YaRN + chunked attention for 10M contexts — Llama 4 Scout, April 2025. Interleaves full-attention layers with chunked-attention layers (no RoPE in the chunked layers) so KV memory stays feasible.

A cheat rule of thumb for your future self: if you fine-tune a Llama-family or Qwen-family model to a longer context in 2026, use YaRN, s = target_len / train_len, and fine-tune on ~1B tokens of long-form data. That recipe has held for two years and nothing simpler works as well.

Further reading:

• YaRN paper — read §3 (NTK-by-parts intuition).
• LongRoPE — the evolutionary-search recipe.
• Eleuther's context-extension survey (2024) — practical benchmarks across PI/NTK/YaRN/LongRoPE.

Retention scaffold

One-line summary (write it in your own words): _______________________________________________

Spaced review: re-read §2 (sinusoidal derivation) and §5 (RoPE preview) in 24 hours. Revisit the full session on day 7, focusing on §9 as the bridge into S043.

Next session (S039): we assemble the full transformer block — attention + FFN + LayerNorm + residual, wired in the specific order that trains well. Pre-norm vs post-norm, RMSNorm, SwiGLU, and why every 2024 model looks almost identical.

Sticky note (keep on your desk): Attention is order-blind. PE injects order. RoPE injects RELATIVE order for free inside the dot product. YaRN scales RoPE to million-token context.

Previous: ← DL S037 · Next: DL S039 →