mlintermediate 110m read

DL S038 · Positional Encodings — Sinusoidal, Learned, and a Sneak Peek at RoPE

Attention is permutation-invariant. Language isn't. We derive sinusoidal positional encodings from the wish for shift-equivariant relative offsets, compare to learned embeddings, and preview RoPE.

🧠SoftwareM07 · Transformers from scratch· Session 038 of 130 110 min

🎯 Derive sinusoidal positional encodings from wanting shift-equivariant relative offsets, and know when to use learned vs sinusoidal vs RoPE in practice.

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 38 / 80 · Module M07 · ~2 hours

The story

Here's an experiment you can run in three minutes: take the S036 attention module, feed it a sentence, then feed it the same sentence with the words shuffled. You'll get the same set of output vectors, just permuted. Attention doesn't know about word order. Permute the input, the output permutes identically. Permutation-equivariance is a beautiful property for set data (deep sets, point clouds) but a disaster for language, where "dog bites man" and "man bites dog" are very different sentences.

Something has to inject position information. The obvious ideas are (1) concatenate a position index onto every embedding, (2) add a learned position embedding at each slot, or (3) use a fixed function of position that has convenient math properties. The original transformer paper picked (3): sinusoidal encodings. GPT-2 picked (2): learned. Llama and every 2023+ frontier model picked a fourth option: rotary position embeddings (RoPE), which cleverly encodes relative position directly inside the Q/K dot product without ever adding to the embeddings.

This session we derive sinusoidal PE from a specific wish — "I want a fixed encoding where relative offsets are computable from any starting position" — and out falls sines and cosines of geometric-frequency waves. Then we contrast with learned embeddings, and give RoPE just enough of a preview that when S043 asks you to implement it, you won't be starting from zero.

You will be able to

Explain why attention needs positional encodings using one concrete counter-example.
Derive the sinusoidal PE formula from the shift-equivariance property.
Implement sinusoidal PE in ~5 lines of PyTorch that matches the reference.
State two advantages and two disadvantages of learned vs sinusoidal PE.
Sketch the RoPE 2-D rotation idea in one paragraph before we build it in S043.
Predict which PE scheme extrapolates to longer sequences at inference time and why.

Prerequisites

Session 036 and Session 037 — where attention comes from.
Basic trig (sine, cosine, angle-addition identities). We'll re-derive angle-addition; if you're rusty, 3B1B's Chapter 4 of Essence of Trigonometry is a 15-minute refresher.

Pre-read · watch and skim (~30 min)

💡 Intuition 16 min

How positional encoding in transformers works

Hedu AI

Best visual walk-through of sinusoidal PE. The waveform diagrams stack from low to high frequency and it just clicks.

🔬 Deep dive 11 min

Rotary Positional Embeddings: Combining Absolute and Relative

Efficient NLP

Cleanest RoPE explanation on YT. You'll come back to this before S043.

📖 Reading 15 min

Transformer Architecture: The Positional Encoding

Amirhossein Kazemnejad

Long-form derivation with all the trig steps written out. If a step feels magical during our derivation, this post writes it out slowly.

📖 Reading 20 min

RoFormer: Enhanced Transformer with Rotary Position Embedding

Su et al. 2021

The RoPE paper. Read only the introduction and Section 3.2 this session — the full paper we'll dissect in S043.

1 · The permutation problem — proved in three lines

Claim: self-attention is permutation-equivariant.

Let P be a permutation matrix (rows of the identity, shuffled). Then PX is X with rows permuted. And:

\text{Attention}(PX) = \operatorname{softmax}\!\left(\frac{(PX)(PX)^\top}{\sqrt{d_k}}\right)(PX) = P \cdot \text{Attention}(X)

because (PX)(PX)ᵀ = P (XXᵀ) Pᵀ and soft-max is row-wise, and pre-multiplying by P and post-multiplying by Pᵀ on V = PX cancels out to leave P · Attention(X).

Consequence: the model's output for shuffled input is just the shuffled output for unshuffled input. It literally cannot distinguish "dog bites man" from "man bites dog" from information in attention alone.

We must inject position info somewhere. The two clean places are:

Additive on the input: X ← X + PE, where PE is a (T, d_model) matrix of position codes.
Inside the Q/K dot product: rotate Q and K by a position-dependent angle so their inner product depends on relative position. This is RoPE.

Original transformer, GPT-2, BERT all use additive. Modern LMs use RoPE. We derive additive first.

2 · Deriving sinusoidal PE from a wish

Wish: "I want a position encoding PE(pos) ∈ ℝ^d such that for any offset k, PE(pos + k) is a linear function of PE(pos)."

Why this wish? Because attention scores are computed via dot products, and if PE(pos + k) = M_k · PE(pos) for some matrix M_k that depends only on k (not on pos), then the model can potentially learn to be sensitive to relative positions — the crucial thing for language.

A wall of piano keys as the position clock

🌍 Real world

💻 Code world

The elegant solution: pair up dimensions and rotate each pair by an angle proportional to pos. For a single 2-D pair at frequency ω:

PE(pos) = \begin{bmatrix} \sin(\omega \cdot pos) \\ \cos(\omega \cdot pos) \end{bmatrix}

Then by the angle-addition formulas:

PE(pos + k) = \begin{bmatrix} \sin(\omega (pos + k)) \\ \cos(\omega (pos + k)) \end{bmatrix} = \begin{bmatrix} \cos(\omega k) & \sin(\omega k) \\ -\sin(\omega k) & \cos(\omega k) \end{bmatrix} PE(pos)

That 2×2 matrix on the right is a rotation by angle ω k. It depends only on k. Wish granted.

Now to cover a large range of positions with different sensitivities, we use d/2 different frequencies. The original paper chose a geometric progression:

\omega_i = \frac{1}{10000^{2i/d}} \quad \text{for } i = 0, 1, \ldots, d/2 - 1

At i = 0: ω_0 = 1 → wavelength 2π.
At i = d/2: ω = 10000^{-1} = 0.0001 → wavelength 20000π ≈ 62800.

So low-index dimensions fluctuate rapidly with position (fine-grained locality), high-index dimensions barely change (broad position bucket). Together they give the model a multi-scale positional "clock".

Final formula:

\begin{aligned} PE_{(pos, 2i)} &= \sin(pos / 10000^{2i/d}) \\ PE_{(pos, 2i+1)} &= \cos(pos / 10000^{2i/d}) \end{aligned}

Read as: "even indices get sines, odd indices get cosines, of a position scaled by a geometric-frequency wavelength".

3 · Implementing sinusoidal PE in PyTorch

import torch, math
 
def sinusoidal_pe(max_len, d_model):
    pe = torch.zeros(max_len, d_model)
    pos = torch.arange(0, max_len).unsqueeze(1).float()        # (T, 1)
    div = torch.exp(torch.arange(0, d_model, 2).float() *
                    -(math.log(10000.0) / d_model))            # (d/2,)
    pe[:, 0::2] = torch.sin(pos * div)
    pe[:, 1::2] = torch.cos(pos * div)
    return pe                                                  # (T, d_model)

Six lines. The exp(arange * -log(10000) / d) trick is just 10000^{-2i/d} in a numerically stable form (computing 10000^{-large} directly underflows in float32).

Add it to your input embeddings:

x = token_embed(input_ids)              # (B, T, d_model)
x = x + sinusoidal_pe(T, d_model)[:T]   # broadcast over batch

That's it. PE is a constant, not a parameter. It's the same for every sequence, every batch, forever.

3.1 · Sanity check — the "PE heatmap"

Plot PE as a (max_len, d_model) heatmap. You'll see the classic barcode pattern: rapid oscillation in the leftmost columns, slow oscillation on the right. This visualisation is in every transformer tutorial ever; it's worth generating yourself once so you internalise that low-index dims move fast, high-index dims move slow.

import matplotlib.pyplot as plt
pe = sinusoidal_pe(100, 128).numpy()
plt.imshow(pe, aspect='auto', cmap='RdBu')
plt.xlabel('embedding dim'); plt.ylabel('position')
plt.colorbar(); plt.show()

Try itProve the two-position dot product depends only on the offset, not the position

Run this in a notebook:

pe = sinusoidal_pe(200, 128)
 
k = 5
for pos in [0, 10, 50, 100, 150]:
    dot = (pe[pos] @ pe[pos + k]).item()
    print(f"pos={pos:3d}, pos+{k}={pos+k:3d}   dot = {dot:8.4f}")

All five dot products should be essentially the same value — the encoding preserves relative offsets even at different absolute positions. Now vary k from 1 to 50 and plot the dot product as a function of k. You'll see a smooth curve that decays with distance — the model gets "distance for free" from this geometry. That's the entire theoretical reason sinusoidal PE beats a random one-hot lookup.

💡 Hint · Compute `PE(pos)·PE(pos+k)` for several `pos` at fixed `k` and confirm the value barely moves.

4 · Learned positional embeddings

Simpler alternative: just make position an nn.Embedding.

class LearnedPE(nn.Module):
    def __init__(self, max_len, d_model):
        super().__init__()
        self.pe = nn.Embedding(max_len, d_model)
 
    def forward(self, x):
        T = x.size(1)
        pos = torch.arange(T, device=x.device)
        return x + self.pe(pos)

Two lines effectively. Trained end-to-end with the rest of the model. GPT-2, GPT-3, BERT all use this.

Learned vs sinusoidal — the tradeoff

Learned: adapts to your data distribution; slightly better in-distribution loss; adds max_len × d_model params.
Learned drawback: FIXED max length. Feed a longer sequence at inference than you trained with → out-of-vocab position → crash (or worse, garbage output if you naively wrap around).
Sinusoidal: zero learnable params; extrapolates to any length without crashing (though performance degrades past training length).
Sinusoidal drawback: fixed inductive bias; can't specialise if the data has weird position statistics (e.g., every 100th token is a section header).

The reason nobody uses either in 2025 is that neither one gets relative position quite right, and RoPE does.

5 · RoPE preview — position by rotation, not addition

Instead of adding a PE to the embedding and then computing q · k, RoPE rotates Q and K in 2-D pairs by a position-dependent angle before the dot product. The magic property: the dot product ⟨R_{pos_q} q, R_{pos_k} k⟩ depends only on the difference pos_q - pos_k, not on the absolute positions. Pure relative encoding, no extra parameters, no length ceiling.

Sketch of the derivation for a single 2-D pair (q_1, q_2):

R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}

\langle R_{m\theta} q, R_{n\theta} k \rangle = q^\top R_{m\theta}^\top R_{n\theta} k = q^\top R_{(n-m)\theta} k

So the score depends only on n - m. Extended to d-dimensional Q and K by pairing consecutive dims and using the same geometric-frequency scheme as sinusoidal PE.

That's the whole idea. We build it in S043 with three lines of tensor manipulation. If you want to preview the code:

def apply_rope(q, cos, sin):
    q1, q2 = q[..., ::2], q[..., 1::2]
    return torch.stack([q1 * cos - q2 * sin,
                        q1 * sin + q2 * cos], dim=-1).flatten(-2)

Don't fully unpack this yet. Just note: RoPE is applied to Q and K inside attention, not added to embeddings.

6 · When each scheme is used (in the real world)

What frontier models actually use (2024)

Original Transformer, T5, most 2017–2019 papers: sinusoidal absolute PE.
GPT-2, GPT-3, BERT, RoBERTa: learned absolute PE.
T5 (position bias): learned bucket-based relative bias added to attention scores.
Llama 1/2/3, Mistral, Qwen, DeepSeek, Gemma: RoPE.
ALiBi (attention-with-linear-biases): a simpler alternative to RoPE, used in some MPT models.

If you're implementing anything from scratch in 2025, use RoPE. If you're studying the original paper for educational clarity, sinusoidal is fine. If you're forking GPT-2, keep the learned embeddings so you don't invalidate the pretrained weights.

7 · Pitfalls

War story Sequence longer than max_len

Learned PE with max_len = 512 and you send in a length-513 sequence: IndexError at best, wrap-around garbage at worst. Sinusoidal handles it (the formula defined for any pos), but the model's attention patterns weren't trained on those positions, so quality degrades from ~T = training_len onwards. Reliable long-context needs RoPE with careful frequency-base tuning (S043).

War story Applying PE inside a residual by mistake

Some tutorials show x = x + PE inside the transformer block, executed once per layer. That's wrong. PE is added ONCE, at the input, before the first block. Adding it every layer amplifies it and mangles training. Add once, forget.

War story Confusing (max_len, d_model) with (d_model, max_len)

Sinusoidal PE is (T, d_model). If you swap the axes you'll broadcast a (d_model, T) against (B, T, d_model) and get a shape error, or (nightmare case) a weird broadcast that runs but produces garbage. Always assert pe.shape == (max_len, d_model) before you use it.

8 · Mermaid — where PE plugs in

One addition. Before block 1. Never inside a block.

9 · Modern-2025 twist — YaRN, LongRoPE, and the million-token context race

When everyone switched to RoPE around 2021–2022, we inherited a new problem. RoPE encodes position via wave frequencies chosen at training time; if you train with 4k context and try to run at 128k, the high-frequency rotations wrap around into positions the model has never seen, and quality collapses hard — what the community calls the train–test length mismatch. The 2023–2025 arc solved this with a small family of clever tricks:

Position Interpolation (PI) — Chen et al., Meta AI, June 2023. Scale the position indices down by L_train / L_target before feeding into RoPE. Trivial change, works OK up to 4× extension after a few hundred fine-tune steps.
NTK-aware scaling — bloc97 on r/LocalLLaMA, July 2023. Adjust the RoPE base (default 10000) instead of interpolating positions. Better preservation of high-frequency detail.
YaRN (Yet another RoPE extensioN) — Peng, Quesnelle, Fan, Shippole, Nov 2023. Combines NTK-by-parts + attention-scale temperature. Shipped in Qwen 2, DeepSeek-V3, Mistral Large, Command-R+, and most open 32k–128k models today. ~400 steps of fine-tuning extends a 4k model to 128k with minimal quality loss.
LongRoPE — Ding et al., Microsoft, Feb 2024. Learns per-dimension frequency rescaling via evolutionary search. First method to hit 2M-token context on a 4k-pretrained model. Powers Phi-3 Long and parts of Gemini 1.5's context extension.
RoPE base = 500,000 — what Llama 3 did instead: retrain from scratch with a much larger base so 128k "just works." Simple, brute-force, only available to labs with pretraining budgets.
YaRN + chunked attention for 10M contexts — Llama 4 Scout, April 2025. Interleaves full-attention layers with chunked-attention layers (no RoPE in the chunked layers) so KV memory stays feasible.

A cheat rule of thumb for your future self: if you fine-tune a Llama-family or Qwen-family model to a longer context in 2026, use YaRN, s = target_len / train_len, and fine-tune on ~1B tokens of long-form data. That recipe has held for two years and nothing simpler works as well.

Further reading:

YaRN paper — read §3 (NTK-by-parts intuition).
LongRoPE — the evolutionary-search recipe.
Eleuther's context-extension survey (2024) — practical benchmarks across PI/NTK/YaRN/LongRoPE.

Common misconception

✗ What most people think

"Positional encodings tell the model where each token is. The vector added at position 5 is the model's representation of the number 5 — an index, encoded as a vector."

✓ What is actually true

What the model actually needs and uses is relative offset, not absolute index. The sinusoidal scheme is built so that the encoding at position p + k is a fixed linear function of the encoding at p, with the transform depending only on k — so "seven tokens back" is expressible as one linear operation regardless of where you are in the sequence. Attention scores then depend chiefly on differences of positions rather than their values. Reading the vector as "the number 5" gets you nothing; reading it as "a point on a set of clocks running at different speeds, where the difference between two points encodes their separation" gets you everything, including why extrapolation beyond the trained length fails.

Why the myth is so sticky

Because "the model needs to know position, so we give it position" is a complete, coherent explanation that predicts the right code. You add a vector indexed by position, it works, nothing contradicts you. And absolute encodings genuinely do carry the index — that part is true. The belief is only exposed when you ask why the frequencies are geometrically spaced rather than, say, a plain binary encoding of the integer, or why a model trained at length 512 collapses at length 2048 rather than degrading gently. Under the index view neither has an answer. Under the relative view both are immediate: the frequencies exist to make offsets linear, and extrapolation fails because the slowest clock has never completed a cycle beyond the training length, so those coordinates are in a regime the model has literally never seen.

Prove it to yourself

Check the linear-shift property directly — the thing that makes the encoding relative:

import numpy as np

def pe(pos, d=64):
    i = np.arange(0, d, 2)
    w = 1.0 / (10000 ** (i / d))
    out = np.zeros(d)
    out[0::2] = np.sin(pos * w)
    out[1::2] = np.cos(pos * w)
    return out

# The dot product between two encodings depends only on the OFFSET,
# not on where the pair sits in the sequence:
for base in (0, 10, 100, 500):
    print(base, [round(float(pe(base) @ pe(base + k)), 3) for k in (1, 2, 5, 20)])
# Each row is essentially identical -> the geometry is translation-invariant.
# That invariance is the property; the index is not.

From first principles

Start with the question

Why does a transformer need positional information at all, when an LSTM and a CNN never did? And why is it added to the embedding rather than concatenated to it?

1
Self-attention computes, for each position, a softmax-weighted sum over all positions. A sum does not care about the order of its terms.
forced by · addition is commutative — that is the whole argument, and it takes three lines to make it rigorous
2
The per-token projections W_q, W_k, W_v are applied identically to every position, so nothing in the layer distinguishes one slot from another either.
forced by · weight sharing across positions is what makes the layer applicable to any sequence length
3
Therefore permuting the input tokens permutes the outputs identically and changes nothing else: attention is a set operation. Recurrence gets order for free from the sequential update, and convolution gets it from the kernel's spatial layout — attention has neither.
forced by · order in the other two architectures is a structural consequence of how they touch the data, and attention touches all of it at once
4
So order must be injected into the token representations themselves, before attention sees them, since after that point the information is unrecoverable.
forced by · a set operation cannot recover an ordering that its input does not encode
5
Concatenation would work but costs dedicated dimensions at every layer and shrinks the width available for content. Addition costs nothing: in a high-dimensional space, and with the positional signal concentrated in a structured low-dimensional set of frequency pairs, the model can learn projections that read position and content largely separately from the same vector.
forced by · high-dimensional spaces have room for near-independent subspaces, so a learned linear map can separate the two components without a hard partition

⇒ Therefore

Therefore positional encoding is not a helpful extra — it is the repair for a structural blindness, and addition is chosen because dimensions are expensive and the model can learn to unmix.

Note two predictions worth checking. First, a transformer with the positional encoding removed should train to a non-trivial loss on tasks that are largely bag-of-words, and fail completely on anything order-sensitive — remove it and watch a copy or reverse task collapse to chance while topic classification barely moves. Second, since the injection happens once at the input while every layer needs order information, the positional signal must survive many residual additions and normalisations. That fragility is exactly what RoPE fixes by re-applying the position at every attention operation instead of trusting it to survive the stack, which is a large part of why it replaced the additive scheme.

Mental modelA bank of clocks

Give every token a panel of clock faces. The first clocks tick fast — a full revolution every couple of tokens. The last ones tick extremely slowly, taking thousands of tokens to complete a revolution. A position is just the reading of all the hands at once.

The fast clocks resolve fine local distinctions but wrap around constantly, so they cannot tell 3 from 103. The slow clocks never wrap within the sequence, so they encode coarse global position but cannot distinguish neighbours. Read together, they pin a position exactly — the same reason a mixed-radix odometer works. And to ask "how far apart are these two tokens", you compare hand angles, which is a difference, not a reading.

Attention is permutation-invariant. Any order information must be injected into the representation, because the mechanism itself cannot see it.
Sinusoidal: fixed, no parameters, geometrically spaced frequencies, offsets are linear maps. Learned: a plain embedding table over positions, more flexible but hard-capped at the trained maximum length.
Extrapolation fails at the slow end. Beyond the training length the low-frequency coordinates enter a range the model never observed, so behaviour there is unconstrained rather than merely approximate.
RoPE moves the position from an additive term at the input to a rotation applied to q and k inside every attention operation, which makes the score depend on relative offset by construction rather than by learned approximation.

🔔 Fires when you see

Fire this model the moment you see: a model that works within its context window and produces nonsense past it · a base or theta constant like 10000 being changed · context-extension methods — interpolation, NTK scaling, YaRN · attention over anything unordered where you were about to add position anyway · a max_position_embeddings field · a bag-of-words result that is suspiciously good.

The tradeoff

You are designing a model that must handle contexts far longer than you can afford to train on. Which positional scheme?

Learned absolute embeddings

+ you gain maximum flexibility — the model discovers whatever positional structure the task needs rather than being handed a prior; simple to implement, and it is just another embedding table with all the tooling that implies

− you pay a hard wall at the trained maximum length, because position n + 1 has no row in the table and there is no principled way to invent one; and every position is learned independently, so rare late positions are undertrained relative to early ones

pick when the maximum length is genuinely fixed and known up front — classification over bounded inputs, fixed-shape structured records — and you will never need to extend it

Sinusoidal absolute

+ you gain no parameters at all, defined for every integer position so it never hard-fails on a longer input, and offsets are exactly linear which gives the model a clean relative structure to exploit

− you pay defined does not mean useful — beyond the training length the slow frequencies enter unseen territory and quality degrades sharply rather than gracefully; and injecting once at the input means the signal must survive the whole residual stack

pick when you want a parameter-free baseline, or you are reproducing the original architecture and want to match it exactly

RoPE with an extension method

+ you gain relative by construction, since the score between two positions depends only on their difference; re-applied at every layer so nothing has to survive the stack; and it admits principled post-hoc extension by rescaling the frequency base, so an already-trained model can be pushed to longer contexts with modest continued training

− you pay naive extension trades resolution for range — squeezing more positions into the same rotational span degrades fine local discrimination, which is why the practical methods scale frequency bands unevenly rather than uniformly; and it adds a rotation to the attention hot path

pick when the target context length is not known at training time, or you expect to extend it later — which is the situation for essentially every general-purpose language model

What a senior engineer actually does

Use RoPE unless you have a specific reason not to, and the reason is not that it performs better in-distribution — at a fixed trained length the schemes are broadly comparable. The reason is optionality: RoPE is the only common scheme where extending the context later is a rescaling plus a short continued-training run rather than a retrain.

When you do extend, expect and measure the resolution tradeoff rather than assuming it away. The failure mode of naive interpolation is not that long-context retrieval breaks — it is that short-range behaviour quietly degrades because neighbouring positions have been pushed closer together in rotational space. Evaluate the short-context tasks after extending, not just the long ones. That is the check people skip, and it is where the regression hides.

Retention scaffold

Quick recall · click to reveal

★ = stretch question

One-line summary (write it in your own words): _______________________________________________

Spaced review: re-read §2 (sinusoidal derivation) and §5 (RoPE preview) in 24 hours. Revisit the full session on day 7, focusing on §9 as the bridge into S043.

Next session (S039): we assemble the full transformer block — attention + FFN + LayerNorm + residual, wired in the specific order that trains well. Pre-norm vs post-norm, RMSNorm, SwiGLU, and why every 2024 model looks almost identical.

Sticky note (keep on your desk): Attention is order-blind. PE injects order. RoPE injects RELATIVE order for free inside the dot product. YaRN scales RoPE to million-token context.

Previous: ← DL S037 · Next: DL S039 →