S107 · Attention Intuition — Why RNNs Failed, Why Attention Won

The idea that changed everything.

Module M13: NLP & Transformers · Session 107 of 130 · Track: LLM 👀

What you'll be able to do after this session

Explain Attention Intuition to a friend in one minute, out loud, no notes.
Recognise it when you see it in real code / systems / papers.
Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

S103 — RNNs & LSTMs — Sequences & the Vanishing Gradient

Pre-read (skim before the session)

🎥 Intuition (5–15 min) · Attention in transformers, visually explained — 3Blue1Brown — The clearest possible intuition for attention.
🎥 Deep dive (30–60 min) · Let's build GPT from scratch — Andrej Karpathy — Karpathy derives attention live in a notebook.
🎥 Hands-on demo (10–20 min) · Illustrated Transformer walkthrough — HeduAI — Follows Jay Alammar's diagrams step by step.
📖 Canonical article · The Illustrated Transformer — Jay Alammar — The most-shared explainer of attention on the internet.
📖 Book chapter / tutorial · Attention Is All You Need (Vaswani 2017) — The original paper — dense but genuinely readable.
📖 Engineering blog · Lilian Weng — Attention? Attention! — Rigorous walk-through of every attention variant.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The idea that changed everything.

When you read the sentence "The animal didn't cross the street because it was too tired", how do you know what "it" refers to? Your brain, without you noticing, weighs every previous word and asks "which of you are relevant to me right now?" — and decides "the animal", not "the street". That's exactly what attention is: a learned mechanism that, for every position in a sequence, computes a weighted sum of information from all other positions, where the weights depend on the content itself.

RNNs and LSTMs (S103) tried to do this by passing information forward through a hidden state, but they were fundamentally serial (each step depends on the previous one) and lossy (long-range info gets diluted). Attention changes both: it's parallel (every position can look at every other position in a single matrix multiplication) and direct (no intermediate steps to dilute a signal from 500 tokens back). That's why transformers train orders of magnitude faster than LSTMs on modern GPUs, and why they don't suffer from vanishing gradients over long contexts.

The whole formula (which you'll derive in S108) reduces to: for each token, compute how similar it is to every other token, softmax the similarities to get weights, and take the weighted average of the "value" vectors. That's it. Everything from GPT-4 to Claude to Gemini is stacks of this operation with more parameters.

Gotcha to carry forever: Attention is O(N²) in sequence length — a 100K-token context needs a 100K × 100K attention matrix, which is 40GB in float32. This is the single reason long-context LLMs need clever tricks (Flash Attention, sliding windows, sparse attention, state-space models). Never say attention is "just a weighted sum" without immediately remembering "…that costs quadratic memory".

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The self-attention operation in words. For each token in a sequence, you compute three vectors: a query ("what am I looking for?"), a key ("what do I offer?"), and a value ("what do I actually contribute if picked?"). Then, for a given query, you take its dot product with every key to get similarity scores, softmax those scores into weights, and use the weights to average the values.

Worked example — attending to context. Consider the toy sentence: [The, cat, sat], 3 tokens. After the linear projections, we get queries, keys, values per token. For the query q_sat, we compute similarities: q_sat · k_the = 0.2, q_sat · k_cat = 3.1, q_sat · k_sat = 0.5. After softmax → [0.05, 0.85, 0.10]. The output for position "sat" is 0.05 · v_the + 0.85 · v_cat + 0.10 · v_sat — heavily weighted toward "cat", because who sat matters more than the or sat itself.

Worked example — matrix shapes. For batch B, sequence length T, and embedding dim d:

X: [B, T, d]
Q, K, V: [B, T, d]
Q @ K^T: [B, T, T] — this is the T×T attention matrix per batch item.
Softmax along the last dim, then multiply by V: [B, T, d] — same shape as input, ready for the next layer.

Operation	Cost	Notes
Q · K transpose	O(T² · d)	The quadratic term everyone fears
softmax	O(T²)	Trivial compute, big memory
· V	O(T² · d)
Total memory	O(T² + T·d)	Attention matrix dominates for large T

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# attention_from_scratch.py — the entire self-attention operation in ~20 lines
import torch, torch.nn.functional as F, math
 
torch.manual_seed(0)
B, T, D = 1, 4, 8                                # batch, seq len, embed dim
X = torch.randn(B, T, D)                         # pretend token embeddings
 
# One linear each for Q, K, V (in real transformers these are learnable)
Wq, Wk, Wv = (torch.randn(D, D) for _ in range(3))
Q, K, V = X @ Wq, X @ Wk, X @ Wv                 # each [B, T, D]
 
# ---- attention core ------------------------------------------------------
scores  = Q @ K.transpose(-2, -1)                # [B, T, T]  raw similarities
scores  = scores / math.sqrt(D)                  # <-- the SCALED bit
weights = F.softmax(scores, dim=-1)              # per-row probability distribution
out     = weights @ V                            # [B, T, D]  weighted values
 
print("attention weight matrix (rows sum to ~1):")
print(weights[0].round(decimals=2))
print("\noutput shape:", out.shape)              # -> torch.Size([1, 4, 8])
 
# ---- causal mask (as used in GPT) ---------------------------------------
mask = torch.triu(torch.ones(T, T), diagonal=1).bool()   # upper triangle True
scores_masked = scores.masked_fill(mask, float("-inf"))
weights_causal = F.softmax(scores_masked, dim=-1)
print("\ncausal weights (each row can only see itself and past):")
print(weights_causal[0].round(decimals=2))
 
# ---- compare against PyTorch's built-in --------------------------------
ref = F.scaled_dot_product_attention(Q, K, V, is_causal=True)
print("\nRef output shape (causal):", ref.shape)

Checklist while it runs:

Confirm the un-masked weight rows sum to ~1.0 — that's the softmax invariant.
Confirm the causal weight matrix is strictly lower-triangular — that's what stops GPT from "cheating" by peeking at future tokens.
Change D to 64 and check that attention weights become less uniform (higher d ⇒ larger raw scores ⇒ sharper softmax; /sqrt(D) scaling counteracts this).
Remove the / math.sqrt(D) step and set D=512; observe softmax saturating into near-one-hot distributions with tiny gradients — that's the "why we scale" story.
Try T = 4096 and watch RAM balloon.

Try this modification: Wrap this into a MultiHeadSelfAttention module by splitting D=8 into nh=2 heads × head_dim=4, running attention independently per head, and concatenating outputs. Preview of S109.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Attention as originally written (Vaswani 2017) is not what runs in production LLMs today — every serious inference stack replaces it with a fused, memory-optimised kernel.

Flash Attention (Dao et al., 2022) rewrites the attention computation to never materialise the full T×T matrix in HBM. Instead it tiles the computation across on-chip SRAM, doing the softmax incrementally with the online-softmax trick. Result: 2–4× speedup and 5–20× less memory. It's the default in PyTorch 2 (F.scaled_dot_product_attention dispatches to it automatically on Ampere/Hopper GPUs) and in every vLLM / TensorRT-LLM serving stack. If you write your own attention loop today, you're leaving 3× performance on the table.

KV cache during generation. At inference, when generating token N+1, you don't need to recompute K and V for tokens 1..N — you already computed them. Every autoregressive LLM keeps a "KV cache" per layer, roughly 2 · n_layers · T · d_head · n_heads · batch · 2 bytes (fp16). For a 70B model this can be tens of GB per user at long context, and it's often the true bottleneck for serving throughput. Techniques like Grouped-Query Attention (Llama-2 70B), Multi-Query Attention (PaLM), and Paged Attention (vLLM) all attack the KV-cache size.

Long-context tricks. True O(N²) attention becomes infeasible past ~32K tokens. Modern solutions: sliding-window attention (Mistral), sparse patterns (Longformer, BigBird), linear attention approximations (Performer, Linformer), and state-space models (Mamba, RWKV) that replace attention entirely with O(N) recurrent cores. Claude 3.5 (200K), Gemini 1.5 (1M+), and Llama 4 all rely on some combination of these.

Common bugs: missing causal mask in a decoder → the model "cheats" during training and produces garbage at inference (loss keeps dropping while quality collapses); wrong softmax dim → the shapes match, so it silently trains nonsense; dropping /sqrt(d) → gradients vanish through the softmax at large model dimensions.

(e) Quiz + exercise · 10 min

In one sentence, describe what the attention mechanism does for each position in a sequence.
Why is attention "parallel" while an LSTM is "serial"?
What is the memory complexity of standard attention in the sequence length T?
What role does the 1/sqrt(d) scaling factor play, and what breaks without it?
What does the causal mask enforce in a decoder-only transformer like GPT?

Stretch (connects to S103 — RNNs & LSTMs): LSTMs vs attention: describe one task where an LSTM is still the better choice today, and one where attention decisively wins.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

What is Attention Intuition? (one sentence, no jargon)
When would you use it? (one concrete example)
When would you NOT use it? (one anti-pattern)

What comes next

Next session (S108): Q/K/V Math — Scaled Dot-Product Attention Derived
Previous (S106): Tokenization — BPE, WordPiece, SentencePiece
Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.