R22 · Week 22 Recall & Drill

Weekly revision · Week 22 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S106 — Tokenization — BPE, WordPiece, SentencePiece
• S107 — Attention Intuition — Why RNNs Failed, Why Attention Won
• S108 — Q/K/V Math — Scaled Dot-Product Attention Derived
• S109 — Multi-Head Attention — Parallel Views
• S110 — Positional Encoding — Sinusoidal, Learned, RoPE

90-min structure

Blank-page reconstruction · 30 min

S106 · Tokenization

1. Say how character-level and word-level tokenisation each fail, and what subword units fix.
2. Describe the merge-training algorithm in three sentences.
3. Explain what byte fallback means and why it matters.

Gotcha you probably forgot: adding a domain-specific token to a pretrained vocabulary does not give the model any knowledge of it. The new entry starts with a randomly-initialised embedding that no pretraining ever shaped, so until it is trained it carries less information than the multi-token spelling it replaced — and it also invalidates any cached tokenisation you had. Vocabulary edits are a training decision, not a configuration one.

S107 · Attention Intuition

1. Describe the bottleneck in the earlier encoder-decoder design that attention was created to fix.
2. Explain attention as a soft dictionary lookup in one paragraph.
3. Say why attention parallelises across the sequence axis where recurrence cannot.

Gotcha you probably forgot: without positional information a transformer is a bag-of-words model — attention is permutation-invariant, so shuffling the input produces a correspondingly shuffled output and nothing else changes. Position is not a refinement bolted on for accuracy; without it the architecture cannot represent order at all.

S108 · Q/K/V Math

1. Write the full formula and name every shape in it.
2. Derive why the scaling constant is the square root of the key dimension rather than the dimension itself.
3. Say what breaks if the normalisation is taken over the wrong axis.

Gotcha you probably forgot: the query and key projections only ever meet inside a bilinear form, so what the model effectively learns is their product as a single matrix — the separation into two matrices is a structural and computational convenience rather than a semantic distinction. The value projection is genuinely separate, because it alone determines what actually gets carried forward.

S109 · Multi-Head Attention

1. Give the head dimension for a stated model dimension and head count, and the shapes after reshaping.
2. Say what the output projection does and what breaks without it.
3. Distinguish the standard, grouped, and single-key-value variants and when each is used.

Gotcha you probably forgot: attention weights that are near one-hot for every query usually indicate a masking or scaling bug rather than a confident model — most often the scaling constant was omitted, so the scores have a variance that grows with dimension and the normalisation saturates. No error is raised; the output is simply nonsense.

S110 · Positional Encoding

1. Prove permutation invariance with a small example.
2. Explain the geometric idea behind the rotary scheme in one paragraph.
3. Say why the rotation is applied to queries and keys but not to values.

Gotcha you probably forgot: the fatal weakness of learned absolute position embeddings is that there is simply no parameter for a position beyond the trained maximum — the model cannot be evaluated there at all, rather than degrading gracefully. Schemes based on a formula are at least defined everywhere, which is a different failure mode and a strictly better starting point for extension.

Hands-on drill · 30 min

Task: run the merge algorithm by hand, measure token ratios across content types, verify the variance argument, and compute a cache budget.

mkdir -p ~/projects/w22-drill && cd ~/projects/w22-drill

Step 1 — the merge algorithm and the token ratio (8 min)

# bpe.py
from collections import Counter
 
corpus = "low low low low low lower lower newest newest newest widest widest"
words = Counter(corpus.split())
vocab = {" ".join(w) + " </w>": c for w, c in words.items()}
 
def pair_counts(vocab):
    pairs = Counter()
    for word, freq in vocab.items():
        syms = word.split()
        for i in range(len(syms) - 1):
            pairs[(syms[i], syms[i + 1])] += freq
    return pairs
 
for step in range(6):
    pairs = pair_counts(vocab)
    if not pairs:
        break
    best, count = pairs.most_common(1)[0]
    merged = "".join(best)
    vocab = {w.replace(" ".join(best), merged): c for w, c in vocab.items()}
    print(f"merge {step+1}: {best[0]!r} + {best[1]!r} -> {merged!r}   (seen {count} times)")
 
print("\nfinal segmentations:")
for w, c in vocab.items():
    print(f"  {w}")

Expected outcome: the algorithm greedily merges the most frequent adjacent pair, and after a handful of steps common stems appear as single units while rare words remain split into pieces. That is exactly the property that makes subword units work: frequent things are cheap, rare things are still representable, and nothing is ever out-of-vocabulary. Note that the merge order is fixed at training time and must be applied identically at inference — a mismatched merge table produces different identifiers for the same text, which is the vocabulary-mismatch bug.

Step 2 — tokens per word is a property of your data (7 min)

# ratio.py
import re
 
# A crude byte-pair-like proxy: count word pieces after splitting on
# case changes, digits, and punctuation the way real tokenizers tend to.
def approx_tokens(text):
    pieces = re.findall(r"[a-z]+|[A-Z][a-z]*|\d|[^\sa-zA-Z\d]", text)
    out = 0
    for p in pieces:
        out += max(1, (len(p) + 3) // 4) if p.isalpha() else 1
    return out
 
samples = {
    "english prose": "The quick brown fox jumps over the lazy dog near the riverbank at dawn.",
    "json payload":  '{"userId": "a3f9-2c1b", "eventType": "checkout", "amountCents": 12995}',
    "source code":   "def get_user_by_id(user_id: int) -> Optional[User]: return db.query(User).get(user_id)",
    "identifiers":   "550e8400-e29b-41d4-a716-446655440000 7c9e6679-7425-40de-944b-e07fc1f90ae7",
}
 
print(f"{'content type':<16} {'words':>6} {'tokens':>7} {'tokens/word':>12}")
for label, text in samples.items():
    w = len(text.split()); t = approx_tokens(text)
    print(f"{label:<16} {w:>6} {t:>7} {t/w:>12.2f}")
 
CONTEXT = 8192
print(f"\nwords that fit in a {CONTEXT}-token window:")
for label, text in samples.items():
    ratio = approx_tokens(text) / len(text.split())
    print(f"  {label:<16} about {int(CONTEXT / ratio):,} words")

Expected outcome: the ratio varies by several times across content types, with identifiers and structured payloads consuming far more tokens per word than prose. The practical consequence is the one to carry: a context budget or a cost estimate computed from word counts on prose will be badly wrong for logs, code, or non-Latin scripts, and the error is in the expensive direction. Always measure the ratio on your actual data with the actual tokeniser rather than assuming a rule of thumb.

Step 3 — why the scaling constant is a square root (8 min)

# scaling.py
import numpy as np
 
rng = np.random.default_rng(0)
 
def softmax(x):
    e = np.exp(x - x.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)
 
print(f"{'d_k':>5} {'var(q·k)':>10} {'var/d_k':>9}  {'max weight, unscaled':>21}  {'max weight, /sqrt(d)':>21}")
for d in (8, 64, 512, 4096):
    q = rng.normal(size=(4000, d)); k = rng.normal(size=(4000, d))
    dots = (q * k).sum(-1)
    scores_u = dots.reshape(200, 20)
    scores_s = scores_u / np.sqrt(d)
    print(f"{d:>5} {dots.var():>10.1f} {dots.var()/d:>9.2f}  "
          f"{softmax(scores_u).max(-1).mean():>21.4f}  {softmax(scores_s).max(-1).mean():>21.4f}")
 
print("\nWith unit-variance components, the dot product of two d-dimensional vectors")
print("has variance proportional to d, so its standard deviation grows like sqrt(d).")
print("Dividing by sqrt(d) restores unit variance -- which is why the constant is the")
print("square root and not d itself, which would over-correct and flatten the distribution.")

Expected outcome: the variance column tracks the dimension almost exactly, and the ratio in the next column stays near one, which is the derivation made visible. Then look at the two rightmost columns: unscaled, the maximum weight approaches one as dimension grows — the normalisation saturates into a near-hard selection, and a saturated softmax has vanishing gradients, so those heads stop learning. Scaled by the square root, the distribution stays usable at every dimension. Dividing by the dimension itself would shrink the scores too far and flatten everything towards uniform, which is the other failure.

Step 4 — head budgets and cache size (7 min)

# heads.py
d_model = 512
 
print(f"{'heads':>6} {'d_head':>7} {'qkv+out params':>16} {'note':>34}")
for h in (1, 2, 8, 32, 128):
    d_head = d_model // h
    params = 4 * d_model * d_model            # W_q, W_k, W_v, W_o are all d_model x d_model
    note = "d_head too small to be expressive" if d_head < 16 else ""
    print(f"{h:>6} {d_head:>7} {params:>16,} {note:>34}")
print("\nParameter count is identical for every row. Head count re-partitions a fixed")
print("budget into more, narrower subspaces -- it does not add capacity.\n")
 
def kv_cache_gb(layers, kv_heads, d_head, seq, batch=1, bytes_per=2):
    return 2 * layers * kv_heads * d_head * seq * batch * bytes_per / 1e9
 
layers, heads, d_head = 80, 64, 128
for seq in (2048, 32768):
    full = kv_cache_gb(layers, heads, d_head, seq)
    grouped = kv_cache_gb(layers, 8, d_head, seq)
    print(f"seq={seq:>6}  all heads keep K/V: {full:7.2f} GB   "
          f"8 shared K/V groups: {grouped:6.2f} GB   ratio {full/grouped:.0f}x")

Expected outcome: the parameter total is constant across head counts, which settles the "more heads means more capacity" question arithmetically — you are choosing how to partition a fixed budget, and beyond some point each subspace is too narrow to represent anything useful. The cache table then shows why the grouped variants exist: at long context the stored keys and values dominate memory and scale linearly with sequence length, so sharing them across query heads is the difference between a servable model and one that will not fit. Both numbers are worth being able to produce on paper.

Gap analysis + next week preview · 10 min

• Did the token-ratio spread in Step 2 change any cost estimate you carry in your head? Measure it on your real data before quoting a number.
• Could you reproduce the variance argument on paper? Quoting the constant is common; deriving it is the differentiator.
• Did the cache table make the grouped variants feel inevitable? That number is why the architecture changed.

Next week (S111–S115) assembles these parts into the full architecture and the training regime: the transformer block end to end, the pretraining objectives, scaling behaviour, and the fine-tuning and alignment stages that turn a pretrained model into a usable one. Everything from this week — tokens, routing, shapes, heads, position — becomes a component rather than a topic.

Part of the 6-month evergreen learning plan.