R23 · Week 23 Recall & Drill

Weekly revision · Week 23 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S111 — Full Transformer Architecture — Encoder + Decoder
• S112 — Encoder (BERT), Decoder (GPT), Enc-Dec (T5) — When Each
• S113 — LLM Sampling — Greedy, Beam, Top-k, Top-p, Temperature
• S114 — Scaling Laws — Chinchilla, Compute-Optimal Training
• S115 — Efficient Attention — Flash, Sparse, Linear

90-min structure

Blank-page reconstruction · 30 min

S111 · Full Transformer

1. List the block's components in order.
2. Say what the position-wise feed-forward network contributes that attention cannot.
3. Explain why normalisation placement relative to the residual connection matters.

Gotcha you probably forgot: tying the input embedding and the output projection weights saves a large fraction of parameters in models with big vocabularies, since both matrices have vocabulary-by-dimension shape. It is one of the highest-leverage architectural choices available and is invisible in a diagram, which is why people rebuild models without it and wonder where the parameters went.

S112 · Model Families

1. Describe the three attention masks in one sentence each.
2. Explain why a bidirectional encoder cannot generate fluent text.
3. Explain why a causal decoder is usually a weaker embedding model at equal size.

Gotcha you probably forgot: the reason encoders remain competitive for retrieval is structural rather than historical. Every token conditions on the full sequence in both directions, so a modest encoder produces better representations per unit of compute than a much larger causal model whose tokens can only see leftward. "Use the biggest model for everything" is a cost mistake, not just an inefficiency.

S113 · Sampling

1. Say what each decoding knob does at the level of the probability distribution.
2. Explain why greedy decoding tends to produce repetition loops.
3. Give a decoding configuration for structured output generation and one for creative variety.

Gotcha you probably forgot: setting both the temperature and a truncation parameter to non-default values makes the interaction hard to reason about, because one reshapes the distribution while the other cuts it, and the effective result depends on the order. Change one at a time, and prefer whichever the serving stack applies first.

S114 · Scaling Laws

1. State the compute relationship between parameters and tokens.
2. State the tokens-per-parameter heuristic and what changed to produce it.
3. Explain why a model may be deliberately trained past that point.

Gotcha you probably forgot: the compute-optimal point minimises training loss for a fixed training budget and says nothing about inference cost. If a model will serve an enormous number of requests, the total lifetime cost is dominated by inference, so a smaller model trained on far more tokens than the heuristic suggests is the rational choice — worse loss per unit of training compute, far better economics overall.

S115 · Efficient Attention

1. Say why vanilla attention is quadratic in both time and memory, and where the true bottleneck sits.
2. Describe the tiling and recomputation trick in one paragraph.
3. Distinguish the sparse family from the linear family and give a workload suited to each.

Gotcha you probably forgot: the bottleneck in attention is not arithmetic, it is traffic between high-bandwidth memory and on-chip memory. The score matrix is written out and read back, and that movement dominates the cost — which is why an implementation performing the same arithmetic in a different order is dramatically faster with no change in the result.

Hands-on drill · 30 min

Task: build the three masks, watch decoding knobs reshape a distribution, size a compute budget, and count memory traffic.

mkdir -p ~/projects/w23-drill && cd ~/projects/w23-drill

Step 1 — the mask is the family (7 min)

# masks.py
import numpy as np
 
T = 6
full = np.ones((T, T), dtype=int)
causal = np.tril(np.ones((T, T), dtype=int))
# Encoder-decoder: encoder is bidirectional, decoder is causal over its own tokens
# and fully attends to the encoder output.
def show(name, m):
    print(f"\n{name}")
    for row in m:
        print("  " + " ".join("#" if v else "." for v in row))
 
show("bidirectional (encoder): every token sees every token", full)
show("causal (decoder): token t sees only positions <= t", causal)
 
print("\nvisible pairs: bidirectional =", full.sum(), " causal =", causal.sum())
print("The causal mask is the entire reason a decoder can be trained to predict")
print("the next token on every position at once without leaking the answer,")
print("and the entire reason its representations are weaker for retrieval.")
 
# Prove permutation behaviour differs.
x = np.arange(T)
print("\nwith a causal mask, position 0 can never be influenced by later tokens:")
print("  attend(0) sees", list(x[causal[0] == 1]))
print("  attend(5) sees", list(x[causal[5] == 1]))

Expected outcome: the two grids make the structural claim concrete. The causal pattern is what permits training on every position simultaneously without a token seeing its own answer, and it is simultaneously why early positions have strictly less context available than late ones. The bidirectional pattern gives every position full context, which is exactly why an encoder of modest size produces stronger representations for retrieval than a much larger causal model — the difference is the mask, not the parameter count.

Step 2 — decoding knobs on a real distribution (8 min)

# decoding.py
import numpy as np
 
rng = np.random.default_rng(0)
V = 20
logits = np.sort(rng.normal(scale=2.0, size=V))[::-1]
 
def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()
 
def apply(logits, temperature=1.0, top_k=None, top_p=None):
    z = logits / max(temperature, 1e-6)
    p = softmax(z)
    if top_k:
        cut = np.sort(p)[::-1][top_k - 1]
        p = np.where(p >= cut, p, 0.0)
    if top_p:
        order = np.argsort(-p)
        cum = np.cumsum(p[order])
        keep = order[:int(np.searchsorted(cum, top_p) + 1)]
        mask = np.zeros_like(p); mask[keep] = 1.0
        p = p * mask
    return p / p.sum()
 
def describe(label, p):
    nz = int((p > 1e-9).sum())
    ent = float(-np.sum(p[p > 0] * np.log(p[p > 0])))
    print(f"{label:<28} candidates={nz:>3}  top prob={p.max():.3f}  entropy={ent:.3f}")
 
describe("baseline (T=1.0)", apply(logits))
describe("T=0.2 (sharpened)", apply(logits, temperature=0.2))
describe("T=1.5 (flattened)", apply(logits, temperature=1.5))
describe("top_k=5", apply(logits, top_k=5))
describe("top_p=0.9", apply(logits, top_p=0.9))
describe("T=0.7 + top_p=0.9", apply(logits, temperature=0.7, top_p=0.9))
 
print("\nTemperature reshapes the whole distribution; truncation deletes the tail.")
print("Rank truncation keeps a fixed count regardless of confidence; mass truncation")
print("keeps few candidates when the model is confident and many when it is not,")
print("which is why the mass-based rule adapts better across positions.")

Expected outcome: lowering the temperature concentrates mass on the top candidate and reduces entropy; raising it does the opposite. Rank-based truncation always leaves the same number of candidates whether the model is certain or not, while mass-based truncation adapts — few candidates at confident positions, many at uncertain ones. That adaptivity is the argument for the mass-based rule as a default. The combined row shows why changing both at once is hard to reason about: the effective candidate set depends on which operation the serving stack applies first.

Step 3 — a compute budget (7 min)

# scaling.py
# Standard approximation: training compute is about 6 * parameters * tokens.
def compute(N, D):
    return 6 * N * D
 
def chinchilla_optimal(C, ratio=20):
    """With D = ratio * N and C = 6*N*D, solve for N."""
    N = (C / (6 * ratio)) ** 0.5
    return N, ratio * N
 
print(f"{'budget (FLOPs)':>16} {'optimal params':>16} {'optimal tokens':>16}")
for C in (1e21, 1e23, 1e24, 1e25):
    N, D = chinchilla_optimal(C)
    print(f"{C:>16.0e} {N/1e9:>14.1f}B {D/1e12:>14.1f}T")
 
print(f"\n{'model':<22} {'params':>9} {'tokens':>9} {'tok/param':>10}  verdict")
for name, N, D in [("early large model", 175e9, 300e9),
                   ("compute-optimal style", 70e9, 1.4e12),
                   ("deliberately over-trained", 8e9, 15e12)]:
    r = D / N
    verdict = ("under-trained" if r < 20 else
               "near the heuristic" if r < 50 else
               "over-trained on purpose (inference economics)")
    print(f"{name:<22} {N/1e9:>8.0f}B {D/1e12:>8.2f}T {r:>10.0f}  {verdict}")

Expected outcome: the optimal size grows only as the square root of the budget, which is why an order of magnitude more compute does not mean an order of magnitude larger model. The verdict table is the reading skill to keep: given a parameter count and a token count you can immediately classify a release as under-trained, near the heuristic, or deliberately over-trained. The last case is not a mistake — a heavily over-trained small model has worse loss per unit of training compute and far better economics when serving many requests, since lifetime cost is dominated by inference.

Step 4 — the bottleneck is traffic, not arithmetic (8 min)

# traffic.py
def naive(T, d, bytes_per=2):
    """Materialise the full score matrix in main memory: write it, read it back."""
    qkv = 3 * T * d * bytes_per
    scores = T * T * bytes_per
    return qkv + 2 * scores + T * d * bytes_per      # write scores, read scores
 
def tiled(T, d, block=128, bytes_per=2):
    """Never materialise the score matrix: stream tiles through on-chip memory."""
    return 3 * T * d * bytes_per + T * d * bytes_per
 
d = 128
print(f"{'seq len':>8} {'naive bytes moved':>20} {'tiled bytes moved':>20} {'ratio':>8}"
      f" {'score matrix':>14}")
for T in (512, 2048, 8192, 32768):
    n, t = naive(T, d), tiled(T, d)
    print(f"{T:>8} {n/1e6:>18.1f}MB {t/1e6:>18.1f}MB {n/t:>8.1f}x"
          f" {T*T*2/1e9:>12.2f}GB")
 
print("\nArithmetic is identical in both columns. Only the movement differs.")
print("The score matrix alone exceeds device memory at long context, which is why")
print("the tiled formulation is not merely faster but is what makes long context possible.")

Expected outcome: the traffic ratio grows with sequence length, and the final column shows the score matrix alone reaching sizes that will not fit in device memory at long context. Both columns perform the same arithmetic — the tiled version simply never writes the intermediate matrix out, keeping tiles in fast on-chip memory and recomputing what it needs. That is why the speedup comes with no accuracy cost, and why the naive implementation's limit is a memory limit rather than a compute limit.

Gap analysis + next week preview · 10 min

• Could you draw the three masks from memory? That single diagram answers most family-choice questions.
• Did the traffic table change how you think about the quadratic cost? "Too much compute" and "too much memory movement" lead to different fixes.
• Can you classify a model release as under- or over-trained from two numbers? That is a thirty-second read worth having.

Next week (S116–S120) turns to applying these models: prompting techniques and their failure modes, retrieval-augmented generation end to end, fine-tuning approaches including the parameter-efficient family, evaluation of generative systems, and the engineering around serving them. The decoding, masking, and cost intuitions from this week are exactly what those decisions rest on.

Part of the 6-month evergreen learning plan.