R20 · Week 20 Recall & Drill

Weekly revision · Week 20 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S096 — Perceptron & Activation Functions
• S097 — Multi-Layer Perceptron — Forward Pass
• S098 — Backpropagation — Derived by Hand on a 2-Layer Net
• S099 — Optimizers — SGD, Momentum, Adam, RMSprop
• S100 — PyTorch Fundamentals — Tensors, Autograd, nn.Module

90-min structure

Blank-page reconstruction · 30 min

S096 · Perceptron & Activations

1. Write the neuron equation and its update rule.
2. State the vanishing gradient problem in one sentence.
3. Give one situation where the saturating sigmoid is still the correct choice.

Gotcha you probably forgot: initialising every weight to zero breaks learning entirely, and not because of a scale problem. All units in a layer then compute the same output, receive the same gradient, and update identically — so they remain identical forever and the layer has the expressive power of a single unit. Symmetry must be broken by the initialisation itself, which is why random initialisation is structural rather than cosmetic.

S097 · MLP Forward Pass

1. Write the shape of every intermediate value through a two-hidden-layer network, including the batch dimension.
2. State the approximation theorem and its practical limitation.
3. Say why the bias vector is needed in each layer.

Gotcha you probably forgot: the feed-forward block inside a transformer is exactly this network — expand the dimension, apply a non-linearity, project back down. Recognising it there is what makes the later architecture sessions feel like assembly rather than new material, and it is also where a large share of a transformer's parameters actually live.

S098 · Backpropagation

1. Write the output-layer gradient for the softmax and cross-entropy combination.
2. Given a layer's pre-activation expression, derive the weight gradient in terms of the incoming activation.
3. Say what shape a weight gradient must have, and why that check catches most bugs.

Gotcha you probably forgot: backpropagation updates nothing. It is reverse-mode automatic differentiation computing gradients on a graph, and that is its entire job. The optimiser is the learning algorithm. Confusing the two makes it impossible to reason about why the same gradients produce different training behaviour under different optimisers — which is the actual content of the next session.

S099 · Optimizers

1. Write the four update rules from memory.
2. Say what dividing by a running gradient-magnitude estimate accomplishes.
3. Explain why bias correction is needed in the adaptive method.

Gotcha you probably forgot: decoupling the weight penalty from the adaptive scaling matters because a penalty folded into the gradient gets divided by the same running magnitude estimate as everything else — so the effective amount of regularisation varies per parameter in a way nobody intended. The decoupled variant applies the decay directly to the weights, which is why it is the default in modern training recipes.

S100 · PyTorch Fundamentals

1. Say what marking a tensor as requiring gradients actually does.
2. Give the five-line training step in order.
3. Explain the difference between training and evaluation mode.

Gotcha you probably forgot: gradients accumulate by default rather than being replaced, so forgetting to zero them means each step uses the sum of every gradient computed since the last reset — training appears to work, loss behaves strangely, and nothing raises an error. It is a deliberate design choice that enables accumulating over several micro-batches, which is why it will never be changed.

Hands-on drill · 30 min

Task: prove that depth without non-linearity is nothing, check a hand-derived gradient numerically, and watch the optimisers behave differently on the same surface.

mkdir -p ~/projects/w20-drill && cd ~/projects/w20-drill

Step 1 — linear layers collapse (7 min)

# collapse.py
import numpy as np
 
rng = np.random.default_rng(0)
W1, W2, W3 = rng.normal(size=(4, 8)), rng.normal(size=(8, 8)), rng.normal(size=(8, 3))
x = rng.normal(size=(5, 4))
 
deep_linear = x @ W1 @ W2 @ W3
single = x @ (W1 @ W2 @ W3)
print("three linear layers vs one equivalent matrix — max difference:",
      f"{np.abs(deep_linear - single).max():.2e}   (i.e. identical)")
 
relu = lambda z: np.maximum(z, 0)
with_nonlin = relu(relu(x @ W1) @ W2) @ W3
print("with a non-linearity between layers — max difference from the single matrix:",
      f"{np.abs(with_nonlin - single).max():.3f}   (i.e. a genuinely different function)")
 
# XOR: the historical example.
X = np.array([[0,0],[0,1],[1,0],[1,1]], float)
y = np.array([0,1,1,0], float)
A = np.c_[np.ones(4), X]
theta, *_ = np.linalg.lstsq(A, y, rcond=None)
print("\nbest linear fit to XOR:", np.round(A @ theta, 3), " target:", y)
print("no line separates these four points; width cannot fix that, only a non-linearity can.")

Expected outcome: the three-layer linear stack is bit-for-bit the same function as a single matrix, so depth bought exactly nothing — the composition of linear maps is a linear map, and adding units to each layer changes the matrix's size but not that fact. Inserting a non-linearity produces a function the single matrix cannot express. The XOR fit lands at one half for every input, which is the best a linear model can do on a problem that needs two regions, and it is precisely the result that stalled the field for years.

Step 2 — shapes and parameter counts (7 min)

# shapes.py
import numpy as np
 
def trace(dims, batch=32):
    total = 0
    print(f"input                         shape ({batch}, {dims[0]})")
    for i in range(len(dims) - 1):
        d_in, d_out = dims[i], dims[i + 1]
        p = d_in * d_out + d_out
        total += p
        print(f"layer {i+1}: W ({d_in:>4}, {d_out:>4})  b ({d_out:>4},)"
              f"   params {p:>9,}   activation shape ({batch}, {d_out})")
    print(f"total parameters: {total:,}\n")
    return total
 
trace([784, 128, 64, 10])
trace([512, 2048, 512])      # the shape of a transformer feed-forward block

Expected outcome: you can read off every intermediate shape and see where the parameters actually live — overwhelmingly in the widest weight matrices, not in the biases. The second call is the transformer feed-forward block: expand, non-linearity, project back. Recognising that this is just the network from earlier in the week is what makes the later architecture material tractable. If you cannot produce these shapes on paper, no amount of framework fluency will save you when a shape error appears mid-training.

Step 3 — gradient check your derivation (8 min)

# gradcheck.py
import numpy as np
 
rng = np.random.default_rng(5)
B, D, H, K = 8, 6, 5, 3
X = rng.normal(size=(B, D))
y = rng.integers(0, K, B)
Y = np.eye(K)[y]
 
params = {"W1": rng.normal(size=(D, H)) * 0.5, "b1": np.zeros(H),
          "W2": rng.normal(size=(H, K)) * 0.5, "b2": np.zeros(K)}
 
def forward(p):
    z1 = X @ p["W1"] + p["b1"]
    a1 = np.maximum(z1, 0)
    z2 = a1 @ p["W2"] + p["b2"]
    z2 = z2 - z2.max(1, keepdims=True)
    probs = np.exp(z2) / np.exp(z2).sum(1, keepdims=True)
    loss = -np.sum(Y * np.log(probs + 1e-12)) / B
    return loss, (z1, a1, probs)
 
def backward(p, cache):
    z1, a1, probs = cache
    dz2 = (probs - Y) / B                       # the clean gradient
    g = {"W2": a1.T @ dz2, "b2": dz2.sum(0)}
    da1 = dz2 @ p["W2"].T
    dz1 = da1 * (z1 > 0)                        # ReLU gate
    g["W1"] = X.T @ dz1
    g["b1"] = dz1.sum(0)
    return g
 
loss, cache = forward(params)
grads = backward(params, cache)
 
eps = 1e-6
print("param  analytic vs numeric — max relative error")
for name in params:
    P = params[name]
    num = np.zeros_like(P)
    it = np.nditer(P, flags=["multi_index"])
    while not it.finished:
        i = it.multi_index
        old = P[i]
        P[i] = old + eps; lp, _ = forward(params)
        P[i] = old - eps; lm, _ = forward(params)
        P[i] = old
        num[i] = (lp - lm) / (2 * eps)
        it.iternext()
    rel = np.abs(num - grads[name]).max() / max(np.abs(num).max(), 1e-12)
    print(f"{name:>4}   {rel:.2e}   {'OK' if rel < 1e-5 else 'WRONG'}")

Expected outcome: every relative error is tiny, confirming the hand derivation. This is the technique to keep: whenever you write a backward pass by hand, gradient-check it before training anything, because a subtly wrong gradient produces training that appears to work while converging to something worse, and you will spend days blaming the data. Note also that the output gradient is simply predicted minus target — the softmax and cross-entropy derivatives cancel, which is why they are always paired and why frameworks fuse them into a single operation for numerical stability.

Step 4 — the optimisers are not interchangeable (8 min)

# optim.py
import numpy as np
 
# A badly-conditioned quadratic bowl: steep in one direction, shallow in the other.
A = np.array([20.0, 1.0])
loss = lambda w: 0.5 * np.sum(A * w**2)
grad = lambda w: A * w
start = np.array([1.0, 1.0])
 
def run(name, lr, steps=200, **kw):
    w = start.copy(); v = np.zeros(2); s = np.zeros(2); best = loss(w)
    for t in range(1, steps + 1):
        g = grad(w)
        if name == "sgd":
            w -= lr * g
        elif name == "momentum":
            v = 0.9 * v + g; w -= lr * v
        elif name == "rmsprop":
            s = 0.9 * s + 0.1 * g**2; w -= lr * g / (np.sqrt(s) + 1e-8)
        elif name == "adam":
            v = 0.9 * v + 0.1 * g; s = 0.999 * s + 0.001 * g**2
            vh, sh = v / (1 - 0.9**t), s / (1 - 0.999**t)
            w -= lr * vh / (np.sqrt(sh) + 1e-8)
        if not np.isfinite(w).all():
            return "diverged", t
        best = min(best, loss(w))
    return f"{loss(w):.2e}", steps
 
print("optimiser   lr=0.005        lr=0.05         lr=0.2")
for name in ("sgd", "momentum", "rmsprop", "adam"):
    row = "  ".join(f"{run(name, lr)[0]:>13}" for lr in (0.005, 0.05, 0.2))
    print(f"{name:<10}  {row}")
print("\nNote which cells diverge. The adaptive methods tolerate a wider band of base step sizes,")
print("but every one of them still has a band — none of them removes the need to choose it.")

Expected outcome: on this ill-conditioned surface plain gradient descent is limited by the steep direction and crawls along the shallow one; momentum accumulates in the consistent direction and gets much further; the adaptive methods rescale per coordinate and handle the conditioning directly. Then read across the columns: each method diverges at some step size. That is the point of the drill — an adaptive optimiser widens the range of workable base learning rates, it does not eliminate the hyperparameter, and treating the default as universally correct is how training runs quietly underperform.

Gap analysis + next week preview · 10 min

• Did the collapse in Step 1 register as obvious or as surprising? If obvious, you can explain why non-linearity is structural rather than a tweak.
• Could you write the backward pass without looking? The gradient check is worth keeping as a habit regardless.
• Did the divergence pattern in Step 4 change how you think about optimiser defaults? "Adam handles it" is the belief that costs the most quietly.

Next week (S101–S105) builds on this foundation into practical deep learning: regularisation for networks including dropout and normalisation layers, convolutional architectures, sequence models, transfer learning, and the training loop concerns that only appear at scale. Every one of them assumes you can write a forward pass, reason about shapes, and separate gradient computation from the update rule.

Part of the 6-month evergreen learning plan.