R21 · Week 21 Recall & Drill

Weekly revision · Week 21 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S101 — Regularization in DL — Dropout, BatchNorm, Weight Decay
• S102 — CNNs — Convolution, Pooling, ImageNet Architectures
• S103 — RNNs & LSTMs — Sequences & the Vanishing Gradient
• S104 — Embeddings — word2vec, GloVe, Contrastive Learning
• S105 — Transfer Learning & Fine-Tuning Classical DL

90-min structure

Blank-page reconstruction · 30 min

S101 · DL Regularization

1. Explain dropout as ensemble training in one sentence, and say why it is inactive at inference.
2. Say which normalisation layer belongs in a convolutional network versus a sequence model, and why.
3. State the difference between folding a penalty into the gradient and decaying weights directly.

Gotcha you probably forgot: batch normalisation and dropout placed together often hurt rather than help, because dropout changes the variance of the activations between training and inference while the normalisation layer's running statistics were estimated under one of those regimes and applied under the other. The mismatch shows up as a validation score noticeably worse than training for reasons that look like overfitting and are not.

S102 · CNNs

1. Write the output-shape formula and apply it to a strided, padded convolution.
2. Name the two structural priors that make convolution suit images.
3. Say what a skip connection does mathematically and why it enabled much deeper networks.

Gotcha you probably forgot: a one-by-one convolution is not a no-op — it is a learned linear projection across the channel dimension applied identically at every spatial position, which is how architectures cheaply change channel count and mix channel information before an expensive spatial convolution. Recognising it explains most of the parameter savings in the landmark architectures.

S103 · RNNs & LSTMs

1. Explain weight sharing across time and why it matters.
2. Derive why gradients decay in a plain recurrent network, referencing the repeated Jacobian product.
3. Say why the sequence axis cannot be parallelised, and what the later architecture did instead.

Gotcha you probably forgot: teacher forcing during training means the model always sees ground-truth history, but at generation time it sees its own outputs — so a single early mistake shifts the input distribution to one the model never trained on, and errors compound. This exposure gap is why generated sequences can start well and degrade, and it is not fixed by more training on the same objective.

S104 · Embeddings

1. Explain why one-hot vectors cannot express similarity.
2. State the skip-gram objective and why negative sampling replaced the full normalisation.
3. Say why vectors are usually normalised before comparison.

Gotcha you probably forgot: distributional training places antonyms extremely close together, because opposites appear in nearly identical contexts — the objective rewarded co-occurrence, not meaning. If your application needs to distinguish a positive from a negative statement, an embedding trained this way will actively work against you, and no amount of dimensionality will fix an objective mismatch.

S105 · Transfer Learning

1. Explain why early layers transfer better than late ones.
2. Give the four-quadrant rule over dataset size and domain distance.
3. Say what discriminative learning rates are and why they work.

Gotcha you probably forgot: you must reproduce the pretraining preprocessing exactly — the same normalisation constants, the same resize interpolation, the same channel order. A frozen backbone fed inputs normalised differently produces activations outside the distribution its later layers expect, and the result looks exactly like "the pretrained weights are not suited to my domain", which sends people down a modelling rabbit hole for a one-line bug.

Hands-on drill · 30 min

Task: do the shape arithmetic by hand, watch recurrent gradients decay, catch a cosine trap, and quantify preprocessing drift.

mkdir -p ~/projects/w21-drill && cd ~/projects/w21-drill

Step 1 — shapes and parameters without a framework (7 min)

# convshapes.py
def out_dim(n, k, p, s):
    return (n + 2 * p - k) // s + 1
 
layers = [
    # (name, in_ch, out_ch, kernel, padding, stride)
    ("conv 3x3",       3,   64, 3, 1, 1),
    ("conv 3x3 stride2", 64, 128, 3, 1, 2),
    ("conv 1x1",       128,  32, 1, 0, 1),
    ("conv 5x5",        32,  64, 5, 2, 2),
]
 
n, total = 224, 0
print(f"input 3 x {n} x {n}")
for name, cin, cout, k, p, s in layers:
    n = out_dim(n, k, p, s)
    params = cin * cout * k * k + cout
    total += params
    print(f"{name:<18} -> {cout:>4} x {n:>3} x {n:<3}   params {params:>9,}")
 
fc_equivalent = 3 * 224 * 224 * 64
print(f"\ntotal conv params: {total:,}")
print(f"a fully-connected layer producing the same first output volume would need "
      f"about {fc_equivalent * 224 * 224 // 1_000_000_000:,} billion weights")
print("that ratio is weight sharing, and it is the reason convolution exists")

Expected outcome: you can predict every intermediate shape from the formula, and the parameter counts show where cost lives — the one-by-one layer is tiny yet changes the channel count, which is exactly why it appears everywhere as a cheap projection before an expensive spatial operation. The comparison against a dense layer is the structural argument: a dense connection learns a separate weight for every input position, so it must relearn the same pattern independently at each location and needs vastly more data and parameters to do it.

Step 2 — why recurrent gradients decay (8 min)

# decay.py
import numpy as np
 
rng = np.random.default_rng(0)
H = 20
 
def gradient_norm_over_time(spectral_radius, steps=60, gated=False, forget=0.95):
    W = rng.normal(size=(H, H))
    W *= spectral_radius / max(abs(np.linalg.eigvals(W)).max(), 1e-9)
    g = rng.normal(size=H); g /= np.linalg.norm(g)
    norms = []
    for t in range(steps):
        if gated:
            g = g * forget                      # gated path: multiply by a scalar gate
        else:
            g = W @ g
            g = g * (rng.random(H) > 0.3)       # activation derivative gating
        norms.append(np.linalg.norm(g))
    return norms
 
print("steps back:            1        10        30        60")
for sr, label in [(0.6, "plain RNN, radius 0.6"), (1.4, "plain RNN, radius 1.4")]:
    n = gradient_norm_over_time(sr)
    print(f"{label:<22} {n[0]:8.2e} {n[9]:9.2e} {n[29]:9.2e} {n[59]:9.2e}")
for f, label in [(0.95, "gated path, gate 0.95"), (0.99, "gated path, gate 0.99")]:
    n = gradient_norm_over_time(1.0, gated=True, forget=f)
    print(f"{label:<22} {n[0]:8.2e} {n[9]:9.2e} {n[29]:9.2e} {n[59]:9.2e}")

Expected outcome: with a contractive recurrent matrix the gradient norm collapses towards zero within tens of steps, so the loss carries essentially no information about distant inputs — that is vanishing, and it is a property of the repeated matrix product, not a bug. With an expansive one it grows without bound, which is why gradient clipping is standard. The gated rows are the important comparison: multiplying by a scalar just below one decays far more slowly, but it still decays geometrically. That is why the honest statement is that gates mitigate the problem and extend the usable range, rather than solving it.

Step 3 — the cosine trap (7 min)

# cosine.py
import numpy as np
 
# Hand-built vectors standing in for distributionally-trained embeddings.
# Words that appear in the same contexts end up close, regardless of meaning.
vecs = {
    "hot":      np.array([0.90, 0.10, 0.42]),
    "cold":     np.array([0.89, 0.12, 0.44]),      # antonym, near-identical contexts
    "warm":     np.array([0.85, 0.20, 0.48]),
    "database": np.array([0.10, 0.95, 0.30]),
}
 
def cos(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
 
print("pair                 cosine")
for a, b in [("hot", "cold"), ("hot", "warm"), ("hot", "database")]:
    print(f"{a:>9} / {b:<9} {cos(vecs[a], vecs[b]):.4f}")
print("\n'hot' and 'cold' score as high as 'hot' and 'warm'. The objective rewarded")
print("co-occurrence, and opposites co-occur in nearly identical contexts.")
 
# Why normalisation matters: magnitude is often frequency, not relevance.
a = np.array([3.0, 4.0]); b = np.array([30.0, 40.0]); c = np.array([4.0, 3.0])
print(f"\ncos(a,b) = {cos(a,b):.4f}  (same direction, 10x magnitude -> identical)")
print(f"dot(a,b) = {a@b:.1f}   vs dot(a,c) = {a@c:.1f}  <-- raw dot product is dominated by magnitude")
print(f"cos(a,c) = {cos(a,c):.4f}  (the actual angular difference)")

Expected outcome: the antonym pair scores as highly as the synonym pair, which is the trap — cosine similarity faithfully reports what the training objective encoded, and that objective was distributional co-occurrence rather than meaning. The second block shows why normalisation is standard: raw dot products are dominated by vector magnitude, which in most training regimes tracks frequency rather than relevance, so the most common items would win every comparison. Neither issue is a bug in the metric; both are reasons to check what your embedding was actually trained to reward before trusting it.

Step 4 — preprocessing drift (8 min)

# drift.py
import numpy as np
 
rng = np.random.default_rng(3)
# Stand-in for a frozen backbone: a fixed random projection plus a rectifier.
W = rng.normal(size=(64, 128)) / np.sqrt(64)
 
PRETRAIN_MEAN = np.array([0.485, 0.456, 0.406])
PRETRAIN_STD  = np.array([0.229, 0.224, 0.225])
 
raw = rng.random((2000, 3))                          # pixel values in [0, 1]
 
def features(x_norm):
    x = np.repeat(x_norm, 64 // 3 + 1, axis=1)[:, :64]
    return np.maximum(x @ W, 0)
 
correct = features((raw - PRETRAIN_MEAN) / PRETRAIN_STD)
wrong_none = features(raw)                            # forgot to normalise at all
wrong_half = features((raw - 0.5) / 0.5)              # used a different convention
 
for label, f in [("correct normalisation", correct),
                 ("no normalisation", wrong_none),
                 ("different convention", wrong_half)]:
    dead = float((f == 0).mean())
    print(f"{label:<24} mean activation {f.mean():7.3f}  std {f.std():6.3f}  "
          f"fraction exactly zero {dead:5.1%}")
 
print("\nThe backbone is identical in all three rows. Only the input convention changed.")
print("A large shift in the dead fraction means later layers receive a distribution")
print("they never saw in pretraining — which presents as 'the pretrained weights")
print("don't suit my domain' and is actually a one-line bug.")

Expected outcome: the three rows differ substantially in activation statistics and in how many units are driven to exactly zero, despite the weights being byte-identical. That is the mechanism behind the most common transfer-learning failure: the frozen layers were tuned for inputs in one distribution and are being fed another, so the features they emit are off-manifold and no amount of head training recovers what was lost upstream. The check to institutionalise is comparing activation statistics against the pretraining recipe before concluding anything about representation quality.

Gap analysis + next week preview · 10 min

• Could you do the shape arithmetic without running the script? That skill turns a class of runtime errors into paper work.
• Did the gated-path numbers in Step 2 match your mental model, or had you filed gates under "solved"?
• Did the drift in Step 4 look large enough to explain a failed fine-tune? It usually is, and it is checked far too rarely.

Next week (S106–S110) moves into the attention era: the attention mechanism itself, the transformer architecture end to end, positional encoding, and the pretraining objectives that produced modern language models. The embedding intuitions and the sequence-modelling limitations from this week are precisely the problems that architecture was built to solve, so this week's material is the setup for that one.

Part of the 6-month evergreen learning plan.