R21 · Week 21 Recall & Drill

Assessment scope

The common thread is structural assumptions: dropout changes a training computation, convolution shares local weights, recurrence shares state transitions, embeddings encode an objective, and transfer reuses a representation with an input contract. None supplies a universal performance guarantee. The CPU examples expose shapes and counterexamples without training a vision model or an LSTM.

Sessions covered

• S101 — Regularization in DL — Dropout, BatchNorm, Weight Decay
• S102 — CNNs — Convolution, Pooling, ImageNet Architectures
• S103 — RNNs & LSTMs — Sequences & the Vanishing Gradient
• S104 — Embeddings — word2vec, GloVe, Contrastive Learning
• S105 — Transfer Learning & Fine-Tuning Classical DL

90-min structure

This is suggested pacing. A real fine-tuning comparison requires its own data, compute and evaluation budget.

Blank-page reconstruction · 30 min

S101 · DL Regularization

1. Explain dropout at training and evaluation.
2. Compare BatchNorm and LayerNorm, including batch-size dependence.
3. Contrast gradient penalties with direct weight decay and their interactions.

Solutions. In inverted dropout, a Bernoulli keep mask m with probability 0 < q <= 1, independent of the activation, multiplies activation a as m*a/q in training; its conditional expectation is a. Evaluation is identity. This does not make the full nonlinear network exactly the mean of every thinned network; the ensemble interpretation is approximate. Excessive dropout can impair useful feature combinations or optimization and cause underfitting. Tune it jointly with other controls rather than stacking regularizers by habit.

PyTorch BatchNorm2d computes per-channel statistics across batch and spatial axes (N,H,W) in training and, by default, uses running estimates in evaluation. Tiny/nonrepresentative batches can make those estimates troublesome. LayerNorm computes statistics within a sample over specified trailing dimensions and uses the same basic computation in train/eval, useful for token features and recurrence without batch-statistic dependence. BatchNorm is common in convolutional models and LayerNorm in transformers, but GroupNorm, RMSNorm and other choices exist: neither “belongs” exclusively to one architecture.

Dropout before a BatchNorm can change the variance distribution on which running statistics are estimated; removing dropout in eval then creates a mismatch. Placement, training protocol and model determine severity, so the pair is not universally harmful. For a scalar random X with mean mu and variance v and an independent keep mask, inverted dropout Y has E[Y]=mu and Var(Y)=v/q+(1/q-1)*mu². For X equally likely to be 1 or 3 and q=1/2, Y takes 0, 2 and 6 with probabilities 1/2, 1/4 and 1/4: its mean is 2 and variance is 6, versus input variance 1. The nonzero-mean term contributes 4 of that output variance. At q=1 there is no masking or inflation; q=0 is outside this division-based formula (an all-dropped implementation needs a separate branch). Even nonzero means contribute to the mismatch. Separately, L2 adds a term to the gradient; AdamW decays weights separately from adaptive scaling. Freezing parameters does not necessarily freeze BatchNorm buffers or switch dropout off—mode must be managed explicitly.

S102 · CNNs

1. Compute convolution/pooling output dimensions, including dilation.
2. Explain locality, sharing and the 1x1 convolution.
3. Write a residual block and distinguish architectural examples.

Solutions. For integer input width n, kernel k, symmetric padding p, stride s and dilation d, the effective kernel is d*(k-1)+1; floor-mode output width is

Apply independently to height and width. This assumes valid positive parameters and at least one valid window. Pooling often uses the same window arithmetic, but ceil-mode, asymmetric padding and adaptive pooling have different rules. A group-g convolution with bias has Cout*(Cin/g)*kh*kw + Cout parameters; groups must divide both channel counts. Pooling normally has no learned weights. These formulas count parameters, not total FLOPs or activation memory.

Locality restricts each output to a neighborhood; weight sharing reuses the same filter at every location. This favors translation-related structure, with stride/padding/boundaries qualifying exact equivariance. Edge-like filters can emerge from training; “it detects edges” is not the definition. A 1x1 convolution is a learned channel projection per spatial position, not a spatial no-op or necessarily an identity. It can reduce channels before a more expensive spatial operation, as in bottleneck/inception-style designs; VGG-style stacks emphasize repeated small kernels, and ResNet adds residual branches.

A residual block computes y=x+F(x) when shapes agree, or y=P(x)+F(x) using a projection when they differ. The derivative includes an identity/projection path in addition to the branch Jacobian. This helps optimization of deep networks in the cited ResNet experiments; it does not guarantee arbitrary depth or fix every gradient problem.

S103 · RNNs & LSTMs

1. Explain time-shared recurrence and its sequential dependence.
2. Derive the repeated-Jacobian gradient argument.
3. Explain the LSTM cell path and the teacher-forcing gap.

Solutions. A plain recurrence is h_t=phi(W_h h_(t-1)+W_x x_t+b) with the same parameters at every step. Sharing makes parameter count independent of sequence length and supplies a repeated transition rule. Within this ordinary nonlinear recurrence, h_t depends on h_(t-1), so time steps cannot all be independently evaluated; batches can still parallelize. Self-attention mixes all known positions by matrix operations rather than that recurrence, though autoregressive generation remains sequential and specialized recurrence algorithms can change implementation tradeoffs.

The gradient from a late state to an earlier one multiplies time-varying Jacobians J_t=diag(phi'(z_t))*W_h; W_h is shared but phi' depends on state/input. Norm bounds multiply singular-norm factors. Products can shrink or grow; the spectral radius of W alone does not determine a nonlinear nonnormal product's finite-time behavior. Gradient clipping limits excessive norms, not recovery of vanished information.

An LSTM has input/forget/output gates and a candidate: c_t=f_t*c_(t-1)+i_t*g_t, h_t=o_t*tanh(c_t). Holding gate values fixed, the direct cell-state path multiplies forget gates instead of a full recurrent Jacobian. Gates near one permit long retention; a constant gate 0.95 yields 0.95**60, still decaying. Real gates depend on prior states, so total derivatives include other paths; this scalar example is not the complete LSTM gradient or a guarantee of a fixed memory horizon.

Teacher forcing feeds ground-truth preceding targets during autoregressive training; generation feeds prior model outputs. This exposure difference can compound errors. More data/training can change it but does not structurally remove the train/generation mismatch. Evaluate rollouts, consider appropriate training/decoding approaches, and distinguish this issue from vanishing gradients.

S104 · Embeddings

1. Why do one-hot vectors not encode learned similarity?
2. Compare skip-gram, negative sampling, GloVe and contrastive learning.
3. Explain cosine, normalization and antonym caveats.

Solutions. Distinct one-hot vectors have zero dot product and equal Euclidean distance; they encode identity, not graded semantic similarity. A learned embedding table maps IDs to dense vectors optimized for a task. Skip-gram predicts context words from a center: maximize sums of log p(context|center), with full-softmax logits v_context.T@u_center. Full normalization sums over the vocabulary. Negative sampling instead optimizes a binary discrimination objective log sigmoid(v_pos.T@u) + sum log sigmoid(-v_neg.T@u) using sampled negatives; it is a different objective, not an exact cheap computation of that full normalized likelihood. GloVe fits weighted relationships to log co-occurrence counts; contrastive methods pull positive pairs together relative to negatives according to their chosen supervision.

Cosine is a.dot(b)/(norm(a)*norm(b)) for nonzero vectors. Normalized vectors have dot product equal to cosine and squared Euclidean distance 2-2*cosine, so their rankings agree. Zero vectors need an explicit policy. Raw norms may carry useful relevance/confidence/popularity signals depending on training; it is false that they always represent frequency or should always be discarded.

Antonyms can occur in similar contexts, so distributional proximity need not mean synonymy or agreement. The hand-built vectors below illustrate that possibility, not a measurement on trained word2vec/GloVe. An objective trained for sentiment/entailment and contextual composition may distinguish opposite statements. More dimensions alone are not a cure, but neither does one objective limitation prove every downstream model must fail.

S105 · Transfer Learning

1. Why can early layers transfer, and when might they not?
2. Make a four-quadrant size/domain-distance decision.
3. Explain preprocessing, differential rates and frozen-state checks.

Solutions. Early vision layers often learn reusable local patterns while late layers specialize to source tasks. That is an empirical tendency: a modality change or very different low-level statistics can defeat it. “Domain distance” must be assessed through representative evaluation, not guessed from a dataset name.

These are starting hypotheses, not four deterministic outcomes. Discriminative learning rates assign smaller steps to pretrained layers and larger ones to a fresh head to limit destructive drift. A small rate can also prevent necessary adaptation; validate it. Capture exact weight version, resize/crop/interpolation, value range, RGB/BGR order and normalization. TorchVision weights carry their inference transforms. Intended training augmentation may differ, but validation must reproduce the chosen inference contract.

Audit requires_grad, optimizer parameter groups, state/buffers and train/eval modes separately. A frozen backbone with active BatchNorm can still change running buffers; a forgotten eval switch can keep dropout active. Test group-safe splits and deduplicate near-identical images. High training/low validation accuracy can reflect overfitting, distribution/preprocessing mismatch, optimization or noise; no evidence here establishes that “most transfer failures” have one cause.

Hands-on drill · 30 min

The examples use NumPy or the standard library only. They are window enumeration, scalar derivatives, vector arithmetic and a fixed random feature map—not CNN/LSTM training or pretrained-model validation.

Step 1 — shapes and parameters without a framework (7 min)

import numpy as np
from itertools import product
def out_dim(n,k,p,s,dilation=1):
    if min(n,k,s,dilation)<=0 or p<0:raise ValueError("invalid dimensions")
    effective=dilation*(k-1)+1
    if n+2*p<effective:raise ValueError("kernel does not fit")
    return (n+2*p-effective)//s+1
def enumerate_windows(n,k,p,s,dilation=1):
    # Independent position enumeration, not the quotient formula.
    effective=dilation*(k-1)+1
    return [tuple(start+j*dilation for j in range(k))
            for start in range(0,n+2*p,s) if start+effective<=n+2*p]
checks=0
for n,k,p,s,dil in product(range(1,9),range(1,4),range(3),range(1,4),range(1,3)):
    if n+2*p>=dil*(k-1)+1:
        assert out_dim(n,k,p,s,dil)==len(enumerate_windows(n,k,p,s,dil));checks+=1
n=224;total=0
for cin,cout,k,p,s in [(3,64,3,1,1),(64,128,3,1,2),(128,32,1,0,1),(32,64,5,2,2)]:
    n=out_dim(n,k,p,s);params=cin*cout*k*k+cout;total+=params
    print("output",(cout,n,n),"parameters",params)
assert n==56 and total==131040
full_dense_weights=(3*224*224)*(64*224*224)
print("dense weights to first output volume",full_dense_weights,"window oracles",checks)
# Dropout expectation/variance, exactly enumerated for a fixed activation.
a,q=3.,.5;values=np.array([0.,a/q]);weights=np.array([1-q,q])
assert np.dot(values,weights)==a
assert np.dot((values-a)**2,weights)==a*a*(1/q-1)
print("inverted dropout mean/variance",a,a*a*(1/q-1))

The successive output volumes are 64×224×224, 128×112×112, 32×112×112 and 64×56×56. A dense map from the whole input to the first output volume uses 483,385,147,392 weights before biases; the first convolution uses 1,792 parameters. This isolates locality plus sharing—not an assertion that the dense and convolutional function classes are equivalent or need a particular ratio of training data.

Step 2 — why recurrent gradients decay (8 min)

from math import prod
for factor in [.6,1.4,.95,.99]:
    sequence=[factor]*60;g=1.;trace=[]
    for f in sequence:g*=f;trace.append(g)
    assert np.isclose(trace[-1],factor**60)
    print("fixed derivative factor",factor,[trace[i-1] for i in [1,10,30,60]])
# Time-varying derivative factors are not the same as a fixed spectral radius.
factors=[1.4,.1]*30
assert np.isclose(prod(factors),(.14)**30)
# A nonnormal matrix can amplify transiently despite eigenvalues inside the unit circle.
W=np.array([[.5,10.],[0.,.5]]);g=np.array([0.,1.])
assert max(abs(np.linalg.eigvals(W)))==.5
assert np.linalg.norm(W@g)>np.linalg.norm(g)
assert np.linalg.norm(np.linalg.matrix_power(W,60)@g)<1e-12
print("transient norm",np.linalg.norm(W@g),"eventual norm",np.linalg.norm(np.linalg.matrix_power(W,60)@g))

The scalar rows make contraction/expansion exact. Gates 0.95 and 0.99 retain about 4.6% and 54.7% respectively over 60 factors. The nonnormal counterexample explains why scaling a random matrix to a chosen eigenvalue radius and multiplying random masks does not prove a universal growth claim. It also distinguishes short-horizon amplification from asymptotic behavior.

Step 3 — the cosine trap (7 min)

vectors={"hot":np.array([.90,.10,.42]),"cold":np.array([.89,.12,.44]),
         "warm":np.array([.85,.20,.48]),"database":np.array([.10,.95,.30])}
def cosine(a,b):
    denominator=np.linalg.norm(a)*np.linalg.norm(b)
    if denominator==0:raise ValueError("cosine undefined for zero vector")
    return float(a@b/denominator)
for b in ['cold','warm','database']:print('hot',b,cosine(vectors['hot'],vectors[b]))
a=np.array([3.,4.]);b=10*a;c=np.array([4.,3.])
assert np.isclose(cosine(a,b),1) and np.isclose(cosine(a,c),.96)
assert np.linalg.norm(a-c)<np.linalg.norm(a-b)  # raw Euclidean reverses angular preference
unit=lambda v:v/np.linalg.norm(v)
for v in [b,c]:assert np.isclose(np.sum((unit(a)-unit(v))**2),2-2*cosine(a,v))
try:cosine(a,np.zeros(2));raise AssertionError("missing zero guard")
except ValueError:pass
print("raw dot",a@b,a@c,"normalized squared distances",np.sum((unit(a)-unit(b))**2),np.sum((unit(a)-unit(c))**2))

Cosine faithfully reports angular closeness of these invented vectors; it does not establish a statement's truth, sentiment or the behavior of a trained embedding model. Normalized Euclidean and cosine agree, whereas raw Euclidean is sensitive to norm. Choose the metric and normalization according to the actual embedding training contract and retrieval evaluation.

Step 4 — preprocessing drift (8 min)

rng=np.random.default_rng(3)
W=rng.normal(size=(3,16))/np.sqrt(3);raw=rng.random((200,3))
mean=np.array([.485,.456,.406]);std=np.array([.229,.224,.225])
def features(z):return np.maximum(z@W,0)
reference=features((raw-mean)/std)
variants={'reference':reference,'none':features(raw),'half':features((raw-.5)/.5),
          'BGR':features((raw[:,::-1]-mean)/std)}
for label,f in variants.items():
    print(label,'activation mean',float(f.mean()),'std',float(f.std()),
          'zero fraction',float(np.mean(f==0)),'delta from reference',float(np.linalg.norm(f-reference)))
assert np.linalg.norm(variants['none']-reference)>0
# A reversible affine preprocessing difference is NOT a proof of irrecoverable information loss.
wrong=(raw-.5)/.5
recovered=wrong*.5+.5
assert np.allclose(recovered,raw)
assert np.allclose(features((recovered-mean)/std),reference)
# Byte-identical weights; only preprocessing changed.
assert W.shape==(3,16)
print("normalization mismatch reversed before backbone: reference features restored")

These weights are random, not a pretrained backbone, so there is no pretrained accuracy to report. The experiment shows a mechanism for activation drift and a reversible counterexample to “no head training can ever recover.” Nonlinear clipping downstream can lose information, and a frozen head may fail under shifted features, but norm/zero-fraction changes alone neither measure task failure nor establish its frequency. Check the real recipe and held-out predictions before changing the representation.

Answer constraints and corrections

Misconception: “Transfer means freeze the backbone; bad accuracy proves bad pretrained weights.” Feature extraction is one option. Verify weight-specific preprocessing, layer-wise learning rates, optimizer membership, buffers/modes, split integrity and label quality before concluding the representation is unsuitable. There is no sourced frequency claim that most failures share one cause.

Quiz answers: (1) Inverted dropout scales surviving activations during training and is identity at evaluation; the full nonlinear ensemble mean is not exact, and too much regularization can underfit. (2) Convolution's locality and sharing explain its structural economy; edge filters are possible learned consequences. (3) Time-varying Jacobian products govern recurrent gradients; LSTM's additive cell path can retain gradients with near-one gates but supplies no universal guarantee. (4) Distributional objectives may place antonyms nearby because their contexts overlap; task-specific supervision and contextual representations determine usefulness. (5) For high train/low validation accuracy on a small target set, inspect preprocessing and mode consistency, rates and frozen-state behavior, group/duplicate leakage, cohort shift and labels. Then compare a frozen-head baseline, selective unfreezing, plausible augmentation and regularization. Reduce capacity or acquire representative labels when evidence supports it; a closer pretrained source is another controlled comparison, not a last-resort certainty.

Gap analysis + next week preview · 10 min

Retest a dilated convolution, a pooling ceil-mode exception, a zero vector, or a frozen BatchNorm buffer that changes in train mode. Explain what was measured and what remains a hypothesis. Next week's tokenization, attention, Q/K/V, multi-head layouts and position mechanisms replace some recurrence constraints but preserve these requirements for explicit shapes and objectives.

Corrective source notes

Primary snapshots checked 2026-10-07: PyTorch v2.9.0 Dropout implementation, 2.9 BatchNorm2d, Conv2d, LSTM, Layer Normalization paper, ResNet paper, recurrent-gradient analysis, word2vec negative sampling, GloVe, transfer tutorial, and TorchVision 0.24 weight transforms. The code executes local arithmetic; no trained embedding, pretrained vision or GPU performance result is claimed.

The skip-gram/negative-sampling equations are supported by the full word2vec paper v1; the simple cosine and scalar-gradient examples are mathematical models, not its experiments.

Part of the 6-month evergreen learning plan.