R20 · Week 20 Recall & Drill

Assessment scope

The week's chain is representation → forward pass → derivative → update → framework bookkeeping. Keeping these separate lets you debug a training run without calling every plateau “bad data.” All executable experiments below use NumPy on CPU. PyTorch 2.9 behavior is source-reviewed; no PyTorch installation, GPU training or distributed benchmark is represented as executed.

Sessions covered

• S096 — Perceptron & Activation Functions
• S097 — Multi-Layer Perceptron — Forward Pass
• S098 — Backpropagation — Derived by Hand on a 2-Layer Net
• S099 — Optimizers — SGD, Momentum, Adam, RMSprop
• S100 — PyTorch Fundamentals — Tensors, Autograd, nn.Module

90-min structure

Treat minutes as adjustable pacing, not an execution benchmark.

Blank-page reconstruction · 30 min

S096 · Perceptron & Activations

1. Write the neuron and perceptron update.
2. Explain saturation, vanishing gradients and dead ReLUs.
3. When is sigmoid appropriate, and when is zero initialization safe?

Solutions. A neuron computes z=w.T@x+b, then a=phi(z). For binary labels in {-1,+1}, a perceptron predicts the sign of z; on an example with y*z <= 0, update w <- w + eta*y*x and b <- b + eta*y. Its classical convergence statement requires linearly separable data and appropriate assumptions; it is not the same as differentiating through a hard threshold.

Sigmoid has derivative p*(1-p) and saturates for large-magnitude logits. Repeated small Jacobian factors can suppress an early layer's gradient. ReLU passes positive z and uses derivative 1 there; for z below zero its derivative is 0. A unit negative on every relevant input may stop receiving useful gradient; leaky activations, initialization and learning-rate adjustments are possible responses. Sigmoid is appropriate for a binary probability output, commonly trained with a fused logit-based binary cross-entropy loss; softmax is appropriate for mutually exclusive multiclass probabilities, while regression may use an unconstrained linear output.

Identical hidden units with identical incoming/outgoing conditions can receive identical gradients and remain symmetric. Randomly initialized weights break that symmetry; biases may often start at zero. Zero initialization is not universally fatal: a single linear/logistic model can learn from zero weights. In an all-zero multilayer ReLU network, zero activations and the chosen zero derivative can additionally block learning. The specific computation matters, not a slogan about all zero values.

S097 · MLP Forward Pass

1. Trace a batched network with two hidden layers.
2. State a precise universal approximation result and its limits.
3. Explain biases and the connection to a transformer's feed-forward block.

Solutions. For batch B and widths D,H1,H2,K, X is (B,D), W1 (D,H1), b1 (H1,), Z1 and A1 (B,H1), W2 (H1,H2), b2 (H2,), Z2 and A2 (B,H2), W3 (H2,K), b3 (K,), logits (B,K). Row-vector convention gives A1=phi(X@W1+b1) and so on; broadcasting b across rows adds no per-row trainable parameters. Bias shifts a boundary away from the origin. Affine layers without nonlinearities compose to another affine map, including the combined bias.

One precise version is Cybenko's 1989 theorem: for a continuous sigmoidal activation, finite sums of activated affine functions are dense in the continuous functions on the unit cube. Thus for each continuous f and positive epsilon there is a finite-width single-hidden-layer network uniformly within epsilon. This existence result supplies neither a practical width bound, a procedure guaranteed to find the weights, nor a finite-data generalization guarantee. Other activation/domain variants need their own assumptions.

XOR cannot be separated by one affine threshold. A hidden layer with nonlinear units can represent it, even without many stacked hidden layers; width with nonlinearity is useful. The transformer's common feed-forward branch similarly expands width, applies a nonlinear or gated transformation, then projects back at each position. It is not merely extra attention, and not every transformer uses an identical two-layer activation recipe.

S098 · Backpropagation

1. Derive softmax/cross-entropy's logit gradient.
2. Derive weight, bias and input gradients for an affine layer.
3. Explain why shapes and numerical checks matter.

Solutions. With one-hot Y and unweighted mean loss, a row's log loss is logsumexp(z)-z[target]. Differentiating logsumexp gives softmax probabilities P, so dZ=(P-Y)/B. For Z=A@W+b, the reverse pass is dW=A.T@dZ, db=sum(dZ,axis=0), dA=dZ@W.T. The shapes are exactly W's shape, b's shape and A's shape. A ReLU layer passes dZ=dA*(Z>0) away from its kink.

Backpropagation is reverse-mode differentiation: accumulate derivative contributions on the computation graph. It does not update parameters; an optimizer consumes the gradients. The clean P-Y expression assumes the specified loss/reduction, normalized targets and no extra class weights. Softmax is useful beyond cross-entropy, so “always paired” is too strong. A log-sum-exp implementation avoids exponential overflow and taking the log of an underflowed probability. Adding an epsilon inside a separate probability logarithm changes the objective slightly; the lab instead checks exactly the loss it differentiates.

Central differences compare (L(w+eps)-L(w-eps))/(2*eps) with the analytic derivative. Check away from nondifferentiable kinks, use double precision and both absolute/relative tolerances. A finite check is strong evidence against an implementation error on that fixture, not a proof for all inputs or all floating-point regimes.

S099 · Optimizers

1. Write SGD, momentum, RMSprop and Adam.
2. Explain per-coordinate scaling and bias correction.
3. Distinguish L2 regularization from decoupled weight decay.

Solutions. Let g be the current gradient and eta the base step size. The following are explicitly defined common variants; libraries offer additional options.

• SGD: w <- w - eta*g.
• Momentum: v <- beta*v + g; w <- w - eta*v. This unnormalized-velocity convention differs from an EMA convention by a scale factor.
• RMSprop: s <- rho*s + (1-rho)*g*g; w <- w - eta*g/(sqrt(s)+eps). Here it is uncentered, without momentum.
• Adam: m <- beta1*m+(1-beta1)*g, v <- beta2*v+(1-beta2)*g*g; use mhat=m/(1-beta1**t) and vhat=v/(1-beta2**t), then w <- w - eta*mhat/(sqrt(vhat)+eps).

Starting moments at zero biases early EMA estimates toward zero; under a stationary moment assumption the denominators correct that initialization factor. They do not make noisy gradients unbiased in every nonstationary setting. Dividing by a running second moment changes relative coordinate step scales; it does not choose a universally safe global eta or guarantee superior generalization.

An L2 term contributes lambda*w to g. In plain SGD this is equivalent to multiplicative decay with the matching convention. Under adaptive scaling that penalty is mixed into the moments and scaled per coordinate, so it differs from AdamW's decoupled operation such as w <- (1-eta*lambda)*w - eta*adaptive_gradient. Parameter groups often exclude bias/normalization parameters; that is an explicit recipe choice, not a theorem that every parameter should decay.

S100 · PyTorch Fundamentals

1. What does requires_grad do, and what are the three useful abstractions?
2. Give a training step in the right order.
3. Distinguish eval mode, no-grad and accumulation.

Solutions (PyTorch 2.9 source review). A Tensor is an array-like object with dtype/device/shape, not necessarily a gradient-tracking one. Autograd records differentiable operations in grad mode when relevant inputs require gradients; leaf parameters receive accumulated .grad. An nn.Module registers parameters/submodules/buffers and exposes train/eval behavior; an optimizer owns an update rule and state. These abstractions let the same forward computation use different optimizers.

The five operations are: optimizer.zero_grad(set_to_none=True), logits=model(x), loss=criterion(logits,y), loss.backward(), optimizer.step(). Call model.train() for training. For evaluation, use model.eval() and a no-grad/inference context when derivatives are unnecessary: eval switches dropout/normalization behavior but does not disable autograd. The loss should receive logits if the criterion expects logits, not an extra softmax.

Gradients accumulate by default. For an additive per-example loss evaluated with the same parameters and the same per-example forward computation, intentional microbatch accumulation weights each microbatch mean loss by its sample count divided by the total, then steps once and clears once. Equal-size microbatches can divide loss by their count. For two microbatches of sizes 2 and 1 with mean gradients 3 and 9, the total-batch mean gradient is (2*3+1*9)/3=5, not the unweighted mean 6. This equivalence is conditional: training-mode BatchNorm changes its statistics when a batch is split, and stochastic layers need matched randomness for exact equality. Correct loss weighting does not by itself reproduce a full-batch forward pass. Accidental accumulation is a bug, but “will never change” is not a technical guarantee about future APIs. A detached tensor or an inappropriate no-grad block can also sever a required gradient path.

Hands-on drill · 30 min

Run the four NumPy fences in order. They use row-vector conventions and float64. No framework autograd is substituted for the printed backward pass.

Step 1 — linear layers collapse (7 min)

import numpy as np
rng=np.random.default_rng(0)
X=rng.normal(size=(5,4))
W1,W2,W3=[rng.normal(size=shape) for shape in [(4,8),(8,8),(8,3)]]
b1,b2,b3=[rng.normal(size=n) for n in [8,8,3]]
deep=((X@W1+b1)@W2+b2)@W3+b3
single=X@(W1@W2@W3)+(b1@W2+b2)@W3+b3
assert np.allclose(deep,single,atol=1e-12)
print("affine collapse max error",abs(deep-single).max())
# XOR with two nonlinear hidden units; the final value is already 0 or 1.
Xxor=np.array([[0,0],[0,1],[1,0],[1,1]],float)
expected=np.array([0,1,1,0])
s=Xxor.sum(1)
pred=np.maximum(s,0)-2*np.maximum(s-1,0)
assert np.array_equal(pred,expected)
A=np.c_[np.ones(4),Xxor]
linear=A@np.linalg.lstsq(A,expected,rcond=None)[0]
assert np.allclose(linear,.5)
print("linear XOR fit",linear,"nonlinear XOR",pred)

The affine identity is exact over real arithmetic, not necessarily bit-for-bit under reordered floating-point matrix products. To prove XOR is not linearly separable, positive (1,0) and (0,1) require w1+b>0 and w2+b>0; negative (0,0) requires b<0, while negative (1,1) requires w1+w2+b<0. Adding the positive inequalities gives w1+w2+2b>0, hence w1+w2+b>-b>0, a contradiction. The nonlinear construction explicitly overcomes that limitation; it is not a historical claim about why research stalled.

Step 2 — shapes and parameter counts (7 min)

def parameter_trace(dims,batch=32):
    total=0
    for i,(din,dout) in enumerate(zip(dims,dims[1:]),1):
        count=din*dout+dout;total+=count
        print("layer",i,"W",(din,dout),"b",(dout,),"activation",(batch,dout),"parameters",count)
    return total
assert parameter_trace([784,128,64,10])==109386
assert parameter_trace([512,2048,512])==2099712
# Independent scalar counter for a small shape, including one bias per output.
assert parameter_trace([2,3,1],batch=2)==sum(1 for _ in range(2*3+3+3*1+1))==13

The first network has 100,480 + 8,256 + 650 = 109,386 parameters. The expansion/projection branch has 1,050,624 + 1,049,088 = 2,099,712. Batch size changes activations and computation, not the number of shared parameters. A framework may store a Linear weight transposed relative to this mathematical convention; inspect the API before copying a matrix blindly.

Step 3 — gradient check your derivation (8 min)

rng=np.random.default_rng(5)
B,D,H,K=4,3,4,3
Xg=rng.normal(size=(B,D)); labels=np.array([0,1,2,1]);Y=np.eye(K)[labels]
params={"W1":rng.normal(size=(D,H))*.3,"b1":np.array([.4,-.3,.2,-.1]),
        "W2":rng.normal(size=(H,K))*.3,"b2":np.zeros(K)}
def forward(p):
    z1=Xg@p['W1']+p['b1'];a1=np.maximum(z1,0)
    z2=a1@p['W2']+p['b2'];shift=z2-z2.max(1,keepdims=True)
    logp=shift-np.log(np.exp(shift).sum(1,keepdims=True))
    return float(-np.sum(Y*logp)/B),(z1,a1,np.exp(logp))
def backward(p,cache):
    z1,a1,prob=cache;dz2=(prob-Y)/B
    dz1=(dz2@p['W2'].T)*(z1>0)
    return {'W2':a1.T@dz2,'b2':dz2.sum(0),'W1':Xg.T@dz1,'b1':dz1.sum(0)}
loss,cache=forward(params);grads=backward(params,cache)
assert np.min(abs(cache[0]))>1e-4  # no central difference straddles a ReLU kink
before={k:v.copy() for k,v in params.items()}
eps=1e-6
for name,p in params.items():
    numeric=np.zeros_like(p)
    for index in np.ndindex(p.shape):
        old=p[index]
        p[index]=old+eps;lp=forward(params)[0]
        p[index]=old-eps;lm=forward(params)[0]
        p[index]=old
        numeric[index]=(lp-lm)/(2*eps)
    assert grads[name].shape==p.shape
    assert np.allclose(numeric,grads[name],atol=1e-8,rtol=1e-5)
    assert np.array_equal(p,before[name])
    print(name,"max absolute gradient error",float(abs(numeric-grads[name]).max()))
assert np.allclose(cache[2].sum(1),1)
assert np.allclose(((cache[2]-Y)/B).sum(1),0)
print("loss",loss,"backprop left parameters unchanged")

Each gradient is checked against a separately computed scalar loss perturbation. Subtracting the row maximum is a stable logsumexp evaluation, not a new objective. The zero row-sum logit gradient is another invariant: adding the same constant to all class logits leaves probabilities unchanged.

Step 4 — the optimisers are not interchangeable (8 min)

curvature=np.array([20.,1.])
def bowl(w):return float(.5*np.sum(curvature*w*w))
def run(name,eta,steps=120):
    w=np.ones(2);m=np.zeros(2);v=np.zeros(2)
    for t in range(1,steps+1):
        g=curvature*w
        if name=='sgd':step=g
        elif name=='momentum':m=.9*m+g;step=m
        elif name=='rmsprop':v=.9*v+.1*g*g;step=g/(np.sqrt(v)+1e-8)
        elif name=='adam':
            m=.9*m+.1*g;v=.999*v+.001*g*g
            step=(m/(1-.9**t))/(np.sqrt(v/(1-.999**t))+1e-8)
        else:raise ValueError(name)
        w=w-eta*step
    return w,bowl(w)
for method in ['sgd','momentum','rmsprop','adam']:
    print(method,[(eta,run(method,eta)[1]) for eta in [.005,.05,.2]])
# Exact GD recurrence oracle for diagonal quadratic.
for eta in [.005,.05,.2]:
    w,_=run('sgd',eta,steps=12)
    assert np.allclose(w,(1-eta*curvature)**12)
# Adam first step after bias correction is g/(abs(g)+epsilon).
g=curvature.copy();expected=np.ones(2)-.05*g/(abs(g)+1e-8)
assert np.allclose(run('adam',.05,steps=1)[0],expected)
# Gradient accumulation is addition, not an optimizer update.
g1=np.array([1.,2.]);g2=np.array([3.,-2.]);acc=g1+g2
assert np.array_equal(acc,[4.,0.])
print("microbatch equal-size average",acc/2)

For plain GD the steep coordinate multiplies by 1-20*eta; eta=0.2 multiplies it by -3, so it explodes even if 120 steps remain finite in float64. “Finite” is not “converged.” The adaptive rows may remain bounded at all three tested rates; the experiment must not claim every row diverged somewhere or prove a wider safe band universally. Compare final loss, trajectory and budget; a large fixed adaptive step can oscillate near a minimum and a small one can be slow.

Answer constraints and corrections

Misconception: “Adam chooses the learning rate for me.” It rescales coordinates through moment estimates; the base eta still multiplies the step. Sweep rates on validation evidence, inspect trajectories and consider schedules/warmup where justified. No optimizer universally removes conditioning, instability or generalization problems.

Quiz solutions: (1) XOR defeats a single affine boundary; a finite nonlinear hidden layer solves it as shown. (2) The stated approximation theorem is on a compact cube with a continuous sigmoidal activation; representation existence is not trainability, a practical width bound or sample efficiency. (3) Backprop differentiates; the optimizer updates. (4) Differentiating logsumexp minus the target logit gives P-Y, with the mean factor 1/B; fused logit losses improve numerical behavior. (5) For a plateau, first check labels/split/preprocessing and expected loss scale, then forward shapes and loss API, finite-difference derivatives, gradient clearing/accumulation and detached paths. Inspect per-layer gradient norms, inactive units and initialization; sweep eta both ways. Try fitting a tiny consistent, representable subset. Failure there can reveal a bug, insufficient capacity, contradictory labels, an excessive regularizer or optimization difficulty—not inevitably a bug. Only then use capacity and feature interventions justified by the evidence.

Gap analysis + next week preview · 10 min

Retest by adding a bias to an affine-collapse proof, changing loss reduction from mean to sum, or deliberately accumulating two microbatches. Predict exactly which gradient or step changes. Next week's dropout/normalization, convolutions, recurrence, embeddings and transfer learning all reuse these shape and mode distinctions.

Corrective source notes

Primary references checked 2026-10-07: Cybenko 1989, Theorem 2, PyTorch 2.9 autograd, modules, CrossEntropyLoss, optimizers, tagged v2.9.0 SGD, RMSprop, Adam, and Decoupled Weight Decay Regularization. The explicit NumPy variants and derivatives are executed; framework/GPU performance and full training behavior are not.

Part of the 6-month evergreen learning plan.