S098 · Backpropagation — Derived by Hand on a 2-Layer Net

Module M12: Deep Learning · Session 98 of 130 · Track: ML 🔙

What you'll be able to do after this session

• Explain Backpropagation to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S097 — Multi-Layer Perceptron — Forward Pass

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · 3Blue1Brown: What is backpropagation really doing? (Ch. 3) — the single best intuition video on backprop, period.
• 🎥 Deep dive (30–60 min) · 3Blue1Brown: Backpropagation calculus (Ch. 4) — the full chain-rule walkthrough.
• 🎥 Hands-on demo (10–20 min) · Andrej Karpathy: micrograd — building autograd from scratch — if you watch one video, watch this.
• 📖 Canonical article · Michael Nielsen: How the backpropagation algorithm works (Ch. 2) — clearest derivation of the four backprop equations.
• 📖 Book chapter / tutorial · CS231n: Backpropagation — Stanford's Karpathy-era notes; computational graphs and local gradients.
• 📖 Engineering blog · Christopher Olah: Calculus on Computational Graphs — Backpropagation — diagram-heavy explanation of why backprop is efficient.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The chain rule that makes deep learning work.

Training a neural network is a search problem: find the weights that minimise the loss. The problem is you have millions of weights and the loss is a complicated function of all of them. Backpropagation is the algorithm that computes the gradient — the direction and magnitude in which every weight should move to reduce the loss — in one efficient pass. It's the reason deep learning works.

The analogy: imagine a mountain range and you're standing on a slope in fog. You want to descend to the lowest valley. Gradient descent = feel which direction is downhill and step that way. Backprop = the machinery that figures out, for every joint in your body and every muscle in every limb, which way to move it to descend most efficiently. Without backprop you could still descend by trial and error (perturb each weight, see if the loss dropped) but for a million weights that would take a million forward passes per step. Backprop gets all million gradients in one backward pass, using the chain rule from calculus.

The insight is the chain rule: if loss depends on w through a chain w → z → h → y → loss, then d loss / d w = (d loss / d y) · (d y / d h) · (d h / d z) · (d z / d w). Each layer knows its own local derivative (dy/dh for that layer). Backprop just multiplies them, right-to-left, accumulating as it goes.

Gotcha to remember forever: the forward pass and backward pass have the same computational cost (up to a constant factor of ~3×). This is called the "cheap gradient principle" and it is the single reason modern AI is affordable. Any training algorithm that doesn't exploit backprop (evolutionary methods, finite differences) is orders of magnitude slower for the same gradient information.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The chain rule on a computational graph

The forward pass computes each node left to right. The backward pass computes derivatives right to left:

1. Start with dL/dy_hat (comes from the loss).
2. Multiply by dy_hat/dz2 (derivative of activation at z2) to get dL/dz2.
3. dL/dW2 = dL/dz2 · h1^T, dL/db2 = dL/dz2, and pass dL/dh1 = W2^T · dL/dz2 back.
4. Repeat: multiply by dh1/dz1, then compute dL/dW1 and dL/db1.

That's the whole algorithm. It's called backpropagation because you propagate the error signal backwards through the graph.

The four fundamental backprop equations (Nielsen's naming)

For a network with pre-activations z^L, activations a^L = sigma(z^L), weights W^L, biases b^L, and MSE loss:

1. Error at output layer: delta^L = (a^L − y) * sigma'(z^L)
2. Error propagated backwards: delta^l = (W^(l+1))^T · delta^(l+1) * sigma'(z^l)
3. Gradient wrt biases: dL/db^l = delta^l
4. Gradient wrt weights: dL/dW^l = delta^l · (a^(l-1))^T

Memorise these four. Every framework (PyTorch, TensorFlow, JAX) computes them automatically via autograd, but knowing them by hand is how you debug real bugs.

Worked example: 1-hidden-unit net

Input x = 1.0, target y = 0.5. Weights w1 = 2.0 (input to hidden), w2 = 3.0 (hidden to output). Activation: sigmoid. Loss: MSE = 0.5(y_hat − y)^2.

Forward:

• z1 = w1 * x = 2.0
• h1 = sigmoid(2.0) = 0.881
• z2 = w2 * h1 = 2.643
• y_hat = sigmoid(2.643) = 0.933
• L = 0.5 * (0.933 − 0.5)^2 = 0.0938

Backward (recall sigmoid'(z) = sigmoid(z) * (1 − sigmoid(z))):

• dL/dy_hat = y_hat − y = 0.433
• dy_hat/dz2 = 0.933 * 0.067 = 0.0625
• dL/dz2 = 0.433 * 0.0625 = 0.0271
• dL/dw2 = dL/dz2 * h1 = 0.0271 * 0.881 = 0.0239
• dL/dh1 = dL/dz2 * w2 = 0.0271 * 3.0 = 0.0812
• dh1/dz1 = 0.881 * 0.119 = 0.105
• dL/dz1 = 0.0812 * 0.105 = 0.00853
• dL/dw1 = dL/dz1 * x = 0.00853

Update: w1 -= lr * 0.00853, w2 -= lr * 0.0239. Do this millions of times → deep learning.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# Backprop by hand in pure NumPy on a 2-layer net. No framework.
import numpy as np
 
rng = np.random.default_rng(42)
def sigmoid(x): return 1 / (1 + np.exp(-x))
def sigmoid_deriv(a): return a * (1 - a)   # given a = sigmoid(z)
 
# XOR dataset (a single perceptron cannot learn it)
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y = np.array([[0],[1],[1],[0]], dtype=float)
 
# Network: 2 -> 4 -> 1
n_in, n_hid, n_out = 2, 4, 1
W1 = rng.normal(0, 0.5, (n_in, n_hid));   b1 = np.zeros((1, n_hid))
W2 = rng.normal(0, 0.5, (n_hid, n_out));  b2 = np.zeros((1, n_out))
lr = 1.0
 
for epoch in range(5000):
    # Forward
    z1 = X @ W1 + b1
    a1 = sigmoid(z1)
    z2 = a1 @ W2 + b2
    a2 = sigmoid(z2)
    loss = ((a2 - y)**2).mean()
 
    # Backward (batched matrix form)
    dz2 = (a2 - y) * sigmoid_deriv(a2)          # (4, 1)
    dW2 = a1.T @ dz2                             # (4, 1)
    db2 = dz2.sum(axis=0, keepdims=True)         # (1, 1)
 
    dh1 = dz2 @ W2.T                             # (4, 4)
    dz1 = dh1 * sigmoid_deriv(a1)                # (4, 4)
    dW1 = X.T @ dz1                              # (2, 4)
    db1 = dz1.sum(axis=0, keepdims=True)         # (1, 4)
 
    # Update
    W1 -= lr * dW1;  b1 -= lr * db1
    W2 -= lr * dW2;  b2 -= lr * db2
 
    if epoch % 500 == 0:
        print(f"epoch {epoch:4d}  loss={loss:.5f}")
 
print("\nFinal predictions (target = [0,1,1,0]):")
print((sigmoid(sigmoid(X @ W1 + b1) @ W2 + b2)).round(3).flatten())
 
# --- Same thing with PyTorch autograd, just to show the abstraction ---
import torch, torch.nn as nn
net = nn.Sequential(nn.Linear(2, 4), nn.Sigmoid(), nn.Linear(4, 1), nn.Sigmoid())
opt = torch.optim.SGD(net.parameters(), lr=1.0)
lossf = nn.MSELoss()
xt = torch.tensor(X, dtype=torch.float32)
yt = torch.tensor(y, dtype=torch.float32)
for epoch in range(5000):
    opt.zero_grad()
    out = net(xt)
    loss = lossf(out, yt)
    loss.backward()   # autograd runs the exact same backprop for you
    opt.step()
print("\nPyTorch autograd XOR:", net(xt).detach().numpy().round(3).flatten())

What to observe when you run it:

1. XOR is solved by the hand-coded backprop — the network converges to something like [0.02, 0.98, 0.98, 0.02].
2. The PyTorch version does exactly the same math via loss.backward(). Autograd is not magic; it's this loop, generalised.
3. Try replacing sigmoid with ReLU by hand — you'll need relu_deriv(z) = (z > 0).astype(float). Convergence is faster.
4. If you set lr = 10.0, the loss diverges within a few epochs — that's the gradient step overshooting.
5. Check shape consistency: dW1.shape == W1.shape, db1.shape == b1.shape. If they don't, your backprop is wrong.

Try this modification: verify the analytical gradient with a numerical gradient check: (loss(w + eps) - loss(w - eps)) / (2*eps) should equal dL/dw to ~5 decimal places for small eps=1e-5. This is the debugging technique used by every serious DL practitioner.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story #1: vanishing gradients. Early deep networks (before ~2012) trained with sigmoid activations often stalled: the loss would drop for a few epochs then flatten out with train accuracy near random. Cause: sigmoid's derivative peaks at 0.25 and drops sharply near ±6. Multiply 10 such derivatives together in a 10-layer backward pass and the gradient becomes ~10^-6 by the time it reaches the first layer. ReLU (derivative 1 for positives), residual connections (skip the multiplication), and batchnorm (renormalise pre-activations) together killed this problem.

War story #2: exploding gradients in RNNs. Recurrent networks unroll into very deep computational graphs (one layer per timestep). Multiplying gradients through 100 timesteps easily produces NaN. The universal fix: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) — rescale gradients whenever their norm exceeds a threshold. Every LLM training script does this.

Common bugs:

• Forgetting optimizer.zero_grad(). PyTorch accumulates gradients across .backward() calls by default. Without a zero-grad, your gradient is a sum of every batch since forever — loss diverges immediately.
• Not calling .backward() — you compute loss, call optimizer.step(), and wonder why nothing changes.
• In-place ops on tensors that require gradient (x += 1) — breaks the computational graph, autograd errors out.
• Detaching by accident (.detach(), .data, torch.no_grad()) — gradient stops flowing at that node; some parameters don't update.
• Loss on the wrong reduction. nn.MSELoss() returns the mean; sum gives different magnitudes and requires a different learning rate.

How top teams handle it:

• Gradient checking (numerical vs analytical) is mandatory when implementing a new layer or custom autograd op. It catches sign-flip and transpose bugs in seconds.
• Gradient clipping by norm (not by value) is standard for transformers and RNNs.
• Mixed precision (bf16/fp16) requires careful loss-scaling to avoid gradient underflow — handled by torch.cuda.amp or bitsandbytes.
• PyTorch's autograd and JAX's grad are just efficient implementations of exactly the four backprop equations above, generalised to arbitrary computational graphs.

(e) Quiz + exercise · 10 min

1. In one sentence, what does backpropagation compute, and why is it more efficient than perturbing each weight individually?
2. State the chain rule and show how it applies to a 2-layer network with input x, hidden z1, activation h1, output y_hat, and loss L.
3. What is the derivative of the sigmoid function sigma(z), expressed in terms of sigma(z) itself? Why is this convenient during backprop?
4. Why must you call optimizer.zero_grad() in every training-step loop iteration in PyTorch?
5. Explain the vanishing gradient problem in one sentence and name two techniques that mitigate it.

Stretch (connects to S097 — forward pass): For the 784→128→64→10 MNIST MLP from S097 with batch size 32, write down the shapes of every gradient tensor (dL/dW1, dL/db1, dL/dW2, dL/db2, dL/dW3, dL/db3) after a full backward pass. Which of these shapes must equal the shape of the corresponding parameter?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Backpropagation? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S099): Optimizers — SGD, Momentum, Adam, RMSprop
• Previous (S097): Multi-Layer Perceptron — Forward Pass
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.