DL S010 · Backprop by Hand

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 10 / 80 · Module M02 · ~2.5 hours

0 · The story — the day deep learning stopped being magic

October 9, 1986. Nature prints Rumelhart–Hinton–Williams's 8-page letter on backpropagation. The math in that letter is exactly what you're going to do today by hand on a 9-parameter network. But the paper is not really about the math — Werbos had the derivation in 1974, Parker had it in 1985, LeCun had it in his 1985 dissertation. What Rumelhart–Hinton–Williams did was make it routinely reproducible. They wrote clean C code, they mailed floppy disks to anyone who wrote and asked, and their small demonstration networks (XOR, a family tree, a symmetry detector) actually trained on the toy 68020-based UNIX workstations of the day.

Andrej Karpathy wrote in 2016 a blog post titled "Yes you should understand backprop" that has become the internal-training doctrine at half the ML companies I've worked with. His argument, compressed: every time a training run goes wrong in a way that's not "loss is NaN," the fix requires understanding what the gradient tensor is doing. Vanishing gradients through a sigmoid stack, exploding gradients through a long RNN, gradients zeroed by a dying ReLU, gradients that don't accumulate across a .detach() you forgot — every one of these is invisible unless the shape of the gradient computation is in your head. And the only way it gets into your head is doing it once, by hand, on numbers you can hold in your head.

That's today. Nine parameters. One example. One BCE loss. Full derivation. Then code. Then numeric check. Then a real training loop that solves XOR. After this session, loss.backward() is not a mysterious PyTorch call; it is code you have written yourself, and you know exactly what bytes it moves.

1 · What you'll build today

Compute, verify, and code every one of the 9 gradients of the 2-2-1 MLP; then train the network on XOR to 100% accuracy in <3000 SGD steps.

2 · Checkpoint · you should already know

• S004 · Derivatives and the chain rule — the chain rule, sum-over-paths, and the linear-layer gradient formulas. Today is a full application of §6 of that session.
• S007 · A single artificial neuron — the clean a − y cancellation for sigmoid + BCE. Today it becomes Y − y_true for a batch.
• S008 · Activations — you need the ReLU derivative (1 if z > 0 else 0) at your fingertips.
• S009 · The Forward Pass for a 2-Layer MLP — you'll differentiate the exact forward pass from that session. The cache tuple (X, H_pre, H, Y_pre, Y) is critical today.

3 · Recap — the forward pass from S009

H_pre = X @ W1 + b1              shape (N, 2)
H     = ReLU(H_pre)              shape (N, 2)
Y_pre = H @ W2 + b2              shape (N, 1)
Y     = σ(Y_pre)                 shape (N, 1)
L     = BCE(Y, y_true)           scalar (mean over batch)

Cache: (X, H_pre, H, Y_pre, Y).

BCE with mean reduction:

Now we walk backwards.

4 · The four cheat-sheet formulas — the whole backprop world in a table

For every graph in this series (and every graph you'll ever meet), gradients flow through linear layers, elementwise activations, and loss endpoints. Three shapes of node, three kinds of formula:

Memorise this table. It is roughly 80% of the practical backprop knowledge you will use for the next ten years of ML work. The remaining 20% is: BatchNorm/LayerNorm (S021), Conv (S025), Attention (S037), all of which are specialised versions of the same three moves.

5 · Backward pass — the full derivation, walked

Start at the very end and walk left.

5.1 dY_pre — the clean sigmoid + BCE gradient

From S007 §6 (the cancellation): the pre-activation gradient for sigmoid + BCE is (Y - y_true), and the mean reduction adds the 1/N:

dY_pre = (Y - y_true) / N       shape (N, 1)

If your loss used sum instead of mean, drop the 1/N. If it used sum and then divided by a fixed constant K outside, divide by K. The rule is: whatever scalar you multiplied the loss by, multiply the gradient by too — because gradient of a scalar-times-loss is that scalar times the gradient.

5.2 Layer 2 — the output linear layer Y_pre = H @ W2 + b2

Apply the cheat-sheet formulas with upstream dZ = dY_pre:

Notice dH — the gradient with respect to layer 1's output — becomes the upstream gradient for the ReLU node next.

5.3 ReLU node — H = ReLU(H_pre)

The derivative of max(0, z) is 1 where z > 0, 0 elsewhere:

dH_pre = dH * (H_pre > 0)       shape (N, 2)

Elementwise multiplication with a boolean mask (which broadcasts as 0/1). Where H_pre was negative, the gradient is killed. That is exactly the "dying ReLU" pathology from S008 §7, seen from the backward-pass side.

5.4 Layer 1 — H_pre = X @ W1 + b1

Same cheat-sheet formulas with upstream dZ = dH_pre:

We usually don't care about dX unless we're doing adversarial examples, saliency maps, or backprop-into-inputs (which shows up in style transfer, DeepDream, and PGD attacks).

Done. We have dW1, db1, dW2, db2 — all 9 gradients. Every one is a matmul, a sum, or an elementwise mask. All of backprop for this network is: 4 matmuls + 2 sums + 1 mask. That's it. That's the algorithm that trains ChatGPT (at a very much larger scale).

6 · By-hand computation on one example

Reuse the numbers from S009 §4:

x   = [1.0, 2.0]                  # single example, N = 1
W1  = [[0.1, 0.2], [0.3, 0.4]]
b1  = [0.1, 0.1]
W2  = [[0.5], [0.6]]
b2  = [0.2]

Forward gave us:

H_pre = [0.8, 1.1]     H = [0.8, 1.1]     Y_pre = [1.26]     Y ≈ [0.7789]

Take y_true = 1.0 — the network was "somewhat right" (predicted 0.78 for a positive example), so the gradients should be moderate.

6.1 dY_pre

dY_pre = (Y - y_true) / N = (0.7789 - 1.0) / 1 = -0.2211        shape (1,)

Negative because Y < y_true: we underpredicted, so increasing Y_pre (and thus Y) would reduce loss.

6.2 dW2, db2, dH

dW2 = H.T @ dY_pre
    = [[0.8], [1.1]] · (-0.2211)
    = [[-0.1769], [-0.2432]]                                     shape (2, 1)
 
db2 = dY_pre.sum() = -0.2211                                     shape (1,)
 
dH  = dY_pre @ W2.T = -0.2211 · [0.5, 0.6] = [-0.1106, -0.1327]  shape (1, 2)

6.3 dH_pre (ReLU mask)

Both entries of H_pre are positive, so the mask is [1, 1]:

dH_pre = dH * (H_pre > 0) = [-0.1106, -0.1327] * [1, 1] = [-0.1106, -0.1327]

6.4 dW1, db1

dW1 = x.T @ dH_pre
    = [[1.0], [2.0]] @ [-0.1106, -0.1327]
    = [[1.0 · -0.1106,  1.0 · -0.1327],
       [2.0 · -0.1106,  2.0 · -0.1327]]
    = [[-0.1106, -0.1327],
       [-0.2211, -0.2653]]                                       shape (2, 2)
 
db1 = dH_pre.sum(axis=0) = [-0.1106, -0.1327]                    shape (2,)

All 9 gradients computed by hand. Type them into NumPy in the next section and confirm agreement to at least 4 dp.

6.5 What do the gradients mean?

dW2[0,0] = -0.177 means: increasing W2[0,0] by a tiny dw would decrease loss by ~0.177 · dw. So to reduce loss, SGD will increase that weight by lr · 0.177. That is what gradient descent will do:

W2[0,0]_new = W2[0,0] - lr · dW2[0,0] = 0.5 - 0.1 · (-0.177) = 0.5177

Interpret every gradient this way. Sign matters (increase or decrease?). Magnitude matters (does this parameter matter right now?). If a whole row of dW1 is zero, that hidden unit is effectively dead and won't update — see the "dying ReLU" war story below.

7 · The backward pass in code

def mlp_backward(y_true, cache, params):
    X, H_pre, H, Y_pre, Y = cache
    W1, b1, W2, b2 = params
    N = X.shape[0]
 
    # sigmoid + BCE fused: pre-activation gradient
    dY_pre = (Y - y_true.reshape(Y.shape)) / N       # (N, 1)
 
    # layer 2 (linear)
    dW2 = H.T @ dY_pre                                # (2, 1)
    db2 = dY_pre.sum(axis=0)                          # (1,)
    dH  = dY_pre @ W2.T                               # (N, 2)
 
    # ReLU
    dH_pre = dH * (H_pre > 0)                         # (N, 2)
 
    # layer 1 (linear)
    dW1 = X.T @ dH_pre                                # (2, 2)
    db1 = dH_pre.sum(axis=0)                          # (2,)
 
    # shape asserts — catch bugs at their source
    assert dW1.shape == W1.shape
    assert db1.shape == b1.shape
    assert dW2.shape == W2.shape
    assert db2.shape == b2.shape
 
    return dW1, db1, dW2, db2

Fifteen lines of algorithm + four asserts. That's your first hand-written backprop. Run it on our example and verify it matches §6:

import numpy as np
X  = np.array([[1.0, 2.0]])
y  = np.array([1.0])
params = (
    np.array([[0.1, 0.2], [0.3, 0.4]]),
    np.array([0.1, 0.1]),
    np.array([[0.5], [0.6]]),
    np.array([0.2]),
)
Y, cache = mlp_forward(X, params)
dW1, db1, dW2, db2 = mlp_backward(y, cache, params)
print("dW1:\n", dW1)
# [[-0.1106 -0.1327]
#  [-0.2211 -0.2653]]     ✅ matches paper
print("dW2:\n", dW2)
# [[-0.1769]
#  [-0.2432]]              ✅

Matches. ✅

8 · Numeric gradient check — the single most useful debug tool in ML

Symbolic gradients are wrong more often than you'd think. Always sanity-check with two-sided finite differences (S007 §6.4):

def num_grad(params, X, y, key, index, h=1e-6):
    """Perturb params[key][index] by ±h, recompute loss, return numeric derivative."""
    p_plus  = tuple(p.copy() for p in params)
    p_minus = tuple(p.copy() for p in params)
    p_plus[key][index]  += h
    p_minus[key][index] -= h
    L_plus  = bce_loss(mlp_forward(X, p_plus)[0], y)
    L_minus = bce_loss(mlp_forward(X, p_minus)[0], y)
    return (L_plus - L_minus) / (2*h)
 
# Check every entry of dW1
Y, cache = mlp_forward(X, params)
dW1_anal, db1_anal, dW2_anal, db2_anal = mlp_backward(y, cache, params)
print("dW1 check:")
for i in range(2):
    for j in range(2):
        num  = num_grad(params, X, y, key=0, index=(i, j))
        anal = dW1_anal[i, j]
        print(f"  dW1[{i},{j}]: analytic={anal: .6f}  numeric={num: .6f}  diff={abs(num-anal):.2e}")

Expected — differences all around ~1e-8:

dW1 check:
  dW1[0,0]: analytic=-0.110616  numeric=-0.110616  diff=3.11e-09
  dW1[0,1]: analytic=-0.132717  numeric=-0.132717  diff=1.24e-09
  dW1[1,0]: analytic=-0.221232  numeric=-0.221232  diff=2.79e-09
  dW1[1,1]: analytic=-0.265434  numeric=-0.265434  diff=8.90e-10

If any difference is bigger than ~1e-5, you have a bug. Do not proceed. In production, use relative error — |a-n| / max(|a|, |n|, ε) — which handles gradients of very different magnitudes uniformly. See Karpathy's CS231n gradient-check notes for the definitive treatment.

8.1 Common numeric-check pitfalls

• h too small → floating-point noise swamps the signal; use 1e-5 to 1e-7.
• h too large → the finite-difference truncation error swamps the answer; don't use 1e-2.
• Using single precision → checks fail even when analytic is right; run gradcheck in float64.
• Random state changing between forward passes (dropout!) → different subgraphs, different gradients. Fix seed or disable stochastic layers during gradcheck.
• Checking on a saturated point → the local slope is legitimately zero; try a different input.

9 · Full training loop — solve XOR

Combine forward + backward + SGD:

def bce_loss(Y, y, eps=1e-9):
    Y = Y.reshape(-1); y = y.reshape(-1)
    return -np.mean(y*np.log(Y+eps) + (1-y)*np.log(1-Y+eps))
 
def train_mlp(X, y, hidden=8, lr=0.5, steps=5000, seed=0):
    rng = np.random.default_rng(seed)
    N, d_in = X.shape
    # He init (S022) — sqrt(2/fan_in) for ReLU nets
    W1 = rng.standard_normal((d_in, hidden)) * np.sqrt(2/d_in)
    b1 = np.zeros(hidden)
    W2 = rng.standard_normal((hidden, 1)) * np.sqrt(2/hidden)
    b2 = np.zeros(1)
    for t in range(steps):
        Y, cache = mlp_forward(X, (W1, b1, W2, b2))
        L = bce_loss(Y, y)
        dW1, db1, dW2, db2 = mlp_backward(y, cache, (W1, b1, W2, b2))
        W1 -= lr * dW1;  b1 -= lr * db1
        W2 -= lr * dW2;  b2 -= lr * db2
        if t % 500 == 0:
            acc = ((Y.flatten() > 0.5) == (y > 0.5)).mean()
            print(f"step {t:4d}  loss={L:.4f}  acc={acc:.2%}")
    return W1, b1, W2, b2

Now try it on the XOR problem a single neuron could not solve (S007 §8):

X = np.array([[0,0], [0,1], [1,0], [1,1]], dtype=float)
y = np.array([0, 1, 1, 0], dtype=float)
train_mlp(X, y, hidden=8, lr=0.5, steps=5000)

Expected:

step    0  loss=0.6931  acc=50.00%
step  500  loss=0.4321  acc=75.00%
step 1000  loss=0.0891  acc=100.00%
step 1500  loss=0.0341  acc=100.00%
...
step 4500  loss=0.0104  acc=100.00%

Hits 100% accuracy on XOR. You just solved the problem Minsky and Papert wrote a book about, using code you wrote from scratch. Take a moment. Every line was yours; every gradient you can defend on a whiteboard. This is the whole magic — nothing more, nothing less.

10 · Diagram — forward and backward together (screenshot this)

Print this, tape it above your monitor. You now understand every training loop ever written. Every deep network's backward pass is this graph, generalised to more layers and more edge types. The exact same diagram, drawn for a 96-layer GPT, is what makes loss.backward() cost 2× a forward pass — one node per layer, one edge per matmul, backwards.

11 · The line from your NumPy code to PyTorch's autograd

PyTorch's loss.backward() does what you just did — with three refinements:

1. The graph is built dynamically during forward. Every operation on a requires_grad=True tensor records its inputs and its local backward function into a graph node. Your cache tuple is PyTorch's ctx.save_for_backward(...).
2. The traversal is a reverse-topological sort. PyTorch walks the graph from the loss node backwards, calling each node's backward function in the right order. Your left-to-right derivation was the manual version of this sort.
3. Local backward functions are looked up in C++. Every op (matmul, relu, sigmoid, ...) has a hand-written backward in the ATen library. Your §4 cheat-sheet table is basically the (Python) header of ATen's derivatives.yaml.

When you write loss.backward(), PyTorch:

1. Sets loss.grad = 1.0.
2. Reverse-topo-sorts the graph.
3. For each node, calls its local backward with the upstream gradient, computes gradients for its inputs, and either accumulates into parameter .grad attributes or propagates further.
4. Frees the graph (unless retain_graph=True).

You just did all four steps manually. You have written a two-node autograd engine. S012 (micrograd) will generalise this to an arbitrary scalar-valued DAG.

12 · The 2025 twist — why hand backprop still matters in the autograd era

Three places in the current stack where knowing backprop by hand is not optional:

• Custom kernels — every FlashAttention v1/v2/v3 (Dao 2022, 2023, 2024) release includes both a forward and a hand-derived backward. If you write a Triton or CUDA kernel for a new attention variant, the forward is fun; the backward is where the paper gets written. See the FlashAttention-2 paper (Dao, 2023, arxiv 2307.08691), §3.
• Gradient checkpointing (Chen et al. 2016) trades compute for memory by not caching intermediates during forward and recomputing them during backward. To decide what to recompute, you have to know the shape and cost of each backward step. Every 100B+ LLM uses this.
• Debugging exploding / vanishing / NaN gradients in production LLM training. When your 70B model's loss spikes at step 47,000, pytorch_lightning will not save you. You need to know that dW = X.T @ dZ scales with the input norm, that the dying-ReLU mask can kill entire columns, and that RoPE positional encoding's backward has non-obvious phase behaviour. Every one of these debug sessions cashes out into a cheat-sheet formula you learned today.

Further reading:

• Rumelhart–Hinton–Williams, "Learning representations by back-propagating errors" (1986) — https://www.nature.com/articles/323533a0
• Baydin et al., "Automatic differentiation in machine learning: a survey" (2018) — https://arxiv.org/abs/1502.05767 — the definitive modern reference on AD.
• Karpathy, "Yes you should understand backprop" (2016) — https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b
• Chen et al., "Training Deep Nets with Sublinear Memory Cost" (gradient checkpointing, 2016) — https://arxiv.org/abs/1604.06174
• Dao, "FlashAttention-2" (2023) — https://arxiv.org/abs/2307.08691

13 · War stories

14 · Try it yourself

import torch, torch.nn as nn, torch.nn.functional as F
class TinyMLP(nn.Module):
    def __init__(self, d_in=2, hidden=8):
        super().__init__()
        self.fc1 = nn.Linear(d_in, hidden)
        self.fc2 = nn.Linear(hidden, 1)
    def forward(self, x):
        return self.fc2(F.relu(self.fc1(x)))
model = TinyMLP()
opt = torch.optim.SGD(model.parameters(), lr=0.5)
X_t = torch.tensor(X, dtype=torch.float32); y_t = torch.tensor(y, dtype=torch.float32).view(-1,1)
for t in range(5000):
    logits = model(X_t)
    loss = F.binary_cross_entropy_with_logits(logits, y_t)
    opt.zero_grad(); loss.backward(); opt.step()

15 · Recap

You did backpropagation on paper for a 2-2-1 MLP: 9 gradients, one input example, four cheat-sheet formulas. You coded the backward pass in 15 lines of NumPy and verified every one of the 9 gradients against a two-sided finite-difference check to ~1e-8. You trained the same network on XOR to 100% accuracy in <3000 SGD steps — solving the problem that ended the first AI winter, in code you wrote from scratch. Along the way you saw the direct line from your NumPy backward to PyTorch's autograd, and heard about the modern echoes: FlashAttention custom backwards, gradient checkpointing, dead-ReLU debugging on 70B-parameter LLMs. Everything else in this series is scale.

import numpy as np
# hand-rolled accumulation, exactly what autograd does
class P:
    def __init__(s, v): s.v, s.grad = v, 0.0
w = P(3.0)
for i in range(3):
    # L = w**2  ->  dL/dw = 2w = 6.0 every time
    w.grad += 2 * w.v            # backward ACCUMULATES
    print(i, w.grad)             # 6.0, 12.0, 18.0  <- not 6,6,6

w.grad = 0.0                     # the zero_grad() you forgot
w.grad += 2 * w.v
print("correct", w.grad)         # 6.0

🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________________________________________

Spaced review: re-derive the §6 hand-trace in 24 hours; revisit §10 (forward-and-backward diagram) on day 7.

Next session (S011): put forward + backward + SGD together and train a 784-256-10 MLP on real MNIST — 60,000 hand-written digits — in NumPy. Target: 96%+ test accuracy in under 60 lines of code. Everything from M01 and M02 pays off.

Sticky note (keep on your desk):

• Cheat-sheet table (linear: dX = dZ@W.T, dW = X.T@dZ, db = dZ.sum(0); ReLU: dZ = dA*(Z>0); sigmoid+BCE: dY_pre = (Y-y)/N). Draw it on an index card.
• Shape rule: dTensor.shape == Tensor.shape. Always. Assert it.
• Debug tool: two-sided numeric gradcheck. h = 1e-6. Every from-scratch backprop starts with this.

Previous: ← DL S009 · The Forward Pass for a 2-Layer MLP · Next: DL S011 · MLP on MNIST — All Numpy, No PyTorch →