DL S009 · The Forward Pass for a 2-Layer MLP
Hand-compute the forward pass of a tiny 2-2-1 network, then a 2-3-1 XOR solver, then a batched (N, 784) → 10 MNIST-scale MLP — the atom of every deep network you will ever build. Story hook (Rumelhart, Hinton, Williams in 1986 solving XOR in a single training pass), row-batch vs column-vector conventions, universal approximation intuition via Chris Olah's manifold picture, and a full parameter count for GPT-2 Small so you feel the same MLP inside every transformer block. Part of the 'Deep Learning & LLMs From Scratch' 80-session self-study series.
🎯 On paper, compute the forward pass of a 2-input, 2-hidden, 1-output MLP with tiny weights; verify in ~10 lines of NumPy; then hand-construct an XOR-solving 2-3-1 network to prove — geometrically — why stacking layers breaks the linear-separability wall from S007.
Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 9 / 80 · Module M02 · ~2 hours
0 · The story — the paper that ended the AI winter
October 1986. Nature publishes an 8-page letter by David Rumelhart, Geoffrey Hinton, and Ronald Williams: "Learning representations by back-propagating errors." Its opening figure is a network of exactly the size you're about to build today — two inputs, two hidden units, one output — solving XOR. The problem Minsky and Papert had proved impossible for a single perceptron seventeen years earlier (S007 §8) falls in a few hundred training steps. The paper is short. Its citation count as of 2025 is over 50,000. Every deep network that has ever been trained descends from the algorithm in that letter.
The trick was not new — Werbos had derived it in his 1974 PhD thesis; LeCun had used it in 1985; the Soviet mathematician Alexey Ivakhnenko had similar ideas in 1965. What Rumelhart–Hinton–Williams did was make it work in practice, on real problems, with a clear demonstration and a clean derivation. The demonstration was XOR. The mechanism was the multi-layer perceptron. The training algorithm — backpropagation — is Session 010's problem. Today you build the forward half of the 1986 network.
Why does it matter that you can compute it by hand? Because every transformer block in every LLM you have ever prompted contains an MLP exactly the shape of what you're about to build — just with d_model = 4096 instead of 2, d_ff = 16384 instead of 2, and no output sigmoid. When you read nanoGPT in S040 you'll see the two lines:
h = self.c_fc(x) # (B, T, d_model) @ (d_model, d_ff)
h = new_gelu(h) # activation
y = self.c_proj(h) # (B, T, d_ff) @ (d_ff, d_model)and you will recognise them as the same forward pass you're about to hand-trace. This is not a metaphor. It is literally the same math.
1 · What you'll build today
Compute the forward pass of a 2-2-1 MLP by hand, then in NumPy, then batch it. Then hand-construct a 2-3-1 network that solves XOR without any training.
- Draw a 2-2-1 MLP with every weight matrix and bias labelled and every shape written next to the arrow.
- Run the forward pass on one input on paper and get the same numbers as NumPy to 4 decimal places.
- State the shape of every intermediate tensor for a batch of N inputs.
- Hand-construct weights for a 2-3-1 MLP that solves XOR — no training required.
- Explain why activations go BETWEEN layers but not at the very end (usually), matched to the loss.
- Predict the parameter count of any MLP given (input_dim, hidden_dims, output_dim), and verify it for GPT-2 Small's FFN block.
- Recognise the same MLP structure inside a transformer FFN.
2 · Checkpoint · you should already know
- S002 · Vectors, matrices, geometry — matmul and shape rules.
- S003 · Broadcasting — bias addition uses broadcasting across the batch axis.
- S007 · A single artificial neuron — one neuron. Today we build many, arranged in layers.
- S008 · Activations — we'll use ReLU as the hidden activation and sigmoid at the output for binary classification.
3 · The 2-2-1 architecture — draw it once, forever
Parameter count: W₁ has 4, b₁ has 2, W₂ has 2, b₂ has 1. Total: 9 parameters. Small enough to write on a napkin. Structurally identical (up to bigger dimensions and different activations) to a 1024-1024-10 MNIST MLP or to the FFN inside any transformer block.
3.1 The forward equations (row-batch convention)
h_pre = x @ W₁ + b₁ shape (2,) — pre-activation of hidden layer
h = ReLU(h_pre) shape (2,) — post-activation of hidden layer
ŷ_pre = h @ W₂ + b₂ shape (1,) — pre-activation of output
ŷ = σ(ŷ_pre) shape (1,) — output probabilityRead those four lines out loud once. That is a neural network. Everything else in this session is either arithmetic on those four lines or a variant of them.
4 · One forward pass — by hand, on numbers you can hold in your head
Pick these values:
x = [1.0, 2.0]
W₁ = [[0.1, 0.2], b₁ = [0.1, 0.1]
[0.3, 0.4]]
W₂ = [[0.5], b₂ = [0.2]
[0.6]]4.1 Layer 1
Using h_pre = x @ W₁ + b₁:
4.2 Layer 2
h @ W₂ = [0.8, 1.1] @ [[0.5], [0.6]]
= [ 0.8·0.5 + 1.1·0.6 ]
= [ 0.4 + 0.66 ]
= [ 1.06 ]
ŷ_pre = [1.06] + [0.2] = [1.26]
ŷ = σ(1.26) = 1 / (1 + e^{-1.26}) ≈ 0.7788Output: 0.78. The model gives about 78% probability to class 1 for this input.
4.3 Verify in NumPy
import numpy as np
x = np.array([1.0, 2.0])
W1 = np.array([[0.1, 0.2], [0.3, 0.4]])
b1 = np.array([0.1, 0.1])
W2 = np.array([[0.5], [0.6]])
b2 = np.array([0.2])
h_pre = x @ W1 + b1
h = np.maximum(0, h_pre)
y_pre = h @ W2 + b2
y = 1 / (1 + np.exp(-y_pre))
print("h_pre:", h_pre) # [0.8 1.1]
print("h :", h) # [0.8 1.1]
print("y_pre:", y_pre) # [1.26]
print("y :", y) # [0.77894181...]Numbers match to machine precision. ✅ You just executed a neural network on paper. If your hand answer disagrees with the NumPy answer, do not proceed — one of your dot products has a sign flipped. Find it before moving on. The number of engineers whose next six months of debugging would have been shorter had they done this exercise once is uncountable.
5 · Batched forward pass — the shape you'll actually use
Real training uses batches. For N inputs at once:
X ∈ (N, 2) H_pre = X @ W₁ + b₁ ∈ (N, 2) H = ReLU(H_pre) ∈ (N, 2)
Y_pre = H @ W₂ + b₂ ∈ (N, 1) Ŷ = σ(Y_pre) ∈ (N, 1)Bias b₁: (2,) broadcasts across the batch axis by NumPy's rules (S003 §5): the shape (N, 2) + (2,) is legal and adds b₁ to every row of the (N, 2) matrix. Same forward code, just with a 2D input:
X = np.array([[ 1.0, 2.0],
[ 0.5, 0.5],
[-1.0, 0.3],
[ 2.0, -1.5]]) # shape (4, 2), a batch of 4 examples
H_pre = X @ W1 + b1 # shape (4, 2)
H = np.maximum(0, H_pre) # shape (4, 2)
Y_pre = H @ W2 + b2 # shape (4, 1)
Y = 1 / (1 + np.exp(-Y_pre)) # shape (4, 1)Same ten lines, N inputs at a time. This is the entire forward pass of a 2-layer MLP. Change the shapes to (1024, 784), (784, 256), (256, 10) and you have an MNIST classifier (S011). Change them to (1, 2048, 4096), (4096, 16384), (16384, 4096) and you have the FFN block of GPT-3 (S040).
5.1 The three inputs traced through
Take row 2: x = [0.5, 0.5].
h_pre = [0.5, 0.5] @ [[0.1, 0.2], [0.3, 0.4]] + [0.1, 0.1]
= [0.5·0.1 + 0.5·0.3, 0.5·0.2 + 0.5·0.4] + [0.1, 0.1]
= [0.20, 0.30] + [0.1, 0.1] = [0.30, 0.40]
h = [0.30, 0.40]
y_pre = [0.30·0.5 + 0.40·0.6] + [0.2] = [0.15 + 0.24 + 0.2] = [0.59]
y = σ(0.59) ≈ 0.6434Take row 3: x = [-1.0, 0.3].
h_pre = [-1.0·0.1 + 0.3·0.3, -1.0·0.2 + 0.3·0.4] + [0.1, 0.1]
= [-0.01, -0.08] + [0.1, 0.1] = [0.09, 0.02]
h = [0.09, 0.02] (ReLU: both still positive)
y_pre = [0.09·0.5 + 0.02·0.6] + [0.2] = [0.057 + 0.2] = [0.257]
y = σ(0.257) ≈ 0.5639Notice h_pre came out just barely positive. A tiny nudge to W₁ — a gradient step during training — could easily push those hidden neurons negative and zero them out. Session 010's backward pass will make this failure mode concrete.
6 · Solving XOR by hand — the payoff of stacking
Recall S007 §8: no single neuron can solve XOR. Here's a 2-3-1 MLP that solves it without any training. I will tell you the weights and let you verify.
X = [[0,0], [0,1], [1,0], [1,1]], y = [0, 1, 1, 0]
W1 = [[ 1, 1, 1], b1 = [0.5, -0.5, -1.5]
[ 1, 1, 1]]
W2 = [[ 1], b2 = [-0.5]
[-2],
[ 1]]Trace x = [0, 1]:
That's on the boundary. Try [1, 0]:
Both [0,1] and [1,0] give h = [1.5, 0.5, 0] because W₁ is symmetric — w·x only depends on x₁ + x₂. Great: the hidden layer has learned the feature x₁ + x₂. Now try [0,0]:
And [1,1]:
h_pre = [1, 1]·W1 + b1 = [2, 2, 2] + [0.5, -0.5, -1.5] = [2.5, 1.5, 0.5]
h = [2.5, 1.5, 0.5]
y_pre = [2.5·1 + 1.5·(-2) + 0.5·1] - 0.5 = [2.5 - 3.0 + 0.5] - 0.5 = 0 - 0.5 = -0.5
y = σ(-0.5) ≈ 0.378Not perfect — the numbers I chose are illustrative rather than optimal — but the point stands: the hidden layer produced three "feature detectors" (roughly: x₁+x₂ > -0.5, x₁+x₂ > 0.5, x₁+x₂ > 1.5) that the output layer combines to distinguish "exactly one of x₁, x₂ is 1" from the other cases. A trained network converges to sharper weights and gets 100% accuracy in a few dozen SGD steps (do this yourself in the Try It below).
The geometric picture: layer 1 bends 2D input space into a 3D feature space where the XOR classes ARE linearly separable, then the output neuron draws the separating line in that bent space. This is exactly Chris Olah's manifold-deformation argument from the pre-read.
7 · Why ReLU between layers, but not at the very end
The activation between layers gives the network nonlinearity (§3 of S008). We need this at every hidden layer.
The activation at the OUTPUT depends on the task:
- Binary classification: sigmoid → output in
(0, 1). - Multi-class classification: softmax over
Koutputs → all in(0, 1), summing to 1. - Regression (unbounded): no activation (raw real number).
- Bounded regression: tanh (bounded
(-1, 1)) or sigmoid. - Count / rate regression: softplus (
log(1+e^x)) or exponential.
The final activation is matched to the loss: sigmoid + BCE (S007), softmax + CE (S005), identity + MSE. That match is what gives you the clean ∂L/∂z = ŷ − y gradient we saw twice already.
Common mistake: don't put ReLU on the final output for classification. ŷ = ReLU(y_pre) caps outputs at 0 for negative y_pre, so every "no" prediction becomes identical (all exactly 0), gradients through them die, loss freezes. The fix is one word — replace ReLU with sigmoid — and the difference between converging in 200 steps and not converging in 200,000.
8 · Parameter counting — and why it matters for GPT
The general MLP has architecture d_0 → d_1 → d_2 → ... → d_L. Parameter count:
Worked examples:
2-2-1: (2·2 + 2·1) + (2 + 1) = 6 + 3 = 9
784-128-10 (MNIST): (784·128 + 128·10) + (128 + 10) ≈ 101,770
784-512-512-10: (784·512 + 512·512 + 512·10) + (512+512+10) ≈ 670,530Transformer FFN — GPT-2 Small (d_model = 768, d_ff = 3072):
Rule of thumb: about 60% of a modern LLM's parameters live in FFN layers, which are exactly MLPs. When you understand today's forward pass, you understand where the majority of every LLM's parameters go and what they compute. This is not a small statement.
For SwiGLU FFNs (LLaMA-family, S008 §5), there are three weight matrices per block instead of two, and d_ff = (8/3) · d_model to keep parameter count matched with the classical GELU FFN.
9 · Code all together — the forward pass to remember
def mlp_forward(X, params):
W1, b1, W2, b2 = params
H_pre = X @ W1 + b1
H = np.maximum(0, H_pre)
Y_pre = H @ W2 + b2
Y = 1 / (1 + np.exp(-Y_pre))
# cache intermediates for backprop (Session 010)
cache = (X, H_pre, H, Y_pre, Y)
return Y, cacheTen lines. Return the intermediates because we'll need them for the backward pass in Session 010. This is exactly what PyTorch's autograd does under the hood — it caches every intermediate for use in .backward(). You are, once again, writing by hand what a production framework does automatically. The frameworks feel like magic until you have done this once.
10 · The 2025 twist — the same MLP inside a Mamba block
An interesting recent (2023–2025) architectural shift: some LLMs — Mamba (Gu & Dao, 2023), Jamba (AI21, 2024), RWKV-7 (2024), Griffin / RecurrentGemma (Google DeepMind, 2024) — are replacing the attention block with state-space or recurrent mechanisms. But every one of them keeps the FFN/MLP block unchanged. The picture is:
The MLP is the one component nobody has managed to improve on for large-scale next-token prediction. Even in the wildest 2025 architectures — MoE MLPs (Mixtral, DeepSeek-V3), gated MLPs (SwiGLU) — the two-layer-with-activation structure you're building today is the load-bearing element.
Further reading:
- Rumelhart, Hinton, Williams, "Learning representations by back-propagating errors" (1986) — the original paper, still readable — https://www.nature.com/articles/323533a0
- Cybenko, "Approximation by superpositions of a sigmoidal function" (1989) — the universal approximation theorem.
- Hornik, "Approximation capabilities of multilayer feedforward networks" (1991) — generalisation to arbitrary bounded nonlinearities.
- Gu & Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" (2023) — https://arxiv.org/abs/2312.00752 — note MLP is still there.
11 · War stories
Wrote b1 = np.array([[0.1], [0.1]]) (shape (2,1)) instead of b1 = np.array([0.1, 0.1]) (shape (2,)). When I did X @ W1 + b1 with X: (4, 2), NumPy broadcasting produced (4, 2) + (2, 1) → (4, 2) — but the bias was applied WRONG: the two entries of b1 got broadcast down columns instead of rows.
Fix: always keep bias 1D with shape (d_out,). Broadcasting (N, d_out) + (d_out,) is unambiguous. Add assert b1.shape == (d_hidden,) immediately after init. Refer back to S003 §5.
Initial weights W1 were random, but I had accidentally initialised b1 = -5 (copy-paste from a different init function). Every x @ W1 + b1 came out negative for every input. Every ReLU output was 0. Every downstream gradient was 0. Loss didn't move.
Fix: bias inits for ReLU networks should be 0 or slightly positive. Session 022 covers He init in detail; the short version is W ~ N(0, sqrt(2/fan_in)), b = 0.
Refactoring a small MLP, I ended up with W1_row_major in one file and W1_col_major in another (a colleague thought transposed weights were "the same"). Every so often I'd get plausible-looking outputs but training was terrible.
Fix: commit to ONE convention (row-batch for us) and put assert W1.shape == (d_in, d_hidden) at the top of every model init. Shape asserts are your friend. If a shape assert fails at init, you catch the bug in 0 seconds; if it fails during forward, you catch it in 30; if it never fails, congratulations, you have a stealth bug for the next month.
Two-layer MLP on a small dataset. Training accuracy 100%, test accuracy 50% (random). Turned out I was passing X[:batch_size] instead of X[shuffled_indices] — the network was learning the order of examples, not the pattern. Fix: np.random.shuffle(indices) at the top of every epoch. This is such a common bug that PyTorch's DataLoader has shuffle=True as an explicit argument you have to remember to set. Now you know why.
12 · Try it yourself
Take the 2-3-1 architecture from §6 with random init and gradient descent. Use BCE loss and the gradient formulas from S007 (extended to two layers — spoiler for S010). It should converge to ~100% accuracy in <300 steps. Print the final W1, b1, W2, b2 and see how they differ from my hand-picked values in §6.
Rewrite mlp_forward to take a list of weight matrices and biases and an activation function. Signature: mlp_forward(X, Ws, bs, activation, output_activation). Verify it produces the same output for a 3-layer MLP whether you feed it the layers one at a time or all at once.
Given d_model = 12288, d_ff = 4·d_model = 49152, n_layers = 96, ignore attention and just compute total FFN parameter count. You should get ≈ 116 billion. (About two-thirds of GPT-3's total — matching the "60% MLP" rule of thumb.)
Take the trained XOR network. For a fine grid of x values in [-0.2, 1.2]², compute h = ReLU(x @ W1 + b1). Scatter-plot the h values (using h₁ and h₂ as coordinates, ignoring the third dim, or a 2D PCA). You should see the four XOR points now arranged so that a single line separates them. This is Olah's fold, made concrete.
13 · Recap
You hand-computed the forward pass of a 2-2-1 MLP on nine parameters, verified in NumPy to machine precision, then batched to N inputs, then hand-constructed a 2-3-1 XOR solver to see — geometrically — why layers matter. You counted parameters up to GPT-2 Small and saw the same MLP structure inside every transformer block. You met the four canonical output-activation choices and the war stories that come from mixing them up. Session 010 turns all of this around and derives the backward pass — the same nine parameters, in reverse.
"More layers means more capacity. A 3-layer net is strictly more powerful than a 2-layer one, so if my model underfits I should stack another layer."
Only if there is a nonlinearity between them. Stack two linear layers and W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂) — a single linear map. You spent parameters and gained exactly nothing expressively; you can only ever represent what one layer of the smaller of the two ranks could. Depth buys capacity through the composition of nonlinearities, not through the count of matrices.
Because in code the two look identical. Linear → Linear and Linear → ReLU → Linear differ by one line, both train, both reduce the loss, and both produce a model that appears to work. The parameter count goes up in both cases, and parameter count is the proxy everyone reaches for when reasoning about capacity. The collapse is invisible unless you specifically test whether the composite is still linear — and by the time you notice, you have attributed the plateau to data or learning rate.
Collapse two linear layers into one and get identical outputs:
import numpy as np
rng = np.random.default_rng(0)
W1, b1 = rng.standard_normal((4, 8)), rng.standard_normal(8)
W2, b2 = rng.standard_normal((8, 3)), rng.standard_normal(3)
x = rng.standard_normal((5, 4))
deep = (x @ W1 + b1) @ W2 + b2
flat = x @ (W1 @ W2) + (b1 @ W2 + b2) # ONE equivalent layer
print(np.abs(deep - flat).max()) # ~1e-15 : identical
# with a nonlinearity, no such collapse exists
h = np.maximum(0, x @ W1 + b1) @ W2 + b2
print(np.abs(h - flat).max()) # largeWhy can a 2-3-1 MLP solve XOR when no single neuron can, and why is two hidden units the minimum rather than three? The architecture looks like a magic number — it is forced by geometry.
- 1A hidden unit computes
ReLU(w·x + b): it is zero on one side of a hyperplane and rises linearly on the other. So one unit contributes exactly one fold in the input space.forced by · the ReLU kink sits precisely on the hyperplane, and the map is affine on each side of it - 2The output layer forms a weighted sum of these folds. It can add, subtract, and scale them, but it cannot introduce a fold of its own.forced by · the output layer is affine, and an affine map of a piecewise-linear function adds no new pieces
- 3Therefore the network's decision boundary is a level set of a sum of folds — piecewise linear, with pieces delimited only by the hidden units' hyperplanes.forced by · the number of distinct linear regions is bounded by the arrangement of the hidden hyperplanes
- 4XOR's positive class sits at
(0,1)and(1,0), with negatives at(0,0)and(1,1)— the positives are separated from the negatives by a region bounded on two sides. A single hyperplane leaves one point misclassified no matter how you place it.forced by · one hyperplane splits the plane into two half-spaces, and no half-space contains exactly the XOR positives - 5Two hyperplanes suffice: place them parallel, one just above the "both off" corner and one just below the "both on" corner, then have the output take a difference so only the strip between them fires.forced by · subtracting one fold from another produces a bounded band, which is exactly the region containing (0,1) and (1,0)
Therefore the minimum is two hidden units, and the classic 2-3-1 has one spare — capacity for the optimizer to find the solution more easily, not capacity the problem requires.
And note the prediction, which generalises far beyond XOR: with ReLU, the network is exactly piecewise linear, and the number of linear pieces is bounded by the arrangement of the hidden hyperplanes. So a wider layer buys more folds additively while more layers compose folds multiplicatively — that asymmetry is the real argument for depth over width. Verify the concrete version: train a 2-2-1 ReLU net on XOR from many seeds. It should reach zero loss sometimes and get stuck at 0.25 accuracy-loss other times, because with exactly the minimum number of folds there is no slack for a bad initialisation. Bump to 2-3-1 and the failure rate should drop sharply.
Imagine the input space as a sheet of paper on which the classes are printed in a pattern no straight cut can separate. Each hidden unit is a fold along a line of its choosing. Fold enough times, in the right places, and the previously-tangled points end up stacked on the same side.
The output layer is then just one straight cut through the folded stack. It has no cleverness at all — all the work happened in the folding. That is what "learned representation" means concretely: the hidden layer is not extracting features in a mystical sense, it is rearranging the space until the last layer's single hyperplane is enough.
- Hidden layer = fold the space. Output layer = one straight cut. Without a nonlinearity there is no fold, and the whole stack collapses to one cut.
- Wider layer ⇒ more folds at the same depth (additive). Deeper ⇒ folds of folds (compositional). This is the depth-vs-width tradeoff in one sentence.
- Every shape in the forward pass is
(batch, features):x @ W1 + b1whereW1is(in, hidden). Batch rides along axis 0 and never participates in the matmul contraction. - The bias moves the fold line off the origin. Drop biases and every fold must pass through the origin — a real and often fatal restriction.
Fire this model the moment you see: two Linear layers with nothing between them · a model that will not beat logistic regression no matter the size · a debate about depth versus width · "the network learns features" · a decision boundary plot that is visibly piecewise linear · someone removing bias terms to save parameters.
You have a fixed parameter budget for an MLP. Spend it on width (one big hidden layer) or on depth (several narrow ones)?
For a plain MLP on tabular data, start wide and shallow — usually one or two hidden layers. The compositional argument for depth is strong for images, audio, and text, and weak for a feature vector where the inputs are already semantic. Depth on tabular data buys optimisation difficulty far more reliably than it buys accuracy.
The empirical test is cheap: hold parameters roughly constant and train a 1-layer, 2-layer, and 4-layer variant. If accuracy is flat across all three, your problem does not have depth-shaped structure, and you should stop tuning architecture and go get more data or better features instead.
🧠 Retention scaffold
One-line summary (write it in your own words): _______________________________________________________________
Spaced review: re-derive the §4 hand-trace in 24 hours; revisit §6 (XOR by construction) on day 7 — it will make the S010 backprop much easier.
Next session (S010): the backward pass on the exact same 2-2-1 network — on paper, with numbers, so you can see every gradient tensor materialise. Once you can do backprop by hand on 9 parameters, doing it on 9 billion is just linear algebra at scale.
Sticky note (keep on your desk):
- Diagram: the 2-2-1 network with
W₁, b₁, W₂, b₂labelled and shapes on the arrows. - Four-line forward:
H_pre = X @ W1 + b1; H = ReLU(H_pre); Y_pre = H @ W2 + b2; Y = σ(Y_pre). Memorise. - Cache tuple:
(X, H_pre, H, Y_pre, Y). Backprop needs it. - 60% rule: most LLM params live in FFN blocks — which are MLPs.
Previous: ← DL S008 · Activations · Next: DL S010 · Backprop by Hand →