DL S009 · The Forward Pass for a 2-Layer MLP

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 9 / 80 · Module M02 · ~2 hours

0 · The story — the paper that ended the AI winter

October 1986. Nature publishes an 8-page letter by David Rumelhart, Geoffrey Hinton, and Ronald Williams: "Learning representations by back-propagating errors." Its opening figure is a network of exactly the size you're about to build today — two inputs, two hidden units, one output — solving XOR. The problem Minsky and Papert had proved impossible for a single perceptron seventeen years earlier (S007 §8) falls in a few hundred training steps. The paper is short. Its citation count as of 2025 is over 50,000. Every deep network that has ever been trained descends from the algorithm in that letter.

The trick was not new — Werbos had derived it in his 1974 PhD thesis; LeCun had used it in 1985; the Soviet mathematician Alexey Ivakhnenko had similar ideas in 1965. What Rumelhart–Hinton–Williams did was make it work in practice, on real problems, with a clear demonstration and a clean derivation. The demonstration was XOR. The mechanism was the multi-layer perceptron. The training algorithm — backpropagation — is Session 010's problem. Today you build the forward half of the 1986 network.

Why does it matter that you can compute it by hand? Because every transformer block in every LLM you have ever prompted contains an MLP exactly the shape of what you're about to build — just with d_model = 4096 instead of 2, d_ff = 16384 instead of 2, and no output sigmoid. When you read nanoGPT in S040 you'll see the two lines:

h = self.c_fc(x)           # (B, T, d_model) @ (d_model, d_ff)
h = new_gelu(h)            # activation
y = self.c_proj(h)         # (B, T, d_ff) @ (d_ff, d_model)

and you will recognise them as the same forward pass you're about to hand-trace. This is not a metaphor. It is literally the same math.

1 · What you'll build today

Compute the forward pass of a 2-2-1 MLP by hand, then in NumPy, then batch it. Then hand-construct a 2-3-1 network that solves XOR without any training.

2 · Checkpoint · you should already know

• S002 · Vectors, matrices, geometry — matmul and shape rules.
• S003 · Broadcasting — bias addition uses broadcasting across the batch axis.
• S007 · A single artificial neuron — one neuron. Today we build many, arranged in layers.
• S008 · Activations — we'll use ReLU as the hidden activation and sigmoid at the output for binary classification.

3 · The 2-2-1 architecture — draw it once, forever

Parameter count: W₁ has 4, b₁ has 2, W₂ has 2, b₂ has 1. Total: 9 parameters. Small enough to write on a napkin. Structurally identical (up to bigger dimensions and different activations) to a 1024-1024-10 MNIST MLP or to the FFN inside any transformer block.

3.1 The forward equations (row-batch convention)

h_pre = x @ W₁ + b₁        shape (2,)         — pre-activation of hidden layer
h     = ReLU(h_pre)        shape (2,)         — post-activation of hidden layer
ŷ_pre = h @ W₂ + b₂        shape (1,)         — pre-activation of output
ŷ     = σ(ŷ_pre)           shape (1,)         — output probability

Read those four lines out loud once. That is a neural network. Everything else in this session is either arithmetic on those four lines or a variant of them.

4 · One forward pass — by hand, on numbers you can hold in your head

Pick these values:

x  = [1.0, 2.0]
W₁ = [[0.1, 0.2],           b₁ = [0.1, 0.1]
      [0.3, 0.4]]
W₂ = [[0.5],                b₂ = [0.2]
      [0.6]]

4.1 Layer 1

Using h_pre = x @ W₁ + b₁:

4.2 Layer 2

h @ W₂ = [0.8, 1.1] @ [[0.5], [0.6]]
       = [ 0.8·0.5 + 1.1·0.6 ]
       = [ 0.4 + 0.66        ]
       = [ 1.06              ]
 
ŷ_pre = [1.06] + [0.2] = [1.26]
ŷ     = σ(1.26) = 1 / (1 + e^{-1.26}) ≈ 0.7788

Output: 0.78. The model gives about 78% probability to class 1 for this input.

4.3 Verify in NumPy

import numpy as np
x  = np.array([1.0, 2.0])
W1 = np.array([[0.1, 0.2], [0.3, 0.4]])
b1 = np.array([0.1, 0.1])
W2 = np.array([[0.5], [0.6]])
b2 = np.array([0.2])
 
h_pre = x @ W1 + b1
h     = np.maximum(0, h_pre)
y_pre = h @ W2 + b2
y     = 1 / (1 + np.exp(-y_pre))
 
print("h_pre:", h_pre)   # [0.8 1.1]
print("h    :", h)       # [0.8 1.1]
print("y_pre:", y_pre)   # [1.26]
print("y    :", y)       # [0.77894181...]

Numbers match to machine precision. ✅ You just executed a neural network on paper. If your hand answer disagrees with the NumPy answer, do not proceed — one of your dot products has a sign flipped. Find it before moving on. The number of engineers whose next six months of debugging would have been shorter had they done this exercise once is uncountable.

5 · Batched forward pass — the shape you'll actually use

Real training uses batches. For N inputs at once:

X ∈ (N, 2)   H_pre = X @ W₁ + b₁ ∈ (N, 2)   H = ReLU(H_pre) ∈ (N, 2)
             Y_pre = H @ W₂ + b₂ ∈ (N, 1)   Ŷ = σ(Y_pre)    ∈ (N, 1)

Bias b₁: (2,) broadcasts across the batch axis by NumPy's rules (S003 §5): the shape (N, 2) + (2,) is legal and adds b₁ to every row of the (N, 2) matrix. Same forward code, just with a 2D input:

X = np.array([[ 1.0,  2.0],
              [ 0.5,  0.5],
              [-1.0,  0.3],
              [ 2.0, -1.5]])          # shape (4, 2), a batch of 4 examples
 
H_pre = X @ W1 + b1                    # shape (4, 2)
H     = np.maximum(0, H_pre)           # shape (4, 2)
Y_pre = H @ W2 + b2                    # shape (4, 1)
Y     = 1 / (1 + np.exp(-Y_pre))       # shape (4, 1)

Same ten lines, N inputs at a time. This is the entire forward pass of a 2-layer MLP. Change the shapes to (1024, 784), (784, 256), (256, 10) and you have an MNIST classifier (S011). Change them to (1, 2048, 4096), (4096, 16384), (16384, 4096) and you have the FFN block of GPT-3 (S040).

5.1 The three inputs traced through

Take row 2: x = [0.5, 0.5].

h_pre = [0.5, 0.5] @ [[0.1, 0.2], [0.3, 0.4]] + [0.1, 0.1]
      = [0.5·0.1 + 0.5·0.3,  0.5·0.2 + 0.5·0.4] + [0.1, 0.1]
      = [0.20, 0.30] + [0.1, 0.1] = [0.30, 0.40]
h     = [0.30, 0.40]
y_pre = [0.30·0.5 + 0.40·0.6] + [0.2] = [0.15 + 0.24 + 0.2] = [0.59]
y     = σ(0.59) ≈ 0.6434

Take row 3: x = [-1.0, 0.3].

h_pre = [-1.0·0.1 + 0.3·0.3,  -1.0·0.2 + 0.3·0.4] + [0.1, 0.1]
      = [-0.01, -0.08] + [0.1, 0.1] = [0.09, 0.02]
h     = [0.09, 0.02]     (ReLU: both still positive)
y_pre = [0.09·0.5 + 0.02·0.6] + [0.2] = [0.057 + 0.2] = [0.257]
y     = σ(0.257) ≈ 0.5639

Notice h_pre came out just barely positive. A tiny nudge to W₁ — a gradient step during training — could easily push those hidden neurons negative and zero them out. Session 010's backward pass will make this failure mode concrete.

6 · Solving XOR by hand — the payoff of stacking

Recall S007 §8: no single neuron can solve XOR. Here's a 2-3-1 MLP that solves it without any training. I will tell you the weights and let you verify.

X = [[0,0], [0,1], [1,0], [1,1]],  y = [0, 1, 1, 0]
 
W1 = [[ 1,  1,  1],       b1 = [0.5, -0.5, -1.5]
      [ 1,  1,  1]]
W2 = [[ 1],               b2 = [-0.5]
      [-2],
      [ 1]]

Trace x = [0, 1]:

That's on the boundary. Try [1, 0]:

Both [0,1] and [1,0] give h = [1.5, 0.5, 0] because W₁ is symmetric — w·x only depends on x₁ + x₂. Great: the hidden layer has learned the feature x₁ + x₂. Now try [0,0]:

And [1,1]:

h_pre = [1, 1]·W1 + b1 = [2, 2, 2] + [0.5, -0.5, -1.5] = [2.5, 1.5, 0.5]
h     = [2.5, 1.5, 0.5]
y_pre = [2.5·1 + 1.5·(-2) + 0.5·1] - 0.5 = [2.5 - 3.0 + 0.5] - 0.5 = 0 - 0.5 = -0.5
y     = σ(-0.5) ≈ 0.378

Not perfect — the numbers I chose are illustrative rather than optimal — but the point stands: the hidden layer produced three "feature detectors" (roughly: x₁+x₂ > -0.5, x₁+x₂ > 0.5, x₁+x₂ > 1.5) that the output layer combines to distinguish "exactly one of x₁, x₂ is 1" from the other cases. A trained network converges to sharper weights and gets 100% accuracy in a few dozen SGD steps (do this yourself in the Try It below).

The geometric picture: layer 1 bends 2D input space into a 3D feature space where the XOR classes ARE linearly separable, then the output neuron draws the separating line in that bent space. This is exactly Chris Olah's manifold-deformation argument from the pre-read.

7 · Why ReLU between layers, but not at the very end

The activation between layers gives the network nonlinearity (§3 of S008). We need this at every hidden layer.

The activation at the OUTPUT depends on the task:

• Binary classification: sigmoid → output in (0, 1).
• Multi-class classification: softmax over K outputs → all in (0, 1), summing to 1.
• Regression (unbounded): no activation (raw real number).
• Bounded regression: tanh (bounded (-1, 1)) or sigmoid.
• Count / rate regression: softplus (log(1+e^x)) or exponential.

The final activation is matched to the loss: sigmoid + BCE (S007), softmax + CE (S005), identity + MSE. That match is what gives you the clean ∂L/∂z = ŷ − y gradient we saw twice already.

Common mistake: don't put ReLU on the final output for classification. ŷ = ReLU(y_pre) caps outputs at 0 for negative y_pre, so every "no" prediction becomes identical (all exactly 0), gradients through them die, loss freezes. The fix is one word — replace ReLU with sigmoid — and the difference between converging in 200 steps and not converging in 200,000.

8 · Parameter counting — and why it matters for GPT

The general MLP has architecture d_0 → d_1 → d_2 → ... → d_L. Parameter count:

Worked examples:

2-2-1:                (2·2 + 2·1) + (2 + 1)                    = 6 + 3      = 9
784-128-10 (MNIST):   (784·128 + 128·10) + (128 + 10)          ≈ 101,770
784-512-512-10:       (784·512 + 512·512 + 512·10) + (512+512+10) ≈ 670,530

Transformer FFN — GPT-2 Small (d_model = 768, d_ff = 3072):

Rule of thumb: about 60% of a modern LLM's parameters live in FFN layers, which are exactly MLPs. When you understand today's forward pass, you understand where the majority of every LLM's parameters go and what they compute. This is not a small statement.

For SwiGLU FFNs (LLaMA-family, S008 §5), there are three weight matrices per block instead of two, and d_ff = (8/3) · d_model to keep parameter count matched with the classical GELU FFN.

9 · Code all together — the forward pass to remember

def mlp_forward(X, params):
    W1, b1, W2, b2 = params
    H_pre = X @ W1 + b1
    H     = np.maximum(0, H_pre)
    Y_pre = H @ W2 + b2
    Y     = 1 / (1 + np.exp(-Y_pre))
    # cache intermediates for backprop (Session 010)
    cache = (X, H_pre, H, Y_pre, Y)
    return Y, cache

Ten lines. Return the intermediates because we'll need them for the backward pass in Session 010. This is exactly what PyTorch's autograd does under the hood — it caches every intermediate for use in .backward(). You are, once again, writing by hand what a production framework does automatically. The frameworks feel like magic until you have done this once.

10 · The 2025 twist — the same MLP inside a Mamba block

An interesting recent (2023–2025) architectural shift: some LLMs — Mamba (Gu & Dao, 2023), Jamba (AI21, 2024), RWKV-7 (2024), Griffin / RecurrentGemma (Google DeepMind, 2024) — are replacing the attention block with state-space or recurrent mechanisms. But every one of them keeps the FFN/MLP block unchanged. The picture is:

The MLP is the one component nobody has managed to improve on for large-scale next-token prediction. Even in the wildest 2025 architectures — MoE MLPs (Mixtral, DeepSeek-V3), gated MLPs (SwiGLU) — the two-layer-with-activation structure you're building today is the load-bearing element.

Further reading:

• Rumelhart, Hinton, Williams, "Learning representations by back-propagating errors" (1986) — the original paper, still readable — https://www.nature.com/articles/323533a0
• Cybenko, "Approximation by superpositions of a sigmoidal function" (1989) — the universal approximation theorem.
• Hornik, "Approximation capabilities of multilayer feedforward networks" (1991) — generalisation to arbitrary bounded nonlinearities.
• Gu & Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" (2023) — https://arxiv.org/abs/2312.00752 — note MLP is still there.

11 · War stories

12 · Try it yourself

13 · Recap

You hand-computed the forward pass of a 2-2-1 MLP on nine parameters, verified in NumPy to machine precision, then batched to N inputs, then hand-constructed a 2-3-1 XOR solver to see — geometrically — why layers matter. You counted parameters up to GPT-2 Small and saw the same MLP structure inside every transformer block. You met the four canonical output-activation choices and the war stories that come from mixing them up. Session 010 turns all of this around and derives the backward pass — the same nine parameters, in reverse.

🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________________________________________

Spaced review: re-derive the §4 hand-trace in 24 hours; revisit §6 (XOR by construction) on day 7 — it will make the S010 backprop much easier.

Next session (S010): the backward pass on the exact same 2-2-1 network — on paper, with numbers, so you can see every gradient tensor materialise. Once you can do backprop by hand on 9 parameters, doing it on 9 billion is just linear algebra at scale.

Sticky note (keep on your desk):

• Diagram: the 2-2-1 network with W₁, b₁, W₂, b₂ labelled and shapes on the arrows.
• Four-line forward: H_pre = X @ W1 + b1; H = ReLU(H_pre); Y_pre = H @ W2 + b2; Y = σ(Y_pre). Memorise.
• Cache tuple: (X, H_pre, H, Y_pre, Y). Backprop needs it.
• 60% rule: most LLM params live in FFN blocks — which are MLPs.

Previous: ← DL S008 · Activations · Next: DL S010 · Backprop by Hand →