S097 · Multi-Layer Perceptron — Forward Pass

Module M12: Deep Learning · Session 97 of 130 · Track: ML ➡️

What you'll be able to do after this session

• Explain Multi-Layer Perceptron to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S096 — Perceptron & Activation Functions

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · 3Blue1Brown: Gradient descent, how neural networks learn (Ch. 2) — sets up how a full MLP maps inputs to outputs.
• 🎥 Deep dive (30–60 min) · Andrej Karpathy: Building makemore (MLP) — 1h+ deep dive on building an MLP from scratch.
• 🎥 Hands-on demo (10–20 min) · sentdex: Neural Networks from Scratch — Forward Pass — codes a 3-layer forward pass with pure NumPy.
• 📖 Canonical article · Michael Nielsen: Chapter 1 (continues into stacked networks) — same free book chapter you started in S096.
• 📖 Book chapter / tutorial · d2l.ai: Multilayer Perceptrons — Dive Into Deep Learning; worked out with code.
• 📖 Engineering blog · Google Developers: Introduction to Neural Networks — clean visual walkthrough of layer composition.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Stacking neurons into a network.

A perceptron is a single decision. A multi-layer perceptron (MLP) is a stack of decisions where the output of one layer becomes the input of the next. The magic is that intermediate ("hidden") layers learn to extract features that the final layer can classify easily.

Think of a factory assembly line. Raw materials (pixel values, tokens, whatever) come in at the input. Each hidden layer refines them: the first layer might detect edges, the second combines edges into shapes, the third combines shapes into objects, the final layer says "this is a cat." Nobody hand-designed those intermediate features — the network invented them because they were useful for the final output. That is representation learning in three sentences, and it is why deep learning replaced 30 years of hand-crafted features.

The forward pass is the calculation you do to get from input to output given fixed weights. Left to right, layer by layer: matrix-multiply, add bias, apply activation, pass to next layer. Nothing more. Training changes the weights (via backpropagation — next session); inference is just a forward pass.

Gotcha to remember forever: everything is a matrix multiply. If you can track the shapes (batch, features) @ (features, next_layer_size) → (batch, next_layer_size) at every step, you can debug 90% of deep-learning errors. When PyTorch screams RuntimeError: mat1 and mat2 shapes cannot be multiplied, that's just you failing to write down the shapes on paper.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

A 3-layer MLP for MNIST (784 → 128 → 64 → 10)

Shape tracking (the only thing that matters)

Assume batch size = 32.

Total trainable parameters: 784*128 + 128 + 128*64 + 64 + 64*10 + 10 = 109,386. Tiny by today's standards (GPT-4 has ~1.8 trillion). But it can already reach ~98% on MNIST.

Worked example: forward pass by hand on a 2-2-1 network

Input x = [1, 0]. Layer 1 weights W1 = [[0.5, -0.3], [0.2, 0.8]], bias b1 = [0.1, -0.2]. Layer 2 weights W2 = [1.0, -1.0], bias b2 = 0.3. Activations: ReLU on hidden, sigmoid on output.

• Layer 1 pre-activation: z1 = x @ W1 + b1 = [0.5, -0.3] + [0.1, -0.2] = [0.6, -0.5]
• Layer 1 activation: h1 = ReLU(z1) = [0.6, 0.0] (the −0.5 dies)
• Layer 2 pre-activation: z2 = h1 @ W2 + b2 = 0.6*1.0 + 0.0*(-1.0) + 0.3 = 0.9
• Layer 2 activation: y = sigmoid(0.9) ≈ 0.711

The output is one number between 0 and 1 — the probability that this input belongs to the positive class. That's the entire forward pass.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# Pure NumPy first — proves there is no magic.
import numpy as np
 
rng = np.random.default_rng(42)
def relu(x): return np.maximum(0, x)
def softmax(x):
    e = np.exp(x - x.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)
 
n_in, n_h1, n_h2, n_out = 4, 8, 6, 3
W1 = rng.normal(0, np.sqrt(2/n_in), (n_in, n_h1));   b1 = np.zeros(n_h1)
W2 = rng.normal(0, np.sqrt(2/n_h1), (n_h1, n_h2));   b2 = np.zeros(n_h2)
W3 = rng.normal(0, np.sqrt(2/n_h2), (n_h2, n_out));  b3 = np.zeros(n_out)
 
X = rng.normal(0, 1, (5, n_in))
print("Input shape:", X.shape)
 
z1 = X @ W1 + b1;  h1 = relu(z1);   print("Layer 1 out:", h1.shape)
z2 = h1 @ W2 + b2; h2 = relu(z2);   print("Layer 2 out:", h2.shape)
z3 = h2 @ W3 + b3; probs = softmax(z3)
print("Output logits:", z3.shape)
print("Softmax probs sum:", probs.sum(axis=1))
print("Predicted classes:", probs.argmax(axis=1))
 
# Same network in PyTorch — 6 lines
import torch, torch.nn as nn
class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_in,  n_h1), nn.ReLU(),
            nn.Linear(n_h1, n_h2), nn.ReLU(),
            nn.Linear(n_h2, n_out),
        )
    def forward(self, x): return self.net(x)
 
net = MLP()
x = torch.tensor(X, dtype=torch.float32)
logits = net(x)
print("\nPyTorch output shape:", logits.shape)
print("Num params:", sum(p.numel() for p in net.parameters()))

What to observe when you run it:

1. Every intermediate shape follows (batch, in) @ (in, out) = (batch, out) — write this on a sticky note.
2. softmax rows sum to exactly 1.0 (or 0.9999… for floating-point).
3. The random-init model's predictions are essentially uniform — as expected before training.
4. He initialisation N(0, sqrt(2/fan_in)) keeps activation magnitudes stable across layers. Try N(0, 1) and print h1.std() vs h2.std() — you'll see them explode or vanish.
5. The PyTorch version is literally a wrapper around the same math — print(net.net[0].weight.shape) gives (8, 4) (PyTorch stores out × in, opposite to our NumPy convention).

Try this modification: replace all nn.ReLU() with nn.Identity() (no activation). Feed the same input. Verify that the entire 3-layer network collapses to a single equivalent linear transformation — you can reproduce it with a single nn.Linear(n_in, n_out). That's why non-linearities matter.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story #1: the shape mismatch that shipped. A team at a large ad-tech company reshaped a batch tensor from (N, T, F) to (N*T, F) for a per-timestep classifier. In one branch they forgot to reshape back before averaging over T — the model happily trained on nonsense-averaged logits for months, hitting a suspicious plateau. Shape assertions (assert x.shape == (batch, 10)) at every model-external boundary would have caught it in an hour.

War story #2: batch normalisation in the wrong mode. MLPs with batchnorm behave differently in train() and eval() modes (batchnorm uses batch stats in train, running stats in eval). Teams have shipped models still in train() mode, causing inference to depend on batch size — including per-batch identity leak in a legal-document classifier. Rule: always model.eval() before serving, always model.train() before optimising.

Common bugs:

• Forgetting to flatten. nn.Linear(784, 128) expects (batch, 784); giving it (batch, 28, 28) crashes. Use nn.Flatten() or x.view(x.size(0), -1).
• Wrong batch dim. PyTorch expects (batch, features); TF used to prefer channels-last for some ops.
• Softmax + nn.CrossEntropyLoss. CrossEntropyLoss applies log-softmax internally — pre-softmaxing causes double softmax and wildly wrong gradients.
• No bias in the final layer, then wondering why the model can't fit the target mean. Include the bias unless you have a strong reason.

How top teams handle it:

• Every serious codebase logs input/output shapes at every layer during a warmup pass (torchinfo.summary(model, input_size)), and adds assert statements at API boundaries.
• Karpathy's rule: "the forward pass should be traceable in your head; if it isn't, write it down until it is."
• Modern architectures (transformers, CNNs) are still carefully-shaped forward passes with matrix multiplies + non-linearities + normalisations. Once MLPs make sense, everything else is variations on the theme.

(e) Quiz + exercise · 10 min

1. In a 3-layer MLP with input dim 100, hidden dims 64 and 32, and 10 output classes, how many trainable parameters does the network have (including biases)?
2. Explain in one sentence what a "forward pass" does, and why it's the same math at training time and inference time.
3. If you remove all non-linear activations from a 5-layer MLP, what simpler model is it equivalent to and why?
4. Given a batch of shape (32, 784) passing through nn.Linear(784, 128), what is the output shape? What is the shape of the weight matrix?
5. Why does softmax always produce outputs that sum to 1, and where in a classifier network is it applied?

Stretch (connects to S096 — perceptron): A single perceptron can't learn XOR. Show, on paper, a 2-hidden-unit MLP with hand-picked weights and biases (using ReLU activation) that does correctly implement XOR on inputs (0,0), (0,1), (1,0), (1,1). Verify the forward pass for each input.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Multi-Layer Perceptron? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S098): Backpropagation — Derived by Hand on a 2-Layer Net
• Previous (S096): Perceptron & Activation Functions
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.