S096 · Perceptron & Activation Functions

Module M12: Deep Learning · Session 96 of 130 · Track: ML ⚡

What you'll be able to do after this session

• Explain Perceptron & Activation Functions to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S085 — Linear Regression from Scratch (numpy)

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · 3Blue1Brown: But what is a neural network? (Ch. 1) — the single best video for building intuition about neurons and layers.
• 🎥 Deep dive (30–60 min) · Andrej Karpathy: micrograd — spelled-out intro to neural networks and backprop — first hour builds up from a single neuron; the definitive from-scratch talk.
• 🎥 Hands-on demo (10–20 min) · StatQuest: Neural Networks Pt. 1 — Inside the Black Box — walks through activation functions with clean visuals.
• 📖 Canonical article · Michael Nielsen: Neural Networks and Deep Learning — Chapter 1 — the free chapter that made deep learning click for a generation.
• 📖 Book chapter / tutorial · Deep Learning Book (Goodfellow) — Chapter 6: Deep Feedforward Networks — definitive reference on activations and why ReLU won.
• 📖 Engineering blog · Distill.pub: Feature Visualization — see what individual neurons actually learn to detect.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The neuron, first principles.

A perceptron is a single artificial neuron, and it is embarrassingly simple. You have inputs (say, pixel values), you multiply each by a weight, add a bias, and pass the sum through a squishing function. That's it. Do this once — perceptron. Stack thousands in layers — GPT-4. Everything else in deep learning is really a variation on this one theme.

The analogy that sticks: a perceptron is a bouncer at a club with a checklist. Age? Weight this heavily (multiply by +0.9). Wearing a suit? Weight moderately (+0.4). Empty bottle in hand? Weight strongly negatively (−1.5). Add them up. If the total crosses a threshold, let them in (output 1); otherwise, no (output 0). Learning is adjusting those weights based on which decisions turned out right or wrong.

The activation function is what makes a stacked network more powerful than a single layer. Without activations, stacking N linear layers is mathematically equivalent to one linear layer (matrix multiplication composes). Activations bend the output — ReLU sets negatives to zero, sigmoid squashes to 0–1, tanh to −1..1. That non-linearity is what lets a deep network learn curves, corners, and concepts.

Gotcha to remember forever: without a non-linear activation between layers, your 100-layer neural net is still a linear model. Every trick in deep learning downstream of this — CNNs, transformers, everything — requires a non-linear activation somewhere. ReLU is the default; use it unless you have a reason not to.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

A perceptron in one picture

Math: y = sigma(w1*x1 + w2*x2 + w3*x3 + b) where sigma is your activation.

The activation zoo

Why ReLU won

Sigmoid and tanh both saturate — for very large or small inputs, their derivative is nearly zero, so gradients vanish and training grinds to a halt. ReLU's derivative is 1 for positive inputs, forever. That single property let us train networks with 100+ layers.

Worked example: a 2-input perceptron doing AND

Suppose w = [1, 1], b = −1.5, activation is a step function.

That's a functional AND gate. Change b to −0.5 and you get OR. Change activation and adjust weights and you can compute any linearly-separable function. XOR? A single perceptron cannot — you need a hidden layer with a non-linear activation. That famous 1969 Minsky–Papert result nearly killed neural nets for a decade until multi-layer perceptrons were rediscovered.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# Pure NumPy — no framework — to show there is nothing magical.
import numpy as np
 
def sigmoid(x):  return 1 / (1 + np.exp(-x))
def relu(x):     return np.maximum(0, x)
def tanh(x):     return np.tanh(x)
def leaky_relu(x, a=0.01): return np.where(x > 0, x, a * x)
 
class Perceptron:
    def __init__(self, n_inputs, activation=sigmoid, seed=42):
        rng = np.random.default_rng(seed)
        self.w = rng.normal(0, 1/np.sqrt(n_inputs), n_inputs)
        self.b = 0.0
        self.act = activation
 
    def forward(self, x):
        return self.act(x @ self.w + self.b)
 
    def train_step(self, x, y_true, lr=0.05):
        y_pred = self.forward(x)
        err = y_true - y_pred
        self.w += lr * err * x
        self.b += lr * err
        return err**2
 
# Learn the AND function
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y = np.array([0, 0, 0, 1], dtype=float)
 
p = Perceptron(n_inputs=2, activation=sigmoid)
for epoch in range(200):
    loss = 0
    for xi, yi in zip(X, y):
        loss += p.train_step(xi, yi, lr=0.5)
    if epoch % 40 == 0:
        print(f"epoch {epoch:3d}  loss={loss:.4f}  w={p.w.round(3)}  b={p.b:+.3f}")
 
print("\nLearned AND gate:")
for xi in X:
    print(f"  {xi} -> {p.forward(xi):.3f}")
 
# Vanishing-gradient demo
for name, fn in [("sigmoid", sigmoid), ("tanh", tanh), ("relu", relu), ("leaky", leaky_relu)]:
    print(f"{name:>8}: f(-8)={fn(-8):+.4f}  f(0)={fn(0):+.4f}  f(8)={fn(8):+.4f}")

What to observe when you run it:

1. The perceptron converges on AND in under 200 epochs — weights end up both positive with a negative bias, matching the manual example.
2. Try y = np.array([0, 1, 1, 0]) (XOR) and it will fail to converge — the famous linear-separability limitation.
3. Sigmoid at ±8 is essentially 0 or 1 (saturated). ReLU at −8 is 0 (dead), at +8 is 8 (linear). That's why ReLU trains better in deep networks.
4. w initialised with 1/sqrt(n_inputs) scale — that's Xavier-style init; poor init makes even this trivial problem oscillate.
5. Learning rate matters: try lr=5.0 (diverges) or lr=0.001 (crawls).

Try this modification: swap activation=sigmoid for activation=relu in the Perceptron class and try to learn AND. It works, but with ReLU one weight can get stuck at zero — a preview of why real training uses proper backprop.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story #1: the dying-ReLU epidemic (~2015). Early practitioners initialised weights with N(0, 0.1) and used a large learning rate. Many ReLU neurons received a big negative gradient early in training, drove their pre-activation permanently negative, and output zero forever — gradient through them was zero, so they never recovered. Diagnosing "50% of my neurons are dead" became a rite of passage. Fixes: He initialisation (N(0, sqrt(2/fan_in))), lower learning rate, or Leaky ReLU / ELU / GELU.

War story #2: sigmoid output on multi-class. A team building an image classifier used sigmoid on the final layer for 10 classes, one per class independently, then took argmax. Trained fine, deployed. Weeks later, evaluation revealed the model was systematically overconfident and poorly calibrated. Cause: sigmoid outputs are independent; softmax outputs are competing probabilities that sum to 1. For mutually-exclusive classes, always use softmax + cross-entropy.

Common bugs:

• Applying softmax twice (once in the model, once in the loss). nn.CrossEntropyLoss in PyTorch expects raw logits; pre-softmaxing silently corrupts gradients.
• Sigmoid on hidden layers. Vanishing gradient territory. Except in output layers or LSTMs, don't.
• Not using bias. Bias lets the perceptron shift the decision boundary; without it, all boundaries pass through the origin.
• All-zero initialisation. Every neuron computes the same thing, no symmetry breaking, network cannot learn.

How top teams handle it:

• GPT / BERT / LLaMA all use GELU in feed-forward blocks. Newer models (LLaMA-2/3, Mistral) use SwiGLU — a gated variant with better empirical performance.
• CNNs (ResNet, EfficientNet) still default to plain ReLU — cheap, effective, hardware-friendly.
• PyTorch / TensorFlow default weight inits (Kaiming He for ReLU, Xavier for sigmoid/tanh) make dying-ReLU rare today unless you deliberately override.

(e) Quiz + exercise · 10 min

1. Write down the formula for a single perceptron's output as a function of inputs, weights, bias, and activation.
2. Why is a non-linear activation function essential between layers, even if every layer is fully connected?
3. Give one advantage and one disadvantage of ReLU compared to sigmoid.
4. When would you use softmax vs sigmoid on the output layer of a classifier?
5. What is the "dying ReLU" problem, what causes it, and name two mitigations.

Stretch (connects to S085 — logistic regression): A logistic regression classifier is mathematically equivalent to a single perceptron with a specific activation and a specific loss. Which activation and which loss? Explain why this means logistic regression is the simplest possible neural network.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Perceptron & Activation Functions? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S097): Multi-Layer Perceptron — Forward Pass
• Previous (S095): Model Selection — CV, Hyperparameter Tuning, Optuna
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.