DL S008 · Activation Functions — sigmoid, tanh, ReLU, GELU, SwiGLU, xIELU

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 8 / 80 · Module M02 · ~2.5 hours

0 · The story — the function max(0, x) and the 2012 turning point

For about twenty years, from Rumelhart-Hinton-Williams (1986) through the mid-2000s, the default activation function in every neural network was the sigmoid. It was smooth, it was probabilistic (outputs looked like probabilities), it was elegant. It also — and nobody had a satisfying story for this at the time — quietly refused to let anyone train networks deeper than about three layers. Loss curves plateaued. First-layer weights barely moved. People blamed the data. People blamed the optimiser. People wrote papers about "unsupervised layer-wise pretraining" (Hinton 2006) that added heroic pre-training stages just to get 5-layer networks off the ground.

Then in 2010, Xavier Glorot and Yoshua Bengio published a paper called "Understanding the difficulty of training deep feedforward neural networks." Their diagnosis was surgical: the derivative of the sigmoid is tiny — it maxes out at 0.25 and is essentially zero once the neuron saturates — so in a deep chain the gradient shrinks by a factor of at least 4 per layer, and often 100 or more when neurons saturate. By the time it reaches layer one it's numerically zero. Vanishing gradients. They also proposed a smarter initialisation (Xavier init, which you'll meet in S022) that partially fixed the problem — but only partially.

The real jailbreak came from Toronto. Vinod Nair and Geoffrey Hinton, 2010, and then Krizhevsky-Sutskever-Hinton (AlexNet, 2012) replaced the sigmoid with the most brutal function anyone had seriously proposed: max(0, x). A ramp. A hinge. A thing that isn't even differentiable at zero. The idea was old (biology, Fukushima's Neocognitron 1980) but the field had dismissed it as inelegant. Krizhevsky's ImageNet result — a 10-point top-5 accuracy jump over the state of the art, using an 8-layer ReLU network on two GTX 580s in a Toronto basement — ended the conversation. Overnight, ReLU became the default.

Then came the refinements. Leaky ReLU (Maas 2013) to fix dying units. ELU (Clevert 2015) for smoothness. Swish / SiLU (Ramachandran 2017, discovered by Google's neural architecture search) for smoother-still. GELU (Hendrycks & Gimpel 2016) — "ReLU with the corner rounded by a Gaussian CDF" — which BERT adopted in 2018 and which became the default in every transformer for the next five years. And then, in the LLM era, SwiGLU (Shazeer 2020, then Chinchilla, LLaMA, Mistral, DeepSeek): a gated activation, where one linear branch modulates another, worth a consistent ~1% perplexity improvement over plain GELU at the same parameter count. And most recently — November 2024 — xIELU, a trainable activation from ETH Zürich that beats both ReLU² and SwiGLU on 1.1B and 3B Llama-family models trained on 125B tokens of FineWeb-Edu.

That's the story: every 4–6 years, a new activation family takes over. Today you learn all of them, why each one dethroned the previous, and how to draw them from memory.

1 · What you'll build today

Compute, plot, and hand-derive the four canonical activations plus the two modern gated ones. Then reproduce the vanishing-gradient failure in 30 lines of NumPy.

2 · Checkpoint · you should already know

• S004 · Derivatives and the chain rule — today is all derivative shapes. You need to be comfortable computing d/dx of composites.
• S007 · A single artificial neuron — you already saw sigmoid+BCE and its clean gradient. Today we compare it against alternatives.
• S003 · Broadcasting — you'll write activations as vectorised NumPy over batches.

3 · Why we need any nonlinearity — the collapse proof

Suppose you stack two linear layers with no activation between them:

Two layers collapse to one: an equivalent single linear layer with weights $W_2 W_1$ and bias $W_2 b_1 + b_2$. Stack a hundred linear layers, you still get one linear layer. Depth is wasted. You cannot approximate sin(x). You cannot approximate XOR (S007 §8). You cannot approximate anything a linear regressor can't approximate.

Insert any nonlinearity $\sigma$ between them:

Now the composition is not linear. In fact, given enough hidden units, it can approximate any continuous function on a compact set — the Universal Approximation Theorem (Cybenko 1989 for sigmoid; Hornik 1991 for arbitrary bounded non-constant activations). Nonlinearity is the whole reason deep learning exists.

4 · The activation zoo — canonical six

We'll cover each with (a) formula, (b) derivative, (c) shape, (d) properties, (e) hand trace on a scalar x = 2.

4.1 Sigmoid

Shape: S-curve, output in (0, 1). Derivative: bell shape, peak of 0.25 at x = 0, decays to 0 at both ends.

Hand trace at x = 2: σ(2) = 1 / (1 + e^{-2}) = 1 / (1 + 0.1353) = 0.8808. σ'(2) = 0.8808 · 0.1192 = 0.1050.

Properties:

• Smooth, differentiable everywhere. ✅
• Probabilistic interpretation (output = probability). ✅ (still used at binary output layers.)
• Saturates both ends. For |x| > 5, derivative is ~0. ⚠️
• Not zero-centred (outputs in (0, 1)), which biases downstream gradients toward always-same-sign updates. ⚠️
• Peak derivative 0.25 → gradients shrink by at least 4× per layer in a deep chain. This is the vanishing-gradient bomb (§6).

4.2 Tanh

Shape: S-curve, output in (-1, 1). Derivative: bell, peak of 1.0 at x = 0.

Hand trace at x = 2: tanh(2) ≈ 0.9640. tanh'(2) ≈ 1 - 0.9294 = 0.0706.

Properties:

• Zero-centred (fixes one sigmoid problem). ✅
• Peak derivative 1.0 (better than sigmoid's 0.25). ✅
• Still saturates for |x| > 3. ⚠️
• Preferred over sigmoid for hidden layers in the 90s and 2000s.
• Still used in modern LSTM cells (S033) for the cell-state update — not because it's better in general, but because the LSTM was designed around it.

4.3 ReLU (Rectified Linear Unit)

Shape: identity for positive inputs, zero elsewhere. Not smooth at 0 — has a kink.

Hand trace at x = 2: ReLU(2) = 2. ReLU'(2) = 1. At x = -2: both are 0.

Properties:

• Does not saturate on the positive side. Gradient of 1 for all x > 0. No vanishing. ✅✅
• Extremely fast: max(0, x) is one instruction on modern hardware. ~10× faster than sigmoid.
• Sparse activations — about half of a random input array is zeroed. Good for representation and for compute (many zero-multiplies).
• Dying ReLU: if a neuron's w·x + b is negative for every input, its output and gradient are both zero forever. The neuron is dead. Can't recover. §7.
• Not differentiable at 0 — theoretically ugly, practically fine because $x = 0$ is measure-zero and any subgradient (0 or 1) works.

4.4 Leaky ReLU / PReLU

PReLU (He et al. 2015): same but α is learned. Gives a tiny positive gradient for negative inputs so neurons can recover. Rarely used in transformers, still common in generative models (StyleGAN uses LeakyReLU).

4.5 GELU (Gaussian Error Linear Unit)

where $\Phi$ is the standard-normal CDF. Interpretation: "multiply x by the probability that a standard-normal random variable is less than x."

Shape: ReLU with a soft, curved corner near zero. For big positive x, ≈ x. For big negative x, → 0. Small negative dip around x = -0.5 (it doesn't kill mildly negative inputs completely — this is the secret sauce).

Hand trace at x = 2: GELU(2) ≈ 2 · Φ(2) ≈ 2 · 0.9772 ≈ 1.9545. At x = -0.5: GELU(-0.5) ≈ -0.154 (the dip).

Properties:

• Smooth (differentiable everywhere). ✅
• Non-monotonic — the small dip helps representations by preserving some information from mildly negative inputs.
• ~3× slower than ReLU per element, but transformers spend >90% of their time in matmul; the activation cost is invisible in the profile.
• Default activation for every transformer between BERT (2018) and PaLM (2022). Still used in the ViT family and in most encoder-only models.

4.6 Swish / SiLU

Discovered by Google's neural architecture search (Ramachandran et al. 2017) and independently proposed as SiLU (Elfwing 2018). Very similar to GELU visually — smooth ReLU with a small negative dip — but slightly cheaper (no polynomial approximation of the Gaussian CDF).

Hand trace at x = 2: Swish(2) = 2 · σ(2) = 2 · 0.8808 = 1.7616.

Used in EfficientNet and — importantly — as the base of SwiGLU (next section).

5 · The gated era — SwiGLU, GeGLU, ReGLU (2020–2025)

The 2020s brought a new idea: instead of one activation, use two linear projections and multiply them elementwise. This is called a Gated Linear Unit (GLU), and the pattern is:

The "activation" branch decides how much to let through of the "gate" branch. Names: ReGLU (ReLU), GeGLU (GELU), SwiGLU (Swish). All three are due to Noam Shazeer (2020, "GLU Variants Improve Transformer") — a delightful 5-page paper that just tries them empirically and finds ~1% perplexity improvement across the board.

SwiGLU in the FFN block of a modern LLM:

def swiglu_ffn(x, W1, W3, W2):
    # x: (batch, d_model)
    a = x @ W1                             # gate branch, (batch, d_ff)
    b = x @ W3                             # value branch, (batch, d_ff)
    h = (a * torch.sigmoid(a)) * b         # Swish(a) ⊙ b, (batch, d_ff)
    return h @ W2                          # (batch, d_model)

Three matrices (W1, W2, W3) instead of the classical two (W1, W2). To keep parameter count fair, the hidden dimension d_ff is scaled by 2/3 (LLaMA convention). Empirically: better perplexity per parameter, better perplexity per FLOP. LLaMA 1/2/3/4, Mistral 7B/8x7B/8x22B, DeepSeek-V2/V3, Qwen 2/2.5, Phi-3/4 all use SwiGLU.

Interpretation (Shazeer's, worth adopting): the gate is a learned dynamic mask on the value channel. Some features are amplified only when other features are present. It's the FFN's cheap version of attention's "which token matters for this position."

5.1 A concrete SwiGLU example — 2 features, 4 hidden units

Notice hidden unit 3 is exactly zero because b[3] = 0: the value branch turned that channel off completely regardless of what the gate said. That is the gating in action.

5.2 xIELU — the November 2024 challenger

xIELU — Fang, Tresp, and Neuhaus, "Deriving Activation Functions via Integration" (arxiv 2411.13010) — introduces a trainable piecewise activation:

with two learnable slope parameters $\alpha_p, \alpha_n$ that let the network adaptively reduce nonlinearity for deeper layers. On 1.1B and 3B Llama-family models trained on 125B tokens of FineWeb-Edu, xIELU beats both ReLU² and SwiGLU on perplexity. It's not yet mainstream (as of mid-2025 no frontier LLM has adopted it) but it's the current best-known result on the "which activation minimises perplexity per token" benchmark and worth watching.

Related note: ReLU² (Squared ReLU, or ShiftedReLU²) — used in Google's Primer (So et al. 2021) and PaLM 2 experiments. Formula: max(0, x)². Cheaper than SwiGLU (no gate matrix), competitive quality. If you ever inherit a PaLM-style codebase, this is what you'll see.

Further reading (recent):

• Shazeer, "GLU Variants Improve Transformer" (2020) — https://arxiv.org/abs/2002.05202
• So et al., "Primer: Searching for Efficient Transformers" (2021, ReLU²) — https://arxiv.org/abs/2109.08668
• Fang et al., "Deriving Activation Functions via Integration" (xIELU, Nov 2024) — https://arxiv.org/abs/2411.13010
• LLaMA 3.1 paper (2024) — activations discussed in §3.1 — https://arxiv.org/abs/2407.21783

6 · The vanishing-gradient problem — walked through with numbers

Suppose you have a 10-layer network of sigmoid neurons. Each layer's Jacobian, by the chain rule, is a factor of at most σ'(z_l) = a_l(1-a_l) ≤ 0.25. Stack ten of them:

By the time gradient reaches the first layer, it's a million times smaller than at the last layer. Weights near the input barely update. Deep networks with sigmoid activations effectively train only their top layers, and then very slowly. This is the reason no one trained networks deeper than about 3 layers in the 1990s without heroic tricks like layer-wise pretraining (Hinton 2006).

Tanh helps (peak derivative 1 rather than 0.25) but still saturates catastrophically for large inputs. ReLU's derivative is exactly 1 for positive inputs, so gradients pass through unchanged — no vanishing. That's the whole story of why ReLU broke deep learning open in 2012.

6.1 See it happen — 30-line NumPy experiment

import numpy as np
 
def deep_forward(x, W, activation):
    """Return activations at every layer for a deep MLP."""
    acts = [x]
    for w in W:
        x = activation(x @ w)
        acts.append(x)
    return acts
 
def sigmoid(x): return 1 / (1 + np.exp(-x))
def relu(x):    return np.maximum(0, x)
 
np.random.seed(0)
N, d, depth = 32, 100, 20
x = np.random.randn(N, d)
W = [np.random.randn(d, d) * 0.1 for _ in range(depth)]
 
for name, act in [("sigmoid", sigmoid), ("ReLU", relu)]:
    acts = deep_forward(x, W, act)
    stds = [a.std() for a in acts]
    print(f"{name:8s} activation std across depth:")
    for i, s in enumerate(stds):
        print(f"  layer {i:2d}: {s:.4e}")

Expected pattern:

Sigmoid activations collapse to a numerical constant after ~20 layers; every neuron outputs approximately the same value; gradients through them are numerically zero. ReLU activations decay but don't collapse. Add proper He init (S022) and ReLU activations preserve variance perfectly across depth.

Do this experiment once, staring at the numbers as they scroll. You will never confuse the two families again.

7 · The dying-ReLU problem — the price of speed

Suppose during initialisation or after a bad gradient step, a ReLU neuron's pre-activation z = w·x + b becomes negative for every input in your dataset. Then:

1. Output = 0 for every input.
2. ReLU derivative = 0 for every input.
3. Gradient through it = 0 for every input.
4. w and b never update.
5. The neuron is dead. Forever.

This happens especially with high learning rates and inputs with a large negative mean. In practice ~10–40% of ReLU units in a trained CNN can be permanently dead. Some sparsity is fine; too much wastes capacity.

Mitigations, in order of increasing sophistication:

• Positive bias init: initialise b = 0.01 for ReLU networks so at least half of neurons produce positive output at step 0.
• He initialisation (Kaiming He, 2015): scale weight variance by 2/n_in for ReLU — keeps activations well-behaved through depth. This is the biggest single fix. (Full treatment in S022.)
• Leaky ReLU / PReLU / ELU: give negative inputs a tiny non-zero gradient so dead neurons can recover.
• GELU: smooth, has the small dip; dying units are extremely rare in practice.

CNNs (S025–S030) mostly still use ReLU because it's fast and dead-neuron rates are tolerable with He init. Transformers use GELU or SwiGLU because attention layers are more sensitive to sparsity patterns.

8 · The one-picture summary (draw this once, refer forever)

sigmoid:  _____/‾‾‾‾‾               deriv:   _/\_
          (both ends flat)                   (peak 0.25 at 0)
 
tanh:     _____/‾‾‾‾‾               deriv:   _/‾\_
          (zero-centred, both flat)          (peak 1.0 at 0)
 
ReLU:     _____/                    deriv:   ___,‾‾‾
          (flat then linear)                 (step: 0 or 1)
 
GELU:     ____,~/                   deriv:   _,-‾‾‾
          (small dip at ~-0.5)               (smooth version of ReLU')
 
Swish:    ____,~/                   deriv:   _,-‾‾‾
          (near-identical to GELU)           (near-identical to GELU')
 
SwiGLU:   [gate branch] ⊙ [value branch]     (two matmuls, one mul)

Draw these on a physical piece of paper, put it under your laptop, and never lose it.

9 · The 2025 activation-choice cheat sheet

Rule of thumb for new architectures: if you don't know, use GELU. If you're optimising for perplexity, use SwiGLU. If you're optimising for wall-clock speed on inference, use ReLU or Swish. If you're writing a research paper, try xIELU.

10 · War stories

11 · Try it yourself

12 · Recap

Activations are the tiny nonlinear valves between linear layers. Sigmoid ruled the 1990s and quietly limited depth to three layers via its 0.25-peak derivative. ReLU (Nair-Hinton 2010, AlexNet 2012) broke depth open by being one-sided saturating and cheap; its dying-neuron pathology was tamed by He init and Leaky variants. GELU (2016, BERT 2018) rounded ReLU's corner with a Gaussian CDF and became the transformer default. SwiGLU (Shazeer 2020, LLaMA 2023) added a learned gate for a consistent 1% perplexity gain; every 2025 frontier LLM uses it. xIELU (Nov 2024) is the current research contender. Along the way, we learned the vanishing-gradient argument well enough to reproduce it in 30 lines of NumPy and to recognise its 21st-century echoes in dying ReLUs and LayerNorm-init bugs.

🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________________________________________

Spaced review: re-read §6 (vanishing) and §5 (SwiGLU) in 24 hours; revisit the full session on day 7. By then S009–S011 will have made the "why" of each activation concrete.

Next session (S009): the 2-layer MLP forward pass — end-to-end with numbers on a tiny dataset — plus the geometric picture of "bending space" that finally makes universal approximation feel real.

Sticky note (keep on your desk):

• Draw: sigmoid / tanh / ReLU / GELU shapes + their derivatives.
• Number: sigmoid's peak derivative is 0.25. That's the vanishing-gradient number. Memorise.
• Rule: post-2012 winning activations are one-sided saturating.
• Default in 2025: GELU for hidden layers of small transformers; SwiGLU for LLMs.

Previous: ← DL S007 · A Single Artificial Neuron · Next: DL S009 · The Forward Pass for a 2-Layer MLP →