Perceptron & Activation Functions

Why this session exists

A neuron computes a weighted sum plus bias and then an activation. That simple formula is worth understanding precisely: the classic perceptron's mistake-driven update is not backpropagation through a hard step. Modern differentiable networks retain the affine computation but change the activation, objective and learning procedure.

You will implement AND and OR, prove why one affine threshold cannot represent XOR, construct a correct two-hidden-unit XOR network, compare six activations and diagnose saturation and inactive ReLUs. You will also distinguish a convergence theorem from a generalization guarantee.

Prerequisites

Know vector dot products, arrays and derivatives. S017 vectors, S019 derivatives and S086 logistic regression are the relevant foundations; session 050 is not a NumPy prerequisite.

(a) Intuition · 5 min

Think of features as votes with learned strengths. A positive weight makes a larger feature favor class 1; a negative weight reverses that effect. The bias shifts the decision independently of the input. The committee analogy explains addition, but does not mean each feature is an independent expert or that a weight is a causal measure of trust.

The neuron equation you will see everywhere

For $x,w\in\mathbb R^d$,

Our perceptron predicts $\hat y=1$ when $z\ge0$, otherwise 0. The tie convention matters, especially at zero initialization. For labels in $\{0,1\}$, on each example use

A false negative has $e=1$, so its new score increases by $\eta(\|x\|^2+1)$; a false positive gets the opposite change. Correct predictions make no update. This is not the gradient of the discontinuous step output. With signed labels one can connect the update to a subgradient of the perceptron surrogate $\max(0,-y z)$, choosing a boundary convention explicitly.

The surface $w^Tx+b=0$ has normal vector $w$. For nonzero $w$, signed distance is $(w^Tx+b)/\|w\|$. Without a bias, the surface must pass through the origin. Bias is useful but not mandatory when the desired separator already passes there.

The seven decades that got us here

The development to understand is from a threshold decision with a special learning rule to multilayer differentiable computation, and then to better activation/initialization/architectural choices. No single activation “made deep learning possible” by itself. Glorot and Bengio (2010) studied sigmoid saturation, activation/gradient propagation and layer-Jacobian singular values, proposing an initialization scheme rather than claiming to invent ReLU. Their abstract even notes that saturated units can slowly leave saturation. He-style initialization (2015), documented in the initialization reference, accounts for rectifier signal scaling; residual learning (2015) reformulates blocks as residual functions to ease deeper optimization. Nonlinear representation, usable derivative paths, initialization and architecture therefore address different obstacles. More compute/data can make larger experiments feasible but cannot turn an affine XOR separator into a nonlinear one. The GELU authors define $x\Phi(x)$ and compare it empirically with ReLU and ELU; this does not prove every transformer uses GELU or that it wins every task. The original Transformer paper uses ReLU in its position-wise feed-forward block, a concrete counterexample to that universal claim.

One neuron, one picture

For $x=(2,-1)$, $w=(0.5,1)$ and $b=0.2$, $z=0.2$. A threshold returns 1, ReLU returns 0.2 and sigmoid returns about 0.5498. The same pre-activation has different semantics under different output functions. A bounded sigmoid output can parameterize a Bernoulli probability model, but calibration is an empirical property, not guaranteed by its range.

The activation function zoo — visualised

Here $\Phi$ and $\varphi$ are the standard-normal CDF and density. Exact GELU uses the CDF; the common tanh expression with coefficient 0.044715 is an approximation. GELU and SiLU are nonmonotonic, and their derivatives are not nonzero at every negative input. Smoothness is not a guarantee against small gradients.

At zero ReLU has no classical derivative; PyTorch chooses zero. Exact zeros occur routinely with zero biases, masks and preceding ReLUs, so the convention can matter. Leaky ReLU reduces one mechanism for a dead unit but does not guarantee useful learning or prevent other forms of saturation.

When to use which

For binary BCE, pass logits to a numerically stable BCE-with-logits loss and apply sigmoid when probabilities are needed. For mutually exclusive classes use logits with categorical cross entropy; for independent multilabel outputs use separate binary logits, not a softmax forcing them to sum to one. Regression may use an identity output, or a transform appropriate to target constraints.

ReLU is a simple hidden-layer baseline. GELU and SiLU are alternatives to test with appropriate initialization. Sigmoid remains useful in recurrent gates and other gating mechanisms; tanh often supplies recurrent candidate/state values. A gated FFN such as SwiGLU multiplies one SiLU-transformed projection by another learned projection. It is not just replacing ReLU in an otherwise identical two-matrix network.

Why the perceptron cannot learn XOR — geometry and a proof

Let the negative inputs be $(0,0),(1,1)$ and positives $(1,0),(0,1)$, with a nonnegative score predicting 1. Correct classification would require

The positive constraints imply $w_1+w_2+2b\ge0$, hence $w_1+w_2+b\ge-b>0$, contradicting the second negative constraint. No learning rate or extra epochs can fix missing representational capacity. A linear threshold can classify at most three of these four examples correctly, and some iterate orders do worse; there is no universal 50–75% oscillation trace.

Affine compositions collapse: $(xW_1+b_1)W_2+b_2=x(W_1W_2)+b_1W_2+b_2$. Calling them affine rather than strictly linear keeps the biases explicit. Multiple thresholded hidden units followed by another combining layer can solve XOR: those threshold operations are nonlinear. “Width never helps” is therefore too broad; width in a nonlinear hidden layer can add capacity, whereas more composed affine maps still collapse.

What the convergence theorem does and does not say

Augment $x$ with a constant 1 for the bias. Suppose signed examples have norm at most $R$ and a unit vector $u$ separates them with margin $\gamma>0$. Starting at zero with unit updates on $y(w^Tx)\le0$, after $M$ updates the projection obeys $w_M^Tu\ge M\gamma$. The squared norm increases by at most $R^2$ per update because the cross term is nonpositive, so $\|w_M\|\le R\sqrt M$. Cauchy–Schwarz gives $M\gamma\le R\sqrt M$, hence

This is a mistake bound under specified separability, boundedness and update conventions. It does not say the first separator maximizes margin, gives calibrated probabilities or generalizes optimally. Nonseparable data cannot achieve zero training error; a cap or pocket algorithm retaining the best iterate makes failure visible instead of looping forever. Logistic regression and soft-margin SVMs offer other objectives, but finite or unique optima require their own assumptions: unregularized logistic regression on separable data need not have a finite minimizer.

Engineer features or learn them?

For XOR, the fixed feature $x_1x_2$ gives score $x_1+x_2-2x_1x_2-0.5$, positive exactly on the mixed inputs. It is cheap and interpretable because we know the interaction. A learned hidden layer is valuable when useful transformations are unknown; it introduces nonconvex optimization and additional parameters. Kernels, splines or random features offer fixed nonlinear representations with different training and scaling costs. Dense kernel storage can grow quadratically in sample count, while explicit finite feature maps need not. None of these choices universally dominates tabular or perceptual data.

(c) Hands-on · 25 min

The complete NumPy/SciPy CPU program tests the truth tables, a correct tanh XOR construction and stable activations. The two broken fixed-weight examples in the earlier lesson are replaced by actual computed logits: a claim of “approximately 0,1,1,0” must survive substitution.

import numpy as np
from scipy.special import expit, ndtr
 
class Perceptron:
    def __init__(self, n_features, lr=.1):
        self.w = np.zeros(n_features, dtype=float)
        self.b, self.lr = 0., lr
    def predict(self, X):
        return (X @ self.w + self.b >= 0).astype(int)
    def fit(self, X, y, epochs=100):
        assert X.ndim == 2 and y.shape == (len(X),)
        assert set(np.unique(y)) <= {0,1}
        history = []
        for _ in range(epochs):
            mistakes = 0
            for xi, yi in zip(X, y, strict=True):
                error = yi - int(xi @ self.w + self.b >= 0)
                self.w += self.lr * error * xi
                self.b += self.lr * error
                mistakes += int(error != 0)
            history.append(float(np.mean(self.predict(X) == y)))
            if mistakes == 0:
                break
        return history
 
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y_xor = np.array([0,1,1,0])
for name,y in [('AND',np.array([0,0,0,1])), ('OR',np.array([0,1,1,1])), ('XOR',y_xor)]:
    p = Perceptron(2)
    history = p.fit(X,y)
    print(name, 'accuracy',history[-1], 'epochs',len(history),'weights',p.w,p.b)
    assert history[-1] == 1 if name != 'XOR' else history[-1] <= .75
 
# h1 detects x1+x2 above .5; h2 detects it above 1.5.
W1 = np.array([[5.,5.],[5.,5.]])
b1 = np.array([-2.5,-7.5])
W2 = np.array([5.,-5.])
b2 = -5.
h = np.tanh(X @ W1 + b1)
logits = h @ W2 + b2
prob = expit(logits)
print('XOR hidden',h,'logits',logits,'probabilities',prob)
assert np.array_equal(prob >= .5, y_xor)
assert np.all(prob[y_xor == 0] < .01) and np.all(prob[y_xor == 1] > .99)
engineered = X[:,0]+X[:,1]-2*X[:,0]*X[:,1]-.5
assert np.array_equal(engineered >= 0,y_xor)
z = np.linspace(-3,3,7)
exact_gelu = z*ndtr(z)
approx_gelu = .5*z*(1+np.tanh(np.sqrt(2/np.pi)*(z+.044715*z**3)))
for name,values in [('sigmoid',expit(z)),('tanh',np.tanh(z)),
    ('relu',np.maximum(z,0)),('leaky',np.where(z>0,z,.01*z)),
    ('gelu',exact_gelu),('silu',z*expit(z))]:
    print(name,np.round(values,6))
assert np.max(abs(exact_gelu-approx_gelu)) < .001
assert np.isfinite(expit(np.array([-1000.,1000.]))).all()
x=np.array([1.,2.,3.]); w=np.full(3,-1.); b=-10.
assert x@w+b == -16 and np.maximum(x@w+b,0) == 0
# Exact affine collapse with biases, not only zero-bias matrix associativity.
rng=np.random.default_rng(5)
A=rng.normal(size=(2,4)); B=rng.normal(size=(4,1))
a=rng.normal(size=4); b=rng.normal(size=1)
assert np.allclose((X@A+a)@B+b, X@(A@B)+a@B+b)

Anatomy of the script

Zero initialization is fine for this single perceptron; there are no parallel hidden units whose symmetry must be broken. The finite iteration cap is essential for XOR. The stable sigmoid comes from scipy.special.expit, avoiding overflow from a naive exponential at large negative inputs. ndtr gives exact GELU's CDF; the separate approximation is correctly labeled.

For the fixed XOR network, the input sums are 0,1,1,2. Hidden values are approximately $(-0.9866,-1)$, $(0.9866,-0.9866)$ twice, and $(1,0.9866)$. The resulting logits are about $-4.9331,4.8661,4.8661,-4.9331$, giving the correct decisions and probabilities near 0.0072,0.9924,0.9924,0.0072. This is representation, not evidence that arbitrary gradient descent always finds these weights.

(d) Production reality · 15 min

Saturation: A sigmoid derivative is at most 1/4, so a chain consisting only of forty such derivative factors has magnitude at most $4^{-40}\approx8.27\times10^{-25}$. A real network also multiplies weight Jacobians and may contain residual branches; this number is not the measured gradient of a ResNet. ReLU's positive derivative avoids that particular factor, but its weight products can still shrink or explode.

Inactive units: A negative pre-activation for one batch is not proof a neuron is permanently dead. Inspect activation frequencies over representative data, gradients and initialization. If all relevant inputs remain negative, its data-loss gradient through ReLU is zero. Upstream changes, optimizer momentum or decay may still move parameters. Leaky activations and careful scaling are interventions to test, not a universal “20% dead” threshold.

Architecture choices: The original Transformer FFN uses ReLU; GELU and gated SiLU-based blocks are other designs. Changing activation can alter training behavior and parameter count for gated variants. Compare validation results under a controlled recipe, not unsupported universal perplexity improvements or invented corporate experiments.

Where this shows up in the rest of the plan

S097 turns these units into matrices; S098 differentiates them; S099 separates gradients from updates. In S103 recurrent networks, sigmoid and tanh still have distinct roles. A transformer FFN acts across feature dimensions; it is not literally just two scalar neurons.

(e) Recall + stretch · 10 min

1. Equation and update? $z=w^Tx+b$, threshold for prediction, $w\leftarrow w+\eta(y-\hat y)x$ and corresponding bias update. Do not differentiate the hard step to obtain that rule.
2. Why no XOR separator? Summing the two positive inequalities contradicts the negative at $(1,1)$.
3. Vanishing gradient? Products of local Jacobians can contract strongly, including saturated activation derivatives and small singular values of weight matrices.
4. Where sigmoid? Binary probabilities and gates; logits-based training remains more stable than explicit probability logs.
5. Dead ReLU? Zero data-loss derivative while the unit stays negative on all relevant inputs; inspect rather than infer from one batch.
6. Transformer activation? Architecture-dependent: ReLU, GELU and gated SiLU variants exist. No universal best choice follows from smoothness.
7. All hidden weights identical? If incoming and outgoing structure preserves symmetry, units can receive identical updates. Random initialization is a practical way to break it; asymmetry elsewhere can also do so.
8. Why do affine layers collapse? Matrix products compose and biases combine as $b_1W_2+b_2$.
9. Derivative at zero? ReLU is nondifferentiable there; frameworks choose a convention. Zeros are not impossible in floating point.
10. Stretch: Reconstruct the four tanh hidden rows and logits above, then verify the engineered interaction alternative. Both solve the representation problem by introducing nonlinearity, not by promising one perfect training trajectory.

Explain-out-loud test

Draw one hyperplane, derive the XOR contradiction, and explain why nonlinearity plus representation matters more than a neuron-count slogan. Separate the mistake bound, the optimizer's behavior and held-out generalization.

What comes next

Multi-layer forward passes preserve the same equation while making shapes and parameter counts explicit.

Sources and execution boundary

PyTorch 2.9 activation definitions, GELU primary paper, sklearn perceptron and logistic objectives, and the original Transformer support the stated definitions and architectural counterexample. The XOR, affine-collapse and convergence-bound reasoning is shown explicitly; the CPU fixtures test the concrete constructions, not large-model performance.