Activation Functions — sigmoid, tanh, ReLU, GELU, SwiGLU, xIELU

Course map · DL008 · Activations, derivatives and gated representations

0 · The story — what the ReLU transition actually teaches

Glorot and Bengio2010 analyzed difficulty in deep feedforward training, including sigmoid saturation and initialization. Nair and Hinton2010 studied rectified units in restricted Boltzmann machines; AlexNet2012 used ReLU in a large convolutional classifier. These are important results, not evidence that every earlier network used sigmoid, no network exceeded three layers, or a single activation caused all benchmark progress.

AlexNet's paper reports15.3% top-five test error versus26.2% for the second-best entry in ILSVRC2012, a10.9 percentage-point error difference for the whole system, not an isolated activation ablation. Hardware, architecture, data augmentation, regularization and optimization also mattered. A ReLU's positive derivative1 is useful, but weights and inactive paths can still shrink or amplify gradients.

GELU, SiLU, gated FFNs and trainable activations offer different mechanisms. We will derive their derivatives and parameter costs, measure actual gradients, and correct the xIELU reference rather than present an ever-changing universal winner list.

1 · What you'll build today

Compute scalar activations and their derivatives; distinguish exact GELU from its approximation; trace a SwiGLU gate; implement xIELU with its source-defined parameters; compare forward statistics and backward sensitivities across depth; and run a bounded matched-data activation experiment. The tests use NumPy/SciPy and existing CPU PyTorch—not downloaded weights, datasets or GPU benchmarks.

2 · Checkpoint · you should already know

DL003 broadcasting, DL004's chain rule and vector–Jacobian products, DL007's sigmoid+BCE cancellation. A hidden activation is not an output likelihood, and SwiGLU is a two-branch module, not merely a scalar function with one input curve.

3 · Why we need any nonlinearity — the collapse proof

For column-vector affine layers, $h=W_1x+b_1$, $y=W_2h+b_2=W_2W_1x+W_2b_1+b_2$. Repeating this gives one affine map. It cannot express XOR in the original coordinates; it can approximate a smooth function locally but not every nonlinear target arbitrarily well on a fixed nontrivial interval.

A suitable nonlinear activation with sufficient width can support universal approximation under the theorem's assumptions. Cybenko's result concerns continuous sigmoidal activations and approximation on a compact domain; it is not a theorem that any nonlinearity, arbitrary finite width or a particular training method works. A quadratic activation in a single hidden layer, for example, still spans only a restricted polynomial family.

The origami analogy is useful only qualitatively: activations introduce changes of slope/shape that affine composition lacks. There is no universal2^100 region count for100 layers. Biases also invalidate the simplistic test “double x and output doubles”: affine maps generally fail that test, while bias-free ReLU networks can satisfy positive homogeneity despite being nonlinear. Test affine interpolation across a region and inspect the actual computation, not one doubled input.

4 · The activation zoo — canonical six

4.1 Sigmoid

$\sigma(x)=1/(1+e^{-x})$, $\sigma'(x)=\sigma(x)(1-\sigma(x))$. The output is in(0,1) in exact arithmetic; the derivative peaks at1/4 at0 and decays in both tails. At2, value≈.880797 and derivative≈.104994. An output in(0,1) is not automatically a calibrated probability; the model/objective gives that interpretation.

The mean shift can affect subsequent layers, but does not prove all parameter gradients have the same sign. A0.25 activation derivative bounds one local factor; the layer Jacobian also contains the weight matrix.

4.2 Tanh

$\tanh(x)=2\sigma(2x)-1$ and derivative $1-\tanh^2x$. At2, value≈.964028 and derivative≈.070651. It is odd and centered around zero, with slope1 at0 and saturation in both tails. LSTM candidate/state transformations commonly use tanh while their gates use sigmoid; “the RNN activation” hides several distinct roles.

4.3 ReLU (Rectified Linear Unit)

ReLU(x)=max(0,x), derivative1 for x>0 and0 for x < 0. At0 no classical derivative exists; our code chooses0. Exact zeros are common in finite computation, not measure-zero events to dismiss. At2, value2 and derivative1; at−2 both0.

ReLU avoids positive-tail activation saturation, but not vanishing gradients from small weights, dead paths or normalization. Sparse outputs do not automatically accelerate dense matrix multiplication. Cost depends on kernel fusion, bandwidth and hardware; no10× sigmoid multiplier is promised.

4.4 Leaky ReLU / PReLU

For a chosen slope α, use x on the positive branch and αx on the negative branch. For0≤α≤1 this equals max(αx,x); outside that range the max formula changes the intended branch. PReLU learns the negative slope. A nonzero slope supplies a backward path for negative inputs, but does not guarantee a useful network.

ELU uses x on the positive side and α(exp(x)−1) on the negative side. With α=1 its first derivative joins continuously at0; with other α values the one-sided slopes differ. Its negative derivative can still become tiny. ReLU²=max(0,x)² has derivative2max(0,x), including a continuous zero slope at0; it is not identical to every shifted squared-ReLU variant.

4.5 GELU (Gaussian Error Linear Unit)

Exact GELU is $x\Phi(x)$, where Φ is the standard normal CDF, with derivative

A second derivative is $(2-x^2)\phi(x)$, so the maximum first derivative occurs at√2 and is about1.1289; the minimum occurs at−√2 and is about−.1289. It is neither bounded by1 nor monotone everywhere. At2, value≈1.954500 and derivative≈1.085232. The negative output at−.5 is about−.154269; the function's minimum is not exactly at−.5.

The common tanh approximation is $\tfrac12x[1+\tanh(\sqrt{2/\pi}(x+.044715x^3))]$. Its derivative must differentiate the cubic and tanh as well; do not combine the approximate forward with the exact derivative and call them identical. GELU appears in documented architectures such as BERT, but the original Transformer used ReLU, so “every transformer2018–2022” was false.

4.6 Swish / SiLU

For β=1, SiLU/Swish is $x\sigma(x)$, with derivative $s+xs(1-s)$. Swish is sometimes written with a configurable/learned β; state which version is used. At2, value≈1.761594 and derivative≈1.090784. The derivative has a small negative region and a maximum about1.09984 near2.39936, not1. Smoothness and a negative tail are properties, not proof of quality superiority. The GELU paper also discusses the sigmoid linear unit; attribution should not erase that in favor of a single discovery story.

import numpy as np
from scipy.special import ndtr, expit
 
def sig(x):return expit(np.asarray(x,float))
def dsig(x):s=sig(x);return s*(1-s)
def tanh(x):return np.tanh(x)
def dtanh(x):return 1-np.tanh(x)**2
def relu(x):return np.maximum(0,x)
def drelu(x):return (np.asarray(x)>0).astype(float)
def gelu(x):return x*ndtr(x)
def dgelu(x):return ndtr(x)+x*np.exp(-x*x/2)/np.sqrt(2*np.pi)
def silu(x):return x*sig(x)
def dsilu(x):s=sig(x);return s+x*s*(1-s)
def gelu_approx(x):
    u=np.sqrt(2/np.pi)*(x+.044715*x**3)
    return .5*x*(1+np.tanh(u))
def dgelu_approx(x):
    c=np.sqrt(2/np.pi);u=c*(x+.044715*x**3);t=np.tanh(u)
    return .5*(1+t)+.5*x*(1-t*t)*c*(1+3*.044715*x*x)
 
x=np.array([-3.,-1.2,-.4,.3,1.4,2.,3.])
for fn,df in [(sig,dsig),(tanh,dtanh),(relu,drelu),(gelu,dgelu),(silu,dsilu),(gelu_approx,dgelu_approx)]:
    np.testing.assert_allclose((fn(x+1e-5)-fn(x-1e-5))/2e-5,df(x),atol=2e-9)
assert dsig(0)==.25 and dtanh(0)==1 and drelu(np.array(0.))==0
assert dgelu(np.sqrt(2))>1.12 and dgelu(-np.sqrt(2))<0
assert dsilu(2.39936)>1.0998
assert not np.allclose(dgelu(x),dgelu_approx(x),atol=1e-6)
print('at x=2:',[(f.__name__,float(f(np.array(2.))),float(df(np.array(2.)))) for f,df in [(sig,dsig),(tanh,dtanh),(relu,drelu),(gelu,dgelu),(silu,dsilu)]])
print('exact and approximate derivatives, maxima and kink convention passed')

These paired curves export the forward value (blue) and derivative (red), with shared x range −4 to4 and y range−1.5 to4.5. Clipping the display is a plotting choice; it does not alter the computed values. SwiGLU is deliberately absent because its value depends on two learned branches.

from pathlib import Path
curves=[('sigmoid',sig,dsig),('tanh',tanh,dtanh),('ReLU',relu,drelu),('GELU',gelu,dgelu),('SiLU',silu,dsilu)]
parts=['<svg xmlns="http://www.w3.org/2000/svg" width="1000" height="250" viewBox="0 0 1000 250"><rect width="1000" height="250" fill="white"/>']
grid=np.linspace(-4,4,201)
for i,(name,fn,df) in enumerate(curves):
    left=200*i
    parts.append(f'<text x="{left+15}" y="20" font-size="13">{name}: blue value, red slope</text>')
    zero_y=220-1.5/6*180
    parts.append(f'<line x1="{left+10}" x2="{left+190}" y1="{zero_y}" y2="{zero_y}" stroke="#bbb"/>')
    parts.append(f'<line x1="{left+100}" x2="{left+100}" y1="40" y2="220" stroke="#ddd"/>')
    for tick in (-4,0,4):
        xp=left+10+(tick+4)/8*180
        parts.append(f'<text x="{xp-3}" y="238" font-size="11">{tick}</text>')
    for tick in (0,1,4):
        yp=220-(tick+1.5)/6*180
        parts.append(f'<text x="{left}" y="{yp}" font-size="10">{tick}</text>')
    for func,color in [(fn,'blue'),(df,'red')]:
        yy=func(grid);assert np.isfinite(yy).all()
        points=' '.join(f'{left+10+(x+4)/8*180:.2f},{220-(np.clip(y,-1.5,4.5)+1.5)/6*180:.2f}' for x,y in zip(grid,yy))
        parts.append(f'<polyline points="{points}" stroke="{color}" fill="none"/>')
parts.append('</svg>')
figure=Path('batch5-dl-foundations-activation-curves.svg');figure.write_text(''.join(parts))
assert figure.read_text().count('<polyline')==10
print('saved',figure)

5 · The gated era — SwiGLU, GeGLU, ReGLU

A GLU-family hidden representation is $h=f(XW_1)\odot(XW_3)$, followed by $hW_2$ in an FFN. A sigmoid gate is bounded; a SiLU gate can be negative or exceed1, so it is not a probability mask. ReGLU uses ReLU, GEGLU uses GELU and SwiGLU uses SiLU. Shazeer's2020 paper compares variants at matched parameter/compute budgets; it does not establish a uniform1% perplexity gain or an immutable recipe for every model family.

A plain FFN with width m and model dimension d has2dm matrix parameters; a gated one with width h has3dh. Equalizing gives h=2m/3. For m=4d this is8d/3, before hardware-friendly rounding or architecture-specific multipliers. Keeping the same width instead adds50% FFN matrix parameters, not a fixed33% total-memory increase. Optimizer states and retained activations have their own counts.

5.1 A concrete SwiGLU example — 2 features, 4 hidden units

For x=(1,−1), W1 rows(1,0,−1,2),(0,1,1,−1), the gate input is(1,−1,−2,3). W3 rows(0,1,1,0),(1,0,1,1) gives values(−1,1,0,−1). Applying SiLU to the first and multiplying gives approximately(−.731059,−.268941,0,−2.857722). The zero is index2, the third coordinate—not b[3]. Both learned branches affect the result and both receive gradients through the product.

For upstream dh, da=dh·value·SiLU'(a) and dvalue=dh·SiLU(a). Therefore dW1=X.T@da,dW3=X.T@dvalue, with corresponding input contributions added. This makes the “gate” more than a decorative second matrix.

5.2 xIELU — a source-corrected trainable activation

The primary paper is Allen Hao Huang and Imanol Schlag, “Deriving Activation Functions Using Integration”, first submitted2024-11-20; the reviewedv3 is2025-01-31. The previous Fang/Tresp/Neuhaus attribution was incorrect. Its function is

The derivatives are2αp x+.5 and αn(exp(x)−1)+.5, joining at.5 at0. Both branches pass through0. The paper learns scalar α values per layer, constraining αp>0 and αn>.5 in its parameterization; β=.5 is fixed here, not another arbitrary learned slope. The positive derivative is unbounded as x grows, so the “all modern activations saturate one side” rule does not classify it properly. The negative branch is not a ReLU-style hard dead region.

The paper reports lower perplexity in its1.1B/3B Llama-model experiments trained on125B FineWeb-Edu tokens at matched compute/parameter count. That is an attributed study, not a current universal frontier ranking or an experiment we reproduced. Primer separately studies squared ReLU and other changes; squaring alone is not its entire architecture.

def swiglu_ffn(x,W1,W3,W2):return (silu(x@W1)*(x@W3))@W2
x=np.array([[1.,-1.]])
W1=np.array([[1,0,-1,2],[0,1,1,-1]],float);W3=np.array([[0,1,1,0],[1,0,1,1]],float)
h=silu(x@W1)*(x@W3)
np.testing.assert_allclose(h,[[-.73105857863,-.26894142137,0.,-2.85772238047]],atol=1e-10)
assert h[0,2]==0
 
def xielu(x,ap=.8,an=.8):
    assert ap>0 and an>.5
    x=np.asarray(x,float);out=np.empty_like(x);p=x>0
    out[p]=ap*x[p]**2+.5*x[p]
    out[~p]=an*(np.expm1(x[~p])-x[~p])+.5*x[~p]
    return out
def dxielu(x,ap=.8,an=.8):
    assert ap>0 and an>.5
    x=np.asarray(x,float);out=np.empty_like(x);p=x>0
    out[p]=2*ap*x[p]+.5;out[~p]=an*np.expm1(x[~p])+.5
    return out
x=np.array([-5.,-.7,0.,.5,3.])
np.testing.assert_allclose((xielu(x+1e-6)-xielu(x-1e-6))/2e-6,dxielu(x),atol=3e-7)
assert xielu(np.array(0.))==0 and dxielu(np.array(0.))==.5
assert np.isfinite(xielu(np.array([-1000.,1000.]))).all()
# Parameter derivatives, distinct from the input derivative.
for param,expected in [('ap',np.where(x>0,x*x,0)),('an',np.where(x<=0,np.expm1(np.minimum(x,0))-x,0))]:
    kwp={param:.8+1e-5};kwm={param:.8-1e-5}
    np.testing.assert_allclose((xielu(x,**kwp)-xielu(x,**kwm))/2e-5,expected,atol=1e-8)
assert 3*8*16==2*8*24  # equal matrix parameters, no biases
print('SwiGLU trace, xIELU input/parameter derivatives and matched parameter count passed')

6 · The vanishing-gradient problem — include the whole Jacobian

For a row-batch layer A_l=f(A_(l−1)W_l), backward multiplies by a diagonal activation derivative and W_l.T. A norm bound for a chain includes both:

Sigmoid supplies ||D_l||≤.25, but large matrix norms can counteract or exceed it. ReLU supplies diagonal0/1 entries, but W can shrink an active path or a zero mask can kill it. Residuals, normalization, topology and initialization change the product. The bare0.25^10≈9.54e−7 is an activation-factor bound, not automatically the complete first-layer gradient.

6.1 See it happen — measure forward and backward separately

The original working forward-only experiment is retained exactly below. It prints global activation standard deviation, which mixes variation across samples and across units. It does not compute gradients. In particular sigmoid activations can become almost identical across samples while different neurons retain different output values. The old displayed global-std≈1e−15 claim did not follow from this program and is not copied.

import numpy as np
 
def deep_forward(x, W, activation):
    """Return activations at every layer for a deep MLP."""
    acts = [x]
    for w in W:
        x = activation(x @ w)
        acts.append(x)
    return acts
 
def sigmoid(x): return 1 / (1 + np.exp(-x))
def relu(x):    return np.maximum(0, x)
 
np.random.seed(0)
N, d, depth = 32, 100, 20
x = np.random.randn(N, d)
W = [np.random.randn(d, d) * 0.1 for _ in range(depth)]
 
for name, act in [("sigmoid", sigmoid), ("ReLU", relu)]:
    acts = deep_forward(x, W, act)
    stds = [a.std() for a in acts]
    print(f"{name:8s} activation std across depth:")
    for i, s in enumerate(stds):
        print(f"  layer {i:2d}: {s:.4e}")

Now hold the same input and matrices fixed between activations, use an explicit scalar readout, and backpropagate it. Compare global standard deviation, mean per-unit batch variation, input gradient and first-layer weight gradient. ReLU under this intentionally uncalibrated initialization can also shrink. A separate scalar negative control shows an all-active ReLU chain with weights.5 has derivative.5^20. Nothing about ReLU's local slope prevents that.

rng=np.random.default_rng(8);xx=rng.normal(size=(12,24));base=[rng.normal(size=(24,24))/np.sqrt(24) for _ in range(40)]
for depth in (1,5,20,40):
    for name,fn,df in [('sigmoid',sig,dsig),('ReLU',relu,drelu)]:
        W=base[:depth];a=xx.copy();acts=[a];zs=[]
        for w in W:z=a@w;zs.append(z);a=fn(z);acts.append(a)
        g=np.ones_like(a)/a.size;dw0=None
        for l in reversed(range(depth)):
            dz=g*df(zs[l]);dw=acts[l].T@dz;g=dz@W[l].T
            if l==0:dw0=dw
        assert np.isfinite(g).all() and np.isfinite(dw0).all()
        print('depth',depth,name,'global_std',float(a.std()),'batch_std',float(a.std(axis=0).mean()),'dx',float(np.linalg.norm(g)),'dW0',float(np.linalg.norm(dw0)))
assert .5**20 < 1e-6  # all-active ReLU can still have a tiny chain derivative
# Fair shared-weight two-layer finite-difference check, away from ReLU kinks.
W=base[:2];x0=xx[:2].copy()
for fn,df in [(sig,dsig),(relu,drelu)]:
    z1=x0@W[0];a1=fn(z1);z2=a1@W[1];a2=fn(z2)
    upstream=np.arange(a2.size).reshape(a2.shape)/a2.size
    analytic=((upstream*df(z2))@W[1].T*df(z1))@W[0].T
    def f(x):return float((fn(fn(x@W[0])@W[1])*upstream).sum())
    for idx in np.ndindex(x0.shape):
        d=np.zeros_like(x0);d[idx]=1e-6
        assert abs((f(x0+d)-f(x0-d))/2e-6-analytic[idx]) < 1e-7
print('matched depth measurements and full-Jacobian finite differences passed')

7 · The dying-ReLU problem — assumptions behind a dead unit

If a unit is negative on all current inputs, its local derivative is0 and data-gradient flow through that activation is0. Under plain gradient updates with fixed upstream inputs it may remain stuck. Momentum, weight decay, changing upstream features or new inputs can change its parameters/preactivations, so “dead forever under every training setup” is too strong.

Inspect per-unit active fractions over representative batches, not just one sample. Large negative biases, input shifts and aggressive updates are possible causes. Positive bias.01 does not guarantee half the examples activate; that depends on the input/preactivation distribution. Leaky/PReLU/ELU provide negative-side derivatives but do not remove all numerical or optimization problems. Smooth GELU/SiLU can also have extremely small tail derivatives.

Why He initialization contains a factor of two

With independent zero-mean weights, preactivation variance is $n\operatorname{Var}(w)E[a^2]$, not necessarily $n\operatorname{Var}(w)\operatorname{Var}(a)$: ReLU outputs have nonzero mean. For symmetric zero-mean z, $E[\operatorname{ReLU}(z)^2]=E[z^2]/2$. Compensating this second-moment loss motivates Var(w)=2/n. For leaky slope α the factor becomes2/(1+α²). Xavier's common fan-in/fan-out compromise uses2/(nin+nout), with additional assumptions about activation and gradient statistics.

For standard normal Z, E[ReLU(Z)]=1/√(2π), E[ReLU(Z)²]=1/2, but Var(ReLU(Z))=1/2−1/(2π)≈.340845, not1/2. Initialization arguments concern approximate distribution propagation, not perfect variance preservation at every layer or after training. Use the correct fan orientation for X@W versus X@W.T.

8 · The one-picture summary (draw this once, refer forever)

Sigmoid and tanh are bounded S-curves with derivatives tending to zero in both tails. ReLU is flat then linear, with a derivative step. GELU and SiLU have smooth negative dips and derivatives that can be negative or above1. SwiGLU combines two learned projections, so there is no single context-free scalar derivative curve for the whole module. xIELU joins a quadratic positive branch to an integrated exponential negative branch with learned coefficients.

To draw each, mark zero, tail behaviour, derivative at zero and any negative-derivative region. A smooth-looking curve alone is not evidence that a deep network's gradients remain healthy.

9 · Activation-choice cheat sheet

For a Bernoulli output use a logits BCE for training and sigmoid for probability display; for mutually exclusive classes use logits CE and softmax when probabilities are needed. An unconstrained real regression output is often affine; constrained outputs need an appropriate link.

For an LSTM preserve its documented sigmoid gates and tanh state transforms unless deliberately redesigning it. ReLU is an inspectable MLP/CNN baseline; GELU/SiLU fit documented smooth-activation recipes; a SwiGLU FFN needs a compatible width/parameter allocation. A pretrained architecture's activation is part of its checkpoint contract, not a free inference-time swap. xIELU is a source-defined experimental option, not a recommended universal replacement.

10 · Failure modes worth reproducing

A removed activation can collapse a stack of affine layers, but one doubled input is not a valid detector. A dead first hidden layer blocks its incoming data gradients, while downstream biases can still learn. A normalization scale initialized to zero can suppress a branch, but some architectures intentionally zero-initialize residual branches; context determines whether it is a bug.

Changing GELU to SwiGLU while retaining width increases FFN matrix parameters by50%. To compare activation mechanisms, match parameter or compute budget explicitly, and report that those are different matching criteria. CPU elementwise timings cannot prove GPU end-to-end speed. Sparse outputs help only if the downstream kernel exploits them.

11 · Try it yourself

Draw and check derivatives: use the exact program's printed x=2 values and finite differences. For ReLU skip the kink in ordinary derivative checks; separately record the chosen zero convention. Compare exact and approximate GELU using their own derivatives.

Reproduce depth behaviour: the program measures depths1,5,20,40 using matched matrices and inputs. Plot the printed norms on a log scale if desired, separating global variance from sample variation. Do not require “six orders” from every initialization; that would preselect the conclusion. A failed gradient check is a bug; an unexpected but finite curve is a result to investigate.

Four-activation two-moons comparison: generate500 points locally with scikit-learn, split before fitting preprocessing, use the same architecture/data order, and record held-out loss for each activation. A fair research comparison then tunes each optimizer rate and initialization and repeats seeds; a single controlled fixture must not be promoted to a ranking.

Mini SwiGLU: d=8,h=16 uses384 matrix entries. A plain GELU FFN must use width24 to match384, not width16. Time warmed-up calls with the same batch and output shape, verify numerical/shape contracts, and report the setting rather than claiming production fusion.

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
import timeit
X,y=make_moons(n_samples=500,noise=.1,random_state=8)
train,test=train_test_split(np.arange(len(X)),test_size=.25,stratify=y,random_state=8)
mu=X[train].mean(0);sd=X[train].std(0);X=(X-mu)/sd
rng=np.random.default_rng(80);initial=[rng.normal(scale=.3,size=(2,16)),np.zeros(16),rng.normal(scale=.3,size=16),np.array(0.)]
for name,act,dact in [('sigmoid',sig,dsig),('tanh',tanh,dtanh),('ReLU',relu,drelu),('GELU',gelu,dgelu)]:
    W,b,v,c=[p.copy() for p in initial]
    def loss(ids):
        z=act(X[ids]@W+b)@v+c
        return float(np.mean(np.maximum(z,0)-y[ids]*z+np.log1p(np.exp(-np.abs(z)))))
    first=loss(train)
    for _ in range(300):
        a=X[train]@W+b;h=act(a);z=h@v+c;dz=(sig(z)-y[train])/len(train)
        da=dz[:,None]*v*dact(a);dW=X[train].T@da;db=da.sum(0);dv=h.T@dz;dc=dz.sum()
        W-=.1*dW;b-=.1*db;v-=.1*dv;c-=.1*dc
    assert np.isfinite(loss(test)) and loss(train)<first
    print('moons fixed recipe',name,'train',loss(train),'heldout',loss(test))
x=rng.normal(size=(32,8));A=rng.normal(size=(8,16));B=rng.normal(size=(8,16));C=rng.normal(size=(16,8))
P=rng.normal(size=(8,24));Q=rng.normal(size=(24,8))
def gated():return swiglu_ffn(x,A,B,C)
def plain():return gelu(x@P)@Q
assert A.size+B.size+C.size==P.size+Q.size==384
assert gated().shape==plain().shape==(32,8)
gated();plain()
print('CPU seconds matched-parameter gated/plain',min(timeit.repeat(gated,number=30,repeat=3))/30,min(timeit.repeat(plain,number=30,repeat=3))/30)

12 · Recap

Judge forward representation, backward derivative, initialization and implementation cost together. A local derivative is only one factor in a layer Jacobian. A gated module is an architecture change with a parameter budget, and an experimental activation's benchmark is bounded to its study. No chronological story substitutes for those checks.

🧠 Retention scaffold

1. Formulas and derivative peaks? Sigmoid s(1−s) peaks.25; tanh1−tanh² peaks1; ReLU is0/1 away from its kink; GELU derivative Φ+xφ peaks≈1.1289 at√2; SiLU derivative s+xs(1−s) peaks≈1.09984 near2.39936. Approximate GELU has its own derivative.
2. Why can sigmoid shrink gradients? Its local factor is at most.25, but multiply by the weight Jacobians too. ReLU does not multiply active paths by a small activation factor, yet small weights and inactive paths can still vanish.
3. SwiGLU trace? Gate inputs(1,−1,−2,3), value(−1,1,0,−1), hidden(−.731059,−.268941,0,−2.857722). Index2 is zero because its value branch is zero.
4. Dying ReLU and three mitigations? Negative preactivations on all relevant inputs block local data gradients. Inspect initialization/feature scale and rate, consider an appropriate bias policy, or use a nonzero-negative-slope/smooth alternative. None is a guarantee of recovery under arbitrary conditions.
5. LSTM, ResNet-50, Llama-3-70B FFN, binary output? Use sigmoid for LSTM gates and tanh for its candidate/state transforms; ReLU for the standard ResNet-50 recipe; and SwiGLU for Llama 3's FFN. The official Llama 3 implementation computes W2(SiLU(W1(x)) * W3(x)), so this is a two-branch module, not a pointwise swap. For the binary head, train raw logits with BCE-with-logits and apply sigmoid only when probabilities are needed. These choices follow architecture/task contracts, not a universal winner.
6. Primary attribution? GELU: Hendrycks and Gimpel2016; GLU variants including SwiGLU: Shazeer2020; xIELU: Huang and Schlag, first2024, reviewedv3 January2025. Do not invent publication venues or the earlier wrong author list.
7. Why match parameter count? Otherwise a wider/double-branch model confounds activation with capacity. Equate2dm=3dh, so h=2m/3 and for m=4d obtain8d/3 before rounding.
8. One-hour xIELU comparison? Predeclare the task/split, paired seeds, matched parameter and separately measured compute budgets, a small rate grid per method, and held-out NLL plus wall time/memory. Count learned α parameters, optimizer state and warmup. A small inconclusive result is honest; one hour cannot reproduce the125B-token paper study.

Spaced review: tomorrow re-derive the two smooth derivatives and the SwiGLU product rule; next week reconstruct the second-moment argument and compare global versus per-unit batch variation. Next, DL009 composes these operations into a multilayer forward pass.

Primary references and boundary

Glorot–Bengio2010, AlexNet2012, GELU, GLU variantsv1, xIELUv3, He et al.v1, Primer, PyTorch2.9 initialization and LLaMA support the scoped claims. The exact CPU programs measure only their small fixtures. No pretrained activation swap, GPU throughput, universal-quality ranking or paper-scale reproduction is claimed.

Previous: One neuron · Next: MLP forward pass