DL S007 · A Single Artificial Neuron

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 7 / 80 · Module M02 · ~2.5 hours

0 · The story — a psychologist, a room full of photocells, and a New York Times headline

It's 1958. Frank Rosenblatt, a psychologist at Cornell Aeronautical Laboratory, wheels a machine the size of a fridge into a room and turns it on for reporters. It's called the Mark I Perceptron. It has a 20×20 grid of cadmium-sulphide photocells wired through a bank of potentiometers whose resistances — the "weights" — are literally spun by small electric motors as the machine learns. The New York Times reports the next day, on July 8, 1958, that the Navy has revealed:

"…the embryo of an electronic computer that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence."

That machine — that fridge full of photocells and motor-driven knobs — is one neuron. A single perceptron with 400 inputs, one weighted sum, one hard-threshold activation, one binary output. The learning rule Rosenblatt used (weight ← weight + error × input) is not quite the one you'll write today, but it's the same shape: a linear score, a nonlinear squish, a target, an error, and a nudge.

The hype was catastrophic. Eleven years later, in 1969, Marvin Minsky and Seymour Papert published Perceptrons — a book of beautiful proofs whose most famous corollary is: a single perceptron cannot compute XOR. Two dots at (0,0) and (1,1) labelled 0. Two dots at (0,1) and (1,0) labelled 1. No straight line separates them. The math is one paragraph and it is unarguable. Funding evaporated. The first AI winter began.

The thaw came from stacking. Rumelhart, Hinton and Williams's 1986 paper "Learning representations by back-propagating errors" showed that if you put more than one layer of neurons together and use a smooth activation (like the sigmoid you're about to code), you can train the whole stack with the chain rule. XOR falls in one epoch. Everything else — every LeNet, every ResNet, every transformer, every LLM you've ever prompted — is a descendant of that fix.

You are, right now, at the exact spot Rosenblatt was standing at. The rest of this module is you replaying, in fast-forward, the next thirty years of the field. This session: one neuron. Session 008: better activations. Session 009: stack them. Session 010: derive backprop by hand. Session 011: MNIST with the whole thing in NumPy. Session 012: micrograd. Session 013: vectorise it. By S013 you will have, in 250 lines of code, everything Hinton's lab had in 1990 — and everything a modern PyTorch nn.Linear layer still is under the hood.

Let's build the neuron.

1 · What you'll build today — the one-sentence spec

Train a single sigmoid neuron with binary cross-entropy on a 2D toy dataset, verify its analytic gradient with a two-sided numerical check, then break it on XOR.

That's the entire session. Everything else is scaffolding to make sure you believe every line.

2 · Checkpoint · you should already know

• S002 · Vectors, matrices, geometry — the dot product w·x is the core operation of the forward pass. If you're shaky on it, re-read §3 of S002 before continuing.
• S003 · Broadcasting in NumPy — how z + b works when z is shape (N,) and b is a scalar.
• S004 · Derivatives and the chain rule — you will differentiate a three-op chain by hand in §5.
• S005 · Probability & information — binary cross-entropy is the negative log-likelihood of a Bernoulli. This is where it comes from.
• S006 · Optimisation intuition — SGD is the update rule at the bottom of the training loop.

If any of those feel foggy: 10 minutes revising S004 §2 (the chain rule) will pay off more than any other prep.

3 · Anatomy of one neuron

A single neuron with d inputs is five objects and three arrows. Draw this on paper before reading the code:

inputs    x  ∈ ℝ^d       (one feature vector for one example)
weights   w  ∈ ℝ^d       (learned parameters, one per input dimension)
bias      b  ∈ ℝ         (a learned scalar offset)
pre-act   z  = w · x + b               (a scalar — one number per example)
activation a = σ(z)                    (a scalar in (0, 1) if σ is sigmoid)

Shapes: x and w are both length-d; their dot product is a scalar; add b, still a scalar; run through σ, still a scalar. One neuron produces one number per input example. Nothing more.

The whole picture:

Read that diagram left to right and it's the forward pass. Read it right to left and it's the backward pass. You will do both today.

3.1 A concrete 3-input example you can hold in your head

Let d = 3. Take these numbers:

x = [1.0,  2.0,  -1.0]
w = [0.5, -0.3,   0.8]
b = 0.1

Then:

z = 1.0·0.5 + 2.0·(-0.3) + (-1.0)·0.8 + 0.1
  = 0.5 - 0.6 - 0.8 + 0.1
  = -0.8
 
a = σ(-0.8) = 1 / (1 + e^{0.8}) ≈ 1 / (1 + 2.2255) ≈ 0.3100

Check that on paper. If your z doesn't come out to -0.8, stop and find the arithmetic bug — because if you can't do it on three numbers, you cannot debug it on three million.

3.2 Batched version — the shape that will haunt you for the next 70 sessions

For a batch of N examples with d features each — X of shape (N, d) — the neuron computes:

z = X @ w + b        shape (N,)
a = σ(z)             shape (N,)

One scalar per example. Broadcasting (S003) handles the bias: b is a scalar, X @ w is shape (N,), and NumPy adds b to every entry.

3.3 The neuron IS logistic regression

If σ is the sigmoid, one neuron trained on binary cross-entropy is exactly logistic regression as any statistician has known it since Berkson coined the term in 1944. You have (unknowingly) already learned deep learning's smallest cell, and it's a mid-twentieth-century statistical model. Don't be disappointed — the miracle of deep learning is that stacks of these things become universal function approximators (Cybenko 1989, Hornik 1991), and that stacks with modern activations, residuals, attention, and mass parallelism become GPT-4.

You should now recognise: every nn.Linear layer in PyTorch is a rectangular grid of these neurons — one row of w per output unit — followed by an optional activation. That's it. Sessions 009 and 013 will make the "rectangular grid" version explicit.

4 · The forward pass — one line of NumPy

import numpy as np
 
def forward(X, w, b):
    z = X @ w + b                       # shape (N,)
    a = 1.0 / (1.0 + np.exp(-z))        # sigmoid, shape (N,)
    return a, z

That's it. Every deep neural network is this line, over and over, with different σs and much bigger ws. Let's break it before we scale it.

4.1 Numerical hygiene — the stable sigmoid

np.exp(-z) overflows for large negative z. At z = -1000, np.exp(1000) returns inf, then 1 / (1 + inf) returns 0.0 — you get lucky and the answer is right. But at z = +1000, np.exp(-1000) returns 0.0 and you get 1 / 1 = 1.0, again lucky. The unlucky case is subtle: intermediate infinities can propagate as NaNs once you multiply by anything derived from them.

The industry-standard fix uses two branches:

def sigmoid(z):
    """Numerically stable sigmoid — no exp overflow either direction."""
    pos = z >= 0
    out = np.empty_like(z, dtype=float)
    # for z >= 0: 1 / (1 + exp(-z))    (exp(-z) is small, safe)
    out[pos]  = 1.0 / (1.0 + np.exp(-z[pos]))
    # for z <  0: exp(z) / (1 + exp(z)) (exp(z) is small, safe)
    e = np.exp(z[~pos])
    out[~pos] = e / (1.0 + e)
    return out

You will not write this daily — every framework has it — but knowing why it exists is the point. This is the same "log-sum-exp trick" you saw for the softmax in S005 §6, in a two-class disguise.

4.2 A hand trace of the forward pass on 4 examples

Let's do a full forward pass on a mini-batch of N = 4 examples with d = 2 features:

Type those numbers into a NumPy REPL and confirm. Every neuron in every model you'll ever build is doing exactly this arithmetic, just at bigger scale.

5 · The loss — binary cross-entropy

For binary targets y ∈ {0, 1} and predicted probability a ∈ (0, 1):

Two cases, one formula. When y = 1 the second term drops out and L = -\log a: the loss is small if a is near 1 (we predicted "yes" correctly) and blows up as a → 0. When y = 0, symmetric: L = -\log(1-a). It's the negative log-likelihood of a Bernoulli we derived in S005 §4, specialised to binary targets. Not a random choice, not an engineering hack — the maximum-likelihood loss that falls out of the Bernoulli assumption.

5.1 Batched implementation

def bce_loss(a, y, eps=1e-8):
    a = np.clip(a, eps, 1 - eps)           # avoid log(0)
    return -(y * np.log(a) + (1 - y) * np.log(1 - a)).mean()

The np.clip is a small robustness cushion so a single a = 1.0 prediction (which happens for z ≥ ~37 in float64) doesn't produce log(0) = -inf. In production you'd instead use binary_cross_entropy_with_logits (see §6.3), which is numerically bulletproof — it never forms a at all.

5.2 Where does -log(0.5) = 0.693 come from?

At the start of training, w = 0, b = 0, so z = 0, so a = σ(0) = 0.5. Every prediction is 50/50. The loss per example is -\log(0.5) = \log 2 ≈ 0.6931. This is your training-run smoke test. If step 0 of your first training loop doesn't print a loss of ~0.693 for a balanced binary problem, something is wrong. Not "maybe wrong." Wrong.

Twenty-six years of ML papers open with a loss curve that starts at \log 2 and drops. If you can spot the \log 2 benchmark in your head, you can eyeball any binary-classification loss curve on Twitter and tell if it's real. Free skill.

6 · The gradient — derived by hand, then checked

We want \partial L/\partial w and \partial L/\partial b. Use the chain rule (S004 §2).

6.1 The chain

The famous cancellation: sigmoid + BCE gives \partial L/\partial z = a - y. Predicted probability minus target. No sigmoid derivatives to remember. This is the same clean pattern as softmax + cross-entropy from S005 §7 — and it is not a coincidence. It's the general result that when you use the natural log-likelihood loss for an exponential-family output (Bernoulli here, Categorical for softmax), the pre-activation gradient collapses to \hat p - y. See Bishop Pattern Recognition and Machine Learning (2006), §4.3.6, for the general derivation across the exponential family.

6.2 Batched gradient — the one-liner you'll memorise

For a batch of N examples with mean loss:

∂L/∂w = (1/N) · X.T @ (a - y)        shape (d,)
∂L/∂b = (1/N) · (a - y).sum()        shape ()

X.T @ (a - y) sums per-example contributions over the batch dim; dividing by N gives the mean-loss gradient. This one line — X.T @ (a - y) / N — is the entire gradient of every logistic regressor ever trained. Facebook's ad-ranking system, your bank's fraud model, the spam classifier that emailed you last week: same line.

6.3 The "logits" trick and why PyTorch uses binary_cross_entropy_with_logits

Once you've seen the cancellation, you realise you never needed to materialise a at all — the gradient depends only on z (through a = σ(z)) and y. And the forward loss can be written directly in z:

(Substitute a = 1/(1+e^{-z}) back in and simplify.) That expression uses log(1+e^z) — the "softplus" — which is numerically stable via log(1 + e^z) = max(z, 0) + \log(1 + e^{-|z|}). Both the forward loss and the backward gradient a - y are computed without ever forming a naively. PyTorch's binary_cross_entropy_with_logits and TensorFlow's sigmoid_cross_entropy_with_logits are exactly this fused operation. Rule of thumb: in production, always pass logits to the loss, never sigmoid outputs. Every serious codebase does this; every buggy one materialises the sigmoid first and eventually NaNs.

6.4 Numeric check — you MUST do this before you trust §7

The single most useful debugging tool for from-scratch neural nets is a two-sided finite-difference check:

def check_grad():
    np.random.seed(0)
    X = np.random.randn(5, 3)
    y = np.array([0, 1, 1, 0, 1], dtype=float)
    w = np.random.randn(3)
    b = 0.5
 
    def L(w_, b_):
        z = X @ w_ + b_
        a = 1 / (1 + np.exp(-z))
        return -(y*np.log(a) + (1-y)*np.log(1-a)).mean()
 
    # analytic
    z = X @ w + b
    a = 1 / (1 + np.exp(-z))
    dw = X.T @ (a - y) / len(y)
    db = (a - y).mean()
 
    # two-sided numeric
    h = 1e-6
    dw_num = np.zeros_like(w)
    for i in range(len(w)):
        w_p = w.copy(); w_p[i] += h
        w_m = w.copy(); w_m[i] -= h
        dw_num[i] = (L(w_p, b) - L(w_m, b)) / (2*h)
    db_num = (L(w, b+h) - L(w, b-h)) / (2*h)
 
    print("dw analytic:", dw)
    print("dw numeric :", dw_num)
    print("max abs err:", np.max(np.abs(dw - dw_num)))
    print("db analytic:", db, " db numeric:", db_num)
 
check_grad()
# max abs err on the order of 1e-8

Both agree to ~1e-8. If yours don't, either the analytic derivation is wrong or you have a shape bug — do not proceed to §7. This 20-line function has saved me more debug hours than any other snippet I own.

7 · One training run — put it all together

def train_neuron(X, y, lr=0.5, steps=200, verbose=True):
    n, d = X.shape
    w = np.zeros(d)
    b = 0.0
    history = []
    for t in range(steps):
        # forward
        z = X @ w + b
        a = 1 / (1 + np.exp(-z))
        loss = -(y*np.log(a+1e-9) + (1-y)*np.log(1-a+1e-9)).mean()
        # backward
        dw = X.T @ (a - y) / n
        db = (a - y).mean()
        # update (SGD)
        w -= lr * dw
        b -= lr * db
        history.append(loss)
        if verbose and t % 20 == 0:
            acc = ((a > 0.5) == y).mean()
            print(f"step {t:3d}  loss={loss:.4f}  acc={acc:.2%}")
    return w, b, history

Thirty lines. That is a trained neural network. Let it sink in.

7.1 Try it on a linearly-separable toy problem

np.random.seed(1)
n = 100
X0 = np.random.randn(n, 2) + np.array([-2, -2])
X1 = np.random.randn(n, 2) + np.array([+2, +2])
X  = np.vstack([X0, X1])
y  = np.concatenate([np.zeros(n), np.ones(n)])
 
w, b, hist = train_neuron(X, y, lr=0.3, steps=200)

Expected output:

step   0  loss=0.6931  acc=50.00%
step  20  loss=0.1204  acc=99.00%
step  40  loss=0.0678  acc=99.50%
step  60  loss=0.0470  acc=99.50%
...
step 180  loss=0.0202  acc=99.50%

Loss drops from 0.6931 (= \log 2, random) to near zero. Accuracy hits 99.5%. Congratulations — you just trained a neural network from first principles. No PyTorch, no autograd, no CUDA. Just the chain rule and NumPy.

7.2 The decision boundary — the money plot

Once trained, the neuron classifies x as class 1 iff σ(w·x + b) > 0.5, i.e., iff w·x + b > 0. That's a hyperplane — a straight line in 2D, a plane in 3D, a (d-1)-dimensional flat in d dimensions. Plot it:

import matplotlib.pyplot as plt
plt.scatter(X[:,0], X[:,1], c=y, cmap='coolwarm', alpha=0.6)
xs = np.linspace(-5, 5, 100)
ys = -(w[0]*xs + b) / w[1]        # solve w·x + b = 0 for x_2
plt.plot(xs, ys, 'k--', lw=2)
plt.axis('equal'); plt.title("Neuron decision boundary")
plt.show()

A diagonal line separates the two blobs. That is what one neuron does — it draws one straight line. Nothing more, nothing less. It's a very good straight line; it's a nothing-else straight line.

7.3 The loss curve — read it like a doctor reads an X-ray

Plot hist:

plt.plot(hist); plt.xlabel("step"); plt.ylabel("BCE loss"); plt.yscale('log')

A well-trained logistic regressor's loss curve on separable data looks like a smooth exponential decay on a log scale — a straight line pointing down. If yours has:

• Spikes → learning rate too big
• A flat plateau at ~0.693 → gradient is zero (bug), or data is not linearly separable (welcome to §8)
• Sudden jump to NaN → sigmoid overflow (revisit §4.1)
• Steady decrease then flat above zero → this is fine and expected for non-separable data

Diagnosing training runs from loss curves alone is a superpower. It's mostly pattern recognition, and it starts here.

8 · Where one neuron breaks — XOR and the 1969 winter

Try the same code on this famous dataset:

# XOR
X = np.array([[0,0], [0,1], [1,0], [1,1]], dtype=float)
y = np.array([0, 1, 1, 0], dtype=float)
w, b, hist = train_neuron(X, y, lr=1.0, steps=1000, verbose=False)
print("final loss:", hist[-1], "final acc:", ((forward(X, w, b)[0] > 0.5) == y).mean())
# final loss: ~0.693   final acc: 0.50

Accuracy plateaus at 50%. Loss stops at \log 2. The neuron cannot solve XOR. Look at the plot: the 1s are at (0,1) and (1,0); the 0s are at (0,0) and (1,1). No straight line separates them. You can rotate, translate, and scale a line all you want — the 1s are on opposite corners.

This is Minsky & Papert's 1969 result, and it froze neural-network research for 15 years, until backpropagation through multi-layer networks became widespread in the mid-1980s. Sit with the plot for a minute. This is the failure that made the field.

8.1 The fix, previewed

Stack two neurons in a hidden layer, then combine them with an output neuron:

With, say, w_1 = [1, 1], b_1 = -0.5 (a line separating (0,0) from the rest) and w_2 = [1, 1], b_2 = -1.5 (a line separating (1,1) from the rest) and v = [1, -1], you get an XOR solver by hand, with no training at all. Session 009 builds exactly this and derives its gradient in Session 010.

9 · The full computation graph, forward and backward

Trace the backward arrows once with your finger. Every deep network's backprop is this graph, repeated for every layer, with more edges. If you can do this graph, you can do backprop on a 100-layer transformer — the topology is bigger but no single node is harder than what you just did. This is not a motivational speech. It is literally true, and Session 010 will prove it to you.

10 · Modern 2025 twist — where the "single neuron" idea reappears in LLM land

Three places in the current stack (2025) where "one neuron trained with a BCE-like loss" is still the entire game:

• Reward models for RLHF. A reward model is a transformer with a single-scalar head (one output neuron) trained with a pairwise-preference loss — a close cousin of BCE. The InstructGPT paper (Ouyang et al., 2022) and every follow-up (Llama 2, Llama 3, Claude, GPT-4 RLHF pipelines) sit on this. The scalar head is a nn.Linear(d, 1) — literally the neuron you just built.
• DPO / KTO / ORPO (2024–2025). DPO (Rafailov et al., 2023, NeurIPS 2023 best-paper runner-up) skips the reward model but keeps a BCE-shaped loss on preference pairs. KTO (Ethayarajh et al., 2024) is even closer to plain per-example BCE. If you understood the a - y cancellation in §6, DPO's log-ratio gradient will feel familiar.
• Router logits in Mixture-of-Experts. The gate that decides which expert a token goes to (Mixtral 8×22B, DeepSeek-V3, GPT-4 rumoured) is a linear layer with a softmax — a many-class generalisation of your neuron with the softmax + CE cancellation.

The scale changes; the shape does not.

Further reading:

• Ouyang et al., "Training language models to follow instructions with human feedback" (2022) — https://arxiv.org/abs/2203.02155
• Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (2023) — https://arxiv.org/abs/2305.18290
• Ethayarajh et al., "KTO: Model Alignment as Prospect Theoretic Optimization" (2024) — https://arxiv.org/abs/2402.01306
• DeepSeek-V3 Technical Report (2024) — https://arxiv.org/abs/2412.19437 — see §2.1 for the multi-head latent attention and §2.2 for the auxiliary-loss-free routing.

11 · War stories

12 · Try it yourself

import torch, torch.nn.functional as F
X_t = torch.tensor(X, dtype=torch.float32)
y_t = torch.tensor(y, dtype=torch.float32)
w = torch.zeros(2, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
opt = torch.optim.SGD([w, b], lr=0.3)
for t in range(200):
    z = X_t @ w + b
    loss = F.binary_cross_entropy_with_logits(z, y_t)
    opt.zero_grad(); loss.backward(); opt.step()

13 · Recap

You built a single sigmoid neuron in 30 lines of NumPy, trained it with SGD on a 2D toy problem, and watched a hand-written decision boundary snap into place. You derived \partial L/\partial z = a - y from the chain rule and verified it to 1e-8 against a two-sided finite-difference check. You broke it on XOR and understood — in your bones, not just on a slide — why the field needed multi-layer networks.

Every line you wrote today reappears at every scale of every model in production in 2025 — as nn.Linear, as binary_cross_entropy_with_logits, as the reward-model head of an RLHF pipeline. The neuron is the atom.

🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________________________________________

Spaced review: re-read §3 (anatomy), §6 (gradient), and §8 (XOR) in 24 hours. Revisit the full session on day 7 — by then S008–S010 will make the "why we care" much sharper.

Next session (S008): activation functions — sigmoid's vanishing gradient, why ReLU took over in 2012, why GELU took over in transformers around 2018, and where SwiGLU and 2025's xIELU fit in.

Sticky note (keep on your desk):

• Diagram: x ─* w─► z ─σ─► a ─L─► loss. Draw it before every from-scratch neural net you write for the next 6 months.
• Equation: ∂L/∂z = a − y for sigmoid + BCE. Memorise.
• Snippet: dw = X.T @ (a - y) / N. The entire gradient of a logistic regressor, in one line of NumPy.
• Smoke test: step-0 BCE loss on a balanced binary problem is log 2 ≈ 0.693. Anything else = bug.

Previous: ← DL S006 · Optimization Intuition + Numerical Stability · Next: DL S008 · Activations — sigmoid, tanh, ReLU, GELU, SwiGLU, xIELU →