One neuron: logistic regression, stable loss, and its boundary

Course map · DL007 · One neuron, inspected completely

0 · The story — what a perceptron does and does not establish

A hard-threshold perceptron and a sigmoid logistic regressor both form an affine score, but have different outputs and learning rules. Rosenblatt's1958 paper describes a probabilistic model of information storage and organization; it should not be reduced to the unsupported claim that the whole historical Mark I machine was exactly one modern sigmoid neuron. Minsky and Papert's1969 Perceptrons studied limitations of perceptron classes; XOR's elementary nonseparability can be proved here without attributing an entire AI winter to one theorem.

The1986 Rumelhart–Hinton–Williams paper is an important primary account of learning internal representations by backpropagation. It is not evidence that all earlier deep networks failed, that backprop first existed in1986, or that XOR trains in one epoch. The useful lesson is representational: composition can express boundaries one affine score cannot. We will inspect that limitation directly rather than repeat invented personal or historical anecdotes.

1 · What you'll build today — the one-sentence spec

Train a sigmoid neuron on a small two-dimensional task using a stable Bernoulli loss, verify every weight and bias derivative, inspect the boundary, compare the exact CPU framework update, and prove why the original-feature model cannot separate XOR. Then solve XOR with an engineered interaction and with a correctly specified hidden representation.

2 · Checkpoint · you should already know

DL002's dot product, DL003 broadcasting, DL004's chain rule, DL005's likelihood and DL006's update/reduction conventions. A probability model and a decision rule are different: first compute p, then choose a threshold based on a task's costs or evaluation policy.

3 · Anatomy of one neuron

For x,w in $\mathbb R^d$ and scalar b,

The score z is a real log-odds parameter, not a probability; a is the modeled Bernoulli probability. A positive coefficient increases the log odds per unit feature change holding other features fixed. Its magnitude depends on feature units and correlations; it is not automatically a causal effect. The intercept is the score at x=0, not necessarily the marginal class prior.

3.1 A concrete 3-input example you can hold in your head

With x=(1,2,−1),w=(.5,−.3,.8),b=.1, z=.5−.6−.8+.1=−.8. Then a≈.3100255. Every operation is visible: three products, a sum, a bias and a nonlinear scalar function. At threshold.5 this is a class0 decision.

3.2 Batched version — shapes are part of the specification

For X:(N,d),w:(d,),b:(), z=X@w+b has(N,), as do a and y. The mean loss is scalar; dw has(d,) and db is scalar. A column target(N,1) must not broadcast against a vector prediction(N,): it makes(N,N). Assert the intended contract before reducing.

3.3 The neuron IS logistic regression

A sigmoid of an affine score trained with Bernoulli negative log-likelihood is logistic regression. torch.nn.Linear(d,1) computes an affine score, with weight shape(1,d), and does not itself apply sigmoid. Deep networks add representations and composition; this one-neuron interpretation does not exhaust attention, normalization or all model architectures.

The statistical model gives a convex loss in these fixed-feature parameters, but convexity does not guarantee identical numerical trajectories, a unique solution under collinearity, a finite optimum on separable data, or freedom from rate tuning.

4 · The forward pass — one affine score and one sigmoid

Use z=X@w+b for a batch. The stable sigmoid below evaluates exp(−abs(z)), never a positive exponent for finite z, then selects an algebraically equivalent branch. Values in extreme tails can round to0 or1; a stable sigmoid alone is not a stable logarithmic loss.

4.1 Numerical hygiene — the stable sigmoid

For z≥0 use1/(1+exp(−z)); for z < 0 use exp(z)/(1+exp(z)). np.where evaluates both branches, so writing one branch with exp(−z) over all z can still overflow. The retained implementation uses a common bounded exponential and is safe from that particular hazard. For negative float32 logits of sufficiently large magnitude, naïve exp(−z) overflows; the original anecdote's positive92 had the sign reversed.

4.2 A hand trace of the forward pass on 4 examples

With rows(1,2),(0,−1),(−2,3),(4,0), w=(.5,−1),b=.2, logits are −1.3,1.2,−3.8,2.2 and sigmoid outputs approximately.214165,.768525,.021881,.900250. Compute one row on paper, then verify all four. The boundary is determined by z, not by the rounded display probabilities.

5 · The loss — binary cross-entropy

For y=1 only−log a remains; for y=0 only−log(1−a). This is Bernoulli negative log-likelihood. Independent binary labels may each use BCE; mutually exclusive classes usually require a joint categorical model. Weights and reduction conventions alter the gradient scale.

5.1 Batched implementation

The stable logit form is

Mean it over the intended examples. Clipping probabilities to[ε,1−ε] changes the mathematical objective; if one then uses the unclipped a−y derivative, forward and backward disagree wherever clipping is active. The logit form avoids that mismatch for finite ordinary inputs, though no floating-point expression is universally bulletproof.

5.2 Where does -log(0.5) = 0.693 come from?

At exactly zero logits every binary label has loss ln2≈.693147, regardless of balance. It is a good fixture for an unweighted objective, not a universal starting-loss test for nonzero initialization, weighting, smoothing or custom reductions. An unbalanced all-zero classifier can have high accuracy; loss and accuracy measure different things.

6 · The gradient — derived by hand, then checked

6.1 The chain

Inside 0 < a < 1, $dL/da=(a-y)/(a(1-a))$. Multiplying by $da/dz=a(1-a)$ cancels the factors, giving $dL/dz=a-y$. Then $dL/dw=(a-y)x$ and $dL/db=a-y$. The stable logit loss has this derivative without numerically dividing by a vanishing probability factor.

For squared probability error $(a-y)^2$, the derivative is $2(a-y)a(1-a)$; a confidently wrong saturated output can receive a tiny correction. This binary formula is not the multiclass softmax-MSE Jacobian from DL005. The BCE cancellation helps the output derivative; hidden sigmoids still contribute their local derivatives multiplied by weight matrices. A blanket4^−depth bound omitting the weights is invalid.

6.2 Batched gradient — reduce once

At x=(2,−1),w=(.5,1),b=0,y=1, z=0,a=.5 and gradients are(−1,.5),−.5. The transpose collects per-example feature contributions. Large feature units amplify dw; normalize using training-fitted statistics when appropriate.

6.3 The logits trick and the framework contract

PyTorch2.9 BCEWithLogitsLoss combines sigmoid and BCE using a numerically stable formulation. Pass raw logits and same-shaped floating targets, not a second sigmoid output. nn.Linear stores output-by-input weights; transpose them when comparing with a NumPy row-batch X@w convention. A probability display can still use sigmoid separately for interpretation.

6.4 Numeric check — test before trusting training

Central differences perturb each parameter on both sides with all data fixed. Their truncation error is second order under smoothness, but digits of agreement depend on scale, dtype and h. A claim of eleven correct digits merely from h=1e−6 is not justified. The test below checks all coordinates against the same stable loss and deliberately rejects shape mismatch.

7 · One training run — put it all together

The exact preserved training program uses four separable rows, not sampled real-world data. It is a useful implementation fixture and nothing more.

import numpy as np
 
X = np.array([[-2., -1.], [-1., 1.], [1., -1.], [2., 1.]])
y = np.array([0., 0., 1., 1.])
w = np.zeros(2); b = 0.
 
def sigmoid(z):
    # exp(-abs(z)) never overflows for finite z
    e = np.exp(-np.abs(z))
    return np.where(z >= 0, 1/(1+e), e/(1+e))
 
def loss(w, b):
    z = X @ w + b
    return np.mean(np.maximum(z, 0) - y*z + np.log1p(np.exp(-np.abs(z))))
 
initial = loss(w, b)
for _ in range(400):
    residual = sigmoid(X @ w + b) - y
    dw = X.T @ residual / len(y)
    db = residual.mean()
    w -= .1 * dw
    b -= .1 * db
p = sigmoid(X @ w + b)
assert loss(w, b) < .05
assert ((p >= .5) == y).all()
assert np.isfinite(sigmoid(np.array([-1000., 1000.]))).all()
print('loss', round(float(initial), 6), '->', round(float(loss(w, b)), 6))

All gradients are computed before updating either weight or bias. On separable data, unregularized logistic loss can approach zero while parameter norms grow without a finite minimizing parameter vector. Perfect training labels therefore do not establish calibration or a converged finite optimum.

7.1 Try it on a linearly-separable toy problem

The original Gaussian blobs were not guaranteed separable: Gaussian tails overlap, even if one small sampled dataset separates. The extended program uses bounded uniform jitter around well-separated centers so the generating supports separate. It records parameter snapshots and independently generated held-out examples. This is still a synthetic model, not a generalization claim about fraud or recommendations.

7.2 The decision boundary — a geometric output

At threshold.5 the boundary is $w_1x_1+w_2x_2+b=0$. If w2≠0, draw $x_2=-(w_1x_1+b)/w_2$; if w2=0,w1≠0 draw the vertical line x1=−b/w1. If w=0 the model is constant and has no ordinary separating line. Multiplying both w and b by a positive scalar preserves the boundary while changing confidence; scaling w alone generally moves it.

At threshold τ strictly between0 and1, replace zero by log(τ/(1−τ)). The boundary remains affine. A sigmoid changes the confidence profile, not the family of level sets.

7.3 The loss curve — measurements, not diagnosis by silhouette

Record loss, gradient norm, parameter norm, class balance, step and boundary coefficients. Spikes can suggest a rate problem, but can also come from changing batches or reduction. A plateau can reflect capacity, regularization, a zero gradient, numerical precision or an optimum. Logistic loss on separable data need not decay exponentially or draw a straight line on a logarithmic plot. Test hypotheses using the same data and pre-step state.

import numpy as np
import torch
import torch.nn.functional as F
 
def neuron_objective(X,y,w,b):
    assert X.ndim==2 and y.shape==(len(X),) and w.shape==(X.shape[1],)
    assert np.isfinite(X).all() and np.isfinite(w).all() and np.isfinite(b)
    assert ((y>=0)&(y<=1)).all()
    z=X@w+b
    value=float(np.mean(np.maximum(z,0)-y*z+np.log1p(np.exp(-np.abs(z)))))
    r=sigmoid(z)-y
    return value,X.T@r/len(y),float(r.mean())
def train_neuron(X,y,lr=.3,steps=200):
    w=np.zeros(X.shape[1]);b=0.;history=[]
    for step in range(steps):
        L,dw,db=neuron_objective(X,y,w,b)
        history.append((L,w.copy(),b))
        w-=lr*dw;b-=lr*db
    return w,b,history
 
x=np.array([1.,2.,-1.]);w=np.array([.5,-.3,.8]);assert np.isclose(x@w+.1,-.8)
rows=np.array([[1,2],[0,-1],[-2,3],[4,0]],float)
np.testing.assert_allclose(rows@np.array([.5,-1.])+.2,[-1.3,1.2,-3.8,2.2])
rng=np.random.default_rng(71)
Xc=rng.normal(size=(5,3));yc=np.array([0,1,1,0,1.]);wc=rng.normal(size=3);bc=.5
L,dw,db=neuron_objective(Xc,yc,wc,bc)
for j in range(3):
    d=np.eye(3)[j]*1e-5
    fd=(neuron_objective(Xc,yc,wc+d,bc)[0]-neuron_objective(Xc,yc,wc-d,bc)[0])/2e-5
    assert abs(fd-dw[j])<1e-8
fd=(neuron_objective(Xc,yc,wc,bc+1e-5)[0]-neuron_objective(Xc,yc,wc,bc-1e-5)[0])/2e-5
assert abs(fd-db)<1e-8
try: neuron_objective(Xc,yc[:,None],wc,bc)
except AssertionError: pass
else: raise AssertionError('all-pairs labels accepted')
 
def blobs(n):
    X=np.vstack([rng.uniform(-.5,.5,(n,2))-2,rng.uniform(-.5,.5,(n,2))+2])
    y=np.r_[np.zeros(n),np.ones(n)];order=rng.permutation(2*n)
    return X[order],y[order]
Xt,yt=blobs(40); Xv,yv=blobs(20)
w,b,hist=train_neuron(Xt,yt)
assert hist[-1][0]<hist[0][0] and ((sigmoid(Xv@w+b)>.5)==yv).all()
# Match the same initialization, order, loss and 200 full-batch CPU updates.
X_t=torch.tensor(Xt,dtype=torch.float64); y_t=torch.tensor(yt,dtype=torch.float64)
w_t=torch.zeros(2,dtype=torch.float64,requires_grad=True);b_t=torch.zeros((),dtype=torch.float64,requires_grad=True)
opt=torch.optim.SGD([w_t,b_t],lr=.3)
for _ in range(200):
    opt.zero_grad();F.binary_cross_entropy_with_logits(X_t@w_t+b_t,y_t).backward();opt.step()
np.testing.assert_allclose(w,w_t.detach().numpy(),atol=1e-12)
assert abs(b-b_t.item())<1e-12
# Inspectable boundary snapshots instead of an unverified interactive/GIF claim.
for step in (0,1,20,199):
    L,ww,bb=hist[step]
    print('boundary',step,'loss',round(L,6),'w',ww.tolist(),'b',round(bb,6))
print('all-coordinate gradients, synthetic held-out fit and exact CPU SGD parity passed')

The following standard-library SVG export makes the boundary and loss curve inspectable without a plotting dependency, server or interactive UI. Open the saved file locally. The vertical/constant cases are handled explicitly; the stored coefficients remain the numerical source of truth.

from pathlib import Path
 
parts=['<svg xmlns="http://www.w3.org/2000/svg" width="760" height="340" viewBox="0 0 760 340">',
       '<rect width="760" height="340" fill="white"/>',
       '<text x="20" y="22">Synthetic neuron: boundary snapshots and mean BCE</text>',
       '<defs><clipPath id="panel"><rect x="20" y="40" width="300" height="260"/></clipPath></defs>']
def px(x):return 20+(x+3)/6*300
def py(y):return 300-(y+3)/6*260
for tick in (-3,0,3):
    parts.append(f'<text x="{px(tick)}" y="315" font-size="11">{tick}</text>')
    parts.append(f'<text x="2" y="{py(tick)}" font-size="11">{tick}</text>')
parts.append('<text x="140" y="335" font-size="12">x1; x2 increases upward</text>')
parts.append('<text x="22" y="38" font-size="11">Gray step 1; orange 20; black 199</text>')
for fraction in (0.,.5,1.):
    yy=300-250*fraction
    parts.append(f'<line x1="380" x2="730" y1="{yy}" y2="{yy}" stroke="#ddd"/>')
    parts.append(f'<text x="335" y="{yy}" font-size="10">{fraction*hist[0][0]:.3f}</text>')
parts.append('<g clip-path="url(#panel)">')
for point,label in zip(Xt,yt):
    color='blue' if label else 'red'
    parts.append(f'<circle cx="{px(point[0]):.3f}" cy="{py(point[1]):.3f}" r="2.5" fill="{color}"/>')
for step,color in [(1,'gray'),(20,'orange'),(199,'black')]:
    _,ww,bb=hist[step]
    if abs(ww[1])>1e-12:
        xx=np.array([-3.,3.]);yy=-(ww[0]*xx+bb)/ww[1]
        parts.append(f'<line x1="{px(xx[0])}" y1="{py(yy[0])}" x2="{px(xx[1])}" y2="{py(yy[1])}" stroke="{color}"/>')
    elif abs(ww[0])>1e-12:
        xline=px(-bb/ww[0]);parts.append(f'<line x1="{xline}" x2="{xline}" y1="40" y2="300" stroke="{color}"/>')
parts.append('</g>')
points=' '.join(f'{380+350*i/(len(hist)-1):.2f},{300-250*L/hist[0][0]:.2f}' for i,(L,_,_) in enumerate(hist))
parts.append(f'<polyline points="{points}" fill="none" stroke="black"/>')
parts.append('<text x="380" y="325">Steps 0 to 199; linear loss axis from 0 to initial BCE</text></svg>')
figure=Path('batch5-dl-foundations-neuron-plot.svg');figure.write_text(''.join(parts))
assert figure.read_text().count('<circle')==len(Xt)
print('saved',figure)

8 · Where one neuron breaks — XOR

For strict class1 score z>0, positives(1,0),(0,1) require w1+b>0 and w2+b>0; negatives(0,0),(1,1) require b≤0 and w1+w2+b≤0. Adding positive inequalities gives w1+w2+2b>0, hence w1+w2+b>−b≥0, a contradiction. A threshold shift can be absorbed into b and cannot help.

Zero-initialized logistic regression on balanced four-row XOR has zero gradient by symmetry and loss ln2. It stays at probability.5 and, with a fixed all-one or all-zero tie rule,50% accuracy. This is not a claim every affine classifier gets at most50%: a line can classify three corners correctly. It cannot get all four. A200,000-sample random parameter search is not a proof of impossibility.

8.1 The fix, previewed

Let s=x1+x2. Hard hidden thresholds $h_1=\mathbf1[s>.5]$, $h_2=\mathbf1[s>1.5]$ yield h1−h2 equal to XOR on the four binary inputs. For smooth hidden units use sigmoid(10(s−.5)) and sigmoid(10(s−1.5)), then output score $10(h_1-h_2-.5)$. The output bias−.5 inside this expression matters: the older difference of two monotone sigmoids with no output bias stays positive and fails.

Alternatively add feature x1x2; score $x_1+x_2-2x_1x_2-.5$ is negative at equal bits and positive at unequal bits. This is linear in the engineered feature coordinates and nonlinear in the original two. A hidden layer can learn interactions, but it need not discover this exact feature or train successfully under any initialization.

Xxor=np.array([[0,0],[0,1],[1,0],[1,1]],float); yxor=np.array([0,1,1,0.])
w0,b0,h0=train_neuron(Xxor,yxor,lr=1.,steps=30)
np.testing.assert_allclose(w0,0); assert b0==0
assert np.isclose(h0[-1][0],np.log(2))
assert ((Xxor@np.array([1.,1.])-.5>0)==yxor).mean()==.75
s=Xxor.sum(1); h1=sigmoid(10*(s-.5));h2=sigmoid(10*(s-1.5))
assert ((10*(h1-h2-.5)>0)==yxor).all()
# Original missing-bias construction: all positive, so it fails.
wrong=sigmoid(s-.5)-sigmoid(s-1.5)
assert (wrong>0).all() and not ((wrong>0)==yxor).all()
features=np.c_[Xxor,Xxor[:,0]*Xxor[:,1]]
assert ((features@np.array([1.,1.,-2.])-.5>0)==yxor).all()
wf,bf,hf=train_neuron(features,yxor,lr=1.,steps=1500)
assert ((features@wf+bf>0)==yxor).all()
print('XOR symmetry, 75-percent affine witness, corrected hidden solver and engineered-feature fit passed')

9 · The full computation graph, forward and backward

Each local derivative belongs to its actual operation. Matrix products, normalization and attention have richer Jacobians than the scalar example; the same compositional principle applies, not the claim that every operation is equally easy. Retain a small numeric oracle when scaling up.

10 · Modern connections — scalar heads and preference losses

InstructGPT's reward model produces scalar scores trained on preference comparisons. A typical pairwise loss is $-\log\sigma(r_w-r_l)$: at equal rewards the preferred-score derivative is−.5 and rejected+.5. The scalar score is not itself a calibrated probability and an arbitrary common offset cancels. The transformer representation is still most of the model; one output head does not make the entire reward model “one neuron.”

DPO applies a sigmoid preference loss to a scaled difference of policy/reference sequence log-probability ratios. Its derivative with respect to that scalar is sigmoid minus1, but policy parameters reach it through many token probabilities, not simply one X.T@residual. KTO's prospect-theoretic desirable/undesirable objective is not ordinary BCE; ORPO uses an odds-ratio penalty plus supervised likelihood. Their shared primitives do not make their full gradients interchangeable.

MoE routers produce expert scores and use routing-specific selection, normalization and balancing. DeepSeek-V3 uses sigmoid affinity scores with auxiliary-loss-free balancing plus a small sequence-level balancing term; calling every router a softmax trained with CE would be wrong. No unsupported GPT-4 architecture rumour is required for this comparison.

11 · Failure cases and model choice

• Initialization: zero weights are a valid starting point for this single logistic unit. Identical hidden-unit parameters can preserve symmetry in a multilayer model; random initialization is one way to break it. Activation and downstream weights also matter.
• Extreme probabilities: sigmoid(50) may round to1. A probability-space 0*log(0) can be NaN even when the intended limiting term is0. Use a stable logits loss, not a fabricated claim that every probability-based codebase fails.
• Rate: feature scale and curvature determine useful step sizes. A tiny rate barely moves; a large one need not immediately produce NaNs with a stable loss. Measure loss and parameter updates for the actual problem.
• Splits: inspect class counts and fit preprocessing on training data only. Shuffling is not sufficient for temporal, grouped or duplicate leakage. A sorted split with one class missing is a sampling problem, not proof that any99% training score was cheating.

A fixed-feature logistic model offers an inspectable convex baseline and cheap inference. Engineered interactions retain convexity in their coefficients but increase feature count and domain assumptions. A hidden layer can learn representations at additional tuning and validation cost. Residual patterns may suggest missing interactions, but scattered errors do not prove irreducible noise. Use held-out evidence before adding capacity or claiming interpretability.

12 · Try it yourself

Boundary animation, with a complete recipe: use the recorded (loss,w,b) snapshots; for each frame choose the nonvertical line formula only when abs(w2) exceeds a tolerance, otherwise draw a vertical line or a constant-model label. Keep scatter data and axes fixed, label the step/loss, and compare first/final boundaries. Saving a GIF is optional presentation, not an optimization test. The program prints those snapshots; no animation artifact is claimed here.

Compare NumPy with PyTorch: the program uses identical float64 data, zeros,200 steps and mean BCE under SGD. It checks final coefficients, not merely similar accuracy. Changing dtype can require looser tolerances; adding weight decay changes the model.

Break it deliberately: compare lr100 and1e−6 from identical initial states; record loss and coefficient norms rather than requiring a predetermined curve. Replace the stable loss by clipped BCE only as a diagnostic and finite-difference its actual objective. Shrinking blob separation creates overlap, so a nonzero optimum can be correct. Never infer implementation failure merely because accuracy is no longer100%.

Multi-label extension: X:(N,d),W:(d,2),b:(2,),Y:(N,2). With a mean over all2N label entries, G=(sigmoid(XW+b)−Y)/(2N), dW=X.T@G and db=G.sum(0). A sum over labels followed by a mean over examples uses1/N instead; name the convention. It remains one affine layer with two Bernoulli outputs, not a two-layer network.

xx=np.array([[1.,2.],[-1.,.3],[.2,-.4]])
yy=np.array([[1.,0.],[0.,1.],[1.,1.]])
ww=np.array([[.2,-.3],[.1,.4]]);bb=np.array([.1,-.2])
z=xx@ww+bb;G=(sigmoid(z)-yy)/yy.size;dw=xx.T@G;db=G.sum(0)
def multilabel(W,b):
    z=xx@W+b
    return np.mean(np.maximum(z,0)-yy*z+np.log1p(np.exp(-np.abs(z))))
for idx in np.ndindex(ww.shape):
    d=np.zeros_like(ww);d[idx]=1e-5
    assert abs((multilabel(ww+d,bb)-multilabel(ww-d,bb))/2e-5-dw[idx])<1e-8
for j in range(2):
    d=np.eye(2)[j]*1e-5
    assert abs((multilabel(ww,bb+d)-multilabel(ww,bb-d))/2e-5-db[j])<1e-8
print('independent multi-label mean reduction and all gradients passed')

13 · Recap

A neuron projects features to one score, converts that score to a Bernoulli probability, and learns by the stable a−y output derivative. Its thresholded boundary remains affine in the supplied features. Computation correctness, representational capacity and generalization are three separate questions. We checked the first, proved a limit on the second, and bounded the synthetic evidence for the third.

🧠 Retention scaffold

1. Two-line forward pass? z=X@w+b, then a=sigmoid(z). Both have(N,), matching y; only the final mean is scalar.
2. The original hand trace? For(1,2),(.5,−1),.2, z=−1.3 and a≈.214. The negative score predicts class0 at threshold.5.
3. Why a−y? The reciprocal a(1−a) in the log-loss derivative cancels sigmoid's derivative. The categorical twin is softmax minus the target distribution, under normalization.
4. Extreme-z failures? Negative z can overflow naïve exp(−z); large-magnitude z can round sigmoid to0/1 and invalidate logarithms. A bounded-exponential sigmoid and a logits BCE address different parts of the issue.
5. Why no XOR? Add the two positive-corner inequalities and contradict the two negative-corner conditions. Minsky–Papert1969 is historical context; the explicit inequalities are the proof, not the year.
6. Two outputs? dW is(d,2), db(2,). Divide by2N for a mean over every binary label; independent BCE is not a mutually exclusive categorical head.
7. Where is DPO's sigmoid? It wraps β times the preferred-minus-rejected sequence log-ratio difference. At zero margin its scalar derivative is−.5, but chain through the policy's token probabilities to obtain parameter gradients.

Spaced review: tomorrow reconstruct the four-example trace and all-coordinate check; next week prove XOR impossibility and repair the missing-bias hidden solver without looking. Your sticky note is dw=X.T@(a-y)/N, plus the loss reduction and feature-shape contract—not a promised accuracy.

Primary references and boundary

PyTorch2.9 BCEWithLogitsLoss, Linear sourcev2.9.0, InstructGPT, DPOv3, KTO, ORPO and DeepSeek-V3v2 support the scoped modern claims. Rosenblatt publisher DOI metadata and Rumelhart–Hinton–Williams publisher DOI metadata identify the1958 and1986 works and the latter cites Minsky–Papert1969 without claiming a single cause of research funding changes. Tests use synthetic data and CPU only; no reward model, preference tuning, animation, pretrained model or real-world service benchmark is reproduced.

Previous: Optimization · Next: Activation functions