S086 · Logistic Regression — Sigmoid, Cross-Entropy, from Scratch

Module M11: Classical ML · Session 86 of 130 · Track: ML 🎯

What you'll be able to do after this session

• Explain Logistic Regression to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S085 — Linear Regression from Scratch (numpy)

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Logistic Regression, Clearly Explained — StatQuest — the intuition in one shot.
• 🎥 Deep dive (30–60 min) · Logistic Regression Details — Maximum Likelihood — StatQuest — where the loss function actually comes from.
• 🎥 Hands-on demo (10–20 min) · Logistic Regression in Python from scratch — Assembly AI — sigmoid + cross-entropy + gradient descent, live-coded.
• 📖 Canonical article · scikit-learn — Logistic Regression — the reference API and solvers.
• 📖 Book chapter / tutorial · ISLR ch.4 — Classification (free PDF) — clean walk-through with worked examples.
• 📖 Engineering blog · Meta — Improving Facebook's Ads Ranking — how logistic regression powered ads ranking at web scale.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Classification, first principles.

Linear regression predicts numbers. Logistic regression predicts probabilities — the chance an email is spam, the chance a click leads to a conversion, the chance a patient has diabetes. The output is always between 0 and 1.

The trick to going from an unbounded linear score to a probability is a single squashing function: the sigmoid, σ(z) = 1 / (1 + e⁻ᶻ). Feed it any real number, get out something in (0, 1). Large positive input → close to 1. Large negative → close to 0. Zero → exactly 0.5. Draw it once, and you'll recognise it forever — it's the "S-curve".

So the whole model is: p = σ(w·x + b). Compute the linear score, squash to probability. If p > 0.5, predict class 1; else class 0. That's it.

The training objective changes though. MSE isn't ideal for probabilities — it gives a non-convex loss surface. Instead we use cross-entropy loss (also called log loss or binary cross-entropy): L = -[y·log(p) + (1-y)·log(1-p)]. Intuitively: if the true label is 1 and you predicted 0.9, you're barely penalised; if you predicted 0.01, you're punished heavily (log(0.01) is a large negative number). Cross-entropy comes directly from maximum likelihood estimation — it's the "right" loss under the assumption that y follows a Bernoulli distribution.

Logistic regression is a linear classifier — the decision boundary is a line (or hyperplane). If your two classes are separable only by a curve, plain logistic regression will underfit. Fix: add polynomial features, or move to a non-linear model (tree, SVM with RBF, neural net).

Gotcha to carry forever: The output of logistic regression is a probability, not a class. Whether you threshold at 0.5, 0.3, or 0.9 is a business decision — usually driven by the cost of false positives vs false negatives.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The whole model in one flow:

Sigmoid table — memorise the shape:

Worked example — spam classifier by hand. Two features: number of suspicious words (x₁) and whether the sender is in your address book (x₂ ∈ ). After training the model gives w = [1.2, -3.0], b = -2.5.

New email: 5 suspicious words, sender not in address book. z = 1.2·5 + (-3.0)·0 + (-2.5) = 3.5. p = σ(3.5) = 1/(1+e⁻³·⁵) ≈ 0.971. Classify as spam.

Same email but from your address book: z = 3.5 - 3.0 = 0.5, p = σ(0.5) ≈ 0.622. Still spam if threshold is 0.5, but you might raise the threshold to 0.9 for known contacts.

Confusion matrix — how you actually evaluate a classifier:

From these: Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = harmonic mean. Accuracy is almost always the wrong metric — on a 99% not-spam dataset, "always predict not spam" scores 99% and is useless.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Logistic regression from scratch — sigmoid, cross-entropy, gradient descent.

# pip install numpy scikit-learn
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
 
rng = np.random.default_rng(42)
data = load_breast_cancer()
X, y = data.data, data.target      # 569 samples, 30 features, y in {0,1}
X = StandardScaler().fit_transform(X)
X = np.hstack([np.ones((len(X), 1)), X])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
 
# ---------- 1. Sigmoid (numerically stable) ----------
def sigmoid(z):
    return np.where(z >= 0, 1.0 / (1.0 + np.exp(-z)),
                            np.exp(z) / (1.0 + np.exp(z)))
 
# ---------- 2. Cross-entropy loss ----------
def bce(y, p, eps=1e-15):
    p = np.clip(p, eps, 1 - eps)
    return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
 
# ---------- 3. Train with gradient descent ----------
def fit(X, y, lr=0.1, epochs=500):
    w = rng.normal(size=X.shape[1]) * 0.01
    for ep in range(epochs):
        p = sigmoid(X @ w)
        grad = X.T @ (p - y) / len(y)     # elegant! same as linear-reg gradient
        w -= lr * grad
        if ep % 100 == 0:
            print(f"  ep={ep:3d}  loss={bce(y, p):.4f}")
    return w
 
w = fit(X_tr, y_tr)
 
# ---------- 4. Evaluate ----------
p_te = sigmoid(X_te @ w)
y_hat = (p_te > 0.5).astype(int)
print("
My from-scratch model:")
print(confusion_matrix(y_te, y_hat))
print(classification_report(y_te, y_hat, digits=3))
print(f"AUC = {roc_auc_score(y_te, p_te):.4f}")
 
# ---------- 5. Verify against sklearn ----------
sk = LogisticRegression(max_iter=1000, C=1e12).fit(X_tr, y_tr)   # C huge = ~no regularisation
print("
Sklearn:")
print(classification_report(y_te, sk.predict(X_te), digits=3))
print(f"AUC = {roc_auc_score(y_te, sk.predict_proba(X_te)[:,1]):.4f}")
 
# ---------- 6. Show threshold trade-off ----------
for t in (0.3, 0.5, 0.7, 0.9):
    yh = (p_te > t).astype(int)
    tp = ((yh==1)&(y_te==1)).sum(); fp = ((yh==1)&(y_te==0)).sum()
    fn = ((yh==0)&(y_te==1)).sum()
    prec = tp/max(tp+fp,1); rec = tp/max(tp+fn,1)
    print(f"  threshold={t}: precision={prec:.3f}  recall={rec:.3f}")

Observe when you run it:

1. The sigmoid uses two branches — this avoids exp(large) overflow.
2. Loss drops smoothly from ~0.7 (log 2) toward ~0.1 — cross-entropy is convex.
3. Sklearn and your from-scratch model produce nearly identical AUC.
4. As threshold rises from 0.3 → 0.9, precision goes up, recall goes down. That's the fundamental trade-off.
5. stratify=y keeps class balance the same across splits — critical on imbalanced data.

Try this modification: train on a severely imbalanced version (y_tr[y_tr==1][:20] — keep only 20 positives). Watch what happens to recall. Then pass class_weight="balanced" to sklearn's LogisticRegression and see it recover.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Where logistic regression is deployed today. Ads CTR prediction (Meta, Google, Criteo used LR at trillion-parameter scale for years before switching to deep nets — and often the deep net is still trained with LR-style cross-entropy at the output layer). Credit scoring (FICO). Fraud detection first-pass. Clinical decision support. Any place where you need calibrated probabilities and interpretable coefficients.

Class imbalance is the #1 gotcha. Fraud is ~0.1% of transactions. Cancer is ~5% of biopsies. A default logistic regression trained on 99% negatives predicts "0" for everyone and looks 99% accurate. Fixes: class_weight="balanced" (up-weight minority class in the loss), SMOTE (synthetic oversampling), threshold tuning post-hoc, or use metrics that don't reward the trivial predictor (precision-recall AUC, F1).

Probability calibration. Even if AUC is high, the probabilities may be miscalibrated — a "70% likely" prediction might actually be right 40% of the time. Check with a reliability diagram; fix with Platt scaling or isotonic regression. Critical for ads (bid × probability of click) and medical decisions.

Feature interactions. Logistic regression can't learn x₁ · x₂ on its own. Meta's legendary hack: use a GBDT (gradient-boosted decision tree) to generate features, then feed them into LR. Documented in the "Practical Lessons from Predicting Clicks on Ads at Facebook" paper. Old but gold.

Regularisation is basically always on. In practice, nobody trains LR without at least a little L2 penalty — it stabilises the coefficients when features are correlated. Sklearn defaults to L2 with C=1.0. (S087 goes deeper.)

Numerical stability. log(0) = -inf will crash your training. Always clip probabilities to [ε, 1-ε] before logging, or use logsumexp-style stable formulations. Every ML framework does this internally.

Multi-class. True logistic regression is binary. For K classes, use softmax regression (aka multinomial logistic regression) — replaces sigmoid with softmax and BCE with categorical cross-entropy. Same math, more dimensions.

(e) Quiz + exercise · 10 min

1. What does the sigmoid function do to a linear score, and what's its output range?
2. Why is cross-entropy preferred over MSE for binary classification?
3. In a spam classifier, would you rather have high precision or high recall, and why?
4. Explain how class imbalance can make a naive accuracy metric useless.
5. What is probability calibration and why does high AUC not guarantee it?

Stretch (connects to S021 — Probability): Show mathematically that cross-entropy loss for logistic regression is the negative log-likelihood of a Bernoulli distribution. What does maximising likelihood correspond to in terms of the loss surface?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Logistic Regression? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S087): Regularization — L1, L2, Elastic Net
• Previous (S085): Linear Regression from Scratch (numpy)
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.