R18 · Week 18 Recall & Drill

Weekly revision · Week 18 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S086 — Logistic Regression — Sigmoid, Cross-Entropy, from Scratch
• S087 — Regularization — L1, L2, Elastic Net
• S088 — Bias–Variance Trade-off & Learning Curves
• S089 — Decision Trees — Gini, Entropy, Splits
• S090 — Random Forest & Bagging

90-min structure

Blank-page reconstruction · 30 min

S086 · Logistic Regression

1. Write the squashing function and say why its output range makes it the right choice here.
2. Derive the classification loss from a likelihood argument in three lines.
3. Explain when the area under the precision-recall curve is more informative than the area under the receiver operating curve.

Gotcha you probably forgot: the default cut point of one half is a convention, not a result. It is only optimal when the two error types cost the same and the classes are balanced, which is almost never true. The threshold should be chosen from the precision-recall curve using the actual relative cost of a false positive versus a false negative — and it is the single cheapest improvement available on most classifiers.

S087 · Regularization

1. Write the two penalty terms and the losses they attach to.
2. Explain geometrically why one produces exact zeros and the other does not.
3. Say how you choose the penalty strength.

Gotcha you probably forgot: features must be scaled before any penalty is applied. The penalty is on coefficient magnitude, and a coefficient's magnitude depends on its feature's units — so a feature measured in millimetres is penalised a thousand times more weakly than the same feature in metres. Without scaling, the regulariser silently encodes your unit choices as a prior about which features matter.

S088 · Bias–Variance

1. State the error decomposition and explain each term.
2. Given training error far below validation error, name the diagnosis and three fixes.
3. Given training and validation error both high and close together, name the diagnosis and three fixes.

Gotcha you probably forgot: the two diagnoses take opposite actions, so guessing wrong actively hurts. More data helps the high-variance case and does nothing at all for the high-bias case — a model that cannot fit the data it already has will not be rescued by more of it. Read the gap between the curves before choosing, because that gap is the diagnosis.

S089 · Decision Trees

1. Write an impurity measure and say what it measures.
2. Describe the greedy split loop in three steps.
3. Name the hyperparameters that control overfitting.

Gotcha you probably forgot: impurity-based feature importance is systematically biased towards high-cardinality features, because a feature with many distinct values offers many more candidate split points and will reduce impurity somewhere by chance. A purely random identifier column can rank near the top. Permutation importance computed on held-out data does not have this bias and is what you should report.

S090 · Random Forest & Bagging

1. Explain bootstrap aggregation and its effect on the variance term.
2. State roughly what fraction of unique rows a bootstrap sample contains, and what the remainder is used for.
3. Say what the forest adds on top of plain bagging, and why.

Gotcha you probably forgot: averaging correlated predictors barely reduces variance. If one feature dominates, nearly every bootstrapped tree splits on it at the root and they all make the same mistakes, so the ensemble is little better than one tree. Restricting each split to a random subset of features forces trees to disagree, and that decorrelation is where the gain actually comes from.

Hands-on drill · 30 min

Task: pick a threshold by cost, watch penalties select versus shrink, read two learning curves, and measure decorrelation.

mkdir -p ~/projects/w18-drill && cd ~/projects/w18-drill

Step 1 — the threshold is a business decision (8 min)

# threshold.py
import numpy as np
 
rng = np.random.default_rng(3)
n = 20_000
prevalence = 0.02                            # rare positive class
y = (rng.random(n) < prevalence).astype(int)
# Scores that separate the classes imperfectly.
score = 1 / (1 + np.exp(-(rng.normal(loc=np.where(y == 1, 1.6, 0.0), scale=1.0))))
 
def confusion(thr):
    pred = score >= thr
    tp = int((pred & (y == 1)).sum()); fp = int((pred & (y == 0)).sum())
    fn = int((~pred & (y == 1)).sum()); tn = int((~pred & (y == 0)).sum())
    return tp, fp, fn, tn
 
COST_FP, COST_FN = 1.0, 40.0                 # missing a positive is far worse here
print(" thr   precision  recall   accuracy   expected cost")
best = None
for thr in np.arange(0.05, 0.96, 0.05):
    tp, fp, fn, tn = confusion(thr)
    prec = tp / max(tp + fp, 1); rec = tp / max(tp + fn, 1)
    acc = (tp + tn) / n
    cost = COST_FP * fp + COST_FN * fn
    print(f"{thr:5.2f}  {prec:9.3f}  {rec:6.3f}  {acc:9.4f}  {cost:12.0f}")
    if best is None or cost < best[1]:
        best = (thr, cost)
print(f"\ncost-minimising threshold: {best[0]:.2f}   (the default 0.50 is not it)")
print(f"always-predict-negative accuracy: {(y == 0).mean():.4f}  <-- beats most thresholds, predicts nothing")

Expected outcome: accuracy is highest for a model that never predicts the positive class, which is worth staring at — it is the clearest demonstration that accuracy is meaningless at low prevalence. The cost-minimising threshold sits well away from one half, because the asymmetric costs pull it towards recall. Change the cost ratio and the optimum moves; that is the point. The threshold is not a model parameter, it is a decision parameter, and it belongs to whoever owns the consequences.

Step 2 — shrink versus select (8 min)

# penalties.py
import numpy as np
 
rng = np.random.default_rng(7)
n, d = 120, 40
X = rng.normal(size=(n, d))
X[:, 5] = X[:, 3] + rng.normal(scale=0.01, size=n)     # near-duplicate feature
true = np.zeros(d); true[[0, 3, 9]] = [2.5, -1.8, 1.1]  # only 3 features matter
y = X @ true + rng.normal(scale=0.5, size=n)
 
Xs = (X - X.mean(0)) / X.std(0)                         # scaling is mandatory
 
def ridge(lam):
    return np.linalg.solve(Xs.T @ Xs + lam * np.eye(d), Xs.T @ y)
 
def lasso(lam, steps=4000, lr=0.01):
    th = np.zeros(d)
    for _ in range(steps):
        th -= lr * (Xs.T @ (Xs @ th - y)) / n
        th = np.sign(th) * np.maximum(np.abs(th) - lr * lam, 0.0)   # soft threshold
    return th
 
print("lambda   ridge: exact zeros   lasso: exact zeros   lasso keeps")
for lam in (0.01, 0.1, 1.0, 5.0):
    r, l = ridge(lam), lasso(lam)
    kept = np.flatnonzero(np.abs(l) > 1e-8)
    print(f"{lam:6.2f}   {int((np.abs(r) < 1e-8).sum()):>15}   {int((np.abs(l) < 1e-8).sum()):>18}   {list(kept)[:8]}")
print(f"\ntruly non-zero features: {list(np.flatnonzero(true))}  (feature 5 is a near-duplicate of 3)")

Expected outcome: the smooth penalty never produces an exact zero at any strength — it shrinks everything and keeps everything — while the absolute-value penalty drives most coefficients to precisely zero and increasingly so as the strength rises. Watch the near-duplicate pair specifically: the selecting penalty tends to keep one of the two arbitrarily and discard the other, which is a real problem when both are genuinely meaningful and is exactly why the combined penalty exists. The geometry behind it is the corner: the absolute-value constraint region has vertices on the axes, and constrained optima land on vertices.

Step 3 — learning curves as a diagnosis (7 min)

# curves.py
import numpy as np
 
rng = np.random.default_rng(2)
N = 3000
x = rng.uniform(-3, 3, N)
y = np.sin(2 * x) + 0.3 * x**2 + rng.normal(scale=0.25, size=N)   # genuinely curved
 
def design(x, degree):
    return np.vstack([x**p for p in range(degree + 1)]).T
 
def curve(degree):
    rows = []
    for m in (30, 60, 120, 300, 800, 2000):
        tr, va = slice(0, m), slice(2000, 3000)
        A = design(x[tr], degree)
        th, *_ = np.linalg.lstsq(A, y[tr], rcond=None)
        rmse = lambda s: float(np.sqrt(np.mean((design(x[s], degree) @ th - y[s])**2)))
        rows.append((m, rmse(tr), rmse(va)))
    return rows
 
for degree, label in [(1, "degree 1  (too simple)"), (14, "degree 14 (too flexible, tiny data)")]:
    print(f"\n{label}")
    print("  n_train   train_rmse   val_rmse    gap")
    for m, tr, va in curve(degree):
        print(f"  {m:>7}   {tr:10.3f}   {va:8.3f}   {va-tr:6.3f}")

Expected outcome: the simple model shows both errors high and the gap small, and crucially the errors stop improving as data is added — that flat, converged, high pair is the signature of insufficient capacity, and more data is wasted money. The flexible model at small sample sizes shows a low training error with a much larger validation error, and the gap narrows as data grows — that shape says more data will help. Read the gap first, then the level: gap large means variance, level high with small gap means bias.

Step 4 — decorrelation is the mechanism (7 min)

# forest.py
import numpy as np
 
rng = np.random.default_rng(11)
n, d = 800, 12
X = rng.normal(size=(n, d))
X[:, 0] *= 4                                    # one dominant feature
y = (X[:, 0] + 0.4 * X[:, 1] + rng.normal(scale=0.5, size=n) > 0).astype(int)
 
def stump(Xb, yb, features):
    """One greedy split, restricted to the given candidate features."""
    best = (1e9, None, None)
    for j in features:
        for t in np.quantile(Xb[:, j], [0.25, 0.5, 0.75]):
            left = Xb[:, j] <= t
            if left.sum() in (0, len(yb)):
                continue
            def gini(v):
                p = v.mean() if len(v) else 0
                return 1 - p**2 - (1 - p)**2
            imp = (left.mean() * gini(yb[left]) + (1 - left.mean()) * gini(yb[~left]))
            if imp < best[0]:
                best = (imp, j, t)
    return best[1], best[2]
 
def ensemble(mtry, trees=40):
    roots, preds = [], []
    for _ in range(trees):
        idx = rng.integers(0, n, n)             # bootstrap
        feats = rng.choice(d, size=mtry, replace=False)
        j, t = stump(X[idx], y[idx], feats)
        roots.append(j)
        left = X[:, j] <= t
        p = np.where(left, y[idx][X[idx][:, j] <= t].mean() if left.any() else 0.5, 0.5)
        preds.append((X[:, j] <= t).astype(float))
    P = np.array(preds)
    corr = np.corrcoef(P)
    off = corr[~np.eye(trees, dtype=bool)]
    return roots, float(np.nanmean(off))
 
for mtry, label in [(d, "bagging only (all features considered)"), (3, "forest (random feature subset)")]:
    roots, mean_corr = ensemble(mtry)
    share = max(roots.count(j) for j in set(roots)) / len(roots)
    print(f"{label:<42} most-common root feature used by {share:5.0%} of trees, "
          f"mean pairwise correlation {mean_corr:.3f}")

Expected outcome: with every feature available, nearly all trees split on the dominant feature at the root and their predictions are highly correlated — bootstrapping the rows was not enough to make them different. Restricting the candidate features forces many trees to use something else, the shared-root share drops sharply, and mean pairwise correlation falls with it. Since averaging reduces variance in proportion to how uncorrelated the members are, that correlation number is the mechanism, not a side effect.

Gap analysis + next week preview · 10 min

• Did the always-predict-negative accuracy in Step 1 land higher than you expected? Keep that number in mind whenever someone quotes accuracy on rare events.
• Could you state the two learning-curve signatures without the drill? That diagnosis saves weeks of collecting data that will not help.
• Did the correlation number in Step 4 change how you explain forests? "Averaging cancels mistakes" is the answer that gets follow-up questions.

Next week (S091–S095) continues into gradient boosting and the modern tabular workhorses, unsupervised methods including clustering and dimensionality reduction, and the practical pipeline work of encoding, imputation, and leakage-free preprocessing. The bias-variance vocabulary and the decorrelation argument from this week are exactly what distinguish boosting from bagging when you get there.

Part of the 6-month evergreen learning plan.