R19 · Week 19 Recall & Drill

Weekly revision · Week 19 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S091 — Gradient Boosting — XGBoost, LightGBM
• S092 — Evaluation Metrics — P/R/F1/ROC/PR/AUC
• S093 — Feature Engineering — Encoding, Scaling, Missing
• S094 — Imbalanced Data — SMOTE, Class Weights, Thresholds
• S095 — Model Selection — CV, Hyperparameter Tuning, Optuna

90-min structure

Blank-page reconstruction · 30 min

S091 · Gradient Boosting

1. Describe the boosting loop in one sentence.
2. Say why it is called gradient boosting rather than just boosting.
3. Name the three hyperparameters that matter most and their interaction.

Gotcha you probably forgot: boosting has no natural stopping point and will happily overfit if you let it, because each additional round reduces training loss further by construction. The number of rounds is therefore not a value you set but one you discover, using a validation set and early stopping. Lowering the learning rate does not remove this — it increases the number of rounds needed, making early stopping more necessary rather than less.

S092 · Evaluation Metrics

1. Draw the confusion matrix and define precision and recall from it in plain language.
2. Say what happens to precision and recall as the threshold rises.
3. Explain why the receiver operating summary is misleading at extreme imbalance.

Gotcha you probably forgot: the false positive rate has the negative count in its denominator, so when negatives outnumber positives by orders of magnitude, thousands of false alarms barely move it. The summary stays impressive while the alert queue is unusable. Precision has the predicted-positive count in its denominator instead, which is why the precision-recall view reflects what the operator actually experiences.

S093 · Feature Engineering

1. Give the decision rule for choosing between the encoders for a categorical column.
2. Say when scaling matters and when it does not.
3. Name three defensible imputation strategies and when each is wrong.

Gotcha you probably forgot: the encoder must be configured to tolerate categories it has never seen, or the first unfamiliar value in production raises an exception on a live request. This is not an edge case — new categories appear constantly in real data — and it is the most common way a model that validated perfectly fails on its first day.

S094 · Imbalanced Data

1. Explain why a default classifier predicts almost only the majority class under heavy imbalance.
2. Say what balanced class weighting does mathematically.
3. Give one situation where synthetic oversampling is clearly the wrong tool.

Gotcha you probably forgot: resampling breaks calibration. After oversampling the minority, the model's outputs no longer correspond to real-world probabilities because it was trained on a distribution you invented — so the numbers can no longer be used for expected-value decisions without recalibrating on the true distribution. If your downstream system multiplies the score by a cost, this matters more than any ranking gain.

S095 · Model Selection

1. Explain what k-fold gains you over a single split.
2. Describe nested cross-validation in one sentence and what it protects.
3. Say when grouped splitting is mandatory rather than optional.

Gotcha you probably forgot: the held-out set must be touched exactly once. Every time you look at it and then change something, you have used it for selection, and its estimate degrades towards the optimism of a validation score. The discipline is unpleasant precisely because the temptation appears at the moment you most want reassurance.

Hands-on drill · 30 min

Task: watch the two curve summaries disagree, leak through a scaler, fix recall without touching the model, and measure the optimism in a tuned score.

mkdir -p ~/projects/w19-drill && cd ~/projects/w19-drill

Step 1 — the two curves disagree (8 min)

# curves.py
import numpy as np
 
rng = np.random.default_rng(4)
 
def build(prevalence, n=200_000, sep=1.5):
    y = (rng.random(n) < prevalence).astype(int)
    s = rng.normal(loc=np.where(y == 1, sep, 0.0), scale=1.0)
    return y, s
 
def roc_auc(y, s):
    order = np.argsort(s)
    ranks = np.empty(len(s)); ranks[order] = np.arange(1, len(s) + 1)
    npos, nneg = y.sum(), (1 - y).sum()
    return float((ranks[y == 1].sum() - npos * (npos + 1) / 2) / (npos * nneg))
 
def pr_auc(y, s):
    order = np.argsort(-s)
    yo = y[order]
    tp = np.cumsum(yo); fp = np.cumsum(1 - yo)
    prec = tp / (tp + fp); rec = tp / y.sum()
    return float(np.sum(np.diff(np.r_[0, rec]) * prec))
 
print("prevalence   ROC-AUC   PR-AUC   precision at recall 0.80")
for prev in (0.5, 0.05, 0.005, 0.0005):
    y, s = build(prev)
    order = np.argsort(-s); yo = y[order]
    tp = np.cumsum(yo); fp = np.cumsum(1 - yo)
    rec = tp / y.sum(); prec = tp / (tp + fp)
    i = int(np.argmax(rec >= 0.80))
    print(f"{prev:10.4f}   {roc_auc(y,s):7.3f}   {pr_auc(y,s):6.3f}   {prec[i]:6.3f}"
          f"   ({int(fp[i]):,} false alarms to catch {int(tp[i]):,} positives)")

Expected outcome: the receiver operating summary is nearly unchanged across all four prevalences — the model's ranking ability genuinely did not change, and that summary measures only ranking. The precision-recall summary collapses as positives become rare, and the last column tells you why: to reach the same recall you must accept an enormous number of false alarms, because there are simply far more negatives available to be wrong about. The ranking is identical; the operational experience is not. Report the summary that reflects what the person reading the alerts will feel.

Step 2 — leak through a scaler (7 min)

# leak.py
import numpy as np
 
rng = np.random.default_rng(1)
n, d = 300, 200
X = rng.normal(size=(n, d))
y = rng.integers(0, 2, n)                     # no relationship by construction
 
def fit_predict(Xtr, ytr, Xte):
    A = np.c_[np.ones(len(Xtr)), Xtr]
    th, *_ = np.linalg.lstsq(A, ytr, rcond=None)
    return (np.c_[np.ones(len(Xte)), Xte] @ th) > 0.5
 
idx = rng.permutation(n); tr, te = idx[:200], idx[200:]
 
# WRONG: scaling parameters computed from all rows, including the held-out ones.
mu, sd = X.mean(0), X.std(0)
Xa = (X - mu) / sd
leaky = (fit_predict(Xa[tr], y[tr], Xa[te]) == y[te]).mean()
 
# RIGHT: fit the transform on training rows only, apply to held-out rows.
mu, sd = X[tr].mean(0), X[tr].std(0)
Xb = (X - mu) / sd
honest = (fit_predict(Xb[tr], y[tr], Xb[te]) == y[te]).mean()
 
print(f"scaler fitted on all rows : {leaky:.3f}")
print(f"scaler fitted on train only: {honest:.3f}")
print("true signal: none. Any gap between these two lines is the leak.")
print("\nThe same argument applies to imputers, encoders, target encoding, and feature selection:")
print("if it has a fit step, it belongs inside the split.")

Expected outcome: both numbers hover near chance because there is no signal, but the leaky one is systematically the higher of the two — and on a real dataset with genuine signal that gap widens into a score you will report and then fail to reproduce. The rule to write down is mechanical rather than statistical: anything with a fitting step is part of the model, so it must be fitted inside the training fold and merely applied to the held-out rows. That is the entire reason preprocessing belongs in a pipeline object rather than in a prior cell of the notebook.

Step 3 — threshold before resampling (8 min)

# imbalance.py
import numpy as np
 
rng = np.random.default_rng(8)
n, prev = 60_000, 0.01
y = (rng.random(n) < prev).astype(int)
score = 1 / (1 + np.exp(-rng.normal(loc=np.where(y == 1, 1.8, 0.0), scale=1.0)))
 
def report(label, thr):
    pred = score >= thr
    tp = int((pred & (y == 1)).sum()); fp = int((pred & (y == 0)).sum())
    fn = int((~pred & (y == 1)).sum())
    prec = tp / max(tp + fp, 1); rec = tp / max(tp + fn, 1)
    print(f"{label:<34} thr={thr:.3f}  precision={prec:.3f}  recall={rec:.3f}  alerts={tp+fp:,}")
 
report("default cut point", 0.50)
 
# Business requirement: recall of at least 0.90. Find the threshold that delivers it.
order = np.argsort(-score); yo = y[order]
rec = np.cumsum(yo) / y.sum()
thr = score[order][int(np.argmax(rec >= 0.90))]
report("threshold tuned for recall 0.90", float(thr))
 
print("\nNo retraining occurred. No resampling occurred. Only the cut point moved.")
print("Order of attempts: (1) move the threshold, (2) class weights, (3) resampling — in that order,")
print("because each step after the first costs calibration, training time, or both.")

Expected outcome: the default cut point produces very high precision and unusably low recall, which is the classic "the model does not work" complaint at low prevalence. Moving the threshold alone reaches the required recall, at a precision cost you can see and quantify. Nothing was retrained. This is the ordering to internalise, because teams routinely reach for synthetic oversampling first — it is the most complex intervention, it damages calibration, and it is frequently unnecessary once the threshold is set to reflect the actual costs.

Step 4 — the optimism in a tuned score (7 min)

# selection_bias.py
import numpy as np
 
rng = np.random.default_rng(21)
 
def experiment(n_configs, n=400, folds=5):
    """No config is better than any other: scores are pure noise around 0.5."""
    X = rng.normal(size=(n, 30))
    y = rng.integers(0, 2, n)
    idx = rng.permutation(n)
 
    def cv(seed):
        r = np.random.default_rng(seed)
        scores = []
        for f in range(folds):
            te = idx[f::folds]; tr = np.setdiff1d(idx, te)
            noise = r.normal(scale=0.04)                # a config's idiosyncratic luck
            scores.append(0.5 + noise + r.normal(scale=0.03))
        return float(np.mean(scores))
 
    inner = [cv(s) for s in range(n_configs)]
    best_reported = max(inner)                          # what people report
    honest = cv(9999)                                   # an untouched estimate
    return best_reported, honest
 
print("configs tried   best CV score reported   honest estimate   optimism")
for k in (1, 10, 50, 200):
    b, h = experiment(k)
    print(f"{k:>13}   {b:>21.3f}   {h:>15.3f}   {b-h:+8.3f}")

Expected outcome: with a single configuration the reported score is honest. As the number of configurations grows, the maximum over noisy estimates drifts steadily upward while nothing about the underlying quality has changed — you are reporting the luckiest fold assignment, not the best model. The gap is the selection bias, and it grows with the size of the search. The fix is an outer loop that never participates in selection: tune inside, evaluate outside, and report the outer number even though it is worse. That worse number is the one that will match production.

Gap analysis + next week preview · 10 min

• Did the false-alarm column in Step 1 change which summary you would put in a report? That column is the argument, not the summary number.
• Could you name all four leakage flavours without the drill? Each one has cost somebody a shipped model.
• Did the optimism in Step 4 look larger than you expected? Note that it grows with search effort, which is the opposite of the intuition most people carry.

Next week (S096–S100) moves from classical machine learning into neural networks: the perceptron and multilayer networks, backpropagation derived by hand, activation functions and initialisation, optimisers beyond plain gradient descent, and the training loop in a real framework. The bias-variance vocabulary, the leakage discipline, and the honest-evaluation protocol from this week carry over unchanged — the models get bigger, the failure modes stay the same.

Part of the 6-month evergreen learning plan.