R19 · Week 19 Recall & Drill

Assessment scope

A model-selection result belongs to a complete procedure: split, preprocessing, estimator, calibration, threshold and selection rule. This revision reconstructs boosting and metrics, then makes the information boundaries executable. NumPy fixtures demonstrate mechanisms; they are not XGBoost/LightGBM/Optuna runs or an operational fraud benchmark.

Sessions covered

• S091 — Gradient Boosting — XGBoost, LightGBM
• S092 — Evaluation Metrics — P/R/F1/ROC/PR/AUC
• S093 — Feature Engineering — Encoding, Scaling, Missing
• S094 — Imbalanced Data — SMOTE, Class Weights, Thresholds
• S095 — Model Selection — CV, Hyperparameter Tuning, Optuna

90-min structure

The allocation is a practice aid, not a measured completion time.

Blank-page reconstruction · 30 min

S091 · Gradient Boosting

1. Describe the boosting loop and the meaning of “gradient.”
2. Contrast boosting with bagging without making the bias/variance slogan absolute.
3. Explain learning rate, tree complexity and number of rounds together.

Solutions. Initialize a prediction function F, compute each training example's negative loss gradient with respect to its current prediction, fit a weak learner h to those pseudo-residuals, then update F <- F + eta*h (possibly with a loss-specific line search or leaf-value optimization). For half-squared error the negative gradient is y-F; for logistic loss on logits it is y-sigmoid(F). This is descent in function space, not simply repeatedly fitting the original labels.

Bagging averages perturbed fitted models and often reduces high tree variance. Boosting incrementally improves a function and often reduces approximation error with shallow learners, but both can affect bias and variance; every extra boosting round does not necessarily add variance or reduce training loss under arbitrary approximate fitting and step size. XGBoost and LightGBM add their own regularization, sampling and tree-building choices; this sketch does not certify every library's defaults.

Learning rate shrinks each contribution; smaller rates often need more rounds. Depth/leaf count controls the interactions each tree can represent. Number of rounds controls total refinement and compute. Select all together inside validation/CV. Early stopping is useful: monitor held-out loss, retain the best checkpoint and stop after a chosen patience. It is not logically mandatory—fixed rounds chosen in honest CV are also legitimate. The final test must not be used to choose the checkpoint. A too-large learning rate can overshoot even for squared loss: if h exactly equals a residual, the next residual is (1-eta)*old, which grows in magnitude when eta > 2.

S092 · Evaluation Metrics

1. Define the confusion counts, precision, recall and F1.
2. How do threshold changes affect them?
3. Explain ROC-AUC versus average precision and an alert queue at low prevalence.

Solutions. TP/FP are positive predictions with positive/negative truth; FN/TN are negative predictions with positive/negative truth. Precision is TP/(TP+FP), recall/TPR is TP/(TP+FN), FPR is FP/(FP+TN), and F1 is 2TP/(2TP+FP+FN). Undefined denominators require a declared convention, not silently “perfect” scores. F1 ignores true negatives and explicit decision costs.

With actual class as the row and predicted class as the column, the confusion matrix is:

Increasing a score >= threshold threshold cannot increase TP or FP, so recall is nonincreasing; precision need not increase at every finite-data step. For scores (0.9, 0.8, 0.7) with labels (0, 1, 0), thresholds 0.7, 0.8 and 0.9 give precisions 1/3, 1/2 and 0, respectively; recalls are 1, 1 and 0. Raising the threshold can remove the only true positive before removing the highest-scoring false positive. AUC summarizes ranking over thresholds, not an action policy. ROC-AUC equals the probability a random positive outranks a random negative, awarding half credit to ties. Average precision (AP) sums precision at score-group boundaries weighted by recall increments. AP is not the trapezoidal area under an interpolated PR curve; call the implementation what it computes and handle tied scores as a group.

At TPR 0.8 and FPR 0.01, a population of 100 positives/900 negatives yields expected TP=80, FP=9, precision=80/89. At 10 positives/9,990 negatives it yields expected TP=8, FP=99.9, precision about 7.4%. Conditional ranking quality can stay the same while the queue becomes much less useful. Report ROC/PR measures appropriate to the question, prevalence, absolute alert counts and uncertainty at the selected operating point. ROC is not inherently dishonest; it just does not include prevalence-dependent precision.

S093 · Feature Engineering

1. Choose categorical encoding by semantics and cardinality.
2. When does scaling matter?
3. Give imputation options and enumerate four leakage routes.

Solutions. Use one-hot for manageable nominal categories, an explicitly meaningful ordinal mapping for ordered categories, and consider hashing or regularized/out-of-fold target encoding for high cardinality. Hashing has collisions; integer codes imply order to models that treat them numerically. Target encoders must not encode a row using its own label. A fitted encoder needs an explicit unknown-category policy; rejecting invalid categories can be intentional, but crashing on ordinary new values is not a deployment design. Preserve the fitted vocabulary/version with the model.

Scaling matters for distances, gradient conditioning and coefficient penalties. Ordinary axis-aligned decision trees depend on order and generally do not require standardization, though implementation precision and preprocessing still matter. Fit all means, variances and category choices on training data. A constant column needs a safe scale, such as 1, rather than division by zero.

Median/mean imputation is simple but can erase subgroups or distort covariance; add a missingness indicator when absence itself carries information. A categorical “missing” level preserves that event but should not conflate it with an unseen category. Model-based/KNN imputation can preserve relationships but adds assumptions, cost and its own fit boundary. Forward filling can suit time-series state but only from past available observations, not across users or from the future. No method repairs a missing-not-at-random mechanism automatically.

Four concrete leaks: target leakage (a chargeback resolution feature unavailable at fraud decision time); preprocessing leakage (selector/scaler/imputer fitted on validation rows); entity leakage (the same patient/customer or near-duplicate in both splits); temporal leakage (future events or aggregates used for earlier predictions). A pipeline packages fit/transform boundaries; it cannot make an inappropriate split or a future-derived feature honest.

S094 · Imbalanced Data

1. Why can a 0.5 rule predict few positives, and when is that not the explanation?
2. Write balanced class weights.
3. Choose thresholds, weighting or resampling and identify a bad SMOTE case.

Solutions. A calibrated rare-event posterior often lies below 0.5 even for useful rankings. But imbalance alone does not force an arbitrary score function to predict mostly negative; the original synthetic sigmoid scores had many negatives above 0.5. First inspect score distributions, calibration, metrics, labels and the deployment requirement.

For K classes with counts n_c, the scikit-learn balanced heuristic is w_c=n/(K*n_c), equalizing each class's total weight. In a weighted Bernoulli loss, the conditional optimum becomes q = w1*p/(w1*p+w0*(1-p)) under an idealized model, so q is not generally the original posterior p. Resampling changes the training prior too. Neither necessarily destroys all calibration irreversibly: use representative held-out calibration or, under pure prior-shift assumptions, a prior-odds correction. Validate the assumptions before cost calculations.

If ranking is adequate but recall at the default threshold is poor, threshold tuning is a cheap first experiment; it trades precision, workload and false-negative cost without retraining. If ranking is inadequate, weighting or data/model changes may help; resampling is another option, not always a final resort. Apply it only inside training folds. Ordinary SMOTE interpolates feature vectors; it can invent impossible one-hot categories, blend distinct fraud modes, or cross entity/time boundaries. Specialized categorical variants do not remove the need for domain validity checks. Test on the natural prevalence, never on an artificially balanced test set.

S095 · Model Selection

1. What does k-fold add over one split?
2. What does nested CV evaluate?
3. When are groups/time constraints more important than stratification?

Solutions. K-fold trains on multiple subsets and evaluates each held-out fold, reducing dependence on one partition and using data efficiently. It is not universally unbiased for the final all-data fit: fold training sizes differ, folds are dependent, and distribution assumptions matter. Stratification stabilizes class composition where compatible with the task; it does not manufacture independent positives.

Nested CV fits/tunes a procedure on each outer training set using inner folds, then evaluates the selected procedure on that untouched outer test fold. The outer score assesses the selection procedure at that training size; it does not prove the single eventual model will achieve exactly that value. A final reserved test is another way to estimate a frozen procedure, subject to uncertainty and deployment shift.

Use grouped splits when future predictions target unseen entities and correlated rows share an entity; group repeated patients, accounts or near-duplicate documents together. Use forward/time splits when predicting the future, including a gap if feature/label windows overlap. Group/time constraints take priority over convenient class balance. In a long project, log feature/model/threshold trials, keep test labels out of tuning, and freeze preprocessing plus calibration plus threshold before final evaluation. Optuna can schedule/prune trials; it cannot supply an untouched evaluation set by itself. Repeatedly adapting to an outer-CV result also turns it into selection data.

Hands-on drill · 30 min

The four programs execute CPU-only models. All score ties, candidate thresholds and split memberships are explicit. Real class-weighted training, SMOTE and service rollout remain separate experiments.

Step 1 — the two curves disagree (8 min)

import numpy as np
from itertools import product
# Exact ranked-pair oracle, including ties.
def roc_auc(y,s):
    positive=[float(v) for v,t in zip(s,y) if t==1]
    negative=[float(v) for v,t in zip(s,y) if t==0]
    if not positive or not negative: raise ValueError("both classes required")
    return sum((p>n)+.5*(p==n) for p,n in product(positive,negative))/(len(positive)*len(negative))
def average_precision(y,s):
    y=np.asarray(y); s=np.asarray(s)
    total=int(y.sum())
    if not total: raise ValueError("positive class required")
    tp=fp=0; area=0.
    for threshold in sorted(set(s),reverse=True):
        group=y[s==threshold]; delta=int(group.sum())
        tp+=delta; fp+=len(group)-delta
        area+=(delta/total)*(tp/(tp+fp))
    return area
positive=np.array([.9,.7,.5,.3])
negative=np.array([.8,.6,.4,.2])
for copies in [1,10,100]:
    s=np.r_[positive,np.tile(negative,copies)]
    y=np.r_[np.ones(4,dtype=int),np.zeros(4*copies,dtype=int)]
    auc=roc_auc(y,s); ap=average_precision(y,s)
    pred=s>=.3
    tp=int(np.sum(pred&(y==1)));fp=int(np.sum(pred&(y==0)))
    assert auc==.625
    print("negative copies",copies,"prevalence",y.mean(),"ROC-AUC",auc,"AP",ap,"TP/FP",tp,fp)
assert roc_auc([1,0],[.5,.5])==.5
assert average_precision([1,0],[.5,.5])==.5
assert average_precision([0,1],[.5,.5])==.5

Replicating the same negative score distribution holds ROC-AUC exactly at 0.625 while prevalence falls. At threshold 0.3 all four positives are caught and FP rises from 3 to 30 to 300; precision changes from 4/7 to 4/34 to 4/304. This is not a claim that all real distribution shifts preserve ROC. Grouping equal scores prevents a favorable tie order from inflating AP.

Step 2 — leak through a scaler (7 min)

rng=np.random.default_rng(1)
X=rng.normal(size=(120,4)); y=2*X[:,0]-X[:,1]+rng.normal(scale=.1,size=120)
train=np.arange(80); test=np.arange(80,120)
def standardize(fit_rows):
    mu=X[fit_rows].mean(0);sd=X[fit_rows].std(0);sd=np.where(sd==0,1,sd)
    return (X-mu)/sd,mu,sd
Xa,ma,sa=standardize(np.arange(len(X)))
Xb,mb,sb=standardize(train)
def ols_predict(Z):
    w=np.linalg.lstsq(np.c_[np.ones(len(train)),Z[train]],y[train],rcond=None)[0]
    return np.c_[np.ones(len(test)),Z[test]]@w
# OLS with an intercept and full-rank columns is invariant to invertible affine feature scaling.
pa,pb=ols_predict(Xa),ols_predict(Xb)
assert np.allclose(pa,pb,atol=1e-10)
assert not np.allclose(ma,mb)
print("different fitted statistics, same OLS predictions; max delta",float(abs(pa-pb).max()))
# Inspectable imputer/encoder fitted ONLY on training rows.
train_values=np.array([10.,np.nan,30.,20.]); med=float(np.nanmedian(train_values))
train_categories=["red","blue","red","blue"]; vocab=sorted(set(train_categories))
def transform(values,categories):
    values=np.where(np.isnan(values),med,values)
    encoded=np.array([[int(c==v) for v in vocab] for c in categories])
    return values,encoded
values,codes=transform(np.array([np.nan,999.]),["green","red"])
assert med==20 and values.tolist()==[20.,999.]
assert codes.tolist()==[[0,0],[0,1]]  # unknown -> all-zero, a declared policy
print("training median/vocabulary",med,vocab,"test transformed",values,codes)

Fitting on all rows is still leakage, even though this exact full-rank OLS fixture has identical predictions under either invertible scaling. That counterexample corrects “the leaked score is systematically higher” and “any gap is the leak.” Regularization, rank deficiency or feature selection can change the response to scaling; absence of a score gap is not permission to use evaluation statistics. The encoder's unknown policy is explicit and must be evaluated for the application.

Step 3 — threshold before resampling (8 min)

validation_scores=np.array([.9,.8,.7,.6,.4,.3,.2,.1])
validation_y=np.array([1,0,1,0,1,0,0,0])
def select_recall_threshold(scores,labels,target):
    if not 0<target<=1 or np.sum(labels)==0: raise ValueError("invalid recall task")
    feasible=[]
    for threshold in sorted(set(scores),reverse=True):
        pred=scores>=threshold
        recall=int(np.sum(pred&(labels==1)))/int(np.sum(labels))
        if recall>=target: feasible.append(float(threshold))
    return max(feasible)
threshold=select_recall_threshold(validation_scores,validation_y,.9)
assert threshold==.4
# A different, untouched fixture measures—not guarantees—the selected rule's transfer.
test_scores=np.array([.85,.65,.45,.35,.25,.15])
test_y=np.array([1,0,0,1,0,0])
def report(scores,labels,threshold):
    pred=scores>=threshold
    tp=int(np.sum(pred&(labels==1)));fp=int(np.sum(pred&(labels==0)))
    fn=int(np.sum(~pred&(labels==1)))
    return tp,fp,fn,int(pred.sum())
assert report(validation_scores,validation_y,threshold)==(3,2,0,5)
assert report(test_scores,test_y,threshold)==(1,2,1,3)
print("chosen threshold",threshold,"validation",report(validation_scores,validation_y,threshold),"test",report(test_scores,test_y,threshold))
counts=np.array([90,10]);weights=counts.sum()/(2*counts)
assert np.allclose(counts*weights,[50,50])
p=.1;w0,w1=weights
q=w1*p/(w1*p+w0*(1-p))
assert np.isclose(q,.5)
print("balanced weights",weights,"weighted optimum for true posterior .1",q)

The validation rule reaches full recall there, but test recall is only 1/2. Selection has not guaranteed a production minimum. The weighted-probability example shows why an uncalibrated 0.5 output after reweighting should not be treated as a 50% natural event probability. Calibration data must reflect the real prevalence, and a scarce alert budget may make the requested recall infeasible at acceptable precision.

Step 4 — the optimism in a tuned score (7 min)

# A repeated finite simulation of selecting the luckiest estimate, not actual CV.
rng=np.random.default_rng(21)
trials,configs=4000,100
inner=.5+rng.normal(0,.04,size=(trials,configs))
outer=.5+rng.normal(0,.04,size=(trials,configs))
for k in [1,10,50,100]:
    winner=np.argmax(inner[:,:k],axis=1)
    reported=inner[np.arange(trials),winner]
    assessed=outer[np.arange(trials),winner]  # SAME selected configuration, independent estimate
    assert np.array_equal(reported,np.max(inner[:,:k],axis=1))
    print("search width",k,"mean selected",reported.mean(),"mean outer",assessed.mean(),"mean optimism",(reported-assessed).mean())
# Real selection protocol on a tiny regression task: outer holdout, inner split, refit.
x=np.linspace(-1,1,60);yr=x*x+.1*np.sin(17*x)
outer_train=np.arange(40);outer_test=np.arange(40,60)
inner_train=outer_train[::2];inner_val=outer_train[1::2]
def design(v,degree): return np.vander(v,degree+1,increasing=True)
def fit(rows,degree):return np.linalg.lstsq(design(x[rows],degree),yr[rows],rcond=None)[0]
choices=[]
for degree in [1,2,5]:
    w=fit(inner_train,degree)
    score=float(np.mean((design(x[inner_val],degree)@w-yr[inner_val])**2))
    choices.append((score,degree))
degree=min(choices)[1];w=fit(outer_train,degree)
outer_mse=float(np.mean((design(x[outer_test],degree)@w-yr[outer_test])**2))
assert set(outer_test).isdisjoint(outer_train)
print("inner choices",choices,"frozen degree",degree,"outer MSE",outer_mse)
# One exact squared-loss boosting step; a large eta disproves unconditional descent.
residual=np.array([1.,-2.,3.])
for eta in [.1,1.,3.]:
    before=float(np.sum(residual**2)/2)
    after=float(np.sum((residual-eta*residual)**2)/2)
    assert np.isclose(after,(1-eta)**2*before)
    print("eta",eta,"loss",before,"->",after)

Selecting a maximum over noisy estimates creates optimism on average when procedures have equal quality. An individual outer score can be higher, and a one-configuration estimate is still noisy—not magically unbiased in all settings. Unlike the original unrelated-seed proxy, the outer simulation evaluates the selected configuration. The concrete holdout example demonstrates selection/refitting boundaries, not a population-accurate generalization estimate; its ordered extrapolation split is intentionally harder than random interpolation.

Answer constraints and corrections

Misconception: “The best tuned CV score is an unbiased production forecast.” Selection reuses the score as an objective. Reserve an outer evaluation boundary; match sampling to the deployment task and report uncertainty. Neither nested CV nor one final test eliminates distribution shift, and an outer result is not necessarily numerically worse on every sample.

Quiz solutions:

1. Bagging often reduces variance; boosting often reduces approximation error, but these are tendencies. Validation-based stopping or honestly chosen rounds control complexity; universal per-round descent is false.
2. A small FPR can generate many false alarms at low prevalence. Report PR/AP, prevalence and counts at an actionable threshold alongside ROC when relevant.
3. Full-data scaler statistics cross the fit boundary even when the loss is invariant. Split before fitting every learned transform.
4. Inspect ranking/calibration and requirements first. Threshold tuning is cheap when ranking is adequate; weighting/resampling are experiments with their own calibration and domain-validity costs, not a universal order of commandments.
5. For months of exploration, establish entity/time-safe train/validation/test partitions, version data/features and log all trials. Tune transformations, models, early stopping, calibrators and thresholds inside selection data; use nested evaluation if assessing the search procedure. Do not repeatedly adapt to outer/test scores. Freeze and refit on allowed non-test data, open the final test, report counts, spread and cohort behavior, and schedule prospective monitoring. Fold spread is descriptive, not automatically a confidence interval because folds share training data.

Gap analysis + next week preview · 10 min

Construct a tied-score AP example, a future-derived feature, or a scaler-leak example whose predictions do not change. Explain why each exposes a different failure. Next week's perceptrons, MLPs, backpropagation, optimizers and PyTorch reuse exactly these split and evaluation obligations; a larger model does not relax them.

Corrective source notes

Primary versioned references, snapshots checked 2026-10-07: scikit-learn 1.7 boosting/ensembles, metrics, average precision, preprocessing, leakage, balanced weights, calibration, cross-validation and nested CV. The labs execute their printed NumPy code, not the named packages' training integrations; the weighting and boosting counterexamples are explicit derivations.

Additional references: imputation, SMOTE variants and interpolation limits.

Part of the 6-month evergreen learning plan.