S095 · Model Selection — CV, Hyperparameter Tuning, Optuna

Module M11: Classical ML · Session 95 of 130 · Track: ML 🎛️

What you'll be able to do after this session

• Explain Model Selection to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S091 — Gradient Boosting — XGBoost, LightGBM

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · StatQuest: Cross Validation — the 6-minute canonical explanation of k-fold CV.
• 🎥 Deep dive (30–60 min) · Abhishek Thakur: Cross Validation Techniques — Kaggle grandmaster walks through k-fold, stratified, group, time-series CV.
• 🎥 Hands-on demo (10–20 min) · Abhishek Thakur: Hyperparameter Tuning with Optuna — live-codes an Optuna study on a real dataset.
• 📖 Canonical article · scikit-learn: Cross-validation — authoritative reference on CV variants and pitfalls.
• 📖 Book chapter / tutorial · Optuna Tutorial: Key Features — the best from-scratch intro to Bayesian HPO.
• 📖 Engineering blog · Preferred Networks: Optuna — a hyperparameter optimization framework — the team who built Optuna explaining why TPE + pruning beats grid search.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Picking the winner honestly.

You've trained a model. It scores 0.87 on your test set. Is that good? Was it luck? Would a different random split give 0.79? Could tweaking max_depth from 6 to 8 push it to 0.90? These are the questions cross-validation and hyperparameter tuning answer, and getting them wrong is the fastest way to embarrass yourself in a code review.

The mental model is a chef testing recipes. If you cook one dish and one friend rates it 9/10, that's noise. Cook it five times for five different friends and average the scores — now you know if the recipe is really a 9. k-fold cross-validation does exactly that: split your data k times, train on k-1 folds, evaluate on the held-out fold, repeat. The average score (and its standard deviation) tells you both the quality and the stability of your model.

Gotcha to remember forever: you have exactly two independent evaluation budgets. Cross-validation is where you compare architectures and tune hyperparameters — you can look at those numbers thousands of times. The final test set is the one you touch once, at the end, to report to the world. If you tune to the test set even a little, you've overfit to it and your published number is a lie. Optuna, GridSearchCV, RandomSearchCV all operate inside the CV budget; the test set stands apart, untouched, sacred.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Cross-validation flavours — pick the right one

Using the wrong flavour is a leak. Common example: patient IDs. If patient P has 3 visits and one lands in train, two in test, your "model" is just recognising patient P — not learning medicine.

HPO — grid vs random vs Bayesian

Optuna uses TPE (Tree-structured Parzen Estimator) by default and adds pruning: it kills unpromising trials early. On a big LightGBM tune, Optuna typically hits a random-search result in 3–5× fewer trials.

Worked example: 100 CV runs is not overfitting; 100 test-set peeks is

Suppose your strategy is 5-fold CV on 10,000 rows, and you run 100 Optuna trials. Total: 500 model fits, 500 CV scores. Take the best config. Refit on all train data. Predict on the held-out test set once. Report that.

If instead you shipped "the model that scored best on the test set out of 100 candidates" — congratulations, your reported metric is now cherry-picked from 100 draws and is optimistically biased by a lot. The nested-CV pattern (outer loop = honest scoring, inner loop = HPO) exists precisely to prevent this.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# pip install optuna lightgbm scikit-learn
import numpy as np
import optuna
import lightgbm as lgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.metrics import roc_auc_score
 
X, y = load_breast_cancer(return_X_y=True)
Xtrval, Xte, ytrval, yte = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
 
def objective(trial):
    params = {
        "objective": "binary", "metric": "auc", "verbosity": -1,
        "learning_rate":     trial.suggest_float("learning_rate", 1e-3, 0.3, log=True),
        "num_leaves":        trial.suggest_int("num_leaves", 8, 128),
        "max_depth":         trial.suggest_int("max_depth", 3, 12),
        "min_data_in_leaf":  trial.suggest_int("min_data_in_leaf", 5, 100),
        "feature_fraction":  trial.suggest_float("feature_fraction", 0.5, 1.0),
        "bagging_fraction":  trial.suggest_float("bagging_fraction", 0.5, 1.0),
        "bagging_freq":      trial.suggest_int("bagging_freq", 0, 10),
        "reg_alpha":         trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True),
        "reg_lambda":        trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True),
        "n_estimators":      500,
    }
    model = lgb.LGBMClassifier(**params, random_state=42)
    scores = cross_val_score(model, Xtrval, ytrval, cv=cv, scoring="roc_auc", n_jobs=-1)
    return scores.mean()
 
study = optuna.create_study(
    direction="maximize",
    sampler=optuna.samplers.TPESampler(seed=42),
    pruner=optuna.pruners.MedianPruner(),
)
study.optimize(objective, n_trials=40, show_progress_bar=True)
 
print("Best CV AUC:", study.best_value)
print("Best params:", study.best_params)
 
best = lgb.LGBMClassifier(**study.best_params, n_estimators=500, random_state=42)
best.fit(Xtrval, ytrval)
test_auc = roc_auc_score(yte, best.predict_proba(Xte)[:, 1])
print(f"HONEST test AUC (touched once): {test_auc:.4f}")

What to observe when you run it:

1. Optuna's TPE quickly narrows the search to promising regions — watch the CV score climb, then plateau.
2. The best trial's params will not be at the middle of your ranges — that's the search working.
3. The test AUC is usually within 0.01–0.03 of the best CV AUC. A large gap either way is a red flag.
4. n_trials=40 on this small dataset takes seconds; on a big dataset, budget as trials × cv_folds × fit_time.
5. study.trials_dataframe() gives a full log — plot learning_rate vs value to see the objective landscape.

Try this modification: replace TPESampler with RandomSampler(seed=42) and re-run with 40 trials. On many datasets random matches TPE at 40 trials; TPE pulls ahead by 100.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story #1: leakage in group-structured data. A speech-recognition team used vanilla KFold on a dataset where each speaker contributed 50 utterances. Their reported CV WER was 7%. Held-out speaker WER: 22%. They were benchmarking "how well do we recognise voices we've heard?" GroupKFold(groups=speaker_id) fixed it.

War story #2: Optuna into a demo, straight into prod. A team ran an Optuna study for 500 trials, took the best config, deployed. Six months later, they added features and re-ran Optuna — best config had radically different num_leaves. Their objective was noisy across seeds; individual "best" configs were within noise of hundreds of others. Fix: average CV scores across at least 3 seeds, or prefer configs from the top 5% band, not the single max.

Common bugs:

• Data leak between CV folds. Fit StandardScaler / TargetEncoder inside the CV loop, not on the full dataset. Use sklearn.Pipeline end to end.
• Tuning n_estimators with a grid. For boosting, fix a generous n_estimators and use early stopping to pick the actual number. Grid-searching it is wasted compute.
• CV score high, test score low. Time leakage, group leakage, or preprocessing leakage. Suspect these three in that order.
• Optuna study not persisted. Add storage="sqlite:///study.db" so a killed job resumes.
• HPO on the wrong metric. Optuna maximises whatever you return; if your business cares about PR-AUC on the minority, don't return accuracy.

How top teams handle it:

• Kaggle grandmasters use nested CV for honest reporting: outer loop = clean test estimate, inner loop = HPO.
• Google Vizier, AWS SageMaker AMT, Azure ML HyperDrive offer hosted Bayesian HPO with parallel workers and early stopping.
• Uber Michelangelo, Meta FBLearner run HPO as part of the training pipeline, not a manual notebook step — configs are versioned and reproducible.

(e) Quiz + exercise · 10 min

1. In one sentence each, describe when you'd use KFold, StratifiedKFold, GroupKFold, and TimeSeriesSplit.
2. Why should hyperparameter search happen on CV within your training set, and not on the held-out test set?
3. What does Optuna's TPE sampler do differently from random search, and why is that usually a win for tabular ML?
4. What is the "pruning" feature of Optuna, and why does it dramatically speed up expensive tunes like deep learning?
5. Your 5-fold CV score is 0.90 ± 0.01 but your test-set score is 0.78. Name three concrete causes to investigate.

Stretch (connects to S091): You want to tune an XGBoost model with 8 hyperparameters on a 1M-row dataset. Design an HPO plan: what search method, how many trials, what CV strategy, how you'll early-stop, and how you'll get an honest test estimate at the end. Justify the compute budget.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Model Selection? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S096): Perceptron & Activation Functions
• Previous (S094): Imbalanced Data — SMOTE, Class Weights, Thresholds
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.