Model Selection — CV, Hyperparameter Tuning, Optuna

Why this session exists

Model selection is itself a learning procedure. Trying architectures, features, resampling methods or thresholds against a score uses information from the labels behind that score. The winning score therefore cannot double as an independent assessment of the search. Cross-validation supplies reusable development comparisons; an outer split or untouched test supplies evaluation of the resulting procedure.

You will choose KFold, stratified, grouped or forward-time splits; explain the benefits and limits of repeated validation; implement nested selection with Optuna; and retune on all development data before one final holdout assessment. Neither CV nor a holdout guarantees future production performance under distribution shift.

Prerequisites

Review metrics, fold-local preprocessing and rare-positive support. Know the difference between model parameters fitted by training and hyperparameters chosen by selection.

(a) Intuition · 5 min

Hiring a chef after one meal risks rewarding a lucky performance. Several meals reveal variation. But if the chef sees the scoring rubric and rehearses those exact meals, the average is now part of preparation. A new tasting evaluates the whole preparation procedure. Similarly, CV averages over validation folds, while nesting protects evaluation from the choices made using those folds.

The analogy has a limit: fold scores are dependent because their training sets overlap. Their standard deviation is descriptive variability, not automatically a standard error or confidence interval. A 0.02 difference cannot be declared either significant or meaningless from the score range alone; paired outcomes, group structure and sampling uncertainty matter.

The three loops of honest evaluation

1. Set aside an optional final holdout before development. It can be a later time period or untouched set of entities, not necessarily random rows.
2. For each outer split, run the complete selection procedure only on outer-training rows.
3. Inside that subset, use inner folds to compare candidate settings. Refit the winner on the whole outer-training subset, then score outer-held-out rows.

Inner folds are subsets of the outer training partition: they do share those rows. The forbidden overlap is with that split's outer-held-out rows. Nested CV does not require a third holdout, but one can be useful as a final independent check if enough data remains. A plain development CV plus a clean final holdout is also valid; nesting is not the only honest design.

Every learned transform, feature selector, calibrator, sampler, stopping decision and tuned threshold belongs within the selection boundary. An early-stopping eval_set must be derived from inner training, not silently borrowed from outer evaluation. No amount of nesting fixes features unavailable at prediction time.

A quick history

The useful distinction among search methods is how they spend a finite budget. Exhaustive grids enumerate settings; random search allocates trials across distributions; Optuna's TPE uses earlier completed trials to propose new candidates. TPE and pruning are separate mechanisms. The versioned Optuna documentation is the authority here, not a claim that one tool replaced every grid search in industry.

K-fold cross-validation

Partition $n$ independent rows into $k$ validation subsets. For each subset fit on the other $k-1$, score it, and summarize. Approximately $(k-1)n/k$ rows train each model, not all $n$. Each row is validation once, while training sets overlap.

For equal-size folds, a mean of per-example additive losses equals the pooled out-of-fold loss. For unequal folds use support weights if the estimand is average per-example loss. A mean of fold AP or AUC is not generally the same as pooling all predictions: these metrics are nonlinear and scores from separate fitted models may have different scales.

Five or ten folds are pragmatic starting points, not a universal bias–variance optimum. Leave-one-out trains on $n-1$ rows but requires $n$ fits, can be unstable for sensitive learners and gives highly overlapping training sets. Larger $k$ does not universally reduce bias or increase variance; both depend on the algorithm, data and metric. Learning curves need not be monotonic on each finite dataset. Repeated K-fold measures sensitivity to partitions but does not create independent new datasets.

Nested CV — the pattern that keeps you honest

Let the estimated utility of candidate $j$ be $\hat s_j=s_j+\epsilon_j$. Selecting $j^*=\arg\max_j\hat s_j$ tends to select favorable noise too. Even if every candidate has the same true utility $s$, $E[\max_j\epsilon_j]\ge0$. With two independent errors taking $-a$ or $a$ equally often, the maximum is $-a$ only one quarter of the time, so its expectation is $a/2$. This is a worked selection-optimism calculation, not a forecast of a fixed percentage gap on a dataset.

Nested CV estimates the selection procedure at the outer training-set size and chosen split distribution. It removes the direct reuse of inner scores for final reporting; it is not universally unbiased for the final full-data model. Retuning search spaces after seeing outer scores makes them development feedback. Preserve a fresh evaluation boundary if doing that.

The four CV strategies you'll actually use

Grouping by patient or user is right when unseen-entity generalization is the goal. Predicting future behavior for already-known users may instead require a temporal split retaining their earlier records. Some problems require both constraints and a custom splitter. StratifiedGroupKFold can approximate class balance while isolating groups, but cannot always meet both perfectly. Do not let class balance override time or entity isolation.

TimeSeriesSplit uses expanding training sets by default. A finite max_train_size creates a bounded rolling history. Equal temporal spacing is needed for its equal-sized test folds to represent comparable durations; irregular events may need explicit calendar windows. A gap in rows is not automatically a gap in days. Exclude overlapping target horizons and fit every feature using only information available at each prediction time.

Search strategies for hyperparameters

A grid with six parameters and five values each has $5^6=15{,}625$ candidates. Three-fold inner CV requires 46,875 candidate fits, plus refits. RandomizedSearchCV limits the number of sampled candidates and can use log-scale distributions. Neither dimension count nor sampling method guarantees a better score.

In Optuna 4.5, TPE models a density $l(x)$ from better objective values and $g(x)$ from the rest, then chooses candidates using their density ratio. Early trials may be random startup trials. A small study using the default ten startup trials may never test adaptive proposals; our lab explicitly uses two startup trials with six total trials. This is an API/mechanism demonstration, not evidence that six trials are enough for deployment.

Log-scale suggest_float is useful when ratios matter: $10^{-3},10^{-2},10^{-1}$ represent equal multiplicative changes. Conditional branches can request different parameters for different model families. Specify a finite budget, objective direction and seed. Parallel/distributed scheduling and underlying numerical kernels can change reproducibility even with a sampler seed.

Pruning is not automatic. An iterative objective must call trial.report(value, step), inspect trial.should_prune() and raise optuna.TrialPruned(). A cross_val_score call that returns only a final mean supplies no intermediate training curve. MedianPruner, successive halving and Hyperband allocate early-stopping budgets; BOHB combines model-based proposals with Hyperband-style resource allocation. Compare them only when intermediate metrics reliably predict later quality.

Choosing an evaluation budget

For small datasets and a report whose deliverable is selection-procedure performance, nested CV can be worth the extra fits. For expensive training or a large representative evaluation set, train/validation/test is simpler. Development CV plus a final holdout offers repeated comparisons without the outer loop. None is automatically best above a fixed row count: the number of positives, independent groups and regimes can matter more than raw rows.

A $5\times5$ nested search over 50 candidates involves 1,250 inner fits plus five outer refits. Final retuning and refitting cost additional work. Record the budget rather than claiming nested CV is free or that a particular optimizer converges several times faster.

(c) Hands-on · 25 min

This bounded CPU experiment uses Optuna 4.5.0, XGBoost-CPU 3.0.5 and sklearn 1.7.2. Eight hyperparameters remain in the study, but the data, trees and fold counts are intentionally small. All estimator and CV work is single-threaded. The untouched holdout is never an eval_set, and the final parameters come from a new search on all development data—not the last outer fold.

import numpy as np
import optuna
from optuna.samplers import TPESampler
from sklearn.datasets import make_classification
from sklearn.model_selection import (StratifiedKFold, cross_val_score,
    train_test_split, GroupKFold, TimeSeriesSplit)
from sklearn.metrics import average_precision_score
from xgboost import XGBClassifier
optuna.logging.set_verbosity(optuna.logging.WARNING)
 
X, y = make_classification(n_samples=900, n_features=12, n_informative=6,
    weights=[.9,.1], flip_y=0, random_state=7)
dev, test = train_test_split(np.arange(len(y)), test_size=.2, stratify=y, random_state=8)
assert not set(dev) & set(test)
 
def model(params):
    return XGBClassifier(**params, tree_method='hist', eval_metric='logloss',
                         random_state=9, n_jobs=1, verbosity=0)
 
def tune(rows):
    inner = list(StratifiedKFold(2, shuffle=True, random_state=10).split(X[rows], y[rows]))
    for a,b in inner:
        assert not set(rows[a]) & set(rows[b])
        assert not (set(rows[a]) | set(rows[b])) & set(test)
    def objective(trial):
        params = {
            'n_estimators': trial.suggest_int('n_estimators', 15, 35, step=10),
            'max_depth': trial.suggest_int('max_depth', 2, 4),
            'learning_rate': trial.suggest_float('learning_rate', .03, .2, log=True),
            'subsample': trial.suggest_float('subsample', .7, 1.),
            'colsample_bytree': trial.suggest_float('colsample_bytree', .7, 1.),
            'min_child_weight': trial.suggest_int('min_child_weight', 1, 4),
            'reg_lambda': trial.suggest_float('reg_lambda', .01, 10., log=True),
            'scale_pos_weight': trial.suggest_float('scale_pos_weight', 1., 5.),
        }
        return float(cross_val_score(model(params), X[rows], y[rows], cv=inner,
            scoring='average_precision', n_jobs=1).mean())
    study = optuna.create_study(direction='maximize',
        sampler=TPESampler(seed=11, n_startup_trials=2))
    study.optimize(objective, n_trials=6)
    assert len(study.trials) == 6 and len(study.best_params) == 8
    return study
 
outer = StratifiedKFold(3, shuffle=True, random_state=12)
scores = []
for a,b in outer.split(X[dev], y[dev]):
    tr, va = dev[a], dev[b]
    assert not set(tr) & set(va)
    study = tune(tr)
    fitted = model(study.best_params).fit(X[tr], y[tr])
    score = average_precision_score(y[va], fitted.predict_proba(X[va])[:,1])
    scores.append(score)
    print('outer AP', score, 'inner selected AP', study.best_value)
final_study = tune(dev)
final = model(final_study.best_params).fit(X[dev], y[dev])
print('outer mean/std (not CI)', np.mean(scores), np.std(scores, ddof=1))
print('development selected AP', final_study.best_value)
print('final test AP', average_precision_score(y[test], final.predict_proba(X[test])[:,1]))
# Independent split-contract fixtures, not evaluation of this iid model.
groups = np.repeat(np.arange(6), 3)
for a,b in GroupKFold(3).split(np.zeros((18,1)), groups=groups):
    assert not set(groups[a]) & set(groups[b])
for a,b in TimeSeriesSplit(3, gap=2, test_size=3, max_train_size=6).split(np.arange(20)):
    assert a.max() + 2 < b.min() and len(a) <= 6
print('group/time split invariants passed')

Anatomy of the script

The objective closes over rows and inner, and it also reads the global X and y; a closure is not an access-control boundary. This implementation restricts fitting and scoring to X[rows] and y[rows], with inner indices constructed from that subset. Since each outer call passes only tr to tune, its outer-held-out rows are excluded by that indexing discipline—not by Python preventing access to them. Indices are local to the chosen subset; mapping back through rows makes the boundary inspectable. No learned preprocessing is needed for these generated numeric tree inputs. If adding imputation, encoding or SMOTE, place them in the estimator pipeline passed into inner CV.

average_precision defines the actual selection objective. XGBoost's eval_metric='logloss' does not change sklearn's scoring function; there is no early stopping here. The printed inner selected values demonstrate which numbers must not be described as independent estimates. The outer standard deviation is not a promise that the final holdout falls within one standard deviation.

Selection-optimism practice with an answer

Exercise: Repeat complete nested and non-nested studies with budgets 10, 30 and 100 across independent generated datasets. What trend must appear?

Answer: There is no mandatory monotonic observed gap. More candidates can increase selection of favorable noise, but can also discover truly better configurations; correlations and adaptive proposals matter. Record both distributions, support and compute, not invented 0.5%, 2% and 5% gaps. A single positive gap does not prove its exact cause. The two-candidate noise calculation above proves a possible optimism mechanism without relying on a lucky simulation.

(d) Production reality · 15 min

Leaderboard feedback: A public score used repeatedly for choices becomes validation feedback. Restrict submissions as part of a budget and preserve a truly independent final assessment. There is no universal safe number of daily submissions.

Recommendation deployment: Use grouped evaluation for cold-start users; use later interactions for known-user forecasting. Neither an invented Netflix incident nor a blanket “shared users always leak” rule is needed to explain the distinction.

Forecasting: Chronology, overlapping labels and point-in-time joins matter together. Forward-time folds cannot repair a feature accidentally computed from the whole series. Monitor later regime changes separately from the original holdout estimate; no historical correlation number establishes profitability.

Where this shows up in the rest of the plan

Learning rates in S099, dropout/decay in S101, prompt selection and calibration all consume evaluation information. Capstone reports should name the procedure, split unit, search budget, metric, support and final evaluation boundary. Production monitoring compares later data to this documented estimand, not to the maximum search score.

(e) Recall + stretch · 10 min

1. What does K-fold add? Several train/validation partitions and a summary, at multiple-fit cost; not independent measurements or universal unbiasedness.
2. Why stratify rare classes? To improve fold support where row-level splitting is justified; groups/time take precedence and very few positives remain a problem.
3. Nested CV in one sentence? Select only inside each outer-training subset, refit there, and evaluate on that split's untouched outer validation subset.
4. What does TPE add? Candidate proposals based on good/bad trial density models after startup; no guaranteed speedup over random search.
5. When GroupKFold? When the target is generalization across groups and group identity must not cross the boundary.
6. Why not quote the best inner score? The selection maximized that noisy score; it is part of fitting.
7. What about 15,625 grid candidates? Set a feasible randomized/adaptive budget or narrow a justified grid; do not promise a smaller search wins.
8. What does “test once” mean? Freeze choices before assessment. Reprinting the same number does not itself train a model, but using it to revise choices spends its independence.
9. What changes in time splits? Successive validation windows move later; training expands or rolls and may exclude a gap matched to the label horizon.
10. Stretch answer: The $a/2$ noise example establishes selection optimism. The larger empirical budget experiment estimates a distribution of effects; it cannot assume fixed percentage inflation or stable nested scores.

Explain-out-loud test

Draw the nested row boundaries; explain when a holdout is enough; distinguish fold standard deviation from uncertainty in future deployment; and state exactly which observations the next prediction is allowed to use.

What comes next

S096 perceptrons changes the model family, not the need for honest selection.

Sources and execution boundary

Use the sklearn 1.7 CV guide, nested-CV example, leakage guide, Optuna 4.5 TPE and pruning tutorial. The exact small nested study and split fixtures are CPU-executed. The proposed large-budget repetition is a practice design, not a reported benchmark.