The ML Mental Model — Features, Labels, Train/Val/Test

Why this session exists

For supervised tabular learning, a useful scaffold is features X, targets y, a fitted predictor and a held-out metric. Unsupervised learning, reinforcement learning and structured data need different representations and feedback; this scaffold is not a definition of all ML.

The important question is not merely “can I call fit?” It is what information would be available when this prediction is made, and what future use does my test reproduce? A technically correct optimizer can solve the wrong problem perfectly. This lesson builds the experiment contract before introducing more elaborate models.

By the end, describe a house-price dataset as 1,000 rows and eight input features, explain why a label recorded after a sale cannot be a prediction-time feature, and compare a fitted model with a mean-price baseline. Then carry that reasoning over to repeated customers and time-dependent observations.

Prerequisites

You need Python functions, NumPy array indexing, and the meaning of a sample mean. A DataFrame is convenient but not necessary: the main experiment runs entirely on small synthetic NumPy arrays. No downloaded housing data, service, GPU or scikit-learn installation is required for that experiment. The separate scikit-learn 1.7 built-in-data example was also tested on CPU; it is not a clinical or California-housing validation.

(a) Intuition · 5 min

Think of training as practice problems, validation as a practice exam used to choose a study strategy, and test as an independent exam used to assess the chosen strategy. Looking at a score is not magical contamination. Changing the strategy in response to it makes the score part of the selection process. A frozen model may be rescored to check reproducibility; that does not create a new independent measurement.

The analogy has limits. Future houses may come from different neighbourhoods; tomorrow's fraud may use a new attack. An independent exam estimates performance on the population it samples, not every possible future population. Similar train, validation and test scores are evidence to interpret, not proof of generalization under distribution shift.

The five nouns to own before you touch a model

• Feature: an input available at the prediction deadline. Square footage may be available before sale; the final sale price is not.
• Label: the outcome the task asks us to predict. State its definition and time horizon: “customer cancels within 30 days” is different from “ever cancelled.”
• Model: a function family and fitted parameters. Linear regression uses weights and an intercept; a tree uses learned splits and leaf predictions.
• Loss: an objective for fitting or comparing predictions. Gradient-based methods need suitable derivatives or subgradients; not every ML fitting algorithm is gradient-based, and not every loss is differentiable everywhere.
• Metric: a reported measure selected to reflect the decision. It may equal the training loss or measure something different, such as recall at a specified false-positive budget.

A useful supervised notation is $\hat y_i=f(x_i;\theta)$. Empirical mean loss is $\frac{1}{n}\sum_i\ell(y_i,\hat y_i)$; a regularized objective may add a penalty on $\theta$. Minimizing that quantity is not the same as maximizing business value unless the task and decision costs connect them.

How ML got here: mechanisms, not inevitability

The perceptron is an early learned linear classifier, not the first possible learning procedure. The 1986 backpropagation paper popularized a way to fit multilayer representations; neural-network research did not then sit idle for 20 years. Breiman's 2001 random-forest paper describes randomized tree ensembles. Scikit-learn's history dates the Google Summer of Code project to 2007 and its first public release to 2010—not a 2007 v0.1 release. The 2012 AlexNet paper combines a large convolutional net, GPU implementation, nonlinearities and regularization on ImageNet; its reported contest numbers are specific to that experiment, not proof of one 10-point effect caused only by GPUs. XGBoost's 2016 paper develops a scalable tree-boosting system; it does not prove that boosting beats neural nets on almost every tabular problem. ChatGPT's consumer popularity likewise supplies no measured “80% of every real system” share for classical ML.

Read this history as a growing toolkit. Trees, linear models and neural models differ in inductive bias and fitting procedure. Choose by the prediction contract and held-out evidence, not by declaring the newest family universally dominant.

The ML lifecycle in one picture

Separate deterministic schema checks from transformations that learn statistics. The latter must be fitted inside the training boundary, including each cross-validation fold.

The held-out rows receive the training-fitted transformation, not their own independently fitted scaler. If selection is complete, refitting the selected recipe on train plus validation uses more development data; the test remains excluded. After deployment, monitor fresh outcomes under a specified protocol rather than repeatedly using the old test to choose improvements.

The X, y matrix — scope and shapes

For 1,000 houses with eight numerical features, X.shape == (1000, 8) and y.shape == (1000,). Row i's target must stay aligned with row i's inputs through every shuffle and join. A multi-output regression has a matrix of targets rather than a single vector. Images and sequences often remain higher-dimensional tensors; there is no requirement to flatten everything into a tabular matrix.

Write a feature contract alongside the shapes:

A correctly shaped array can still violate every information boundary in this table.

Splits — the three-way divide

Train determines fitted parameters and preprocessing statistics. Validation selects hyperparameters, thresholds, feature recipes or stopping points. Test evaluates the frozen choice. Percentages such as 70/15/15 are planning examples, not universal rules: choose enough observations and positive cases to support the precision you need.

Repeatedly searching against validation also fits its noise. Cross-validation reduces dependence on one arbitrary split but does not make unlimited adaptive search unbiased. In k-fold CV, fit the entire pipeline on k−1 folds and score the remaining fold, repeating k times. The fold-score standard deviation describes variation between folds; it is not automatically a confidence interval because training sets overlap. Nested CV can separate hyperparameter selection from outer evaluation.

Choose a split from the deployment question:

1. Independent examples from the same population: a seeded random split may be appropriate.
2. New patients, accounts or devices: hold out whole entities, not random measurements from the same entity.
3. Predicting future outcomes: train on earlier data, validate later, test later still. Respect label maturation: a 30-day churn label is not available on the feature timestamp itself.
4. Future activity of known customers: customer overlap may be legitimate, but future events still cannot enter earlier features. This is a different question from performance on unseen customers.

For Jan 2023–Dec 2024 churn data, one possible design uses training feature dates through August 2024, September–October for validation, and November–December for test. A 30-day outcome horizon requires observing outcomes beyond each feature date and may require an embargo at cutoffs. Document those dates; merely sorting rows is insufficient.

Loss vs metric — a concrete table

For residuals 1, −1 and 4, MSE is 6, RMSE is approximately 2.449, and MAE is 2. Changing only the largest residual from 4 to 8 changes MSE to 22 and MAE to 10/3. This explains the different penalties without claiming that MAE is always the correct business objective or always slower to optimize.

If 500 of 500,500 transactions are fraudulent, predicting “not fraud” has accuracy about 99.9001% and fraud recall zero. Accuracy is not meaningless, but alone it hides the failure. Report precision and recall at a capacity/cost-aware threshold, the confusion counts, and a precision-recall summary with its convention named. Average precision and trapezoidal PR area are not interchangeable. None of these metrics is automatically “imbalance-safe.”

Deriving why selection needs an independent check

Suppose two equally good candidates have true error 10. Sampling noise makes validation errors 9 and 11. Choosing the first and reporting 9 as its expected future error is optimistic even though neither candidate trained on those validation rows. Searching more candidates creates more chances to select a favourable fluctuation. That is why selection is a form of fitting.

Independence does not guarantee a favourable result, and a test estimate still has uncertainty. It gives you a measurement that was not optimized by the procedure being measured. A fresh prospective test is particularly useful when the intended deployment differs from the historical dataset.

Rules, learned models and hybrids

For an exact requirement such as “block a transfer above an account's authorized limit,” implement the rule directly and test its boundary conditions. A classifier approximating that rule introduces needless uncertainty. For a noisy ranking task such as prioritizing potentially fraudulent transactions for review, compare a learned score against simple rules on the same held-out cases.

A hybrid can use a learned score for prioritization while deterministic authorization gates enforce permitted actions. This adds integration work: a gate may mask a model regression, so measure both score quality and end-to-end decisions. Neither a model nor a rule system is inherently fair or infallible. Inspect how labels were produced, who is missing from the data, and error rates in relevant groups. Prediction from observational associations alone does not establish causal effects.

(c) Hands-on · 25 min

This bounded CPU program constructs a linear signal with noise, creates disjoint indices, fits training-only normalization and least squares, and measures a mean baseline. The assertions check mechanics and this particular experiment, not a universal quality promise.

import numpy as np
 
rng = np.random.default_rng(84)
X = rng.normal(size=(240, 3))
y = 4 + X @ np.array([2.0, -1.0, 0.5]) + rng.normal(0, 0.2, 240)
idx = rng.permutation(len(y))
train, val, test = idx[:144], idx[144:192], idx[192:]
assert not (set(train) & set(val) or set(train) & set(test) or set(val) & set(test))
 
def fit(rows, data=X):
    mu = data[rows].mean(axis=0)
    scale = data[rows].std(axis=0)
    scale = np.where(scale == 0, 1.0, scale)
    design = np.column_stack([np.ones(len(rows)), (data[rows] - mu) / scale])
    weights = np.linalg.lstsq(design, y[rows], rcond=None)[0]
    return mu, scale, weights
 
def predict(state, rows, data=X):
    mu, scale, weights = state
    design = np.column_stack([np.ones(len(rows)), (data[rows] - mu) / scale])
    return design @ weights
 
def rmse(actual, predicted):
    return np.sqrt(np.mean((actual - predicted) ** 2))
 
state = fit(train)
baseline = np.full(len(val), y[train].mean())
base_error = rmse(y[val], baseline)
val_error = rmse(y[val], predict(state, val))
assert val_error < base_error
print('validation baseline/model:', base_error, val_error)
 
# Development-only learning curve, no assumed monotonicity.
for n in (8, 24, 72, 144):
    small = fit(train[:n])
    print('curve:', n, rmse(y[train[:n]], predict(small, train[:n])),
          rmse(y[val], predict(small, val)))
 
# Selection is now frozen. Refit development data, evaluate test once.
final = fit(np.concatenate([train, val]))
print('final test:', rmse(y[test], predict(final, test)))
 
# Separate intentionally invalid experiment, not a deployable feature.
leaky_X = np.column_stack([X, y])
leaky = fit(train, leaky_X)
assert rmse(y[val], predict(leaky, val, leaky_X)) < 1e-10
 
# Ordinary least squares with intercept is invariant to invertible scaling.
a = np.column_stack([np.ones(len(train)), X[train]])
b = np.linalg.lstsq(a, y[train], rcond=None)[0]
raw_predictions = np.column_stack([np.ones(len(val)), X[val]]) @ b
assert np.allclose(raw_predictions, predict(state, val))
print('split, baseline, target-leak and scaling checks passed')

What each block does

The baseline uses the training target mean. It does not peek at validation labels to choose the constant. lstsq solves a small overdetermined linear system without explicitly inverting a matrix. Each smaller learning-curve model fits its own mean and scale. The final refit excludes test rows.

Appending y to X creates a near-zero-error predictor because the answer is now an input. A train/test split cannot repair this feature definition. For this full-column-rank design, unregularized least squares with an intercept gives the same predictions after invertible affine feature scaling in exact arithmetic. The fitted coefficients simply express the same unique predictor in new coordinates. With a rank-deficient design, training fitted values still agree, but a solver’s minimum-norm choice can change predictions away from the training feature relationships. NumPy’s lstsq contract specifies that minimum-norm choice. For example, if two training columns are identical, equally splitting their coefficient is minimum-norm; rescaling only one column changes that preference. A future row on which the columns differ can then receive a different prediction. The earlier “20–40% optimistic from global scaling” figure was neither measured nor generally valid. Global scaling still violates the proper experiment boundary; this particular model may hide the mistake because its predictions do not change. Regularization, feature selection or other estimators can make preprocessing choices consequential.

The learning curve is an observation, not a forced plot shape. Eight training points do not guarantee zero error, and more training points do not guarantee a monotonically improving validation score on every sample. Large gaps suggest investigating variance, distribution differences and implementation errors; small gaps can also mean both scores are poor.

Optional scikit-learn 1.7 workflow

The official 1.7 documentation supports the following API pattern. This small built-in diabetes dataset avoids a network fetch, and this framework example was executed in the existing scikit-learn 1.7.2 CPU runtime. Its random split illustrates the mechanics, not a clinical validation design. No printed performance values are promised.

from sklearn.datasets import load_diabetes
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import root_mean_squared_error
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
 
X, y = load_diabetes(return_X_y=True, scaled=False)
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.2, random_state=84
)
pipe = make_pipeline(StandardScaler(), LinearRegression())
folds = KFold(n_splits=5, shuffle=True, random_state=84)
errors = -cross_val_score(pipe, X_dev, y_dev,
                         cv=folds, scoring='neg_root_mean_squared_error')
print('CV RMSE mean and fold spread:', errors.mean(), errors.std())
# Freeze the recipe before this final fit and evaluation.
pipe.fit(X_dev, y_dev)
baseline = DummyRegressor(strategy='mean').fit(X_dev, y_dev)
print('test baseline:', root_mean_squared_error(y_test, baseline.predict(X_test)))
print('test model:', root_mean_squared_error(y_test, pipe.predict(X_test)))

root_mean_squared_error was added in scikit-learn 1.4 and is documented in 1.7. It replaces this article's obsolete mean_squared_error(..., squared=False) usage. The pipeline is cloned and fitted within each CV fold; it cannot protect against incorrect labels, duplicated patients or future information already inside X.

(d) Production reality · 15 min

Five leakage mechanisms and their repairs

1. Preprocessing leakage: fitting a scaler, vocabulary, feature selector or imputer on held-out data. Put the fitting operation inside each training fold; reuse that fitted transform for its validation rows.
2. Target leakage: joining the eventual refund or fraud outcome into the feature table. Define the prediction deadline and audit every feature's lineage and availability timestamp.
3. Temporal leakage: rolling statistics include the current target or future events. Use lagged/as-of computations and test them with a tiny event timeline.
4. Entity leakage: multiple observations of one patient or duplicate document appear in supposedly independent folds. Split by the deployment entity and group duplicates before splitting.
5. Selection leakage: choosing features or thresholds after seeing the final test score. Treat that dataset as development data thereafter and obtain a genuinely fresh final evaluation.

A business failure does not, by itself, identify one of these mechanisms. For a hypothetical price model, changes in input frequencies are covariate shift, while changes in the relationship between inputs and outcomes are concept drift. Both require investigation, and neither is proved by a single aggregate metric.

A deployment record should include feature/label definitions, data versions, split dates or entity keys, model and preprocessing versions, baseline and model scores, subgroup counts, and a response plan for regression. Decide when to abstain or route to a human. Monitoring feature drift alone is not monitoring correctness: acquire delayed outcome labels and compare operational error costs.

Three production case analyses

Adaptive leaderboard use: choosing among hundreds of submissions on a public score is another fitting loop. A private split can disagree because of selection noise or a different population; a 200-place crash is not a universal law. Keep a stable local development protocol and a fresh final assessment. Even local CV can be overfit by repeated choices.

Price-model operations: separate a company’s financial disclosures from a causal diagnosis of its model. In a hypothetical buying system, examine price forecast residuals by time/region, inventory exposure, transaction delays, acquisition policy and operational constraints. Covariate shift changes P(X); concept drift changes the conditional relationship P(y|X). Losses alone establish neither. Retraining without diagnosing label delay or decision economics can repeat the failure.

Recruitment and label bias: labels such as “previously hired” encode a historical selection process, not automatically job performance. A proxy feature can reproduce exclusion. Removing a word does not remove all correlated proxies. Compare a job-relevant outcome definition, representative sampling, error rates and human review under appropriate legal governance; a work-sample score is another measurement to validate, not an automatic fairness cure.

Where this shows up in the rest of the plan

Linear and logistic regression instantiate the function and loss. Regularization changes the fitted objective. Trees and ensembles use different fitting algorithms but still require valid information boundaries. Neural models do not escape any of these constraints. The next study task is to derive a least-squares gradient while keeping this experiment contract fixed; continue to linear regression, logistic regression, regularization, bias–variance, trees, forests, boosting and model selection. The original S096=MLOps label was wrong: that local lesson is the perceptron. Use the explicit topic rather than that obsolete cross-reference.

(e) Recall + stretch · 10 min

Attempt these before opening the answers.

1. Give X/y shapes for 1,000 houses and eight features. What changes for two target values per house?
2. Why can selecting the best of 100 validation scores overfit without a gradient touching validation data?
3. Calculate MSE, RMSE and MAE for residuals 1, −1, 4. Which changes most sharply if 4 becomes 8?
4. A mean baseline beats your model. Must the data be defective?
5. Can a pipeline prevent a feature computed from the future? Can a random split fix it?
6. Why is 99.9% accuracy inadequate evidence in the fraud example?
7. Design a split for unseen patients, then for future visits of already-known patients.
8. Interpret a learning curve with high, similar train and validation errors.
9. What evidence would distinguish input drift from a changed input–outcome relationship?
10. Explain why the intentionally leaky NumPy model is invalid even though validation error is nearly zero.

Explain-out-loud test

Draw the lifecycle from memory, trace exactly which rows fit a scaler, and explain why an accurate score on the wrong population can be irrelevant. Then give one task suited to a deterministic rule, one to a learned ranking model, and one needing both.

Reviewed sources and execution boundary

• scikit-learn 1.7.2: common pitfalls, especially data leakage and train-only preprocessing.
• scikit-learn 1.7.2: cross-validation, including selection, pipelines, grouped and temporal splits.
• scikit-learn 1.7.2: root mean squared error, signature and version history.

The NumPy scaffold is a bounded local arithmetic/mechanics test, not a reproduced housing benchmark. The built-in-data scikit-learn example is executed for API/pipeline mechanics, not clinical validity. Historical anecdotes, absolute learning-curve predictions and invented numerical outputs from the earlier version are not retained as facts.

Optional pre-read and recall bridge

Original supplementary resources: 3Blue1Brown neural-network intuition, StatQuest cross-validation, scikit-learn crash course, official introduction, Google terminology, and Kaggle model introduction. They are optional, not independently revalidated video/course claims.

The original recall also asked about heavy outliers and CV: MAE penalizes residual magnitude linearly and MSE quadratically, so MAE is less dominated by one huge residual, but the right loss follows error costs and data assumptions; its constant subgradient away from zero does not universally imply slower convergence. K-fold CV rotates validation folds through development data with preprocessing refitted each time; grouped/time splits and nested selection may be required regardless of dataset size. Neither a pipeline nor “test once” alone repairs a wrong deployment population.