S084 · The ML Mental Model — Features, Labels, Train/Val/Test

Module M11: Classical ML · Session 84 of 130 · Track: ML 🧠

What you'll be able to do after this session

• Explain The ML Mental Model to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S022 — Statistics — CLT, Hypothesis Testing, Confidence Intervals

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · But what is machine learning? — 3Blue1Brown — the neural-net intro doubles as the best ML intro anywhere.
• 🎥 Deep dive (30–60 min) · The Machine Learning Landscape — StatQuest full playlist opener — Josh Starmer's "Machine Learning Fundamentals: Cross Validation".
• 🎥 Hands-on demo (10–20 min) · Scikit-learn in 15 minutes — Python Simplified — end-to-end train/test split + fit + score.
• 📖 Canonical article · scikit-learn — An introduction to machine learning — the reference every course borrows from.
• 📖 Book chapter / tutorial · ISLR ch.2 — Statistical Learning (free PDF) — the beginner-friendly chapter of the field's standard textbook.
• 📖 Engineering blog · Google — Rules of Machine Learning — 43 hard-won rules Google engineers wish they'd known on day 1.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: What ML actually is.

Machine learning, stripped of hype, is exactly one idea: instead of writing rules to map inputs to outputs, we give the computer examples and let it fit a function. That's it. "Given these 10,000 emails labelled spam/not-spam, learn a function that classifies new emails."

Three ingredients, always:

1. Features (X) — the inputs. For an email: word counts, sender domain, hour of day, has-link. Each row is a sample; each column is a feature.
2. Labels (y) — the answers you want to predict (spam=1, not=0). If you have labels, it's supervised learning. If you don't, it's unsupervised (clustering, dimensionality reduction).
3. A model — a family of functions with tunable parameters (a line, a decision tree, a neural net). Training = finding the parameters that make the model's predictions match the labels best.

The single most important discipline: you must evaluate on data the model never saw during training. This is why we split data into train / validation / test. Train the model on train, tune hyperparameters on val, and — only once, at the very end — measure honesty on test. If you evaluate on training data, you're grading your own homework; a big enough model will memorise it perfectly and generalise terribly.

That gap between train performance and test performance has a name: generalisation. It's what all of ML is trying to maximise. Everything else — regularisation, cross-validation, learning curves — is machinery to detect and improve generalisation.

Gotcha to carry forever: If your test-set score looks amazing, you probably leaked test data into training. 90% of "too good to be true" ML results are data leakage.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The end-to-end ML pipeline looks the same whether you're doing housing prices or GPT-4:

Worked example — predicting house prices.

Suppose you have 10,000 house sales in Bengaluru with columns sqft, bedrooms, locality, age, price. The task is regression (predict a continuous price).

Worked example 2 — task taxonomy.

Every ML problem you meet fits into one of these boxes. Identify the box first, and 80% of the choices are made for you.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

End-to-end supervised ML in 40 lines. This is the entire recipe you'll reuse for the rest of the course.

# pip install scikit-learn pandas
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np
 
# ---------- 1. Load data ----------
data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target       # X: 8 numeric features, y: median house value ($100k)
print("shape:", X.shape, "  target head:", y.head().tolist())
 
# ---------- 2. Split — always first! ----------
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=42)
print(f"train={len(X_train)}  val={len(X_val)}  test={len(X_test)}")
 
# ---------- 3. Train a baseline (linear regression) ----------
lr = LinearRegression().fit(X_train, y_train)
val_rmse_lr = np.sqrt(mean_squared_error(y_val, lr.predict(X_val)))
print(f"linear reg   val RMSE = {val_rmse_lr:.3f}")
 
# ---------- 4. Train a stronger model ----------
rf = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1).fit(X_train, y_train)
val_rmse_rf = np.sqrt(mean_squared_error(y_val, rf.predict(X_val)))
print(f"random forest val RMSE = {val_rmse_rf:.3f}")
 
# ---------- 5. Pick the winner by val, evaluate ONCE on test ----------
best = rf if val_rmse_rf < val_rmse_lr else lr
name = "RF" if best is rf else "LR"
test_pred = best.predict(X_test)
print(f"CHOSEN {name}  test RMSE = {np.sqrt(mean_squared_error(y_test, test_pred)):.3f}")
print(f"CHOSEN {name}  test R^2 = {r2_score(y_test, test_pred):.3f}")
 
# ---------- 6. Sanity: train vs val vs test scores ----------
for split_name, Xs, ys in [("train", X_train, y_train), ("val", X_val, y_val), ("test", X_test, y_test)]:
    rmse = np.sqrt(mean_squared_error(ys, best.predict(Xs)))
    print(f"  {split_name:5s} RMSE = {rmse:.3f}")

Observe when you run it:

1. X.shape is (20640, 8) — 20k homes, 8 features. Always print shapes first.
2. Split sizes: 12,384 / 4,128 / 4,128 = 60/20/20.
3. Linear reg val RMSE ~0.73, Random Forest val RMSE ~0.50 — the RF is dramatically better.
4. Train RMSE on RF is much lower than val/test — that's slight overfitting (topic of S087).
5. Test RMSE ≈ val RMSE — good sign, no distribution shift between the two.

Try this modification: replace the final train_test_split with KFold(n_splits=5) cross-validation and report mean±std of val RMSE. You'll see the variance in your metric — a single train/val split can be misleading on small data.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

The single biggest ML production failure mode: data leakage. Someone accidentally includes a feature that's derived from the label, or splits data before deduplication, or fits the scaler on the whole dataset before splitting. Result: 99% offline accuracy, model is useless in production. Kaggle has a decade of stories.

Train/serve skew. The features you compute at training time (batch SQL over the last 6 months) are subtly different from what you compute at serve time (real-time from a feature store). Uber famously invested in Michelangelo, Airbnb built Zipline, Meta has FBLearner Flow — all to make training features and serving features come from the same code. If you take one thing from this session: the feature transformation is part of the model. Serialise both together.

Distribution shift. The world in Jan is not the world in July. Users, prices, behaviours drift. A model trained on pre-COVID travel data cratered in April 2020. Ops teams monitor input-feature distributions (KL divergence, PSI) and output-prediction distributions in prod, alerting when they wander. Retraining cadence (nightly? weekly? monthly?) is a real design decision.

Baselines matter more than SOTA. A well-tuned logistic regression beats a badly-tuned neural net 80% of the time. Always report your baseline. If your fancy model beats "predict the mean" by 1%, you don't have a model, you have a fancy way to predict the mean.

Reproducibility. Seed every RNG (random_state=42 everywhere), pin library versions, checkpoint data snapshots. "It worked on my machine last Tuesday" is not shippable.

How top teams structure it. Feature store (Feast, Tecton, Vertex, Databricks) → training pipeline (Kubeflow, Airflow, Metaflow) → model registry (MLflow) → serving (TF Serving, Seldon, KServe, or just a FastAPI + Docker) → monitoring (Evidently, Arize, WhyLabs). Same 5 components at every FAANG.

Google's "Rules of Machine Learning" doc (in the pre-read) is basically 43 hard-won versions of these lessons. It is the single highest-value document in applied ML.

(e) Quiz + exercise · 10 min

1. What are features, labels, and a model — in one sentence each?
2. Why do we split into train/val/test instead of just train/test?
3. What is data leakage and give a concrete example of how it happens.
4. When should you use a regression model vs a classification model? Give one example of each.
5. What is train/serve skew and why does it usually only get noticed after deployment?

Stretch (connects to S015 — Big-O): Random Forest is O(n·log n) per tree at training time and O(log n) per prediction, times ~100 trees. Estimate wall-clock training time on 1 M rows with 100 features if a single tree on 10k rows takes 200 ms. What's the memory footprint you should expect?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is The ML Mental Model? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S085): Linear Regression from Scratch (numpy)
• Previous (S083): OWASP Top 10, Secrets Mgmt & Threat Modelling
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.