S088 · Bias–Variance Trade-off & Learning Curves

Module M11: Classical ML · Session 88 of 130 · Track: ML ⚖️

What you'll be able to do after this session

• Explain Bias–Variance Trade-off & Learning Curves to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S087 — Regularization — L1, L2, Elastic Net

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Machine Learning Fundamentals: Bias and Variance — StatQuest — the darts-on-a-target picture that makes it stick.
• 🎥 Deep dive (30–60 min) · Bias-Variance and Learning Curves — Andrew Ng CS229 — the classic Stanford lecture treatment.
• 🎥 Hands-on demo (10–20 min) · Learning curves with scikit-learn — Sebastian Raschka — plotting them to diagnose your model.
• 📖 Canonical article · scikit-learn — Validation curves and learning curves — the reference and API.
• 📖 Book chapter / tutorial · ISLR ch.2.2 — Assessing Model Accuracy (free PDF) — the canonical bias-variance derivation.
• 📖 Engineering blog · Google — Rules of Machine Learning, Rules 12-24 — practical bias-variance advice from Google engineers.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The fundamental ML trade-off.

There are exactly three ways your model can be wrong on new data, and the total error decomposes into them exactly:

Total Error = Bias² + Variance + Irreducible Noise

• Bias is systematic error from the model being too simple to capture the pattern. A straight line trying to fit a curve. It predicts the same wrong thing every time, no matter what training data you give it. This is underfitting.
• Variance is error from the model being too sensitive to the specific training sample. A wildly wiggly curve that fits your 100 training points perfectly but predicts nonsense on the 101st point. Re-sample the training data and you get a completely different model. This is overfitting.
• Noise is the inherent randomness in the data (measurement error, unmodelled variables). You can't eliminate it, but you can stop pretending to. If two houses at the exact same address sold for different prices, no model on Earth can predict that difference.

The dartboard analogy: bias is where the center of your darts lands (are they on the bullseye on average?), variance is how tightly they cluster. Ideal: low bias and low variance. High bias low variance = missing the bullseye consistently. Low bias high variance = darts averaging around the bullseye but scattered wildly.

You can trade bias for variance by changing model complexity: simpler models (linear reg, shallow trees) → high bias, low variance. Complex models (deep nets, very deep trees) → low bias, high variance. Regularisation adds bias to reduce variance. The sweet spot is where the total error curve bottoms out.

Learning curves are your diagnostic tool: plot train and val error as a function of training-set size. Big gap that doesn't close = high variance (need more data, more regularisation, simpler model). Both curves flat and high = high bias (model too simple; use richer features or a bigger model).

Gotcha to carry forever: More data helps variance problems. It does not help bias problems. If your linear model is underfitting a non-linear pattern, another million rows won't save you.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The classic double-U diagram:

The math (memorise this). For a fixed x, if we train many models on different samples:

• Bias(x) = E[ŷ(x)] - y_true(x) (average prediction minus truth)
• Variance(x) = E[(ŷ(x) - E[ŷ(x)])²] (spread of predictions)
• MSE(x) = Bias²(x) + Variance(x) + σ² (σ² = irreducible noise)

Diagnosis table — learn to read learning curves:

Worked example — degree-of-polynomial sweep.

Fit polynomials of degrees 1 to 15 to noisy y = sin(x) + ε with 20 training points.

Total error is minimised at degree 3. The train curve keeps going down forever; the val curve is a U.

Ensembling as a variance killer. Take 100 high-variance models, average their predictions → predictions cluster around the truth (variance drops by ~1/n if models are independent). This is why Random Forest (S090) works.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Reproduce the classic bias-variance picture by fitting polynomials of different degrees to a noisy sine wave, then plot learning curves.

# pip install numpy scikit-learn matplotlib
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split, learning_curve
from sklearn.metrics import mean_squared_error
 
rng = np.random.default_rng(1)
 
# ---------- 1. Noisy sine ----------
def gen(n=200, noise=0.3):
    x = np.sort(rng.uniform(0, 2*np.pi, n))
    y = np.sin(x) + rng.normal(0, noise, n)
    return x.reshape(-1, 1), y
 
X, y = gen()
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
 
# ---------- 2. Sweep polynomial degree ----------
print(f"{'deg':>4} {'train MSE':>10} {'test MSE':>10}")
for deg in [1, 3, 5, 8, 12, 20]:
    m = make_pipeline(PolynomialFeatures(deg), LinearRegression())
    m.fit(X_tr, y_tr)
    tr = mean_squared_error(y_tr, m.predict(X_tr))
    te = mean_squared_error(y_te, m.predict(X_te))
    print(f"{deg:>4} {tr:>10.4f} {te:>10.4f}")
 
# ---------- 3. Empirical bias-variance decomposition ----------
def bv_decomp(deg, n_trials=200, n_train=30):
    x_test = np.linspace(0, 2*np.pi, 100).reshape(-1, 1)
    preds = np.zeros((n_trials, 100))
    for i in range(n_trials):
        xi, yi = gen(n_train)
        preds[i] = make_pipeline(PolynomialFeatures(deg), LinearRegression()).fit(xi, yi).predict(x_test)
    truth = np.sin(x_test).ravel()
    bias_sq = np.mean((preds.mean(axis=0) - truth) ** 2)
    variance = np.mean(preds.var(axis=0))
    return bias_sq, variance
 
print(f"
{'deg':>4} {'bias^2':>10} {'variance':>10} {'sum':>10}")
for deg in [1, 3, 5, 8, 12, 20]:
    b, v = bv_decomp(deg)
    print(f"{deg:>4} {b:>10.4f} {v:>10.4f} {b+v:>10.4f}")
 
# ---------- 4. Learning curves — the field's go-to diagnostic ----------
m = make_pipeline(PolynomialFeatures(5), LinearRegression())
sizes, train_scores, val_scores = learning_curve(
    m, X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 10),
    scoring="neg_mean_squared_error")
print(f"
{'n_train':>8} {'train MSE':>10} {'val MSE':>10}")
for n, tr, va in zip(sizes, -train_scores.mean(axis=1), -val_scores.mean(axis=1)):
    print(f"{int(n):>8} {tr:>10.4f} {va:>10.4f}")

Observe when you run it:

1. Degree 1: high train MSE + high test MSE → underfitting (high bias).
2. Degree 3-5: both low, close together → sweet spot.
3. Degree 20: train MSE near 0, test MSE explodes → overfitting (high variance).
4. Empirical bias² decreases with degree; variance increases. The sum is a U-shape.
5. Learning curve: with only 20 training points, val MSE is high; adds points → val drops and converges to train. That's the "variance shrinks with more data" law made visible.

Try this modification: double the noise (noise=0.6) and rerun the learning curve. Both curves flatten at a higher floor — you've just hit the irreducible noise term. No amount of data or model complexity can push below it.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Diagnosing production models. When a stakeholder says "the model is bad", your first job is to figure out which problem: bias, variance, or distribution shift. Look at:

• Train vs val vs test scores side by side.
• Learning curves.
• Prediction error distribution across slices (age, geography, product category). A model with low overall error but terrible error on one segment has a fairness or slice problem, not a bias-variance problem.

Adding more data is expensive. Labelling costs money (Scale AI, Surge, Mechanical Turk). Before commissioning 100k more labels, always run a learning curve — if the curve has flattened, more data won't help; you need a better model or better features.

The "high-bias trap" in production. Team ships a logistic regression, hits ceiling, adds more data for a year — no improvement. Someone finally tries XGBoost and gets +8% overnight. The lesson: when adding data doesn't help, stop adding data.

Deep learning myth-buster. "Bigger model always wins" is approximately true only if you have (a) enough data and (b) enough regularisation to control the variance explosion. GPT-3 has 175 B parameters but was trained on 300 B tokens and heavily regularised. Naively adding parameters to a small dataset is the fastest way to overfit.

Cross-validated learning curves for production planning. "If we double our training data, how much better does our model get?" Fit a power-law to your learning curve and extrapolate. Google Brain and Baidu have published papers showing test error decays as a predictable power law in dataset size — you can plan your data acquisition.

Ensembling. In ML competitions (Kaggle, KDD Cup), the winning model is almost always an ensemble of models with different bias-variance profiles: a boosted tree + a neural net + a linear model, averaged. The ensemble has lower variance than any single model. Random Forest (S090) is the industrial version of this idea.

The "curse of dimensionality" angle. In very high dimensions, variance explodes because data points get sparse — every point is far from every other point. Regularisation and dimensionality reduction (PCA) tame it.

(e) Quiz + exercise · 10 min

1. Write the bias-variance decomposition of MSE and explain each term in one sentence.
2. Give the dartboard analogy for high bias vs high variance.
3. From a learning curve with train and val error both high and flat, what do you diagnose and how do you fix it?
4. Why does adding more data help variance but not bias?
5. Name three techniques that trade increased bias for reduced variance.

Stretch (connects to S087 — Regularization): Explain, using the bias-variance decomposition, why Ridge regression usually beats plain OLS on out-of-sample data even though OLS has zero bias and Ridge does not.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Bias–Variance Trade-off & Learning Curves? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S089): Decision Trees — Gini, Entropy, Splits
• Previous (S087): Regularization — L1, L2, Elastic Net
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.