S088 · Bias–Variance Trade-off & Learning Curves
The single most important theoretical idea in ML — decompose error into bias, variance, and noise, then read learning curves to know which one is killing you.
🎯 Decompose prediction error into bias + variance + noise, read learning curves fluently, and choose the right next action when your model is stuck.
Why this session exists
The bias–variance decomposition is the single lens through which every experienced ML engineer looks at a failing model. You look at the learning curve, you know instantly whether the model is underfitting (bias) or overfitting (variance), and you know exactly what to change next — more data, more features, more regularization, or a different model family. Without this lens, ML debugging is guessing. With it, you can systematically improve any pipeline.
- State the bias-variance decomposition formula and explain each term.
- Read a learning curve and diagnose high bias vs high variance in under 30 seconds.
- List the correct next action for each pathology (more data, more features, more regularization, etc).
- Explain double descent — the modern twist that overturns the classical U-curve.
- Compute bias and variance empirically by bootstrapping a model.
Prerequisites
- S084 · ML mental model — train / val / test.
- S085 · Linear regression from scratch.
- S087 · Regularization.
(a) Intuition · 5 min
Archer A has a strong bow arm, always pulls the same way, but her sight is misaligned. Her arrows form a tight cluster — off-centre. High bias, low variance. Her problem is systematic; you fix it by adjusting the sight (or getting a better archer).
Archer B has a perfectly calibrated sight but shaky hands. Her arrows are all over the target, average roughly at the bullseye, but no single shot lands close. Low bias, high variance. Her problem is noise; you fix it with more practice (more data) or steadier hands (regularization).
An ML model's prediction error, averaged over datasets drawn from the same distribution, decomposes into three pieces:
E[(y - ŷ)²] = Bias(ŷ)² + Var(ŷ) + σ²
Bias = how far the average prediction is from the truth (systematic error). Variance = how much the prediction wobbles as training data changes (fitting noise). σ² = irreducible noise in the labels themselves.
The three pieces of every prediction error
- Bias — the model can't represent the true function. Fixes: more expressive model (polynomial features, tree, NN), more features.
- Variance — the model overreacts to noise in the training data. Fixes: more data, regularization, simpler model, ensemble.
- Noise (σ²) — irreducible. Even a perfect model can't beat it. Recognise when you've hit the noise floor and stop chasing it.
The intellectual history
- 1948Bias-variance in signal processingStatistical estimation theory formalises MSE = bias² + variance. Applied to filter design.
- 1992‘Neural networks and the bias/variance dilemma’ · GemanThe canonical ML paper. Frames overfitting for the ML community.
- 2001Random Forest paper · BreimanEnsembles as variance reduction. Bagging cuts variance without touching bias.
- 2019‘Deep Double Descent’ · Nakkiran et alThe classical U-curve gets a second dip for overparameterised models. Rewrites the textbook.
- 2020Neural scaling laws · Kaplan et alBias, variance, and noise as functions of model size, data size, compute. The theory of ‘just make it bigger.’
(b) Visual walkthrough · 15 min
The classical U-curve
Traditional textbook says: total error = bias² + variance + noise. Complexity increases → bias falls monotonically, variance rises. Total error is U-shaped. Pick the model at the U's bottom.
Reading a learning curve — the debugging playbook
What a learning curve tells you
The action matrix — what to change based on diagnosis
Model too simple
- → Add features (polynomial, interactions, engineered).
- → More expressive model (deeper tree, NN, kernel).
- → Reduce regularization.
- → Train longer / more epochs.
- ❌ NOT: more data (won't help).
Model too complex
- → More training data (single biggest lever).
- → Regularization (L1/L2/dropout).
- → Ensemble methods (bagging, random forest).
- → Simpler model (fewer features, shallower tree).
- → Data augmentation (for vision/text).
Both errors low, close, and stuck
- → Get better labels (harder to game).
- → Add features that reduce label noise.
- → Report the ceiling to the business honestly.
- → Focus on latency, cost, robustness instead.
- ❌ NOT: keep tuning hyperparameters (waste of time).
The double descent surprise
"Bias and variance trade off against each other, so there's always a sweet spot in the middle. More model capacity means less bias and more variance — that's a law, and the U-shaped test-error curve always applies."
The decomposition is exact, but the "capacity up ⇒ variance up" rule is not a law — it is an observation about the underparameterised regime. Past the interpolation threshold, where the model has enough capacity to fit the training data exactly, test error can fall again. That double-descent behaviour is why massively overparameterised networks generalise at all, and it is why "the model is too big, it will overfit" is not a valid argument by itself.
The myth is sticky because the classic U-curve is genuinely correct everywhere you first encounter it: polynomial degree on a small dataset, tree depth, k in kNN. In all those settings you stay well below the interpolation threshold, so the observed relationship holds perfectly and gets encoded as a universal principle. It also survives because the bias-variance decomposition is a theorem — true always — and people transfer that certainty onto the U-shaped curve, which is only an empirical regularity in one regime.
Measure both components directly instead of reasoning about them. Bootstrap-resample the training set, refit, and look at how predictions for a fixed test point behave:
import numpy as np
preds = []
for _ in range(200):
idx = np.random.randint(0, len(Xtr), len(Xtr))
m = make_model().fit(Xtr[idx], ytr[idx])
preds.append(m.predict(Xte))
P = np.array(preds) # (200, n_test)
variance = P.var(axis=0).mean() # spread across refits
bias2 = ((P.mean(axis=0) - yte)**2).mean() # systematic offset
print(bias2, variance)
# high variance, low bias2 -> more data or more regularisation
# high bias2, low variance -> more capacity or better featuresWhy does the expected squared error split cleanly into exactly three terms — bias², variance, and irreducible noise — with no cross terms? That the messy sum of all errors decomposes so tidily is not an accident of algebra.
- 1Fix a test point x. The label is y = f(x) + ε, where f is the true function and ε is noise with mean 0. Your prediction ĝ(x) is a random variable, because it depends on which training sample you happened to draw.forced by · two sources of randomness exist — the label noise and the training set — and they must be tracked separately
- 2Write the error as (y − ĝ) and insert the average prediction over all possible training sets, ḡ = E[ĝ]:
y − ĝ = (f + ε − ḡ) + (ḡ − ĝ). Nothing has changed; you added and subtracted the same quantity.forced by · the mean prediction is the natural pivot separating "systematically wrong" from "unstable" - 3Square it and take expectations. The three squared terms give E[ε²] = σ², (f − ḡ)² = bias², and E[(ḡ − ĝ)²] = variance.forced by · squaring a three-part sum yields three squares plus three cross terms
- 4Every cross term vanishes. Those involving ε vanish because E[ε] = 0 and ε is independent of the training draw. The term (f − ḡ)·E[ḡ − ĝ] vanishes because ḡ is defined as E[ĝ], so that expectation is exactly zero.forced by · subtracting the mean makes the deviation orthogonal to any constant — the same orthogonality that makes least squares a projection
- 5The σ² term contains no model quantity at all. No algorithm, no amount of data, and no capacity can touch it — it is the noise in the labels themselves.forced by · ε is independent of everything you control
Therefore the decomposition is exact and the cross terms vanish by construction, because deviations from a mean are orthogonal to that mean. Error = (wrong on average) + (unstable across samples) + (unknowable).
And note what this predicts: the clean three-way split depends on the loss being squared error, since that is what made the orthogonality argument work. For 0–1 loss or cross-entropy there is no equally clean decomposition — the analogues are approximate and the terms can interact. So "bias-variance tradeoff" is a precise statement for regression and a useful metaphor for classification, and treating the metaphor as arithmetic is where people go wrong.
Each retraining on a fresh sample from the same distribution is one throw. Bias is where the cluster of darts is centred relative to the bullseye — a systematic aiming error that stays even if you throw a thousand times. Variance is how wide the cluster is spread — the throw-to-throw instability. Irreducible noise is the board itself wobbling.
The critical insight: you cannot diagnose either one from a single throw. A single model's test score tells you the total error and nothing about its composition — which is why the diagnosis requires either a learning curve or multiple refits, and why "add more data" and "add more capacity" are opposite prescriptions that a single score cannot distinguish between.
- High bias = high train error AND high validation error, close together. Adding data will not help. Add capacity, add features, weaken regularisation.
- High variance = low train error, much higher validation error. Adding capacity will hurt. Add data, add regularisation, simplify, or ensemble.
- More data reduces variance and does nothing for bias. This is the single most useful asymmetry in the whole framework.
- Ensembling attacks variance directly by averaging many high-variance models (bagging); boosting attacks bias by sequentially fitting the residuals. That is the whole difference between them.
Fire this the moment you see: someone proposing more data for a model that already underfits · someone proposing a bigger model for a train/val gap of 30 points · a learning curve nobody plotted before deciding · a demand to hit an accuracy target with no discussion of the noise floor · wildly different scores across CV folds.
Your validation error is too high and you have one sprint. Do you buy more data, add capacity, or invest in regularisation and ensembling?
Do not choose from intuition; plot the learning curve first. It costs one afternoon and it distinguishes the three cases unambiguously: converged-and-high means bias, a persistent gap means variance, and a still-falling validation curve means data will pay. Teams routinely spend a quarter collecting data for a model that was bias-limited from day one, and the curve would have said so immediately.
In practice the ordering that wins most often is: better features first (reduces bias cheaply, and is where domain knowledge actually converts into accuracy), then regularisation and ensembling (fast variance reduction), then more data (slow but the only durable fix). And know your noise floor — if two human labellers disagree 8% of the time, chasing 97% accuracy is chasing σ², and no engineering effort will ever reach it.
(c) Hands-on · 25 min
We're going to (1) plot a real learning curve, (2) empirically decompose bias and variance via bootstrapping, and (3) reproduce the classical U-curve by varying polynomial degree.
# bias_variance.py — the debugging playbook in ~130 lines.
import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split, learning_curve
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error
np.random.seed(42)
# --- 1. Synthetic non-linear data: y = sin(2·x) + noise ---
def make_data(n=300, noise=0.3):
X = np.random.uniform(-3, 3, size=(n, 1))
y = np.sin(2 * X.ravel()) + np.random.normal(0, noise, size=n)
return X, y
X, y = make_data(n=300)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
# =====================================================
# 1. LEARNING CURVE — vary training set size
# =====================================================
print("=== Learning curve — model = poly-deg-9 + ridge ===")
model = Pipeline([
("poly", PolynomialFeatures(degree=9)),
("scaler", StandardScaler()),
("ridge", Ridge(alpha=1e-3)),
])
sizes, train_scores, val_scores = learning_curve(
model, X_train, y_train,
train_sizes=np.linspace(0.1, 1.0, 8),
cv=5, scoring="neg_root_mean_squared_error", random_state=42
)
print(f" {'train_size':>10} {'train_RMSE':>12} {'val_RMSE':>10} {'gap':>8}")
for n, tr, va in zip(sizes, -train_scores.mean(1), -val_scores.mean(1)):
gap = va - tr
diag = "high var" if gap > 0.2 else ("high bias" if tr > 0.4 else "OK")
print(f" {int(n):>10} {tr:>12.4f} {va:>10.4f} {gap:>8.4f} {diag}")
# =====================================================
# 2. CLASSICAL U-CURVE — vary polynomial degree
# =====================================================
print("\n=== U-curve — vary complexity (poly degree) ===")
print(f" {'degree':>7} {'train_RMSE':>12} {'test_RMSE':>10} {'diag':>15}")
for degree in [1, 2, 3, 5, 7, 9, 12, 15, 20]:
m = Pipeline([
("poly", PolynomialFeatures(degree=degree)),
("scaler", StandardScaler()),
("ridge", Ridge(alpha=1e-6)), # tiny alpha so complexity, not reg, dominates
])
m.fit(X_train, y_train)
tr_err = mean_squared_error(y_train, m.predict(X_train), squared=False)
te_err = mean_squared_error(y_test, m.predict(X_test), squared=False)
if degree <= 2:
d = "high bias"
elif degree >= 15:
d = "high variance"
else:
d = "balanced"
print(f" {degree:>7} {tr_err:>12.4f} {te_err:>10.4f} {d:>15}")
# =====================================================
# 3. EMPIRICAL BIAS + VARIANCE via bootstrapping
# =====================================================
def empirical_bias_variance(model_ctor, X_train, y_train, X_test, y_test, n_boot=50):
"""
Fit n_boot models on bootstrap samples of train.
At each test point, compute bias² + variance across the ensemble.
"""
predictions = np.zeros((n_boot, len(X_test)))
n = len(X_train)
for b in range(n_boot):
idx = np.random.choice(n, size=n, replace=True)
m = model_ctor()
m.fit(X_train[idx], y_train[idx])
predictions[b] = m.predict(X_test)
mean_pred = predictions.mean(axis=0)
bias_sq = np.mean((mean_pred - y_test) ** 2)
variance = np.mean(predictions.var(axis=0))
total_mse = np.mean((predictions - y_test) ** 2)
noise = total_mse - bias_sq - variance # residual — approximates σ²
return bias_sq, variance, noise, total_mse
print("\n=== Empirical bias-variance across model complexity ===")
print(f" {'degree':>7} {'bias²':>10} {'variance':>10} {'noise':>10} {'total':>10}")
for degree in [1, 3, 5, 9, 15]:
def ctor(d=degree):
return Pipeline([
("poly", PolynomialFeatures(degree=d)),
("scaler", StandardScaler()),
("ridge", Ridge(alpha=1e-6)),
])
b2, v, n_, t = empirical_bias_variance(ctor, X_train, y_train, X_test, y_test, n_boot=30)
print(f" {degree:>7} {b2:>10.4f} {v:>10.4f} {n_:>10.4f} {t:>10.4f}")What each block does
Anatomy of the script
Take a degree-15 polynomial (the high-variance case). Fit 20 of them on 20 bootstrap samples. Average their predictions. Now:
- Individual model test RMSE: ~0.6
- Ensemble prediction RMSE: ~0.4
You've reduced variance by ensembling — same idea as random forest. Bias stays the same; only variance drops. This is why bagging works.
(d) Production reality · 15 min
Teams that overfit early — huge stacked ensembles with insufficient regularization — climbed the public leaderboard fast, then collapsed on the private leaderboard. The teams that eventually won carefully monitored train vs val error gap and stopped adding models when the gap widened.
Early CTR models used simple logistic regression on ~100 features. Bias was high — the model couldn't capture rare feature interactions. Adding polynomial features exploded dimensionality and made training infeasible.
Classical bias-variance theory predicts that a 175B-parameter model trained on 300B tokens should overfit catastrophically. It doesn't. Instead, val loss keeps decreasing with more parameters even in the ‘overparameterised’ regime.
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
If you can't teach these three without notes, redo the session:
- What is bias, what is variance, what is noise? (one-sentence each)
- How do you read a learning curve to diagnose which one is your problem?
- When would you pick regularization over more data, and vice versa?
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.