Bias–Variance Trade-off & Learning Curves

Why this session exists

When validation error is disappointing, collecting more data, adding features and reducing model complexity are different interventions with different costs. Bias and variance provide a language for the problem, but a single learning curve is not an infallible diagnosis. Optimization failure, distribution shift, leakage and measurement noise can resemble a capacity problem.

We will derive the squared-error decomposition, distinguish three kinds of curves, measure retraining variability on a controlled problem, and choose interventions as testable hypotheses. The important skill is not naming a curve in 30 seconds; it is identifying what evidence would distinguish competing explanations before spending another sprint.

Prerequisites

Review train/validation/test roles, linear regression and regularization. Know expectation and variance. Keep MSE separate from RMSE: the additive decomposition is about squared error, and taking a square root does not preserve the sum.

(a) Intuition · 5 min

Picture an archer repeatedly shooting at one fixed target. The center of the arrow cluster may be displaced from the target—systematic bias—and individual arrows may spread around that center—variance. In our setting, one arrow is a prediction after training on a fresh random dataset, not a prediction for a different test example. Mixing these two sources of variation is a common conceptual error.

The target itself can have noise: two otherwise identical cases can have different outcomes. That is a property of the conditional outcome distribution given the available inputs. Better measurements or additional relevant features may reduce that uncertainty; no model using the same inputs and information can predict independent noise.

The three pieces of every prediction error

Fix an input $x$. Let the future label be $Y=f(x)+\epsilon$, where $f(x)=E[Y\mid x]$, $E[\epsilon\mid x]=0$ and $\operatorname{Var}(\epsilon\mid x)=\sigma^2(x)$. A trained prediction $\hat f_D(x)$ varies with the random training dataset $D$ and, if relevant, training randomness. Assume future noise is independent of the training procedure conditional on $x$, and the required second moments exist.

Write $\bar f(x)=E_D[\hat f_D(x)]$. Add and subtract this mean:

Squaring produces three squared terms and three cross terms. The cross terms involving $\epsilon$ vanish by its zero conditional mean and independence. The remaining cross term vanishes because $E_D[\bar f-\hat f_D]=0$. Thus

Average this identity over the deployment input distribution to get integrated error. Homoscedastic noise replaces $\sigma^2(x)$ by a constant; that is an extra assumption, not part of the definition of bias or variance.

Bias is not only a model's inability to represent the truth. Regularization, finite-sample estimation and optimization choices can affect the average fitted prediction. Variance concerns sensitivity across training draws, not simply “having many parameters.” This particular arithmetic is for squared loss; classification losses have other decompositions, and the displayed formula must not be copied unchanged to accuracy or cross-entropy.

Worked calculation. At a fixed $x$, let $f(x)=2$, mean prediction 1.5, prediction variance 0.16 and independent noise variance 0.09. Expected squared error is $0.5^2+0.16+0.09=0.50$. Corresponding root expected squared error is $\sqrt{0.50}\approx0.707$, not $0.5+0.4+0.3=1.2$.

The intellectual history

The official scikit-learn bias–variance example compares predictions over independently drawn learning samples and separates the known Bayes function from observation noise. It is a concrete reference for what the expectations mean.

Nakkiran and colleagues' 2019 Deep Double Descent paper reports a later empirical complication: model size and training epochs can produce nonmonotonic test error, including a second descent. This adds conditions to the classical complexity story; it does not invalidate the squared-error identity. The paper is not evidence that parameter count alone explains the generalization of every large language model.

The classical U-curve

In a familiar underparameterized example, a rigid model cannot represent a curved signal, while a flexible estimator reacts strongly to noise. Increasing complexity can lower squared bias and increase variance, yielding a U-shaped expected-error curve. This is a possible behavior of a model/data/algorithm family—not a theorem that either component must vary monotonically with parameter count.

Even training error need not be monotonically decreasing across a collection of practical fits: model families may not be nested, regularization can change, and optimization can stop at different tolerances. For nested unregularized hypothesis classes solved globally under the same loss, the best achievable training loss cannot rise; that narrower statement follows from set inclusion.

Reading a learning curve — the debugging playbook

Name the horizontal axis before diagnosing:

• Learning curve: vary training-set size, keeping the model specification and evaluation scheme fixed. Plot mean train and validation loss, variation across folds, and the number of examples actually fitted.
• Validation/complexity curve: vary a parameter such as polynomial degree or ridge strength with data size fixed.
• Optimization curve: vary epochs or boosting stages on the same training data. A validation minimum may guide early stopping.

High train and validation error close together is consistent with underfitting, but also an optimization bug, strong regularization, weak inputs or high outcome noise. First compare with a constant baseline and a model that should fit a small subset. A tiny train error with much worse validation error suggests overfitting, but can also expose a time shift, an inappropriate group split or mismatched preprocessing. Low, close errors deserve checks for leakage and subgroup failures before celebration.

If validation error keeps falling as more representative independent data is added, collecting more is a plausible next experiment. It is not a guarantee that the next data source will match the distribution. An epoch curve improving on both train and validation can suggest training longer; a size curve cannot establish that the optimizer needs more epochs.

The action matrix — what to change based on diagnosis

More data can affect bias as well as variance for a finite-sample learning algorithm. Better features can reduce both by simplifying the relationship; regularization can sometimes alter both in ways the simple story does not predict. These are hypotheses to measure. Annotator disagreement is not itself a hard accuracy ceiling: it depends on the labeling protocol, error correlations and the definition of truth.

The double descent surprise

Interpolation means fitting the training labels exactly, or nearly so under a stated loss. Near an interpolation threshold, an estimator can become sensitive to noise; beyond it, a training rule may select a different, lower-error interpolating solution. Test error can therefore rise and then fall again as capacity increases. The location is not universally “number of parameters equals number of rows”; architecture, regularization, optimization and noise matter.

Nakkiran et al. report model-wise and epoch-wise double descent and even regimes where additional samples hurt test performance. Those observations refute simplistic monotonic claims, not the need for validation. The small polynomial experiment below is not a reproduction of their deep-learning study, and a U-shaped curve or second descent is not asserted as a passing test.

(c) Hands-on · 25 min

This bounded lab uses $f(x)=\sin(2x)$ with Gaussian noise standard deviation 0.3. Because the noiseless function is known, squared bias can be estimated relative to $f$ rather than a single noisy target. It prints curve data and compares independent retraining with conditional bootstrap variation.

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, learning_curve
from sklearn.metrics import root_mean_squared_error
 
rng = np.random.default_rng(42)
SIGMA = .3
 
def make_data(n=120):
    X = rng.uniform(-3, 3, (n, 1))
    return X, np.sin(2*X[:, 0]) + rng.normal(0, SIGMA, n)
 
 
def model(degree=5, alpha=.01):
    return Pipeline([('poly', PolynomialFeatures(degree, include_bias=False)),
                     ('scale', StandardScaler()), ('ridge', Ridge(alpha=alpha))])
 
 
X, y = make_data(240)
cv = KFold(n_splits=4, shuffle=True, random_state=7)
sizes, train, val = learning_curve(model(9), X, y, cv=cv,
    train_sizes=[.25, .5, .75, 1.], scoring='neg_root_mean_squared_error',
    shuffle=True, random_state=8, n_jobs=1, error_score='raise')
for n, tr, va in zip(sizes, -train, -val):
    print('size', n, 'train RMSE', tr.mean(), 'validation RMSE', va.mean(),
          'fold SD', va.std())
 
# Separate synthetic validation sample; these are complexity diagnostics, not final-test claims.
Xval, yval = make_data(200)
for d in [1, 3, 5, 9, 12]:
    m = model(d).fit(X, y)
    print('degree', d, 'train', root_mean_squared_error(y, m.predict(X)),
          'validation', root_mean_squared_error(yval, m.predict(Xval)))
 
# Fixed evaluation inputs and known conditional mean.
grid = np.linspace(-3, 3, 151).reshape(-1, 1)
f = np.sin(2*grid[:, 0])
 
def summarize(P, truth):
    mean = P.mean(axis=0)
    bias2 = np.mean((mean-truth)**2)
    variance = np.mean(P.var(axis=0, ddof=0))
    noiseless = np.mean((P-truth)**2)
    assert np.allclose(noiseless, bias2+variance, atol=1e-10)
    # Known noise variance is added, NOT estimated as a residual of this identity.
    return bias2, variance, noiseless + SIGMA**2
 
 
N_REFIT = 120
Xboot, yboot = make_data(N_REFIT)  # one fixed observed sample for the bootstrap
print('refit rows: independent', N_REFIT, 'bootstrap', len(Xboot), 'fits per degree', 32)
for d in [1, 5, 9]:
    independent, bootstrap = [], []
    for _ in range(32):
        Xi, yi = make_data(N_REFIT)
        independent.append(model(d).fit(Xi, yi).predict(grid))
        idx = rng.integers(0, len(Xboot), len(Xboot))
        bootstrap.append(model(d).fit(Xboot[idx], yboot[idx]).predict(grid))
    P = np.asarray(independent)
    B = np.asarray(bootstrap)
    print('independent degree', d, 'estimated bias2/variance/noisy MSE', summarize(P, f))
    print('conditional bootstrap degree', d,
          'mean-offset2/resampling variance/noisy MSE', summarize(B, f))
 
    # The former 'noise residual' is algebraically zero for ANY fixed target vector.
    fixed_noisy_target = f + rng.normal(0, SIGMA, len(f))
    old_bias = np.mean((P.mean(axis=0)-fixed_noisy_target)**2)
    old_total = np.mean((P-fixed_noisy_target)**2)
    assert abs(old_total-old_bias-np.mean(P.var(axis=0))) < 1e-10
 
    # Averaging can improve squared error relative to the average member's error.
    ensemble_error = np.mean((B.mean(axis=0)-f)**2)
    average_member_error = np.mean((B-f)**2)
    assert ensemble_error <= average_member_error + 1e-10
    print('bootstrap mean / average member noiseless MSE', ensemble_error, average_member_error)

What each block does

learning_curve refits the entire polynomial/scaling/ridge pipeline for each training subset and fold. shuffle=True makes the supplied random state meaningful for subset selection. Negative scorer values are negated for readable RMSE. Fold standard deviation describes variation among these folds; folds share observations, so it is not automatically a confidence interval from independent replications.

The degree sweep prints actual errors without labeling a degree as “high variance” before measuring it. Polynomial columns of large degree can be ill-conditioned even after scaling; degree, input range, ridge strength and solver all affect the result. A numerically unstable model is not a clean demonstration of a statistical theorem.

For the decomposition, independent fits use fresh training draws. Bootstrap fits resample one observed dataset; they estimate a conditional resampling distribution, not the exact population distribution of independently trained models. Both now fit 120 rows per refit, and the program prints the sample sizes and number of refits. Equal sizes remove a separate source of difference, but do not turn conditional resampling into independent population sampling. The bootstrap mean-offset is measured against known $f$ for this one observed sample; it is not population squared bias. For independent fits, the finite ensemble mean itself has Monte Carlo error, so its squared discrepancy from $f$ is only an estimate of population squared bias.

The old computation used one fixed noisy y_test, defined “bias squared” as the squared error of the mean prediction to that vector, and subtracted variance from total squared error to estimate noise. That residual is identically zero (up to floating-point error):

It cannot recover $\sigma^2$. The corrected lab uses known $f$ and adds the known $0.3^2=0.09$. For unknown real data, population bias/noise estimation needs additional assumptions, repeated measurements or a defensible reference; a test-label vector does not supply them.

To plot the learning curve, put sample counts on x and mean RMSE on y with bands for the printed fold variation. Plot degree versus validation RMSE separately. For the retraining plot, overlay several prediction functions on the fixed grid, their mean, and $\sin(2x)$. These pictures answer different questions; do not combine their axes into one “learning curve.”

(d) Production reality · 15 min

Choosing one intervention with limited time

Suppose train RMSE is 0.05 and validation RMSE 0.90 on the same target scale. First confirm that the validation population, preprocessing and label definition match training. If they do, try a controlled capacity reduction or regularization change on the same split, then add independent training data if available. A gap is a reason to investigate variance, not proof of it.

If train RMSE is 0.85 and validation 0.90, compare both with a simple baseline and inspect convergence. A model unable to fit even a small clean subset may have a bug; a converged model with suitable inputs may need more capacity or a weaker penalty. More epochs will not fix a representational limitation, while more expressive features may reduce the amount of capacity needed.

Data collection has labeling and latency costs, and data from a different source can introduce shift. Ensembling adds training/serving memory and can obscure explanations. Capacity changes can increase inference cost. Better labels and useful representations may be high-leverage, but their expected benefit should be tested rather than asserted from the diagram.

Sparse features and online regularization

High-dimensional categorical crosses can make a linear model more expressive without making every row dense. For instance, a device-by-hour indicator exposes an interaction that two separate additive indicators cannot represent. Hashing can bound the feature space at the cost of collisions; L1 can limit the number of stored nonzero coefficients. This changes representation, optimization and regularization together, not merely one isolated bias term.

McMahan et al.'s 2013 CTR paper describes FTRL-Proximal online logistic learning with sparse examples and per-coordinate learning rates. Each observation contributes gradient $(p-y)x_i$ on active coordinates. Its lazy coefficient representation permits a thresholded L1 update, unlike simply adding a sign subgradient and expecting exact zeros. A model can therefore combine many possible crossed features with sparse serving weights. That does not establish that more data makes variance irrelevant: rare categories, changing populations, collisions and noisy outcomes remain limitations. The paper also treats calibration and engineering constraints rather than attributing success to parameter count alone. No web-scale FTRL experiment is run in this lesson.

Averaging and the correlated-error floor

For $B$ predictors with equal variance $v$ and equal pairwise correlation $\rho$,

This follows by retaining covariance terms in the variance of a sum; it does not need the central limit theorem. If $v=0.25$, $\rho=0.6$, and $B=20$, average variance is $0.25(0.6+0.4/20)=0.155$, not $0.25/20$. Increasing $B$ cannot remove the shared component in this model. Equal-mean averaging preserves that mean's bias, but training base learners on bootstrap samples changes their distribution relative to a single model trained on the full dataset; bagging can change bias too.

The lab's Jensen inequality compares the squared error of one ensemble mean with the average squared error of its constituent predictions at the same evaluation points. It does not guarantee beating the best member or reducing error by half. Boosting instead changes a running prediction sequentially; it is not a procedure that modifies only bias while leaving variance untouched.

Where this shows up in the rest of the plan

Decision trees show greedy, unstable partitions. Random forests deliberately diversify them before averaging. Boosting adds corrections with validation-controlled complexity. Feature engineering changes the representation, which can alter both approximation difficulty and estimation variability. Each connection is a mechanism, not a universal prescription to use one algorithm.

(e) Recall + stretch · 10 min

1. State the decomposition and assumptions. Answer: conditional expected squared prediction error equals squared bias plus training-draw variance plus conditional noise variance; require zero-mean future noise independent of training and finite second moments.
2. One-line intuition? Answer: bias is the offset of the average trained prediction from $f(x)$; variance is its spread over training draws. Neither is directly the error across different test rows.
3. Train 0.05, validation 0.90: three actions? Answer: verify split/preprocessing/shift, test regularization or reduced capacity, then test additional representative data or averaging. Overfitting is a hypothesis, not the only explanation.
4. Train 0.85, validation 0.90: three actions? Answer: inspect baseline and optimizer, reduce excessive penalty, add informative features or capacity. More training time helps only an optimization-limited fit.
5. What is the noise floor? Answer: $\operatorname{Var}(Y\mid X)$ for the chosen information set. Adding relevant information can lower it; an error plateau or annotator disagreement alone does not measure it.
6. How do we estimate bias/variance here? Answer: train repeatedly, predict at the same inputs, compare the average to known $f$ and measure prediction variance. Bootstrap estimates are conditional and cannot conjure a known truth on real data.
7. What is double descent? Answer: a possible decrease–increase–decrease in test error along a specified complexity or epoch axis, often related to interpolation. Not “all models improve once parameters exceed rows.”
8. Does bagging only reduce variance? Answer: averaging equal-mean predictors preserves their mean but reduces the independent component of variance; the bootstrap base-fitting procedure can change bias relative to a full-data fit.
9. Does boosting only reduce bias? Answer: it fits sequential loss corrections. Both bias and variance can change, and validation is required to assess total error.
10. Stretch: 'always use XGBoost.' Answer: compare a regularized linear model, a controlled tree ensemble and a meaningful baseline using the same splits. If the signal is nearly linear or data are scarce, extra flexibility may add little benefit; no sample-size cutoff guarantees which family wins. Include serving cost and calibration in the decision.

Explain-out-loud test

Draw the same test input receiving predictions from many independently trained models. Mark the known $f(x)$, the prediction mean, the spread and future-label noise. Then explain why a learning curve generates hypotheses but cannot numerically separate all three components from one fit.

What comes next

Build a Decision Tree, where local split choices create a concrete source of instability that we can later average in a forest.

Sources and execution boundary

Checked 2026-10-07: scikit-learn 1.7 bias–variance example, learning/validation curves, cross-validation and Nakkiran et al., 2019. The local CPU lab checks algebraic identities and finite synthetic fits. It does not estimate the noise floor of a real application, reproduce deep double descent, or establish benchmark performance.