S087 · Regularization — L1, L2, Elastic Net
Add a penalty to the loss and watch overfitting die — ridge shrinks, lasso selects, elastic net compromises. The single most useful technique in classical ML.
🎯 Add L1, L2, and Elastic Net penalties to your loss — understand why one shrinks, the other selects, and how to tune λ properly with cross-validation.
Why this session exists
Almost every real dataset has more features than you have signal for. Fit a plain linear or logistic regression on 200 features and 500 rows and you'll get glorious training accuracy that collapses on new data. Regularization is the one trick that lets you throw features at your model without paying the overfitting tax — and choosing between L1, L2, and Elastic Net is a design decision that recurs in every model family (deep learning included).
- Write the loss functions for ridge, lasso, and elastic net from memory.
- Explain geometrically why L1 produces sparse solutions and L2 shrinks smoothly.
- Tune the regularization strength λ using k-fold cross-validation.
- Choose between L1 and L2 based on whether you want feature selection or shrinkage.
- Recognise regularization in disguise — weight decay, dropout, early stopping.
Prerequisites
- S085 · Linear regression from scratch — you can implement OLS + GD.
- S086 · Logistic regression — same gradient shape, extended loss.
- S040 · Linear algebra — vector norms.
(a) Intuition · 5 min
You're moving to a new apartment. Without a rule, you'll stuff every closet and end up with junk you never use. So you set a rule: total weight of stuff you bring must be under X kilos. Now you're forced to decide what's really worth carrying.
Two ways to enforce the rule: (1) charge $1 per kilo (soft budget — you can bring more, but it costs) — that's L1. (2) charge $1 per kilo squared (harder as you get bigger) — that's L2. Both stop you from bringing 500 kilos of stuff; they penalise differently.
Regularization adds a penalty on the size of θ to the loss: L(θ) = data_loss(θ) + λ · penalty(θ). The optimiser now has to balance ‘fit the data’ against ‘keep θ small.’ Small θ = simpler model = less overfitting.
L2 penalty = λ·||θ||² = shrinks every coefficient smoothly toward zero. L1 penalty = λ·||θ||₁ = pushes small coefficients to exactly zero (feature selection). Elastic Net = combine both, tuned by a mixing ratio.
The three penalties in one comparison
- Ridge (L2): L = data_loss + λ·Σθⱼ². Shrinks all coefficients smoothly. Never zeros them out. Handles correlated features gracefully.
- Lasso (L1): L = data_loss + λ·Σ|θⱼ|. Zeros out small coefficients → feature selection built in. Struggles with highly correlated features (picks one arbitrarily).
- Elastic Net: L = data_loss + λ·(α·Σ|θⱼ| + (1-α)·Σθⱼ²). Combines both. α controls the mix. Default sklearn α = 0.5.
A brief history
- 1943Tikhonov regularizationAndrey Tikhonov invents it in the USSR to solve ill-posed inverse problems. Ridge is renamed western import.
- 1970Ridge regression · Hoerl & KennardFormal introduction in statistics literature. Still a fringe technique for decades.
- 1996Lasso · Robert Tibshirani‘Regression Shrinkage and Selection via the Lasso’ — one of the most-cited stats papers ever. Solved the ‘too many features’ problem.
- 2005Elastic Net · Zou & HastieCombines L1 + L2. Handles correlated features better than pure lasso.
- 2012Dropout · Hinton et alDeep learning gets its own regularizer. Randomly zero out neurons during training — an implicit ensemble.
- 2019Double descent · OpenAI/BelkinThe classical bias-variance curve gets a second dip in the overparameterised regime. Regularization reshapes it.
(b) Visual walkthrough · 15 min
The geometry — why L1 gives zeros and L2 doesn't
The loss minimum lies where the elliptical loss contour first touches the constraint region. For a diamond (L1), the touching point is almost always at a corner where one coordinate is zero — that's the sparsity effect. For a circle (L2), the touching point is anywhere on the boundary — no reason to prefer zero over any other value.
Ridge vs Lasso vs Elastic Net — full comparison
Shrink everything · closed-form solvable
- Loss: MSE + λ·Σθⱼ². Convex + smooth.
- Closed-form: θ = (XᵀX + λI)⁻¹ Xᵀy — always invertible.
- All coefficients get smaller, none zero.
- Handles correlated features well (spreads weight).
- Default in most sklearn classifiers.
Feature selection built in
- Loss: MSE + λ·Σ|θⱼ|. Convex but non-smooth at 0.
- No closed form — solve with coordinate descent or subgradient.
- Small coefficients pushed to EXACTLY zero.
- With correlated features, picks one arbitrarily.
- Great for high-dimensional data (d ≫ n).
Best of both
- Loss: MSE + λ(α·L1 + (1-α)·L2).
- Sparsity from L1 + stability from L2.
- Handles correlated features better than pure lasso.
- Two hyperparameters (λ, α) — more tuning.
- Default in DeepMind's tabnet, glmnet.
The bias-variance trade-off, one dial
No regularization. High variance, low bias. Overfits — perfect train, terrible test.
Gentle shrinkage. Slight bias, big variance reduction. Usually helpful.
The sweet spot on the bias-variance trade-off. Found via cross-validation.
All coefficients → 0. High bias, low variance. Model predicts the mean.
The regularization path — every coefficient as λ shrinks
"Regularization shrinks coefficients toward zero, which makes the model simpler and therefore less accurate on the training data but more accurate on new data. So it's a fixed tax I pay for generalisation."
Regularization is not a tax with a fixed benefit — it is a deliberate bias-for-variance trade whose value depends entirely on where you currently sit. On an already-underfit model, adding L2 makes both training and test error worse. The penalty only pays when the variance it removes exceeds the bias it introduces, which is why λ is a hyperparameter you must tune per dataset rather than a good practice you always apply.
The myth is sticky because in practice you almost always meet regularization in the regime where it helps — the model was overfitting, you turned on L2, test error dropped, story confirmed. You never see the counterfactual where the model was already too rigid. It is also reinforced by defaults: sklearn's LogisticRegression applies L2 with C=1.0 unless told otherwise, so many people have never observed an unregularised fit at all and assume the penalty is part of the model rather than an addition to it.
Sweep λ and watch both curves. The point is that train error rises monotonically while test error is U-shaped — and that the minimum is not at zero and not at infinity:
from sklearn.linear_model import Ridge
import numpy as np
for lam in [0, 1e-3, 1e-1, 1, 10, 100, 1e4]:
m = Ridge(alpha=lam).fit(Xtr, ytr)
tr = ((m.predict(Xtr) - ytr)**2).mean()
te = ((m.predict(Xte) - yte)**2).mean()
print(lam, round(tr, 4), round(te, 4), round(np.abs(m.coef_).sum(), 2))
# train MSE only ever increases with lam
# test MSE falls then rises -- the U is the whole story
# coefficient magnitude collapses toward 0 as lam -> infWhy does L1 produce exactly-zero coefficients while L2 only shrinks them toward zero but never reaches it? Both penalise size. The difference in outcome is qualitative, not just a matter of degree — and it is forced by geometry.
- 1Both penalties can be written as constrained optimisation: minimise the squared error subject to the coefficient vector lying inside a budget region. L2 constrains ‖β‖₂ ≤ t (a ball); L1 constrains ‖β‖₁ ≤ t (a diamond/cross-polytope).forced by · every penalised objective has an equivalent constrained form by Lagrangian duality
- 2The squared-error surface forms concentric ellipses centred on the unconstrained least-squares solution. The regularised answer is the first point where an expanding ellipse touches the constraint region.forced by · you want the lowest-error point that is still inside the budget, which is the tangency point
- 3The L2 ball is perfectly smooth — no corners, no flat faces. An ellipse touching a sphere generically touches at a point with all coordinates non-zero, and the tangency location moves continuously as you shrink the budget.forced by · a smooth boundary has a unique tangent plane everywhere, so nothing distinguishes the axes
- 4The L1 diamond has vertices lying exactly on the axes, and at a vertex a coordinate is precisely zero. Vertices are corners, so a whole cone of ellipse orientations touches there first — the vertices capture tangency with disproportionate probability.forced by · non-differentiable corners attract the optimum; a range of gradients is compatible with a single corner
- 5Equivalently in gradient terms: the L2 gradient is 2β, which shrinks proportionally and therefore vanishes as β approaches zero — an asymptote it never crosses. The L1 subgradient is a constant ±λ regardless of magnitude, so it keeps pushing with undiminished force right up to zero and then holds the coefficient pinned there.forced by · a penalty gradient that does not vanish at the origin can drive a coefficient exactly to it and keep it there
Therefore sparsity is a consequence of the L1 ball's corners — equivalently, of its constant-magnitude gradient. L2's smoothness makes exact zeros a measure-zero accident; L1's corners make them the common case.
And note what this predicts: with a group of highly correlated features, L1 must behave unstably — the corner it lands on is nearly arbitrary among the group, so tiny data perturbations flip which feature survives. L2, having no corners, spreads weight smoothly across all of them instead. That is exactly the observed behaviour, and exactly why Elastic Net exists: the L2 term restores the grouping effect while the L1 term keeps the sparsity.
The loss function pulls coefficients outward toward whatever fits the training data best, including its noise. The penalty is a leash pulling them back toward the origin. The fitted model sits where the two forces balance, and λ is the leash's tension.
Short leash (large λ) ⇒ coefficients pinned near zero, model too rigid, high bias. Long leash (λ → 0) ⇒ coefficients chase noise, high variance. The shape of the leash determines the character of the answer: L2 pulls proportionally to size and so mostly disciplines the big coefficients, while L1 pulls with constant force and so eliminates the small ones outright.
- You must standardise features first. The penalty is applied to coefficient magnitude, and magnitude depends on the feature's units — an unscaled feature is penalised by an accident of measurement.
- Never penalise the intercept. It is not a slope; shrinking it biases the model's overall level toward zero for no reason.
- L1 selects (zeros out), L2 shares (spreads across correlated features), Elastic Net does both. Choose by whether you need a shorter feature list or a stabler one.
- Early stopping, dropout, data augmentation, and adding data are all regularisers too. Any constraint that limits how thoroughly the model can fit the training noise belongs to this family.
Fire this the moment you see: a large gap between train and validation scores · coefficients with enormous magnitudes or wild sign flips across CV folds · more features than rows · perfectly separable classification data · a penalty applied without standardising · λ chosen once and never re-tuned after the feature set changed.
You have several hundred features, many correlated, and you are overfitting. Ridge, Lasso, or Elastic Net?
Start with Ridge as the default, because unless you have a real reason to believe the signal is sparse, dense-and-shrunk is the safer prior and it is the most numerically well-behaved. Reach for L1 when a shorter feature list has concrete operational value — every feature you delete is a pipeline you no longer maintain and a service that can no longer take you down.
The failure mode worth naming: treating Lasso's selected features as a scientific claim about which variables matter. With correlated inputs that list is an artefact of the sample, and it will change if you refit next month. If you need stable selection, use Elastic Net or check selection stability across bootstrap resamples — and either way tune λ by cross-validation on your actual data, since the optimum has no default and moves whenever the feature set does.
(c) Hands-on · 25 min
We're going to fit OLS, ridge, lasso, and elastic net on a dataset with lots of noise features, then cross-validate the regularization strength.
# regularization.py — L1/L2/EN comparison on noisy data, ~130 lines.
import numpy as np
import pandas as pd
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet, RidgeCV, LassoCV, ElasticNetCV
from sklearn.model_selection import train_test_split, KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error
np.random.seed(42)
# --- 1. Data: 200 rows, 50 features, but only 5 are actually informative ---
# The other 45 are pure noise — a great stress test for lasso.
X, y, true_coef = make_regression(
n_samples=200, n_features=50, n_informative=5,
noise=15.0, coef=True, random_state=42
)
print(f"True nonzero coefficients: {np.sum(true_coef != 0)} of {len(true_coef)}")
print(f"True nonzero features: {np.where(true_coef != 0)[0].tolist()}")
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
# --- 2. Baseline: plain OLS ---
pipe_ols = Pipeline([("scaler", StandardScaler()), ("model", LinearRegression())])
pipe_ols.fit(X_train, y_train)
rmse_ols = mean_squared_error(y_test, pipe_ols.predict(X_test), squared=False)
nonzero_ols = np.sum(np.abs(pipe_ols.named_steps["model"].coef_) > 1e-4)
print(f"\n[OLS · λ=0] test RMSE = {rmse_ols:6.2f} nonzero coefs = {nonzero_ols}/50")
# --- 3. Ridge (L2) — cross-validated over λ ---
pipe_ridge = Pipeline([
("scaler", StandardScaler()),
("model", RidgeCV(alphas=np.logspace(-3, 3, 50), cv=5)),
])
pipe_ridge.fit(X_train, y_train)
best_alpha_ridge = pipe_ridge.named_steps["model"].alpha_
rmse_ridge = mean_squared_error(y_test, pipe_ridge.predict(X_test), squared=False)
nonzero_ridge = np.sum(np.abs(pipe_ridge.named_steps["model"].coef_) > 1e-4)
print(f"[Ridge · CV] test RMSE = {rmse_ridge:6.2f} nonzero = {nonzero_ridge}/50 λ = {best_alpha_ridge:.4f}")
# --- 4. Lasso (L1) — cross-validated over λ ---
pipe_lasso = Pipeline([
("scaler", StandardScaler()),
("model", LassoCV(alphas=np.logspace(-3, 2, 50), cv=5, max_iter=10000)),
])
pipe_lasso.fit(X_train, y_train)
best_alpha_lasso = pipe_lasso.named_steps["model"].alpha_
rmse_lasso = mean_squared_error(y_test, pipe_lasso.predict(X_test), squared=False)
lasso_coefs = pipe_lasso.named_steps["model"].coef_
nonzero_lasso = np.sum(np.abs(lasso_coefs) > 1e-4)
print(f"[Lasso · CV] test RMSE = {rmse_lasso:6.2f} nonzero = {nonzero_lasso}/50 λ = {best_alpha_lasso:.4f}")
# --- 5. Elastic Net — CV over both α (mix) and λ ---
pipe_en = Pipeline([
("scaler", StandardScaler()),
("model", ElasticNetCV(l1_ratio=[.1, .3, .5, .7, .9], alphas=np.logspace(-3, 2, 50), cv=5, max_iter=10000)),
])
pipe_en.fit(X_train, y_train)
best_l1 = pipe_en.named_steps["model"].l1_ratio_
best_alpha_en = pipe_en.named_steps["model"].alpha_
rmse_en = mean_squared_error(y_test, pipe_en.predict(X_test), squared=False)
nonzero_en = np.sum(np.abs(pipe_en.named_steps["model"].coef_) > 1e-4)
print(f"[ElasticNet] test RMSE = {rmse_en:6.2f} nonzero = {nonzero_en}/50 λ = {best_alpha_en:.4f}, α = {best_l1}")
# --- 6. Compare recovered nonzero features to truth ---
true_nonzero = set(np.where(true_coef != 0)[0].tolist())
lasso_nonzero = set(np.where(np.abs(lasso_coefs) > 1e-4)[0].tolist())
print(f"\nTrue nonzero features: {sorted(true_nonzero)}")
print(f"Lasso-recovered : {sorted(lasso_nonzero)}")
print(f"Correctly recovered : {sorted(true_nonzero & lasso_nonzero)}")
print(f"False positives : {sorted(lasso_nonzero - true_nonzero)}")
print(f"False negatives : {sorted(true_nonzero - lasso_nonzero)}")
# --- 7. Regularization path for lasso — watch coefs shrink to zero ---
print("\n=== Lasso regularization path ===")
for alpha in [100, 30, 10, 3, 1, 0.3, 0.1, 0.01]:
m = Lasso(alpha=alpha, max_iter=10000)
m.fit(StandardScaler().fit_transform(X_train), y_train)
active = np.sum(np.abs(m.coef_) > 1e-4)
print(f" λ = {alpha:>6}: {active:2d} active features")What each block does
Anatomy of the script
Add three copies of an existing informative column (X[:, 0] + tiny noise) to your feature matrix. Refit lasso and elastic net. You'll see:
- Lasso: keeps one of the four correlated columns, zeros the other three — arbitrarily.
- Ridge: spreads weight across all four correlated columns.
- Elastic Net: spreads across all four but with smaller coefs on the copies.
For prediction accuracy, elastic net beats lasso here. For interpretability, lasso is misleading.
(d) Production reality · 15 min
Early teams naively fit high-dimensional matrix factorisations without regularization. Training RMSE looked amazing; test (private LB) RMSE was catastrophic. The models had memorised the training ratings.
L = ||R - PQᵀ||² + λ(||P||² + ||Q||²) became standard. It's still the default in every collaborative filtering library today.Common winning pattern: gradient-boosted trees + a linear model with heavy regularization, then blend them. Teams that skipped regularization on the linear model always did worse — the blend requires each ingredient to generalise.
Beginners routinely reported ‘but my lasso model has RMSE 100!’ Root cause: forgot to scale features. The regularization was penalising the wrong things.
Transformer training has always used weight decay (L2 regularization on the weights). Without it, the models overfit small subsets of the training data. With classic Adam + weight decay, the effective regularization was subtly wrong (AdamW paper, 2019).
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
If you can't teach these three without notes, redo the session:
- What does regularization do in one sentence? (limits θ magnitude to prevent overfitting)
- Difference between ridge and lasso in behaviour? (shrink vs select)
- What breaks if you skip scaling before regularization? (penalty applies unequally across features)
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.