S087 · Regularization — L1, L2, Elastic Net

Module M11: Classical ML · Session 87 of 130 · Track: ML 🎚️

What you'll be able to do after this session

• Explain Regularization to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S086 — Logistic Regression — Sigmoid, Cross-Entropy, from Scratch

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Regularization Part 1: Ridge (L2) — StatQuest — the shrinkage intuition in one napkin.
• 🎥 Deep dive (30–60 min) · Ridge, Lasso, Elastic Net — StatQuest playlist — L1 vs L2 vs both, geometrically.
• 🎥 Hands-on demo (10–20 min) · Ridge and Lasso in Python — Krish Naik — end-to-end with sklearn.
• 📖 Canonical article · scikit-learn — Ridge, Lasso, ElasticNet — API + when to pick which.
• 📖 Book chapter / tutorial · ISLR ch.6 — Linear Model Selection & Regularization (free PDF) — the reference chapter for this topic.
• 📖 Engineering blog · Netflix Tech Blog — Interpretable Machine Learning — where sparse (L1) models pay their rent.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Fighting overfitting.

Overfitting has a signature: your training accuracy climbs, your validation accuracy stalls or drops. The model has memorised noise. Regularisation is the counter-move — you deliberately make training harder by adding a penalty on the size of the weights. The model now has to choose between (a) fitting the training data perfectly and (b) keeping its weights small. Forced to compromise, it generalises better.

Three common flavours:

• L2 (Ridge): penalty = λ · Σ wᵢ². Shrinks all weights toward zero but rarely to exactly zero. Good default, mathematically clean (still convex, still closed-form), stabilises correlated features.
• L1 (Lasso): penalty = λ · Σ |wᵢ|. Pushes weights all the way to zero, so it performs implicit feature selection. Great when you suspect most features are irrelevant.
• Elastic Net: α · L1 + (1-α) · L2. Best of both — shrinks correlated features together (L2) while also zeroing out irrelevant ones (L1).

The single knob you tune is λ (called alpha in sklearn, C = 1/λ in LogisticRegression). Too small → no regularisation, overfits. Too big → underfits, everything close to zero, predictions constant. You find the sweet spot with cross-validation: try a log-spaced grid of λ values, pick the one with the best val score.

Why does L1 zero-out coefficients but L2 doesn't? Geometry. The L1 penalty region is a diamond with corners on the axes; the loss contour often first touches the diamond at a corner (where a coordinate is zero). L2 is a smooth sphere — you can touch it anywhere. Watch StatQuest for the picture; it clicks instantly.

Gotcha to carry forever: Regularisation only works if features are on the same scale — the penalty treats all wᵢ equally. Always standardise your features before regularised models.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The two penalty shapes explain why L1 sparsifies and L2 doesn't:

Comparison table:

Worked example — the effect of λ on a 100-feature problem.

The classic U-shape: too little regularisation overfits, too much underfits, the sweet spot is somewhere in the middle. Find it with cross-validation.

Bias-variance angle (preview of S088). Regularisation increases bias (the model can no longer fit as flexibly) but decreases variance (predictions on new data are more stable). Small bias increase, big variance decrease → net win on val/test.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Compare no-reg, Ridge, Lasso, and Elastic Net on a synthetic problem where only 5 of 50 features actually matter — perfect setup to see L1 do its thing.

# pip install numpy scikit-learn
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet, RidgeCV, LassoCV
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
 
rng = np.random.default_rng(0)
 
# ---------- 1. Make a sparse, noisy problem ----------
X, y, true_w = make_regression(n_samples=400, n_features=50, n_informative=5,
                               noise=15.0, coef=True, random_state=0)
print("True nonzero coefficients:", (true_w != 0).sum(), "of", len(true_w))
X = StandardScaler().fit_transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
 
def score(name, model):
    model.fit(X_tr, y_tr)
    tr = np.sqrt(mean_squared_error(y_tr, model.predict(X_tr)))
    te = np.sqrt(mean_squared_error(y_te, model.predict(X_te)))
    nz = int(np.sum(np.abs(model.coef_) > 1e-5))
    print(f"{name:20s} train={tr:7.3f}  test={te:7.3f}  nonzero={nz:2d}/50")
    return model
 
# ---------- 2. No regularisation — will overfit ----------
score("Plain LinReg",       LinearRegression())
 
# ---------- 3. Fixed-alpha versions ----------
score("Ridge alpha=1",      Ridge(alpha=1.0))
score("Lasso alpha=0.1",    Lasso(alpha=0.1, max_iter=5000))
score("ElasticNet",         ElasticNet(alpha=0.1, l1_ratio=0.5, max_iter=5000))
 
# ---------- 4. Cross-validated versions — let sklearn pick alpha ----------
ridge_cv = score("RidgeCV",  RidgeCV(alphas=np.logspace(-3, 3, 20)))
lasso_cv = score("LassoCV",  LassoCV(alphas=np.logspace(-3, 3, 20), max_iter=5000, cv=5))
print(f"  RidgeCV picked alpha={ridge_cv.alpha_}")
print(f"  LassoCV picked alpha={lasso_cv.alpha_:.4f}")
 
# ---------- 5. Did LassoCV find the right features? ----------
true_nz = set(np.where(true_w != 0)[0])
found_nz = set(np.where(np.abs(lasso_cv.coef_) > 1e-5)[0])
print("True informative features:  ", sorted(true_nz))
print("Lasso-selected features:    ", sorted(found_nz))
print("Precision:", len(true_nz & found_nz) / max(len(found_nz), 1))
print("Recall:   ", len(true_nz & found_nz) / len(true_nz))

Observe when you run it:

1. Plain LinReg has low train RMSE and huge test RMSE — textbook overfitting.
2. Ridge tightens train slightly but slashes test error — bias-variance win.
3. Lasso zeros out ~40 of 50 features while achieving similar or better test error.
4. CV-selected alphas beat any fixed guess — this is why you always cross-validate.
5. LassoCV usually recovers 4-5 of the 5 true informative features. Automatic feature selection.

Try this modification: raise noise=50 and rerun. Regularisation matters more under high noise. Then set n_informative=45 (dense problem). Now L1 loses to Ridge — L1 is only better when the true model is sparse.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Why every production model quietly uses regularisation. Coefficients from unregularised linear models are unstable across retraining runs — flip sign, blow up, correlate wildly with random seed. Business partners see a coefficient of +₹2000/sqft today and -₹500/sqft tomorrow and lose trust immediately. Even a tiny Ridge penalty stabilises things.

Feature scaling is table stakes. Because the penalty treats all wᵢ equally, features on huge scales (salary in ₹) get shrunk far more than features on tiny scales (years_of_experience). Fit a StandardScaler on train only, apply to val/test, or you leak.

Lasso for interpretability at Netflix / Spotify. Recommender-explanation models use L1 to force the output to depend on a handful of user features so the product team can say "this recommendation is because you like X, Y". A dense 1000-feature model is uninterpretable; a Lasso model with 8 non-zero features tells a story.

Elastic Net for genomics. DNA microarray data: 20,000 genes, 100 patients. Correlated features abound (genes in the same pathway). Pure Lasso arbitrarily picks one and drops the rest; Elastic Net keeps the group together. The paper by Zou & Hastie (2005) launched a decade of biology ML.

Regularisation in deep learning. Same idea, different names: weight decay = L2, dropout = randomly zero out activations at train time (a stochastic ensemble regulariser), early stopping = halt training before it starts memorising (an implicit regulariser). All achieve the same "prefer simpler models" goal.

Common bug: standardising target y. Standardise X, not y (for regression). If you scale y, remember to un-scale the predictions. Every ML engineer has burned an afternoon on this.

Choosing λ. Log-spaced grid (np.logspace(-4, 4, 30)) + 5-fold CV + 1-SE rule (pick the largest λ within one standard error of the best — favours simpler models). This is Hastie/Tibshirani gospel.

Business impact. At scale, a 1% RMSE reduction from proper regularisation on a pricing model is worth millions. Regularisation is arguably the highest-ROI ML technique after "clean your data".

(e) Quiz + exercise · 10 min

1. What is overfitting, and how does regularisation fight it?
2. Contrast L1 and L2 penalties: what does each do to the coefficient values?
3. Why must features be standardised before applying L1/L2 regularisation?
4. What's the practical difference between Lasso and Elastic Net when features are correlated?
5. Name three deep-learning techniques that are secretly forms of regularisation.

Stretch (connects to S084 — ML mental model): Design a cross-validation procedure to pick λ that avoids both (a) data leakage from val into train and (b) optimistic bias from picking the best λ on val. Where does the test set fit in?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Regularization? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S088): Bias–Variance Trade-off & Learning Curves
• Previous (S086): Logistic Regression — Sigmoid, Cross-Entropy, from Scratch
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.