Regularization — L1, L2, Elastic Net

Why this session exists

A flexible model can fit useful signal and accidental detail in the same training sample. Regularization changes which fit we prefer: we accept some training error in exchange for a constraint on the coefficient vector. This can stabilize predictions when predictors are correlated or numerous, but it does not make unlimited feature generation free and cannot repair leakage or an unsuitable model family.

Ridge, lasso and elastic net answer different operational questions. Do we want to retain many weak signals, select a smaller feature set, or share weight among correlated groups while retaining some sparsity? We will derive their objectives, see why exact zeros arise, tune penalties without contaminating validation folds, and distinguish explicit penalties from related deep-learning techniques.

Prerequisites

Know linear least squares, logistic loss, vector norms and train/validation/test separation. In the derivations below, columns of $X$ and $y$ are centered so an intercept is handled separately. Our teaching convention uses mean half squared error. Library parameter names must be translated to that convention, not copied blindly.

(a) Intuition · 5 min

Imagine a budget on the coefficients. A model can spend that budget to improve fit, but must justify large slopes. This is not literally a budget on the number of input columns: ridge may retain every column, and lasso only saves serving cost if zero-weight features and their upstream computations are actually removed.

The three penalties in one comparison

With an unpenalized intercept $b$ and $r=y-Xw-b\mathbf1$, define

Ridge is smooth and shrinks toward zero without generally selecting exact zeros. Lasso's nonsmooth L1 term can produce exact zeros. Elastic net combines L1 sparsity with L2 stabilization; $\rho=1$ is lasso and $\rho=0$ gives the ridge objective in this convention. In ElasticNet, alpha is the overall strength and l1_ratio is $\rho$; alpha is not the mixing parameter.

A brief history

The linear-model guide places ridge, lasso and elastic net in a common family and cites their statistical literature. Their conceptual progression matters here: penalizing squared magnitude stabilizes least squares; introducing absolute magnitude adds sparse selection; combining the two permits sparsity with a grouping tendency under correlation. This progression is not a claim that the later penalty supersedes the earlier one.

The later Decoupled Weight Decay Regularization paper, published at ICLR 2019, explicitly distinguishes weight decay from L2 regularization for adaptive optimizers. That distinction is taught below instead of repeating the unsupported claim that AdamW “makes weight decay a true L2 penalty.”

The geometry — why L1 gives zeros and L2 doesn't

For a suitable budget corresponding to a penalty optimum, minimize data loss subject to $\|w\|_1\le c$ or $\|w\|_2^2\le c$. In two dimensions, the L1 feasible set is a diamond and the L2 set a disk. An expanding loss contour may first touch an L1 corner, where a coefficient is zero. It need not do so: lasso can retain every feature for a small enough penalty. If the unconstrained minimum is inside the budget, no boundary contact is required.

The decisive calculation is one-dimensional. Let a normalized independent coordinate have unpenalized coefficient $a$. Ridge minimizes $(w-a)^2/2+\lambda w^2/2$. Differentiation gives $(1+\lambda)w-a=0$, hence $w=a/(1+\lambda)$.

Lasso minimizes $(w-a)^2/2+\lambda|w|$. For $w>0$, stationarity gives $w=a-\lambda$; for $w<0$, it gives $w=a+\lambda$. At zero, the L1 subgradient is the interval $[-\lambda,\lambda]$, so zero is optimal whenever $|a|\le\lambda$. Together:

For $a=(3,0.4,0)$ and $\lambda=1$, ridge gives $(1.5,0.2,0)$ while lasso gives $(2,0,0)$. This both demonstrates sparsity and refutes “ridge never has zeros”: an already-zero coordinate can remain zero. Ordinary gradient descent with a sign term is not a reliable exact-sparsity solver; coordinate descent or proximal updates explicitly implement the thresholding logic.

For elastic net in these normalized coordinates,

This separates thresholding from shrinkage. For correlated columns the coordinates interact, so this closed form does not solve the whole problem independently; coordinate descent repeatedly updates one coefficient given the current others.

Ridge vs Lasso vs Elastic Net — full comparison

Ridge's stationary equation for centered data is

The factor $n$ follows from the mean-loss normalization. With positive $\lambda$ and all slopes penalized, $v^T(X^TX+n\lambda I)v=\|Xv\|^2+n\lambda\|v\|^2>0$ for nonzero $v$, so the system is invertible. Use solve, an SVD or a suitable iterative solver instead of forming an explicit inverse. An unpenalized intercept and zero penalty require their own rank conditions.

The scikit-learn Ridge objective is sum squared error plus alpha times squared norm. Thus Ridge(alpha=n*lambda) matches our teaching convention on a fixed sample of size $n$. Lasso(alpha=lambda) and ElasticNet(alpha=lambda, l1_ratio=rho) already use the mean half squared error convention. The same numeric alpha across all three estimators is not an equal-strength comparison.

Identical standardized columns illustrate the difference. For a fixed sum $w_1+w_2=s$, L2 is minimized at $w_1=w_2=s/2$. L1 has equal penalty for any same-sign split of $s$, so multiple coefficient assignments can make identical predictions. A solver choosing one column does not prove that column caused the outcome. Claims that lasso can never select more than $n$ features require conditions on the solution/design; nonunique degenerate solutions need qualification.

The bias-variance trade-off, one dial

Increasing a penalty limits how strongly a fitted model reacts to training detail. In many settings it raises bias while lowering variance, improving expected error if the variance reduction outweighs the squared-bias increase. An already underfit model can become worse. Zero penalty does not guarantee overfitting, and no theorem says every finite validation curve is U-shaped.

For an exactly optimized objective with a fixed dataset, stronger penalty cannot lower the unpenalized training loss along the usual penalized trade-off. Finite optimization error can obscure that property. Test/validation error has no corresponding monotonic guarantee.

As a slope penalty grows very large and the intercept is free, a squared-error model approaches the training mean. Logistic regression approaches an intercept-only probability model rather than a numeric target mean prediction in an arbitrary scale. Never interpret the “best lambda” without naming the objective, feature scaling and dataset.

The regularization path — every coefficient as λ shrinks

A path fits a sequence of strengths and records coefficients and validation performance. It reveals which coordinates are stable and which fluctuate. For correlated features, lasso variables can enter and leave; active-set size and individual coefficient magnitudes need not change monotonically. At alpha=0, use the unpenalized solver rather than expecting lasso coordinate descent to be the right OLS implementation.

Feature units change the penalty. Replacing income in dollars by income in millions divides its required coefficient scale differently; an equal numeric penalty then expresses a different prior. Standardizing is a sensible default for comparable slope penalties, not a universal requirement that all binary features must be scaled. Some domain penalties intentionally operate in original units. Sparse centering can densify matrices, so use an appropriate non-centering scaler or another representation when needed.

(c) Hands-on · 25 min

The lab fits OLS and three penalized alternatives on synthetic data with known coefficients. The cross-validation search wraps the whole pipeline. A scaler followed by LassoCV inside a pipeline would fit the scaler once before the estimator's internal CV, leaking inner-validation statistics. GridSearchCV(Pipeline(...)) avoids that boundary error.

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, KFold, GridSearchCV
from sklearn.metrics import root_mean_squared_error
 
X, y, truth = make_regression(n_samples=200, n_features=50,
    n_informative=5, noise=15, coef=True, random_state=42)
Xdev, Xtest, ydev, ytest = train_test_split(X, y, test_size=.25, random_state=42)
cv = KFold(n_splits=5, shuffle=True, random_state=13)
configs = [
    ('OLS', LinearRegression(), {}),
    ('Ridge', Ridge(), {'model__alpha': np.logspace(-2, 3, 9)}),
    ('Lasso', Lasso(max_iter=20000), {'model__alpha': np.logspace(-2, 2, 9)}),
    ('ElasticNet', ElasticNet(max_iter=20000), {
        'model__alpha': np.logspace(-2, 2, 9), 'model__l1_ratio': [.2, .6, .9]})]
fits = {}
for name, estimator, grid in configs:
    pipe = Pipeline([('scale', StandardScaler()), ('model', estimator)])
    search = GridSearchCV(pipe, grid, cv=cv,
        scoring='neg_root_mean_squared_error', n_jobs=1, error_score='raise')
    search.fit(Xdev, ydev)
    fits[name] = search
    w = search.best_estimator_.named_steps['model'].coef_
    print(name, 'CV RMSE', -search.best_score_, 'settings', search.best_params_,
          'nonzero', int(np.count_nonzero(np.abs(w)>1e-4)))
 
# Select model family without opening test scores.
winner = max(fits, key=lambda name: fits[name].best_score_)
chosen = fits[winner].best_estimator_
print('chosen', winner, 'final test RMSE', root_mean_squared_error(ytest, chosen.predict(Xtest)))
 
# Selection recovery is a synthetic-data diagnostic, not causal inference.
lasso_w = fits['Lasso'].best_estimator_.named_steps['model'].coef_
real = set(np.flatnonzero(truth))
selected = set(np.flatnonzero(np.abs(lasso_w)>1e-4))
print('recovered', sorted(real & selected), 'false positives', sorted(selected-real),
      'missed', sorted(real-selected))
 
# The path is descriptive on development data, not a new test-set search.
Z = StandardScaler().fit_transform(Xdev)
for alpha in [100, 30, 10, 3, 1, .3, .1, .01]:
    m = Lasso(alpha=alpha, max_iter=20000).fit(Z, ydev)
    print('path', alpha, 'active', np.count_nonzero(np.abs(m.coef_)>1e-4))
 
# Verify the hand-derived normalized-coordinate solutions.
a = np.array([3., .4, 0.])
soft = lambda a, lam: np.sign(a)*np.maximum(np.abs(a)-lam, 0)
assert np.allclose(a/(1+1), [1.5, .2, 0])
assert np.allclose(soft(a, 1), [2, 0, 0])
 
# Verify the ridge normal equation against sklearn under matching normalization.
Xc = Z - Z.mean(axis=0)
yc = ydev - ydev.mean()
lam = .2
w = np.linalg.solve(Xc.T@Xc + len(yc)*lam*np.eye(Xc.shape[1]), Xc.T@yc)
ref = Ridge(alpha=len(yc)*lam, fit_intercept=False).fit(Xc, yc).coef_
assert np.allclose(w, ref, atol=1e-8)
 
# Concrete correlated-column experiment, using an actually informative column.
j = int(np.flatnonzero(truth)[0])
rng = np.random.default_rng(9)
Xdup = np.column_stack([Xdev, Xdev[:, j]+rng.normal(0, .001, len(Xdev))])
for model in [Ridge(alpha=1), Lasso(alpha=1, max_iter=20000),
              ElasticNet(alpha=1, l1_ratio=.5, max_iter=20000)]:
    p = Pipeline([('scale', StandardScaler()), ('model', model)]).fit(Xdup, ydev)
    print(type(model).__name__, 'original/copy', p.named_steps['model'].coef_[[j, -1]])

What each block does

make_regression supplies five truly nonzero coefficients, but its shuffled informative columns need not include column zero. The duplicate-column exercise therefore selects a nonzero entry of truth explicitly. It only examines coefficients on development data; it does not claim elastic net must win prediction RMSE.

The shared fold assignments support a fair comparison; the scaler is refit inside each training fold. A log grid explores orders of magnitude economically, then a narrower grid can refine the selected region if the validation budget permits. Searching many alternatives itself risks validation overfitting, so use nested CV for an unbiased selection-procedure estimate when a separate test set is unavailable.

All reported strengths are library alpha values. The code separately checks the ridge scaling equation to prevent an accidental normalization mismatch. root_mean_squared_error is the current API; the former mean_squared_error(..., squared=False) call is not valid in scikit-learn 1.7. Exact coefficient recovery is not asserted because finite noise, correlated columns and the chosen penalty affect it.

(d) Production reality · 15 min

Regularization beyond linear regression

A Gaussian prior $p(w)\propto\exp(-\|w\|^2/(2\tau^2))$ combined with Gaussian observation noise variance $\sigma^2$ gives MAP objective $\|y-Xw\|^2/(2\sigma^2)+\|w\|^2/(2\tau^2)$, up to constants. Multiplying by $\sigma^2/n$ identifies our ridge strength as $\lambda=\sigma^2/(n\tau^2)$. This is a MAP connection, not a claim that ridge supplies the full Bayesian posterior. A Laplace prior gives an L1 term by the same negative-log calculation.

For plain SGD with loss $L+\lambda\|w\|^2/2$, the update is

That matches multiplicative weight decay under the stated convention. Adam rescales gradient coordinates using moment estimates. Adding L2 to that gradient also subjects the penalty to adaptive rescaling; applying a separate multiplicative shrinkage does not. AdamW decouples weight decay from the adaptive loss-gradient update, rather than making it identical to ridge. Optimizer, learning-rate schedule and parameter-group exclusions therefore belong in the experimental specification.

Dropout randomly masks activations during training and changes the effective learning problem. Early stopping restricts the optimization trajectory using validation behavior. Data augmentation imposes task-appropriate invariances through transformed examples. These can regularize learning, but are not interchangeable L2 penalties or guaranteed improvements. More independent relevant data can reduce estimation uncertainty, but adding rows is not literally an explicit penalty term.

Regularizing a matrix-factorization model

The same penalty principle extends beyond a single coefficient vector. Suppose a rating model predicts observed entry $(u,i)$ as $\hat r_{ui}=\mu+b_u+c_i+p_u^Tq_i$. A possible regularized objective is

Only observed ratings belong in $\mathcal O$; an unobserved rating is not automatically zero. The penalties discourage large latent factors and entity-specific offsets estimated from little evidence. Unlike the global intercept, user/item offsets may reasonably be penalized toward a shared baseline—so “never penalize any bias parameter” would be too broad. Holding one factor matrix fixed gives a regularized least-squares subproblem for the other, but joint bilinear fitting is not the convex ridge problem. Validate according to whether future ratings involve known users/items or cold-start entities, and tune strengths under the chosen sum/mean convention. This is a derived modeling example, not a claimed Netflix training result.

Choosing a model under a serving budget

If acquiring each feature involves a remote service, a sparse model can reduce latency and failure dependencies. First verify zeros and their stability, then remove the unused transformations and measure that the deployed pipeline actually avoids those calls. With correlated features, selection instability may matter more than a small average RMSE improvement: repeat fits or bootstrap development samples and record selection frequencies and prediction differences.

With 10,000 features and 500 rows, ridge is still a valid candidate; positive regularization makes its slope system well-posed even when the design is rank deficient. Start with ridge for a dense-signal hypothesis, lasso for a sparse hypothesis, and elastic net for correlated sparse groups. Pick by appropriate validation and operational constraints, not the bare inequality “features exceed rows.” Coefficient selection alone is not a hypothesis test or proof of causality.

Where this shows up in the rest of the plan

Bias–variance supplies the error decomposition. Decision trees constrain depth and leaf support rather than coefficient norms. Gradient boosting adds regularized leaf values, shrinkage and stage selection. In neural training, distinguish optimizer-specific weight decay from dropout and early stopping instead of translating one numeric lambda across model families.

(e) Recall + stretch · 10 min

1. Write all three objectives. Answer: mean half squared error plus respectively $\lambda\|w\|^2/2$, $\lambda\|w\|_1$, or $\lambda\rho\|w\|_1+\lambda(1-\rho)\|w\|^2/2$. State whether the intercept is penalized.
2. Why can lasso return exact zeros? Answer: at zero its subgradient interval can cancel a nonzero loss gradient. The soft-threshold condition is $|a|\le\lambda$ in normalized independent coordinates.
3. Derive ridge. Answer: setting $X^T(Xw-y)/n+\lambda w=0$ gives $(X^TX+n\lambda I)w=X^Ty$. Omitting $n$ silently changes the strength convention.
4. What breaks without scaling? Answer: equal coefficient penalties act differently on different measurement units. Scaling encodes a choice of comparable feature effects, not proof that all inputs must be Gaussian.
5. How is strength selected? Answer: fit the complete preprocessing/model pipeline within each fold of an outer hyperparameter search. A pipeline around an internally cross-validating estimator does not automatically isolate its internal preprocessing folds.
6. Lasso or ridge? Answer: sparse representation and feature cost favor trying lasso; dense weak signal and coefficient stability favor trying ridge. Validate both when uncertain.
7. Why elastic net? Answer: L1 can remove coordinates while L2 favors sharing among correlated columns; it adds a mixing-ratio search and does not guarantee stable scientific selection.
8. Weight decay equals L2? Answer: under the displayed plain-SGD update, with matched factors. Not generally for adaptive Adam; AdamW implements decoupled decay.
9. Name three other regularizing mechanisms. Answer: dropout masks activations, early stopping selects a trajectory point, augmentation encodes invariance. Each changes training differently.
10. Stretch: correlated copies. Run the final experiment under several seeds. Answer: compare the sum of the group's coefficients, individual coefficients and validation predictions. Lasso can change which copy carries weight while predictions remain similar; no fixed 50/50 split or guaranteed elastic-net victory follows.

Explain-out-loud test

Explain shrinkage versus selection using the scalar derivative/subgradient calculation, then show how a feature's units alter the penalty. Finish by drawing the fold boundary around both preprocessing and fitting. If the explanation stops at “L1 has corners,” include the condition that pins a coefficient to zero.

What comes next

Continue to Bias–Variance Trade-off & Learning Curves to see exactly what is averaged in an error decomposition and what a single train/validation gap cannot prove.

Sources and execution boundary

The scikit-learn 1.7 linear-model guide, Lasso API, common pitfalls and Loshchilov and Hutter, ICLR 2019 were checked on 2026-10-07. The lab is a bounded CPU experiment on generated data with numerical equivalence assertions, not a benchmark of feature-selection reliability on real high-dimensional data. No neural-network training is implied by the optimizer algebra.