Dropout and generalization — masks, scaling and controlled experiments

Course map · Lesson 7 of 18 · Original URL identifier 020

0 · Random masks and generalization

Dropout randomly masks activations during training. The mask changes on each call; it does not permanently delete a neuron. The hard parts are the expectation being preserved, what is not preserved, where randomness enters the model and how to evaluate whether it helps.

A team whose members are sometimes absent is a useful co-adaptation analogy: a strategy depending on one fragile partnership becomes less reliable. It is an analogy, not evidence for a detailed bank-window story or a guarantee that every feature becomes independent. The 2014 JMLR paper motivates preventing co-adaptation and approximate model averaging; it does not prove every model benefits. DropConnect dates to 2013 and stochastic depth to 2016, not 2016 and 2019 respectively.

The existing split-audit, NumPy dropout and normalization programs remain intact below. They are expanded into a full dropout treatment, not replaced by a miniature generalization summary.

Prerequisites

Understand tensor multiplication, expectation and .train()/.eval(). The training-loop lesson is the intended prerequisite, not a nonexistent session-dl-018-training-loop URL. The debugging lesson explains why a train/validation gap is a hypothesis rather than a diagnosis.

Define the evaluation unit first

Imagine ten photographs of each of 100 people. A random image split can put the same person in both training and validation. If deployment concerns unseen people, split by person instead. For future forecasting, a chronological split may be appropriate. The split must reflect the claim you plan to make.

Fit normalization constants, vocabulary choices, feature selectors and other learned preprocessing on training data only. Use validation for model and hyperparameter decisions. Keep a final test set outside that loop. Repeatedly choosing what looks best on validation is itself adaptation to that set; recording how many alternatives you tried is more honest than treating one final score as independent evidence.

Lee et al.'s deduplication study documents how duplicate text and train-test overlap distort language-model evaluation. Its reported improvements belong to the studied datasets, but the basic experimental warning applies here: related samples are not automatically independent evidence.

A tiny split audit

import numpy as np
 
people = np.repeat(np.arange(10), 3)
train = people < 6
valid = (people >= 6) & (people < 8)
test = people >= 8
assert not (set(people[train]) & set(people[valid]))
assert not (set(people[train]) & set(people[test]))
X = np.arange(60, dtype=float).reshape(30, 2)
mu = X[train].mean(axis=0)
scale = X[train].std(axis=0)
scaled_train = (X[train]-mu) / scale
scaled_valid = (X[valid]-mu) / scale
np.testing.assert_allclose(scaled_train.mean(axis=0), 0., atol=1e-14)
assert not np.allclose(scaled_valid.mean(axis=0), 0.)
print('split sizes', int(train.sum()), int(valid.sum()), int(test.sum()))

Validation not having zero mean is not a preprocessing error. Re-centering validation with its own statistics would answer a different question. The synthetic values here are not evidence of a real distribution shift; they let us test the intended data flow.

1 · The idea in one paragraph, then in one picture

Let $p$ be the drop probability, $q=1-p$ the keep probability, and $h_i$ a fixed activation. Draw independent masks $m_i\sim\operatorname{Bernoulli}(q)$. For ordinary elementwise inverted dropout,

That formula assumes $p<1$. At $p=1$, define training output as zero directly instead of dividing by zero. Real implementations also validate probability bounds. Standard elementwise dropout independently masks tensor elements, typically including separate masks per example; channel dropout and recurrent shared masks deliberately use different mask shapes.

The light-bulb picture is useful: surviving bulbs are brighter to preserve expected input to the next linear map. One draw need not preserve its mean, variance or final prediction. New masks are sampled at each forward call; resetting the RNG before every call accidentally freezes the mask.

2 · Deriving the scale factor from scratch

Given fixed $h$ and $q>0$,

The constant $c$ in $cmh$ must satisfy $cq=1$ if this expectation is the requirement. It is not a tunable correction. At $h=2,p=.5$, output is 0 or 4, mean 2, variance 4. At $p=.1$, noise standard deviation is $|h|/3$; at $p=.9$ it is $3|h|$. Higher p increases mask noise, but generalization quality is not a monotone function of p.

Two conventions are valid: the original mask-at-training / scale-by-q-at-evaluation convention and the inverted scale-at-training / identity-at-evaluation convention. They match the appropriate local expectations under their conventions; this does not make independently trained nonlinear networks pointwise identical. Legacy evaluation scaling is not inherently a bug—mixing conventions is. Inverted dropout is convenient because evaluation becomes an identity operation: the inference path needs neither random masks nor a dropout-specific rescaling step. That engineering advantage does not prove one convention generalizes better.

import numpy as np
 
rng = np.random.default_rng(12)
h, drop = 2., .5
samples = h * (rng.random(100000) >= drop) / (1-drop)
assert abs(samples.mean()-2) < .03
assert abs(samples.var()-4) < .03
print('dropout mean and variance', round(samples.mean(), 3), round(samples.var(), 3))

Expectation does not commute with a nonlinear function. If dropout makes $H$ either 0 or 2, then $\mathbb E[H]=1$, yet $\mathbb E[H^2]=2\ne(\mathbb E[H])^2=1$. Thus turning dropout off is not exact arithmetic averaging of every nonlinear subnetwork's prediction.

3 · Worked numeric example on a 2×3 activation tensor

Take

Then $\widetilde h=\begin{bmatrix}2&0&6\\0&10&12\end{bmatrix}$. Its sample mean is 5, not the original 3.5. Over all independent masks the expected mean is 3.5. The variance of the six-element sample mean is $91/36$, because independent variances add and the averaging coefficient is $1/6$; its standard deviation is about 1.589899, not 3.5.

Do not set the scale from the realized surviving fraction. With configured p=.5, five survivors out of eight are still scaled by 2, not 8/5. The original mask diagram's 1.6 factor corresponds to configured keep probability .625, not an automatic count-based normalization.

import itertools
import math
import torch
h = torch.arange(1.,7.,dtype=torch.float64).reshape(2,3)
mask = torch.tensor([[1.,0.,1.],[0.,1.,1.]])
y = h*mask/.5
assert torch.equal(y, torch.tensor([[2.,0.,6.],[0.,10.,12.]]))
means = torch.tensor([(h.flatten()*torch.tensor(bits)/.5).mean().item()
                      for bits in itertools.product([0.,1.], repeat=6)], dtype=torch.float64)
assert abs(means.mean().item()-3.5) < 1e-12
assert abs(means.var(unbiased=False).item()-91/36) < 1e-6
assert abs(math.sqrt(91/36)-1.589898669) < 1e-8
print("all 64 masks: exact expectation and sample-mean variance verified")

4 · Implementation, mode contract and tests

The core is three operations—draw mask, multiply, divide—but production semantics include validation and edge cases. This teaching implementation assumes finite floating inputs and uses elementwise masks. It is not a NaN/Inf sanitizer: IEEE multiplication makes inf*0 or nan*0 nonfinite, including in the p=1 branch. Diagnose nonfinite upstream activations rather than expecting masking to repair them. A supplied generator makes experiments reproducible without resetting the global RNG on every forward.

import math
import torch
from torch import nn
import torch.nn.functional as F
 
def dropout(x, p=.5, training=True, generator=None):
    if not math.isfinite(p) or not 0 <= p <= 1: raise ValueError("p must be in [0,1]")
    if not x.is_floating_point(): raise TypeError("floating input required")
    if not training or p == 0: return x
    if p == 1: return x*0
    mask = torch.rand(x.shape, device=x.device, generator=generator) >= p
    return x*mask.to(x.dtype)/(1-p)
 
class MyDropout(nn.Module):
    def __init__(self, p=.5):
        super().__init__(); self.p = p
    def forward(self, x):
        return dropout(x, self.p, self.training)
 
generator = torch.Generator().manual_seed(0)
x = torch.ones(100_000)
for p in [.1,.3,.5,.7]:
    z = dropout(x, p, generator=generator)
    assert abs(z.mean().item()-1) < .015
    assert abs((z==0).float().mean().item()-p) < .01
    assert abs(z.var(unbiased=False).item()-p/(1-p)) < .05
assert torch.equal(dropout(x, .3, False), x)
assert dropout(x, 0) is x and not dropout(x,1).any()
for invalid in [-.1,1.1,float("nan")]:
    try: dropout(x,invalid)
    except ValueError: pass
    else: raise AssertionError("invalid probability accepted")
a = torch.randn(3,4,5,dtype=torch.float64,requires_grad=True)
b = dropout(a,.3); b.sum().backward()
assert b.shape == a.shape and b.dtype == a.dtype
assert torch.all((a.grad == 0) | torch.isclose(a.grad, torch.tensor(1/.7,dtype=a.dtype)))
# model.eval() affects only registered descendants, not an unrelated layer.
layer = nn.Dropout(.2); model = nn.Sequential(layer)
model.eval(); assert torch.equal(layer(x),x)
model.train(); assert not torch.equal(layer(x),x)
custom = MyDropout(.3); custom.eval(); assert torch.equal(custom(x),x)
# Negative control: normal noise is not a Bernoulli mask with keep probability .7.
wrong = (torch.randn(100_000,generator=generator) > .3).float()/.7
assert abs(wrong.mean().item()-1) > .3
print("dropout edge, moment, derivative, registration and negative controls passed")

F.dropout(x,p) defaults to training=True; inside a module pass training=self.training. model.eval() does not disable autograd, and torch.no_grad() does not disable dropout. Finite-difference gradient checks must hold the mask/RNG fixed across perturbations; otherwise the numerator changes both weights and sampled function.

5 · Why can random masking help?

5.1 Co-adaptation prevention

A unit cannot rely on the same partners always being present. Minimizing expected loss over masks puts pressure on representations that remain useful under missing features. This is a mechanism-level intuition, not a guarantee that neurons become independent or that larger models always benefit more. A model short on fitting capacity can become harder to optimize.

5.2 Shared-weight subnetwork ensemble

There are $2^n$ possible binary masks for n maskable units, but the subnetworks share weights and are not trained independently like bagged trees. The original paper discusses approximate model averaging; special linear/log-linear cases give stronger identities than arbitrary deep nonlinear networks. Evaluation with the full scaled convention is an approximation, not a general geometric-mean theorem for every ReLU network or a promise of reduced out-of-distribution variance.

5.3 Noise and a local regularization derivation

For a linear predictor with squared loss, masking input coordinates has an exact useful identity. Let $\widehat y=w^T x$ and independently apply inverted dropout to fixed x. Then

The cross term vanishes because the masked predictor's mean is unchanged; the remaining term is its variance. This is a feature-scaled quadratic penalty in this model, not proof that all deep dropout is identical to ordinary weight decay or Gaussian noise. For a smooth nonlinear loss a Taylor expansion gives a local curvature-weighted noise term, with approximation assumptions.

import itertools
import torch
x = torch.tensor([1.,2.],dtype=torch.float64)
w = torch.tensor([.3,-.4],dtype=torch.float64); target = .7
actual = sum((target-(w*x*torch.tensor(m,dtype=x.dtype)/.5).sum()).square().item()
             for m in itertools.product([0.,1.],repeat=2))/4
oracle = (target-(w*x).sum()).square().item()+(w*w*x*x).sum().item()
assert abs(actual-oracle) < 1e-12
print("linear squared-loss dropout penalty", actual)

6 · Placement: representations, attention and residuals

For a small MLP, Linear → activation → Dropout is a reasonable starting ablation; keep the output Linear unmasked unless deliberately studying a different objective. A CNN may use head dropout, channel dropout or structured spatial masks; convolutional dropout is not universally forbidden. Neighboring pixels can make independent element masks less effective at removing spatial information.

For a residual block, x + dropout(branch(x)) preserves the identity path. dropout(x + branch(x)) also corrupts the skip signal and defines a different architecture; it is not the original Transformer branch-dropout recipe. The 2017 Transformer uses dropout on sublayer outputs before residual addition, on embedding-plus-position representations, and its attention construction can apply dropout to attention probabilities. If attention probabilities are masked and rescaled, their row sum is not necessarily one on each draw; expectation is the preserved quantity.

Choose the actual model's configuration rather than claiming every GPT, Llama or Gemma block contains identical dropout sites. There is no 'Llama 4 8B decoder' premise to build the explanation on. Fine-tuning adapter dropout and backbone attention dropout are distinct knobs.

7 · Choosing p and comparing other regularizers

For an overfitting small MLP, compare p=0, .1, .3 and .5; these are experimental candidates, not universal defaults. In a ResNet, first reproduce its normalization/augmentation recipe, then compare a head or structured mask if the held-out objective warrants it. For an LM, start from the actual configuration and test dropout only with a comparable token/compute budget. TrainingArguments is not a universal source of model dropout=.1; model config and adapter config have their own fields/defaults.

More data and valid augmentation can encode task invariances, but a horizontal flip is not valid for every label. Weight decay shrinks parameters; with AdamW it is decoupled from adaptive gradient normalization and has its own LR/update interaction. It is not always harmless or mandatory. Dropout changes sampled computation; it does not create new labeled information. There is no rule limiting a model to two regularizers. Tune combinations with controlled budgets and report the entire recipe.

High p can be too noisy for a small model, but p=.9 is not mathematically unusable. Its survivor scale is 10 and conditional variance is nine times the squared activation; assess that against width/data/optimizer. Placement and p both matter, not a theorem that one always dominates.

8 · Dropout and normalization: variance shift

If a random activation X has mean $\mu$ and variance $\sigma^2$, then inverted dropout gives

A BatchNorm layer downstream can accumulate statistics of this noisier training distribution and then receive a clean distribution at eval. Putting dropout after a nearby BatchNorm may reduce a direct mismatch, but a later BatchNorm can still see transformed noise. Li et al.'s variance-shift paper studies this interaction; it does not prove a universal fixed ImageNet penalty or prohibit every combined design.

LayerNorm/RMSNorm do not use batch-running estimates in their usual forms, so they avoid that particular running-statistics mismatch. They still interact with masking through input distribution, axes, nonlinearities and residual scale. 'No batch statistics' does not mean 'no interaction'.

Normalization is about axes

For a batch matrix $(B,D)$, a simple BatchNorm-style calculation computes one mean and variance per feature over the batch. LayerNorm instead computes statistics over the specified features of each observation. Layer Normalization describes this per-case normalization and the same calculation at training and test time. Framework BatchNorm typically maintains running statistics for evaluation; this NumPy demonstration shows only the instantaneous normalization calculation, not a complete BatchNorm implementation.

For rows $(1,3)$ and $(5,7)$, feature means are $(3,5)$, while row means are 2 and 6. Ignoring epsilon, column normalization gives rows $(-1,-1)$ and $(1,1)$; row normalization gives $(-1,1)$ for both rows. Neither axis choice is “the same thing but faster.”

import numpy as np
 
x = np.array([[1., 3.], [5., 7.]])
batch = (x-x.mean(axis=0)) / np.sqrt(x.var(axis=0)+1e-5)
layer = (x-x.mean(axis=1, keepdims=True)) / np.sqrt(x.var(axis=1, keepdims=True)+1e-5)
np.testing.assert_allclose(batch, [[-1,-1],[1,1]], atol=1e-5)
np.testing.assert_allclose(layer, [[-1,1],[-1,1]], atol=1e-5)
print('different normalization axes verified')

9 · Six failure scenarios and how to test them

1. Forgot eval: repeated calls can differ. Test registered dropout in train and eval separately. A persistent loss gap is not a unique signature; augmentation and timing also matter.
2. Too much masking: log fitting capacity, loss and mask distribution. Compare p=0 before blaming the optimizer. Do not demand a predetermined U-shaped curve.
3. Missing survivor scale: the mean test on ones rejects mean near q instead of one. Both train-scale and eval-scale conventions are valid when consistently implemented.
4. Masking logits unintentionally: this changes the training objective and score distribution. Use representation dropout as the controlled baseline; no invented eight-point accuracy loss is needed.
5. Masking the residual sum: compare the two equations. The identity route is no longer clean; when a branch mask is zero, x+0 preserves x while masking the sum does not.
6. Different rates across DDP ranks: independent masks at the same configured rate are expected. Mismatched configs optimize a different mixture objective. Compare gathered configuration hashes/all rates; checking only that a sum equals a local value times world size is not a universal all-equal proof. No distributed run is executed here.

10 · Variants: choose what is masked

10.1 DropConnect — weights, 2013

Mask weights rather than activation values. For a Linear map, use a sampled W_masked=W*M/q during training and clean W at evaluation under the inverted convention. A mask shared across all examples induces different correlations from per-example activation masks. This is not guaranteed better for any model family.

10.2 Stochastic depth and DropPath — residual branches, 2016

With branch keep probability q, one modern inverted implementation is x + b*f(x)/q; b can be one per example broadcast over all remaining dimensions. A dropped branch is exactly the identity for that example. Depth-dependent survival probabilities are a recipe choice; the original stochastic-depth paper and later per-example DropPath implementations should not be treated as identical mask granularity. ConvNeXt V2 is a 2023 paper, not a 2025 invention.

10.3 Zoneout — retain an old recurrent state

Instead of setting a state coordinate to zero, randomly retain its previous value: h_next = m*h_previous + (1-m)*h_candidate for a mask that means retain. This preserves an identity route through time. It is still meaningful for recurrent architectures; transformers did not make the mechanism mathematically obsolete.

10.4 DropBlock — contiguous feature regions

DropBlock masks contiguous spatial regions, targeting correlated spatial information that independent pixels may fail to remove. Its sampling probability is adjusted for feature/block geometry and boundary effects; setting a seed-mask probability equal to the desired final drop fraction is generally wrong because blocks overlap. Use a versioned implementation for training and verify achieved fraction/normalization. The conceptual distinction is locality, not a promise that most vision recipes use it.

import torch
import torch.nn.functional as F
 
torch.manual_seed(6)
x = torch.ones(4,3); w = torch.arange(6.).reshape(2,3)
mask = torch.tensor([[1.,0.,1.],[0.,1.,0.]])
out = F.linear(x,w*mask/.5)
assert torch.equal(out[0],torch.tensor([4.,8.]))
branch = torch.full_like(x,2.)
keep = torch.tensor([0.,1.,0.,1.]).reshape(4,1)
y = x+keep*branch/.5
assert torch.equal(y[0],x[0]) and torch.equal(y[1],torch.full((3,),5.))
previous = torch.tensor([1.,2.]); candidate = torch.tensor([9.,8.])
retain = torch.tensor([True,False])
assert torch.equal(torch.where(retain,previous,candidate),torch.tensor([1.,8.]))
print("DropConnect, branch-mask and Zoneout arithmetic passed")

11 · Large-model pretraining: configuration is not a causal theorem

Some large LM recipes use zero dropout. That observation does not prove dropout is universally useless above a token/parameter ratio, nor that a model with 100k fine-tuning examples is 'capacity-limited' in the sense of too little capacity. A small fine-tuning dataset relative to model capacity may create overfitting risk, but the benefit of an adapter/dropout setting still needs validation.

Chinchilla studies compute-optimal allocation between parameter count and tokens; a roughly 20-token-per-parameter heuristic belongs to a particular fitted regime. It does not give a dropout phase transition. Data-constrained scaling studies repeated data and finite unique-token budgets. More compute cannot create fresh information by itself. For arithmetic, 15.6T/405B is about 38.52 tokens per total parameter, not 30–100 times a 20-token baseline; an MoE's total and active parameter counts give different ratios and must be named.

An inference-oriented training budget may deliberately train a smaller model longer than a training-compute-optimal recommendation. That is a different objective, not evidence that compute is cheap or every model is incapable of overfitting. Use train/validation diagnostics and matched regularization ablations rather than an invented 'Rethinking Chinchilla' result to infer causation.

13 · Try it yourself — a controlled local ablation

The original MNIST exercise remains a valid optional protocol when an authorized local dataset already exists: hold split, initialization policy, optimizer and data order fixed; compare p=0, .3 and a sweep; compare built-in versus custom implementations statistically, not by requiring identical RNG kernels. Do not download MNIST for this review. The complete synthetic CPU version below exercises the same ablation mechanics. It is not a reproduction of MNIST accuracy or evidence that dropout must win.

import copy
import torch
from torch import nn
import torch.nn.functional as F
 
torch.manual_seed(19)
x = torch.randn(40,6); y = (x[:,0]+x[:,1] > 0).long()
vx = torch.randn(120,6); vy = (vx[:,0]+vx[:,1] > 0).long()
results = []
for p in [0.,.1,.3,.5,.7,.9]:
    torch.manual_seed(10)
    model = nn.Sequential(nn.Linear(6,24),nn.ReLU(),nn.Dropout(p),nn.Linear(24,2))
    opt = torch.optim.AdamW(model.parameters(),lr=.02,weight_decay=.01)
    for _ in range(100):
        model.train(); opt.zero_grad()
        F.cross_entropy(model(x),y).backward(); opt.step()
    model.eval()
    with torch.no_grad():
        row=(p,F.cross_entropy(model(x),y).item(),F.cross_entropy(model(vx),vy).item())
    assert all(torch.isfinite(torch.tensor(row)))
    results.append(row)
print("p, clean-train CE, held-out CE", results)

To compare custom and built-in dropout, run several seeds and compare mask fraction, mean, gradient contract and held-out distributions; same seed need not generate the same masks across distinct implementations. To distinguish intuitions, hold compute/parameters fixed, measure robustness to feature removal and compare deterministic eval with Monte Carlo mask-averaged predictions. Co-adaptation and ensemble explanations are not mutually exclusive, so one small ablation cannot conclusively choose only one.

14 · Recap and worked recall answers

Q1 — Where is the scale? During training, multiply by a Bernoulli keep mask and divide by q. Evaluation is identity. Handle p=1 separately, validate bounds and connect self.training to functional calls.

Q2 — Reproduce the example. The given mask produces rows (2,0,6) and (0,10,12), sample mean 5; expected mean 3.5. Independent-mask sample-mean variance is 91/36. A realized survivor fraction does not set the scale.

Q3 — Two intuitions? Co-adaptation: features must tolerate missing partners. Ensemble: shared weights train many masked subnetworks. Neither proves exact nonlinear ensemble averaging or universal generalization improvement; the squared-loss calculation supplies one exact special case.

Q4 — Choose p for an MLP, ResNet and LM. For an overfitting MLP compare 0/.1/.3/.5; for ResNet reproduce its BN/augmentation baseline before a head/structured-mask ablation; for an LM inspect the actual backbone/adapter config and token-budget regime. These are experiments, not architecture-wide constants.

Q5 — Why zero dropout in a large recipe? The authors selected that recipe under their data/compute/model objective. Configuration alone does not establish the cause. Reconsider when a valid held-out objective worsens while fitting improves, after checking split/mode/reduction, and test against other regularizers.

Q6 — Can an experiment separate the intuitions? Measure feature-removal sensitivity and Monte Carlo prediction aggregation under matched training. A deep wide network can show both effects, so report which measurements support which mechanism instead of asserting mutually exclusive predictions.

Additional preserved checks: near-zero training loss alone does not establish generalization; for h=3,p=.25 variance is 3; tiny-batch BN changes its statistic estimator; .eval() does not disable autograd. These answers preserve the original current generalization lesson's exit questions.

Sources and execution boundary

Srivastava et al., JMLR 2014, PyTorch dropout v2.9.0, DropConnect 2013, stochastic depth 2016, Zoneout 2016, DropBlock 2018, variance-shift study, Chinchilla and data-constrained scaling support the bounded mechanisms. Exact CPU examples and local ablation are executed. No named-model dropout performance, MNIST download, DDP job or GPU benchmark is reproduced.

Optional pre-read resources

These inherited resource links are optional, not the primary evidence for this review. Video identity, duration and current availability are not certified.

• Dropout Regularization (C2W1L06)
• Dropout in PyTorch
• Dropout: A Simple Way to Prevent Neural Networks from Overfitting
• Neural Networks 2 — Regularization

Previous: ← DL S019 · Debugging · Next: DL S021 · BatchNorm and LayerNorm →

Optional earlier local-lab route: Previous: A complete neural network: learn XOR and audit every gradient · Next: Convolution: shared local weights, shapes, and backward accumulation