DL S020 · Dropout — Why Random Kills the Smart

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 20 / 80 · Module M04 · ~3 hours

The story · a bank, a busy year, and a bag of dice

It is somewhere around late 2011. Geoffrey Hinton is walking into a bank in Toronto. He is standing at a teller window, and the teller in front of him is unfamiliar. He asks a routine question, is told "sorry, the person who normally handles that is at a different branch today," and moves on with his morning. But something about the exchange sticks. Later he mentions it in a talk: banks deliberately rotate their tellers between branches, because if the same three employees always work the same window with the same customers, small conspiracies become easy. A regular teller who quietly skims a penny off every transaction is invisible; a stranger doing the same thing gets caught the first time a customer notices the receipt looks weird. Randomness in staffing prevents collusion.

Neural networks in 2011 have a collusion problem. They overfit spectacularly. A 4-layer MLP on MNIST can drive its training loss to essentially zero while its test error hovers stubbornly above 1.5%. Weight decay helps a little. Early stopping helps a lot but feels like giving up. L1 sparsity works for some architectures, breaks others. The field has half a dozen tricks, none of them dominant.

Hinton walks back to his office and tries something absurd. What if, on every training minibatch, we picked a random 50% of the neurons in each hidden layer and just set their outputs to zero for that step? No gradient flows through them. The remaining neurons have to do the whole job. Next minibatch, a different random 50% get zeroed. Every neuron is now, on average, working alongside a random subset of its usual collaborators.

The results are so good that even Hinton's collaborators think he made an error. Nitish Srivastava reproduces them. The 2012 preprint shows 20-30% reductions in test error across MNIST, CIFAR, ImageNet, TIMIT phoneme recognition, and Reuters text classification — every benchmark the group runs. The final JMLR paper (Srivastava et al. 2014) is one of the ten most cited ML papers of the decade. The technique gets a one-word name that any programmer immediately understands.

Dropout.

The name is misleading. Neurons are not "dropped out" in a permanent sense — they come back next step. A better name would be "stochastic thinning" or "Bernoulli masking." But "dropout" won because it captures the feel of the operation: at each step, you drop a random subset. Everyone still calls it that. Everyone still uses it. In 2025, when you open the source of Meta's Llama 4 8B decoder, or Google's Gemma 3, or the Qwen 2.5 series, you will find nn.Dropout(0.0) in the transformer block — the setting is zero by default, but the slot is there because the architecture was designed around dropout being available. When Llama 4 fine-tuners want a little regularization at low-data SFT, they set it to 0.05 and move on.

Today you are going to (1) implement dropout in three lines, (2) derive from first principles why those three lines contain an extremely subtle scale factor that half of every intro tutorial gets wrong, (3) internalize two competing intuitions for why random destruction reduces overfitting, and (4) walk out with defensible defaults for p on MLPs, CNNs, and transformers — the three regimes where the correct answer differs by 5×.

Prerequisites

• Session 013 — vectorized ops on tensors. Dropout is one broadcast multiply.
• Session 015 — nn.Module, .train() / .eval(). Dropout obeys this switch.
• Session 018 — the outer loop where you toggle train/eval mode.
• Session 019 — the "val loss oscillates" war story. Foreshadow of §9.

1 · The idea in one paragraph, then in one picture

At every training step, for each neuron in a chosen layer independently, flip a biased coin (heads with probability p, tails with probability 1-p). If heads: zero this neuron's output for this step. If tails: keep it, but scale it up by 1/(1-p). At test time: use all neurons, unscaled. That is dropout.

Picture the hidden layer as a row of light bulbs. Normally every bulb is on and contributes some brightness to the next layer. Dropout walks the row with a coin. Bulb 1 stays on, brighter than usual to compensate. Bulb 2 off. Bulb 3 off. Bulb 4 on, brighter. Bulb 5 on, brighter. Bulb 6 off. The next layer sees a jittery version of the input where a random 30-50% of the sources have vanished. The net has to learn to still get the answer right despite this. Next minibatch: a completely different pattern of on/off. Over thousands of steps, no small clique of bulbs can be counted on to always fire together, so no clique can encode a fragile joint pattern. Every bulb has to carry meaningful, independent information.

Formally, for a layer's output vector h ∈ ℝ^d:

Two lines of tensor math. That is the entire operation. Everything else in this session is either "why the scale factor is what it is" or "what happens when you get placement / rate / interaction with other layers wrong."

2 · Deriving the scale factor from scratch

This is the one paragraph in the session that is worth reading three times. Let it stick.

Consider a single scalar activation h. During training with dropout at rate p, we replace h with a random variable H defined by:

Compute the expectation:

The expected value of the dropped-and-scaled activation equals the original activation. That is exactly the property we want. The next layer's weights W see an input whose mean is unchanged. Only the variance changes:

So dropout adds multiplicative Gaussian-flavored (actually Bernoulli-flavored) noise to each activation with variance proportional to p/(1-p). At p=0.5 the variance is h², i.e. as large as the signal itself. At p=0.1 (transformer default), the variance is 0.111 h² — noise about a third of the signal in amplitude. That is the "how much noise am I injecting?" knob.

If you had scaled by anything other than 1/(1-p) — say by 1 (naive dropout) — then E[H] = (1-p) h, and the next layer sees inputs that are (1-p)× smaller at train time than at eval time. The weights adapt to the smaller inputs. At inference, the inputs are suddenly 1/(1-p)× larger, so pre-activations are 1/(1-p)× larger than the ReLU/sigmoid saw during training. For ReLU nothing dramatic breaks. For sigmoid you saturate. For a transformer's softmax you get a distribution collapsed onto one token. Bad.

3 · Worked numeric example on a 2×3 activation tensor

Nothing beats a concrete example. Let

Apply dropout with p = 0.5. Sample masks (imagine the random draw came out this way):

Compute h_train = h * m / (1 - p) = h * m * 2:

Sanity: the mean of h is (1+2+3+4+5+6)/6 = 3.5. The mean of h_train is (2+0+6+0+10+12)/6 = 5.0. That is not 3.5. Why isn't the expectation preserved?

Because expectation is over the random mask, not over one specific draw. If you average h_train across many draws, it converges to 3.5. On any single draw it can be anywhere from 0 (all zeros in the mask) to 7.0 (all ones × 2). The expected mean is preserved. The sample mean fluctuates by ~sqrt(p/(1-p)) × 3.5 ≈ 3.5. That is a large fluctuation. Dropout is a lot of noise per step; it works because you take a lot of steps.

Let's verify in code:

import torch
torch.manual_seed(42)
 
h = torch.tensor([[1., 2., 3.], [4., 5., 6.]])
p = 0.5
 
# Single draw
mask = (torch.rand_like(h) > p).float()
h_train = h * mask / (1 - p)
print("mask:\n", mask)
print("h_train:\n", h_train)
print("this draw's mean:", h_train.mean().item())
 
# Average over 10,000 draws
means = []
for _ in range(10_000):
    mask = (torch.rand_like(h) > p).float()
    means.append((h * mask / (1 - p)).mean().item())
print("avg over 10k draws:", sum(means) / len(means))    # ≈ 3.5
print("std over 10k draws:", torch.tensor(means).std().item())

Run this. You will see the single-draw mean bounces around wildly, but the average across 10,000 draws converges to 3.5 — the mean of the original tensor. That is the E[H] = h property working in practice.

4 · The 3-line implementation, and the test that catches every bug

Three lines of body. Now the test — this is the test I wish I had written the first time I reimplemented dropout by hand for a research project:

def test_dropout():
    torch.manual_seed(0)
    x = torch.ones(100_000)                       # constant tensor for clarity
 
    # 1. Identity at eval
    assert torch.equal(dropout(x, p=0.3, training=False), x)
 
    # 2. Identity at p=0
    assert torch.equal(dropout(x, p=0.0, training=True), x)
 
    # 3. E[output] == input, empirically
    for p in [0.1, 0.3, 0.5, 0.7]:
        y = dropout(x, p=p, training=True)
        assert abs(y.mean().item() - 1.0) < 0.01, f"failed for p={p}"
 
    # 4. Fraction zeroed ≈ p
    for p in [0.1, 0.3, 0.5, 0.7]:
        y = dropout(x, p=p, training=True)
        frac = (y == 0).float().mean().item()
        assert abs(frac - p) < 0.01, f"failed for p={p}"
 
    # 5. Shape and dtype preserved
    x2 = torch.randn(3, 4, 5, dtype=torch.float64)
    y2 = dropout(x2, p=0.3)
    assert y2.shape == x2.shape and y2.dtype == x2.dtype
 
    print("all dropout tests passed")
 
test_dropout()

If your implementation ever silently breaks (a colleague adds a channel dimension and you forgot to broadcast the mask; someone swaps torch.rand_like for torch.randn_like — a real bug I have seen), test 3 or 4 will fire. Ten lines of test, one afternoon saved.

PyTorch's built-in:

import torch.nn as nn
layer = nn.Dropout(p=0.2)          # uses inverted-dropout
y = layer(x)                       # active in train mode
model.eval()
y = layer(x)                       # identity in eval mode

Under the hood, nn.Dropout does exactly what your 3-liner does, but it hooks into model.train() / model.eval() so the training flag flips automatically. Use nn.Dropout in real code. Reimplement from scratch only for teaching and for research where you want a variant.

5 · Why does dropout work? Two competing intuitions

Neither is proven; both are useful. The truth is probably a combination.

5.1 Co-adaptation prevention (Hinton's original story)

Without dropout, neuron A can learn a pattern that only works when neuron B is also firing (they "co-adapt"). Together they detect "cat" — but neither alone means anything. With dropout, B disappears sometimes. A has to be able to detect something on its own, or the pair will fail whenever B drops. This forces neurons to be individually informative.

This intuition is closest to the original paper's motivation. It predicts:

• Higher-capacity models benefit more from dropout (they have more capacity to co-adapt).
• Small models with barely enough capacity for the task should use less dropout, or none.
• Dropout should hurt representational efficiency (spread information across more units) but improve generalization. Both are observed.

5.2 Implicit ensemble (Baldi & Sadowski 2013 formalization)

Every training step, dropout gives you a randomly-thinned subnetwork. Over the course of training you are effectively training an exponential number of subnetworks (there are 2^n possible masks for n neurons) that all share weights. At test time (no dropout, all neurons active), you are approximating the geometric mean of predictions from all these subnetworks — an ensemble.

Both intuitions predict roughly the same thing: dropout should reduce variance of predictions, especially on data far from the training distribution. Empirically this is exactly what it does. The ensemble view has a nice mathematical guarantee (Baldi & Sadowski 2013) that the eval-time forward pass is exactly the geometric mean under the linear approximation and a good approximation under ReLU.

5.3 A third view: noise injection ≈ smoothing the loss landscape

A more modern take (Wei et al. 2020, Mianjy & Arora 2019): dropout is a specific form of Gaussian-like noise injection, and noise injection acts like an implicit L2 regularizer on the inputs to each layer, favoring functions that are smooth (Lipschitz) in the noise direction. This connects dropout to weight decay, data augmentation, and mixup — all of which reduce test error by penalizing overly-sensitive predictions.

6 · Where to put dropout in a modern (2025) model

Example MLP with dropout:

model = nn.Sequential(
    nn.Linear(784, 256), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(256, 128), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(128, 10),
)

Example transformer block (schematic — we build this properly in Session 038-039):

# Inside a transformer block
attn_out = self.attn(self.ln1(x))
attn_out = self.attn_dropout(attn_out)          # dropout on attention output
x = x + attn_out                                 # residual
ffn_out = self.ffn(self.ln2(x))
ffn_out = self.ffn_dropout(ffn_out)              # dropout on FFN output
x = x + ffn_out                                  # residual

Two placements, both after the sub-layer output and before the residual add. This is the pattern from the original 2017 Attention is All You Need paper. Every mainstream transformer still uses it — GPT-3, PaLM, Llama, Gemma — though many set the rate to 0 for large-scale pretraining (see §11).

A less common but powerful variant: attention dropout, applied to the softmax weights themselves before multiplying by V. This is a sparser form of dropout — it drops entire (query, key) pairs rather than dropping neurons. It shows up in some vision transformers (DeiT-III 2022) and in a few recent LLM fine-tuning recipes as a way to prevent the model from over-attending to any single key.

7 · Choosing p — defaults that survive contact with reality

The parameter p is the dropout rate — probability of a neuron being dropped. Larger p = more regularization = harder training. Rules of thumb from the field, cross-checked against 2024-2025 recipes:

• MLPs, small models (<10M params): 0.2–0.5. Start at 0.3.
• CNNs (in the FC head): 0.3–0.5. The 2015 VGG paper uses 0.5.
• Transformers, pretraining from scratch: 0.1 was universal in 2017-2020. Modern large-scale pretraining (Llama 3, DeepSeek-V3) sets it to 0.0 because the model is data-limited, not overfitting-limited — see §11.
• Transformers, fine-tuning: 0.05-0.1 is a common LoRA/QLoRA default. HuggingFace's TrainingArguments defaults to 0.1.
• Overfitting is bad: try higher p first, before adding more data — data is expensive, p is a keyword argument.
• Underfitting: reduce p (or remove dropout entirely). Model doesn't have capacity spare to regularize.

Don't stack too much: dropout + batchnorm + weight-decay + heavy augmentation on a small model = underfitting. Pick two regularizers max, tune them, don't pile. The 2019 "Bag of Tricks" paper for image classification is a great case study in what combinations actually stack usefully.

8 · Dropout vs BatchNorm — an important interaction (foreshadowing S021)

BatchNorm (Session 021) is also a regularizer (weakly). And it has weird interactions with dropout, first characterized by Li et al. 2018 "Understanding the Disharmony Between Dropout and Batch Normalization":

• If you put dropout BEFORE BatchNorm, dropout adds noise to BN's mean/variance estimates. BN accumulates weird running stats. At test time, the running stats are computed from the noisy train distribution but you apply them to the clean test distribution — a "variance shift."
• The 2018 paper demonstrates that this shift can cost 1-2 percentage points on ImageNet. Not huge, but consistent.
• Rule: use dropout AFTER BatchNorm, if at all. In practice, most CNNs since 2015 use BN and skip dropout entirely (or use it only in the FC head where there is no BN).

Transformers use LayerNorm and RMSNorm (see S021), which are per-example — no batch stats, no interaction. That is one of the reasons transformers can keep dropout everywhere while modern CNNs cannot.

9 · War stories from four years of production ML code review

x = x + self.dropout(self.attn(x))   # correct: dropout on branch
x = self.dropout(x + self.attn(x))   # WRONG: dropout on the residual sum

10 · Variants: DropConnect, DropPath, ZoneOut, DropBlock

The 2014 dropout is one point in a design space of "randomly zero out things." A short tour of the useful cousins:

10.1 DropConnect (Wan et al. 2013)

Drop weights instead of activations. Same idea, one level deeper. Mathematically similar but empirically slightly better on some MLPs, slightly worse on most CNNs, and rarely used in transformers. Interesting mainly as a lineage note.

10.2 DropPath / Stochastic Depth (Huang et al. 2016)

In a network with residual connections, at each step, randomly drop entire blocks — set the whole branch to zero and pass only the identity through. Used pervasively in modern vision transformers (Swin, DeiT, ConvNeXt v2). The dropout rate here is typically small (0.1-0.2 per block) but grows with depth. The 2025 ConvNeXt-V2 recipe uses DropPath but not vanilla dropout — the ensemble intuition scales up naturally when you're dropping blocks not neurons.

10.3 ZoneOut (Krueger et al. 2016)

For RNNs. Instead of zeroing a hidden state, hold it constant at its previous value. Preserves temporal signal while still regularizing. Superseded in the LLM era by LayerNorm + transformers, but useful conceptually.

10.4 DropBlock (Ghiasi et al. 2018)

For CNNs specifically: zero out a contiguous spatial region of a feature map, not scattered pixels. The intuition is that scattered pixel dropout is too easy for a CNN to compensate for (neighboring pixels are correlated). Dropping a 5×5 block forces the network to actually rely on other spatial regions.

Most 2025 vision recipes use DropPath (in the transformer/hybrid variants) or DropBlock (in pure convnets). Vanilla per-activation dropout has quietly become a small piece of the toolbox rather than the default.

11 · The 2025 twist — why big pretraining runs set dropout to zero

If you crack open the config for Meta's Llama 3 tech report (2024) or DeepSeek's DeepSeek-V3 tech report (December 2024) or the Gemma 3 model card (2025), one thing jumps out: dropout is set to 0 during pretraining. This flips half of what the field taught for a decade. What changed?

The short answer: at frontier scale, models are data-limited, not capacity-limited. Llama 3 405B trained on 15.6 trillion tokens. DeepSeek-V3 671B trained on 14.8T. Under the Chinchilla scaling laws (Hoffmann et al. 2022) and the more recent Muennighoff et al. 2023 "Scaling Data-Constrained Language Models", you want ~20 tokens per parameter for compute-optimal training; frontier models train on 30-100× that ratio because compute is cheap and web data is (barely) enough. In this regime, the model is not overfitting — it is fighting to memorize. Adding dropout (which is an anti-memorization regularizer) makes the model worse per training FLOP.

The 2024 Kaplan-revisit "Rethinking Chinchilla" analysis confirms this: for models trained past 20 tokens/param, regularization moves loss up not down. Dropout survives in these architectures only because fine-tuning is a different regime — a 100k-example SFT run on a 70B model is capacity-limited, and there p=0.05-0.1 genuinely helps.

The lesson generalizes: regularization is a tool for the overfitting regime. If your training loss is going down and val loss is going down together, you are underfitting and dropout is a tax. If train is going down and val is going up, you are overfitting and dropout is medicine. Always look at the curves before dialing knobs.

Further reading

• Hoffmann et al. (2022), Training Compute-Optimal Large Language Models (Chinchilla).
• Muennighoff et al. (2023), Scaling Data-Constrained Language Models.
• Llama 3 team (2024), The Llama 3 Herd of Models.
• DeepSeek-AI (2024), DeepSeek-V3 Technical Report.
• Li et al. (2018), Understanding the Disharmony Between Dropout and Batch Normalization.

12 · Diagram — dropout in action

And the tensor-level picture across time:

Every step: a different random subnetwork, scaled to preserve expected magnitude. Ensemble of exponentially many nets, all sharing weights.

13 · Try it yourself

14 · Recap

Dropout is Bernoulli masking on activations at train time, scaled by 1/(1-p) so E[output] = input. At eval, it is identity. Three lines of code, one subtle scale factor, one deep intuition (co-adaptation prevention and implicit ensembling — both are correct). It shaped a decade of neural network design, still lives in every modern transformer block, but is quietly set to zero for frontier pretraining because those models are data-limited, not overfitting-limited. Use it when you are overfitting, skip it when you are underfitting, put it after norm layers not before, and never on the output logits.

🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: re-read §2 (scale factor derivation) and §3 (worked example) in 24 hours. Revisit the full session on day 7 with a focus on §11 (2025 twist) and the war stories.

Next session (S021): we cover BatchNorm and LayerNorm — the other dominant "layer that changes activation distributions" in every modern net. BN made 100-layer CNNs possible in 2015. LN made transformers work. Plus we look at RMSNorm, which quietly ate LN's lunch in 2024-2025 LLMs (Llama, Gemma, Qwen all use it), and the 2025 RMSNorm-vs-LayerNorm ablation that gives clean numbers on the trade-off.

Sticky note (keep on your desk):

• 3-line dropout: mask ~ Bernoulli(1-p), then x * mask / (1 - p).
• p defaults: 0.3 for MLPs, 0.1 for transformer training-from-scratch, 0.0 for frontier pretraining, ~0.05 for fine-tuning, none in CNN conv layers.
• Always .eval() in val. Dropout in eval is the single most common bug in this session.

Previous: ← DL S019 · Debugging · Next: DL S021 · BatchNorm and LayerNorm →