DL S026 · LeNet → AlexNet → ResNet — the three architectures that mattered

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 26 / 80 · Module M05 · ~2 hours

The story

Story hook · the night of September 30, 2012

Picture the scene. It is late evening on September 30, 2012. Alex Krizhevsky, a 26-year-old PhD student in Geoff Hinton's Toronto lab, is refreshing an obscure webpage — the results of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). Every year since 2010, computer-vision teams from Oxford, Xerox PARC, NEC, the University of Amsterdam, and roughly 30 others have submitted their best object-classification models. Every year the top-5 error rate crawls down by half a point. The 2011 winner (Xerox's Fisher-vector approach) hit 25.8%. The consensus is that 2012's winner will land somewhere around 24%. Hand-designed feature pipelines. SIFT + Fisher vectors + SVMs. That is how you win.

The leaderboard refreshes. Toronto's entry — a neural network of all things, something the vision community had loudly declared dead ten years earlier — has top-5 error of 15.3%. The next best team is at 26.2%. Krizhevsky's model beat the field by eleven percentage points on a task where the previous year's winner beat the runner-up by 0.6.

That gap crashed a paradigm. Within twelve months, the entire ICCV/CVPR community had thrown out SIFT and started training convnets. Within twenty-four, Google, Microsoft, Facebook, and Baidu had bought or built deep-learning labs. Yann LeCun — who had been quietly maintaining the convolutional flame for two decades — later called the AlexNet moment "the Big Bang of modern AI." Ilya Sutskever, one of the paper's authors, would go on to co-found OpenAI. Alex would join Google Brain. Hinton would sell his three-person startup, DNNresearch, to Google for $44M six months later.

What was AlexNet actually? A slightly deeper LeNet trained on 1.2 million images with three tricks nobody had bothered combining: ReLU (which trained 6× faster than tanh), dropout (which crushed overfitting), and GPU training (which made all of this fit in a reasonable clock). The architecture itself was almost embarrassing in retrospect — five conv layers, three fully-connected layers, 60 million parameters, hard-coded for two GTX 580s because a single GPU didn't have enough VRAM.

But the result was unarguable. And that is the theme running through today's session: the architecture is often a footnote. The recipe — what you optimise, on what data, with what regulariser — is usually where the wins live. LeNet → AlexNet → ResNet → ConvNeXt is a story of increasingly disciplined recipes wrapped around fundamentally similar core primitives.

Let's build them.

The three moments that made vision work

Everything you know about image models traces back to three papers. Not three eras — three specific papers.

LeNet-5 (LeCun et al., 1998). Yann LeCun had already published the convolutional layer idea in 1989. LeNet-5 was the version that reached bank-cheque quality on digit recognition and got deployed. It ran on a CPU, had ~60K parameters, and demonstrated the recipe — conv → pool → conv → pool → fc → softmax — that every image model still uses in some form.

AlexNet (Krizhevsky, Sutskever, Hinton, 2012). LeNet-5 sat on the shelf for 14 years. Nobody cared. Then Alex Krizhevsky took the same recipe, scaled it up 100×, trained it on 1.2M ImageNet images across two GPUs for a week, added ReLU + dropout + GPU tricks, and won ImageNet 2012 by an unheard-of 10 percentage points. That single result triggered the modern deep-learning boom. Every paper you read today, on any topic, is downstream of AlexNet convincing the world that "just make it bigger" works.

ResNet (He et al., 2015). By 2014 people were trying to stack 30+ conv layers and finding that deeper networks got worse than shallower ones — even on the training set. Kaiming He's team realized the layers weren't failing to learn; they were failing to learn the identity function. So they added a shortcut connection that let each block learn a residual on top of the identity. Suddenly 152-layer networks trained fine, and every architecture since — DenseNet, Transformer, ViT, GPT — has borrowed the trick.

Two hours today. We'll implement LeNet on MNIST, dissect AlexNet, and write a residual block by hand.

Prerequisites

• Session 025 (convolution mechanics, shape formulas, receptive fields).
• Session 018–019 (softmax, cross-entropy, backprop). We'll use them without re-derivation.

1 · LeNet-5: the 1998 recipe

LeNet-5 was designed to read hand-written digits on cheques. Input: 32×32 grayscale. Output: one of 10 digit classes. Around 60,000 parameters, which was already considered a lot in 1998.

Architecture (in modern PyTorch idiom)

The pattern — conv → nonlinearity → downsample → conv → nonlinearity → downsample → flatten → fc → fc → out — is the template that everything from AlexNet to ResNet-50 still follows. Only the block widths and the activation function have changed.

PyTorch implementation

Parameter count (do this by hand)

• conv1: 6 · 1 · 5·5 + 6 = 156
• conv2: 16 · 6 · 5·5 + 16 = 2,416
• fc1: 400 · 120 + 120 = 48,120
• fc2: 120 · 84 + 84 = 10,164
• fc3: 84 · 10 + 10 = 850

Total: 61,706. Notice how 80% of parameters are in the fully-connected layers, even though 99% of the work is done by the convolutions. That FC-heavy tail is a running theme — AlexNet has it too, ResNet finally kills it.

Training on MNIST (60 seconds on CPU)

The training loop — five lines that look identical to every training loop you'll write for the rest of this series:

for epoch in range(5):
    model.train()
    for xb, yb in train_dl:
        opt.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        opt.step()
 
    # eval
    model.eval()
    with torch.no_grad():
        correct = sum((model(xb).argmax(1) == yb).sum().item()
                      for xb, yb in test_dl)
    print(f"epoch {epoch}: {correct/len(test):.4f}")

You should hit ~99.1% by epoch 5. On CPU. In under a minute.

2 · AlexNet: what changed in 2012

AlexNet is architecturally not radically different from LeNet — it's LeNet, but bigger, with a few carefully chosen tricks. Here's the delta.

2.1 The four things AlexNet added

1. ReLU instead of tanh. Faster to compute, doesn't saturate for positive inputs, and — this is the underrated part — gradients don't decay through the depth. Enables training a deeper network.
2. Dropout on the FC layers. 50% dropout on the 4096-unit fully-connected layers. Was borderline heretical at the time; is now standard practice.
3. Overlapping max-pooling (3×3 stride 2). A small win over 2×2 stride 2, ~0.3% top-5.
4. GPU training. Two GTX 580s in parallel — the architecture is literally drawn in two halves in the paper because half the channels lived on each GPU. This was the moment training on GPUs became normal.

2.2 The scale jump

Scale up input by 50×, parameters by 1,000×, dataset by 20×, compute by ~100,000×. That is the entire recipe. No new theory. Just — do it, and it works.

2.3 What AlexNet did NOT solve

Depth was still hard. AlexNet has 8 layers total. VGG (2014) got to 19. Beyond that, training just… stopped working. Even on the training set. That's the problem ResNet was invented to solve.

3 · The problem ResNet solved

By 2014 people had noticed something baffling: a 56-layer conv net had higher training error than a 20-layer one on CIFAR-10. Not test error — training error. The deeper network was strictly worse at fitting.

That's weird, because a 56-layer network can trivially emulate a 20-layer one: just make the extra 36 layers the identity function. So the 56-layer network's optimal solution is at least as good as the 20-layer one's. If SGD can't find it, that's an optimization problem, not a representational problem.

Kaiming He's insight: identity is hard to learn. If you initialize a conv layer with small random weights, you get "small random function" — very far from the identity. To become the identity, every weight has to migrate to a very specific configuration. That's a long journey through weight space, and SGD often doesn't make it.

The fix: add a shortcut that carries the input directly to the output, and let the conv layers learn a residual on top:

Now if the block wants to do nothing, its weights just need to go to zero, which is a very short journey from a small-random initialization. Doing "nothing" is now the easy default, and doing "something" is a small perturbation of it.

3.1 The BasicBlock (ResNet-18 / ResNet-34)

class BasicBlock(nn.Module):
    expansion = 1
 
    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 3,
                               stride=stride, padding=1, bias=False)
        self.bn1   = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3,
                               stride=1, padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_channels)
 
        # If shape changes (channel count or stride > 1), project the shortcut.
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, 1,
                          stride=stride, bias=False),
                nn.BatchNorm2d(out_channels),
            )
        else:
            self.shortcut = nn.Identity()
 
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + self.shortcut(x)      # the residual add
        return F.relu(out)

Two things to notice:

• bias=False on every conv. BatchNorm applies its own shift; the conv bias would be redundant (see Session 021).
• self.shortcut. When the two convs change the spatial resolution or channel count, we can't just add x back — the shapes don't match. We use a 1×1 conv as a cheap projection to bring x into the same shape. This is called the "projection shortcut".

3.2 ResNet-18 in ~30 lines

class ResNet18(nn.Module):
    def __init__(self, num_classes=1000):
        super().__init__()
        self.stem = nn.Sequential(
            nn.Conv2d(3, 64, 7, stride=2, padding=3, bias=False),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
            nn.MaxPool2d(3, stride=2, padding=1),
        )
        self.layer1 = self._make_layer(64,  64,  2, stride=1)
        self.layer2 = self._make_layer(64,  128, 2, stride=2)
        self.layer3 = self._make_layer(128, 256, 2, stride=2)
        self.layer4 = self._make_layer(256, 512, 2, stride=2)
        self.avgpool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Linear(512, num_classes)
 
    def _make_layer(self, in_c, out_c, n_blocks, stride):
        layers = [BasicBlock(in_c, out_c, stride=stride)]
        for _ in range(n_blocks - 1):
            layers.append(BasicBlock(out_c, out_c, stride=1))
        return nn.Sequential(*layers)
 
    def forward(self, x):
        x = self.stem(x)                         # (N, 64, 56, 56)
        x = self.layer1(x); x = self.layer2(x)
        x = self.layer3(x); x = self.layer4(x)   # (N, 512, 7, 7)
        x = self.avgpool(x).flatten(1)           # (N, 512)
        return self.fc(x)

Notice the two design decisions that make ResNet's parameter count so lean:

• AdaptiveAvgPool2d(1) collapses the whole final feature map to a single vector per channel. Replaces the giant fc layers of AlexNet/VGG. This is where the fc-heavy tail dies.
• Stage transitions double channels and halve resolution. The information content of the tensor stays roughly constant across stages.

Total parameters: ~11.7M. Compare with VGG-19 at ~144M — ResNet-18 is 12× smaller and better.

model = ResNet18(num_classes=1000)
total = sum(p.numel() for p in model.parameters())
print(f"total params: {total/1e6:.2f}M")   # expect ~11.7M
 
# Break it down
for name, module in model.named_children():
    n = sum(p.numel() for p in module.parameters())
    print(f"{name:10s}  {n/1e6:.3f}M")

4 · The parameter-count table you should remember

Two observations:

1. AlexNet and ResNet-152 have the same param count (~60M). Depth and design bought all the accuracy.
2. ResNet-50 is smaller than VGG-16 and better. That's what the residual trick unlocked.

5 · Why the residual actually helps — a gradient argument

Consider a stack of blocks y = x + F(x). Compute dL/dx:

That leading 1 is the whole point. Even if ∂F/∂x is tiny (vanishing gradient), the derivative through the shortcut is a clean 1 — so gradient signals reach the earliest layers undamped. In a plain deep network without shortcuts, gradients get multiplied by many small numbers and disappear.

War stories

6 · The 2025 pulse · what came after ResNet

ResNet (2015) held the crown for a surprisingly long time — the "ResNet-50 baseline" was still the default vision backbone in most 2020 papers, five years after publication. What actually replaced it? A cascade of architectures each of which added maybe 2-3% ImageNet top-1 while trading off compute, latency, or data-hunger differently.

Here is the compressed post-ResNet family tree you should carry in your head:

2016–2019 · the incremental era.

• DenseNet (Huang et al. 2017) — instead of y = x + F(x), concatenate all previous layer outputs. Fewer parameters, more feature reuse. Popular in medical imaging but died out for general-purpose vision.
• ResNeXt (Xie et al. 2017) — add a "cardinality" axis: run 32 parallel small convs and concatenate. FAIR's answer to "how do I get more accuracy without more depth?" Precursor to grouped convolutions everywhere.
• SENet (Hu et al. 2018) — Squeeze-and-Excitation. Learn per-channel gating: global-avg-pool → tiny MLP → sigmoid → multiply back. Won ILSVRC 2017 — the last human-organized ILSVRC before ImageNet was retired as a benchmark. Every efficient architecture since has a variant of this attention block.
• EfficientNet (Tan & Le 2019) — systematic "compound scaling": jointly scale depth, width, and resolution using a single φ knob. B0-B7 became the reference Pareto frontier for accuracy-vs-FLOPs until ConvNeXt showed up.

2020–2022 · the ViT eclipse (see S030 for the full story).

• ViT and DeiT briefly convinced everyone convolutions were done.
• Swin Transformer (Liu et al. 2021) — hierarchical shifted windows; won ICCV Marr Prize. Made ViT-style architectures work at multiple scales, which is what dense-prediction (segmentation, detection) needs.

2022 · the ConvNeXt counter-attack.

• ConvNeXt (Liu, Mao, Wu, Feichtenhofer et al. 2022) — the paper we covered in S025. Ports the ViT training recipe (AdamW, 300 epochs, MixUp/CutMix/RandAugment/stochastic-depth) onto a modernised ResNet with 7×7 depthwise convs, LayerNorm, GELU, and an inverted bottleneck. 87.8% top-1 at 22M params.
• ConvNeXt V2 (Woo et al. 2023) — adds fully convolutional masked autoencoder pretraining + Global Response Normalization. 88.9% top-1 at 200M params, and unusually well-behaved when scaled up.

2023–2025 · the hybrid consensus.

• MaxViT (Tu et al. 2022) — alternate windowed self-attention with grid self-attention, sandwiched with MBConv blocks. Turns out mixing conv and attention beats either alone at most scales.
• Hiera (Ryali et al., FAIR 2023) — strip a hierarchical ViT (like Swin) of all its bells and whistles, train it with MAE. Faster, simpler, better. "The best hierarchical vision model is a plain one."
• EVA-02 (Fang et al. 2023) and InternImage (Wang et al. 2023) — the current dense-prediction champions on COCO/ADE20K, both hybrid conv-attention.
• MobileNetV4 (Qin, Howard et al., ECCV 2024) — the current on-device Pareto king. 87% top-1 in 3.8 ms on a Pixel 8.

The 2025 mental model. Nobody talks about "the ResNet era" or "the ViT era" anymore. They talk about backbones on a spectrum, and the game is picking one that fits your data scale, compute, and latency target. What all of them share, in modernised form, is exactly the ResNet trinity: conv (or attention) blocks, some normalization, and a residual add. That trinity has now survived a decade and looks nowhere close to being replaced.

Further reading · backbones in 2025

• Liu et al. (2022) — A ConvNet for the 2020s · the paper that saved conv architectures.
• Woo et al. (2023) — ConvNeXt V2.
• Qin et al. (2024) — MobileNetV4.
• Wightman et al. (2021) — ResNet Strikes Back: An Improved Training Procedure in timm · shows a plain ResNet-50 can hit 80.4% top-1 with a modern recipe (vs the original 76.2%). Recipe > architecture.
• Kaiming He's original 2015 ResNet talk (ICML slides) — the "degradation problem" figure is the single most-referenced plot in modern vision.

Retention scaffold

One-line summary (write it in your own words): ______________________________________

Spaced review: re-read section 3 (the BasicBlock) and section 5 (the gradient argument) in 24 hours. On day 7, redraw the ResNet-18 stage table from memory (stem → 4 stages → pool → fc) with correct channels and strides.

Next session (S027): Take the ResNet-18 you just wrote and actually train it. Full CIFAR-10 pipeline in under 20 minutes on a single GPU, targeting 93%+.

Sticky note:

y = x + F(x) — the six characters that made 100-layer networks trainable. Every Transformer block, every LLM, every diffusion U-Net still has this line somewhere inside.

Legacy recall drills

Recall

Stretch

Train LeNet-5 on MNIST and match >99%. Then swap tanh for ReLU, remove the FC layers, add a BasicBlock, and see how few parameters you can use while still hitting 99%. Set a 90-min timer.

In your own words

If a colleague asks "what problem does the residual connection solve?" — answer in 3 sentences.

Spaced review

• S021 (BatchNorm) — every conv in ResNet is followed by BN. Understanding BN vs LayerNorm will come back in S038 (positional encodings) and S036 (attention).
• S025 (convolution) — the shape formula and receptive field logic underpin every architecture we just discussed.

Next session

S027 — Training a real CNN on CIFAR-10 end to end. We take the ResNet-18 we just built, wire up a proper training pipeline (augmentation, LR schedule, weight decay, mixed precision), and target ~93% on CIFAR-10 in under 20 minutes on a single GPU.

Bring back tomorrow

• LeNet template = conv → nonlin → pool, repeat, then flatten → fc → out.
• Residual add: y = x + F(x) — leading 1 in the gradient is why deep nets suddenly worked.
• ResNet stages: double channels, halve resolution.

Previous: ← DL S025 · Next: DL S027 →