S102 · CNNs — Convolution, Pooling, ImageNet Architectures

Module M12: Deep Learning · Session 102 of 130 · Track: ML 🖼️

What you'll be able to do after this session

• Explain CNNs to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S100 — PyTorch Fundamentals — Tensors, Autograd, nn.Module

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · But what is a convolution? — 3Blue1Brown — The math of convolution animated end-to-end.
• 🎥 Deep dive (30–60 min) · Stanford CS231n — Convolutional Neural Networks — Andrej Karpathy's classic CNN lecture.
• 🎥 Hands-on demo (10–20 min) · Build a CNN in PyTorch — Aladdin Persson — Types out a CNN for CIFAR-10 live.
• 📖 Canonical article · CS231n — Convolutional Networks notes — The single best CNN explainer online.
• 📖 Book chapter / tutorial · PyTorch CIFAR-10 tutorial — Official walkthrough of a working CNN.
• 📖 Engineering blog · Google Research — Advancing instance-level recognition — How production CNNs are pushed at Google scale.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Vision, cracked.

Picture how you actually look at a photograph. Your eyes don't process every pixel at once with equal attention — they scan in patches, first noticing edges and corners, then combining those into shapes (an eye, a wheel), then combining shapes into things (a face, a car). A Convolutional Neural Network does exactly this in code. Instead of one giant matrix multiplication that touches every pixel, a CNN slides a tiny filter (e.g. 3×3) across the image, detecting local patterns, then stacks layers that build progressively more abstract features on top.

The big win over a plain fully-connected net is parameter sharing. A fully-connected layer on a 224×224 RGB image would need 224·224·3 = 150,528 weights per output neuron. A convolutional filter uses the same 3·3·3 = 27 weights everywhere on the image, because "edge-ness" looks the same in the top-left as in the bottom-right. This is called translation invariance and it's why CNNs took over vision in 2012 with AlexNet.

Gotcha to carry forever: The number of output channels of one conv layer must equal the number of input channels of the next. Almost every "shape mismatch" error you'll ever see in a CNN traces back to this one line.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

A CNN is a stack of the same three operations repeated: convolve → activate → downsample.

Worked example — output shape of a conv layer. The formula every CNN engineer memorises: out = floor((in + 2·padding − kernel) / stride) + 1. Given input H = 224, kernel = 3, padding = 1, stride = 1: (224 + 2 − 3)/1 + 1 = 224. Same spatial size — "same padding". If instead stride = 2: (224 + 2 − 3)/2 + 1 = 112. Half the spatial size. That's how you downsample.

Worked example — parameter count. A Conv2d(64, 128, kernel_size=3) layer has 64 · 128 · 3 · 3 = 73,728 weights + 128 biases = 73,856 parameters. Compare to a fully-connected layer mapping 64·56·56 inputs to 128·56·56 outputs: (64·56·56) · (128·56·56) ≈ 8·10¹¹ weights. A 10-million-fold reduction. This is why CNNs even fit on a GPU.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# cifar_cnn.py — a real CNN for CIFAR-10, ~78% test accuracy in 10 epochs
import torch, torch.nn as nn, torch.nn.functional as F
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
 
device = "cuda" if torch.cuda.is_available() else "cpu"
tfm = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465),
                         (0.2470, 0.2435, 0.2616)),
])
train = DataLoader(datasets.CIFAR10("./data", train=True, download=True,
                                    transform=tfm), batch_size=128, shuffle=True)
test  = DataLoader(datasets.CIFAR10("./data", train=False, transform=tfm),
                   batch_size=256)
 
class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.b1 = nn.Sequential(nn.Conv2d(3,   64, 3, padding=1), nn.BatchNorm2d(64),  nn.ReLU(),
                                nn.Conv2d(64,  64, 3, padding=1), nn.BatchNorm2d(64),  nn.ReLU(),
                                nn.MaxPool2d(2))
        self.b2 = nn.Sequential(nn.Conv2d(64, 128, 3, padding=1), nn.BatchNorm2d(128), nn.ReLU(),
                                nn.Conv2d(128,128, 3, padding=1), nn.BatchNorm2d(128), nn.ReLU(),
                                nn.MaxPool2d(2))
        self.b3 = nn.Sequential(nn.Conv2d(128,256, 3, padding=1), nn.BatchNorm2d(256), nn.ReLU(),
                                nn.AdaptiveAvgPool2d(1))
        self.fc = nn.Linear(256, 10)
    def forward(self, x):
        return self.fc(self.b3(self.b2(self.b1(x))).flatten(1))
 
model = SmallCNN().to(device)
opt = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9, weight_decay=5e-4)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=10)
 
for epoch in range(10):
    model.train()
    for x, y in train:
        x, y = x.to(device), y.to(device)
        opt.zero_grad(); F.cross_entropy(model(x), y).backward(); opt.step()
    sched.step()
    model.eval(); correct = 0
    with torch.no_grad():
        for x, y in test:
            x, y = x.to(device), y.to(device)
            correct += (model(x).argmax(1) == y).sum().item()
    print(f"epoch {epoch}: acc={correct/len(test.dataset):.4f}")

Checklist while it runs:

1. Count total params via sum(p.numel() for p in model.parameters()) — should be ~1.1M.
2. Watch test accuracy climb past 70% by epoch 5.
3. Comment out the augmentations — accuracy drops 3–5%.
4. Change AdaptiveAvgPool2d(1) to Flatten() + Linear(256*8*8, 10) — param count explodes.
5. Add print(x.shape) between blocks to see the spatial dims shrink 32 → 16 → 8 → 1.

Try this modification: Add a residual skip connection inside b2. You've just built a mini-ResNet — accuracy should tick up another 1–2%.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Real vision systems live and die by three things: data augmentation, transfer learning, and inference latency.

Nobody trains from scratch anymore. For any real product classification task, you start from ImageNet-pretrained weights (torchvision.models.resnet50(weights="DEFAULT")) and fine-tune. The bottom layers already know edges, textures, and object parts — retraining them wastes weeks of GPU and gives worse results. This is the transfer-learning story we do in depth in S105.

Augmentation is the real regularizer for vision. Random crops, flips, colour jitter, Mixup, CutMix, RandAugment — the modern recipe is "throw everything at the training loader". Torchvision's Improved ResNet-50 recipe shows that swapping in aggressive augmentation moves a stock ResNet-50 from 76% to 80%+ ImageNet top-1 with no architecture change.

Inference latency is where research meets reality. A ResNet-50 is 4 GFLOPs — fine on a V100 GPU, painful on a phone. Google's MobileNet and EfficientNet families use depthwise-separable convolutions to cut FLOPs 10× at 1–2% accuracy cost. For real-time video (self-driving, AR filters), you'll also see NCHW → NHWC layout tuning, TensorRT / ONNX Runtime kernel fusion, and INT8 post-training quantisation.

Common bugs: forgetting to normalise inputs to the mean/std the pretrained model expects; confusing nn.CrossEntropyLoss (wants raw logits) with nn.NLLLoss (wants log-softmax); leaving BatchNorm in .train() mode when fine-tuning on a small dataset — corrupts running stats. Google's Advancing Instance-Level Recognition blog details how Landmarks-v2 fixed exactly these issues to hit production-grade retrieval accuracy.

(e) Quiz + exercise · 10 min

1. Write the formula for the output spatial size of a Conv2d given input, kernel, padding, and stride.
2. Why does parameter sharing in a conv layer produce translation invariance?
3. What does AdaptiveAvgPool2d(1) do to a [B, 256, 8, 8] tensor, and why is it useful before a classifier head?
4. A Conv2d(128, 256, kernel_size=3, padding=1) on a [B, 128, 56, 56] input produces what output shape and how many parameters?
5. Name one architectural change AlexNet introduced that is still standard today.

Stretch (connects to S101 — Regularization): In modern CNNs, why has BatchNorm mostly replaced Dropout inside conv blocks, and when would you still add dropout back?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is CNNs? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S103): RNNs & LSTMs — Sequences & the Vanishing Gradient
• Previous (S101): Regularization in DL — Dropout, BatchNorm, Weight Decay
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.