CNNs — Convolution, Pooling, ImageNet Architectures

Why this session exists

A CNN trades unrestricted connections for a useful spatial prior: apply the same local transformation at many locations. That can greatly reduce parameters, but neither guarantees better accuracy than an MLP nor makes every vision system convolutional. This lesson derives the operation, output sizes, receptive fields and residual gradients, then builds a CIFAR-shaped classifier with a bounded CPU lab. It also explains how to run the same training function on locally supplied CIFAR-10 without treating synthetic data as a benchmark.

Prerequisites

Use S100 PyTorch, S098 backpropagation and S097 MLPs. Know NCHW tensors, logit cross entropy and validation-only model selection.

(a) Intuition · 5 min

Think of a reusable stencil, not a literal model of how eyes saccade. Place the stencil over each image patch and compute a response. Reusing it encodes the hypothesis that a useful local pattern can occur at many positions. Multiple learned stencils make multiple feature maps; later layers combine them over larger input regions. Edges, textures and object parts are possible learned responses, not prescribed layer numbers or guaranteed meanings.

The three ideas that make CNNs work

Locality: a kernel reads nearby spatial positions, but all connected input channels. Sharing: its coefficients are reused at every output position. Translation equivariance: on an infinite grid, a stride-one convolution shifts its output when its input shifts. This is not translation invariance. Padding boundaries, strides, subsampling and cropping can break exact equivariance. Pooling gives some local tolerance, not a universal guarantee; global averaging is invariant to permutations of its feature-map entries, while the preceding finite-image network may still change under a translated crop.

An MLP can learn image features and fixed-window sequence functions. CNNs impose a narrower function class whose bias often helps with limited data. Fewer weights do not imply equal expressive power. Activation memory still grows with image resolution even though convolutional parameter count does not.

A timeline of the vision revolution

The meaningful progression is hierarchical local features, trainable shared filters, and then improvements in depth, optimization and data. Do not conflate early handwritten-digit convolution work with the later LeNet-5 architecture. The AlexNet paper (2012) describes five convolutional and three fully connected layers, ReLU units, dropout and two-GPU training; it did not invent ReLU or dropout. VGG (2014) systematically studies depth with small filters. ResNet (2015) addresses degradation of training accuracy with residual learning. Vision Transformers later explore a different image prior; no fixed 100-million-image threshold decides which family wins. These papers do not establish that hand-crafted vision disappeared or that every camera uses the same architecture.

The convolution operation

PyTorch 2.9 Conv2d uses cross-correlation: it does not reverse the spatial kernel. For one output channel, sum the products over its connected input channels and kernel positions, then add one bias. A learned kernel could represent the reversed mathematical-convolution kernel, but fixed filters require the convention to be explicit.

At the top-left output, the five selected entries are 1, 1, 1, 0, 1, summing to 4. There are nine legal placements, hence a 3×3 result. The code checks every output entry against an independent patch-sum oracle, not just its shape.

The output shape formula — memorise this

For each spatial axis, with symmetric padding $P$, stride $S$, dilation $D$ and kernel width $K$:

The effective kernel width is $D(K-1)+1$. The floor matters: a 32-wide input with kernel 5, padding 2, stride 2 gives $\lfloor31/2\rfloor+1=16$, not 16.5. A 224 input with kernel 3, padding 1, stride 1 stays 224. PyTorch's string padding='same' in Conv2d supports stride one; do not infer support for strided same-padding from another framework. MaxPool2d's default floor rule has the same geometry, but ceil_mode=True changes boundary handling. Odd sizes do not always halve exactly. Average pooling's padding/divisor policy also matters.

The full CNN pipeline

NCHW input → Conv2d → activation → spatial reduction → another block → classifier logits. Channels need not double and pooling need not follow every block. ReLU is elementwise; pooling acts on spatial windows independently per channel. Max selects the largest value; average includes all values with the chosen divisor. Neither learns kernel parameters. Flatten preserves batch but fixes the classifier's input size; adaptive/global pooling can support variable spatial sizes, subject to minimum sizes and deployment validation.

Anatomy of a conv layer's weight tensor

For groups $G$, the weight shape is $(C_{out},C_{in}/G,K_h,K_w)$ and parameter count is $C_{out}(C_{in}/G)K_hK_w+C_{out}$ with bias. Both channel counts must be divisible by $G$. With RGB input, 64 outputs and 3×3 filters, this is 1,728 weights + 64 biases = 1,792, independent of resolution. A dense layer to only 64 outputs costs 9,633,792 weights; that is a different output geometry. A dense map to 224×224×64 outputs costs 483,385,147,392 weights, plus biases. This is the fair same-output-size comparison, not the earlier erroneous 1.5e11 estimate.

A 1×1 convolution mixes channels at each spatial position; it does not mix neighboring positions by itself. Depthwise convolution uses one group per input channel, often followed by a pointwise 1×1 projection. Sharing constrains spatial behavior, not filter diversity: two learned filters may become redundant.

Landmark architectures — recognise on sight

Inception-v3's paper discusses factorizing larger convolutions, including asymmetric factors. Parameter counts depend on the exact implementation/head; recognizing the computation is more useful than memorizing disputed layer counts or vendor adoption anecdotes.

Why ResNet's skip connection was revolutionary

For $y=x+F(x)$, $J_y=I+J_F$. The direct term can improve optimization without proving gradients never vanish: $J_F=-I$ cancels it. For a post-activation block, the ReLU derivative also multiplies the sum. A projection shortcut gives $J_P+J_F$, not an identity. Different channel counts or resolutions must match before addition. The ResNet paper distinguishes degradation from simply blaming vanishing gradients in already normalized networks. Skip paths permit useful near-identity parameterizations; they are not an invention claim covering every earlier shortcut architecture.

Receptive fields and the small-kernel tradeoff

Let jump $j_0=1$ and receptive width $r_0=1$. For layer $l$:

Three stride-one 3×3 layers have widths 3, 5, 7. For constant channels and spatial resolution, their weights cost $27C^2$, versus $49C^2$ for one 7×7, ignoring bias. Extra nonlinearities prevent collapse into one linear convolution. This does not prove strict dominance: intermediate activations, sequential launches, bandwidth, bottleneck widths and hardware change latency/memory; some large-kernel functions cannot be represented by a particular narrow stack. Pooling/stride increases later jumps, and dilation increases coverage without downsampling, but may produce sparse sampling patterns. The theoretical receptive field is the possible dependency set, not proof each pixel strongly influences the trained output.

Choosing scratch, frozen features or fine-tuning

A frozen pretrained backbone is an inexpensive baseline when its representation is useful. Fine-tuning can adapt it but costs gradients/state and may overfit. Training from scratch gives architecture/channel freedom but still has data-license and provenance responsibilities. A failed linear probe is evidence about that probe, preprocessing and representation—not proof fine-tuning cannot work. Choose by target validation, domain, license and compute, not a universal image-count cutoff. S105 implements the state-management details.

(c) Hands-on · 25 min

This complete CPU program restores the three-block, six-convolution CIFAR-shaped architecture. The original docstring incorrectly called it four blocks; its parameter total was also understated. The default synthetic fixtures exercise forward/backward, validation checkpointing, normalization and residual dimensions. They are not CIFAR training, natural-image generalization or a runtime benchmark.

import copy
import torch
from torch import nn
from torch.nn import functional as F
from torch.utils.data import TensorDataset, DataLoader
 
torch.set_num_threads(1)
torch.manual_seed(7)
class SmallCNN(nn.Module):
    def __init__(self, classes=10, residual=False):
        super().__init__()
        self.blocks=nn.ModuleList()
        for cin,cout in [(3,64),(64,128),(128,256)]:
            self.blocks.append(nn.Sequential(nn.Conv2d(cin,cout,3,padding=1),nn.ReLU(),
                nn.Conv2d(cout,cout,3,padding=1),nn.ReLU(),nn.MaxPool2d(2)))
        self.project=nn.Conv2d(64,128,1) if residual else None
        self.fc1=nn.Linear(256*4*4,512);self.fc2=nn.Linear(512,classes)
        self.drop=nn.Dropout(.3)
        self.register_buffer('mean',torch.zeros(1,3,1,1))
        self.register_buffer('std',torch.ones(1,3,1,1))
    def forward(self,x):
        if x.ndim!=4 or tuple(x.shape[1:])!=(3,32,32):
            raise ValueError('expected N,3,32,32')
        x=(x-self.mean)/self.std
        x=self.blocks[0](x);old=x;x=self.blocks[1](x)
        if self.project is not None:x=x+F.avg_pool2d(self.project(old),2)
        x=self.blocks[2](x)
        return self.fc2(self.drop(F.relu(self.fc1(x.flatten(1)))))
 
def run(m,loader,opt=None):
    m.train(opt is not None);total=correct=n=0
    with torch.set_grad_enabled(opt is not None):
        for x,y in loader:
            if opt is not None:opt.zero_grad(set_to_none=True)
            z=m(x);loss=F.cross_entropy(z,y)
            if opt is not None:loss.backward();opt.step()
            total+=loss.item()*len(y);correct+=(z.argmax(1)==y).sum().item();n+=len(y)
    if not n:raise ValueError('empty loader')
    return total/n,correct/n
 
def fit(train_x,train_y,val_x,val_y,test_x,test_y,epochs=2,residual=False):
    m=SmallCNN(residual=residual)
    with torch.no_grad():
        m.mean.copy_(train_x.mean((0,2,3),keepdim=True))
        m.std.copy_(train_x.std((0,2,3),unbiased=False,keepdim=True).clamp_min(1e-5))
    tr=DataLoader(TensorDataset(train_x,train_y),batch_size=4,shuffle=True,
        generator=torch.Generator().manual_seed(4),num_workers=0)
    va=DataLoader(TensorDataset(val_x,val_y),batch_size=4)
    te=DataLoader(TensorDataset(test_x,test_y),batch_size=4)
    opt=torch.optim.Adam(m.parameters(),lr=1e-3)
    best=float('inf');state=None
    for epoch in range(epochs):
        train=run(m,tr,opt);valid=run(m,va)
        if valid[0]<best:best=valid[0];state=copy.deepcopy(m.state_dict())
        print(epoch,train,valid)
    m.load_state_dict(state);print('fixture test only',run(m,te))
    return m
 
image=torch.tensor([[1,1,1,0,0],[0,1,1,1,0],[0,0,1,1,1],
                    [0,0,1,1,0],[0,1,1,0,0]],dtype=torch.float64)
kernel=torch.tensor([[1,0,1],[0,1,0],[1,0,1]],dtype=torch.float64)
actual=F.conv2d(image[None,None],kernel[None,None])[0,0]
oracle=torch.stack([torch.stack([(image[i:i+3,j:j+3]*kernel).sum()
    for j in range(3)]) for i in range(3)])
assert torch.equal(actual,oracle)
assert actual.tolist()==[[4.,3.,4.],[2.,4.,3.],[2.,3.,4.]]
# Asymmetric kernel detects an accidental mathematical-convolution flip.
asym=torch.arange(9,dtype=torch.float64).reshape(3,3)
assert not torch.equal(F.conv2d(image[None,None],asym[None,None]),
                       F.conv2d(image[None,None],asym.flip((0,1))[None,None]))
assert nn.Conv2d(64,128,5,2,2)(torch.zeros(1,64,32,32)).shape==(1,128,16,16)
x=torch.rand(20,3,32,32);y=torch.arange(20)%10
m=fit(x[:12],y[:12],x[12:16],y[12:16],x[16:],y[16:])
assert sum(p.numel() for p in m.parameters())==3248202
r=SmallCNN(residual=True);assert r(x[:2]).shape==(2,10)
try:m(torch.zeros(1,3,28,28));raise AssertionError('shape accepted')
except ValueError:pass
print('cross-correlation, shape, parameter, residual and training contracts passed')

Anatomy of the script

Conv2d(3,64,3,padding=1) preserves 32×32, and the three pools produce 16, 8, 4. flatten(1) gives 4,096 features. The classifier dominates the count: 4,096×512 + 512 weights/biases in fc1; the exact full model has 3,248,202 parameters. The residual practice projects 64 to 128 channels and average-pools to 8×8 before adding; no accuracy gain is presumed. Loss is weighted by actual batch size, and validation selects a cloned state before one test evaluation. Normalization buffers use training data only.

For locally supplied official CIFAR-10 arrays, retain the official 50,000-training/10,000-test roles; split validation from training (stratify if appropriate), convert NHWC uint8 to NCHW float in [0,1], and call fit with those disjoint arrays. Do not use test accuracy to select dropout, epochs or residuals. A torchvision CIFAR10(..., download=False) adapter may read already provisioned data; this lesson does not download it. Training-only padded crops and horizontal flips are optional label-preserving augmentations for suitable images; deterministic validation gets no random augmentation. They encourage robustness rather than prove invariance. Compare Adam and scheduled momentum SGD with a controlled search budget, not a promised one-point improvement.

(d) Production reality · 15 min

Multi-camera fusion: preserve camera calibration, timestamp alignment and missing-camera policy. Shared per-camera weights do not themselves produce a bird's-eye-view; geometric projection or a learned cross-view mechanism is additional computation. Compare early/late fusion under occlusions and synchronized splits. No Tesla recall result is asserted.

Weak labels: hashtag supervision can encode language/popularity shortcuts. Define a visual vocabulary, measure label noise, choose weighting/sampling on validation, and report clean downstream transfer separately from weak-label fitting. Neither billions of images nor a normalization switch automatically solves noise; no invented Instagram collapse/recovery is evidence.

Clinical domain shift: site, camera, capture quality and workflow change together. Evaluate patient/site-held-out data, calibration, ungradable-image handling and manual-review workload. An OOD detector is not a universally reliable safety gate, and saliency does not certify causal features. These are explicit engineering scenarios, not the original unsupported clinic-count and accuracy anecdotes.

Where this shows up in the rest of the plan

Spatial sharing leads to temporal sharing; pretrained image features lead to transfer. CLIP can use a CNN or a vision transformer as its image encoder. Diffusion systems can use convolutional U-Nets with skips, but not every diffusion model does. Serving requires separately measured export/kernel compatibility and latency; no automatic 10× speedup follows from a CNN definition.

(e) Recall + stretch · 10 min

1. CNN versus dense pixels? Locality and shared weights reduce parameters under a spatial prior, not equal expressive power or guaranteed invariance.
2. 32 input, 5 kernel, 2 padding, 2 stride? Floor gives 16; output is (B,128,16,16) for 128 filters.
3. Max or average pooling? Max routes gradient to selected maxima (ties need a convention); average distributes it by the divisor. Test which loses important information.
4. Why residuals? A direct Jacobian term supports optimization; post-activation/projection/cancellation prevent an unconditional gradient guarantee.
5. Three 3×3 versus one 7×7? Same theoretical field at stride one, 27 versus 49 channel-squared weights, but different nonlinearities, activation memory and hardware latency.
6. Receptive field? Use the jump/field recurrence above; dilation and later downsampling change the increment.
7. Normalization? Match input scale and the particular pretrained-weight contract, or fit training-only statistics for scratch. Not all models use one ImageNet pipeline.
8. Memory pressure on 512×512 scans? Try smaller batches/crops, supported mixed precision, checkpointing or a smaller model. These alter compute/numerics; accumulation does not reproduce BatchNorm of a large batch. No fixed 2× saving or unchanged accuracy is guaranteed.
9. 1×1 conv? A learned channel projection at each position; useful for bottlenecks and compatible shortcuts.
10. 95% train, 72% held-out? First check duplicates, preprocessing, metric/split comparability and domain shift; then compare augmentation, decay, capacity and transfer on validation. Do not repeatedly tune on the test set.

Explain-out-loud test

Derive a grouped-convolution parameter count, trace the receptive field through a strided block, and differentiate both an identity and projected residual. Explain why the CPU run establishes implementation contracts but says nothing about CIFAR accuracy.

What comes next

RNNs and LSTMs reuse a transformation across time rather than space. Learning-plan hub.

Primary references

• PyTorch 2.9 Conv2d: cross-correlation, groups and floor/dilation geometry.
• VGG, ResNet, and Inception-v3: architectural mechanisms, not universal speed/quality claims.