CNN architectures: LeNet, AlexNet, residual learning and modern backbones

Three landmark designs make a useful progression: LeNet learns local shared features, AlexNet combines a much larger vision model with a particular training system, and ResNet reparameterizes blocks as corrections to a shortcut. None licenses the claim that all vision history came from three papers or that depth became risk-free.

Historical results need a comparison contract

LeCun et al.'s1998 document-recognition paper describes LeNet-5 and practical recognition systems. LeNet did not sit unused until2012. Its original design used learned subsampling, a partly connected C3 layer and a specialized output formulation; the fully connected two-convolution tanh network below is a modern LeNet-like teaching variant, not a parameter-exact museum reproduction.

The AlexNet2012 paper describes five convolutional and three fully connected layers, about60M parameters and training on two GTX580 GPUs. Its competition entry's15.3% top-five error is compared with26.2% for the runner-up in that competition:10.9 percentage points. Do not substitute a2011 score and call it the same comparison, equate error with accuracy, or attribute a whole-system result to ReLU alone. The original lesson's detailed late-night narrative and speculative investment story are not established by the paper.

ResNet's central experiment is degradation: a deeper plain network can have higher training error than its shallower counterpart, despite additional representational capacity. This differs from overfitting, which requires considering held-out performance, and from diagnosing small gradient norms. Initialization and normalization help gradient propagation but do not prove an optimizer can find every representable identity map.

Outcomes and prerequisites. Use convolution shape arithmetic and train/eval modes. Build the LeNet-like classifier, a projected BasicBlock and ResNet18; restore a complete local MNIST recipe; count parameters by stage; and distinguish optimization evidence from universal architectural claims.

LeNet-like network: shapes, parameters and logits

For padded32×32 grayscale input the spatial route is32→28→14→10→5. Six channels follow the first convolution,16 the second, then flatten to400. Tanh activations precede the two average pools. The classifier is400→120→84→10. Return raw logits for cross-entropy; adding softmax before CrossEntropyLoss changes the objective.

import torch
import torch.nn as nn
import torch.nn.functional as F
 
class LeNet5(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, kernel_size=5)      # 32 → 28
        self.pool  = nn.AvgPool2d(2, 2)                  # halve
        self.conv2 = nn.Conv2d(6, 16, kernel_size=5)     # 14 → 10
        self.fc1   = nn.Linear(16 * 5 * 5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, num_classes)
 
    def forward(self, x):
        x = self.pool(torch.tanh(self.conv1(x)))         # (N, 6, 14, 14)
        x = self.pool(torch.tanh(self.conv2(x)))         # (N, 16, 5, 5)
        x = x.flatten(1)                                 # (N, 400)
        x = torch.tanh(self.fc1(x))
        x = torch.tanh(self.fc2(x))
        return self.fc3(x)                               # raw logits

The parameter counts are156,2,416,48,120,10,164 and850, totaling 61,706. The three linear layers hold59,134, or 95.83%, not80%. Counting MACs is a different calculation: conv1 uses117,600 and conv2 uses240,000 per image, versus58,920 for the dense layers; the original99%-of-computation statement was also wrong. Bias, pooling and activation operations are additional to these MACs.

Complete local MNIST experiment

The recipe below is an opt-in function, not an automatic download. It requires a separately installed compatible torchvision0.24 and already present, trusted MNIST data. Validation comes from training data; the official test split is evaluated only after selection. Five epochs and Adam1e−3 are an experiment specification, not a99.1%-in-sixty-seconds guarantee. The CPU audit tests the exact network and train/eval helpers on synthetic images, not this missing local dataset.

from copy import deepcopy
from torch.utils.data import DataLoader, TensorDataset, Subset
 
def epoch_metrics(model, loader, optimizer=None):
    training = optimizer is not None
    model.train(training)
    loss_sum = correct = count = 0
    for xb,yb in loader:
        with torch.set_grad_enabled(training):
            logits = model(xb)
            loss = F.cross_entropy(logits,yb)
            if training:
                optimizer.zero_grad(set_to_none=True)
                loss.backward(); optimizer.step()
        loss_sum += loss.item()*len(yb)
        correct += (logits.argmax(1)==yb).sum().item(); count += len(yb)
    if count == 0:
        raise ValueError('empty loader')
    return {'loss':loss_sum/count,'accuracy':correct/count,'count':count}
 
def local_mnist(root, epochs=5):
    from torchvision import datasets, transforms
    tfm = transforms.Compose([transforms.Pad(2),transforms.ToTensor(),
                              transforms.Normalize((0.1307,),(0.3081,))])
    ds = datasets.MNIST(root,train=True,download=False,transform=tfm)
    test = datasets.MNIST(root,train=False,download=False,transform=tfm)
    perm = torch.randperm(len(ds),generator=torch.Generator().manual_seed(26))
    train_ids,val_ids = perm[5000:].tolist(),perm[:5000].tolist()
    train_dl = DataLoader(Subset(ds,train_ids),batch_size=128,shuffle=True,num_workers=0)
    val_dl = DataLoader(Subset(ds,val_ids),batch_size=512,num_workers=0)
    test_dl = DataLoader(test,batch_size=512,num_workers=0)
    torch.manual_seed(26); model = LeNet5()
    opt = torch.optim.Adam(model.parameters(),lr=1e-3)
    best = float('inf'); state = None; history=[]
    for epoch in range(epochs):
        tr=epoch_metrics(model,train_dl,opt); va=epoch_metrics(model,val_dl)
        history.append({'epoch':epoch,'train':tr,'validation':va})
        if va['loss'] < best:
            best=va['loss'];state=deepcopy(model.state_dict())
    if state is None:
        raise ValueError('need at least one epoch')
    model.load_state_dict(state)
    return model,history,epoch_metrics(model,test_dl)
 
# Bounded model/helper check, not MNIST accuracy.
torch.manual_seed(26)
small = DataLoader(TensorDataset(torch.randn(6,1,32,32),torch.arange(6)%10),batch_size=4)
m = LeNet5(); assert sum(p.numel() for p in m.parameters()) == 61706
before = m.fc3.weight.detach().clone()
tr = epoch_metrics(m,small,torch.optim.Adam(m.parameters(),lr=1e-3))
va = epoch_metrics(m,small)
assert tr['count']==va['count']==6 and not torch.equal(before,m.fc3.weight)
assert 0 <= va['accuracy'] <= 1 and torch.isfinite(torch.tensor(va['loss']))
print('LeNet parameters',sum(p.numel() for p in m.parameters()),'synthetic loss',va['loss'])

The familiar MNIST normalization constants above are declared reference preprocessing, not newly estimated validation statistics. For a strict independent protocol compute any learned preprocessing using train indices only. To compare tanh and ReLU, or remove the dense tail and insert a residual block, hold split, seed, budget and selection metric constant; log the new parameter count and achieved score rather than inventing a guaranteed99% endpoint.

AlexNet: what actually changed

The paper's useful mechanisms are more specific than “just make it bigger”:

1. Rectified units: positive inputs have derivative1, avoiding tanh's positive saturation. Products of weights, dead units and negative inputs still permit vanishing gradients; ReLU does not guarantee a healthy deep Jacobian.
2. Dropout in early fully connected layers: stochastic unit omission during training regularizes co-adaptation. Evaluation disables dropout. A reported ablation is not a fixed reward on every dataset.
3. **Overlapping max pooling:**3×3 windows at stride2 share input locations. This differs from nonoverlapping2×2 pooling and from a learned strided convolution.
4. Two-GPU placement: some layers connect only to channels on the same device, while others connect across devices. This is a systems constraint reflected in connectivity, not evidence that GPUs were invented for this experiment.
5. Data augmentation and normalization: image crops/reflections and RGB intensity perturbations are part of the reported recipe; local-response normalization is historically distinct from later BatchNorm.

The paper states224×224 input while common reproductions use227 or suitable padding to obtain55 after the first11×11,stride4 convolution. State which implementation you mean rather than silently mixing diagrams and library code. A common original grouped-channel parameter calculation has2,334,080 conv parameters and58,631,144 classifier parameters, totaling60,965,224. The first dense9216→4096 map dominates, not the4096→4096 map. Current torchvision's AlexNet is a particular variant with a different width/connectivity contract; do not equate every model called AlexNet byte-for-byte.

Residual learning and the degradation problem

Write the desired block function as H(x). Instead of learning H directly, learn $F(x)=H(x)-x$ and output $x+F(x)$. When a useful block is close to identity, a small residual can be an easier target. This is a reparameterization with empirical optimization benefits, not proof that more depth can never hurt.

For a pure additive block the Jacobian is $I+J_F$. Taking $F(x)=-x$ makes it zero: identity contribution does not guarantee nonvanishing gradients. A post-activation BasicBlock outputs ReLU(x+F(x)), so its Jacobian includes the final ReLU mask. A zero residual branch yields ReLU(x), not identity on negative x. A projected shortcut is another learned linear map, not I.

BasicBlock with an explicit projection

class BasicBlock(nn.Module):
    expansion = 1
 
    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 3,
                               stride=stride, padding=1, bias=False)
        self.bn1   = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3,
                               stride=1, padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_channels)
 
        # If shape changes (channel count or stride > 1), project the shortcut.
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, 1,
                          stride=stride, bias=False),
                nn.BatchNorm2d(out_channels),
            )
        else:
            self.shortcut = nn.Identity()
 
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + self.shortcut(x)      # the residual add
        return F.relu(out)

Convolutions followed directly by train-mode BatchNorm commonly omit biases: normalization removes a spatially constant channel shift, while BN has an affine shift. Eval uses stored statistics, so this is a design rationale, not license to delete a bias from an arbitrary already-trained network without folding its effect.

Projection is needed when the branches have incompatible channels or spatial sizes. Use a1×1 convolution with the same transition stride and normalization. Do not add one in every block without a reason. In a bottleneck block,1×1 reduces channels,3×3 processes the narrow representation, and1×1 expands (often by4); if the intermediate width is b, the spatial kernel costs9b² instead of9(4b)². Projection and expansion counts must still be included.

ResNet18 stages and the global-average head

class ResNet18(nn.Module):
    def __init__(self, num_classes=1000):
        super().__init__()
        self.stem = nn.Sequential(
            nn.Conv2d(3, 64, 7, stride=2, padding=3, bias=False),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
            nn.MaxPool2d(3, stride=2, padding=1),
        )
        self.layer1 = self._make_layer(64,  64,  2, stride=1)
        self.layer2 = self._make_layer(64,  128, 2, stride=2)
        self.layer3 = self._make_layer(128, 256, 2, stride=2)
        self.layer4 = self._make_layer(256, 512, 2, stride=2)
        self.avgpool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Linear(512, num_classes)
 
    def _make_layer(self, in_c, out_c, n_blocks, stride):
        layers = [BasicBlock(in_c, out_c, stride=stride)]
        for _ in range(n_blocks - 1):
            layers.append(BasicBlock(out_c, out_c, stride=1))
        return nn.Sequential(*layers)
 
    def forward(self, x):
        x = self.stem(x)                         # (N, 64, 56, 56)
        x = self.layer1(x); x = self.layer2(x)
        x = self.layer3(x); x = self.layer4(x)   # (N, 512, 7, 7)
        x = self.avgpool(x).flatten(1)           # (N, 512)
        return self.fc(x)

At224 input, stem gives64×56×56 after conv/pool; stages produce64×56×56,128×28×28,256×14×14,512×7×7. Doubling channels while halving both spatial axes halves the activation element count, not preserves it. Adaptive pooling averages each final channel to one scalar before the512→classes head.

model = ResNet18(num_classes=1000)
total = sum(p.numel() for p in model.parameters())
print(f"total params: {total/1e6:.2f}M")   # expect ~11.7M
 
# Break it down
for name, module in model.named_children():
    n = sum(p.numel() for p in module.parameters())
    print(f"{name:10s}  {n/1e6:.3f}M")

The exact1000-class total is11,689,512. Stage counts are: stem9,536; layer1 147,968; layer2 525,568; layer3 2,099,712; layer4 8,393,728; fc513,000. Replacing pooling plus the512→1000 head by flatten plus25088→1000 adds24,576,000 parameters, yielding36,265,512. Large image size raises activations and compute even when global pooling keeps parameter count unchanged.

Direct tests of shortcut and mode semantics

torch.manual_seed(26)
block = BasicBlock(3,5,stride=2).eval()
x = torch.randn(2,3,8,8,requires_grad=True)
y = block(x)
assert y.shape == (2,5,4,4)
y.square().mean().backward(); assert torch.isfinite(x.grad).all()
# Pure additive zero branch is identity; post-ReLU identity is conditional.
z = torch.tensor([-2.,3.],requires_grad=True)
assert torch.equal(z + torch.zeros_like(z),z)
assert not torch.equal(F.relu(z),z)
(z-z).sum().backward(); assert torch.equal(z.grad,torch.zeros_like(z))
# Fifty pure additive zero branches are identity; this does not include post-ReLU.
branches=nn.ModuleList([nn.Linear(2,2) for _ in range(50)])
for branch in branches:
    nn.init.zeros_(branch.weight); nn.init.zeros_(branch.bias)
carried=z.detach().clone()
for branch in branches: carried=carried+branch(carried)
assert torch.equal(carried,z.detach())
# Freezing affine parameters alone does not freeze BN buffers.
bn = nn.BatchNorm2d(3)
for p in bn.parameters(): p.requires_grad_(False)
before = bn.running_mean.clone(); bn(torch.ones(2,3,4,4)*5)
assert not torch.equal(before,bn.running_mean)
bn.eval(); before = bn.running_mean.clone(); bn(torch.ones(2,3,4,4)*9)
assert torch.equal(before,bn.running_mean)
res = ResNet18().eval()
assert sum(p.numel() for p in res.parameters()) == 11689512
with torch.no_grad(): assert res(torch.randn(1,3,32,32)).shape == (1,1000)
print('projection, cancellation, BN and ResNet counts: pass')

For50 pure additive linear branches, zeroing each branch's final map makes the stack identity. The conclusion changes with post-ReLU, projection or normalization outside the branch. BN train mode uses N×H×W values per channel; a batch of one is not automatically all zeros when spatial dimensions remain. Evaluation mode and gradient disabling are separate controls.

After ResNet: what each family changes

A family tree is useful when it explains mechanisms rather than declaring a dated winner.

• VGG: stacks small3×3 filters with nonlinearities, trading depth against large kernels; its dense tail is parameter-heavy. ResNet18 can be smaller without being more accurate than every VGG checkpoint.
• DenseNet: concatenates earlier features into subsequent layers. Reuse can reduce parameters but increases feature storage and bandwidth. “Died out” is not an architectural property.
• ResNeXt: aggregates parallel transformations, commonly implemented with grouped convolutions and a sum of transformed branches; the original description as simply concatenating32 convs was wrong. Cardinality is distinct from depth and width.
• SENet: global average pooling→small channel MLP→sigmoid gates→channelwise multiplication recalibrates features. ImageNet did not stop existing as a benchmark after its2017 competition.
• EfficientNet: compound scaling allocates depth,width,resolution with an explicit resource rule; EfficientNetV2 adds training-oriented operator/search changes and progressive learning.
• DeiT: stronger augmentation and teacher distillation make ViT training on ImageNet-only data viable; Swin uses shifted local windows and hierarchy. These contradict a hard “ViT always needs100M images” threshold.
• ConvNeXt: modernizes a convnet and its recipe; ConvNeXtV2 adds masked-autoencoder co-design and GRN. Neither87.8% at22M nor88.9% at200M should be repeated without identifying the actual configuration.
• MaxViT: combines convolutional blocks, window attention and grid attention. Hiera: simplified hierarchical transformer trained with masked pretraining; not every hierarchical transformer needs Swin's full set of components.
• EVA-02: masked visual representation learning with strong vision-language teacher targets. InternImage: builds around deformable convolution, not a generic conv-attention hybrid. Their model type is more useful than a claim to be permanent COCO/ADE20K champion.
• MobileNetV4: configurable inverted bottlenecks and mobile multi-query attention in hybrid models. Export support and latency require testing the chosen variant, device and precision.

ResNet Strikes Back studies improved recipes and reports80.4% top1 for ResNet50 under its strongest recipe. This does not prove every preceding architecture gain was illusory or identify the same metric/protocol as every76.2% baseline. Read data, epochs, resolution, optimizer, augmentation and inference rules before subtracting scores.

Choosing a backbone and designing a fair comparison

Start with a tractable reference such as ResNet18 to validate labels, splits and metrics. A modern CNN/hybrid may improve deployment efficiency but needs operator/export testing. A pretrained ViT may offer stronger transfer features but carries checkpoint, preprocessing, license and resolution costs. No option wins merely by age or parameter count.

For a reproducible parameter table, distinguish the teaching LeNet-like variant from pinned torchvision0.24 models. These are parameter inventories, not a common-training accuracy comparison:

The old table mixed single-model,ensemble and different-training top-five results. Read a named weight enum's metric,recipe and evaluation transform before making an accuracy comparison; MNIST accuracy does not belong in an ImageNet top-five column.

Compare at two levels: (1) matched training recipe and data to isolate architecture, and (2) best tuned recipe per architecture under a matched tuning budget to compare practical systems. Report parameter counts, MAC convention, peak memory, measured latency and held-out quality separately. Reusing a weight at many pixels increases operations per parameter; it is not proof of fewer FLOPs or faster inference.

Worked answers and retention

1. **AlexNet result:**15.3% versus26.2% top-five error in the paper's2012 competition comparison, a10.9-point gap. A comparison with2011 is a different historical question, not a controlled ablation.
2. BasicBlock from memory: conv→BN→ReLU→conv→BN; match the shortcut's shape if needed; add; final ReLU. The addition precedes that final activation.
3. Two depth failures: small early gradient norms suggest attenuated credit; higher training error in a deeper plain net is degradation. A loss curve alone does not prove one or exclude the other; inspect gradients, optimization settings and a shallower control. Vanishing does not inherently imply later divergence.
4. Where parameters live: this LeNet-like net95.83% in linear layers; common AlexNet about96% in its dense tail. ResNet50's2048→1000 head alone has2,049,000 parameters, roughly8% of25.6M, so “99% convolutions” is false. Weight reuse and hardware decide compute cost separately.
5. Projection rule: mismatch of channel or spatial shape requires compatible mapping, not necessarily stride2 in every case. A channel-only change uses stride1.
6. What does improved ResNet training show? Recipe is a confounder that must be controlled or reported. It does not show depth cannot improve representation, or that all new architecture results vanish under retuning.
7. Why no conv bias before BN? Batch centering cancels it; BN supplies its own shift. Why global pooling? It replaces a large spatially flattened linear map by per-channel averages, sacrificing explicit spatial arrangement at the head.
8. Three-sentence explanation: residual learning supplies a shortcut and asks the branch to add a correction. This can make identity-like solutions easier to optimize. It neither guarantees gradients nor makes arbitrary depth free.

For the original stretch, run the local MNIST function, then compare activation swaps and a pooling head at the same budget. Write achieved accuracy and counts;99% is a target to measure, not a check already passed here. Tomorrow redraw the stage shapes and reconstruct the projected BasicBlock; next week reproduce the cancellation counterexample.

Sources and execution boundary

Primary sources: LeCun1998, AlexNet2012, ResNet, VGG, torchvision0.24 ResNet source, BatchNorm2.9, DenseNet, ResNeXt, SENet, EfficientNet, MaxViT, Hiera, EVA02, InternImage.

Displayed model/helper tests run on synthetic CPU tensors. Local MNIST, pretrained weights, ImageNet comparisons and timing are not executed. Previous: convolution · Next: CIFAR training.