DL S027 · Training a Real CNN on CIFAR-10, End to End

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 27 / 80 · Module M05

The story

Story hook · the recipe war of 2018–2022

Here is a fact that should make you slightly angry, or possibly relieved: the ResNet-50 in the original 2015 paper reached 76.2% top-1 on ImageNet. The same architecture, unmodified, retrained in 2021 with a modern recipe, reaches 80.4%.

That's not a typo. Four full percentage points — more than most "new architecture" papers of the intervening decade claimed — came from doing nothing more than swapping the training recipe. Ross Wightman documented this in "ResNet Strikes Back" (Wightman, Touvron, Jégou 2021), and the ML community collectively had a very awkward afternoon.

What changed in the recipe? Every ingredient we're about to touch today:

• AdamW / LAMB replacing plain SGD in some regimes (though for CIFAR + ResNet, SGD with the right momentum still wins).
• Cosine schedule with warmup replacing step decays. No more "drop LR by 10× at epochs 30, 60, 90" hand-tuning.
• Longer training — 300 epochs instead of 90. Sounds obvious in retrospect, but people had been running short schedules because they thought the model "converged".
• RandAugment + MixUp + CutMix + label smoothing + stochastic depth — five regularisers stacked, none of which existed in usable form in 2015.
• EMA of model weights — keep a Polyak-style running average of weights and use that for evaluation. Free 0.3-0.8% in most vision tasks.
• Weight decay excluded from BN and biases — the subject of section 3.1 today, and a common silent bug.
• Mixed-precision training (fp16 / bf16 / fp8) — 3× faster iteration on the same GPU, no accuracy loss.

This is why the field's centre of gravity shifted from "invent new architectures" to "invent better training recipes" around 2020. The timm library — Wightman's collection of reference vision models and recipes — became the de-facto ImageNet baseline. Every new paper today either uses timm's recipe directly or explains why they didn't.

Today we build the CIFAR-10 version of that recipe. It's simpler than the ImageNet one but every ingredient is the same. Do this session slowly. The knobs you turn today are the same knobs you'll turn when you fine-tune Llama-3 in twenty sessions.

Today we actually train something that works

The last two sessions were about mechanics. This one is about training runs — the messy, empirical craft of getting a network to actually reach a good number on a real dataset. CIFAR-10 is the perfect first target: 50,000 training images, 10 classes, small enough to fit on a laptop GPU, big enough that shortcuts and augmentation actually matter.

The competitive baseline for ResNet-18 on CIFAR-10 is around 93% test accuracy. We're going to hit it. Not by copying a script — by understanding each ingredient and why it's there.

Prerequisites

• S025 + S026 (conv layers and ResNet basic block).
• S020–S024 (dropout, batchnorm, weight init, LR schedules, grad clipping).

1 · The dataset and its normalizations

CIFAR-10 is 50,000 training + 10,000 test images, each 32×32 RGB, in 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck).

import torch
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
 
CIFAR_MEAN = (0.4914, 0.4822, 0.4465)
CIFAR_STD  = (0.2470, 0.2435, 0.2616)
 
train_tfm = transforms.Compose([
    transforms.RandomCrop(32, padding=4, padding_mode="reflect"),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(CIFAR_MEAN, CIFAR_STD),
])
 
test_tfm = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(CIFAR_MEAN, CIFAR_STD),
])

The three important things about this block:

1. RandomCrop(32, padding=4). Pads the 32×32 image out to 40×40, then randomly crops a 32×32 window. This is a mild spatial jitter — the network sees each image with slightly different framing every epoch, which acts as regularization.
2. RandomHorizontalFlip(). Reflects the image left-right with 50% probability. A cat is still a cat when mirrored; a "7" is not still a "7" when mirrored — that's why we don't do this on MNIST.
3. Normalization. Subtract dataset mean, divide by dataset std, per channel. Puts inputs in roughly [-2, 2] — which is where ReLU + BatchNorm want to see them. Only applied to ToTensor output (i.e. after conversion to floats in [0, 1]).

train = datasets.CIFAR10("./data", train=True,  download=True, transform=train_tfm)
test  = datasets.CIFAR10("./data", train=False, download=True, transform=test_tfm)
 
train_dl = DataLoader(train, batch_size=128, shuffle=True,
                       num_workers=4, pin_memory=True, drop_last=True)
test_dl  = DataLoader(test,  batch_size=512, shuffle=False,
                       num_workers=4, pin_memory=True)

pin_memory=True and num_workers=4 are the two flags that keep the GPU fed. Skip them and your GPU sits idle 60% of the time waiting for CPU→GPU copies.

2 · The model — a small adaptation of S026's ResNet-18

Vanilla ResNet-18 is designed for 224×224 ImageNet. On 32×32 CIFAR, its 7×7 stride-2 stem instantly kills the spatial resolution. Community-standard CIFAR variant: replace the stem with a plain 3×3 stride-1 conv, drop the max-pool.

import torch.nn as nn
import torch.nn.functional as F
from typing import Type
 
class BasicBlock(nn.Module):
    def __init__(self, in_c, out_c, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_c, out_c, 3, stride, 1, bias=False)
        self.bn1   = nn.BatchNorm2d(out_c)
        self.conv2 = nn.Conv2d(out_c, out_c, 3, 1, 1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_c)
        self.short = (nn.Sequential(
                        nn.Conv2d(in_c, out_c, 1, stride, bias=False),
                        nn.BatchNorm2d(out_c))
                      if stride != 1 or in_c != out_c else nn.Identity())
 
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return F.relu(out + self.short(x))
 
class ResNet18_CIFAR(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.stem = nn.Sequential(
            nn.Conv2d(3, 64, 3, 1, 1, bias=False),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
        )
        self.layer1 = self._make(64,  64,  2, stride=1)
        self.layer2 = self._make(64,  128, 2, stride=2)
        self.layer3 = self._make(128, 256, 2, stride=2)
        self.layer4 = self._make(256, 512, 2, stride=2)
        self.pool = nn.AdaptiveAvgPool2d(1)
        self.fc   = nn.Linear(512, num_classes)
 
    def _make(self, in_c, out_c, n, stride):
        blocks = [BasicBlock(in_c, out_c, stride)]
        blocks += [BasicBlock(out_c, out_c, 1) for _ in range(n-1)]
        return nn.Sequential(*blocks)
 
    def forward(self, x):
        x = self.stem(x)
        x = self.layer1(x); x = self.layer2(x)
        x = self.layer3(x); x = self.layer4(x)
        return self.fc(self.pool(x).flatten(1))

3 · Optimizer + scheduler — the classic recipe

Three components. Together they are the CIFAR reference recipe.

from torch.optim import SGD
from torch.optim.lr_scheduler import CosineAnnealingLR
 
model = ResNet18_CIFAR().cuda()
 
opt = SGD(model.parameters(),
          lr=0.1,
          momentum=0.9,
          weight_decay=5e-4,
          nesterov=True)
 
epochs = 50
sched = CosineAnnealingLR(opt, T_max=epochs)

Why each hyperparameter is what it is:

• lr=0.1 with SGD + momentum. The historical CIFAR default. Adam works too but tends to reach ~1% lower on CIFAR-10 with a ResNet. SGD wins for image classification, still.
• momentum=0.9. Smooths the gradient direction, escapes small ravines. 0.9 is the near-universal default; you almost never tune it.
• weight_decay=5e-4. L2 penalty on weights. Prevents them from growing without bound. Crucial for ResNet — dropping it costs 2–3 accuracy points. But be careful: don't apply it to BatchNorm parameters or biases (see below).
• nesterov=True. Lookahead momentum. Free ~0.2%. Turn it on.
• CosineAnnealingLR(T_max=epochs). Smoothly decays LR from 0.1 to 0 over the full run. No step drops, no re-tuning. Just works. Almost every modern paper uses cosine.

3.1 The weight-decay-on-BN gotcha

Weight decay on BatchNorm scale and biases is a bug that most beginner code has. The fix — separate the parameter groups:

decay, no_decay = [], []
for name, p in model.named_parameters():
    if not p.requires_grad: continue
    if p.ndim <= 1 or name.endswith(".bias"):
        no_decay.append(p)
    else:
        decay.append(p)
 
opt = SGD([
    {"params": decay,    "weight_decay": 5e-4},
    {"params": no_decay, "weight_decay": 0.0},
], lr=0.1, momentum=0.9, nesterov=True)

You lose accuracy by regularizing BN — its scale/shift parameters are supposed to be free. Skip this and you cap around ~91%.

4 · The training loop with mixed precision

from torch.cuda.amp import autocast, GradScaler
 
scaler = GradScaler()
loss_fn = nn.CrossEntropyLoss()
 
for epoch in range(epochs):
    model.train()
    for xb, yb in train_dl:
        xb, yb = xb.cuda(non_blocking=True), yb.cuda(non_blocking=True)
        opt.zero_grad(set_to_none=True)
 
        with autocast():
            logits = model(xb)
            loss = loss_fn(logits, yb)
 
        scaler.scale(loss).backward()
        scaler.step(opt)
        scaler.update()
 
    sched.step()

Three things to notice:

• set_to_none=True on zero_grad — sets gradients to None instead of zero-tensors. Slightly faster and forces bugs where you forgot to call .backward() to surface as NoneType errors instead of silently wrong zeros.
• autocast() casts eligible ops to float16 automatically. Convs and matmuls run at 2× throughput on modern GPUs (Ampere onwards use tensor cores in fp16).
• GradScaler scales the loss up before backward pass to keep small gradients from underflowing to zero in fp16, then scales gradients back down before the optimizer step. Without it, your loss will silently plateau.

4.1 The eval loop

def evaluate(model, dl):
    model.eval()
    correct = total = 0
    with torch.no_grad(), autocast():
        for xb, yb in dl:
            xb, yb = xb.cuda(non_blocking=True), yb.cuda(non_blocking=True)
            correct += (model(xb).argmax(1) == yb).sum().item()
            total   += yb.size(0)
    return correct / total
 
for epoch in range(epochs):
    # ... training loop above ...
    acc = evaluate(model, test_dl)
    print(f"epoch {epoch:2d}  lr={sched.get_last_lr()[0]:.4f}  test_acc={acc:.4f}")

Expected trajectory:

epoch  0  lr=0.1000  test_acc=0.4823
epoch  5  lr=0.0976  test_acc=0.7841
epoch 15 lr=0.0805  test_acc=0.8712
epoch 30 lr=0.0331  test_acc=0.9155
epoch 45 lr=0.0033  test_acc=0.9302

5 · Reading the log — the three failure modes

5.1 "Loss stuck near log(10) ≈ 2.30"

You never actually started learning. Causes, in order of likelihood:

1. LR too high — divergence. You'll see loss = NaN within a few steps.
2. LR too low — no visible progress. Bump 10×.
3. Data pipeline bug: labels shuffled independently from images. Check by hand — xb, yb = next(iter(train_dl)); plt.imshow(xb[0].permute(1,2,0)); print(yb[0]).
4. Model outputs raw nan — check for x.mean().item() at each layer, find the first layer that produces NaN, look at its inputs. Usually an underflow in fp16 or an NaN-producing operation like log(0).

5.2 Train accuracy → 100%, test accuracy stuck at 80%

Overfitting. Fixes: stronger augmentation (RandAugment, CutMix), more weight decay, less capacity, more data. On CIFAR-10 with ResNet-18, an >10% gap almost certainly means augmentation isn't turned on.

5.3 Test accuracy 55%, "seems fine but doesn't improve"

Silently broken. Common causes:

• You set model.eval() inside the training loop and forgot to put it back into .train() — BatchNorm now uses running stats during training.
• You forgot optimizer.step(). Loss goes down for a bit (from initialization luck), then plateaus.
• You loaded pretrained weights for num_classes=1000 and only re-initialized the FC layer, but forgot to unfreeze the rest.

War stories

6 · The 2025 pulse · how far this recipe scales

CIFAR-10 with ResNet-18 is a tiny problem — 5 minutes on a laptop GPU. But the recipe scales all the way up to trillion-parameter models. Here's what changes as you go up:

~100k params (this session): SGD + momentum 0.9 + cosine 100 epochs + RandomCrop/Flip is enough. 93%+ on CIFAR-10.

~10M params (ResNet-18 on ImageNet): Add RandAugment + MixUp + label smoothing + longer schedule + LR warmup. Recipe from Wightman's "ResNet Strikes Back". ~80% top-1.

~100M params (ViT-B, ConvNeXt-B): Switch to AdamW. Add stochastic depth + EMA weights + longer warmup. Layer-wise LR decay for fine-tuning. Recipe from DeiT (Touvron et al. 2020) and ConvNeXt. ~85-87% top-1.

~1B params (ViT-L, ConvNeXt-L, EVA-02): Same recipe but with self-supervised pretraining before supervised fine-tuning — MAE (He et al. 2021), DINOv2 (Oquab et al. 2023), or SigLIP (Zhai et al. 2023). Supervised loss alone stops being enough data-efficient. Recipe from EVA-02 (Fang et al. 2023). ~90% top-1.

~10B+ params (ViT-22B, DINOv2, InternVL 2.5): FSDP or DeepSpeed ZeRO-3 sharding, gradient checkpointing everywhere, careful LR schedule with linear warmup + cosine decay tail, mixed-precision fp8 or bf16, and a data pipeline that streams from S3. Recipe from ViT-22B (Dehghani et al. 2023). At this scale training-run engineering (checkpointing, restart, monitoring) becomes as much work as the model itself — we'll cover this in M09.

2024–2025 developments to know:

• torch.compile (PyTorch 2.0+, stable in 2.4) — wraps your model in Inductor's compiler graph. Free 20-40% throughput speedup for typical CNN + attention hybrids. Should be your default in 2025.
• bf16 as default — fp16 requires GradScaler because its range is small. bf16 has fp32's range with fp16's memory footprint. On any Ampere+ GPU (A100, H100, H200, RTX 4090, 5090) bf16 has no downsides and simpler code.
• FP8 training (H100+) — NVIDIA Transformer Engine. Another 2× throughput on H100/H200. Still requires careful loss scaling and per-tensor scaling factors. Standard for large LLM pretraining in 2025.
• Sophia optimizer (Liu et al. 2023) and Lion (Chen et al. 2023) — candidate replacements for AdamW. Both save memory (Lion needs only momentum, not second moment). Empirically strong for LLMs, mixed results for vision.
• schedule-free optimizers (Defazio et al. 2024) — remove the need for a LR schedule entirely by using a Polyak-style iterate averaging trick. Actively deployed inside Meta's LLM training as of late 2024.

For this session, stick with SGD + momentum + cosine — it's the reference that every deep-learning course uses. But know that if you're training anything > 100M params on real data in 2025, AdamW + bf16 + torch.compile is the modern default.

Further reading · training recipes in 2025

• Wightman, Touvron, Jégou (2021) — ResNet Strikes Back. Section 3 is the reference recipe.
• Bello et al. (2021) — Revisiting ResNets: Improved Training and Scaling Strategies. The Google Brain sibling paper with detailed ablations.
• Loshchilov & Hutter (2019) — Decoupled Weight Decay Regularization. The AdamW paper. Section 4 explains why decoupled decay matters.
• PyTorch (2024) — torch.compile tutorial.
• Andrej Karpathy — A Recipe for Training Neural Networks (2019). Still the best general-purpose debugging checklist ever written.

Retention scaffold

One-line summary (write it in your own words): ______________________________________

Spaced review: re-run the training loop from memory in 24 hours — no copy-paste. On day 7, take the log from your best run and label each line with which recipe ingredient produced that curve shape.

Next session (S028): Transfer learning — take an ImageNet-pretrained ResNet, freeze the backbone, retrain the head on a tiny 200-image dog-breed dataset. Beat from-scratch accuracy in 5 epochs instead of 50.

Sticky note:

SGD(0.1, 0.9, nesterov, wd=5e-4) · Cosine · Crop+Flip · AMP — memorise this like a poem. It's the CIFAR baseline every deep-learning class starts from.

Legacy recall drills

Recall

Stretch

Take your trained ResNet-18 and add Mixup + CutMix (torchvision.transforms.v2). Aim for 94%. Then try RandAugment. Aim for 95%. Set a 3-hour timer and log every experiment.

In your own words

Write out the CIFAR recipe from memory: model, optimizer, scheduler, augmentation, precision. No googling.

Spaced review

• S021 (BatchNorm) — the bias=False and weight-decay-exclusion tricks trace back here.
• S023 (LR schedules) — cosine is the schedule we just used.
• S024 (grad clipping / accumulation) — GradScaler is the fp16 cousin of grad clipping.

Next session

S028 — Transfer learning. Take an ImageNet-pretrained ResNet, freeze the backbone, retrain the head on a tiny custom dataset (like 200 dog-breed images). Match from-scratch accuracy in 5 epochs instead of 50.

Bring back tomorrow

• The recipe: SGD(0.1, mom=0.9, nesterov, wd=5e-4) + cosine + RandomCrop/Flip + mixed precision.
• Weight decay excludes BN and biases.
• Three failure modes: not learning, overfitting, silently broken.

Previous: ← DL S026 · Next: DL S028 →