S105 · Transfer Learning & Fine-Tuning Classical DL

Standing on giants' shoulders.

Module M12: Deep Learning · Session 105 of 130 · Track: ML 🎓

What you'll be able to do after this session

Explain Transfer Learning & Fine-Tuning Classical DL to a friend in one minute, out loud, no notes.
Recognise it when you see it in real code / systems / papers.
Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

S102 — CNNs — Convolution, Pooling, ImageNet Architectures

Pre-read (skim before the session)

🎥 Intuition (5–15 min) · Transfer Learning — Sebastian Raschka — Clean overview of feature extraction vs fine-tuning.
🎥 Deep dive (30–60 min) · Stanford CS231n — Transfer Learning — The rules of thumb for when to freeze what.
🎥 Hands-on demo (10–20 min) · Fine-tune ResNet in PyTorch — Aladdin Persson — Live-codes the exact recipe you'll use in industry.
📖 Canonical article · CS231n — Transfer Learning notes — The decision matrix every CV practitioner memorises.
📖 Book chapter / tutorial · PyTorch — Transfer Learning for Computer Vision Tutorial — Official end-to-end recipe with code.
📖 Engineering blog · Google Research — Advancing instance-level recognition — Real production fine-tuning of ImageNet backbones.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Standing on giants' shoulders.

If you needed to build a rare-cat-breed classifier and you had 500 labelled photos, would you train a network from random weights? No — you'd start from a network that already spent weeks staring at 14 million ImageNet photos and knows what a cat, a whisker, a fur pattern is, and then teach it the last 1% about your specific breeds. That's transfer learning. You take a model pretrained on a huge, generic dataset and reuse its learned features for your small, specific task.

The insight is that early layers in a deep net learn generic features (edges, textures, colours) useful for almost any visual task, middle layers learn object parts (eyes, wheels, leaves), and only the last few layers learn task-specific concepts. Since you're only really changing the last part, you get away with tiny datasets and hours of training instead of weeks.

There are two flavours: feature extraction (freeze all pretrained layers, train only a new head on top — fast, works with tiny datasets) and fine-tuning (unfreeze some or all layers and train with a small learning rate so the pretrained knowledge isn't destroyed — more powerful, needs more data).

Gotcha to carry forever: The single biggest transfer-learning bug is using a learning rate that's too high on the pretrained backbone. The old weights are precious — hit them with lr=0.1 and you'll wipe out ImageNet in three batches. Industry standard is a differential learning rate: lr=1e-4 for the pretrained layers, lr=1e-3 for the new head.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The decision of "how much to fine-tune" depends on dataset size and similarity to the pretraining source:

Your dataset	Similar to pretraining?	Recipe
Small (<1000)	Yes	Freeze everything, train a linear classifier on features
Small (<1000)	No	Freeze early layers, train last few + head. Risky.
Large (>10K)	Yes	Fine-tune the whole network with small lr
Large (>10K)	No	Fine-tune the whole network, maybe train from scratch

Worked example — differential learning rates. In PyTorch:

opt = torch.optim.SGD([
    {"params": model.backbone.parameters(), "lr": 1e-4},   # gentle
    {"params": model.head.parameters(),     "lr": 1e-3},   # aggressive
], momentum=0.9)

Two parameter groups, two learning rates. Adam and AdamW support the same pattern.

Worked example — feature extraction as sklearn. A shockingly good baseline for any image task:

Pass every training image through resnet50(pretrained=True) up to the penultimate layer → a 2048-dim vector per image.
Stack them into an (N, 2048) matrix.
Fit sklearn.linear_model.LogisticRegression on that matrix.

That's it — no PyTorch training loop. On small-data problems this often beats a from-scratch CNN by 20+ accuracy points and trains in seconds.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# transfer_learn.py — fine-tune ResNet-18 on the hymenoptera dataset
# Data: https://download.pytorch.org/tutorial/hymenoptera_data.zip
import torch, torch.nn as nn, torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, models, transforms
 
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
DATA_DIR = "./hymenoptera_data"
 
tfm = {"train": transforms.Compose([
           transforms.RandomResizedCrop(224),
           transforms.RandomHorizontalFlip(),
           transforms.ToTensor(),
           transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]),
       "val": transforms.Compose([
           transforms.Resize(256), transforms.CenterCrop(224),
           transforms.ToTensor(),
           transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])}
loaders = {k: DataLoader(datasets.ImageFolder(f"{DATA_DIR}/{k}", tfm[k]),
                         batch_size=32, shuffle=(k=="train"), num_workers=2)
           for k in ("train", "val")}
n_classes = len(loaders["train"].dataset.classes)
 
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)
model.fc = nn.Linear(model.fc.in_features, n_classes)
model = model.to(DEVICE)
 
opt = optim.SGD([
    {"params": [p for n, p in model.named_parameters() if not n.startswith("fc.")],
     "lr": 1e-4},
    {"params": model.fc.parameters(), "lr": 1e-3},
], momentum=0.9)
sched = optim.lr_scheduler.StepLR(opt, step_size=5, gamma=0.1)
 
for epoch in range(10):
    for phase in ("train", "val"):
        model.train() if phase == "train" else model.eval()
        loss_sum, correct = 0.0, 0
        for x, y in loaders[phase]:
            x, y = x.to(DEVICE), y.to(DEVICE)
            opt.zero_grad()
            with torch.set_grad_enabled(phase == "train"):
                logits = model(x); loss = nn.functional.cross_entropy(logits, y)
                if phase == "train": loss.backward(); opt.step()
            loss_sum += loss.item() * x.size(0)
            correct  += (logits.argmax(1) == y).sum().item()
        n = len(loaders[phase].dataset)
        print(f"[{epoch}] {phase} loss={loss_sum/n:.4f} acc={correct/n:.4f}")
    sched.step()

Checklist while it runs:

First-epoch validation accuracy should already be 85%+ — that's pretraining working for you.
Change weights=...DEFAULT to None (random init) — accuracy crashes to 50% (chance).
Set both learning rates to 1e-1 — training diverges within an epoch.
Freeze the entire backbone (p.requires_grad = False) before replacing fc — trains ~1000 params, still hits ~90%.
Print sum(p.numel() for p in model.parameters()) — ~11M.

Try this modification: Replace resnet18 with models.efficientnet_b0(weights="DEFAULT"). The final layer is model.classifier[1] instead of model.fc. Same recipe, ~2% higher accuracy.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Transfer learning isn't just a trick — it's the only way most production CV systems get built. Almost nobody has 14M labelled examples of their exact problem, so the entire industry stands on top of ImageNet, COCO, and increasingly self-supervised foundation models like DINO, CLIP, and SAM.

The BatchNorm-in-fine-tuning trap (see also S101). Pretrained models have BN running statistics from ImageNet. When you fine-tune on a small dataset, tiny batches can corrupt those stats and blow up validation accuracy in a way that looks like "the model just forgot everything". Standard fix: freeze BN layers in .eval() mode, or replace with GroupNorm.

Domain shift is the real killer. ImageNet is photos of consumer objects taken in natural light. If your task is medical X-rays, satellite imagery, or industrial defect detection, ImageNet features help less than intuition suggests. The modern answer is domain-specific pretraining: RadImageNet for radiology, BigEarthNet for satellite. Or self-supervised pretraining on your own unlabelled data (SimCLR, MAE) followed by supervised fine-tuning.

Common bugs in code review:

Wrong normalisation. Feeding [0, 255] pixel values to a model that expects normalised [0, 1] inputs.
Forgotten requires_grad. You "froze" the backbone but forgot to actually set requires_grad = False, so the optimizer silently updates all 25M parameters at lr=1e-3 — 10× the intended.
Class imbalance ignored. You have 10,000 healthy X-rays and 200 diseased. Naive fine-tuning gives you a 98%-accurate model that always predicts "healthy". Use WeightedRandomSampler or a weighted loss.
Test-set leakage. You split by image but two images from the same patient/session end up in train and test — model looks great, deploys, fails.

LLM world. The exact same pattern applies to language models: pretrain on the internet, fine-tune on your task. LoRA, QLoRA, and PEFT are the modern equivalents of "differential learning rates" — they let you fine-tune a 70B model with a laptop-friendly memory footprint. You'll meet them in M15.

(e) Quiz + exercise · 10 min

What's the difference between feature extraction and fine-tuning?
Given a small dataset that's very different from ImageNet, which layers should you freeze and why?
What is a "differential learning rate" and why is it standard practice in fine-tuning?
Why must you normalise inputs to a pretrained model with the pretraining mean and std?
Name the BatchNorm bug that commonly surfaces during fine-tuning on small datasets.

Stretch (connects to S102 — CNNs): Why does replacing a CNN's classifier head with a Linear(feature_dim, num_classes) work at all — what property of the feature vector coming out of AdaptiveAvgPool2d(1) makes this a sensible thing to do?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

What is Transfer Learning & Fine-Tuning Classical DL? (one sentence, no jargon)
When would you use it? (one concrete example)
When would you NOT use it? (one anti-pattern)

What comes next

Next session (S106): Tokenization — BPE, WordPiece, SentencePiece
Previous (S104): Embeddings — word2vec, GloVe, Contrastive Learning
Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.