S105 · Transfer Learning & Fine-Tuning Classical DL
Never train from scratch again. How ImageNet-pretrained CNNs learn your custom task in an hour instead of a week — and the failure modes that catch every junior.
🎯 Take a pretrained ResNet, freeze the backbone, fine-tune the head on your dataset, and understand exactly which layers to unfreeze when and why.
Why this session exists
Almost nobody trains a deep network from scratch anymore. Every practical vision problem starts with a model pretrained on ImageNet (or a bigger dataset), then adapts it to a specific task with a fraction of the data. Same story for NLP with BERT/LLaMA and audio with Wav2Vec2. Transfer learning is the reason a solo developer can build a production classifier with 1000 labelled images instead of 1 million. Miss the details — layer freezing, learning-rate schedules, catastrophic forgetting — and you'll ship a model that's 10 points worse than necessary.
- Explain why the early layers of a CNN transfer better than late layers.
- Distinguish feature extraction, fine-tuning, and full retraining — and pick the right one.
- Set differential learning rates for the head vs the backbone.
- Fine-tune a torchvision ResNet on a small custom dataset with correct data preprocessing.
- Diagnose catastrophic forgetting and the top-3 fine-tuning failure modes.
Prerequisites
- S102 · CNNs — you need to know what a ResNet block looks like.
- S100 · PyTorch Fundamentals — you'll manipulate
nn.Moduleinternals. - S104 · Embeddings — the intuition of "pretrained representations are useful."
(a) Intuition · 5 min
You need someone to cook Andhra biryani for your restaurant. Two options: hire a total beginner and spend 5 years training them from knife skills up, or hire an experienced Michelin chef who's cooked French and Italian for 20 years, and spend 3 months teaching them the specifics of biryani.
The Michelin chef already knows knife technique, heat control, seasoning, timing, plating. All you need to teach is the biryani-specific stuff — the masala ratios, dum cooking, layering. Six months and they're outperforming your 5-year trainee.
A pretrained CNN is that Michelin chef. It already knows how to see — edges, textures, shapes, object parts — from staring at 1.4M ImageNet images for a week. Give it your custom dataset (500 chest X-rays, or 2000 satellite tiles) and it just needs to learn the last mile: how the general visual features map to your classes.
Result: 1000 images gets you 85% accuracy in an hour, when training from scratch would need 100K images and a week.
The three transfer-learning strategies
- Feature extraction — freeze the entire backbone, train only a new head. Use when you have very little data (\<1000 images) or your task is close to the pretraining domain. Fastest, safest.
- Fine-tuning — unfreeze the top layers (or all layers) at a small learning rate. Use when you have moderate data (1K–100K) and the domain differs from pretraining. Best trade-off.
- Full retraining — initialise from pretrained weights but train everything at full LR. Use when you have massive data (\>100K) AND the domain is very different (medical scans, satellite images at different wavelengths).
The four-quadrant decision matrix
- small + similarFeature extractionFreeze backbone, train head only. E.g. classifying dog breeds after ImageNet pretraining.
- small + differentFeature extraction from EARLY layersBackbone is still useful for low-level features but not for high-level. Extract features from block 2-3, not the final layer.
- large + similarFine-tune with low LR on backboneUnfreeze all, backbone LR = 1/10 of head LR. E.g. medical X-ray classification (grayscale, but still 'photos').
- large + differentFine-tune aggressively OR train from scratchSatellite imagery in multispectral bands, cryo-EM microscopy — pretraining might hurt. Test both.
(b) Visual walkthrough · 15 min
Why early layers transfer, late layers don't
The green blocks are portable — an edge detector or texture detector works for chest X-rays as well as it works for cat photos. The red blocks are ImageNet-specific — "this is a golden retriever" logic isn't useful when you're classifying skin lesions.
The universal fine-tuning recipe
torchvision.models.resnet50(weights=ResNet50_Weights.IMAGENET1K_V2). Weights auto-downloaded (~100MB).
model.fc = nn.Linear(2048, num_your_classes). The 1000-class ImageNet head is useless for your task.
for p in model.parameters(): p.requires_grad = False. Then unfreeze head: for p in model.fc.parameters(): p.requires_grad = True.
Use a normal LR (1e-3 for Adam). Warm the new head up so it makes reasonable predictions before you touch the backbone.
Unfreeze layer4 (last ResNet stage). Use 10× lower LR on it (1e-4). Train another 5 epochs.
Unfreeze layer1-3 too, at 100× lower LR (1e-5). Optional — only if you have enough data. Train 5-10 more epochs.
Data preprocessing is the silent killer
What torchvision's weights.transforms() gives you
Feature extraction vs fine-tuning at a glance
Freeze everything but head
- Fastest — no backbone gradients
- Can precompute features once
- Needs \<1000 samples
- 80% of top possible accuracy usually
Unfreeze last block, low LR
- 10× slower per epoch
- Needs 1K–100K samples
- 90-95% of top accuracy
- Standard industry recipe
Discriminative LRs
- Full-speed backbone gradients
- Best accuracy on target task
- Needs 10K+ samples
- Risk of overfit / catastrophic forgetting
Random init, full LR
- Slowest, needs most data
- Only wins if domain is truly alien
- Baseline to compare against
- Rare in practice
"Transfer learning means I freeze the backbone and train a new head. If accuracy is bad, the pretrained weights just weren't good enough for my domain."
Most transfer-learning failures are preprocessing and learning-rate failures, not representation failures. A frozen ImageNet backbone fed images normalised with the wrong mean/std, or resized with the wrong interpolation, produces activations off the manifold it was trained on — and the head cannot recover that. Likewise, unfreezing with the head's learning rate destroys the backbone in the first few hundred steps.
Because the myth is true in the one case people meet first: a tutorial on cats-vs-dogs, where the target domain is ImageNet, so preprocessing is copied verbatim and it works. The moment you move to grayscale medical scans, satellite tiles, or screenshots, the same code silently mis-normalises and the only visible symptom is "accuracy is low" — which reads as "the features don't transfer".
Compare the feature statistics your data produces against what the backbone expects. If the activations are near-dead or saturated, it is preprocessing, not transfer:
import torch, torchvision as tv
w = tv.models.ResNet18_Weights.IMAGENET1K_V1
print(w.transforms()) # the EXACT resize/crop/normalise the weights assume
m = tv.models.resnet18(weights=w); m.fc = torch.nn.Identity(); m.eval()
with torch.no_grad():
f_ok = m(good_batch) # preprocessed with w.transforms()
f_bad = m(raw_0_1_batch) # no normalisation
print(f_ok.mean().item(), f_ok.std().item())
print(f_bad.mean().item(), f_bad.std().item()) # collapsed / shiftedWhy does the standard recipe use a lower learning rate for early layers than for late ones? "Discriminative learning rates" sounds like a heuristic — it is forced.
- 1A pretrained network is a stack of feature detectors whose usefulness is ordered: early layers encode edges, colour opponency and texture; late layers encode class-specific compositions.forced by · this ordering is an empirical regularity of gradient-trained convnets and transformers alike — early features are shared across almost all natural-image tasks
- 2Your new task changes the label space and usually the object semantics, but almost never changes what an edge is.forced by · the low-level statistics of natural images are task-independent
- 3Therefore the amount of required weight change is small at the bottom and large at the top.forced by · only the layers whose target function actually changed need to move
- 4But the gradient magnitude reaching a layer is not proportional to how much that layer needs to change — it is set by backprop through the loss and the randomly-initialised head, which at step 0 emits a large, meaningless error signal.forced by · a fresh head has no useful direction yet, so its gradient is noise with real magnitude
- 5A single global learning rate therefore applies noise-driven updates of similar scale to layers that needed to move a lot and layers that needed to move barely at all — destroying the latter. This is catastrophic forgetting.forced by · SGD has no notion of "this weight was expensive to learn"
Therefore the LR must be scaled down with depth-from-the-output: it is the only knob that encodes the prior "the bottom is already right". The usual geometric decay (each block ~2–3× lower than the one above) is just a smooth version of freezing.
And note what this predicts: warming up the head first with the backbone frozen, then unfreezing, should recover most of the same benefit — because it removes the noisy-head phase before any backbone gradient flows. That is exactly the two-stage recipe practitioners converged on independently.
The pretrained backbone is a finished house: foundation, plumbing, wiring. Your task is a new tenant with different furniture. You do not re-pour the foundation. You repaint the top floor, and you touch the plumbing only gently, only if the new tenant genuinely needs different pipes.
The learning rate is how hard you swing the hammer. One rate everywhere means you swing at the foundation as hard as at the wallpaper.
- Data size decides the strategy: hundreds of samples ⇒ frozen features + linear head; thousands ⇒ unfreeze the last block; tens of thousands ⇒ full fine-tune with discriminative LRs.
- Domain distance decides how deep you unfreeze. Close to pretraining data ⇒ shallow. Alien (spectrograms, seismic, X-ray) ⇒ deeper, or reconsider pretraining source.
- Always warm up the head before unfreezing anything. A random head is a gradient bomb.
- Preprocessing must match the weights, not your intuition. Wrong normalisation looks exactly like "features don't transfer".
Fire this model the moment you see: a new model starting from weights= anything · accuracy that plateaus far below a published baseline · training loss that rises in the first epoch after unfreezing · a teammate saying "we should just train from scratch" · a domain-shift complaint with no preprocessing check.
You have a pretrained backbone and a modest labelled dataset. Freeze and train a head, partially fine-tune, or fine-tune everything?
Start frozen, always. It costs an hour and gives you the number that every later decision is measured against. Then unfreeze upward one block at a time and stop the moment the validation gain stops paying for the compute — that stopping point is the measurement of how far your domain sits from the pretraining distribution, and it is more informative than the accuracy itself.
The failure mode to avoid is skipping straight to full fine-tuning: it usually works, it always costs the most, and it leaves you with no idea whether you needed it.
(c) Hands-on · 25 min
Fine-tune a pretrained ResNet-18 on a custom two-class dataset (bees vs ants, from Hymenoptera — a classic small dataset). ~100 train images per class. Get to \>90% val accuracy in 10 epochs on CPU.
Anatomy of the script
What the interesting lines do
Replace build_model with a version that doesn't freeze anything, and use a flat optimiser Adam(model.parameters(), lr=1e-3). Train for 10 epochs. Compare final val_acc against the two-phase version. Difference is usually 15-25 points on tiny datasets like this.
(d) Production reality · 15 min
Google's screening model was fine-tuned from Inception-v3 pretrained on ImageNet. Works great on carefully-captured research images. Deployed to Thai clinics, accuracy dropped ~15 points. Investigation: the domain gap was too big. ImageNet = photos of everyday objects; retinal scans = specialised fundus images with entirely different colour distribution, lighting, and structure.
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
If you can't teach these three without notes, redo the session:
- Why can 1000 images beat 100K images (from scratch) with the right pretrained backbone?
- What is catastrophic forgetting, and what's the specific hyperparameter change that prevents it?
- When should you NOT use transfer learning?
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.