DL S028 · Transfer Learning — Frozen Backbones and Fine-Tuning

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 28 / 80 · Module M05

The story

Story hook · the ImageNet moment nobody expected

In 2013, one year after AlexNet won ImageNet, Matthew Zeiler and Rob Fergus published "Visualizing and Understanding Convolutional Networks" — the paper that first cracked open a trained convnet and asked what did it actually learn? They ran a technique called deconvnet to project neurons back to input space. Layer 1 filters looked like Gabor edges (surprise: same thing V1 in your visual cortex does). Layer 2 detected corners, colour blobs, T-junctions. Layer 3 detected textures. Layer 4 started detecting object parts — dog faces, car wheels. Layer 5 detected whole categories.

The practical implication landed hard. If layers 1–4 of a network trained on ImageNet learned features useful for detecting any natural image concept — not just the 1000 ImageNet classes — then you could take the trained network, chop off the last layer, and use it as a general-purpose feature extractor for any vision task. Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson at KTH proved this brutally clearly with "CNN Features off-the-shelf: an Astounding Baseline for Recognition" (CVPR 2014). They took a pretrained AlexNet, ran every image through it, extracted the 4096-dimensional penultimate-layer activation, and trained a linear SVM on top. On 15+ vision benchmarks — object classification, scene classification, fine-grained bird species, face verification, attribute prediction — this trivial "linear probe" pipeline beat or matched every specialised hand-designed system.

The subtitle of that paper ("an astounding baseline") was not a joke. The community's reaction was a mixture of excitement and existential dread. Excitement because suddenly every small-data vision problem became tractable overnight. Dread because a decade of hand-engineered features had just been rendered obsolete by 5 minutes of feature extraction from someone else's network.

That paper is why every modern vision pipeline starts with torchvision.models.resnet18(weights='DEFAULT') or timm.create_model('convnext_base.fb_in22k_ft_in1k', pretrained=True). The transferability of learned features is the reason the entire deep-learning ecosystem is downloadable — HuggingFace Hub, torch.hub, timm. Every one of those weights files is someone else's compute that you get to reuse for free.

And — this is the important bit — the trick is not specific to vision. The exact same argument extends to language models. When you fine-tune Llama-3 or Gemma-3 on your domain, you're doing transfer learning: the model spent millions of GPU-hours learning generic English, and you spend a few GPU-hours specialising it for medical Q&A or SQL generation. Sessions 058–065 in this series will do that in detail. What you learn today about vision transfer maps directly to LLM fine-tuning.

The most useful trick in applied deep learning

Here's a claim that would have been controversial in 2013 and is now completely uncontroversial: you should almost never train a vision model from scratch on your own data. Not "usually". Almost never. If you have <100,000 labelled images (which is basically everyone outside FAANG), start from a model someone else pretrained on ImageNet, LAION, or DINOv2, and fine-tune. You'll get better accuracy in less time with less compute.

This is transfer learning. It works because the early layers of any well-trained convnet learn edge detectors, blob detectors, colour opponents — features that are useful for any natural-image task. Only the last few layers are task-specific. So you keep 95% of the pretrained network and retrain only the 5% that cares about "cat vs dog" vs "malignant vs benign melanoma" vs "wing icing on a Cessna".

Today we do it three ways and understand when each is right.

Prerequisites

• S027 (a working CIFAR training pipeline). We'll adapt it directly.
• S026 (ResNet architecture — you should know why layer4 is the "last" backbone stage).

1 · What "transfer" actually means

Pretraining on ImageNet gives you a network whose layers look, in order:

• Early conv layers (stem + layer1): edge detectors, colour blobs, orientation-selective filters. Universal — useful for every natural-image task.
• Middle layers (layer2, layer3): texture, motif, part detectors. Semi-universal — useful for most tasks but slowly get more ImageNet-specific.
• Late layers (layer4): object-level features (dog-face-ness, wheel-ness, plant-ness). ImageNet-specific — may not help if your task is X-rays or satellite imagery.
• Final FC: literal ImageNet class scores. Always thrown away.

1.1 The three regimes

2 · The scaffold — pretrained ResNet-18 with a swapped head

import torch
import torch.nn as nn
import torchvision
from torchvision import transforms
 
# 1. Get a pretrained model
model = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.IMAGENET1K_V1)
 
# 2. Replace the final classifier
num_features = model.fc.in_features         # 512 for ResNet-18
model.fc = nn.Linear(num_features, num_classes_you_have)   # e.g. num_classes=37 for Oxford-Pets
 
model = model.cuda()

That's it. The 25MB of weights you just downloaded are the result of 90 GPU-days of ImageNet training that someone else paid for. You'd never train that from scratch on your own hardware.

2.1 Preprocessing must match pretraining

Critical: the input pipeline must match the mean, std, and pixel range that the pretrained model saw. For torchvision's ImageNet models, that's:

IMAGENET_MEAN = (0.485, 0.456, 0.406)
IMAGENET_STD  = (0.229, 0.224, 0.225)
 
train_tfm = transforms.Compose([
    transforms.Resize(256),
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(IMAGENET_MEAN, IMAGENET_STD),
])
 
test_tfm = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(IMAGENET_MEAN, IMAGENET_STD),
])

If you feed the model images normalized with your own dataset's mean, the BatchNorm running statistics from ImageNet will be badly mis-calibrated and your accuracy will be inexplicably low. Every torchvision model has a weights.transforms() method that returns the correct preprocessor — use it if you're unsure.

3 · Regime 1 — Linear probe (freeze the backbone)

for p in model.parameters():
    p.requires_grad = False
# Unfreeze only the new head
for p in model.fc.parameters():
    p.requires_grad = True
 
opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3, weight_decay=1e-4)

Because only the head is trainable, memory usage drops sharply — no gradients or optimizer state for the frozen weights. On a 1000-image target dataset this can train in ~2 minutes and reach 85–90% of what full fine-tuning achieves.

3.1 The BatchNorm gotcha in the frozen regime

Setting requires_grad = False on BN weights does NOT put BN in eval mode. BN in .train() mode still updates its running statistics from your target dataset's batch stats, which will drift the model. Fix:

def freeze_bn(model):
    for m in model.modules():
        if isinstance(m, nn.BatchNorm2d):
            m.eval()
            for p in m.parameters():
                p.requires_grad = False
 
model.train()
freeze_bn(model)   # must call AFTER model.train()

Call this at the start of every epoch (because model.train() re-arms all modules). Without it, your linear probe will slowly degrade over training as BN drifts. Silent bug, ~1–2% accuracy loss.

4 · Regime 2 — Full fine-tune

Everything trainable, small LR, short schedule:

for p in model.parameters():
    p.requires_grad = True
 
opt = torch.optim.SGD(model.parameters(),
                       lr=1e-2,          # 10× smaller than "from scratch"
                       momentum=0.9,
                       weight_decay=1e-4,
                       nesterov=True)
 
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=10)   # 10 epochs

Two key rules:

• LR ~10× smaller than from scratch. If from-scratch used lr=0.1, fine-tune uses 1e-2. Too high and you catastrophically forget the pretrained features.
• Fewer epochs. 5–20 is standard, vs 90–300 for from-scratch.

5 · Regime 3 — Discriminative learning rates (the modern default)

Head gets 10× the backbone's LR, because the head is randomly initialized (needs big steps) while the backbone is near-optimal (only needs small nudges).

backbone_params, head_params = [], []
for name, p in model.named_parameters():
    if name.startswith("fc."):
        head_params.append(p)
    else:
        backbone_params.append(p)
 
opt = torch.optim.SGD([
    {"params": backbone_params, "lr": 1e-3},
    {"params": head_params,     "lr": 1e-2},
], momentum=0.9, weight_decay=1e-4, nesterov=True)

Even better: geometrically increasing LR per stage. The layer1 (very early features) gets the smallest LR, layer4 (more task-specific) gets more, and the head gets the most.

groups = []
for i, module in enumerate([model.stem if hasattr(model,'stem') else model.conv1,
                            model.layer1, model.layer2, model.layer3, model.layer4,
                            model.fc]):
    lr = 1e-4 * (10 ** (i * 0.5))    # ~3× per stage
    groups.append({"params": list(module.parameters()), "lr": lr})
opt = torch.optim.SGD(groups, momentum=0.9, weight_decay=1e-4)

This is the "gradually unfreeze" recipe from fast.ai and it consistently wins by ~1% over a single-LR fine-tune.

6 · When does transfer NOT help?

Transfer helps ~always in the 100 → 10K sample regime. It stops mattering (and can even hurt) in two cases:

1. Target task is very different from source. ImageNet is natural photos. If your task is medical grayscale (chest X-rays), satellite multispectral, or grayscale document images, ImageNet features may transfer poorly. Try domain-specific pretraining (e.g., RadImageNet for chest, SentSat for satellite).
2. Target dataset is huge. With >1M task-specific images, from-scratch is fine and sometimes better (no risk of inheriting bad biases).

For everything in between — which is 99% of applied projects — transfer is the right default.

War stories

7 · The 2025 pulse · pretraining has eaten transfer learning

Here is the shift that has happened over the last three years and that most tutorials haven't caught up to: for serious vision work in 2025, you don't fine-tune from ImageNet supervised pretraining anymore. You fine-tune from a self-supervised backbone.

The reason: supervised ImageNet pretraining maxes out around 88% ImageNet top-1 with ~1M labelled images. Self-supervised methods have shown they can learn features that transfer better by consuming unlabelled data at 100×–10000× the scale.

The three self-supervised paradigms that matter:

1. Contrastive learning — SimCLR / MoCo / DINO. Two augmented views of the same image should have similar embeddings; two random images should not. SimCLR (Chen et al. 2020) was the breakthrough. DINO (Caron et al., FAIR 2021) refined the idea for ViT and became the reference approach.
2. Masked image modelling — MAE. Mask out 75% of the patches of an image, ask the model to reconstruct them. MAE (He, Chen et al. 2021). Trivially scalable, no negative pairs needed. Now the default pretraining objective for ViT-scale models.
3. Vision-language contrastive — CLIP / SigLIP. Pair each image with its caption; maximise the dot product between matched image-text embeddings. CLIP (Radford et al., OpenAI 2021) trained on 400M image-text pairs; SigLIP (Zhai et al., DeepMind 2023) uses a sigmoid loss instead of softmax for better scalability; **SigLIP-2 (Tschannen et al., DeepMind 2025)** adds decoder-based captioning objectives and self-distillation — the current SOTA multilingual vision-language backbone.

The 2024–2025 recommended stack — what to actually fine-tune from:

• DINOv2 (Oquab et al., Meta AI 2023) — ViT-g/14 trained self-supervised on 142M curated images. Frozen features + linear probe already matches most supervised backbones. Fine-tuned features beat everything on segmentation, depth estimation, and dense-prediction tasks. If you're doing dense-prediction vision in 2025, this is the default.
• SigLIP-2 (2025) — vision encoder used inside Gemma 3, PaliGemma 2, and a growing list of open multimodal models. Great when you need image↔text alignment (VQA, captioning, multimodal retrieval).
• EVA-02 (Fang et al. 2023) — combines MAE pretraining with CLIP-based feature distillation. Strong classification + dense-prediction.
• SAM 2 (Ravi et al., Meta AI 2024) — Segment Anything's video-and-image successor. Not a classification backbone but the reference for segmentation transfer.

The practical decision tree in 2025:

1. Small custom dataset (< 5K labelled images), simple classification: torchvision.models.resnet18(weights='DEFAULT') + linear probe. Still works. Still fine.
2. Medium dataset (5K–100K), classification or moderately complex task: timm.create_model('convnext_base.fb_in22k_ft_in1k', pretrained=True) + discriminative-LR fine-tune. Reliable 85-92% territory.
3. Any dense-prediction task (segmentation, depth, detection): DINOv2 frozen features + task-specific head. Or fine-tune DINOv2 end-to-end with a very small LR.
4. Multimodal (image + text): SigLIP-2 encoder + LLM head. This is how PaliGemma 2, Molmo, Qwen2-VL, and the newer Llama 4 vision variant are built.

A note on when transfer stops helping: if your target domain is very different from natural images — medical microscopy, satellite imagery, radar, molecular structure — ImageNet or LAION features are less useful. Domain-specific pretrained models exist: RadImageNet, SatMAE, MoleculeNet. Use those when available. If you have >1M labelled samples in your domain, you're now in "pretrain your own" territory.

Further reading · transfer & self-supervision in 2025

• Sharif Razavian et al. (2014) — CNN Features off-the-shelf. The paper that started it all.
• He et al. (2021) — Masked Autoencoders Are Scalable Vision Learners (MAE).
• Oquab et al. (2023) — DINOv2: Learning Robust Visual Features without Supervision.
• Tschannen et al. (2025) — SigLIP-2: Multilingual Vision-Language Encoders with Improved Semantic Understanding.
• Ravi et al. (2024) — SAM 2: Segment Anything in Images and Videos.
• Karpathy — "A Recipe for Training Neural Networks", section on "transfer learning". Old but still gold.

Retention scaffold

One-line summary (write it in your own words): ______________________________________

Spaced review: re-write the discriminative-LR loop from memory in 24 hours. On day 7, take a small target dataset of your choice and run all three regimes end-to-end; log the accuracy differences.

Next session (S029): Data augmentation deep dive — RandAugment, MixUp, CutMix, and why they're worth 3-5% on any vision task.

Sticky note:

Head LR = 10× backbone LR · weights.transforms() · Freeze BN properly — the three-line summary of applied transfer learning in 2025.

Legacy recall drills

Retention scaffold

Recall

Stretch

Fine-tune a pretrained ResNet-18 on Oxford-IIIT Pets (37 classes, 200 imgs per class). Compare linear probe vs full fine-tune vs discriminative LR. Aim for 90%+ with discriminative LR, in 10 epochs, on a laptop GPU.

In your own words

Write 3 sentences: "when would I fine-tune vs train from scratch?"

Spaced review

• S021 (BatchNorm) — the .eval() gotcha lives at the intersection of BN and freezing.
• S023 (LR schedules) — cosine, applied over 10 epochs instead of 100.

Next session

S029 — Data augmentation deep dive. RandAugment, Mixup, CutMix, and why they are worth 3–5% on any vision task.

Bring back tomorrow

• Discriminative LR: backbone 1e-3, head 1e-2, cosine over ~10 epochs.
• weights.transforms() for the preprocessing.
• Freeze BN with .eval() explicitly, not just requires_grad=False.

Previous: ← DL S027 · Next: DL S029 →