Transfer Learning & Fine-Tuning Classical DL

Why this session exists

Transfer learning reuses a learned representation and adapts it to a new task. It can reduce target-data needs, but a pretrained model is neither universally suitable nor immune to preprocessing, split and optimizer mistakes. We will distinguish frozen features, partial/full fine-tuning and scratch training, implement separate learning-rate groups and normalization-state policies, and specify a local-only torchvision ResNet workflow. CPU fixtures verify the training machinery; they do not reproduce an ImageNet/Hymenoptera result.

Prerequisites

S102 CNNs, S100 modules and state and S104 representations. Know that a parameter, a buffer and a module's training flag are different objects.

(a) Intuition · 5 min

Hiring an experienced chef can save basic training, but their cuisine and equipment may not match your restaurant. Likewise, low-level filters may transfer while task-specific outputs need replacement; none of this yields a numerical guarantee like '1,000 images gives 85% in an hour'. Renovating a house is a useful second analogy: preserve useful structure, but inspect whether the foundation actually suits the new use before deciding how gently to modify it.

The three transfer-learning strategies

1. Frozen feature extraction: hold backbone parameters and chosen buffers fixed; train a new head. Deterministic features can be cached.
2. Partial or full fine-tuning: initialize from pretrained weights and update selected or all parameters. Full fine-tuning is still transfer, regardless of whether its learning rate is small.
3. Scratch training: random initialization, rather than the earlier contradictory definition 'full retraining from pretrained weights'. Compare when architecture/domain/data justify it.

Catastrophic forgetting is loss of previously useful behavior after adaptation. Poor target validation by itself is not proof of forgetting: preprocessing errors, overfitting and unsuitable features can look similar. Evaluate a retained/source task when that behavior matters. Lower backbone rates, head warmup, early stopping and parameter-efficient updates are tools, not a rule that the same LR destroys every backbone within 100 steps.

The four-quadrant decision matrix

These are hypotheses, not thresholds at 1K/10K/100K images. Patient count, label quality and correlation can matter more than nominal image count. X-rays are not 'similar to ImageNet because both are photos'. Early-layer features have many channels and large spatial extent: a probe over block 2 is not free or automatically best for distant domains.

Transferability varies across layers and domains

Yosinski et al. (2014) identify both specialization and optimization difficulty from splitting co-adapted features. Transferability decreases with task distance in their experiments; 'early universally useful, late useless' is too strong. A source head may remain useful for an unchanged task or as features; replace it when target outputs require a new mapping, not merely because it has 1,000 outputs.

The universal fine-tuning recipe

Treat this as a measured staged recipe, not a universal guarantee. Record checkpoint identity/license and preprocessing, split target data by the true deployment unit, replace the head, establish a frozen baseline, then optionally unfreeze higher blocks or everything with declared rates and buffers. Compare checkpoints by validation and assess the reserved test only after choices are fixed.

For SGD group $l$, $\Delta\theta_l=-\eta_l g_l$. The useful diagnostic is the relative update $\|\Delta\theta_l\|/(\|\theta_l\|+\epsilon)$, not LR alone: gradient norms and adaptive preconditioning vary by layer. A random head can send poorly adapted gradients, but they are not necessarily meaningless noise. ULMFiT studies discriminative fine-tuning and gradual unfreezing; it does not prove these are mandatory for all CNNs or that LR is the only way to preserve representations.

Data preprocessing is the silent killer

Use the selected weight enum's inference preset as the documented evaluation contract. It is not literally the full stochastic pretraining pipeline. Torchvision 0.24.0 ResNet18_Weights.IMAGENET1K_V1 uses the ImageClassification preset with default shorter-side resize 256 and center crop 224; ResNet50_Weights.IMAGENET1K_V2 explicitly uses resize 232, not 256. The preset converts to float and normalizes with its configured mean/std. Training recipes can add validated augmentations; identical preprocessing at every stage is not an absolute law.

Inspect RGB/channel order, integer versus float scale, interpolation/antialiasing, class indices and transformed samples. A float tensor already in [0,255] is not equivalent to uint8 conversion; a library cannot infer your units. Medical channels may require replication or a redesigned first layer, neither guaranteeing preserved transfer. Shifted activation statistics suggest a diagnostic, not proof that preprocessing rather than domain mismatch caused the error.

Feature extraction vs fine-tuning at a glance

Freezing requires_grad blocks parameter-gradient accumulation, not BatchNorm running-stat updates or dropout randomness. Decide mode separately. A frozen deterministic prefix can be cached even if later blocks train; unfreezing the final block does not force recomputation of every earlier gradient. Caches are invalid if preprocessing, augmentation policy or prefix state changes. Frozen weights eliminate parameter forgetting there, but drifting buffers can still change behavior.

Partial tuning adapts some features with fewer optimizer states; full tuning increases flexibility and risk, not guaranteed highest validation accuracy. Scratch is a valid baseline even on related domains. No fixed '80% of possible quality' or 10× epoch factor is defensible without a measured workload.

(c) Hands-on · 25 min

The first standalone program implements the exact phase-management and epoch functions used by the local ResNet recipe below. A small network with named layer4 and fc modules is a contract fixture, not a pretrained ResNet or a transfer-performance experiment. The tests verify unchanged frozen parameters/buffers, changed head/last block, unique optimizer membership and the BatchNorm negative control.

import copy
import torch
from torch import nn
from torch.nn import functional as F
from torch.utils.data import DataLoader,TensorDataset
 
torch.set_num_threads(1);torch.manual_seed(13)
def configure(model,phase):
    if phase not in ['head','last','all']:raise ValueError('unknown phase')
    for p in model.parameters():p.requires_grad_(False);p.grad=None
    for p in model.fc.parameters():p.requires_grad_(True)
    if phase=='last':
        for p in model.layer4.parameters():p.requires_grad_(True)
    if phase=='all':
        for p in model.parameters():p.requires_grad_(True)
    head=list(model.fc.parameters());head_ids={id(p) for p in head}
    body=[p for p in model.parameters() if p.requires_grad and id(p) not in head_ids]
    groups=[{'params':head,'lr':1e-2}]
    if body:groups.append({'params':body,'lr':1e-3})
    ids=[id(p) for g in groups for p in g['params']]
    assert len(ids)==len(set(ids))
    assert set(ids)=={id(p) for p in model.parameters() if p.requires_grad}
    return torch.optim.SGD(groups,momentum=.9)
 
def epoch(model,loader,opt=None):
    model.train(opt is not None)
    # Chosen policy: retain pretrained BN statistics in EVERY phase.
    for m in model.modules():
        if isinstance(m,nn.modules.batchnorm._BatchNorm):m.eval()
    total=correct=n=0
    with torch.set_grad_enabled(opt is not None):
        for x,y in loader:
            if opt:opt.zero_grad(set_to_none=True)
            logits=model(x);loss=F.cross_entropy(logits,y)
            if opt:loss.backward();opt.step()
            total+=len(y)*loss.item();correct+=(logits.argmax(-1)==y).sum().item();n+=len(y)
    if n==0:raise ValueError('empty split')
    return total/n,correct/n
 
def staged_fit(model,train,val,epochs=(2,2)):
    best=float('inf');best_state=None
    for phase,count in zip(['head','last'],epochs):
        opt=configure(model,phase) # deliberately resets momentum at phase boundary
        for e in range(count):
            tr=epoch(model,train,opt);va=epoch(model,val)
            print(phase,e,tr,va)
            if va[0]<best:best=va[0];best_state=copy.deepcopy(model.state_dict())
    if best_state is None:raise ValueError('need at least one epoch')
    model.load_state_dict(best_state);return model
 
class ContractNet(nn.Module):
    def __init__(self):
        super().__init__();self.stem=nn.Sequential(nn.Linear(4,8),nn.BatchNorm1d(8),nn.ReLU())
        self.layer4=nn.Sequential(nn.Linear(8,8),nn.BatchNorm1d(8),nn.ReLU())
        self.fc=nn.Linear(8,2)
    def forward(self,x):return self.fc(self.layer4(self.stem(x)))
 
m=ContractNet();x=torch.randn(24,4);y=(x[:,0]>0).long()
ld=DataLoader(TensorDataset(x,y),batch_size=6)
initial=copy.deepcopy(m.state_dict());opt=configure(m,'head');epoch(m,ld,opt)
for k,v in initial.items():
    if not k.startswith('fc.'):assert torch.equal(v,m.state_dict()[k]),k
assert not torch.equal(initial['fc.weight'],m.fc.weight)
before=copy.deepcopy(m.state_dict());opt=configure(m,'last');epoch(m,ld,opt)
assert torch.equal(before['stem.0.weight'],m.stem[0].weight)
assert not torch.equal(before['layer4.0.weight'],m.layer4[0].weight)
for name,buffer in m.named_buffers():assert torch.equal(before[name],buffer)
# Negative control: requires_grad alone does NOT freeze buffers.
bad=ContractNet();configure(bad,'head');stats=bad.stem[1].running_mean.clone()
bad.train();bad(x);assert not torch.equal(stats,bad.stem[1].running_mean)
# Exact group rates and coverage.
assert [g['lr'] for g in opt.param_groups]==[.01,.001]
staged_fit(ContractNet(),ld,DataLoader(TensorDataset(x[:6]+.1,y[:6])),(1,1))
print('fixture only: freezing, BN, staged-fit and optimizer contracts passed')

Locally provisioned ResNet and custom images

This separate recipe is source-checked, not executed here: torchvision, the checkpoint and image dataset are not installed/downloaded for the audit. Run it only after provisioning trusted compatible local artifacts. It uses the functions from the first block; nothing auto-downloads. Dataset directories must be split beforehand by patient/object/site as appropriate. It deliberately rejects class-map drift.

# Define after the first block. Calling this requires local torchvision 0.24.0,
# a trusted official ResNet-18 state dict, and train/val/test ImageFolder trees.
def local_resnet_recipe(root,checkpoint,epochs=(5,5)):
    from pathlib import Path
    from torchvision import datasets,models,transforms
    from torchvision.models import ResNet18_Weights
    weights=ResNet18_Weights.IMAGENET1K_V1
    evaluation=weights.transforms()
    training=transforms.Compose([transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),transforms.ToTensor(),
        transforms.Normalize(evaluation.mean,evaluation.std)])
    root=Path(root)
    ds={split:datasets.ImageFolder(root/split,transform=training if split=='train' else evaluation)
        for split in ['train','val','test']}
    assert ds['train'].class_to_idx==ds['val'].class_to_idx==ds['test'].class_to_idx
    if any(len(d)==0 for d in ds.values()):raise ValueError('empty split')
    model=models.resnet18(weights=None) # never contacts model hosting
    model.load_state_dict(torch.load(checkpoint,map_location='cpu',weights_only=True),strict=True)
    model.fc=nn.Linear(model.fc.in_features,len(ds['train'].classes))
    loaders={s:DataLoader(d,batch_size=8,shuffle=s=='train',num_workers=0) for s,d in ds.items()}
    staged_fit(model,loaders['train'],loaders['val'],epochs)
    print('reserved target test',epoch(model,loaders['test']))
    return model,ds['train'].class_to_idx

Anatomy of the script

ResNet-18's original 1,000-class model has 11,689,512 parameters in the pinned source. Replacing its 512-input classifier by a binary head makes 1,026 head parameters; inspect fc.in_features rather than hardcoding it for another family. New modules are trainable by default. Parameter groups must include newly unfrozen parameters exactly once. Rebuilding SGD resets momentum here; preserving optimizer state with add_param_group is another deliberate policy, and schedulers must match the chosen lifecycle.

The frozen-BatchNorm policy is explicit and reapplied after each train() call. BatchNorm affine parameters can train while running statistics remain fixed. Dropout elsewhere still uses the model's training flag; if a fully frozen feature extractor contains dropout, set that prefix to eval as well before caching. A full-model no_grad() would wrongly disable head learning; a frozen prefix with ordinary input often needs no parameter gradient graph anyway.

Validation uses eval/no-grad and sample-weighted metrics, and all phases compete for the same best cloned state. Real checkpoint persistence must retain architecture, class map, preprocessing, checkpoint hash/license, dependency versions and, for resumption, optimizer/scheduler/RNG state. weights_only=True is restricted deserialization, not permission to trust arbitrary artifacts.

Forgetting experiment with an answer

Compare frozen, last-block, all-layer and scratch baselines with fixed splits and matched tuning budgets. Sweep rates; measure target validation and a retained source-domain task, representation drift and layer update norms. If all-layer tuning is worse, inspect class mapping, buffer drift and overfitting before naming forgetting. If it is better, report that rather than forcing a 15–25-point collapse. The CPU fixture proves a state bug can exist without gradient updates, not that a pretrained network has forgotten knowledge.

(d) Production reality · 15 min

Clinical adaptation: evaluate patient/site/time shift and image-quality workflows; domain-adaptive self-supervision may help but requires independent validation and permissible data. The earlier Google clinic-count, accuracy-drop and retinal-pretraining recovery story is not evidence here.

Language adaptation and LoRA: a low-rank update writes $W'=W+(\alpha/r)BA$, with $A\in\mathbb R^{r\times d_{in}}$, $B\in\mathbb R^{d_{out}\times r}$. Trainable matrices use $r(d_{in}+d_{out})$ parameters versus $d_{in}d_{out}$ for a full update. This can reduce gradients/optimizer state, but total memory still includes base weights and activations. Frozen base weights do not prevent the adapted function from forgetting or behaving badly. Model size alone does not determine the correct LR; no unsupported GPT-2 tutorial incident or 0.1%-of-total-memory promise is retained.

Spurious photo-quality cues: split by listing/object, balance evaluation slices and test counterfactual nuisance changes. Saliency/Grad-CAM can suggest a shortcut but cannot prove causal dependence or its absence. Compare actual task outcomes rather than an invented Airbnb accuracy/conversion story.

Where this shows up in the rest of the plan

BERT, audio encoders, CLIP and two-tower systems reuse the same questions—what representation, what target loss, what trainable state—but not identical preprocessing or heads. Parameter-efficient adaptation changes the parameterization, not the need for splits, validation and retained-behavior checks.

(e) Recall + stretch · 10 min

1. Early transfer better? Often more generic in studied CNNs, but co-adaptation and task distance matter; not universal.
2. Extraction versus tuning? Fixed features versus changing some representation parameters/state; define buffers too.
3. Forgetting? Loss of retained behavior after adaptation; compare that behavior and tune constraints rather than assume one LR ratio prevents it.
4. Preprocessing? Match the particular weight preset and input units, with separately chosen training augmentations; not a universal constant pipeline.
5. 200 X-rays? Frozen probe and conservative domain-appropriate tuning are baselines; patient-level uncertainty and leakage dominate. There is no theorem forbidding unfreezing.
6. 500K lesion images? Compare source/domain pretraining and scratch with patient/site splits; nominal image count alone does not decide.
7. Discriminative rates? Separate parameter groups encode different update priors; inspect actual relative updates.
8. LoRA? Useful under adaptation-state/storage constraints, including smaller models; base-weight/activation memory remains.
9. 95% train, 72% validation? Check split/metric/transform mismatch, then tune regularization, augmentation, freezing and capacity. A smaller LR plus longer training is not automatically a cure.
10. 20K labeled and 5M unlabeled satellite tiles? Hold out geography/time before any target adaptation, compare ImageNet/domain initialization, self-supervise on permitted training-region tiles, and fine-tune on labels. Audit multispectral channels, spatial overlap and license; no assumed 5–10-point gain.

Explain-out-loud test

Explain why changing no weights can still change BatchNorm outputs, trace two optimizer groups through unfreezing, and design a test distinguishing forgetting from target overfitting.

What comes next

Tokenization studies the representation boundary that must remain compatible with pretrained text weights. Learning-plan hub.

Primary references

• Yosinski et al., ULMFiT, LoRA: transfer constraints and adaptation methods.
• torchvision 0.24 ResNet and presets: exact evaluation contracts.