Transfer learning: frozen features, fine-tuning and buffer discipline

Transfer learning changes the starting representation and which parts may adapt. A useful experiment compares a frozen feature extractor, a partly unfrozen model and a full fine-tune on the same split. It does not assume an image-count threshold or guaranteed improvement.

What transfers, and why that is an empirical question

Zeiler and Fergus visualized intermediate convnet features and studied layer contributions. Razavian et al.'s2014 CNN Features off-the-shelf used OverFeat, not simply an AlexNet checkpoint, as a4096-dimensional generic representation with a linear SVM or retrieval distance. Their strong results motivated reusable visual features; they do not prove every future medical or satellite task is solved by ImageNet.

Yosinski et al.'s transfer study identifies two obstacles: higher-layer task specialization and optimization difficulties from splitting co-adapted neurons. Early Gabor-like/color features often transfer broadly, but “every early layer is universal” and “specificity is strictly monotone at every layer” are too strong. Domain distance often matters; even distant transfer can outperform random features. Compare frozen probes at several depths if the final representation is poor, rather than diagnosing domain distance from one score alone.

The vocabulary analogy is useful: learning Italian after Spanish may reuse patterns, but the shared patterns can also mislead. ImageNet features likewise encode source-data biases. Freezing limits what moves; it does not eliminate downstream overfitting, licensing constraints or source/target leakage.

Outcomes. Replace a ResNet18 head, match a checkpoint's preprocessing, implement linear/partial/full/discriminative-LR adaptation, control BatchNorm buffers, verify optimizer ownership and investigate forgetting. Prerequisites are DL026's stages and DL027's split/evaluation discipline. This lesson provides a complete opt-in local checkpoint and local image-folder recipe; no weights or dataset are fetched.

Three adaptation regimes, not three sample-count laws

Discriminative LR is a parameter-group policy, not a mutually exclusive amount of unfreezing. Gradual unfreezing changes which parameters train over time. Layerwise LR decay changes how fast existing trainable parameters move. Neither guarantees a1% gain.

A head LR ten times the backbone LR is a starting trial, not a theorem that2× destroys features or100× prevents learning. Effective updates depend on gradient scale,optimizer,weight decay and normalization. Keep a linear-probe baseline,then tune on validation. A randomly initialized frozen backbone is a useful random-features control; a fully trainable randomly initialized network is a different from-scratch baseline.

Checkpoint identity, swapped head and preprocessing

Torchvision0.24's ResNet18 ImageNet1K_V1 has a512-feature final representation and1000-way classifier. Its standard float32 parameter storage is about46.8MB (11,689,512 parameters times4), not25MB; no fixed “90 GPU-days” follows from the file. The local loader below instantiates weights=None, loads a caller-supplied trusted file, then replaces its head. It refuses missing files or a digest mismatch. It neither contacts a hub nor treats synthetic tests as pretrained execution.

The weights' evaluation preset resizes the shorter side to256,crops224,converts to float and normalizes RGB with means(.485,.456,.406),stds(.229,.224,.225). Read the chosen weight enum's actual transform: another checkpoint can have a different crop,interpolation,antialias or normalization policy. Training may add appropriate random crops/flips, but evaluation stays deterministic.

Matching the reference preprocessing establishes a baseline. It is not a theorem that no alternative normalization can ever be learned during full adaptation. Wrong preprocessing can shift features silently, so change it only as an explicit experiment. A frozen eval backbone plus deterministic input pipeline is cacheable; training dropout,stochastic transforms or buffer updates invalidate that cache contract.

import copy, hashlib
from pathlib import Path
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset
 
BN_TYPES=(nn.BatchNorm1d,nn.BatchNorm2d,nn.BatchNorm3d)
 
def load_local_resnet(checkpoint, expected_sha256, classes, from_scratch=False):
    from torchvision.models import resnet18
    if classes < 2: raise ValueError('need at least two classes')
    model=resnet18(weights=None)
    if not from_scratch:
        path=Path(checkpoint)
        if not path.is_file(): raise FileNotFoundError(path)
        if hashlib.sha256(path.read_bytes()).hexdigest()!=expected_sha256:
            raise ValueError('checkpoint digest mismatch')
        state=torch.load(path,map_location='cpu',weights_only=True)
        model.load_state_dict(state,strict=True)  # explicit 1000-class state_dict contract
    model.fc=nn.Linear(model.fc.in_features,classes)
    return model
 
def freeze_bn(model):
    for module in model.modules():
        if isinstance(module,BN_TYPES):
            module.eval()
            for p in module.parameters(): p.requires_grad_(False)
 
def configure_transfer(model, regime, freeze_stats=True):
    if regime not in ('linear','last','full','discriminative'):
        raise ValueError('unknown transfer regime')
    for name,p in model.named_parameters():
        p.grad=None  # discard stale gradients before changing optimizer ownership
        p.requires_grad_(regime in ('full','discriminative') or name.startswith('fc.')
                         or (regime=='last' and name.startswith('layer4.')))
    if freeze_stats: freeze_bn(model)
 
def transfer_mode(model, regime, training, freeze_stats=True):
    model.train(training)
    if training and regime=='linear':
        model.eval(); model.fc.train()  # immutable whole backbone, not just no gradients
    elif training and freeze_stats:
        freeze_bn(model)  # train() above recursively re-enabled all children
 
# Every trainable parameter appears exactly once, including the stem's bn1.
def transfer_groups(model, regime, head_lr=.01, backbone_lr=.001):
    groups={}
    for name,p in model.named_parameters():
        if not p.requires_grad: continue
        lr=head_lr if name.startswith('fc.') else backbone_lr
        if regime=='full': lr=backbone_lr
        decay=0.0 if p.ndim <= 1 else 1e-4
        groups.setdefault((lr,decay),[]).append(p)
    ids=[id(p) for ps in groups.values() for p in ps]
    expected={id(p) for p in model.parameters() if p.requires_grad}
    assert len(ids)==len(set(ids)) and set(ids)==expected
    return [{'params':ps,'lr':lr,'weight_decay':wd} for (lr,wd),ps in groups.items()]
 
def stage_groups(model, head_lr=.01, ratio=.5):
    # torchvision names; include conv1 AND bn1, unlike the old omitted-bn1 example.
    stage={'conv1':0,'bn1':0,'layer1':1,'layer2':2,'layer3':3,'layer4':4,'fc':5}
    groups={}
    for name,p in model.named_parameters():
        if not p.requires_grad: continue
        prefix=name.split('.')[0]
        if prefix not in stage: raise ValueError('unassigned parameter '+name)
        lr=head_lr*ratio**(5-stage[prefix])
        wd=0.0 if p.ndim<=1 else 1e-4
        groups.setdefault((lr,wd),[]).append(p)
    rows=[{'params':ps,'lr':lr,'weight_decay':wd} for (lr,wd),ps in groups.items()]
    ids=[id(p) for row in rows for p in row['params']]
    assert len(ids)==len(set(ids))==sum(p.requires_grad for p in model.parameters())
    return rows

When changing regimes between optimizer steps, clear existing gradients before freezing parameters: requires_grad_(False) prevents future gradient construction but does not erase a populated .grad. A retained optimizer can otherwise update a newly frozen tensor from that stale gradient, including momentum or weight decay. configure_transfer clears every parameter's gradient; call it at a stage boundary, not halfway through an accumulated update. Rebuild the optimizer from the newly selected groups, or deliberately reconcile its groups and state. Newly unfrozen parameters must be added; rebuilding discards old optimizer state unless migrated. A low LR in a group the optimizer never received cannot update anything.

BatchNorm: parameters, buffers and mode are separate

requires_grad=False freezes affine parameters, not the forward-pass updates to running_mean, running_var and num_batches_tracked. Call eval on frozen BN modules after model.train each epoch. Keep normal track_running_stats=True when using the checkpoint's stored statistics. The old quiz instruction to turn tracking off was wrong: a BN constructed without tracked statistics uses batch statistics even at evaluation; mutating an existing flag is also not a sound freezing recipe.

Small target batches may yield noisy estimates, so frozen statistics are a reasonable baseline. If domain shift is large and batches are representative, recalibration or trainable statistics may help; this changes the feature extractor and must be validated. Dropout is another mode-dependent source of stochastic features, even without mutable buffers.

torch.manual_seed(28)
bn=nn.BatchNorm1d(4)
for p in bn.parameters(): p.requires_grad_(False)
before=bn.running_mean.clone()
bn.train(); bn(torch.randn(16,4)*2+5)
assert not torch.equal(before,bn.running_mean)
bn.eval(); before_mean=bn.running_mean.clone(); before_var=bn.running_var.clone()
bn(torch.randn(16,4)*2+9)
assert torch.equal(before_mean,bn.running_mean) and torch.equal(before_var,bn.running_var)
# Negative control: a BN constructed without tracked stats is batch-dependent in eval.
no_stats=nn.BatchNorm1d(4,track_running_stats=False).eval()
a=torch.arange(16,dtype=torch.float32).reshape(4,4)
b=a.clone(); b[1:]+=100
assert not torch.allclose(no_stats(a)[0],no_stats(b)[0])

This repairs the original vacuous test that compared a just-cloned running mean with itself. The before snapshot must be taken before the forward pass.

Complete local-only adaptation recipe

Use root/train/<class>/, root/val/<class>/, root/test/<class>/ with class names identical across splits. Split by independent subject/object/source where appropriate, not merely random near-duplicate frames. The helper detects exact duplicate files across splits; it cannot detect visually near-identical images or patient-level leakage without external metadata.

def local_transfer_loaders(root, batch_size=16, seed=28):
    from torchvision import datasets,transforms as T
    from torchvision.models import ResNet18_Weights
    root=Path(root)
    weights=ResNet18_Weights.IMAGENET1K_V1  # enum access/transform does not fetch weights
    evaluation=weights.transforms()
    training=T.Compose([T.RandomResizedCrop(224),T.RandomHorizontalFlip(),T.ToTensor(),
                        T.Normalize((.485,.456,.406),(.229,.224,.225))])
    ds={s:datasets.ImageFolder(root/s,transform=training if s=='train' else evaluation)
        for s in ('train','val','test')}
    mapping=ds['train'].class_to_idx
    if any(d.class_to_idx!=mapping for d in ds.values()):
        raise ValueError('split class mapping mismatch')
    seen={}; hashes={}
    for split,dataset in ds.items():
        hashes[split]=[]
        for file,label in dataset.samples:
            digest=hashlib.sha256(Path(file).read_bytes()).hexdigest()
            if digest in seen and seen[digest]!=split: raise ValueError('cross-split duplicate')
            seen[digest]=split; hashes[split].append([str(Path(file).relative_to(root)),label,digest])
    gen=torch.Generator().manual_seed(seed)
    loaders={s:DataLoader(d,batch_size=batch_size,shuffle=s=='train',num_workers=0,
                         generator=gen if s=='train' else None) for s,d in ds.items()}
    return loaders,{'class_to_idx':mapping,'files':hashes,'seed':seed,
                    'weights_contract':'ResNet18_Weights.IMAGENET1K_V1'}
 
def transfer_epoch(model, loader, regime, optimizer=None, freeze_stats=True):
    training=optimizer is not None
    transfer_mode(model,regime,training,freeze_stats)
    loss_sum=correct=count=0
    for x,y in loader:
        with torch.set_grad_enabled(training):
            logits=model(x); loss=F.cross_entropy(logits,y)
            if not torch.isfinite(loss): raise FloatingPointError('nonfinite loss')
            if training:
                optimizer.zero_grad(set_to_none=True); loss.backward(); optimizer.step()
        loss_sum+=loss.item()*len(y); correct+=(logits.argmax(1)==y).sum().item();count+=len(y)
    if count==0: raise ValueError('empty split')
    return {'loss':loss_sum/count,'accuracy':correct/count,'count':count}
 
def adapt_local(root,checkpoint,sha256,regime='linear',epochs=10,from_scratch=False):
    if epochs<1: raise ValueError('epochs must be positive')
    if from_scratch and regime!='full':
        raise ValueError('from-scratch comparison trains the full model')
    torch.manual_seed(28)
    loaders,manifest=local_transfer_loaders(root)
    model=load_local_resnet(checkpoint,sha256,len(manifest['class_to_idx']),from_scratch)
    freeze_stats=not from_scratch
    configure_transfer(model,regime,freeze_stats)
    opt=torch.optim.SGD(transfer_groups(model,regime),momentum=.9,nesterov=True)
    sched=torch.optim.lr_scheduler.CosineAnnealingLR(opt,T_max=epochs)
    best_loss=float('inf'); best=None; history=[]
    for epoch in range(epochs):
        lr_used=[g['lr'] for g in opt.param_groups]
        tr=transfer_epoch(model,loaders['train'],regime,opt,freeze_stats)
        va=transfer_epoch(model,loaders['val'],regime,freeze_stats=freeze_stats)
        history.append({'epoch':epoch,'lr_used':lr_used,'train':tr,'validation':va})
        if va['loss']<best_loss: best_loss=va['loss']; best=copy.deepcopy(model.state_dict())
        sched.step()
    model.load_state_dict(best)
    final_test=transfer_epoch(model,loaders['test'],regime,freeze_stats=freeze_stats)
    return model,history,final_test,manifest

Call this function separately per chosen regime only after local files exist. The audit does not call it. For the original200-image/Pets exercise, stratify an already-local training subset, keep validation/test fixed,record the chosen200 file IDs and compare probe,last-stage,full/discriminative and from-scratch at the same compute budget. Oxford-IIIT Pets has37 categories; do not invent a guaranteed70–85% or90%-in-ten-epochs outcome. Report class balance and several seeds; a200-image total subset differs substantially from200 images per class.

Exact CPU tests of freezing and update ownership

A miniature network supplies the same naming and BatchNorm contracts without requiring a checkpoint. This test cannot establish pretrained feature quality; it can establish that the displayed functions freeze and update what they claim.

class TinyTransfer(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1=nn.Conv2d(3,4,1,bias=False);self.bn1=nn.BatchNorm2d(4)
        self.layer1=nn.Sequential(nn.Conv2d(4,4,1),nn.ReLU())
        self.layer2=nn.Identity();self.layer3=nn.Identity()
        self.layer4=nn.Sequential(nn.Conv2d(4,4,1),nn.ReLU())
        self.fc=nn.Linear(4,3)
    def forward(self,x):
        x=F.relu(self.bn1(self.conv1(x)));x=self.layer1(x)
        x=self.layer4(self.layer3(self.layer2(x)))
        return self.fc(x.mean((2,3)))
 
x=torch.randn(8,3,6,6);y=torch.arange(8)%3
loader=DataLoader(TensorDataset(x,y),batch_size=4)
for regime in ('linear','last','full','discriminative'):
    torch.manual_seed(28); model=TinyTransfer();configure_transfer(model,regime)
    groups=transfer_groups(model,regime); opt=torch.optim.SGD(groups,momentum=.9)
    before={k:v.clone() for k,v in model.state_dict().items()}
    metrics=transfer_epoch(model,loader,regime,opt)
    after=model.state_dict()
    assert metrics['count']==8 and not torch.equal(before['fc.weight'],after['fc.weight'])
    for name,p in model.named_parameters():
        if not p.requires_grad: assert torch.equal(before[name],after[name])
    assert torch.equal(before['bn1.running_mean'],after['bn1.running_mean'])
    if regime=='linear':
        assert all(torch.equal(before[k],after[k]) for k in before if not k.startswith('fc.'))
    if regime=='last': assert not torch.equal(before['layer4.0.weight'],after['layer4.0.weight'])
# The full stage grouping includes bn1 when its affine tensors are deliberately trainable.
model=TinyTransfer();configure_transfer(model,'full',freeze_stats=False)
rows=stage_groups(model)
assert any(any(p is model.bn1.weight for p in g['params']) for g in rows)
# Deterministic frozen features can be cached; dropout would need eval too.
configure_transfer(model,'linear');transfer_mode(model,'linear',True)
with torch.no_grad(): assert torch.equal(model(x),model(x))
print('all transfer update/buffer/group controls passed')

Forgetting, negative transfer and experiment design

A very high backbone LR can overwrite useful source features; a tiny target set can also simply overfit. Diagnose using a frozen-probe control, validation loss,parameter-update norms and (where permitted) a separate source-domain retention evaluation. A falling target training loss with poor validation is not by itself proof of catastrophic forgetting. Smaller LR,head warmup,selective unfreezing,regularization or better data may help.

Feature caching trades repeated forward compute for storage. With N examples,D features and float32, storage is4ND bytes plus labels/metadata. Cache keys should include checkpoint hash,preprocessing hash,example ID,feature layer and model mode. Reusing a cache after augmentation or normalization changes silently compares different experiments.

Medical grayscale,multispectral imagery and documents may require channel adaptation,domain-specific pretraining or from-scratch controls. Repeating grayscale into RGB is a baseline,not new spectral information. Large target datasets can narrow the pretraining advantage; small datasets can still benefit from full tuning with appropriate regularization. Dataset size alone is not a reliable switch at1k,10k,100k or1M.

Modern pretraining objectives and backbone choices

These named methods restore the original survey without pretending they share one loss or one commercial deployment.

• SimCLR: contrasts two augmented views using positives and other examples as negatives. MoCo: a momentum encoder and queue support a large consistent dictionary. View construction determines which information is preserved.
• DINO: teacher–student self-distillation with an EMA teacher,centering/sharpening and multiple views, not a standard explicit-negative contrastive loss. DINOv2: combines image-level self-distillation and patch-level masked learning with curated large-scale data. Frozen features can serve classification or dense heads; no single dataset result proves it beats everything.
• MAE: an asymmetric encoder/decoder reconstructs masked image patches,typically masking a large proportion in the studied recipe. It differs from predicting teacher features. EVA-02: masked visual representation learning reconstructs language-aligned teacher features; it is not merely MAE pixel reconstruction plus a generic classifier.
• CLIP: aligns image/text embeddings with a batchwise contrastive objective. SigLIP: pairwise sigmoid image-text training removes the need for global softmax normalization. SigLIP2: adds multilingual training,self-supervision and dense/localization-oriented objectives including decoder-based training; variants support different resolution/aspect-ratio handling.
• SAM and SAM2: promptable segmentation systems; SAM2 extends to images and videos using streaming memory. They are not drop-in classification heads nor unqualified examples of purely self-supervised pretraining.
• SatMAE: masked pretraining adapted to multispectral/temporal satellite data. RadImageNet: a domain-specific radiology resource, whose modalities/license and target match need checking. MoleculeNet: a molecular-learning benchmark, not a ready-made image backbone. The old substitution of all three as interchangeable pretrained encoders was misleading.

The architecture of a named VLM must come from its own paper/model card. SigLIP2 does not retroactively power every2024 model: PaliGemma2 uses a SigLIP vision encoder with Gemma2; Molmo's paper describes an OpenAI CLIP-based vision encoder; Qwen2-VL has its own dynamic-resolution ViT design. Gemma3 uses a SigLIP-based vision component, not evidence of the generic SigLIP2 attribution made in the old text. Llama4 is a separate model family,not verified by listing it beside these encoders.

For classification,compare a manageable ResNet/ConvNeXt baseline and a suitable pretrained ViT. For dense prediction,inspect spatial-feature resolution and head requirements rather than a classification top1 score. For retrieval/VQA,check image–text alignment and projector/token interfaces. Strong frozen features may reduce label needs, but task labels,held-out evaluation and domain-specific safety checks remain necessary.

Worked answers and retained practice

1. Three regimes: probe freezes the backbone; full tuning updates it; discriminative LR changes update rates and may accompany partial or full unfreezing. Treat image-count cutoffs as heuristics,not definitions.
2. Preprocessing contract: channel/range,resize,crop,interpolation and normalization. Ask the precise weight enum's transforms; store its identity with the checkpoint.
3. Freeze BN correctly: eval after train; retain tracked running statistics; decide separately about affine gradients. Turning track_running_stats=False is not the fix. The before/after buffer test above is the worked proof.
4. Why a10× LR ratio? A random head and pretrained backbone have different adaptation needs;10× is a candidate setting. Compare2×,10×,100× on validation rather than declaring two of them categorically wrong. Likewise, a fine-tuning LR ten times smaller than a scratch LR is only a trial, not a universal conversion rule: match the optimizer, batch normalization policy, effective batch and schedule before comparing.
5. When does transfer fail? Source/target mismatch,task-specific lost information,bias,poor preprocessing or optimization can hurt. Domain distance and large target data are considerations,not sufficient proofs.
6. What do self-supervised features show about labels? Useful structure can be learned from unlabeled images or weak image–text pairs under designed objectives. That does not show labels have no future role or every unlabeled method dominates every supervised baseline.
7. Three-sentence scratch decision: establish whether a licensed matching checkpoint exists. Compare its frozen/partly tuned baseline with a controlled scratch model if feasible. Choose using target validation,compute and deployment requirements,not a universal rule.

Tomorrow rewrite parameter grouping and the BN mode order without looking. On day7 run the local comparison only if trusted files exist,record every achieved score and investigate update/buffer differences before increasing model size.

Sources and execution boundary

Primary sources: official transfer tutorial, torchvision0.24 ResNet, preset transforms, transferability study, off-the-shelf features, SimCLR, MoCo, DINO, MAE, DINOv2, SigLIP2, SAM2, PaliGemma2, Molmo, Qwen2-VL, Gemma3.

The original optional video URL (HGwBXDKFk9I) was labeled a Stanford CS231n transfer-learning lecture with a specific discriminative-LR segment. That title, channel, duration and segment are not verified here, so they are not retained as an endorsement or evidence. The official tutorial and transferability paper above are the governing reading for the same objectives.

Exact CPU tests execute all displayed pure transfer/grouping/epoch functions with TinyTransfer,including wrong-freeze controls. The torchvision/local-image/checkpoint wrapper is syntax/source reviewed,not invoked: this environment has no torchvision or downloaded checkpoint. No actual transfer-quality comparison is claimed. Previous: CIFAR · Next: augmentation.