Data augmentation: tested mixing, task assumptions and calibration

An augmentation is a claim about a task: some transformation preserves the target, or a constructed input should receive a specified constructed target. Counted augmented tensors are not independent new observations. The useful question is what assumption each transformation introduces—and whether that assumption survives your labels,distribution and evaluation.

From examples to an invariance assumption

The mixup paper proposes convex combinations of examples and labels as a regularization principle. It does not establish the old lesson's invented lunch story,universal calibration improvement or additive3–5% gain on every vision task. Ordinary augmentation trains against transformed views; mixup supplies a target between examples. Both can help or harm.

If T is a label-preserving transformation, the training objective can average $\ell(f(T(x)),y)$ over T. This encourages similar answers under those transformations without proving perfect invariance. For segmentation,object detection or coordinate regression, transform targets too: image rotation without rotating masks/boxes is label corruption. Crops can remove the only labeled object. A flip may preserve “cat” but change “left hand,”text orientation or medical laterality.

Prerequisites are DL027's controlled training/evaluation and DL028's preprocessing contract. Outcomes: distinguish geometric,photometric,occlusion and label-mixing operations; use the actual RandAugment API; implement and test mixup/CutMix; compute soft-target loss and calibration; and design controlled ablations with worked answers rather than prefilled gains.

Taxonomy and pipeline order

• Geometric: crop,resize,flip,rotate,affine warp. These change spatial support and often pixel values too through interpolation/fill, so “preserve pixel values” is not universally true.
• Photometric: brightness,contrast,saturation,hue,grayscale and blur. Diagnostic color may be label-bearing. Blur also combines neighboring pixels.
• Occlusion: Cutout/RandomErasing masks regions while normally retaining the original label. It does not mix labels by definition.
• Policy selection: RandAugment chooses operations from a pool; it is not itself a label mixer.
• Example mixing: mixup blends full inputs/targets; CutMix replaces a rectangle and weights labels by retained area. They are commonly batch operations.

A torchvision-v1 PIL pipeline often uses crop/flip/policy→ToTensor→Normalize→tensor erasing→batch mixing. This is an API-specific practical ordering,not a universal law: v2 supports tensor inputs,and supported dtype/range matters. RandomErasing works on tensors; after normalization its fill0 means the channel mean in raw image space, not black. Before normalization the same literal fill has another meaning. Hue jitter on normalized values outside its expected range is not interchangeable with jitter before conversion.

def local_augmentation_transforms(mean=(.4914,.4822,.4465),std=(.2470,.2435,.2616),
                                  policy='baseline'):
    # Recipe only: requires an already installed torchvision0.24.
    from torchvision import transforms as T
    steps=[T.RandomCrop(32,padding=4,padding_mode='reflect'),T.RandomHorizontalFlip()]
    if policy=='randaugment': steps += [T.RandAugment(num_ops=2,magnitude=9)]
    elif policy=='photometric': steps += [T.ColorJitter(.2,.2,.2)]
    elif policy!='baseline': raise ValueError('unknown policy')
    steps += [T.ToTensor(),T.Normalize(mean,std)]
    if policy=='photometric': steps += [T.RandomErasing(p=.25,scale=(.02,.2),value=0)]
    return T.Compose(steps),T.Compose([T.ToTensor(),T.Normalize(mean,std)])

RandomResizedCrop32 with scale(.7,1) is an alternative crop policy: it samples area and aspect ratio then resizes,not simply a translated32-pixel window. Compare it as a separate change. ColorJitter.2 and RandomErasing.25 are candidate strengths,not a safe rule for every tomato,document or medical task.

RandAugment and the limits of automatic policies

RandAugment's N is the number of sequential sampled operations; M chooses their magnitude on the implementation's discrete scale. It removes a separate expensive proxy search by reducing the search space,not by making magnitude untunable. The paper explicitly discusses adjusting strength to model and dataset size. (2,9) is a candidate baseline,not “safe everywhere.”

Torchvision0.24's pool includes Identity,ShearX/Y,TranslateX/Y,Rotate,Brightness,Color,Contrast,Sharpness,Posterize,Solarize,AutoContrast and Equalize. The precise pool is version-specific: copying an old paper list including Invert would misdescribe this implementation. Some operations have no meaningful magnitude; signed operations may randomly choose direction. Interpolation and fill affect geometry and need to match image semantics.

AutoAugment searches subpolicies with operation probabilities/magnitudes. TrivialAugment samples a single operation and magnitude; its “parameter-free” result is relative to its defined search space,not absence of all implementation choices. AugMix blends augmentation chains and studies robustness/uncertainty; it does not require changing the semantic class label. DeiTIII's3-Augment recipe explores grayscale,solarization and blur as a simplified training design,not proof that three operations always beat every other policy.

Mixup: derive the image and label contract

For paired samples i,j,

Labels here are distributions,not integer IDs. With cat/dog and λ.7,the target is(.7,.3). Alpha.2 is U-shaped,concentrating near endpoints; alpha1 is uniform and uses more middle mixtures. Negative alpha is invalid. Mixing two same-class examples still gives a one-hot target,so the original “never hard labels” claim is false.

For logits z and softmax p,the CE loss is $-\sum_c\tilde y_c\log p_c$. Because CE is linear in the target, this equals λ CE(z,y_i)+(1−λ)CE(z,y_j). The logit gradient is $p-\tilde y$ under a normalized target. A stable log-softmax should operate on logits; do not apply softmax first and then feed its probabilities to cross-entropy, which would normalize them a second time.

import numpy as np
import torch
import torch.nn.functional as F
 
def check_batch(x,y,num_classes):
    if x.ndim!=4 or not x.is_floating_point() or min(x.shape)<1:
        raise ValueError('positive-sized floating NCHW batch required')
    if y.shape!=(len(x),) or y.dtype!=torch.long or y.device!=x.device:
        raise ValueError('same-device integer labels required')
    if num_classes<2 or y.min()<0 or y.max()>=num_classes:
        raise ValueError('label outside class range')
 
def batch_permutation(x,perm):
    if perm is None: return torch.randperm(len(x),device=x.device)
    if (not isinstance(perm,torch.Tensor) or perm.dtype!=torch.long
            or perm.device!=x.device or perm.shape!=(len(x),)):
        raise ValueError('same-device one-dimensional long permutation required')
    if not torch.equal(perm.sort().values,torch.arange(len(x),device=x.device)):
        raise ValueError('not a permutation')
    return perm
 
def mixup_batch(x,y,alpha=.2,num_classes=10,*,rng=None,lam=None,perm=None):
    check_batch(x,y,num_classes)
    if not np.isfinite(alpha) or alpha<0: raise ValueError('alpha must be finite and nonnegative')
    rng=np.random.default_rng() if rng is None else rng
    if lam is None: lam=float(rng.beta(alpha,alpha)) if alpha>0 else 1.0
    if not 0<=lam<=1: raise ValueError('invalid lambda')
    perm=batch_permutation(x,perm)
    targets=F.one_hot(y,num_classes).to(x.dtype)
    return lam*x+(1-lam)*x[perm],lam*targets+(1-lam)*targets[perm]
 
x=torch.arange(2*1*4*4,dtype=torch.float64).reshape(2,1,4,4)
y=torch.tensor([0,1]); swap=torch.tensor([1,0])
xm,ym=mixup_batch(x,y,num_classes=2,lam=.7,perm=swap)
torch.testing.assert_close(ym,torch.tensor([[.7,.3],[.3,.7]],dtype=x.dtype))
assert torch.equal(mixup_batch(x,y,num_classes=2,lam=1,perm=swap)[0],x)
logits=torch.tensor([[2.,-1.],[-.2,.8]],dtype=torch.float64,requires_grad=True)
loss=F.cross_entropy(logits,ym)
manual=-(ym*F.log_softmax(logits,dim=1)).sum(1).mean()
weighted=.7*F.cross_entropy(logits,y)+.3*F.cross_entropy(logits,y[swap])
torch.testing.assert_close(loss,manual);torch.testing.assert_close(loss,weighted)
loss.backward();torch.testing.assert_close(logits.grad,(logits.softmax(1)-ym)/2)
same_y=torch.zeros(2,dtype=torch.long)
assert torch.equal(mixup_batch(x,same_y,num_classes=2,lam=.5,perm=swap)[1][:,0],torch.ones(2,dtype=x.dtype))

The helpers require positive NCHW dimensions and a same-device, one-dimensional torch.long permutation. Duplicate indices, float or Boolean index tensors and nonfinite alpha are rejected instead of silently changing pairing semantics. alpha=0 disables sampled mixup; an explicitly supplied lam is a deterministic test override. For reproducibility, seed both the NumPy generator (coefficients/boxes) and PyTorch (pairing), or supply perm explicitly.

Mixup encourages approximate linearity on sampled segments. It does not force globally linear logits or guarantee boundaries lie farther from every training sample. It also changes optimization difficulty and may need a different schedule. Applying a shared affine normalization before or after mixup is mathematically equivalent; nonlinear color policies generally do not commute.

CutMix: area after clipping, not the sampled proposal

CutMix pastes a spatial region from another image rather than making all pixels translucent. Let A be the actual pasted rectangle area out of H×W. The original image's label receives λ_actual=1−A/(HW). The area heuristic does not prove the rectangle contains that fraction of the semantic object; it is the training assumption to evaluate.

def cutmix_batch(x,y,alpha=1.,num_classes=10,*,rng=None,box=None,perm=None):
    check_batch(x,y,num_classes)
    if not np.isfinite(alpha) or alpha<=0: raise ValueError('CutMix alpha must be finite and positive')
    rng=np.random.default_rng() if rng is None else rng
    H,W=x.shape[-2:]
    perm=batch_permutation(x,perm)
    if box is None:
        proposed=float(rng.beta(alpha,alpha))
        h=int(H*np.sqrt(1-proposed));w=int(W*np.sqrt(1-proposed))
        cy=int(rng.integers(H));cx=int(rng.integers(W))
        box=(cy-h//2,cx-w//2,cy+(h-h//2),cx+(w-w//2))
    y1,x1,y2,x2=box
    y1,y2=int(np.clip(y1,0,H)),int(np.clip(y2,0,H))
    x1,x2=int(np.clip(x1,0,W)),int(np.clip(x2,0,W))
    if y2<y1 or x2<x1: raise ValueError('reversed rectangle')
    mixed=x.clone();mixed[:,:,y1:y2,x1:x2]=x[perm,:,y1:y2,x1:x2]
    lam=1-((y2-y1)*(x2-x1)/(H*W))
    target=F.one_hot(y,num_classes).to(x.dtype)
    return mixed,lam*target+(1-lam)*target[perm],lam
 
before=x.clone()
cm,cy,lam=cutmix_batch(x,y,num_classes=2,box=(-1,-1,2,2),perm=swap)
assert lam==.75  # proposal3x3 clipped to actual2x2 on4x4
assert torch.equal(x,before)  # not an in-place corruption of the source batch
assert torch.equal(cm[:,:,:2,:2],x[swap,:,:2,:2])
assert torch.equal(cm[:,:,2:,2:],x[:,:,2:,2:])
torch.testing.assert_close(cy,torch.tensor([[.75,.25],[.25,.75]],dtype=x.dtype))
wrong_lam=1-9/16
assert wrong_lam!=lam  # negative control: using unclipped area mislabels the pixels
full,full_y,l=cutmix_batch(x,y,num_classes=2,box=(0,0,4,4),perm=swap)
assert l==0 and torch.equal(full,x[swap])
empty,_,l=cutmix_batch(x,y,num_classes=2,box=(2,2,2,2),perm=swap)
assert l==1 and torch.equal(empty,x)

The original cut-size calculation used center±half and silently reduced odd side lengths. This version preserves the proposed integer size before clipping,then recomputes area. Alpha1 is more uniform than alpha.2,not “more extreme.” A50/50 choice between mixup/CutMix is one policy,not a universal timm default or theorem that the combination wins.

Label smoothing and batch training without missing steps

PyTorch's uniform smoothing is $y'=(1-\epsilon)y+\epsilon/K$. For K10 and ε.1,the true class gets.91 and each other class.01. It distributes mass over all classes,not just the nine wrong ones. Applied after mixup,it further smooths the already mixed target; the effects overlap and can overregularize. Select ε jointly with mixing strength rather than assuming a free gain.

def train_mixed_batch(model,opt,x,y,method,rng,smoothing=0.,num_classes=10):
    model.train();opt.zero_grad(set_to_none=True)
    classes=num_classes
    if method=='mixup': xx,yy=mixup_batch(x,y,num_classes=classes,rng=rng)
    elif method=='cutmix': xx,yy,_=cutmix_batch(x,y,num_classes=classes,rng=rng)
    elif method=='none': xx,yy=x,F.one_hot(y,classes).to(x.dtype)
    else: raise ValueError('unknown mixing policy')
    loss=F.cross_entropy(model(xx),yy,label_smoothing=smoothing)
    if not torch.isfinite(loss): raise FloatingPointError('nonfinite loss')
    loss.backward();opt.step()
    return loss.item()
 
# Known distribution oracle and a deliberate invalid-simplex example.
one=F.one_hot(torch.tensor([2]),10).double()
smoothed=.9*one+.1/10
assert abs(smoothed[0,2].item()-.91)<1e-12 and abs(smoothed.sum().item()-1)<1e-12
zz=torch.arange(10,dtype=torch.float64).reshape(1,10)
torch.testing.assert_close(F.cross_entropy(zz,one,label_smoothing=.1),
                          -(smoothed*zz.log_softmax(1)).sum())
bad=smoothed*2
assert not torch.allclose(bad.sum(1),torch.ones(1,dtype=bad.dtype))
# PyTorch accepts float targets; it does not validate their probability simplex.
model=torch.nn.Sequential(torch.nn.Flatten(),torch.nn.Linear(16,2)).double()
model.num_classes=2
opt=torch.optim.SGD(model.parameters(),lr=.001)
for method in ('none','mixup','cutmix'):
    assert np.isfinite(train_mixed_batch(model,opt,x,y,method,np.random.default_rng(29),.05,num_classes=2))

Use clean,unmixed evaluation for accuracy. Training accuracy against original hard labels is not directly comparable once labels are mixed; optionally report a weighted-label diagnostic with its definition. Pass the known dataset class count explicitly; do not infer it with an extra one-sample training forward in a network with batch-dependent layers. The synthetic test uses2 classes.

Calibration is measurable, not a request for peaked confidence

A confident but wrong predictor is not desirable simply because a triage system uses thresholds. Calibration compares confidence with empirical correctness; operating-point sensitivity/specificity and costs are separate. Mixup or smoothing can improve or worsen calibration,including underconfidence. Evaluate on held-out data from the intended population and relevant subgroups.

For confidence bins B_m, ECE is $\sum_m |B_m|/n\,|accuracy(B_m)-confidence(B_m)|$. It depends on binning,has sampling error and can hide subgroup failures. Report NLL and a proper score such as multiclass Brier alongside it. Empty bins contribute zero.

def calibration_metrics(logits,labels,bins=10):
    if len(labels)==0 or bins<1: raise ValueError('empty data or invalid bins')
    p=logits.softmax(1);confidence,pred=p.max(1)
    correct=(pred==labels).to(p.dtype)
    index=torch.clamp((confidence*bins).long(),max=bins-1)
    ece=logits.new_tensor(0.)
    for b in range(bins):
        mask=index==b
        if mask.any(): ece+=mask.to(p.dtype).mean()*(correct[mask].mean()-confidence[mask].mean()).abs()
    target=F.one_hot(labels,logits.shape[1]).to(p.dtype)
    return {'accuracy':correct.mean().item(),'nll':F.cross_entropy(logits,labels).item(),
            'ece':ece.item(),'brier':((p-target)**2).sum(1).mean().item()}
 
uniform=calibration_metrics(torch.zeros(2,2,dtype=torch.float64),torch.tensor([0,1]))
assert uniform['accuracy']==.5 and uniform['ece']==0
assert abs(uniform['nll']-np.log(2))<1e-12 and uniform['brier']==.5
certain_wrong=calibration_metrics(torch.tensor([[20.,-20.]]),torch.tensor([1]))
assert certain_wrong['ece']>.99 and certain_wrong['nll']>39
print('mixup, CutMix, smoothing and calibration controls passed')

Controlled experiments and failure cases

The original four-way exercise becomes: no augmentation;crop/flip;crop/flip+RandAugment;that stack+mixup. Fix splits,model,optimizer,schedule,seeds and tuning budget. Log clean validation accuracy,NLL,ECE,Brier,training time and examples of transformed batches. A short run may underfit stronger augmentation; distinguish matched compute from best-tuned quality. Keep final test data for the frozen decision. No5–10-point gain or halved ECE is predetermined.

Do not augment blindly in these cases:

• Orientation/color carries the label. Reflecting an L glyph changes the character,even if reflecting a cat preserves its category.
• Limited fit capacity or excessive strength. There is no five-layer/5000-image cutoff; inspect whether optimization can fit the intended augmented task.
• Regression. Mixup can be meaningful if the input–target interpolation models the task; it is not categorically nonsense. Blending face pixels labeled ages20/60 does not automatically imply age40. A linear measurement task can have exactly the opposite conclusion.
• Imbalance. Uniform pairing may dilute rare-class signal; examine conditional metrics and a controlled sampling policy. Resampling also changes training priors and calibration.
• Evaluation leakage. Ordinary validation transformations should be deterministic. Explicit TTA averages predictions of declared views at extra inference cost; one random crop per evaluation is not secretly a five-view ensemble.

L=torch.tensor([[1.,0.,0.],[1.,0.,0.],[1.,1.,1.]])
assert not torch.equal(L,L.flip(1))
# A valid regression interpolation: y=2*x+3 obeys mixup exactly.
a,b,weight=2.,6.,.3
assert abs((2*(weight*a+(1-weight)*b)+3)-(weight*(2*a+3)+(1-weight)*(2*b+3)))<1e-12

Augmentation, self-supervision and synthetic data

In SimCLR,two augmented views define a positive relation;changing views changes the learned invariance. DINO-style self-distillation also uses multiple views but differs from explicit-negative contrastive loss. Augmentation is a central part of the objective design,not literally the entire learning algorithm independent of teacher,loss or anti-collapse mechanisms.

Azizi et al.'s diffusion-augmentation study generates class-conditional samples and evaluates their use with real data. That does not justify a generic “hundreds of millions of images” recipe or5–10% gain on medical tasks. Generated data can preserve generator biases,leak evaluation images,mislabel classes or reduce diversity. Check provenance,rights,deduplication,class balance and held-out distribution shift. It is not mathematically the same as linear pixel interpolation.

Model soups average compatible weights of fine-tuned models sharing an initialization/architecture,not arbitrary networks from different runs. This can avoid inference-time ensemble cost but still spends training and validation compute. Repeated augmentation shows multiple views of an example during training; a mixup-off final phase is a schedule choice. Neither implies a universal80/20 policy or fixed.5% reward.

Text transforms require their own semantics: synonym replacement may change technical meaning,back-translation can erase details,sentence shuffling can destroy order,and synthetic instruction/response generation needs factual and license checks. The claim that every named2024–25 model used millions of GPT4 pairs in pretraining is unsupported; supervised post-training and pretraining are distinct stages. Do not use a vision invariance analogy as evidence of any model's undisclosed corpus.

Worked answers and retention

1. Mixup's insight: paired input interpolation and matching soft targets impose a sampled smoothness/linearity prior. Improvement is an empirical result,not a guarantee.
2. CutMix coefficient: λ_actual=1−actual clipped area/(HW). The worked4×4 case pastes2×2 after clipping,so λ=.75 rather than.4375 from the untrimmed3×3 proposal.
3. Three ways to hurt: label-changing transformations,overregularized/underfit training and shifted subgroup/calibration behavior. Regression is allowed when interpolation is justified; confident medical outputs are not inherently calibrated.
4. Order: follow each operation's dtype/range and target contract. The supplied v1 recipe uses per-image transforms then tensor normalization and batch mixing; this is not a theorem that all geometric and photometric operations commute or that all libraries require PIL.
5. RandAugment N/M: operation count and magnitude index; inspect the versioned pool and tune both on validation.
6. Self-supervised views: views define information the representation should retain across transformations. The loss/teacher determines how agreement is learned. Destructive crops may remove the only semantic evidence,and not all objectives use negative pairs.
7. Why smoothing plus mixup is not a free lunch: both soften targets;ε changes an already mixed optimum. Measure calibration and task quality rather than summing advertised gains.
8. When to turn augmentation off: deterministic reference evaluation,invalid task transformations or a measured harmful training policy. TTA is a separate declared exception; no universal regression ban.

Stretch: integrate the displayed batch mixers into DL027's training loop after batching,before loss,and leave validation untouched. Compare policies at matched seeds/budget;94.5% is a possible chosen target,not a result here. Tomorrow derive CE linearity in the target and the clipped-area formula. On day7 inspect actual transformed local images if available before spending a long run.

Sources and execution boundary

Primary sources: mixup, CutMix, RandAugment, torchvision0.24 operation pool, CrossEntropyLoss2.9, AutoAugment, AugMix, TrivialAugment, DeiTIII, SimCLR, synthetic data study, model soups.

The original optional video IDs bNb2fEVKeEo and ArPaAX_PhIs carried specific channel, title, duration and timestamp claims. These are unverified and are not endorsed here. The versioned transform source and the mixup/RandAugment papers above cover the associated mechanisms without relying on those attributions.

CPU execution covers the exact displayed mixing,loss,calibration and counterexample programs. The torchvision transform factory and full CIFAR/paper-scale experiments are unexecuted. No generator,weights,datasets or GPU resources were downloaded. Previous: transfer · Next: ViT.