DL S029 · Data Augmentation — RandAugment, Mixup, CutMix, and Why They Matter

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 29 / 80 · Module M05

The story

Story hook · Hongyi Zhang's weird idea

In late 2017, a PhD student named Hongyi Zhang at MIT was staring at CIFAR-10 loss curves and wondering: what if, instead of showing the network one image and one label per training example, we showed it a blend of two images and a blend of two labels? Not as a data-augmentation trick where the labels stay clean — but literally as a mixture. Image A weighted 0.7, image B weighted 0.3, and the target label is [0.7 · one_hot(A_label), 0.3 · one_hot(B_label)].

The idea sounds bizarre. You are showing the network things that aren't real. A ghost dog superimposed on a ghost cat, labelled 70% dog + 30% cat. Every intuition about supervised learning says this should confuse it. The training loss shouldn't even be well-defined — what does "correct" mean when the target isn't a real class?

The mixup paper (Zhang, Cissé, Dauphin, Lopez-Paz, ICLR 2018) showed the opposite. Mixup networks generalise better. They calibrate better (predicted probabilities more closely match empirical accuracies). They are more robust to label noise and adversarial attacks. They outperform baseline networks on CIFAR-10, CIFAR-100, ImageNet, and even speech-recognition tasks. The gain is usually 1-2% top-1 with zero architectural changes and effectively zero compute overhead.

Why? Because mixup implicitly enforces a linearity constraint: the model's predictions should be roughly linear between training samples in input space. That linearity reduces the model's tendency to memorise sharp decision boundaries around each training point, which is what overfitting looks like geometrically.

And it opened a floodgate. In the two years after mixup, the community produced a small industry of "mix images somehow" augmentations — CutMix (Yun et al. 2019) which pastes a rectangular patch of one image onto another, AugMix (Hendrycks et al. 2020) which blends multiple augmentations of the same image, RandAugment (Cubuk et al. 2019) which searched the augmentation-policy space so you don't have to, and TrivialAugment (Müller & Hutter 2021) which showed that literally picking one augmentation uniformly at random works about as well.

By 2022 the modern regularisation stack — crop + flip + RandAugment + MixUp + CutMix + label smoothing + stochastic depth — was worth 3-5% top-1 on ImageNet vs the 2015 default. That is more than every architectural improvement from AlexNet→ResNet→ConvNeXt combined. Augmentation is the single most underrated component of modern deep learning.

Today we build that stack piece by piece, understand why each piece helps, and end with the modern reference recipe you should paste into every vision project.

Why we torture the training data

Every image in your training set is one specific view of the world. A photo of a cat is that cat, in that pose, at that focal length, under that lighting. If the network overfits to that specific view, it fails on new cats.

Data augmentation is the trick where we generate infinite variations of each training image — slightly cropped, slightly rotated, colour-shifted, brightness-jittered, sometimes literally mixed with another image — and force the network to be right on all of them. The network learns the invariant content ("cat-ness") instead of the incidental details ("this cat is in the bottom-right and slightly overexposed").

The effect is systematic. On CIFAR-10, going from no augmentation to RandomCrop + Flip alone buys ~4%. Adding RandAugment buys another 1–2%. Adding CutMix buys another 0.5–1%. These are not opinion — they are consistent, replicated, no-hyperparameter-hell wins.

Prerequisites

• S027 (a CIFAR-10 training loop you can plug this into).
• S028 (fine-tuning pipelines also benefit from augmentation).

1 · The augmentation taxonomy

There are three families, applied in order:

1. Geometric. RandomCrop, RandomHorizontalFlip, RandomResizedCrop, RandomRotation, RandomAffine. Preserve pixel values, change spatial layout.
2. Photometric. ColorJitter (brightness, contrast, saturation, hue), Grayscale, GaussianBlur, RandomErasing. Preserve spatial layout, change pixel values.
3. Composition / label-mixing. Mixup, CutMix, Cutout, RandAugment. These change both the image AND the label distribution.

Standard training-time order:

Mixup/CutMix happen after batching because they combine two samples.

2 · The baseline stack (geometric + photometric)

from torchvision import transforms as T
 
train_tfm = T.Compose([
    T.RandomResizedCrop(32, scale=(0.7, 1.0)),  # zoom-crop 70–100%
    T.RandomHorizontalFlip(),
    T.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),
    T.ToTensor(),
    T.Normalize(CIFAR_MEAN, CIFAR_STD),
    T.RandomErasing(p=0.25, scale=(0.02, 0.2)),   # applied on tensor
])

Two things to notice:

• RandomErasing operates on the tensor, so it comes AFTER ToTensor and Normalize. It zeroes out a random rectangle — a very cheap form of Cutout.
• ColorJitter values around 0.1–0.3 are safe. Bigger values (0.5+) can invert colour semantics ("this used to be a red tomato and is now a green one") and can hurt.

3 · RandAugment — one policy that replaces all the tuning

RandAugment's insight: instead of hand-tuning which augmentations to use and how strong each one should be, just pick N random augmentations from a fixed list and apply each with strength M. Two hyperparameters, that's it.

train_tfm = T.Compose([
    T.RandomResizedCrop(32, scale=(0.7, 1.0)),
    T.RandomHorizontalFlip(),
    T.RandAugment(num_ops=2, magnitude=9),      # N=2 ops of magnitude 9/30
    T.ToTensor(),
    T.Normalize(CIFAR_MEAN, CIFAR_STD),
])

The 14-op menu (from the paper): AutoContrast, Equalize, Invert, Rotate, Posterize, Solarize, Color, Contrast, Brightness, Sharpness, ShearX, ShearY, TranslateX, TranslateY. Two of them, sampled uniformly per image.

Sensible defaults for CIFAR: num_ops=2, magnitude=9. Sensible defaults for ImageNet + ResNet-50: num_ops=2, magnitude=9. Sensible defaults for larger models (RegNet, EfficientNet): num_ops=2, magnitude=15.

Yes — the same defaults work everywhere. The whole point of RandAugment was to eliminate the hyperparameter search of its predecessor AutoAugment.

4 · Mixup — mixing images AND labels

Given two training samples (x1, y1) and (x2, y2), mixup makes a new one:

where $\lambda \sim \text{Beta}(\alpha, \alpha)$ with $\alpha \approx 0.2$ (concentrated near 0 or 1 — most mixes are gentle).

The labels are mixed in one-hot / probability space. If y1 = "cat" and y2 = "dog" and $\lambda=0.7$, the new label is {cat: 0.7, dog: 0.3}. The network is asked to produce those exact soft probabilities.

4.1 Implementation (~15 lines)

import numpy as np
import torch
 
def mixup_batch(x, y, alpha=0.2, num_classes=10):
    """
    x: (N, C, H, W) batch of images
    y: (N,)          integer labels
    returns: mixed x, mixed y (as soft probability targets)
    """
    lam = np.random.beta(alpha, alpha) if alpha > 0 else 1.0
    perm = torch.randperm(x.size(0), device=x.device)
    x2, y2 = x[perm], y[perm]
 
    x_mixed = lam * x + (1 - lam) * x2
 
    # Convert labels to soft probabilities
    y_onehot  = torch.nn.functional.one_hot(y,  num_classes).float()
    y2_onehot = torch.nn.functional.one_hot(y2, num_classes).float()
    y_mixed = lam * y_onehot + (1 - lam) * y2_onehot
 
    return x_mixed, y_mixed

4.2 Loss that accepts soft targets

Standard nn.CrossEntropyLoss since PyTorch 1.10 accepts soft-probability targets natively — just pass (N, num_classes) instead of (N,):

loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)   # smoothing pairs well
 
for xb, yb in train_dl:
    xb, yb = xb.cuda(), yb.cuda()
    xb, yb_soft = mixup_batch(xb, yb, alpha=0.2, num_classes=10)
    logits = model(xb)
    loss = loss_fn(logits, yb_soft)
    loss.backward()
    opt.step(); opt.zero_grad()

4.3 Why Mixup works (short version)

Two intuitions:

1. Convex hull augmentation. By training on convex combinations of pairs, you're forcing the network to be linear-ish in input space between class pairs. That's a much smoother function than what unaugmented training produces, and smoother = better generalization.
2. Label smoothing on steroids. Networks trained without mixup tend to become overconfident — the softmax outputs get very peaked. Mixup breaks this by never showing the network a hard label. Calibration improves noticeably (see the ECE metric — expected calibration error).

5 · CutMix — the spatial variant

Instead of blending pixel values, CutMix cuts out a rectangle from one image and pastes it into another. The label is mixed proportionally to the area.

def cutmix_batch(x, y, alpha=1.0, num_classes=10):
    lam = np.random.beta(alpha, alpha)
    perm = torch.randperm(x.size(0), device=x.device)
    x2, y2 = x[perm], y[perm]
 
    # Pick a random box
    H, W = x.size(2), x.size(3)
    cut_h = int(H * np.sqrt(1 - lam))
    cut_w = int(W * np.sqrt(1 - lam))
    cy = np.random.randint(H); cx = np.random.randint(W)
    y1 = np.clip(cy - cut_h//2, 0, H); y2b = np.clip(cy + cut_h//2, 0, H)
    x1 = np.clip(cx - cut_w//2, 0, W); x2b = np.clip(cx + cut_w//2, 0, W)
 
    x_mixed = x.clone()
    x_mixed[:, :, y1:y2b, x1:x2b] = x2[:, :, y1:y2b, x1:x2b]
 
    # Actual box area may differ from intended lam because of clipping
    lam = 1 - ((y2b - y1) * (x2b - x1) / (H * W))
 
    y_onehot  = torch.nn.functional.one_hot(y,  num_classes).float()
    y2_onehot = torch.nn.functional.one_hot(y2, num_classes).float()
    y_mixed = lam * y_onehot + (1 - lam) * y2_onehot
 
    return x_mixed, y_mixed

Two design choices:

• alpha=1.0 for CutMix (compared to alpha=0.2 for Mixup). CutMix uses more extreme mixing ratios because the resulting images are more realistic — a chunk of a dog on a cat still contains real dog pixels and real cat pixels, unlike a 50/50 alpha-blend.
• Post-hoc lam correction. The intended mixing ratio and the actual box area disagree when the box is clipped by the image edge. Always recompute lam from the final box dimensions before mixing labels.

5.1 Random choice per batch

In practice, use CutMix and Mixup with 50/50 probability per batch:

if np.random.rand() < 0.5:
    xb, yb = mixup_batch(xb, yb, 0.2, 10)
else:
    xb, yb = cutmix_batch(xb, yb, 1.0, 10)

This is what timm's default augmentation policy does and it consistently beats either alone.

6 · Label smoothing — the free lunch

Even without mixing, label smoothing usually helps:

loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)

Instead of one-hot target [0, 0, 1, 0, ..., 0], use [0.01, 0.01, 0.91, 0.01, ...] — subtract 0.1 from the true class, distribute uniformly across the rest. Costs one line, buys ~0.5% on most classification tasks. Pairs beautifully with Mixup — both discourage overconfidence, and their effects don't overlap much.

7 · When augmentation hurts

• Very small models on small datasets. A tiny model can't fit even the augmented distribution — augmentation just adds noise. Cutoff is somewhere around a 5-layer network on <5,000 images.
• Regression tasks. Especially label-mixing methods make no sense for continuous targets. Stick to geometric + photometric.
• Calibration-sensitive tasks. Mixup makes networks less confident. If you actually want peaked predictions (e.g. medical triage with a strict threshold), Mixup can hurt operating-point accuracy.
• Class-imbalanced datasets. Mixup between a rare class and a common class can drown the rare class's signal. Consider per-class re-sampling first.

8 · The 2025 pulse · augmentation, self-supervision, and synthetic data

Data augmentation started as a hack — "we don't have enough images so let's flip them". By 2025 it's grown into three intertwined ideas that shape almost every modern training pipeline.

1. Augmentation ≡ the loss. Under contrastive self-supervision, augmentation is not a preprocessing step — it is the entire training signal. Two augmented views of the same image are labelled "positive pair"; the model learns whatever representation makes those two look alike. SimCLR (Chen et al. 2020) demonstrated that the choice of augmentation is the single most important hyperparameter of contrastive pretraining — more than model depth, batch size, or loss temperature. Their figure 5 is required viewing: colour jitter + random crop dominates every other combination by a huge margin.

2. Data-centric > model-centric. Andrew Ng started calling this shift out around 2021 ("Data-Centric AI"), and it has become a running theme. When ResNet Strikes Back added 4% top-1 to ResNet-50 by upgrading the recipe (mostly augmentation), no architecture change could match that. Nowadays when someone claims a new architecture win, the first review question is "did you match the augmentation recipe?"

3. Synthetic data as augmentation. The newest twist: generate training examples with diffusion models. "Synthetic Data from Diffusion Models Improves ImageNet Classification" (Azizi et al. 2023) showed that adding a few hundred million generated images to real ImageNet gives ~2% top-1. For low-data regimes (e.g. medical, satellite) diffusion-generated augmentations can be worth 5-10%. This is the same principle as MixUp — hallucinated samples in the manifold-interpolation direction — taken to a semantic level.

Current-year augmentation recipes worth knowing:

• TrivialAugment (Müller & Hutter, ICCV 2021) — pick one random augmentation per image with a random magnitude. As good as RandAugment. Zero hyperparameters. In 2025 papers this is increasingly the default.
• SimpleAug / 3-Augment (Touvron et al., DeiT-III 2022) — the DeiT-III paper stripped augmentation down to grayscale + solarize + gaussian blur, and matched heavier stacks. Less can be more.
• RepeatedAugment / MixUp+CutMix scheduling (timm library, 2023+) — apply MixUp for the first 80% of training, turn it off for the last 20% ("cool-down phase"). Recovers 0.5% top-1 by giving the model a chance to sharpen calibration at the end.
• Model Soups (Wortsman et al., ICML 2022) — not augmentation strictly but train multiple copies with different augmentation seeds and average their weights. Free 0.5-1% top-1 on ImageNet if you can afford the training.
• AutoAugment vs learned policies — RL-searched policies from 2018 have been mostly replaced by fixed simple recipes because the cost/benefit tilted toward simplicity.

A note on augmentation for LLMs. Text augmentation is much harder than image augmentation — words don't have translation equivariance or colour jitter. The LLM analogues are: (a) synonym replacement, (b) back-translation, (c) sentence shuffling, (d) synthetic data generation from a larger model (this is how Llama 3, DeepSeek-V3, and every 2024–2025 model was trained — with millions of GPT-4-generated instruction-response pairs mixed into pretraining). We'll cover this in sessions M08 and M10.

Further reading · augmentation in 2025

• Zhang et al. (2018) — mixup: Beyond Empirical Risk Minimization. The paper.
• Yun et al. (2019) — CutMix.
• Cubuk et al. (2019) — RandAugment.
• Müller & Hutter (2021) — TrivialAugment.
• Touvron et al. (2022) — DeiT III: Revenge of the ViT. The 3-Augment recipe.
• Chen et al. (2020) — SimCLR. Section 3 (augmentation study).
• Azizi et al. (2023) — Synthetic Data from Diffusion Models Improves ImageNet Classification.

Retention scaffold

One-line summary (write it in your own words): ______________________________________

Spaced review: re-derive the MixUp and CutMix mixing math in 24 hours. On day 7, take your S027 CIFAR pipeline and incrementally add each augmentation, logging the accuracy delta per addition — you should recover ~3% top-1 total.

Next session (S030): The Vision Transformer — same task, radically different architecture. Understand why ViT beats CNNs at scale and loses at low data.

Sticky note:

Order: geometric → photometric → normalize → mix. RandAugment(2, 9) + MixUp(0.2) + CutMix(1.0) + label_smoothing(0.1) is the modern default.

Legacy recall drills

Retention scaffold

Recall

Stretch

Add Mixup + CutMix to your S027 CIFAR pipeline. Target 94.5% test accuracy in 50 epochs. Log the accuracy delta from each augmentation added incrementally.

In your own words

If you had to explain Mixup to a colleague in one paragraph and one equation, how would you do it?

Spaced review

• S020 (dropout) — augmentation is regularization in input space; dropout is regularization in weight space. Understanding both is understanding modern regularization.
• S027 (CIFAR training) — plug today's stack into yesterday's pipeline.

Next session

S030 — The Vision Transformer. Same task, different architecture — throw away every convolution and use pure attention. Understand why ViT beats CNNs at scale and loses at low data.

Bring back tomorrow

• Order: geometric → photometric → normalize → mixing.
• RandAugment(2, 9) is a safe default everywhere.
• Mixup + CutMix + label smoothing = the modern regularization stack.

Previous: ← DL S028 · Next: DL S030 →