DL S056 · Common Pretraining Bugs and Their Fingerprints

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 56 / 80 · Module M09 · ~2 hours

The story we're starting with

Everyone who has trained more than three real language models has a mental Rolodex of "curves I have seen before." A loss that plateaus at 7.0 forever. A curve that looks perfect for 4000 steps and then suddenly spikes to 12 and stays there. A tokens-per-second graph that falls off a cliff at step 500 and never recovers. A validation loss that goes up while training loss goes down.

Each of these is a specific bug with a specific fingerprint. Recognizing them by shape saves you hours per debug cycle. This session catalogs the eight most common ones — every one of which I (or someone I trust) has hit in a real run — with the fingerprint, the root cause, and the exact fix.

You will encounter at least three of these in your first year of pretraining. This session is a photo album so you recognize them when they show up.

Prerequisites

• S053 (100M pretraining) — you have a real loop that could exhibit any of these.
• S054 (loss curves) — you have the vocabulary for what a "healthy" curve looks like.

Bug 1: Loss stuck at exactly 10.4 (= log(vocab))

Fingerprint: completely flat line at 10.4 from step 0 to step 100+. No downward trend at all.

What's happening: the optimizer isn't updating the model. Either:

• optimizer.step() never called (I've done this).
• loss.backward() never called on a loss that has grad_fn.
• Model params not in optimizer (built optimizer before moving model to GPU).
• param.requires_grad = False on all params (accidentally frozen).

Fix: in a REPL:

before = {k: v.clone() for k, v in model.state_dict().items()}
train_one_step()
after = model.state_dict()
for k in before:
    if not torch.equal(before[k], after[k]):
        print(k, "changed")

If nothing prints → params aren't being touched. Trace back through backward + optimizer.

Prevention: the overfit-one-batch test from S053 catches this in 30 seconds.

Bug 2: Loss stuck at ~7.0

Fingerprint: loss drops fast from 10.4 to ~7.0 in the first 20-50 steps, then flatlines. Never gets below 6.5.

What's happening: the model has learned the token frequency distribution (which is why loss dropped from log(vocab) to ~7 — the entropy of English token frequencies) but can't learn anything more. Almost always:

• Tokenizer mismatch: data was tokenized with a different vocab than the model uses. Model sees random ID sequences with no learnable structure.
• All-padding data: dataloader is yielding batches of [PAD, PAD, PAD, ...] from a broken collate function.
• Loss mask bug: you're masking out all tokens instead of just prompt tokens.

Fix:

# Print one batch
x, y = train_ds.get_batch(4)
print("x[0][:20] =", x[0][:20].tolist())
print("decoded:", tokenizer.decode(x[0][:20].tolist()))

Should look like real text. If it's mostly repeated IDs or the decoded text is gibberish, you've found the bug.

Bug 3: NaN at step ~50

Fingerprint: loss curve looks great for 30-80 steps, then suddenly becomes NaN. All subsequent losses are NaN too.

What's happening: bf16 overflow. Usually in attention (exp(qk^T / sqrt(d)) can overflow bf16 when qk^T is large). Sometimes in the loss itself.

Fix stack:

1. Add gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0).
2. Init the head projection at reduced scale — Llama uses std=0.02/sqrt(2·n_layers), which keeps outputs modest.
3. Ensure your attention uses fp32 for the softmax: attn = attn.float().softmax(-1).to(dtype).
4. If it still NaNs, watch grad_norm — a spike from 0.5 to 500 one step before NaN is the smoking gun. Alert on grad_norm > 10.

FlashAttention (S043) does most of this in fp32 internally and is much more numerically stable — a good reason to use it.

Bug 4: Loss decreases fine for 2 hours, then spikes hard

Fingerprint: perfect cosine-like descent for thousands of steps. Then at step ~4500 a sudden spike from loss 4.0 to 6.5, followed by some recovery but never back to the original trajectory.

What's happening: a bad data batch. Some pathological document (all-caps repeated word, all-punctuation) triggers an anomalous gradient. The optimizer takes a big step in the wrong direction; the model has to relearn.

Fix:

• Log the batch content when loss > previous + 1.0. Add a if loss.item() > running_avg + 1: torch.save({"batch": x, "step": step}, f"bad_batch_{step}.pt").
• Skip abnormal-gradient steps: if grad_norm > 5.0: optimizer.zero_grad(); continue.
• Improve data cleaning to catch these upstream (S046 Gopher rules should help).

Meta's OPT-175B training log famously logged dozens of these spikes with manual data-batch skipping. It's a real, ongoing operational cost of pretraining.

Bug 5: Val loss goes UP while train loss keeps dropping

Fingerprint: train loss on smooth cosine descent. Val loss decreases initially, then inflects and starts rising around 30–60% through training.

What's happening: overfitting. Very rare in true pretraining (corpora are big enough that you never see an example twice). Almost always means:

• Your "held-out validation" wasn't actually held out. Split the val set from a separate corpus, not from a random shuffle that leaked shard neighbors.
• Your corpus is too small — you're epoching >2 times through it. Increase corpus size or reduce total tokens.
• You're accidentally training on your validation set (paths crossed in a config).

Fix: deduplicate carefully between train and val (MinHash from S049 across the boundary), and ensure val comes from timestamps or sources not in train.

Bug 6: Tokens/sec drops off a cliff at step 500

Fingerprint: throughput graph is flat at 24000 tok/s for the first 500 steps. Then drops to 8000 tok/s and stays there.

What's happening: almost always the first checkpoint save. torch.save on 100M+ params takes 5-15 seconds. If it blocks the training loop (single process, no async), your average tok/s drops.

Fix:

• Async save: save_checkpoint from a background thread.
• Reduce ckpt_every: don't save every 500 steps if your step is 1 second.
• Save to fast local disk (NVMe), not network storage. On Runpod: /workspace/checkpoints if the network volume is NVMe-backed; if not, save locally then rsync in background.

Also possible: gpu-cpu memory transfer inefficiency (missing pin_memory=True on dataloader tensors). Check nvidia-smi — if GPU util is <90% during training, comms is bottlenecking.

Bug 7: Loss curve identical to yesterday's run

Fingerprint: you changed a hyperparameter, restarted training, and the new curve overlays exactly on the old curve. Same values at every step, to 4 decimal places.

What's happening: something is being loaded from cache/checkpoint that overrides your new config.

Common causes:

• --resume_from latest.pt was left in your launch command — you loaded the old optimizer state and continued the old run.
• wandb picked up your old sweep config from sweep.yaml.
• Environment variables persist from a previous shell (LR=1e-4 set once, then you edited the file but env still wins).

Fix: rename or delete latest.pt before a fresh run. Print cfg at step 0 and confirm every value is what you expect.

Bug 8: Model outputs are gibberish despite low loss

Fingerprint: loss reaches expected value (~3.8) but samples from the model are nonsense — random tokens, repetition loops, or exclusively-punctuation streams.

What's happening: several possibilities:

• Sampling code is wrong: greedy decoding of the last token position on a batch-first tensor when the model expects seq-first. Print logits.shape and argmax positions.
• You've trained the wrong forward path: e.g., model has an if training: use_teacher_forcing() branch. Sampling calls the other path.
• Tokenizer decode mismatch: trained on tokenizer A, generating tokens for tokenizer B. Byte offsets go wild.
• Rotary/RoPE not applied at inference: you applied it during training via a helper but forgot in the generate path.

Fix: print logits[0, -1, :].topk(5) for a real prompt. If the top-5 tokens look reasonable ("The cat sat on the → mat, floor, chair, ground, ..."), the model is fine and only the generation loop is broken. Isolate.

Bonus Bug 9: Loss curve is beautiful but downstream evals are barely above random

Fingerprint: val loss dropped to 3.8 as expected. HellaSwag stays at 25.5% (random). MMLU at 25%. Not improving with more training.

What's happening:

• Data too monotonous: e.g., all news-wire. Model learned "news writing" but not general reasoning. Fix mix (S048).
• Context too short: trained on seq_len=128 when the eval needs 512+ context. The model can't fit the question.
• BOS/EOS/format mismatch: eval prompts use <|user|> template your model never saw. Format matters more than most people think.

Fix: rerun eval with --gen_kwargs num_fewshot=5 (few-shot examples usually help). Or match the prompt template to your training data distribution.

The 10-minute debug protocol

When wandb looks wrong, follow this every time:

Ten questions, ten minutes. Nine times out of ten, this narrows it to one of the eight bugs above.

Wandb alerts you should set

Before every real run:

wandb.alert(
    title="Loss spike",
    text="Loss increased more than 1.0 from previous log",
    level=wandb.AlertLevel.WARN,
    wait_duration=60,
)

Do this on:

• loss > previous_loss + 1.0
• grad_norm > 10.0
• tok_per_sec \< 0.6 × mean_tok_per_sec
• val_loss - train_loss > 0.5
• val_loss stops improving for 3 consecutive evals

Push alerts to your phone. Catching a bad-batch spike five minutes in beats catching it three hours in.

War stories

The 2025 debugging toolkit

A few tools that make pretraining debugging dramatically easier in 2025 vs 2023:

The wandb tricks that pay off

• Grad-norm per parameter tensor (wandb.log({"grad_norm/embedding": g.norm(), ...})). Catches per-layer explosions before the aggregate norm shows anything.
• Attention entropy (H(softmax(qk^T)) averaged over heads). Healthy at ~log(seq_len) early, drops to ~2–4 nats mid-training. If it collapses to 0 in one layer, that layer has degenerated.
• Weight norm per layer over time. Sudden jumps flag numerical instability. Sudden zeroes flag dead layers.
• Custom system panels: GPU memory + tokens/sec + nvidia-smi per rank, plotted per-rank not just aggregate. The BLOOM lesson: a single hot GPU shows up per-rank but averages out in the aggregate.

The Pattern-of-Life dashboard

Steal MosaicML Composer's default wandb dashboard layout:

1. Row 1: train/loss, val/loss, train/perplexity, val/perplexity.
2. Row 2: grad_norm, learning_rate, param_norm, train/tokens_per_sec.
3. Row 3: per-layer grad norm heatmap (imshow), attention entropy per layer.
4. Row 4: GPU memory %, temperature, throughput per rank.
5. Row 5: sample generations (logged as wandb.Table), eval scores.

Every healthy run has a recognizable shape. Once you've stared at 5–10 such dashboards, unhealthy runs pop out visually within seconds.

torch.utils.flop_counter and MFU tracking

Since PyTorch 2.2, torch.utils.flop_counter.FlopCounterMode gives per-op FLOPs cheaply. Compute your theoretical MFU each step: mfu = actual_tokens_per_sec / (peak_flops / model_flops_per_token). Log it. If MFU drops from 40% to 15% mid-run, your dataloader stalled, your grad-accum count changed, or your VRAM fragmented.

torch.cuda.memory._record_memory_history

Since PyTorch 2.1, you can snapshot memory alloc/free events and visualize the timeline in Chrome's chrome://tracing (or the newer memory_viz.html tool). When your OOM shows up out of nowhere, this tool tells you exactly which line allocated the offending tensor. Use before you cargo-cult torch.cuda.empty_cache() calls.

Further reading

• PaLM tech report §5.1 (instability) — arxiv:2204.02311
• Grokking paper — arxiv:2201.02177
• BLOOM chronicles — github.com/bigscience-workshop/bigscience
• OPT chronicles — github.com/facebookresearch/metaseq
• MosaicML MPT-7B blog (spike protocol) — mosaicml.com/blog/mpt-7b
• torch memory viz — pytorch.org/blog/understanding-gpu-memory-1

🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: re-read the bug-fingerprint table (§Bug 1–9) in 24 hours. Revisit the full session on day 7 with focus on the 2025 debugging toolkit.

Next session (S057): zoom out — given the 100M we just debugged, what changes at 1B? 7B? 70B? The scaling mental model.

Sticky note (keep on your desk): "Bugs have fingerprints. Fingerprints live on wandb. Learn to read them like an EKG."

Legacy recall drills (kept for spaced-review continuity)

Stretch

Write a wandb_alerts.py helper that installs all five alerts (loss spike, grad spike, throughput drop, val gap, val plateau) with reasonable defaults. Use it at the top of every training script. Test each alert by artificially triggering it.

In your own words

"When a loss curve looks wrong, the first thing I check is ______, because ______."

Spaced-review pointer

• S054 (loss curves) — "healthy" reference to compare against.
• S049 (dedup) — val-set contamination is a subtle version of Bug 5.
• S055 (checkpointing) — Bug 7 (identical curve) usually traces to checkpoint hygiene.

Next-session teaser

Session 057 zooms out: given everything we've built for 100M, what changes at 1B? At 7B? At 70B? We map the compute/memory/comms scaling from toy to frontier — so you can look at a Llama or Mistral release and confidently say "I could reproduce this if I had ~X GPU-months."

Bring back tomorrow

• Flowchart: the 10-minute debug protocol.
• Numbers: flat-at-10.4 vs flat-at-7.0 diagnoses.
• Habit: wandb alerts on loss+1, grad_norm>10, throughput drop.

Previous: ← DL S055 · Next: DL S057 →