Pretraining bugs: hypotheses, counterexamples and tests

A curve gives hypotheses; a test separates them

Goal: investigate all eight original failure families plus the downstream-evaluation case using reproducible interventions, rather than diagnose from a screenshot alone. The useful mental model is a differential diagnosis: record the symptom first, propose competing causes, then choose a cheap test that could falsify each cause.

A descending loss proves that the implementation finds a learnable signal, not that it solves the intended task. Copying an unshifted input, attending to future targets or including prompt/padding targets can all make a wrong pipeline look excellent. A loss near ln(V) or a unigram entropy is a baseline to compare against, not a unique root-cause fingerprint.

Baselines and deliberately wrong objectives

Uniform logits give CE=ln(V). For target frequencies p_i, the best context-independent predictor achieves entropy H(p)=-sum(p_i log p_i). A nonuniform randomly initialized head need not produce ln(V). A correctly trained model on independent uniformly random targets can remain near ln(V) despite real updates. All-pad targets are easy to memorize if scored, not inherently a loss7 plateau. All-ignored mean CE can be NaN. A consistent permutation of token IDs preserves learnable structure even if a mismatched tokenizer makes decoding gibberish.

This bounded CPU program demonstrates those counterexamples instead of asserting that every broken loop is exactly10.4. Save it as debugging.py.

import math, copy
from collections import Counter
import torch
import torch.nn.functional as F
 
def entropy(ids):
    c=Counter(ids); n=sum(c.values())
    if not n: raise ValueError('empty corpus')
    return -sum(v/n*math.log(v/n) for v in c.values())
 
def check_batch(x,y,vocab):
    if x.shape!=y.shape or x.ndim!=2: raise ValueError('B,T shapes')
    if x.dtype!=torch.long or y.dtype!=torch.long: raise ValueError('integer IDs')
    if not torch.all((x>=0)&(x<vocab)&(y>=0)&(y<vocab)): raise ValueError('range')
    if not torch.equal(x[:,1:],y[:,:-1]): raise ValueError('next-token shift')
 
class Alerts:
    def __init__(self,cooldown=5):
        self.cooldown=cooldown; self.last={}; self.best=float('inf'); self.stale=0
    def check(self,step,loss,previous,grad,tps,baseline_tps,gap,val=None):
        metrics=[loss,previous,grad,tps,baseline_tps,gap]
        if val is not None: metrics.append(val)
        if not all(math.isfinite(value) for value in metrics):
            # Invalid numerics are a stop/investigate signal, never a quiet comparison.
            # Do not advance plateau/best state with an invalid observation.
            return ['nonfinite']
        conditions={'loss_spike':loss>previous+1.,'grad_spike':grad>10.,
                    'throughput_drop':tps<.6*baseline_tps,'validation_gap':gap>.5}
        if val is not None:
            if val<self.best: self.best=val; self.stale=0
            else: self.stale+=1
        conditions['validation_plateau']=self.stale>=3
        fired=[]
        for name,flag in conditions.items():
            if flag and step-self.last.get(name,-10**9)>=self.cooldown:
                fired.append(name); self.last[name]=step
        return fired
 
if __name__=='__main__':
    torch.set_num_threads(1); torch.manual_seed(5)
    v=7; y=torch.tensor([0,1,2,3])
    assert math.isclose(float(F.cross_entropy(torch.zeros(4,v),y)),math.log(v),rel_tol=1e-6)
    logits=torch.tensor([[5.,0.,0.]]).repeat(2,1)
    assert not math.isclose(float(F.cross_entropy(logits,torch.tensor([1,2]))),math.log(3))
    m=torch.nn.Linear(3,3); opt=torch.optim.AdamW(m.parameters(),lr=.02)
    x=torch.randn(5,3); labels=torch.tensor([0,1,2,1,0]); before=copy.deepcopy(m.state_dict())
    fixed=[]
    for _ in range(3):
        opt.zero_grad(); loss=F.cross_entropy(m(x),labels); loss.backward(); fixed.append(float(loss.detach()))
        # Deliberately no optimizer.step: constant fixed-batch loss, not necessarily ln(3).
    assert fixed[0]==fixed[1]==fixed[2]
    assert all(torch.equal(before[k],z) for k,z in m.state_dict().items())
    opt.step(); assert any(not torch.equal(before[k],z) for k,z in m.state_dict().items())
    all_ignored=F.cross_entropy(torch.zeros(2,3),torch.full((2,),-100),ignore_index=-100)
    assert torch.isnan(all_ignored)
    extreme=torch.tensor([[1000.,-1000.,0.]])
    assert torch.isfinite(F.cross_entropy(extreme,torch.tensor([0])))
    assert not torch.isfinite(extreme.exp()/extreme.exp().sum()).all()
    # Relabel both logit columns and labels by the same bijection: CE is unchanged.
    a=torch.randn(4,v); perm=torch.tensor([3,0,6,2,1,5,4]); inv=perm.argsort()
    torch.testing.assert_close(F.cross_entropy(a,y),F.cross_entropy(a[:,perm],inv[y]))
    ids=torch.tensor([[1,2,3,4,5]])
    check_batch(ids[:,:-1],ids[:,1:],v)
    try: check_batch(ids[:,:-1],ids[:,:-1],v)
    except ValueError: pass
    else: raise AssertionError('unshifted targets accepted')
    # A fake copy model scores wrong same-token labels well but fails next-token prediction.
    copy_logits=F.one_hot(ids[:,:-1],v).float()*20
    wrong=F.cross_entropy(copy_logits.reshape(-1,v),ids[:,:-1].reshape(-1))
    right=F.cross_entropy(copy_logits.reshape(-1,v),ids[:,1:].reshape(-1))
    assert wrong<1e-6 and right>10
    assert entropy([0,0,0,0])==0
    alerts=Alerts(); fired=alerts.check(0,4,2,20,10,100,1,2)
    assert set(fired)=={'loss_spike','grad_spike','throughput_drop','validation_gap'}
    assert alerts.check(1,4,2,20,10,100,1,2)==[]
    alerts.check(2,2,2,1,100,100,0,2)
    assert 'validation_plateau' in alerts.check(3,2,2,1,100,100,0,2)
    print('uniform, no-step, all-ignored, overflow, permutation, shift and five-alert controls passed')

Alert thresholds here are testable example policies, not universal healthy values or causal diagnoses. NaN comparisons are false, so finite-threshold checks alone can silently miss a broken run. The explicit nonfinite event bypasses cooldown and leaves best/plateau state unchanged; its caller must stop updates and investigate rather than merely log the event. Apply cooldown only to ordinary finite anomalies. This object produces events, not optimizer control. Persist cooldown/plateau state if restart continuity matters. A notifier such as W&B can consume fired events after explicit configuration; an unconditional wandb.alert() call does not itself monitor a condition. This lesson neither contacts a tracker nor creates a dashboard.

Bug 1: no observable parameter updates

Check actual used LR, optimizer membership by parameter identity, requires_grad, graph connectivity, finite gradients, clipping factor, scaler skipped steps and the update call. Constructing the optimizer before a device move can be problematic in some transformations; inspect actual identities rather than asserting every .to() replaces parameters. Save tensors with .clone()/deep copy, not a live state_dict alias.

Compare loss and weights on the same batch. With dropout or changing samples, missing .step() need not produce a perfectly flat curve. Nonzero gradient does not imply nonzero parameter movement: zero LR, underflow or optimizer state can intervene. Fit a bounded fixed batch, then run a no-step negative control. Passing does not validate label meaning; a sufficiently expressive model can memorize fixed random labels.

Bug 2: context is not helping

Measure a unigram/count baseline using the exact target distribution and mask. Decode both x and y; verify x[:,1:]==y[:,:-1] where boundary policy permits. Check ID range and tokenizer hash: equal vocabulary sizes can still map IDs to different text, while out-of-range IDs should fail loudly rather than clip/wrap.

Inspect all-pad and all-ignored fractions, selected loss positions, duplicate batches, sequence lengths, EOS boundaries, positional offsets and attention masks. A consistent ID permutation preserves information, so “wrong tokenizer makes random unlearnable IDs” is false in general. Inconsistent tokenization or semantic train/serve mismatch is a different failure.

Shuffle-context interventions must hold targets/protocol fixed and compare enough samples. Unchanged loss after scrambling can suggest limited use of context, but model/task uncertainty and small sample power remain alternatives. A biased output head is not required to learn marginals; the architecture here omits linear biases.

Bug 3: early nonfinite loss or gradients

Locate the first nonfinite tensor in this order: input/preprocessing, embeddings, Q/K scores, softmax, output logits, scalar loss, unscaled gradients, optimizer moments and updated parameters. Save a small failing batch with configuration/hash, avoiding raw private data in logs. Use stable cross_entropy on logits and numerically stable attention. Float32 softmax cannot undo infinities already produced by an FP16 score matmul. Scaling before a matmul can help magnitude but is not a universal fix.

Underflow produces rounded zeros; it does not directly mean NaN. FP16 overflow, inf-inf, division by zero or all-masked reductions can produce nonfinites. BF16 has a wide exponent range but still limited precision. Reduce a reproduction to eager FP32, then re-enable dtype/compiler features one at a time. Use clip_grad_norm_(...,error_if_nonfinite=True) before stepping; clipping bounds finite gradients, not a corrupted forward. Do not claim a fixed Llama head-init formula when the referenced scheme actually scales residual projections.

Bug 4: a late spike

Correlate batch identity, LR, used token denominator, preclip norm, optimizer state, recent checkpoint restore, data shard and system events. Replaying the same batch from the same last-good state is stronger evidence than the visual shape. An unusual but valid batch is not necessarily dirty data. A finite spike can recover; a NaN update should not be allowed to contaminate future steps.

PaLM reports an empirical rewind/skip recovery procedure in its training-instability discussion. That is an attributed result, not a universal skip200/500 recipe or proof that every1B run needs automatic rewind. Arbitrarily skipping every gradient above5 changes the training distribution and update clock. If a skip is used as emergency containment, log it and investigate; distributed ranks must agree whether the update happened. Preserve the original run branch for comparison.

Bug 5: train improves while validation worsens

Overfitting is possible whenever repeated exposure and capacity fit training-specific structure; no exact two-epoch or100M-token boundary makes it rare or inevitable. Distribution shift, sample weighting, augmentation/dropout mode and an eval bug can create a gap too. Leakage normally biases validation down, so rising validation is not a fingerprint of overlap.

Audit train↔validation deduplication, source/entity grouping and paths. Compare per-source metrics in the same model mode and loss denominator. A different timestamp/source can be a valuable out-of-distribution test, but it is not interchangeable with an IID holdout. Use a separate validation selection rule and untouched final test; do not “fix” poor validation by training on it.

Bug 6: throughput changes

Separate data wait, host→device, forward/backward, optimizer, evaluation, serialization, upload and logging. A synchronous first checkpoint delays its interval; it cannot alone explain permanently slower training-only steps after the save ends. Async work can contend for CPU/RAM/I/O, and live state_dict writes can race with updates. Increase the save interval to save less frequently—the original advice to reduce ckpt_every reversed this.

Low utilization is not proof of a communication bottleneck; small kernels, CPU stalls, throttling, memory pressure and instrumentation matter. Pinned memory and nonblocking copies can help supported transfers but need profiling. Do not repeatedly empty CUDA's cache as an OOM cure; it cannot free live tensors. Plot throughput against actual sequence/token counts, since variable lengths or packing can change real work.

Bug 7: yesterday's identical curve

Deterministic reruns with the same inputs are sometimes exactly what we want. Check raw log files, run/config/code/data hashes, checkpoint path and environment precedence before changing the seed. A stale plot or a resumed optimizer group can conceal a requested LR change. Create a new output directory for a fresh experiment; preserve the old latest instead of deleting the only recovery point.

When deliberately branching from a checkpoint, state which configuration changed and whether moments/schedule/data were retained. Identical outputs alone cannot distinguish stale resume from successful reproducibility.

Bug 8: low teacher-forced loss, poor generation

Audit batch/sequence axes, last-position logits, target shift, causal mask, tokenizer decode, BOS/EOS/stop IDs, temperature/top-k, template, context truncation and RoPE/cache offsets. Compare full-prefix recomputation with cached next-token logits on a tiny deterministic sequence. A future-token perturbation must not change earlier logits in a causal decoder.

Reasonable top-five next tokens for one prompt do not prove the model is fine. Teacher forcing supplies correct history; free generation can drift after one error. Greedy loops can be a model/data issue rather than a coding bug. Test fixed prompts, seeds and maximum generation bounds in addition to likelihood.

Bonus: good likelihood but weak task scores

The corpus, context length and task format may not support the capability being measured. Prompt/template mismatch, evaluation normalization, option boundaries and contamination checks come before a claim of emergence. Few-shot count is a harness-level --num_fewshot option, not --gen_kwargs num_fewshot. More examples can also consume needed context.

Grokking studies delayed generalization on particular algorithmic tasks; it is not evidence that an arbitrary15-point jump in a web-pretrained model is a phase transition. A monotone-after-jump curve does not prove legitimacy. Repeat a frozen protocol, inspect item-level changes and continuous margins, and test exposure before attributing mechanism. No unverified Cottier-scale anecdote is needed to teach the distinction.

A finite debug protocol and instrumentation

The “ten-minute” checklist is a prioritization tool, not a diagnosis guarantee: freeze evidence; compute baselines; inspect batch/shift/mask; inspect optimizer/deltas; replay in FP32; compare one-step restart; profile; change one factor and record prediction/result. For a finite benign anomaly, a short monitored continuation can be justified; “watching costs nothing” is false because GPU time and opportunity cost continue. A small parallel experiment needs authorized spare capacity and may not reproduce scale-dependent behavior.

A useful local log contains train/validation NLL/count, used LR, preclip norm, parameter norm/delta, per-layer finite/norm summaries, data hash/cursor and phase times. Attention entropy depends on the number of allowed keys: uniform causal attention at position t has entropy ln(t+1), not ln(full sequence length) at every position. A specialized head can have low entropy without being broken. Inspect heads/tasks rather than enforce a2–4nat healthy band.

PyTorch profiler and torch.utils.flop_counter.FlopCounterMode can characterize supported operations; missing/custom/fused operator coverage must be reported. Model FLOPs utilization divides estimated model FLOPs by a hardware peak with the same dtype/sparsity convention. A drop does not uniquely identify a dataloader or memory allocator. CUDA memory history is inspected through PyTorch's memory snapshot visualizer, not assumed interchangeable with Chrome trace format; profiler trace and allocator snapshot are separate artifacts. Underscored CUDA APIs are version-sensitive. No CUDA timeline or multi-rank dashboard was run here.

Worked quiz, recall and alert exercise

• Flat10.4: calculate ln(V), then inspect gradients/updates and the data optimum. It is not uniquely a missing .step().
• Early NaN: distinguish all-ignored targets, invalid arithmetic and overflow; inspect first nonfinite, then correct the actual operator/precision/denominator. Underflow is not a synonym for NaN.
• Late spike: preserve/replay, test LR/state/data, recover from last valid state when warranted. A named paper's rewind is not an automatic recipe.
• Val up/train down: investigate overfit/shift/eval implementation; overlap is an independent audit, not the inevitable cause.
• Step500 throughput drop: compare the phase logs around the first scheduled eval/checkpoint; mode alone may not affect a dropout-free transformer.
• Plateau5.2 first five steps: inspect decoded inputs/targets and counts; verify used LR; verify optimizer identities and finite gradients; measure deltas; attempt a tiny fixed-batch fit with a negative control. Failures guide the next test, not a blanket0.5-in100-steps requirement.

Legacy recall uses the same corrected answers: ln(V) and entropy are hypotheses; clipping cannot repair invalid forward values; bad-batch skipping changes the objective; no-step curves need fixed data/mode to be constant. The alert stretch is implemented above for loss, gradient, throughput, gap and plateau, with cooldown and tests. Add a notifier only after choosing privacy-safe event fields and authorization. Review tomorrow: explain a healthy run that stays near ln(V) and a broken run whose loss falls rapidly.

Sources and execution boundary

CrossEntropyLoss2.9, AMP examples2.9, gradient clipping source, PaLM, Grokking, profiler, CUDA memory source.

The bounded CPU counterexamples and alert rules are executed. No fabricated personal incident, GPU/large-model diagnosis, tracker notification, hardware fault or service benchmark is claimed. Optional reading and next steps are listed below.

Inherited optional resources and navigation

These pre-read links are optional further reading; their timestamps, video claims and optional pacing are not accepted technical evidence.

• Karpathy — GPT-2 reproduction (debugging section)
• Debugging deep networks — overfit one batch
• A Recipe for Training Neural Networks

Previous: ← DL S055 · Next: DL S057 →