Loss curves and evaluation: likelihood, tasks and controls

Three measurements, three different questions

Goal: compute token-weighted perplexity, diagnose curves through interventions, and construct a reproducible downstream evaluation rather than infer quality from a screenshot. Training NLL asks how the model fits sampled training targets. Held-out NLL asks how it predicts a specified unseen distribution. Task evaluation asks whether it performs a specified task under a specified prompting/scoring protocol. None alone proves general intelligence, safety or uncontaminated performance.

The small, previously verified NumPy experiment remains intact below, including both exact Python payloads. It is a concrete correctness and synthetic-generalization lab, not a replacement for downstream evaluation or a natural-language benchmark. Its five-symbol sequences do not become “English pretraining” when the training curve descends.

Perplexity from likelihood and denominators

For scored targets with probabilities p_i and N nonignored tokens:

PPL is reciprocal geometric-mean assigned probability, often described as an equivalent branching factor. It does not mean exactly PPL candidates are tied at each position. Uniform probabilities over V give ln(V) and PPL=V; random initialized logits need not be uniform, and a model can have PPL greater than V.

At loss3.8, PPL≈44.7012; at3.85,≈46.9931; at3.4,≈29.9641. Equal absolute loss decreases have equal relative PPL improvement: ΔL=-ln2 halves PPL anywhere on the scale. A0.5 decrease is not a larger fractional improvement at low loss than high loss.

Aggregate NLL sums and counts, then exponentiate. Two batches with losses2 and4 over10 and90 tokens yield (20+360)/100=3.8, not the unweighted batch mean3.0, and not the mean of the two perplexities. If examples deserve equal weight instead, explicitly report mean per-example NLL as a different metric. Masking/truncation/context changes alter the measured population.

Tokenization changes the units. Compare the same text, tokenizer, BOS/EOS policy, context, stride and target mask; if tokenizers differ consider a documented common byte/character-normalized likelihood with its own caveats. WikiText, PG-19, Pile and FineWeb are not a universal easy-to-hard ordering. The original table's GPT-4 size/tokens/perplexity estimates and promised100M/2B target were unsupported and are not retained as facts.

Read curves as hypotheses, not diagnoses

A healthy stochastic training trace need not be monotone, logarithmically straight or cosine-shaped. Plot raw and smoothed observations, token/update axes, LR and confidence/variation; smoothing must not erase spikes. Here are all twelve original symptom categories with defensible follow-up tests:

Do not use nan_to_num to hide failed attention or continue updating corrupted optimizer state. Gradient clipping cannot repair a NaN already created in the forward pass. A train/validation gap has no universal0.1/0.3 alarm threshold: dropout, regularization, difficulty and sampling differ. Train loss above validation can be legitimate. Leakage often makes evaluation optimistic; rising validation is not proof of a leak.

Complete local likelihood and task scorers

Save as evaluation.py. The scorer requires explicit token IDs and candidate boundaries; tokenization belongs to the caller's pinned tokenizer contract. A logit at position t scores token t+1. We reject an empty context/continuation rather than silently giving a zero score.

import math, re
import numpy as np
import torch
import torch.nn.functional as F
 
@torch.no_grad()
def sequence_nll(model,ids,labels=None):
    if ids.ndim!=2 or ids.shape[1]<2: raise ValueError('need shifted sequence')
    labels=ids if labels is None else labels
    if labels.shape!=ids.shape: raise ValueError('label shape')
    old=model.training
    try:
        model.eval(); logits=model(ids)[:,:-1]; targets=labels[:,1:]
        n=int((targets!=-100).sum())
        if n==0: raise ValueError('no scored targets')
        total=F.cross_entropy(logits.reshape(-1,logits.shape[-1]),targets.reshape(-1),
                              ignore_index=-100,reduction='sum')
        if not torch.isfinite(total): raise ValueError('nonfinite NLL')
        return float(total),n
    finally: model.train(old)
 
def aggregate(parts):
    if not parts or any(n<=0 or not math.isfinite(s) for s,n in parts):
        raise ValueError('invalid likelihood parts')
    n=sum(n for _,n in parts); loss=sum(s for s,_ in parts)/n
    return {'nll':loss,'ppl':math.exp(loss),'targets':n}
 
def candidate_scores(model,prompt,candidates,normalize=False):
    if not prompt or not candidates or any(not c for c in candidates):
        raise ValueError('empty candidate contract')
    scores=[]
    for c in candidates:
        ids=torch.tensor([prompt+c],dtype=torch.long,device=next(model.parameters()).device)
        labels=ids.clone(); labels[:,:len(prompt)]=-100
        s,n=sequence_nll(model,ids,labels); scores.append(-s/n if normalize else -s)
    return scores
 
def wilson(successes,n,z=1.96):
    if n<1 or not 0<=successes<=n: raise ValueError('binomial sample')
    p=successes/n; denom=1+z*z/n
    center=(p+z*z/(2*n))/denom
    half=z*math.sqrt(p*(1-p)/n+z*z/(4*n*n))/denom
    return center-half,center+half
 
def paired_bootstrap(a,b,seed=8,repeats=2000):
    a=np.asarray(a,dtype=float); b=np.asarray(b,dtype=float)
    if a.ndim!=1 or a.shape!=b.shape or not len(a): raise ValueError('paired results')
    rng=np.random.default_rng(seed)
    means=np.array([(b-a)[rng.integers(0,len(a),len(a))].mean() for _ in range(repeats)])
    return np.quantile(means,[.025,.975]).tolist()
 
def ngrams(text,n=13):
    if n<1: raise ValueError('ngram order')
    words=re.findall(r'\w+',text.casefold())
    return {tuple(words[i:i+n]) for i in range(len(words)-n+1)}
 
def overlap_audit(train_docs,items,n=13):
    grams=set().union(*(ngrams(d,n) for d in train_docs))
    return [{'id':i,'has_ngrams':bool(ngrams(s,n)),
             'overlap':bool(ngrams(s,n)&grams)} for i,s in enumerate(items)]
 
if __name__=='__main__':
    from architecture import Config,Decoder
    torch.set_num_threads(1); torch.manual_seed(1)
    m=Decoder(Config(vocab=8,width=8,layers=1,heads=2,kv_heads=1,hidden=16,context=16))
    ids=torch.tensor([[1,2,3,4]]); labels=ids.clone(); labels[:,:2]=-100
    s,n=sequence_nll(m,ids,labels); assert n==2 and s>0 and m.training
    v=aggregate([(20.,10),(360.,90)]); assert v['nll']==3.8
    assert not math.isclose(v['ppl'],(math.exp(2)+math.exp(4))/2)
    assert len(candidate_scores(m,[1,2],[[3],[4,5]]))==2
    lo,hi=wilson(2700,10000); assert lo<.27<hi and lo>.25
    assert paired_bootstrap([0,1,1],[0,1,1])==[0.,0.]
    hits=overlap_audit(['a b c d'],['A b c d','a b'],3)
    assert hits[0]['overlap'] and not hits[1]['has_ngrams']
    try: sequence_nll(m,ids,torch.full_like(ids,-100))
    except ValueError: pass
    else: raise AssertionError('empty targets accepted')
    print('NLL, candidate boundaries, uncertainty and overlap controls passed')

Sum log likelihood favors different candidates than length-normalized likelihood can. Use the task's declared acc/acc_norm rule rather than switching to whichever score is flattering. Multiple-choice chance is1/k per item under uniform choice; variable numbers of options require averaging1/k_i. MMLU's four-choice chance is25%, not a coin flip; GPQA does not have0% random chance. A27% score on10,000 four-choice items can be statistically distinguishable from25%; “small model” is not a statistical test.

Wilson intervals quantify binomial uncertainty under assumptions; paired bootstrap compares itemwise correctness on the same sample. If examples cluster by source/entity, bootstrap groups instead. Fixed deterministic prompts do not create seed variation by magic; few-shot selection, generation randomness and evaluation sampling can. Repeatedly choosing checkpoints/prompts on a test suite turns it into validation feedback.

Preserved synthetic experiment: task and audit

Take four distinct symbols from five and repeat the cycle: (0,2,4,1,0,2,4,1,0). Eight positions predict the next eight tokens. There are120 ordered cycles, split80/20/20 before windows. Rotations can represent related cycles across splits, so this is a modest compositional diagnostic, not a strong independent-data claim. A stricter extension groups rotations first. Early targets are ambiguous; after a complete cycle, continuation is deterministic.

Save the complete NumPy TinyDecoder module from the transformer-block lesson as tiny_transformer.py beside the following exact preserved programs. The first audits all192 coordinates and future-token causality. The second trains all parameters for300 fixed updates, compares the same-token bigram baseline and proves a next-update in-memory resume. These payloads are kept byte-for-byte; the audit snapshots their dependency separately.

import numpy as np
from tiny_transformer import TinyDecoder
 
m = TinyDecoder(vocab=4, width=4, context=4)
x = np.array([[0,1,0], [2,1,2]])
y = np.array([[1,0,1], [1,2,1]])
_, g = m.loss_grad(x, y)
errors = []
for name, value in m.p.items():
    for index in np.ndindex(value.shape):
        old = value[index]
        value[index] = old+1e-5; plus = m.loss_grad(x,y)[0]
        value[index] = old-1e-5; minus = m.loss_grad(x,y)[0]
        value[index] = old
        errors.append(abs((plus-minus)/2e-5 - g[name][index]))
assert len(errors) == 192 and max(errors) < 1e-6
changed = x.copy(); changed[:,-1] = 3
np.testing.assert_allclose(m.forward(x)[0][:,:-1], m.forward(changed)[0][:,:-1])
for bad_ids in (np.array([[-1]]), np.array([[4]]), np.array([1,2]),
                np.array([[1.5]]), np.zeros((1,5), dtype=int)):
    try:
        m.forward(bad_ids)
    except ValueError:
        pass
    else:
        raise AssertionError('invalid token/shape/context accepted')
print('checked', len(errors), 'parameters; maximum absolute error', max(errors))

A central difference near a nondifferentiable kink can disagree with a chosen subgradient; inspect rather than relaxing every tolerance. This fixture supports its tested shapes, not all configurations.

Preserved fixed-budget run and interpretation

import copy
import itertools
import numpy as np
from tiny_transformer import TinyDecoder
 
rng = np.random.default_rng(1)
seq = np.array([[*cycle, *cycle, cycle[0]]
                for cycle in itertools.permutations(range(5), 4)])
rng.shuffle(seq)
train, valid, test = seq[:80], seq[80:100], seq[100:]
x, y = train[:,:-1], train[:,1:]
model = TinyDecoder()
mom = {k: np.zeros_like(v) for k,v in model.p.items()}
var = {k: np.zeros_like(v) for k,v in model.p.items()}
 
def step(t):
    rows = rng.integers(0, len(x), 16)
    loss, grads = model.loss_grad(x[rows], y[rows])
    norm = np.sqrt(sum(np.sum(g*g) for g in grads.values()))
    factor = min(1., 1./max(float(norm), 1e-12))
    for k in model.p:
        g = grads[k] * factor
        assert np.isfinite(g).all()
        mom[k] = .9*mom[k] + .1*g
        var[k] = .999*var[k] + .001*g*g
        model.p[k] -= .01*(mom[k]/(1-.9**t))/(np.sqrt(var[k]/(1-.999**t))+1e-8)
    return loss
 
initial = model.loss_grad(x,y)[0]
for t in range(1, 301):
    step(t)
 
counts = np.ones((5,5))  # additive-one smoothing
np.add.at(counts, (x.ravel(), y.ravel()), 1)
base = counts / counts.sum(axis=1, keepdims=True)
for name, split in [('train',train), ('validation',valid), ('test',test)]:
    inputs, targets = split[:,:-1], split[:,1:]
    nll = model.loss_grad(inputs, targets)[0]
    baseline = -np.log(base[inputs, targets]).mean()
    print(name, 'NLL', round(nll,6), 'PPL', round(np.exp(nll),6),
          'bigram NLL', round(float(baseline),6))
assert model.loss_grad(x,y)[0] < initial
 
# In-memory checkpoint/resume test: no pickle, no file and no untrusted input.
snapshot = copy.deepcopy((model.p, mom, var, rng.bit_generator.state))
step(301)
expected = {k: v.copy() for k,v in model.p.items()}
model.p, mom, var, state = copy.deepcopy(snapshot)
rng.bit_generator.state = state
step(301)
for k in model.p:
    np.testing.assert_array_equal(model.p[k], expected[k])
print('resume reproduced next update exactly in this environment')

The earlier published Python3.12.3/NumPy2.4.5 reference reported train NLL0.977094, validation1.243897, test1.166045, versus bigram1.388380/1.401240/1.439238; its all-coordinate max error was about3.75e-8. The batch reruns the exact code on the existing NumPy2.2.6 CPU environment and saves raw output. Historical numbers are labelled as such, not silently asserted as newly measured or cross-platform bitwise guarantees.

The evaluation interface computes gradients unnecessarily but does not update weights. Keep the test split for final reporting; tuning on it makes it validation. The preserved lab's four exercises have these answers: the bigram tests whether additional context earns its cost; omitting sampler RNG changes the next update; later-cycle positions specifically test contextual copying; lower five-symbol PPL establishes no factual knowledge or instruction following.

A real downstream harness: pinned and local-only

The v0.4.9 lm-evaluation-harness README uses lm_eval, not the original unversioned lm-eval spelling. A custom decoder checkpoint is not automatically HF-compatible: supply a reviewed HF export/config/tokenizer or a harness adapter implementing loglikelihood/generation. Inspect rendered prompts and continuation boundaries before trusting scores.

This unexecuted reference assumes the harness and compatible model plus all required task data already exist locally. Offline variables prevent dataset/model fetching; missing cache is a blocker, not permission to download. Use a fresh output directory per run and inspect actual emitted result filenames rather than assuming a fixed /tmp/eval-out.json.

# Resource-dependent reference; not executed in this CPU audit.
HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
  lm_eval --model hf \
  --model_args pretrained=/absolute/local/hf-export,dtype=float32 \
  --tasks hellaswag,arc_easy,arc_challenge,piqa,winogrande,mmlu \
  --num_fewshot 0 --device cpu --batch_size 1 \
  --log_samples --output_path /absolute/local/new-eval-run

Record harness commit/version, task YAML revisions, model/tokenizer hashes, dataset split/item IDs, number of examples, prompt/chat template, few-shot count/selection, seeds, normalization, generation settings, raw responses, scoring exceptions and wall time. A10-shot HellaSwag or5-shot MMLU comparison needs the corresponding explicit protocol; --gen_kwargs num_fewshot=5 is not the few-shot control. No20-minute/USD0.30 performance is claimed.

Selecting an evaluation suite and newer methods

• HellaSwag: plausible continuation discrimination. Useful for some small-model comparisons, not universally the only useful benchmark or proof of reasoning.
• ARC-Easy/Challenge: science questions with task-specific option counts and normalization. “Challenge” names the benchmark selection, not an exact parameter threshold at which skill emerges.
• PIQA and WinoGrande: physical commonsense and coreference-style tests. Neither is contamination-proof.
• MMLU/MMLU-Pro: broad subject knowledge; Pro makes the setting harder with revised questions/options. Statistical power depends on items/protocol and effect size, not a blanket300M/7B cutoff.
• GSM8K, HumanEval and BBH: math answers, executable code correctness and heterogeneous difficult tasks. Code execution needs a separate sandbox/resource policy; this audit does not execute generated benchmark code. Few-shot/CoT/extraction and pass@k definitions matter.
• GPQA-Diamond and LiveBench: difficult expert questions and refreshed evaluation respectively. Freshness reduces some contamination opportunity but cannot make a benchmark immune to exposure or repeated tuning.
• MT-Bench, AlpacaEval and Arena-Hard: instruction-output comparison using specified judges/protocols. Length-controlled win rate is not ordinary accuracy. Report judge identity/date, order swaps, prompts, failures and calibration against human labels. No universal0.85 correlation is asserted.
• Chatbot Arena: human pairwise preferences with selection/population effects and uncertainty. A rank is not a universal best-model declaration or an application-specific acceptance test.

Schaeffer et al. show how nonlinear/discrete metrics can create apparent emergence; that does not prove every capability transition is an artifact. Plot log-prob margins and per-domain likelihood alongside accuracy at fixed checkpoints. A benchmark can improve while aggregate NLL barely moves without contamination, and a declining NLL with flat accuracy can be genuine improvement below an argmax threshold.

Base models can be tested with appropriate continuation prompts; instruction judges may be unsuitable for an untuned base but are not categorically meaningless for all small models. Keep a fixed probe set covering grammar, facts, commonsense, arithmetic, reasoning, instruction following and relevant languages. Read failures under greedy and controlled sampling. Twenty examples can reveal bugs but cannot guarantee detecting every safety/capability problem.

Contamination audit and report card

Exact hashes detect exact normalized documents, not equivalence between a whole-question digest and arbitrary5-grams. Choose word/token shingling and thresholds deliberately; scan questions, answers and context, record flagged fraction and sample identities, inspect false positives and separately report full versus filtered scores. Short questions can have no13-grams, as the test demonstrates. Absence of overlap in a sampled corpus does not clear the entire corpus; paraphrases and translated copies remain hard.

Do not automatically remove every overlap after seeing scores: predeclare the filter and disclose the changed denominator/distribution. A newly written domain eval helps but can still overlap known facts/templates or leak through development. The original alleged Phi-3 scandal and fixed score-drop story were unsupported; the valid lesson is protocol and provenance, not an accusation about a named lab.

A report card should include: exact checkpoint/config, actual training tokens, data contract, NLL sum/count/PPL, per-task protocol/n/chance/score/interval, paired deltas, contamination method and flagged IDs, qualitative failures, runtime and all unperformed checks. Do not fill it with invented Pythia/Llama reference scores or a “40% fewer parameters,5× less compute” conclusion with unmatched baselines.

Worked recall and extension answers

1. PPL at3.8? 44.7012 using natural logs; aggregate by targets before exponentiation.
2. Four useful small-model tasks? HellaSwag/ARC-Easy/PIQA/WinoGrande are candidates, not a whitelist. Run a pilot and check uncertainty, task fit and cost; harder tasks can still expose useful failures.
3. Opening gap? Investigate overfit, shift, modes and aggregation. A closed gap does not prove compute-bound status or mandate a bigger model.
4. Why overlap checks? Shared web sources can invalidate independence. No fixed13-gram threshold or3–5% overlap maps to a guaranteed score inflation.
5. What replaced MMLU? Nothing universally. MMLU-Pro/GPQA/Arena-Hard/LiveBench emphasize different difficulties; maintain a task-relevant suite rather than chase a dated ranking.
6. 500M six-task suite: choose the six in the reference command, fix shots per task, measure a bounded pilot, report intervals and matched baselines, then decide frequency from actual runtime. Do not promise35minutes on an unspecified A100 setup.

Legacy answers: no PPL40–50 health certificate; an early NaN needs first-nonfinite localization, not a nan guard; HellaSwag's value depends on the task/protocol; contamination requires actual normalized content matching. Four-checkpoint plots should show observations with uncertainty, not a required monotone curve. Review tomorrow: hand-calculate a weighted NLL and explain one false inference from a train/val gap.

Sources and execution boundary

CrossEntropyLoss2.9, lm-evaluation-harness0.4.9, emergence, MMLU-Pro, GPQA, LiveBench, judge study, Arena.

Executed: preserved synthetic NumPy programs and bounded metric/uncertainty/mask controls. Unexecuted: natural-language corpus training, real benchmark sweep, model-judge APIs and application UI. Optional original resources and legacy course navigation are suggestions for further study.

Inherited optional resources and navigation

These pre-read links are optional further reading; their timestamps, video claims and optional pacing are not accepted technical evidence.

• Karpathy — Intro to LLMs (eval section)
• Deep learning training loss curves
• lm-evaluation-harness README
• Are Emergent Abilities of LLMs a Mirage?

Previous: ← DL S053 · Next: DL S055 →