DL S054 · Loss Curves and Eval Harness

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 54 / 80 · Module M09 · ~2 hours

The story we're starting with

Your 100M-param model finished training. wandb shows a loss curve that goes from 10.4 to 3.8 in a smooth-looking cosine. Beautiful. Done, right?

Not quite. Training loss alone lies. It lies because your model can memorize training data (loss goes down, generalization doesn't). It lies because your data mix determines what "easy loss" looks like (a code-heavy corpus gets lower loss because code is more predictable — a fact, not a virtue). It lies because two models with the same training loss can have wildly different downstream capabilities.

You need three things to know whether your model is any good: (1) held-out validation loss (perplexity on data the model didn't see), (2) shape of the loss curve (spot bugs even if the final number looks fine), and (3) downstream evaluations (HellaSwag, ARC-easy, MMLU-tiny — tasks that ask the model to actually do things). This session covers all three.

By the end you'll be able to look at any wandb screenshot and diagnose it, and you'll have run your first lm-evaluation-harness sweep against your fresh 100M model. Spoiler: it's going to score barely above random on most benchmarks — and that's fine, we're at 0.001% of GPT-4's compute.

Prerequisites

• S053 (100M pretraining) — you have a checkpoint to eval.
• S005 (probability + cross-entropy) — for perplexity intuition.

1 · Perplexity — the number to know

Cross-entropy loss and perplexity are the same information in two units:

PPL = exp(cross_entropy_loss)

Perplexity is the effective number of choices the model considers per token. PPL = 40 means "on average, the model is as confused as if it had to pick uniformly among 40 candidate tokens for each position." PPL = 1 means perfect prediction. PPL = vocab_size (32000) means random guessing.

Reference numbers on English pretraining:

Your 100M model getting to PPL ~45 with 2B tokens (vs GPT-2 small's ~25 with 10B tokens) is what a Chinchilla-suboptimal small model looks like. Feed it 10× more tokens and you'd close most of that gap.

2 · Reading a loss curve

The gold standard: smooth monotonic drop in log-scale, decelerating. Draw it on log-y with steps on x. It should look like a shallow parabola opening down-right.

Twelve real pathologies to recognize:

Save these patterns in your head like an ECG chart. Pattern-recognition is the whole skill.

3 · The train-vs-val gap

Log train_loss at every step and val_loss every 500 steps on a held-out slice. The two should track each other within ~0.1 for a well-mixed pretraining.

Interpreting the gap:

• train ≈ val: you're using data efficiently. Keep going.
• train < val - 0.2: overfitting. Rare in pretraining but happens if you have <100M unique tokens.
• train > val: something weird — usually a bug in your eval sampler. Rerun eval with a different seed.

Set a wandb alert: val_loss - train_loss > 0.3. Catches memorization before it becomes chronic.

4 · Downstream evaluation — lm-evaluation-harness

Perplexity tells you the model has learned language. Downstream evals tell you whether it can do anything with that knowledge. The industry standard is EleutherAI's lm-evaluation-harness.

4.1 Install and run

pip install lm-eval
# On the pod, with the model loaded:
lm-eval --model hf \
        --model_args pretrained=/workspace/checkpoints/final,dtype=bfloat16 \
        --tasks hellaswag,arc_easy,arc_challenge,piqa,winogrande,mmlu \
        --batch_size 32 \
        --output_path /workspace/eval-results.json

Takes ~20 minutes on an A100 for these tasks. Costs ~$0.30.

4.2 What the numbers mean

Baseline reference — a 100M model like ours should hit roughly:

Your 100M model should score slightly above random on most tasks and roughly match Pythia-160M (which was Chinchilla-suboptimal too). Anything much higher and you probably contaminated your training data with eval data — a very common bug we cover in S056.

4.3 Why HellaSwag is the go-to benchmark at small scale

HellaSwag has ~10k examples. Multiple choice (4 options). Sensitive to model quality across a huge range — random 25%, worst frontier LLM ~40%, best 95%. Predictive of downstream generalization. And fast to run (~5 min on A100).

If you only have time for one eval during training, run HellaSwag every 1000 steps. Its curve should track val loss with a lag of a few thousand steps — reasoning skills lag pure language modeling.

5 · Beyond the harness — sample and read

Automated benchmarks lie. The best eval is to actually read the model's output.

Standard practice: keep a probes.txt with ~30 prompts covering:

• Grammar: "The cat sat on the"
• Factual: "The capital of Japan is"
• Common sense: "If I drop a glass on the floor it will"
• Arithmetic: "The sum of 47 and 25 is"
• Reasoning: "If all cats are mammals and all mammals are animals, then all cats are"
• Instruction (usually fails at 100M): "Write a haiku about the moon:"
• Multi-lingual: "El presidente de México es" (if trained on any Spanish)

Sample all 30 at temperature=0 (greedy) after every training pass, save to a versioned file. Diff between runs. When your quantization or fine-tune breaks, the diff will scream.

At 100M/2B tokens, expect:

• Grammar: excellent.
• Facts: hit-and-miss, hallucination heavy.
• Common sense: OK on training-frequent scenarios, wild on novel ones.
• Arithmetic: terrible (needs code + math data + scale).
• Reasoning: terrible.
• Instructions: won't follow — you haven't SFT'd yet (S058).

6 · The eval-harness script for your project

Put this in eval.py and run after every checkpoint:

import subprocess, json, argparse, wandb
 
def run_lm_eval(ckpt_path: str, tasks: list[str], batch_size: int = 32) -> dict:
    cmd = [
        "lm-eval", "--model", "hf",
        "--model_args", f"pretrained={ckpt_path},dtype=bfloat16",
        "--tasks", ",".join(tasks),
        "--batch_size", str(batch_size),
        "--output_path", "/tmp/eval-out.json",
    ]
    subprocess.run(cmd, check=True)
    with open("/tmp/eval-out.json") as f:
        return json.load(f)
 
if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--ckpt", required=True)
    parser.add_argument("--tasks", default="hellaswag,arc_easy,piqa,winogrande")
    args = parser.parse_args()
    results = run_lm_eval(args.ckpt, args.tasks.split(","))
    # Log to wandb
    wandb.init(project="dl-llms-100m-eval", name=f"eval-{args.ckpt.split('/')[-1]}")
    for task, scores in results["results"].items():
        for k, v in scores.items():
            if isinstance(v, (int, float)):
                wandb.log({f"{task}/{k}": v})

Now compare runs on wandb by benchmark. Bar charts across your different training checkpoints show you exactly how each 500 more training steps translates to downstream capability.

7 · When benchmarks lie — the contamination trap

Contamination = your model saw eval data during pretraining. Detection is nontrivial but essential.

Signs of contamination:

• HellaSwag score jumps 10+ points between two adjacent checkpoints.
• MMLU score at 100M above 35% (random is 25%; there is no way a real 100M model beats 30%).
• The model completes a benchmark question verbatim when you feed it the first half.

Prevention:

• Explicitly URL-blocklist known benchmark sources during S046 data cleaning (huggingface.co/datasets, ai2-website).
• Check contamination with SHA-256 of eval questions vs 5-gram lookup in your training shards.

Fixing after the fact: retrain. There is no way to "unlearn" contaminated eval data cleanly.

8 · The one-page report card

After eval, produce a one-page summary. Template:

# 100M-Param Model — Eval Report (2026-XX-XX)
 
Config: 100M params, 2B training tokens, Chinchilla-optimal.
Val loss: 3.82 (PPL 45.6)
 
| Task           | Score | Random | Pythia-160M |
|----------------|-------|--------|-------------|
| HellaSwag (10-shot) | 30.1% | 25% | 33% |
| ARC-easy       | 36.4% | 25% | 40% |
| ARC-challenge  | 25.2% | 25% | 26% |
| PIQA           | 60.2% | 50% | 62% |
| Winogrande     | 52.1% | 50% | 53% |
 
Sample outputs on probes.txt: see /workspace/probes-500.txt
Overall: at par with Pythia-160M on most tasks despite 40% fewer params
         and 5× less compute — good showing.

Commit this to git alongside the checkpoint. Future-you will thank present-you when you're wondering "wait, was that run good or bad?"

9 · War stories

10 · The 2025 eval landscape — what you should actually be running

Benchmarks have quietly become the most contested part of the LLM stack. As models saturated MMLU (top-5 now within 1 point of each other) and HellaSwag (95%+), the community rotated to harder, more contamination-resistant evals. Here's the mid-2025 shortlist.

10.1 · The 2025 eval starter pack

At 100M, only 4 of these give signal: HellaSwag, ARC-Easy, PIQA, WinoGrande. The rest are all-noise. Save yourself the compute and skip them.

10.2 · The 'emergence' curve — what should be flat, what should climb

Run lm_eval at 4 checkpoints (say steps 1500, 3000, 4500, 6000/final). You should see:

• HellaSwag, ARC-Easy, PIQA: monotonic climb from ~random → modest above-chance. Slope tapers.
• WinoGrande, ARC-Challenge: flat near chance for the entire run. Emerges much later (~1B+ params).
• MMLU: flat at 25% ± 1 point. If it's climbing, you have contamination.
• Perplexity on held-out FineWeb-Edu: smooth log-linear decrease. This is your real quality signal.

Wei et al. (2022, "Emergent Abilities of LLMs", arxiv 2206.07682) drew this picture originally. Schaeffer et al. (2023, "Are Emergent Abilities a Mirage?", arxiv 2304.15004) argued the abrupt emergences are metric artifacts — use continuous metrics (log-prob margins, not accuracy) and the curves become smooth. Both are right in different senses; for your 100M run, expect smooth log-linear on continuous metrics and noise-floor on discrete metrics.

10.3 · LLM-as-a-judge and Chatbot Arena — the new default

For instruction-tuned models (you'll get there in M10), automated benchmarks are increasingly replaced by:

• LMSys Chatbot Arena (lmarena.ai): humans vote on pairwise blind comparisons. As of mid-2025, ~4M votes accumulated. The ELO leaderboard is the closest thing the field has to consensus on 'which model is best'. Downside: hobbyist models can't get on it without significant traffic.
• LLM-as-a-judge (Zheng et al. 2023, arxiv 2306.05685): use GPT-4o or Claude 4.5 to score outputs pairwise. Correlates ~0.85 with human ELO on most task categories. Standard framework: MT-Bench (80 multi-turn questions across 8 categories) or AlpacaEval 2.0 (LC-win-rate against GPT-4 responses).
• Arena-Hard (2024): a filtered subset of Arena questions that discriminates best among top models. What every frontier lab reports now.

For your 100M base model, none of these apply — you have no instruction following to measure. But note that Dinesh's day job (LLM-as-a-Judge eval for OneNote Copilot video) is exactly this frontier; the tooling you'll build for M10 is the same tooling MSFT builds in production.

10.4 · Contamination in 2025 — you cannot outsource this

Every major 2024–2025 model release now includes a decontamination report. Standard is Meta's Llama-3 protocol: check every eval sample against training data at 13-gram overlap, report the % of eval samples flagged, and re-run affected benchmarks with those samples removed. Do this for your run. Even at 100M/2B tokens, HellaSwag has ~50 samples that appear verbatim in Common Crawl. Your model 'remembering' 3–5% of a benchmark inflates your score by 2–4 points.

Good tools:

• decontaminate (github.com/allenai/decontamination) — AllenAI's Dolma pipeline decontamination tool.
• lm-eval-harness --predict_only + custom n-gram filter — flexible.
• HuggingFace evaluate library's decontamination module — easy for HellaSwag/ARC.

Further reading (10.x)

• lm-evaluation-harness — github.com/EleutherAI/lm-evaluation-harness
• MMLU-Pro paper — arxiv:2406.01574
• GPQA (Diamond) — arxiv:2311.12022
• LiveBench — livebench.ai
• Chatbot Arena methodology — arxiv:2403.04132
• MT-Bench / LLM-as-judge — arxiv:2306.05685
• Are Emergent Abilities a Mirage? — arxiv:2304.15004
• Llama-3 model card (decontamination §) — ai.meta.com/blog/meta-llama-3

11 · 🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: re-read §1 (perplexity intuition) and §10 (2025 eval landscape) in 24 hours. Revisit the full session on day 7.

Next session (S055): checkpointing — the boring infrastructure that turns a fragile pretrain into a resumable, safe, spot-friendly training pipeline.

Sticky note (keep on your desk): "Held-out perplexity is your only truth. Downstream benchmarks are wobbly proxies below 1B. Decontaminate or the numbers lie."

Legacy recall drills (kept for spaced-review continuity)

Stretch

Run lm-eval on your checkpoint at 4 different points: step 1000, 3000, 5000, and 7500. Plot each benchmark's score over training. Which benchmarks emerge earliest? Which stay flat? Which look noisy? Write one paragraph on what "emergence" looks like at 100M scale.

In your own words

"Training loss lies about model quality because ______, so we also need ______ and ______."

Spaced-review pointer

• S005 (cross-entropy) — loss ↔ perplexity.
• S053 (pretraining) — the run that produced the checkpoint you're evaluating.
• S048 (data mixing) — your mix determines which downstream tasks improve.

Next-session teaser

Session 055 tackles the boring-but-critical infrastructure: checkpointing. What to save, what shape to save it in, how to resume mid-batch after a spot kill, and why every serious training script has a --resume_from latest that Just Works.

Bring back tomorrow

• Formula: PPL = exp(loss).
• Rule: HellaSwag > MMLU at small scale for detecting real improvements.
• Number: Your 100M model target — PPL ~45, HellaSwag ~30%.

Previous: ← DL S053 · Next: DL S055 →