S114 · Scaling Laws — Chinchilla, Compute-Optimal Training

Module M13: NLP & Transformers · Session 114 of 130 · Track: LLM 📈

What you'll be able to do after this session

• Explain Scaling Laws to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S112 — Encoder (BERT), Decoder (GPT), Enc-Dec (T5) — When Each

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Chinchilla explained in 5 minutes — The 'compute-optimal' rule of thumb in a nutshell.
• 🎥 Deep dive (30–60 min) · Neural scaling laws — Yannic Kilcher paper walkthrough — 40 min line-by-line through the Kaplan + Chinchilla papers.
• 🎥 Hands-on demo (10–20 min) · How LLMs are trained (compute, data, params) — StatQuest — 15-min hands-on tour of what a training run actually costs.
• 📖 Canonical article · Training Compute-Optimal Large Language Models (Chinchilla) — DeepMind's paper — the ~20 tokens per parameter rule.
• 📖 Book chapter / tutorial · Scaling Laws for Neural Language Models (Kaplan et al.) — The OpenAI paper that started it all. Read §3 for the power-law intuition.
• 📖 Engineering blog · PaLM: scaling language models to 540B parameters — Google's public post on what breaks when you 10× a model.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Why bigger models keep winning.

You have a fixed training budget — say $10M of GPU time. You can spend it on a bigger model (more parameters), or more data (more tokens), or longer training (more steps). What mix minimizes final loss? For years everyone assumed "bigger model always wins" (Kaplan 2020). Then DeepMind's Chinchilla paper (2022) said the opposite: most models were undertrained. For every parameter, you want roughly 20 tokens of training data.

Analogy: think of training a student for an exam. You can pay for a bigger brain (parameters), more textbooks (data), or more study hours (compute). Kaplan said "just get a bigger brain." Chinchilla measured actual test scores and found: a 70B-param model trained on 1.4T tokens beats a 175B-param model trained on 300B tokens using the same compute — because the smaller model got more practice problems per neuron. GPT-3 (175B / 300B tokens) turned out to be a 4× undertrained model.

Gotcha to carry forever: loss follows a power law in compute, params, and tokens. Doubling any one input gives you predictable — not linear — improvement. If you're bottlenecked on tokens (data), adding more params buys nothing. If you're bottlenecked on params, adding more tokens buys little. The magic ratio (Chinchilla): tokens ≈ 20 × parameters. And it's why Llama-3 8B was trained on 15T tokens (~1875 tokens per param) — modern practice is "overtrain small models" because inference is cheap forever but training is one-time.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Worked example — you have 10²² FLOPs (roughly $1M compute at 2024 GPU prices).

Chinchilla formula: C = 6 · N · D (rough training FLOPs for a Transformer). Solve for compute-optimal split: N_opt ≈ (C / 120)^0.5 and D_opt ≈ 20 · N_opt.

Kaplan (2020) vs Chinchilla (2022) — same budget, different splits:

Same FLOPs, half the parameters, 5× the data — better model. Also 2.5× faster/cheaper at inference. That's why Llama, Mistral, and Gemma all shipped small-and-overtrained.

Power-law intuition. Loss ≈ A / N^α + B / D^β + irreducible. The exponents α, β are small (~0.3), which means you need to double your resources for each linear drop in loss. Compute grows exponentially, wins shrink logarithmically — hence the trillion-dollar GPU market.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Plot the Chinchilla curve yourself and see the trade-off:

# pip install numpy matplotlib
import numpy as np, matplotlib.pyplot as plt
 
def loss_estimate(N, D):
    # Chinchilla-style approximation (Table A9 of arXiv:2203.15556)
    # L(N, D) = E + A/N^alpha + B/D^beta
    E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28
    return E + A / (N ** alpha) + B / (D ** beta)
 
compute_budgets = [1e20, 1e21, 1e22, 1e23, 1e24]
for C in compute_budgets:
    # Grid-search the (N, D) that minimises loss subject to 6*N*D <= C
    Ns = np.logspace(7, 12, 200)                    # 10M to 1T params
    best = None
    for N in Ns:
        D = C / (6 * N)                             # spend all compute
        if D < 1e6: continue
        L = loss_estimate(N, D)
        if best is None or L < best[2]:
            best = (N, D, L)
    N, D, L = best
    print(f"C={C:.0e} | optimal N={N/1e9:6.1f}B params | D={D/1e9:6.1f}B tokens "
          f"| tokens/param={D/N:5.1f} | loss={L:.3f}")
 
# Also: how much loss do you leave on the table if you go 4x undertrained?
C = 1e23
N_opt = 30e9; D_opt = C / (6 * N_opt)                 # roughly Chinchilla-optimal
N_bad = 120e9; D_bad = C / (6 * N_bad)                # Kaplan-style overparam
print()
print(f"Optimal:      N={N_opt/1e9}B  D={D_opt/1e9:.0f}B  L={loss_estimate(N_opt,D_opt):.3f}")
print(f"4x undertrained: N={N_bad/1e9}B D={D_bad/1e9:.0f}B  L={loss_estimate(N_bad,D_bad):.3f}")

Checklist:

1. tokens/param should land near 20 for every compute budget — that's Chinchilla.
2. Loss decreases monotonically as compute grows (power law).
3. The "4× undertrained" run has a higher loss with the same total FLOPs.
4. Doubling compute roughly reduces loss by a small, fixed absolute amount — not linearly.
5. Try N = 1e12 (1T params) at C = 1e22 — loss gets worse because you're starving the model of data.

Try this: re-fit the constants using a small toy language model on wikitext-103 for compute budgets {1e15, 1e16, 1e17} — plot loss vs compute on log-log axes. You should see a straight line. That's a scaling law in the wild.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Scaling laws are not academic curiosity — they are the product roadmap of every frontier lab. Every $100M training run starts with a spreadsheet that says "at this compute budget, this (N, D) will hit this loss." If they're wrong by 10%, someone gets fired.

War stories. (1) GPT-3 was undertrained by 4× — one of the most valuable insights of the decade, and it took two years to spot. Chinchilla showed a 70B model trained on 1.4T tokens beats a 175B model trained on 300B tokens at the same compute. Everyone (OpenAI, Anthropic, Google) quietly re-baselined. (2) Kaplan overweighted the params term because their small-scale runs weren't near the loss floor. Lesson: extrapolating scaling laws from small runs is dangerous — you must sweep multiple orders of magnitude. (3) Llama 2 → Llama 3 ratio jumped from ~30 tokens/param to ~1800 — Meta explicitly overtrained past Chinchilla-optimal because inference cost dominates lifetime cost. If your model will serve 1B queries, a 5% smaller model saves more than the extra training. (4) The "data wall" — trillion-token models are running out of high-quality human text. Synthetic data (rephrasing, self-play, model-generated Q&A) is the current hack; nobody knows if it'll hold.

How top teams operate. DeepMind, OpenAI, and Anthropic fit their own scaling laws internally (constants differ from Chinchilla depending on tokenizer + data mix + architecture). They then use IsoFLOP plots — "at fixed compute, what N gives lowest loss?" — to decide their next training run's shape. Emergent capabilities (in-context learning, chain-of-thought, tool use) show up as phase transitions on scaling curves — often at the trillion-token scale. It's why every serious lab is currently racing for 10²⁶ FLOPs (roughly $1B of GPUs) and above.

(e) Quiz + exercise · 10 min

1. State the Chinchilla rule of thumb in one line (tokens per parameter).
2. Given a fixed compute budget, what happens to loss if you double params without adding data?
3. Why is Llama-3 8B trained on 15T tokens (way past Chinchilla-optimal)?
4. Roughly what formula relates training FLOPs to N and D?
5. Name one factor scaling laws don't predict.

Stretch (connects to S102 — Gradient Descent): Scaling laws assume you actually reach the loss floor. What optimizer / batch-size / learning-rate choices from S102 must you get right for a $100M run to hit its predicted loss? Give three.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Scaling Laws? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S115): Efficient Attention — Flash, Sparse, Linear
• Previous (S113): LLM Sampling — Greedy, Beam, Top-k, Top-p, Temperature
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.