DL S047 · Chinchilla Scaling Laws

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 47 / 80 · Module M08 · ~2 hours

The story we're starting with

In 2020, OpenAI published Scaling Laws for Neural Language Models — the Kaplan et al. paper. The takeaway everyone repeated at parties: "make models bigger, don't worry as much about data." For two years the field trained enormous, undertrained models. GPT-3 (175B params, ~300B tokens) is the poster child. Megatron-Turing NLG (530B params, ~270B tokens) is another. Gopher (280B, ~300B tokens) too.

Then in 2022 DeepMind published Training Compute-Optimal Large Language Models — a.k.a. Chinchilla — and quietly detonated a bomb under the whole field. Their claim: given a fixed compute budget, you should scale model parameters and training tokens in equal measure. Specifically, for compute-optimal training you want ≈20 tokens per parameter. That means Gopher (280B params) should have seen ~5.6 trillion tokens, not 300 billion. It was under-trained by a factor of ~18×.

They proved it by training Chinchilla — 70B params, 1.4T tokens — with the same FLOP budget as Gopher, and Chinchilla beat Gopher on almost every benchmark. Same compute, quarter the parameters, five times the data, way better model.

The whole industry pivoted. Llama 1 (7B, 1T tokens), Llama 2 (7B, 2T), Llama 3 (8B, 15T) — each generation pours more tokens per parameter, going well past the Chinchilla "optimum" because inference cost dominates deployment economics. This session teaches you why, gives you the actual power laws, and by the end you'll be able to look at any published model and say "under- or over-trained, and by how much."

Prerequisites

• S041 (Training your decoder on Tiny Shakespeare) — you should have seen a real loss curve on a real model.
• S006 (Optimization intuition) — for the "why does loss follow a power law?" intuition.

1 · The one equation that runs the entire field

Before any scaling laws, you need this:

Training a decoder transformer with N parameters on D tokens costs approximately C ≈ 6ND FLOPs.

Where the 6 breaks down as:

• 2ND for the forward pass (one multiply + one add per parameter per token — a matrix multiply is 2 FLOPs per weight)
• 4ND for the backward pass (backprop is roughly 2× the forward)

This is the formula. Every cost estimate, every scaling law plot, every "how much would it cost to train GPT-4" tweet reduces to this.

1.1 Where does the 6 come from?

Take a linear layer y = Wx where W is [d_out, d_in]. Forward: d_out * d_in multiplies + d_out * d_in adds ≈ 2 · d_out · d_in FLOPs per token. That's 2 FLOPs per parameter, per token. Sum across all layers → 2N FLOPs per token forward.

Backward pass computes both dL/dW (gradient w.r.t. weights) and dL/dx (gradient w.r.t. input, needed to keep chaining backward). Each is another 2N FLOPs per token. Total backward = 4N per token. Grand total per token = 6N. Multiply by D tokens processed → 6ND.

Attention adds a small overhead (~2 · n_layers · seq_len · d_model extra per token) but for models where d_model >> seq_len (any modern LLM) it's <10% of the total and everyone ignores it in first-cut estimates.

1.2 Sanity-check on a real number

GPT-3 175B trained on 300B tokens:

C = 6 · N · D = 6 · 175e9 · 300e9 = 3.15e23 FLOPs

The published number in the GPT-3 paper: 3.14e23 FLOPs. Matches to 0.3%. The formula is that precise for any real transformer training run.

The point: if you know N and D you know the cost. And if you know your GPU budget you know N and D. Everything else is derived.

2 · Kaplan (2020) — the paper we now think was mostly wrong

Kaplan et al. fit a power law to a bunch of small-model training runs and concluded that for a fixed compute budget C, optimal parameters scale as:

N_opt ∝ C^0.73
D_opt ∝ C^0.27

Read that carefully. N (params) grows nearly 3× faster than D (tokens) as compute grows. In practical terms: if you 10× your compute, Kaplan says you should scale params ~5.4× and tokens ~1.9×.

This gave the field permission — even the directive — to build enormous, undertrained models. GPT-3, Gopher, MT-NLG, PaLM all followed this recipe. In hindsight the Kaplan fit was drawn on a slice of the loss surface that was plateaued in D-dimension — his runs had way too few tokens to see the "keep training" gains that dominate at scale.

3 · Chinchilla (2022) — the three approaches

The Chinchilla paper's genius is that it fits the scaling law three different ways and gets the same answer from all three. When your empirical estimate matches under three independent methodologies, you trust it.

3.1 Approach 1 — IsoFLOP curves

Fix a compute budget C. Train many models of different sizes N (holding D = C/(6N) so total FLOPs is constant). Plot final loss vs N. It's a U — too small and the model can't fit the data; too big and you starved it of tokens. Read the bottom of the U → that's the optimal N for this C.

Do this for many C values → plot optimal N vs C → power law fit.

3.2 Approach 2 — Loss curves from all runs

Fit a single joint model:

L(N, D) = E + A/N^α + B/D^β

Where:

• E is the "irreducible loss" (data entropy — the ceiling nothing can beat)
• A/N^α is the loss due to finite model capacity
• B/D^β is the loss due to finite training data

DeepMind fit this on ~400 runs. Their numbers:

E = 1.69     α = 0.34     β = 0.28
A = 406.4    B = 410.7

Notice α ≈ β. That's the key insight. Loss falls at roughly equal rates in N and D. So if you want to minimize L(N, D) subject to 6ND = C, calculus gives you N ∝ C^0.5, D ∝ C^0.5.

3.3 Approach 3 — Extrapolating from all training runs

Fit the same L(N,D) form but using the loss along the entire training trajectory, not just the final loss. This wrings way more signal out of the same compute.

All three give the same answer to within a few percent:

4 · The 20-tokens-per-param rule, in numbers

Let's tabulate a few compute budgets. All at MFU=40% on A100s.

For our M09 100M-param toy model the optimum is ~2B tokens — right in the ballpark our S046 corpus (1B tokens) approaches. At 1B tokens for 100M params (10 tok/param) we're at half-Chinchilla-optimal, which is fine for a learning project — we'll pay a small loss penalty in exchange for cheap iteration.

4.1 The calculus, if you want to see it

Minimize L(N, D) subject to 6ND = C. Lagrangian:

∂/∂N [ A/N^α + B/D^β + λ(6ND - C) ] = 0
∂/∂D [ same ] = 0

Working it out:

-αA/N^(α+1) + 6λD = 0
-βB/D^(β+1) + 6λN = 0

Divide:

Substituting C = 6ND and solving for N in terms of C gives:

N_opt = G · C^{β/(α+β)}
D_opt = G'· C^{α/(α+β)}

With α=0.34, β=0.28: N_opt ∝ C^0.45, D_opt ∝ C^0.55. Chinchilla rounded to 0.5/0.5 for simplicity — the exact exponents drift slightly with the specific fit.

5 · The over-training era — why real models blow past 20 tokens/param

Chinchilla optimizes training compute only. But real deployed models are served — you pay inference cost for every user request forever. Inference cost scales linearly with N (bigger model = slower per token). So for a fixed quality target, if you can make a smaller model reach the same quality by giving it more tokens, your inference bill drops permanently.

Llama 2 7B was trained on 2T tokens (286 tok/param — 14× past Chinchilla). Llama 3 8B was trained on 15T tokens (1875 tok/param — 94× past Chinchilla). Phi-3-mini 3.8B was trained on ~3.3T tokens (868 tok/param — 43× past).

They knew they were "wasting" training compute vs Chinchilla-optimal. They did it on purpose because inference savings dominate at scale.

The rule of thumb from Meta's Llama 3 paper:

"For a model that will serve ≥10 billion inference tokens, over-training past Chinchilla by 10× pays back."

For your 100M pretraining run, this doesn't matter — you're not serving it to millions. Aim for Chinchilla (~2B tokens) and don't stress the last 20% of loss.

6 · Reading a scaling law plot

Every scaling paper draws IsoFLOP curves. Here's the mental picture:

Each U has:

• Left side (too small): model can't fit the data — capacity-limited.
• Bottom: optimum for this compute — the compute-optimal size.
• Right side (too big): you had to cut D too much to fit N → data-limited.

Plotting the minima of all these Us against C gives you N_opt(C) — a straight line on log-log, slope ~0.5 → the Chinchilla relation.

7 · A cheap Python "Chinchilla calculator"

Handy to have. Save as chinchilla.py:

def chinchilla(compute_flops: float) -> tuple[float, float]:
    """Return (N_opt, D_opt) for a given compute budget in FLOPs."""
    # Fit constants from Hoffmann et al. 2022, Approach 3.
    N = 0.6 * (compute_flops / 6e21) ** 0.5 * 1e10
    D = 20 * N
    return N, D
 
def cost_from_gpu_hours(a100_hours: float, mfu: float = 0.4) -> float:
    return a100_hours * 312e12 * mfu * 3600
 
# Usage
compute = cost_from_gpu_hours(100)     # 100 A100·hours
N, D = chinchilla(compute)
print(f"N_opt = {N/1e6:.0f}M params, D_opt = {D/1e9:.1f}B tokens")

Output: N_opt = 520M params, D_opt = 10.4B tokens. Use this before every training run. Saves you from training a beautifully-architected model with no data.

8 · Where the scaling laws break

Scaling laws are extrapolations from empirical fits. Places they mislead:

• Data quality dominates near the optimum. Chinchilla assumes "generic web text at fixed quality." Filter your data harder and the effective β goes up — you get more per token. See RefinedWeb, FineWeb-Edu, Phi papers.
• Repeating epochs is not free. Chinchilla assumes fresh tokens. Epoch-2 tokens give you diminishing returns (~50% as valuable), and by epoch 4 you're near zero improvement. This caps how far you can push D on a fixed corpus.
• Small models (<10M params) don't follow the law. The fits break down below the scale where you have enough capacity to see structure.
• Very-long-context degrades throughput. The +2 · n_layers · seq_len · d_model term stops being negligible past 8k context. At 128k context, attention is ~30% of FLOPs, invalidating the 6ND shortcut.

9 · Worked example — pricing your own 100M-param run

You're going to pretrain a 100M-param model (M09). What should you budget?

N = 1e8
D_opt = 20 · N = 2e9 tokens
C_opt = 6 · N · D = 6 · 1e8 · 2e9 = 1.2e18 FLOPs

On an A100 at MFU=40%:

seconds = 1.2e18 / (312e12 · 0.4) = 1.2e18 / 1.25e14 = ~9600 seconds
        = ~2.7 A100-hours

At spot pricing ($1–2/hr for A100 40GB on Runpod/Vast):

$3–$6 for the pretraining pass.

That's the entire flagship module. Six bucks. This is why we can do this in a personal project. A 1B-param run would be ~100× more compute (still under $600) and you'd hit real-world scaling territory.

10 · Diagram — the whole compute triangle

Whenever you're stuck deciding "bigger model or more data?", redraw this triangle. Two of the three fix the third.

11 · War stories

12 · Retention scaffold

Recall — answer from memory

The post-Chinchilla era (2023–2025): overtraining and the death of the "optimal" model

Chinchilla told us the compute-optimal ratio is ~20 tokens per parameter. The industry read that, nodded politely, and then trained everything to 100–500× that ratio. Why? Because Chinchilla optimizes for training loss at fixed training FLOPs — and nobody actually cares about that. What you care about is inference cost per query for the next 3 years and loss at your deployed model size.

A parade of 2023–2025 papers formalized this:

• Hoffmann et al. 2022 (Chinchilla, arxiv 2203.15556) — the origin. D/N ≈ 20 for compute-optimal.
• Touvron et al. 2023 (Llama 2) — 7B trained on 2T tokens (D/N ≈ 285). "Chinchilla-optimal would have been 140B tokens. We trained on 14× more because we want a small model at inference time."
• Muennighoff et al. 2023, Scaling Data-Constrained Language Models (arxiv 2305.16264) — what if you don't have infinite unique data? Answer: you can repeat data up to ~4 epochs before returns diminish sharply. This paper reshaped the mixing decisions in S048.
• Sardana et al. 2024, Beyond Chinchilla-Optimal (arxiv 2401.00448) — explicitly formulates inference-aware scaling. If you plan to serve N_infer queries over the model's lifetime, the optimal (N, D) shifts smaller-and-longer. Recovers exactly the Llama 2/3 recipe.
• Gadre et al. 2024, Language Models Scale Reliably with Over-Training (arxiv 2403.08540) — predicts downstream benchmark accuracy from training loss and (N, D), showing the extrapolation is remarkably clean up to 900× overtraining.
• Llama 3.1 8B: 15T tokens on 8B params = D/N ≈ 1875 — the loudest "we ignored Chinchilla" statement in the industry. And the model is fantastic, because it will be served trillions of times.

The modern mental model:

Chinchilla only minimizes the first term. In 2025 you minimize the sum. For any model you plan to deploy at scale (Llama 3, GPT-4o, Mistral Large), $N_{\text{infer}} \cdot C_{\text{infer}}(N)$ dwarfs $C_{\text{train}}$ within months of release. Overtraining a smaller model is the rational choice.

Stretch

Take three real published models (any of GPT-3, Chinchilla, Llama 1/2/3, Mistral 7B, Phi-3, DeepSeek V3, Qwen 2.5, GPT-4 if you can find a leak) — look up (N, D) from their papers, compute D/N, C = 6ND, and rank them by "how far past Chinchilla optimum". Draw a bar chart. This exercise will change how you read model release blog posts forever.

In your own words

Fill in the sentence: "For a fixed compute budget of C FLOPs, spending more on parameters instead of tokens is worse because ______________________."

Spaced-review pointer

• S041 (Tiny Shakespeare) — the loss curve you got there was pinned by a similar (though much smaller) L(N,D) surface. Re-plot it if you have the run.
• S046 (Pretraining dataset) — the corpus you built is your D. This session tells you whether you have enough for a given N.

Next-session teaser

You now know how much data. Next: what kind? A 20 tok/param budget spent on pure Common Crawl is worse than the same budget split 40% web / 30% code / 15% books / 10% arxiv / 5% wiki. Session 048 — data mixing — is where we learn how to design the mix and how to run cheap ablations to defend it.

What to bring back tomorrow — sticky note

• Equation: C ≈ 6ND.
• Ratio: D/N ≈ 20 for compute-optimal.
• Snippet: the 3-line chinchilla(compute_flops) calculator.

Previous: ← DL S046 · Next: DL S048 →