Scaling Laws — Chinchilla, Compute-Optimal Training

Why this session exists

At fixed training compute, a larger model leaves fewer tokens to train it. Scaling studies fit that tradeoff rather than prove that the largest model wins. A valid calculator must distinguish an empirical loss fit from a rough tokens-per-parameter heuristic, preserve units and actually spend the supplied budget.

The original calculator's N=0.6*sqrt(C), D=20*N spent 43.2 times C under its own 6ND cost model. This lesson derives two internally consistent calculators, shows their different answers, and explains why neither reveals a private model's design or guarantees future benchmark accuracy.

Outcomes: compare Kaplan and Chinchilla; derive a constrained optimum; compute allocations and verify FLOPs; interpret public parameter/token counts without an absolute “undertrained” label; and incorporate serving demand, data limits and architecture into a sizing decision.

Prerequisites

Use raw parameter and token counts, not billions unless explicitly converted. One TFLOP/s is 10¹² floating-point operations per second; a GPU count or a day is not itself a FLOP count. Natural-log cross-entropy is measured in nats per token and depends on the tokenizer and evaluation distribution.

(a) Intuition · 5 min

A house budget can buy floor space or finish quality. A training budget can buy capacity or exposure. The analogy helps explain tradeoffs but does not derive a universal optimum: data quality, optimization, architecture and deployment constraints matter. A smooth fitted curve is a model of observed experiments, not a physical law.

The two-paper history

Kaplan et al. (2020) study power-law behavior in model size, data and compute, using a particular experimental/compute convention. Their compute-efficient scaling allocates roughly N proportional to C^0.73 and training tokens proportional to C^0.27 in the relevant regime [1]. It is not a constant 1.5–2 tokens per parameter recommendation. GPT-3's 175B/300B ratio is one model point, not the law.

Hoffmann et al. (2022) train over 400 models spanning 70M–16B parameters and 5B–500B tokens and compare several methods for the compute frontier [2]. Their headline result is approximately equal scaling of parameters and tokens. Chinchilla (70B, 1.4T) is compared at the same training budget as Gopher, not the same budget as GPT-3. Learning-rate schedule choices and fitting methods are relevant to the difference; reducing the whole history to “Kaplan forgot the schedule” overstates a single explanation.

LLaMA (2023) explicitly discusses models that are economical at inference, and later Llama 3 reports much larger training token counts [3,4]. “Overtrained relative to a training-only frontier” does not mean overfitted or badly designed. A tokens/parameter ratio alone cannot identify the optimum for every corpus or serving objective.

The compute–loss universe

For a dense, parameter-matmul-dominated Transformer, a common approximation is

The leading factor comes from about 2 operations per multiply-add in the forward matrix products and about twice the forward work for backward input/weight gradients. It is not three identical passes called forward, backward and optimizer. Optimizer updates, embedding/attention terms, activation recomputation, padding and implementation details can change accounting. Kaplan's N excludes vocabulary and positional embeddings in its convention; do not silently mix that with an arbitrary total-parameter figure.

The two rules head-to-head

These are not interchangeable. The rounded published fit α=.34, β=.28 does not imply an exactly constant D/N=20. The headline equal-scaling approximation summarizes multiple approaches; it is not the precise minimizer of every reported rounded fit.

The core formulas

Ratio-constrained allocation. Choose a positive ratio r=D/N. Substituting into C=6ND gives

For r=20 and C=10²⁴, N≈91.287B and D≈1.82574T. Check: 6ND=10²⁴. The old 600B/12T answer instead costs 4.32×10²⁵, exactly 43.2× the budget.

Fitted-loss allocation. Use the paper's rounded illustrative parameters A=406.4, B=410.7, E=1.69, α=.34, β=.28 [2]. Substitute D=C/(6N):

Differentiate with respect to log N. At the optimum:

The exponents are approximately .451613 for N and .548387 for D. In log N the sum of two positive exponentials is strictly convex, so this idealized unconstrained surface has a unique minimum. For C=10²⁴ this fit gives N≈41.297B, D≈4.03583T and D/N≈97.728. This differs substantially from the ratio-20 heuristic. Reporting both is more honest than presenting one formula as both calculations.

A budget worked all the way through

Assume 100 devices × 30 days × sustained 300 TFLOP/s per device. This is a hypothetical sustained rate, not a measured H100 benchmark or peak-to-usable conversion. The budget is 100×30×86400×300×10¹² = 7.776×10²² FLOPs.

• Ratio 20: N≈25.456B, D≈509.117B tokens.
• Rounded parametric fit: N≈13.031B, D≈994.577B tokens.
• Both satisfy 6ND=C. Which estimate is useful depends on whether the fit describes the actual architecture, schedule, data and tokenization.

At fixed C, halving N means doubling D. Halving N and quadrupling D doubles the budget. Multiplying D/N by r while preserving compute requires N divided by √r and D multiplied by √r. Dividing N by five and multiplying D by five produces a 25× ratio increase, not fivefold.

What the papers plotted

A log-log line for L−E versus N corresponds to multiplicative error reduction with multiplicative resources; log L itself need not remain linear near the floor. IsoFLOP curves sweep N while adjusting D to maintain compute, and their minima trace a frontier. Learning curves at a fixed N answer a different question. Fitted curves should retain experimental endpoints, schedule choices, parameter definitions and uncertainty before extrapolating many orders of magnitude.

The decomposition into capacity error, data error and irreducible entropy is an interpretation of the empirical fit, not three proven independent error sources. Cross-entropy against a true target distribution is minimized at its entropy. But E=1.69 is a fitted constant for a specific regime, not “the entropy of all natural language.” A heavy tail can motivate scaling models, but it does not force this exact additive form. “Multiply resources by k to subtract a fixed loss amount” describes logarithmic behavior, not a power law; a power law multiplies the excess loss by a fixed factor.

One budget, two levers, and a fitted floor

Use the fit to find the current bottleneck, but do not say improving the smaller term is literally useless: both terms have nonzero derivatives. Repeated data is not equivalent to unlimited fresh data. Token quality, deduplication, contamination, optimizer convergence and domain mismatch may invalidate a fit. Task capabilities can move differently from aggregate loss. A synthetic optimizer demonstration does not prove emergent abilities or a universal inference-time-compute law.

(c) Hands-on · 25 min

This is a pure mathematical calculator, not model training. It implements both formulas, checks the budget identity, compares the analytic stationary point to SciPy's independent scalar minimizer and rejects the original formula as a negative control. The public model rows are rounded reference numbers for arithmetic, not a leaderboard or a universal fitted-loss prediction.

import math
import numpy as np
from scipy.optimize import minimize_scalar
A,B,E,alpha,beta=406.4,410.7,1.69,.34,.28
 
def positive(*xs):
    if any(not math.isfinite(x) or x<=0 for x in xs):raise ValueError('finite positive inputs required')
def compute_flops(n,d):positive(n,d);return 6*n*d
def chinchilla_loss(n,d):positive(n,d);return E+A/n**alpha+B/d**beta
def ratio_split(c,ratio=20):
    positive(c,ratio);n=math.sqrt(c/(6*ratio));return n,ratio*n
def fitted_split(c):
    positive(c)
    logn=(math.log(alpha*A/(beta*B))+beta*math.log(c/6))/(alpha+beta)
    n=math.exp(logn);return n,c/(6*n)
 
for c in (1e18,1e21,1e24,100*30*86400*300e12):
    n,d=fitted_split(c);rn,rd=ratio_split(c)
    assert math.isclose(compute_flops(n,d),c,rel_tol=2e-15)
    assert math.isclose(compute_flops(rn,rd),c,rel_tol=2e-15)
    # Independent numerical optimizer in log-parameter space.
    obj=lambda logn:chinchilla_loss(math.exp(logn),c/(6*math.exp(logn)))
    result=minimize_scalar(obj,bounds=(math.log(n)-5,math.log(n)+5),method='bounded',options={'xatol':1e-10})
    assert abs(math.log(n)-result.x)<1e-5
    assert obj(math.log(n))<obj(math.log(n)+math.log(2))
    assert obj(math.log(n))<obj(math.log(n)-math.log(2))
    eps=1e-4;der=(obj(math.log(n)+eps)-obj(math.log(n)-eps))/(2*eps)
    assert abs(der)<1e-7
    assert math.isclose(alpha*A/n**alpha,beta*B/d**beta,rel_tol=1e-12)
    print('C',c,'ratio20 N,D',rn,rd,'fit N,D',n,d,'fit ratio',d/n,'loss',chinchilla_loss(n,d))
# Reject the original 0.6 sqrt(C) recommendation: it spends 43.2 times the budget.
c=1e24;wrongn=.6*math.sqrt(c);wrongd=20*wrongn
assert math.isclose(compute_flops(wrongn,wrongd)/c,43.2)
# Halving N and doubling D preserves compute; quadrupling D doubles it.
n,d=ratio_split(c)
assert compute_flops(n/2,d*2)==compute_flops(n,d)
assert compute_flops(n/2,d*4)==2*compute_flops(n,d)
# Multiplying the tokens/parameter ratio by r at fixed compute uses sqrt(r).
r=5;nn,dd=n/math.sqrt(r),d*math.sqrt(r)
assert math.isclose(dd/nn,r*d/n) and math.isclose(compute_flops(nn,dd),c)
models=[('GPT-3 reference',175e9,300e9),('Chinchilla',70e9,1.4e12),
        ('LLaMA 65B',65e9,1.4e12),('Llama2 70B',70e9,2e12),
        ('Llama3 8B rounded 15T',8e9,15e12),('Llama3 70B rounded 15T',70e9,15e12),
        ('Llama3 405B rounded 15T',405e9,15e12)]
for name,n,d in models:print(name,'tokens/parameter',d/n,'ratio relative to 20',d/n/20,'6ND',compute_flops(n,d))
c=25000*90*86400*400e12;n,d=ratio_split(c)
print('hypothetical large cluster',c,'ratio20',n,d,'ratio100',n/math.sqrt(5),d*math.sqrt(5))
# Lifetime break-even: extra training FLOPs divided by per-token inference saving.
extra=1e22;big,small=20e9,10e9
q=extra/(2*(big-small));assert q==5e11
for bad in (0,-1,float('nan'),float('inf')):
    try:fitted_split(bad)
    except ValueError:pass
    else:raise AssertionError('invalid budget accepted')
print('analytic allocation, numerical optimization, budget identities and negative controls passed')

What each block does

compute_flops enforces positive finite input. ratio_split solves the stated ratio constraint. fitted_split computes in log space and obtains D from the remaining budget, preventing silent constraint drift. The numerical oracle minimizes the substituted loss independently rather than calling the analytic formula again. Perturbing log N both upward and downward increases the modeled loss.

Reference ratios are GPT-3 175B/300B ≈1.714; Chinchilla 70B/1.4T=20; LLaMA 65B/1.4T≈21.538; Llama 2 70B/2T≈28.571. Using a deliberately rounded 15T for Llama 3 gives 1875 for 8B, 214.286 for 70B and 37.037 for 405B. The v3 report explicitly gives 405B parameters and 15.6T text tokens for the flagship, while describing the corpus as about 15T [4]. It also calls the flagship approximately compute-optimal under its own scaling laws while deliberately training the smaller models longer. The calculator's rounded 15T rows do not claim exact equality across checkpoints/stages or override that local fit. The earlier unverified Qwen row is not used as source evidence for a universal release comparison.

Under 6ND, GPT-3's reference run is 3.15×10²³ FLOPs and Chinchilla's is 5.88×10²³. This alone disproves the page's old “same compute as GPT-3” statement. Actual paper accounting can include additional terms; the paper's controlled same-budget comparison is Gopher.

Practice: a hypothetical large cluster, not a private-model forecast

For 25,000 devices, 90 days and an assumed sustained 400 TFLOP/s, C=7.776×10²⁵. Ratio-20 allocation gives about 804.984B parameters and 16.0997T tokens. To increase the tokens/parameter ratio fivefold at fixed compute, use about 360B and 36T. These are scenario calculations; they reveal nothing about GPT-5's undisclosed hardware utilization, active parameters, data or architecture. Compare any future disclosure only if its units and definitions match.

(d) Production reality · 15 min

Training-optimal versus lifetime-optimal

Suppose a smaller model requires ΔC extra training FLOPs to meet a quality target. In a rough dense inference model, switching from N_big to N_small saves about 2(N_big−N_small) FLOPs per generated token, ignoring context attention and systems effects. Break-even inference volume is

With ΔC=10²² and a reduction from 20B to 10B, Q_break=5×10¹¹ tokens. This is FLOP accounting, not a currency/payback promise. Real latency depends on bandwidth, batching, quantization, prefill/decode mix and utilization. A smaller N does not uniquely determine KV-cache size; layer count, head dimensions and KV sharing do. There is no universal 10× or 1000× “sane overtraining” ceiling.

Data, MoE and operational constraints

MoE separates total expert parameters from the subset used per token. Leading compute tracks activated paths plus routing and shared work; storage includes all required weights unless offloaded or sharded. Communication, load balance and capacity change both training and serving. Substituting total N into a dense fit is not a valid MoE scaling law, and a public flagship name is not evidence that it uses MoE.

Fit local small runs for the intended data/schedule and reserve held-out budgets to test extrapolation. Track training and validation curves, failure/restart/checkpoint overhead, memory and sustained throughput. A flattening curve could signal optimization problems or repeated-data limits, not only insufficient N. No actual GPU run, rental quote, energy audit or modern-model loss reproduction is claimed by this calculator.

Where this shows up next

Efficient attention reduces memory traffic and can improve usable throughput; it does not automatically reduce the 6ND parameter-matmul coefficient or remove dense quadratic attention arithmetic. Fine-tuning and embedding-model scaling need their own regimes; LoRA is not simply “the fine-tune analogue of overtraining.” Serving demand and sustainability add costs beyond training FLOPs. Closed-model names without disclosed N/D cannot be ranked by plugging guesses into Chinchilla.

(e) Recall + stretch · 10 min

1. Compute formula? Approximate dense training C≈6ND; forward matmul work plus roughly twice that backward work, not an optimizer pass equal to a forward pass.
2. Twenty tokens? A rough allocation heuristic related to the Chinchilla result; choose r=20 and solve N=√(C/120). It is not a universal exact optimum.
3. Kaplan versus Chinchilla? Earlier compute-frontier scaling favored parameter growth more strongly; Chinchilla's methods favored roughly joint scaling. Neither is a universal fixed-ratio theorem.
4. GPT-3 ratio? About 1.714 versus 20, suggesting relatively fewer tokens under that heuristic. The matched Chinchilla compute comparator is Gopher, not GPT-3.
5. Llama 3 smaller models? Longer training can buy a deployable small checkpoint with lower serving cost. The report supports the inference-efficiency motivation, not a universal payback in weeks.
6. 10²⁴ FLOPs? Ratio-20 gives 91.287B/1.82574T; the rounded loss fit gives 41.297B/4.03583T. State which model of allocation you used.
7. Why additive fit? It models capacity/data limitations plus a floor; it is empirically fitted and not forced by Zipf's law. Multiplicative resource increases imply multiplicative excess-loss changes under a power law.
8. Why refit? Data, tokenizer, architecture and schedule change the regime and uncertainty. No universal .02 exponent shift or dollar penalty is established here.
9. Overtraining limits? Decide using marginal quality, fresh/repeated data and lifetime cost. There is no proven 100× practical ceiling.
10. MoE? Separate activated compute from total storage/capacity and routing/communication. Dense N/D equations do not transfer unchanged.

Explain-out-loud test

Derive the stationary condition, verify both allocations at 10²⁴ FLOPs, explain the 43.2× negative control and state the extra information needed before converting an allocation into a real training plan.

What comes next

Efficient attention separates mathematical cost from execution and memory. Learning-plan hub.

Primary references

1. Kaplan et al., Scaling Laws for Neural Language Models, 2020.
2. Hoffmann et al., Training Compute-Optimal Large Language Models, 2022, §3 and Appendix D.
3. LLaMA, 2023.
4. The Llama 3 Herd of Models, v3, 2024.