Chinchilla scaling: fit assumptions, budgets and serving cost

What question is being optimized?

A scaling law is an empirical relationship fitted under a model family, data distribution, tokenizer, training schedule and compute convention. Chinchilla asks how to choose model size N and processed training tokens D to minimize final loss at a fixed training compute budget. Serving-aware optimization asks a different question: how cheaply can a quality target be trained and deployed over an expected workload?

Bring loss curves, optimization and a versioned corpus/tokenizer from the preceding lessons. We will derive the dense-transformer FLOP approximation, distinguish Hoffmann's three fitting approaches, implement two explicitly different calculators, verify numerical budgets, and reason about data limits and serving demand. No GPU training, rental-price measurement or benchmark reproduction is performed.

1 · Deriving the six and its limits

For a dense matrix W with N matrix entries, one forward matrix-vector operation costs about2N floating-point operations when a multiply and an add each count as one. Backprop computes a weight-gradient product and an input-gradient product, each of comparable leading cost. Over D tokens, the common approximation is:

This approximates dominant dense parameter-matrix work, not every operation a training system performs. State the parameter convention: embedding lookups, output projection, tied weights, biases and inactive MoE experts do not all have identical per-token use. A total parameter count for a sparse MoE is not automatically the N in this dense formula.

Attention adds pairwise work beyond parameter projections. For a layer, both QKᵀ and PV together cost about4*B*T*T*d_model forward FLOPs for a full dense square. Training derivatives add further comparable work; causal kernels can avoid masked pairs. Long context can make this term important. “d_model is much larger than context in any modern LM” is false, so there is no universal less-than10% attention overhead or fixed8k cutoff. Activation recomputation, optimizer steps, communication, padding and hardware inefficiency also change actual work/time.

The approximate GPT-3 calculation6*175e9*300e9=3.15e23 is close to a reported training-FLOP estimate, but agreement of two approximate accounting methods does not prove0.3% accuracy for every real transformer run.

2 · Kaplan, Hoffmann and three actual methods

Kaplan et al.(2020) studied empirical power laws and a compute-efficient regime favoring relatively fast growth of parameters; the commonly cited allocation exponents are about.73 for N and.27 for D. A tenfold compute increase implies factors10^.73 and10^.27, approximately5.37 and1.86. That is a fitted allocation, not a directive from which every subsequent model's recipe can be inferred.

Hoffmann et al.(2022) reconsidered parameter/token allocation with hundreds of runs and training horizons. The paper's methods are:

1. Approach1: minimum over training curves. Train model sizes with several token horizons/schedules, construct the lower envelope of loss at a given compute and fit the compute-optimal frontier.
2. Approach2: IsoFLOP profiles. At each chosen compute budget, compare model sizes trained for corresponding token counts and locate a profile's minimum.
3. Approach3: parametric loss surface. Fit a joint formE+A/N^α+B/D^β to final losses and analytically optimize it under the compute approximation.

The original lesson swapped method labels and called another trajectory fit “Approach3.” The estimates broadly support more balanced N/D scaling but are not three independent datasets agreeing on identical constants. Learning-rate decay tied to training horizon, optimization choices, parameter accounting and fit range affect the comparison. Do not explain the entire historical disagreement by an invented “data-plateaued ridge” story.

The illustrative Chinchilla model has70B parameters and1.4T training tokens, whereas Gopher has280B parameters and about300B tokens. Similar training-compute scale and better reported performance from the smaller, longer-trained model were the empirical demonstration. Saying Gopher “should just have seen5.6T tokens” changes its compute budget: multiplying D at fixed N does not hold C fixed.

3 · Optimize the fitted loss surface

Let N,D,A,B,α,β be positive. For a fixed compute C, substituteD=C/(6N):

Differentiating with respect to log N gives the balance condition:

At the optimum, parameter and data contributions have equal elasticity-weighted marginal effects; they need not be equal in magnitude. Solving:

The second derivative with respect to log N is positive, so this positive-domain surrogate has a unique minimum. Equal exponents give square-root scaling and a constant D/N depending on A/B; similar exponents yield approximately balanced growth but a drifting ratio. E is a fitted asymptote, not a measured universal language entropy.

The paper's Approach3 coefficients often quoted are E=1.69,A=406.4,B=410.7,α=.34,β=.28. They give N's compute exponent.28/.62≈.4516 and D's.34/.62≈.5484, not exactly .5/.5. Do not use this surface and a fixed20:1 ratio as if they were the same model.

Besiroglu et al.'s2024 Chinchilla Scaling: A replication attempt challenges the third fit, including reconstructed-data fit and implausibly narrow confidence intervals. It is an external replication study, not an official erratum proving α=β=.35 or a universal22:1 constant. Fit uncertainty and source version belong beside the calculator.

4 · Two calculators, each obeying its own budget

The first calculator is a 20:1 heuristic: set D=rN and solve C=6rN² exactly, givingsqrt(C/(6r)). The second optimizes the stated parametric surface. Both check positivity and reconstruct the budget. The old.6*(C/6e21)^.5*1e10 constant did not satisfy its own20:1 budget; a “fudge factor” cannot excuse that algebraic inconsistency.

import math
import numpy as np
from scipy.optimize import minimize_scalar
 
def fixed_ratio(compute, ratio=20., factor=6.):
    if not all(math.isfinite(x) and x>0 for x in (compute,ratio,factor)):
        raise ValueError('positive finite compute, ratio and factor required')
    n=math.sqrt(compute/(factor*ratio))
    return n,ratio*n
 
def surface_loss(n,d,A=406.4,B=410.7,alpha=.34,beta=.28,E=1.69):
    if n<=0 or d<=0:raise ValueError('positive parameters and tokens required')
    return E+A/n**alpha+B/d**beta
 
def surface_optimum(compute,A=406.4,B=410.7,alpha=.34,beta=.28):
    if not all(math.isfinite(x) and x>0 for x in (compute,A,B,alpha,beta)):
        raise ValueError('positive finite fit inputs required')
    logn=(math.log(alpha*A/(beta*B))+beta*math.log(compute/6))/(alpha+beta)
    n=math.exp(logn);d=compute/(6*n)
    return n,d
 
def gpu_hours_to_flops(hours, peak_flops_per_second, utilization):
    if (not all(math.isfinite(x) for x in (hours, peak_flops_per_second, utilization))
            or hours<0 or peak_flops_per_second<=0 or not 0<utilization<=1):
        raise ValueError('invalid hours/throughput/utilization')
    return hours*3600*peak_flops_per_second*utilization
 
for c in (1e19,1e20,1e21,6e20):
    n,d=fixed_ratio(c)
    assert math.isclose(6*n*d,c,rel_tol=1e-14)
    a,b=surface_optimum(c)
    numerical=minimize_scalar(lambda logn:surface_loss(math.exp(logn),c/(6*math.exp(logn))),
        bounds=(math.log(a)-5,math.log(a)+5),method='bounded')
    assert abs(numerical.x-math.log(a))<1e-4
    assert math.isclose(.34*406.4/a**.34,.28*410.7/b**.28,rel_tol=1e-12)
    assert surface_loss(a,c/(6*a)) < surface_loss(a*2,c/(12*a))
    print('C',c,'20:1 N,D',n,d,'surface N,D',a,b)
C=gpu_hours_to_flops(100,312e12,.4)
n,d=fixed_ratio(C)
assert C==4.4928e19
print('100 assumed accelerator-hours',C,n,d)
old_n=.6*(C/6e21)**.5*1e10
assert not math.isclose(6*old_n*(20*old_n),C,rel_tol=.1)
print('old formula budget fraction',6*old_n*(20*old_n)/C)

Under the assumed312TFLOP/s peak and40% sustained fraction,100 accelerator-hours provide4.4928e19 FLOPs; the20:1 result is about611.9M parameters and12.238B tokens. This is neither the original520M answer nor a measured machine capacity. The tokens needed to grow from1B to12.238B are not “twice the corpus,” and WARC download counts cannot be inferred without a measured yield.

5 · IsoFLOP curves and a reproducible plot

A fixed-budget line is a hyperbola in ordinary N,D coordinates and a straight line in log coordinates. On it, too-small N leaves model capacity expensive to improve via more D; too-large N starves the run of processed tokens. The parametric surrogate gives a bowl in log N. How flat it is depends on coefficients, range and metric; being off by “a factor of a few” has no universal negligible-quality guarantee.

from pathlib import Path
 
def isoflop_svg(computes=(1e19,1e20,1e21)):
    curves=[]; all_loss=[]
    grid=np.geomspace(1e7,1e11,90)
    for c in computes:
        ys=[surface_loss(float(n),c/(6*n)) for n in grid]
        curves.append(ys);all_loss.extend(ys)
    low,high=min(all_loss),max(all_loss)
    svg=['<svg xmlns="http://www.w3.org/2000/svg" width="720" height="360" viewBox="0 0 720 360">',
         '<rect width="720" height="360" fill="white"/>',
         '<text x="30" y="24">Illustrative fitted IsoFLOP loss; not trained models</text>',
         '<path d="M60 40V300H690" fill="none" stroke="black"/>']
    for i,(c,ys) in enumerate(zip(computes,curves)):
        color=['#165ca1','#a13a16','#26833b'][i]
        pts=' '.join(f'{60+630*j/(len(grid)-1):.2f},{300-250*(y-low)/(high-low):.2f}' for j,y in enumerate(ys))
        svg.append(f'<polyline points="{pts}" fill="none" stroke="{color}" stroke-width="2"/>')
        svg.append(f'<text x="{90+190*i}" y="345" fill="{color}">C={c:.0e}</text>')
    svg.extend(['<text x="230" y="321">log-spaced parameters: 10 million to 100 billion</text>',
                '<text x="8" y="45">loss</text>','</svg>'])
    return '\n'.join(svg)
Path('batch7-dl-pretraining-isoflop.svg').write_text(isoflop_svg())

This plots a mathematical surrogate from stated coefficients. It is not experimental evidence for a new scaling law. A real fitting study needs multiple model sizes/horizons, fair learning-rate schedules, token definitions, repeated seeds or uncertainty analysis, held-out predictions and residual/error inspection. Never fit and validate only on the same endpoints.

6 · Wall time, memory and the100M worked example

For N=100M,D=2B, the dense approximation is1.2e18 FLOPs. Dividing by assumed sustained124.8TFLOP/s gives9615.38 seconds, about2.671 hours. This arithmetic is correct conditional on sustained throughput; it does not establish a3–6-dollar pretraining job. Small models may achieve very different utilization, and data loading, validation, checkpointing, startup and failures take time.

Likewise312TFLOP/s times86400 seconds is2.69568e19 FLOPs per ideal device-day.3.14e23 FLOPs require about11,648 ideal device-days or31.9 ideal device-years, and about29,121 device-days at40%, not355 or880 days. “A100-years” and “A100-days” cannot be used interchangeably.

peak=312e12;sustained=peak*.4
small_flops=6*100e6*2e9
print('100M/2B idealized hours',small_flops/sustained/3600)
print('3.14e23 ideal device days',3.14e23/(peak*86400))
assert math.isclose(small_flops/sustained/3600,2.670940170940171)
# Three proxy runs: do not divide the total and then multiply by three again.
proxy_total=3*6*20e6*500e6
assert proxy_total==1.8e17
print('three proxy runs assumed minutes',proxy_total/sustained/60)

Memory has independent terms: parameters, gradients, optimizer states, activations, attention/communication workspace and allocator reservations. FLOPs alone do not show that a chosen model fits. For a future paid run, measure a short representative configuration, include overheads and use a dated provider/region/storage/transfer quote. No new resource is requested or created here.

7 · Serving-aware allocation and a real crossover equation

A simplified lifetime-compute model is6ND+2NS, where S is total served tokens and2N is a leading dense forward cost. Actual prefill/decode costs depend on prompt lengths, cache, batching, context and hardware; “per token” must say which tokens. Optimize at a quality constraint before comparing costs.

Suppose a large option costs training C-big and per-served-token I-big, while a smaller longer-trained option costs C-small and I-small. If C-small>C-big and I-big>I-small, the break-even served count is:

For the explicit options below, training the 10B/200B option costs 1.2e22 FLOPs and the 4B/1T option costs 2.4e22. The smaller model costs an extra 1.2e22 to train but saves 12e9 FLOPs per served token under this approximation, giving a crossover of 1e12 served tokens. Below that count the larger option is cheaper in this model; above it the smaller option is cheaper. Neither pair has established equal quality, so this is conditional accounting, not a deployment recommendation.

If two options have identical training FLOPs, they tie at S=0 and the smaller one has lower simplified serving FLOPs for any positive S. That does not make their quality equal. The original code's claim that a compute-optimal model “wins at S=0 by construction” contradicted its equal-cost setup.

def lifetime(n,d,served):return 6*n*d+2*n*served
large=(10e9,200e9);small=(4e9,1000e9)
c_big,c_small=lifetime(*large,0),lifetime(*small,0)
crossover=(c_small-c_big)/(2*large[0]-2*small[0])
assert crossover>0
assert math.isclose(lifetime(*large,crossover),lifetime(*small,crossover),rel_tol=1e-14)
assert lifetime(*small,crossover*2)<lifetime(*large,crossover*2)
print('hypothetical equal-quality options crossover',crossover,
      'quality equivalence must be established separately')

Llama2's roughly2T training tokens and Llama3's heavily trained small models motivate this distinction. The Llama3 report explicitly says its flagship is near its compute-optimal allocation while smaller models are trained longer for the same inference budget. It does not supply the invented “10 billion tokens →10x overtraining pays back” universal rule. Phi3's small-model data recipe and capability goals differ again. A D/N bar chart describes processing ratios; it does not rank quality or establish wasted compute.

A source-backed three-model exercise can use the openly reported nominal sizes below. LLaMA1's original small-model table uses6.7B rather than rounding it to7B; Phi3-mini reports3.8B and3.3T. These are descriptive ratios, not a capability ranking.

published_examples={'Chinchilla':(70e9,1.4e12),
                    'LLaMA1-small':(6.7e9,1e12),
                    'Phi3-mini':(3.8e9,3.3e12)}
for name,(params,tokens) in published_examples.items():
    print('attributed nominal pair',name,'D/N',tokens/params,'6ND estimate',6*params*tokens)
assert math.isclose(published_examples['LLaMA1-small'][1]/published_examples['LLaMA1-small'][0],149.2537313432836)
assert math.isclose(published_examples['Phi3-mini'][1]/published_examples['Phi3-mini'][0],868.421052631579)
bars=['<svg xmlns="http://www.w3.org/2000/svg" width="700" height="230"><rect width="700" height="230" fill="white"/>',
      '<text x="20" y="25">Reported token/parameter ratios, not quality scores</text>']
for i,(name,(params,tokens)) in enumerate(published_examples.items()):
    ratio=tokens/params;y=55+i*50
    bars.append(f'<text x="15" y="{y+16}">{name}</text><rect x="170" y="{y}" width="{ratio/2:.2f}" height="23" fill="#285a91"/><text x="{180+ratio/2:.2f}" y="{y+16}">{ratio:.2f}</text>')
bars.append('</svg>')
Path('batch7-dl-pretraining-model-ratios.svg').write_text('\n'.join(bars))

The corresponding6ND estimates are5.88e23,4.02e22 and7.524e22 FLOPs. Chinchilla's reported accounting budget is about5.76e23, so the leading approximation differs by about2.1%; do not overwrite the reported budget with the surrogate. Attributed source pairs and the separate estimate remain distinguishable.

Distillation is another option: teacher distributions or generated targets can help a student, but teacher inference/training has costs and errors. A student is not mathematically forbidden to outperform its teacher on a particular task. Count which costs are sunk and compare under the target quality/evaluation contract.

8 · Data constraints and limits of extrapolation

Processed D includes repeated tokens; unique corpus size is a different variable. Repeated data can continue to help but returns diminish in a setting-dependent way. Scaling Data-Constrained Language Models studies that regime; it does not establish a hard four-epoch cliff, a universally half-valuable second epoch or zero benefit by epoch4. Track train-versus-held-out divergence and repeat counts by source.

Data quality/filtering changes the distribution, tokenization changes the unit and schedule/architecture changes optimization. None guarantees that a fitted β increases. Very small models and very long contexts can lie outside the measured range, but there is no general10M boundary at which all scaling laws stop working. Report extrapolation distances and prediction intervals rather than announcing a physical law.

Beyond Chinchilla-Optimal explicitly adds inference demand to the cost objective. Language Models Scale Reliably with Over-Training and on Downstream Tasks studies extrapolation across overtraining and downstream metrics; a token/parameter ratio such as900 is not “900 times beyond a20:1 ratio.” Both are study-bounded results, not evidence that every deployed model's serving cost dominates within months.

Failure cases: a low-token20B plan can be data-limited, but a loss plateau may also reflect optimization bugs; check both before buying more data. Repeating a10B corpus eight times produces80B processed tokens, not80B unique tokens; monitor clean validation rather than infer a mandatory stop at epoch3. The original workplace anecdotes and exact HumanEval gains are not presented as witnessed events.

9 · Recall and worked answers

1. Where does6ND come from? Roughly2ND forward plus two comparable gradient products. It omits architecture/context/system details and requires a declared N convention.
2. Why did Kaplan/Chinchilla differ? Different empirical protocols, horizons/schedules and fits changed the estimated allocation. Hoffmann's three methods support more balanced growth; an external replication challenges the third fit's precision. No single invented bug story resolves every difference.
3. What does20 tokens/parameter mean? A useful approximate training-compute allocation heuristic associated with Chinchilla, not a universal optimum, serving rule or fixed-quality prescription.
4. C=1e21 under20:1? N=sqrt(C/120)=2.88675B,D=57.735B. The old400M/8B pair spends only1.92e19, not1e21.
5. C=6e20 under20:1? About2.23607B and44.7214B. Substitute back into6ND; this quiz arithmetic was sound even when other tables were not.
6. Why train a smaller model longer? It may reach the required quality with lower serving cost. Establish quality equivalence and estimate serving demand; lower N alone does not prove the full trade worthwhile.
7. Is training beyond20 wasted? Not necessarily. Fixed-N training can improve with more D; fixed-training-compute optimization and fixed-quality lifetime-cost optimization are different problems.
8. Stretch answer: choose three openly documented models, record exact source/version,N definition,D/tokenizer and whether parameters are active or total; calculate D/N and approximate6ND only where applicable. Exclude rumored private-model sizes. Plot ratios with units and avoid interpreting them as quality rankings.

The durable picture is a budget line and a loss contour, not a sacred ratio. Tomorrow derive the marginal-balance equation; after a week verify both calculators by substituting their outputs back into the budget.

Sources and optional navigation

Kaplan2020, Hoffmann/Chinchilla, replication attempt, data-constrained scaling, Sardana et al.2024, overtraining/downstream scaling, Llama3 report, Phi3 report. Exact primary passages and independent arithmetic are separate in the review evidence.

Optional inherited explainers: Chinchilla video, Karpathy overview, Chinchilla implications essay. Course map · Previous DL046 · Next DL048.