Scaling from 100M to 1B — A Mental Model

Scaling is several budgets, not a bigger model name

A small reproducible training run is a useful baseline, but completing one does not establish the cost or quality of a 7B run. Compute, persistent state, activations, communication, data supply and recovery do not scale in the same way. More devices can reduce elapsed time, distribute state, or add coordination overhead; know which problem you are solving.

Prerequisites are the dense FLOP estimate, distributed state accounting, resource lifecycle, and a training/evaluation/checkpoint/debugging loop. If you have not run a pilot, the worksheet below is an assumption-driven plan, not measured evidence. You will calculate 100M through 400B examples, derive a wall-clock range, rank infrastructure risks, compare training-optimal with inference-aware design, and interpret named-model receipts without fabricating prices. No GPU, cluster or pretraining run was executed here.

The scaling table: derive it instead of memorizing prices

For a dense transformer in a regime where parameter matmuls dominate, a common approximation is

N is parameters and D is processed training tokens, not unique documents or sequences. A matmul uses about two FLOPs per multiply-accumulate in the forward pass; the corresponding backward work motivates the approximate factor six. Embeddings, attention at long sequence lengths, recomputation, padding and architecture change the count. A FLOP formula must state what it counts before its efficiency can be compared with a hardware peak.

Hoffmann et al., v1, 2022 investigated compute-optimal allocations by varying model size and tokens. Their conclusion is approximately equal scaling of model size and training tokens in the studied setting. D=20N is a useful illustrative allocation, not a universal law or a definition of a finished model. Under that assumption C=120N²: increasing N tenfold costs one hundred times the arithmetic, not ten times.

The following table assumes sustained 124.8 TFLOP/s per device, obtained by assuming 40% of a 312 TFLOP/s dense BF16 peak for a specified A100 configuration. These are equivalent A100-days under an assumed utilization, not measured rental durations. All state memory uses the explicit 16-byte representation, excluding activations and peaks.

Do not read the last column as permission to run a memory-impossible single-device job. It is a compute unit. Similarly, a 1B row is about eleven equivalent days, not four days as in the old inconsistent table. The exact calculator below is authoritative for rounding.

From FLOPs to wall-clock and uncertainty

If each of G devices has peak arithmetic F, measured model-FLOPs utilization u, and the same counting convention, then

If u is measured for the whole distributed configuration, do not multiply by a second scaling-efficiency penalty that already contributed to u. Alternatively, use a single-device sustained rate and a separately measured scaling efficiency e. Add initialization, evaluation, saving, waiting and restarts outside the training-only estimate. Model-FLOPs utilization (MFU) and hardware-FLOPs utilization differ when recomputation or other extra work is counted; high device occupancy is neither.

At 7B, 140B tokens and eight devices sustaining 124.8 TFLOP/s each, idealized training takes about 68.16 days, not fifteen. At u=.25 versus .45 and F=312 TFLOP/s, the range is about 109.06 to 60.59 days before other overhead. A claim of “within 30%” is unjustified without an appropriate pilot and error analysis. If observed utilization is .22 rather than assumed .40, elapsed time multiplies by .40/.22=1.818, even before retries.

# CPU scale calculator. Hardware and money inputs are explicit assumptions.
import math
 
def plan(params, tokens_per_param=20., gpus=1, peak_flops=312e12,
         mfu=.4, rate_per_gpu_hour=0., mode="fsdp", activation_gb=0.,
         gather_gb=0., other_gb=0., overhead_hours=0.):
    if isinstance(gpus, bool) or not isinstance(gpus, int) or gpus < 1:
        raise ValueError("device count must be a positive integer")
    values = (params, tokens_per_param, peak_flops, mfu, rate_per_gpu_hour,
              activation_gb, gather_gb, other_gb, overhead_hours)
    if any(isinstance(x, bool) or not isinstance(x, (int, float)) or not math.isfinite(x)
           for x in values):
        raise ValueError("assumptions must be finite real numbers")
    if mode not in {"ddp", "zero1", "zero2", "fsdp"}:
        raise ValueError("unknown state-sharding mode")
    if params <= 0 or tokens_per_param <= 0 or not 0 < mfu <= 1:
        raise ValueError("invalid model, data, device count or utilization")
    if peak_flops <= 0 or min(rate_per_gpu_hour, activation_gb, gather_gb,
                             other_gb, overhead_hours) < 0:
        raise ValueError("invalid throughput or resource term")
    factors = {"ddp": 16., "zero1": 4+12/gpus,
               "zero2": 2+14/gpus, "fsdp": 16/gpus}
    tokens = params * tokens_per_param
    compute = 6 * params * tokens
    hours = compute / (gpus * peak_flops * mfu) / 3600
    return {"tokens": tokens, "compute": compute, "training_hours": hours,
            "wall_hours_with_assumed_overhead": hours + overhead_hours,
            "gpu_hours": gpus * hours,
            "per_gpu_gb": params * factors[mode] / 1e9 + activation_gb + gather_gb + other_gb,
            "assumed_compute_cost": gpus * (hours + overhead_hours) * rate_per_gpu_hour}
 
assert plan(1e8)["compute"] == 1.2e18
assert plan(7e9, gpus=8)["per_gpu_gb"] == 14.
assert abs(plan(1e9)["training_hours"] / 24 - 11.1289) < .0001
# A hypothetical H100 case can pass its chosen variant's peak explicitly.
# No device performance or rental price is looked up by this function.

The calculator rejects NaN/infinite inputs and noninteger or boolean device counts: an unknown measurement is not a valid zero-cost or fractional-device plan. Its arithmetic assumes representable, physically meaningful input magnitudes; it is not a capacity reservation or a full cost model.

Use actual tokens/sec as the strongest direct pilot quantity when the full target configuration is available. For global token rate v, model-MFU estimate is 6Nv/(GF) under the same approximation. Comparing MFU across architectures can still mislead if FLOP accounting, precision, sparsity assumptions or peaks differ. Comparing tokens/sec without model/context/batch is also incomplete.

Memory scaling: state is sharded, activations are not automatically divided

DDP keeps the persistent state on every rank. Fully sharded training ideally stores 16N/G state bytes per rank in our illustrative representation. Add local activations, unsharded units, buffers, workspace and reserve after dividing the sharded term. For 7B on eight devices with 10 GB of local activations and 3 GB of other/gather peaks, the estimate is 112/8+10+3=27 GB, not (112+10+3)/8.

For 70B on eight 80 GB devices, persistent state alone is 140 GB per rank. Activation checkpointing cannot make that fit. For 400B on 32 devices, it is 200 GB before peaks; the old “32+ H100s suffice” statement was not supported. Even the state-only 80-device limit of 80 GB leaves zero room for useful work.

A 24 GB device's state-only ceiling is 1.5B at 16 bytes/parameter, not a practical capacity guarantee. At 40 GB it is 2.5B before overhead. Lower microbatch/context and activation checkpointing relieve activation pressure; ZeRO stages and offload alter state; tensor/pipeline/context parallelism change the work and storage layout. A different dtype/optimizer or adapter-only training creates a different worksheet. There is no immutable “7B personal ceiling” or “70B team ceiling”: budgets, data, goals and operational constraints define it.

Communication scaling and the topology decision

At fixed local tokens per step, dense compute is proportional to N and a gradient payload is also proportional to N. Their ratio is therefore roughly proportional to local tokens, not automatically worse for every larger N. The difficulty appears when capacity forces a different sharding strategy, or strong scaling reduces local tokens at fixed global batch, leaving less compute to hide communications.

For an eight-rank ring and a BF16 7B payload, two parameter all-gathers and a same-dtype reduce-scatter move about 36.75 GB sent per rank under the explicit schedule from the distributed lesson. If effective bandwidth is hypothetically 50 GB/s, serialized transfer time is .735 seconds plus latency. Do not substitute a bidirectional NVLink marketing total into this unidirectional calculation. Inter-node NIC sharing, topology and simultaneous collectives matter.

A candidate 2D layout uses tensor parallelism on fast intra-node links and data-parallel sharding across nodes, but this is not universally optimal. TP adds intra-layer collectives; PP introduces pipeline bubbles and needs a microbatch schedule; MoE expert parallelism routes tokens via all-to-all. Increasing microbatch may improve compute/communication balance until memory binds. Increasing sequence length also increases attention work and memory, so it is not a free utilization fix. Profile before declaring that NVLink “stops being enough at 70B.”

Seven infrastructure knobs: rank by symptoms

For an early 100M pilot, rank correctness/data and restart parity ahead of aggregate throughput. For a state-overflowing model, memory feasibility is first; after it fits, communication or data can dominate. At many nodes, failure frequency and recovery can dominate useful throughput. Several limits can bind together; it is false that exactly one budget always binds or that everything except compute is trivial at small scale.

A practical six-stage ladder is: (1) small baseline and held-out evaluation; (2) deliberately interrupted/resumed baseline; (3) larger model with measured state inventory; (4) two-device gradient/throughput parity; (5) adaptation/serving evaluation against a strong permitted baseline; (6) only then a costed larger pretrain. Each stage has a pass criterion, not a calendar or price guarantee. A six-month calendar can allocate time to these stages, but no spending ladder promises an educational credential or a particular model capability.

Named-model receipts: keep denominators and boundaries straight

Llama 3 8B and overtraining

The Llama 3 report v3 describes a roughly 15T-token corpus and explicitly says the small models are trained longer than compute-optimal to improve performance at a fixed inference budget. Its table distinguishes April Llama 3 from July Llama 3.1; the paper's results are for 3.1. Do not silently mix their GPU-hour model cards.

For a calculation with N=8B and D=15T, 6ND=7.2e23 FLOPs. The 20N reference is 160B tokens, making 15T 93.75 times as many tokens, not ten or twelve. A model card's total GPU-hours do not reveal its elapsed time unless the actual concurrently allocated devices over time are known. Dividing them by a headline cluster size used elsewhere in the program invents a wall clock. Neither a paper nor hardware peak yields Meta's internal electricity/amortization cost. The April Llama 3 model card, snapshotted on 2026-10-08, reports 1.3M GPU-hours for 8B and 6.4M for 70B, identifying pretraining time on H100-80GB hardware. These are historical model-card figures, distinct from a later 3.1 paper; at an explicitly hypothetical 2 units/device-hour they would multiply to 2.6M and 12.8M compute-only units, not audited bills or elapsed-time measurements.

DeepSeek-V3: a disclosed estimate, not a market quote

DeepSeek-V3 v2 reports 671B total parameters, 37B activated per token and 14.8T pretraining tokens. Its Table 1 separates 2,664K H800 GPU-hours pretraining, 119K context extension and 5K post-training: total 2,788K. At the paper's assumed 2 USD/H800-hour, those rows become 5.328M, .238M and .010M USD, totaling 5.576M. The report explicitly excludes prior research/ablation costs. This is a source-attributed training-cost estimate, not a bill independently reproduced here or all-in organizational cost.

The report discusses FP8 mixed precision, MLA, expert parallelism and DualPipe. Those techniques change different terms: activated compute, cache/state layout, numeric formats, routing and overlap. A comparison with Llama 3 cannot assign a causal “tenfold cheaper at equal quality” result without matched tasks, training stages, hardware and total experimental budgets. H800 is not safely modeled as a universal percentage haircut to H100 arithmetic. No current vendor price enters this lesson.

Training-optimal versus deployment-optimal

Sardana and Frankle v1 study how inference demand changes the optimal train-and-deploy allocation. In a simple dense arithmetic model, total work is 6ND + 2NI, where I is lifetime inference tokens; real serving adds context, cache, batching and utilization. A smaller model trained longer may pay extra training work to save on every later request. That does not prescribe 100–300 tokens/parameter for every application, nor prove that every user-facing system is inference-dominated.

Data-constrained scaling v1 studies repeated data and diminishing returns; it is a different paper from inference-adjusted scaling. Do not treat repeated tokens as unlimited fresh information. Porian et al. v1 explain Kaplan/Hoffmann discrepancies through last-layer compute, warmup and scale-dependent optimizer tuning and obtain agreement with Chinchilla after corrections; they do not simply establish a new universal 18–25 constant.

For MoE, active parameters help estimate part of per-token arithmetic while all stored experts contribute to memory/checkpoints. Routing, load imbalance, shared layers and attention prevent substituting active count blindly into every dense formula. There is no established universal “quality equals sqrt(active×total)” law. Measure a task/data/architecture-specific fit. Distillation introduces teacher-generation cost and a different data distribution; a smaller model can improve on selected tasks, but no generic 5–10-point gain is implied.

Precision, interconnect and the limits of a fit

Transformer Engine v2.0's FP8 primer explains E4M3 versus E5M2 and scaling. More low-precision arithmetic peak does not double end-to-end throughput: casts, sensitive operations, communication, data wait and kernels remain. BF16 and FP8 training require different numeric contracts. A hardware generation or network specification changes the worksheet, not an automatic model-size ceiling or a guaranteed 5–10× economic improvement every eighteen months.

Scaling fits predict a measured loss under an experimental regime. Downstream exact-match and thresholded scores may look discontinuous even when loss changes smoothly. Evaluate both continuous loss and task metrics with uncertainty; do not promise either inevitable emergence or universal smoothness. Tokenization changes loss units, and contamination, data quality or domain shifts can invalidate comparisons. A fit trained below a certain scale does not abruptly become worthless above it, but extrapolation uncertainty grows and should be checked with intermediate pilots.

Exercises, diagnostics and worked answers

1. Compute-optimal worksheet for 1B: under D=20N, D=20B and C=1.2e20. At the assumed 124.8 TFLOP/s, about 11.1289 equivalent device-days. Price is GPU-hours × explicit rate, plus separately identified costs.
2. 7B versus 3B: C scales as N² only while D=20N. The 3B job uses 1.08e21 FLOPs. A 7B job stopped at 40B tokens uses 1.68e21; that is a hypothetical token budget, not a factual account of Llama 1 training.
3. Which memory formula is wrong? (state+local_activation)/G for FSDP. The correct simplified formula is state/G+local_activation+peaks. Activation checkpointing cannot rescue a state-only overflow.
4. Plan a 100,000-unit budget over 30 days. Let q be a dated all-in device-hour quote. Available device-hours are at most (budget-reserve-storage-egress)/q. Derive feasible G and wall time, then state/model/data candidates and held-out evaluation. No budget guarantees “Llama-3-8B quality,” and an MoE choice needs a routing/serving capacity plan.
5. Diagnose an unfinished 7B run after repeated spot interruptions. Compare useful token progress, lost work, reacquisition and checkpoint write/restore costs. Do not relaunch indefinitely or blame the FLOP estimate alone; remaining expected completion cost may exceed the stop budget.
6. How to compare pilot predictions? Record predicted versus measured full-update time, validation cost and restart time. A consistent 1.818× discrepancy from .40 versus .22 utilization reprices the run; it does not prove model quality or data sufficiency.
7. What did Module 09 establish? Corpus construction and quality (046–049), architecture (050), parallelism/resources (051–052), training/eval/recovery (053–055), debugging/scaling (056–057) form a dependency chain. Reading it is not evidence that you have executed the chain. SFT next changes the objective and data contract rather than magically making every base model useful.

Retention: tomorrow rebuild the five-row table from formulas, including units, and explain why “personal ceiling” is a budget decision. In a week rank the seven knobs for a concrete measured failure and distinguish an inference-aware objective from data-constrained scaling. Bring back a pilot error range and a recovery criterion, not a memorized market price.

Previous: ← DL S056 · Next: DL S058 →

Original optional resource links

The original optional video/reading destinations are retained here for continuity, not as current API, price, benchmark or publication-date evidence. The versioned primary citations in the teaching above govern the corrected claims. These supplementary destinations were not all fetched or watched; a missing or changed resource does not replace the local derivations, procedures and answers.

• https://www.youtube.com/watch?v=zjkBMFhNj_g
• https://www.youtube.com/watch?v=nOxKexn3iBo
• https://arxiv.org/abs/2407.21783
• https://blog.eleuther.ai/transformer-math/
• https://arxiv.org/abs/2412.19437
• https://arxiv.org/abs/2305.16264
• https://arxiv.org/abs/2401.00448
• https://arxiv.org/abs/2406.19146
• https://arxiv.org/abs/2101.03961
• https://github.com/NVIDIA/TransformerEngine