DL S057 · Scaling from 100M to 1B — A Mental Model

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 57 / 80 · Module M09 · ~2 hours

The story we're starting with

Your 100M model cost $3 and 3 hours. You are now, technically, "someone who has pretrained a foundation model." The next question everyone asks is: what does it take to scale?

The honest answer has three axes: compute (mostly a function of N × D, from Chinchilla), memory (which architecture and infrastructure you can use, from the 16-bytes-per-param rule), and throughput (which depends on interconnect and MFU). Each axis scales differently as you go up model sizes, and understanding which becomes the bottleneck at what scale is the difference between "I could reproduce Llama 2" and "I would run out of memory on step 1."

This session builds a spreadsheet-in-your-head for the full range: 100M → 1B → 7B → 70B → 400B. At each scale we ask: what does the config look like? How much compute? How much memory per GPU? How many GPUs? How long? How much money? And what infrastructure do you actually need? By the end you'll close any tab about the "next big model release" and know exactly what it took.

We also close Module 09 — you've now covered the entire "train your own foundation model" pipeline end to end. Module 10 (fine-tuning + alignment) starts turning that base model into an instruction-follower.

Prerequisites

• S047 (Chinchilla) — the compute formula.
• S051 (DDP/FSDP) — the memory + comms formulas.
• S052 (renting GPU) — the dollar prices.
• S053, S054, S055, S056 — you've now trained, evaluated, checkpointed, and debugged a real model.

1 · The scaling table — the whole picture on one page

Read down the table. Every column changes by roughly 10× as N grows 10×, and each pair of columns changes at different rates — so the binding constraint shifts:

• 100M → 1B: compute becomes the pain. Wall clock jumps 30×.
• 1B → 7B: memory becomes the pain. Doesn't fit on one GPU.
• 7B → 70B: interconnect becomes the pain. Need multi-node with InfiniBand.
• 70B → 400B: operations become the pain. Node failures, cost management, human coordination.

2 · The N → wall clock formula, from scratch

Chinchilla: D = 20N and C = 6ND = 120 N².

At MFU=40%, an A100 sustains 312e12 × 0.4 = 1.25e14 FLOPs/s.

Wall clock on 1 A100:

seconds = 120 N² / 1.25e14
        = 9.6e-13 × N² seconds

Or in A100-days:

A100-days = 9.6e-13 × N² / 86400 = 1.1e-17 × N² days

Plug in:

Beyond ~3B, single-GPU training is impractical. Beyond ~30B, single-node (8-GPU) training is impractical.

Cost formula: $ ≈ A100-days × 24 × $1.10 (spot) ≈ 30 × A100-days.

So a 1B model costs 11 × 30 = $330 on spot. A 7B costs 540 × 30 ≈ $16000. That matches published estimates for training a Llama 2-class 7B from scratch on rented hardware (~$10k-20k depending on efficiency).

3 · Memory scaling — what fits where

Recall from S051: 16 bytes/param for bf16 Adam training steady state.

Add activations on top. Rough activation formula per layer (with checkpointing): ~2 × batch × seq × d_model bytes. Sum over layers, multiply by ~2 for backward.

Concrete numbers with batch=8, seq=2048:

7B doesn't fit on any single GPU. Even H100 80GB is ~120 GB short. You must shard.

Under FSDP on 8 GPUs:

• 7B: 122/8 ≈ 15 GB per GPU. Fits nicely on A100 40GB.
• 70B: 1150/8 ≈ 143 GB per GPU. Still doesn't fit on H100 80. Need 16+ GPUs, or add activation checkpointing (cut activations 4×).
• 400B: needs 32+ H100 80s with FSDP + activation checkpointing.

4 · Comms scaling — why NVLink stops being enough at 70B

Recall FSDP comms: ~6N bytes per step (all-gathers + reduce-scatters).

Step compute time: C_per_step / TFLOP/s = 6ND_micro_batch / (MFU × 312e12).

For a 7B model, 8× A100, batch=8, seq=4096:

• Step compute: 6 × 7e9 × 8 × 4096 / (0.4 × 312e12) ≈ 11 s
• Comms: 6 × 7e9 × 2 / 600e9 (NVLink3) ≈ 0.14 s
• Overlap ratio: 1.3% — comms is basically free.

For 70B on 8× A100 (assuming it fit):

• Step compute: 6 × 70e9 × 8 × 4096 / (0.4 × 312e12) ≈ 110 s
• Comms: 6 × 70e9 × 2 / 600e9 ≈ 1.4 s
• Overlap ratio: 1.3% — still fine.

But inter-node (across nodes) doesn't have NVLink. It has InfiniBand at 25-50 GB/s (200-400 Gbps). At 70B on 16× GPUs across 2 nodes:

• Inter-node comms: 6 × 70e9 × 2 / 25e9 = 33.6 s per step
• Step compute (halved by 2× GPUs): 55 s
• Comms is now 60% of compute — that's a real slowdown.

Fix: hybrid parallelism. Use tensor parallelism within a node (fast NVLink), FSDP across nodes (fewer comms). Megatron-LM's key insight; every serious multi-node run uses it.

For a 400B run on 8 nodes × 8 GPUs, you need TP + PP + FSDP simultaneously — three axes of parallelism. This is where hobby-scale ends and industry-scale begins.

5 · The seven infrastructure knobs, ranked by scale

Read this every time you're planning a new run. It tells you what to expect to spend engineering time on.

6 · Real numbers from Llama 3 8B

From the Llama 3 paper (§3):

• Params: 8B
• Tokens: 15T (over-trained ~10× past Chinchilla)
• Compute: ~7.4 × 10²³ FLOPs (12× more than Chinchilla-optimal for 8B)
• Wall clock: ~1.3M H100-hours on a 16000-H100 cluster
• Wall clock in days: ~3.4 days
• **Cost (if rented at $2.50/hr H100 spot):** ~$3.25M
• Cost (Meta internal): perhaps $500k in electricity + amortized hardware

That's your "what does frontier open-source look like" reference. 100× our 7B baseline in compute (from over-training + more careful data) and 3.4 days on 16000 H100s.

7 · A realistic 6-month "personal frontier" ladder

If you wanted to spend a year seriously moving up from 100M, here's a sane ramp:

Total budget: ~$17k over 6 months. Compare to a Master's degree in ML. Similar cost, ridiculously more capability.

Almost nobody does this — most stop at 1B or move to fine-tuning bigger open models — but the ladder is real. The gate isn't skill; it's dollars and time.

8 · Where the mental model breaks

Things this scaling model doesn't capture:

• Data quality wins. A well-curated 1T tokens might beat a poorly-curated 5T. This session assumes fixed quality.
• Architecture innovations. MoE, MLA, and various attention variants can shift the effective compute curve. Sometimes 30% improvements.
• Fine-tuning is much cheaper than pretraining. A 7B LoRA fine-tune is $10, not$10k. Most practical LLM work happens here, not at pretraining.
• Distillation. A 100M student trained on outputs of a 7B teacher can beat a 100M peer trained from scratch by 5-10 points. We cover it in S068.
• Emergent scaling. Some capabilities don't appear until N ≥ some threshold. You can't buy them incrementally at smaller scales.

9 · Closing Module 09

You have now covered:

• S046-S049: how to build a pretraining corpus.
• S050: how to design a modern LLM architecture.
• S051-S052: how to make it run on real hardware.
• S053-S055: how to train, eval, and checkpoint it.
• S056-S057: how to debug it and scale it.

You could — literally — kick off a personal 1B pretraining run tomorrow. You know what it costs, what to watch for, what will break, and how to recover.

That was the "make your own foundation model" module. The rest of the series (M10 onward) turns your model, or someone else's open weights, into something that does things: follows instructions, gets aligned to preferences, runs efficiently at inference, and eventually powers a real product.

10 · War stories

12 · The 2025 scaling landscape — what the frontier learned this year

Six things changed the scaling calculus in 2024–2025. If your mental model is stuck at Chinchilla-2022, update.

12.1 · Overtraining beats Chinchilla for inference-heavy models

Muennighoff et al. (2023, arxiv 2305.16264, 'Scaling Data-Constrained LMs') and follow-ups showed: Chinchilla-optimal minimizes training compute for a target loss, but not total (train + inference) compute over a model's lifetime. If you're going to serve N tokens of inference per parameter, the optimal training tokens-per-parameter grows from Chinchilla's ~20 toward 100–300 for popular inference-heavy models. Llama-3-8B was trained on 15T tokens (~1875 tokens/param, ~90× Chinchilla). Phi-3 mini went ~1000 tokens/param. Under-training is the new over-training.

For your 100M model: at 2B tokens (20 tokens/param), you're Chinchilla-optimal for training cost. Push to 10–20B tokens if you plan to use the model.

12.2 · MoE changed the param-to-compute-to-quality frontier

The old rule 'quality scales with N and D' silently assumed dense. For MoE:

• Active params (Nactive) determine per-token compute (≈ 6 · Nactive · D).
• Total params (Ntotal) determine model 'capacity' and disk footprint.
• Quality lies somewhere between the two, closer to sqrt(Nactive · Ntotal) per Fedus et al. Switch Transformer scaling laws and DeepSeek's own scaling analysis.

Rule of thumb 2025: a well-tuned MoE with 4–10× total:active ratio matches a dense model with active_params · 1.5–2 params at the same training compute. Mixtral 8×7B (12.9B active, 46.7B total) matches Llama-2-34B. DeepSeek-V3 (37B active, 671B total) matches Llama-3-405B dense-equivalent quality.

12.3 · FP8 and BF16 are your leverage

H100 bf16 = 989 TFLOPS. H100 fp8 = 1979 TFLOPS. If your training loop is compute-bound and your framework supports Transformer Engine, fp8 doubles your throughput (in practice ~1.5–1.8× after all-reduce overhead). This alone is why 2024–2025 pretrains got so much cheaper. Every serious training framework now supports fp8 (torchao, TransformerEngine, MosaicML Composer). Use it on H100+.

12.4 · Interconnect is the new ceiling

At 8B, one 8×H100 node with NVLink is enough. At 70B dense, you need 2D parallelism (TP inside node, FSDP across). At 405B, you need 3D (TP + PP + FSDP) plus InfiniBand NDR (or better, XDR). At MoE 600B+, you need expert-parallel with careful all-to-all optimization. Each order of magnitude in params adds a new dimension of parallelism.

See §4; add: 2025 Blackwell (B200) doubles NVLink5 to ~1.8 TB/s per GPU, pushing the single-node ceiling from ~70B to ~150B dense. If you rent 8×B200 (available on CoreWeave/Nebius mid-2025), you can train a Llama-3-70B-class model on a single node without TP. That's a big deal.

12.5 · The 'personal frontier' has moved up

What a solo dev/small team could pretrain in 2022 vs 2025:

Roughly a 5–10× improvement in 'what you can afford' every 18 months, driven by hardware (H100 → B200), formats (bf16 → fp8), and architecture (dense → MoE). Your 100M/2B run in 2025 is what a 10M/200M run was in 2020.

12.6 · Where scaling laws stop predicting — the frontier's dirty secret

Hoffmann's Chinchilla laws were fit on <100B-param dense models. Extrapolating to 1T dense or 10T MoE, the laws stop being predictive. Frontier labs have their own internal scaling laws refit yearly on their own architectures + data recipes; those laws are proprietary. What's public:

• DeepMind's revised Chinchilla (Porian et al. 2024, arxiv 2406.19146) reanalyzed the original data and found the exponents differ from Hoffmann's estimates. The 'optimal tokens/param = 20' is closer to 18–25 depending on how you fit.
• Sardana & Frankle (2024, arxiv 2401.00448) 'Beyond Chinchilla-Optimal': derives the inference-adjusted optimum. Shows for models with >100M inference tokens per param served, optimal training tokens is 100–300× params, not 20×.
• Kaplan-style vs Chinchilla-style laws: still an active fight. See Muennighoff for the definitive 2024 synthesis.

Practical takeaway: for your 100M–1B "personal frontier", the classical formulas are close enough. Don't extrapolate them to 100B+.

Further reading (12.x)

• DeepSeek-V3 Tech Report — arxiv:2412.19437
• Llama-3 herd of models paper — arxiv:2407.21783
• Muennighoff data-constrained scaling — arxiv:2305.16264
• Sardana & Frankle inference-adjusted scaling — arxiv:2401.00448
• Porian et al. Chinchilla replication — arxiv:2406.19146
• Switch Transformer scaling (Fedus) — arxiv:2101.03961
• DualPipe (DeepSeek-V3 pipeline parallel) — tech report §2.3
• Transformer Engine (fp8) — github.com/NVIDIA/TransformerEngine

13 · 🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: re-read §1 (the scaling table) and §12 (2025 landscape) in 24 hours. Revisit the full session on day 7. This session and §1 combined ARE your working mental model of pretraining scale.

Next session (S058, Module M10): Supervised Fine-Tuning — how to turn your base model (gibberish completer) into something that follows instructions.

Sticky note (keep on your desk): "FLOPs = 6ND. Chinchilla D=20N (or 100–300N if inference-heavy). MoE quality ≈ sqrt(Nactive · Ntotal). +1 dimension of parallelism per 10× in N."

Legacy recall drills (kept for spaced-review continuity)

Stretch

Build a scaling calculator. Function plan(N, tokens_per_param=20, gpus=1, gpu="A100_40", mfu=0.4) returns \{compute, wall_clock, per_gpu_mem, cost\}. Test on the 5 sizes in §1's table and confirm the numbers match. Extend to handle gpu="H100" and mfu=0.5 for a realistic H100 sanity check. Post the code as a gist.

In your own words

"The four scale-dependent bottlenecks are ______, ______, ______, ______ — and each becomes binding at roughly the ______, ______, ______, ______ scale."

Spaced-review pointer

• S047 (Chinchilla) — you're applying it at every row of the table.
• S051 (DDP/FSDP) — memory formulas.
• S053 (real training) — the 100M baseline is your only data point that isn't extrapolation.

Next-session teaser

Module 10 opens with Supervised Fine-Tuning (SFT) — how to take your (or Llama's) base model and teach it to follow instructions. Suddenly your gibberish-completer becomes something you can chat with. Session 058 turns "language model" into "assistant."

Bring back tomorrow

• Table: the 5-row scaling table (100M/1B/7B/70B/400B) — memorize the compute + memory + cost row.
• Rule: 24 GB VRAM = ~1.5B params ceiling for single-GPU training.
• Question: at your target scale, which of the seven infrastructure knobs is the binding constraint?

Previous: ← DL S056 · Next: DL S058 →