Training a 100M model: complete local recipe and budget

From an architecture to an auditable run

Goal: assemble the decoder, local token shards, optimizer, schedule, validation, sampling and restart state into a complete training experiment. The 100M architecture has 100092672 parameters, not an assumed round number. A two-billion-token run is a proposed budget, not an executed experiment or a promise of coherent assistant behavior.

The original launch story implied 24,000 tokens/s, two billion tokens and 2.7 hours simultaneously. Those numbers cannot all be true: 2e9/24000/3600 = 23.1481 hours before overhead. At an illustrative, not quoted, USD1.10/hour that is USD25.46, not USD3. Fixing the arithmetic does not validate that throughput, price, hardware availability or final loss. Measure sustained throughput on the intended stack and obtain a dated quote before authorizing a real run.

A launch checklist is more useful than a rocket metaphor that discourages intervention. Freeze a run identity/config/data manifest; inspect telemetry; stop at a finite budget when invariants fail. A documented branch from a checkpoint is valid experimentation, not necessarily an invalid run. Quietly changing a schedule while preserving the old experiment label is the problem.

Token, update, compute and money clocks

For one rank with microbatch 32, eight accumulation batches and sequence length 1024, one complete update supervises 262144 tokens. Two billion tokens require 7629 complete updates plus a final 103424-token update: 7630 updates in total. Flooring to 7629 undertrains; taking 7630 full updates processes 2000158720 tokens. State which policy you implement. Our loop masks the unused tail and divides by the actual supervised count.

For a dense linear layer Y=XW, forward costs about 2P FLOPs/token; the two backward matrix multiplies cost about 4P. Hence the common approximation

Embeddings, attention score/value products, output projections, sparse activation, optimizer work and recomputation complicate the approximation. Not every parameter is used in a matrix multiply once per token. Whole-layer activation recomputation can add roughly another forward, yielding an 8ND-style model, but overhead is not a universal 33% wall-time penalty. Decode can be bandwidth-bound and sequential: forward-only FLOPs do not imply inference tokens/s are three times training tokens/s.

If D scales as 20N, compute scales as 120N². Moving from 100M/2B to 300M/6B multiplies approximate compute by nine; 1B/20B multiplies it by 100. At the same 262144-token update budget, ceil counts are 22889 and 76294. Chinchilla's fitted compute-optimal allocation is regime-dependent; 20 tokens/parameter is not a ceiling, a quality floor or a theorem for every domain. Repeated data has different value from fresh data. Serving demand can favor a smaller model trained longer, but the improvement must be measured.

Configuration and optimizer invariants

The starting proposal is width768, depth12, 12 query/4 KV heads, SwiGLU2048, tied32K embedding, context1024, peak LR6e-4, floor6e-5, warmup500 updates, AdamW betas(0.9,0.95), decay0.1 and clip1.0. These are initial choices, not a universal LR-size equation. The original alleged Chinchilla formula also did not produce its stated number: 1e-3*sqrt(1e6/1e8)=1e-4.

Our update index is zero-based: the first used LR is peak/warmup, update warmup-1 reaches peak, and the final used update reaches the floor. A run too short to include decay is rejected rather than dividing by zero. Warmup can help early optimization, but Adam bias correction already removes the simple zero-initialization bias: at the first scalar step, corrected m=g and v=g². A noisy early second-moment estimate and network-scale dynamics are subtler than “Adam always divides by a meaningless zero.” A plot of raw exp_avg_sq alone does not identify an optimal warmup length.

Dimension-based decay is a reproducible policy, not a law that norm decay always destroys generalization. On this architecture all 2D tensors, including the tied embedding once, enter the decayed group; norm scales do not. Expect 100073472 decayed and19200 nondecayed parameters, not85M/200K. Assert disjoint membership, full coverage and unique identities. Fused AdamW or torch.compile can help supported workloads; neither is a free fixed-percentage speedup.

Local data contract

Use existing little-endian uint16 .bin shards only when the entire tokenizer ID range fits 0..65535. Each shard must contain at least T+1 tokens. The number of legal windows is length-T, with start indices through length-T-1 inclusive. Sampling shards uniformly biases short shards; we sample uniformly over all legal starts by weighting each shard by its window count. This still samples overlapping windows with replacement, not a no-repeat epoch.

A tokenizer hash, EOS policy, dtype, corpus license, preprocessing/split identity and each shard hash are required provenance. Different file paths are not proof of independent content; group/deduplicate before making shards. The loader catches identical train/validation files and ID range errors, but cannot prove semantic decontamination. EOS-separated packing allows earlier documents as context unless an additional block attention mask is supplied. This recipe intentionally uses ordinary causal packed-token pretraining and records that choice.

Complete local-only training recipe

Save the decoder block from Architecture Choices as architecture.py and the persistence block from Checkpointing as checkpointing.py, beside this block saved as pretrain.py. These are exact lesson programs, not pip packages. The code uses only NumPy and PyTorch already in the CPU runtime. train is opt-in: nothing downloads data, rents a GPU, starts a tracker or launches a large model. Its small synthetic smoke test is executed separately.

import copy, hashlib, json, math, time
from dataclasses import dataclass, asdict
from pathlib import Path
import numpy as np
import torch
import torch.nn.functional as F
from architecture import Config, Decoder
from checkpointing import atomic_save, trusted_load, capture_rng, restore_rng
 
@dataclass(frozen=True)
class Run:
    tokens: int=2_000_000_000
    micro: int=32
    accum: int=8
    length: int=1024
    peak: float=6e-4
    floor: float=6e-5
    warmup: int=500
    decay: float=0.1
    clip: float=1.0
    evaluate_every: int=500
    save_every: int=500
    val_batches: int=20
    seed: int=42
 
def lr_at(i, steps, run):
    if not 0<=run.warmup<steps or not 0<=run.floor<=run.peak:
        raise ValueError('schedule configuration')
    if i<0: raise ValueError('negative update')
    if run.warmup and i<run.warmup: return run.peak*(i+1)/run.warmup
    if i>=steps-1: return run.floor
    span=steps-1-run.warmup
    p=(i-run.warmup)/span if span else 1.0
    return run.floor+0.5*(1+math.cos(math.pi*p))*(run.peak-run.floor)
 
class Shards:
    def __init__(self, paths, length, vocab, seed):
        if not paths or not 1<vocab<=65536 or length<1: raise ValueError('data config')
        self.paths=[Path(p) for p in paths]; self.length=length
        self.arrays=[]; self.manifest=[]
        for p in self.paths:
            if p.stat().st_size%2: raise ValueError('uint16 byte length')
            digest=hashlib.sha256()
            with p.open('rb') as source:
                for chunk in iter(lambda:source.read(1024*1024),b''): digest.update(chunk)
            a=np.memmap(p,mode='r',dtype='<u2')
            if len(a)<=length or int(a.max())>=vocab: raise ValueError('shard length or ID')
            self.arrays.append(a)
            self.manifest.append({'name':p.name,'sha256':digest.hexdigest(),'tokens':len(a)})
        self.ends=np.cumsum([len(a)-length for a in self.arrays])
        self.rng=np.random.default_rng(seed)
    def batch(self,n,device):
        if n<1: raise ValueError('batch')
        starts=self.rng.integers(0,int(self.ends[-1]),size=n)
        rows=[]
        for j in starts:
            s=int(np.searchsorted(self.ends,j,side='right'))
            start=int(j-(self.ends[s-1] if s else 0))
            rows.append(np.array(self.arrays[s][start:start+self.length+1],dtype=np.int64))
        a=torch.from_numpy(np.stack(rows)).to(device)
        return a[:,:-1].contiguous(),a[:,1:].contiguous()
 
@torch.no_grad()
def evaluate(model,data,batches,micro,device):
    mode=model.training; state=copy.deepcopy(data.rng.bit_generator.state)
    total=0.; count=0
    try:
        model.eval()
        for _ in range(batches):
            x,y=data.batch(micro,device); logits=model(x)
            loss=F.cross_entropy(logits.reshape(-1,logits.shape[-1]),y.reshape(-1),reduction='sum')
            if not torch.isfinite(loss): raise FloatingPointError('evaluation')
            total+=float(loss); count+=y.numel()
        if not count: raise ValueError('empty evaluation')
        return {'nll':total/count,'ppl':math.exp(total/count),'tokens':count}
    finally:
        model.train(mode); data.rng.bit_generator.state=state
 
def train(train_paths,val_paths,out,model_cfg=Config(context=1024),run=Run(),
          resume=None,stop_after=None,device='cpu'):
    if min(run.tokens,run.micro,run.accum,run.length,run.evaluate_every,
           run.save_every,run.val_batches)<=0 or run.clip<=0: raise ValueError('run bounds')
    if run.length>model_cfg.context: raise ValueError('context')
    out=Path(out); out.mkdir(parents=True,exist_ok=True)
    # One writer per directory; never overwrite a completed run or adopt old output.
    if (out/'DONE.json').exists() or (resume is None and any(out.iterdir())):
        raise FileExistsError('use an empty output directory or explicitly resume an unfinished run')
    torch.manual_seed(run.seed)
    data=Shards(train_paths,run.length,model_cfg.vocab,run.seed)
    val=Shards(val_paths,run.length,model_cfg.vocab,run.seed+1)
    if {x['sha256'] for x in data.manifest}&{x['sha256'] for x in val.manifest}:
        raise ValueError('identical train/validation shard')
    contract={'model':asdict(model_cfg),'run':asdict(run),'train':data.manifest,
              'validation':val.manifest,'packing':'ordinary causal; cross-document context allowed'}
    model=Decoder(model_cfg).to(device)
    params=list(model.parameters())
    decay=[p for p in params if p.ndim>=2]; nodecay=[p for p in params if p.ndim<2]
    assert len({id(p) for p in decay+nodecay})==len(params)
    opt=torch.optim.AdamW([{'params':decay,'weight_decay':run.decay},
                          {'params':nodecay,'weight_decay':0.}],lr=run.peak,betas=(.9,.95))
    tokens_per_update=run.micro*run.accum*run.length
    steps=math.ceil(run.tokens/tokens_per_update); lr_at(0,steps,run)
    next_update=seen=0; history=[]
    if resume is not None:
        payload=trusted_load(resume)
        if payload['contract']!=contract: raise ValueError('resume contract mismatch')
        model.load_state_dict(payload['model']); opt.load_state_dict(payload['optimizer'])
        next_update=payload['next_update']; seen=payload['tokens_seen']; history=payload['history']
        data.rng.bit_generator.state=payload['sampler']; restore_rng(payload['rng'])
    def save():
        atomic_save({'format':1,'contract':contract,'model':model.state_dict(),
                     'optimizer':opt.state_dict(),'next_update':next_update,'tokens_seen':seen,
                     'history':history,'sampler':copy.deepcopy(data.rng.bit_generator.state),
                     'rng':capture_rng()},out/'latest.pt')
    model.train()
    while seen<run.tokens and (stop_after is None or next_update<stop_after):
        count=min(tokens_per_update,run.tokens-seen); remaining=count
        lr=lr_at(next_update,steps,run)
        for group in opt.param_groups: group['lr']=lr
        opt.zero_grad(set_to_none=True); nll=0.; start=time.perf_counter()
        while remaining:
            x,y=data.batch(run.micro,device); labels=y.clone()
            used=min(remaining,labels.numel()); labels.view(-1)[used:]=-100
            logits=model(x)
            summed=F.cross_entropy(logits.reshape(-1,model_cfg.vocab),labels.reshape(-1),
                                   ignore_index=-100,reduction='sum')
            if not torch.isfinite(summed): raise FloatingPointError('training loss; no update committed')
            (summed/count).backward(); nll+=float(summed.detach()); remaining-=used
        norm=torch.nn.utils.clip_grad_norm_(params,run.clip,error_if_nonfinite=True)
        opt.step(); next_update+=1; seen+=count
        if str(device).startswith('cuda'): torch.cuda.synchronize()
        elapsed=time.perf_counter()-start
        row={'update':next_update,'tokens':seen,'nll':nll/count,'lr':lr,
             'preclip_norm':float(norm),'train_seconds':elapsed,'train_tokens_per_second':count/elapsed}
        if next_update%run.evaluate_every==0 or seen==run.tokens:
            row['validation']=evaluate(model,val,run.val_batches,run.micro,device)
        history.append(row)
        if next_update%run.save_every==0 or seen==run.tokens: save()
    save()
    (out/'metrics.json').write_text(json.dumps({'contract':contract,'history':history},indent=2))
    if seen==run.tokens:
        # Completion records the actual final checkpoint digest, not a stale empty sentinel.
        (out/'DONE.json').write_text(json.dumps({'tokens':seen,'next_update':next_update,
            'sha256':hashlib.sha256((out/'latest.pt').read_bytes()).hexdigest()}))
    return model,history

Use one writer per output directory. A fresh run requires an empty directory; an explicit resume may reuse an unfinished directory but refuses one containing DONE.json. To branch from a completed checkpoint, choose a new empty output directory and pass the trusted checkpoint explicitly. This prevents a partial rerun from overwriting latest.pt while leaving an old completion receipt. The guard is not a cross-process lock: concurrent writers are unsupported.

The code intentionally trains in FP32 on CPU by default. A full 100M/2B run would be resource-dependent and was not executed. The same function can be explicitly pointed at authorized existing local shards and an available device; its unfused FP32 reference is not a throughput-optimized GPU prescription. Before optional BF16 autocast, test support and keep master parameter/state semantics explicit. FP16 uses the supported GradScaler sequence: scale backward, unscale, finite-check/clip, step, update; a skipped optimizer step must not silently advance an optimizer-step schedule. DDP additionally needs global token denominators, disjoint rank sampling, synchronized update/skip decisions and all-rank restart state. This single-process code is not torchrun-ready merely because a launcher exists.

Pre-flight, monitoring and sampling

1. Validate model count, optimizer membership, IDs, shifts, masks, tokenizer round-trip and local shard hashes. Record free storage and peak memory rather than assuming 20GB is enough.
2. Fit a tiny fixed batch and compare with a deliberately disabled update. There is no universal hundred-step threshold or80% bug coverage. Memorizing inconsistent labels does not validate their semantics.
3. Test interrupted versus uninterrupted continuation before committing the budget. Include the next sampler state and next update, not an ambiguous “step.”
4. Check a fixed validation probe with mode restoration. A no-dropout/no-BN model may behave identically in train/eval; forgetting .train() does not guarantee a five-percent loss penalty.
5. Run the decoder's bounded generation method with the exact tokenizer, stop IDs and context contract. Greedy means argmax, not division by a zero temperature. Compare fixed prompts and seeds; do not invent stage-specific English completions as measured outputs.
6. Decide finite stopping/alert conditions and logging cadence. Local JSON is sufficient; remote trackers are optional and were not invoked.

Loss need not decrease monotonically and is not cosine-shaped because LR is cosine. ln(32000)=10.373491 is the loss of uniform predictions, not every random initialization. A lower validation NLL can reflect genuine improvement, a changed denominator, easier data or leakage; no3.3–3.6 target range distinguishes them. Retain token-weighted train/validation and per-domain metrics, preclip norms, used LR, update/token clocks, phase timings and a small generation probe.

Time only completed work. The original first logging interval multiplied one update by a ten-update factor. This program reports per-update training throughput; end-to-end throughput must include evaluation, persistence, startup and restarts. A checkpoint pause cannot by itself explain permanently slower compute steps. SSH or nvidia-smi is not known to cost5–10% here; profile rather than prohibit ordinary monitoring.

Reading published small-model runs without inventing a benchmark

Karpathy's nanoGPT/llm.c and the modded-nanoGPT speedrun are useful because they expose code, hardware, data and evaluation rules. They are different experiments. A FineWeb tokenization/split loss threshold is not directly comparable to another tokenizer's OpenWebText or FineWeb-Edu loss. The evolving speedrun includes optimizer, numerical, architecture and systems changes; freeze a commit and rules before attributing a gain to Muon, FP8, value residuals or embedding sharing. A bundled benchmark code path is not a guarantee of supported production behavior.

FineWeb-Edu is an educational-quality-filtered corpus with documented construction, not a permanent winner over every dataset. Data-constrained scaling studies examine repetition and compute tradeoffs; they do not promise a0.15–0.25 loss decrease from doubling this run. Nor are unpublished GPT-4 perplexity, parameter and token counts legitimate comparators.

Worked practice and retention answers

• Updates: floor7629, ceil7630, exact tail103424. Explain which clock drives the scheduler and where it is saved.
• Initial CE: uniform logits produce ln(V); nonuniform logits can produce larger or smaller batch loss. Confirm the loss reduction before diagnosing architecture.
• Overfit test: establish that gradients can change parameters and fit a chosen fixture. Include no-step, wrong-shift and leak controls; passing does not prove correct labels or generalization.
• Healthy final loss: no universal number. Compare matched tokenizer/split/context/mask against a count baseline and earlier checkpoints; inspect downstream capability and uncertainty.
• PPL at3.4: exp(3.4)=29.9641, reciprocal geometric-mean probability. It is not a claim that exactly30 candidates are tied or that this is a particular “intelligence tier.”
• Double tokens: approximate linear training compute at fixed architecture and sequence regime; fixed overhead, data repeats and schedule changes break exact cost scaling. Measure quality; do not pre-fill a gain.
• 300M versus1B stretch: under D=20N, compute ratios9 and100 relative to100M. Divide each FLOP budget by an actually measured sustained rate and separately add storage/recovery/eval cost; parameter scaling alone is not a rental quote.
• Warmup and decay recall: smaller LR at first can protect optimization, but no universal inverse-square-root model-size LR law is used. Decay grouping is an explicit policy, not a proof that all1D decay is harmful.

Sources and execution boundary

Primary support: Chinchilla, data-constrained scaling, Adam implementation, torch2.9, FineWeb-Edu card, llm.c, modded-nanoGPT. Reports pin retrieved bytes/dates, not an everlasting leaderboard.

Only the bounded generated-token CPU run and independent arithmetic/control tests are executed. Full local training is supplied but unexecuted; no dataset/weights, paid API, GPU, tracker or infrastructure was obtained. Optional reading and next steps are listed below. Review tomorrow: reconstruct the four clocks and trace the final partial update through save/resume.

Inherited optional resources and navigation

These pre-read links are optional further reading; their timestamps, video claims and optional pacing are not accepted technical evidence.

• Reproducing GPT-2 (124M) from scratch — 4-hour walkthrough
• Let's build GPT: from scratch, in code, spelled out
• nanoGPT train.py
• GPT-3 Model Sizes

Previous: ← DL S052 · Next: DL S054 →