DL S080 · CAPSTONE — Train and Serve Your Own 100M Instruct Model

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 80 / 80 · Module M12 · The Last One · ~3 hours

The story

Eight months ago you opened a Python REPL, typed import numpy as np, and started multiplying matrices you didn't fully trust. Today you sit down for the last session of the series. Between those two moments: eighty sessions, seven modules, one working GPT-family model that fits in your laptop's RAM, and probably several late-night moments where a training run finally stopped NaN-ing and you almost cried. This session isn't new material. It's a graduation ceremony — with homework. You'll take everything you've built and turn it into a thing other people can download and run.

Stop for a second before you scroll on. Open the very first notebook you made — the numpy warmup from S001, the one where you built an outer product with x[:, None] * y[None, :] and stared at the shape broadcasting rules until they made sense. That person, eight months ago, did not know what a residual stream was, could not have named the three matrices inside attention, would have blinked at the word "tokenizer." That person did the work anyway. Every session since has been a small act of showing up: reading the paper, running the code, being confused for an hour, and pushing through. The 100M-parameter model you're about to ship is not what changed you. It's the artifact that proves what changed — the fact that eighty times in a row you sat down and did the thing. That's the muscle. The model is the receipt.

You did it

Take a breath. Look at your directory. Somewhere in there — probably in model.py, train.py, a checkpoints/ folder that ate way too much disk — is a program that eight months ago you would have called witchcraft. It takes text in. It produces text out. It knows the difference between Python and JavaScript. It can write a haiku. It can explain the softmax function while also being a softmax function, at multiple points in its own forward pass.

You built that. From numpy.array up. Through matrix multiplies, through autograd, through your own micrograd, through the first perceptron, through the MLP, through your first fully working backprop, through the first transformer block that finally didn't NaN, through nanoGPT reproduced from scratch, through pretraining runs that OOMed, through SFT that overfit and then didn't, through DPO that felt like it wasn't doing anything and then suddenly was, through quantization, through vLLM, through your first agent, through your first RAG, through YaRN, through your own red-team session where you found a jailbreak in your own model and had a small existential moment about the whole project.

Eight months. Eighty sessions. One model. One person. You. This session is a victory lap.

It is also a real session with real deliverables. By the end of the next three hours you will have:

1. A 100M-parameter instruct model trained, evaluated, quantized, and uploaded to Hugging Face Hub under your username.
2. A model card honestly documenting what it can and can't do.
3. A public writeup — blog post or long-form thread — that explains what you built and how.
4. A plan for what to build next, because you're not done, you're just done with this series.

Let's ship.

Prerequisites

• All 79 prior sessions. This is the final. If you skipped modules, this is where the shortcuts catch up with you. Go back and skim; the payoff is real.
• A Hugging Face account (free) with an API token that has write access.
• A cheap GPU (rented T4 or A10 works; even CPU is fine for the final finetune passes if you're patient).
• ~50 GB of disk. Checkpoints add up.
• Two afternoons. One for training + evaluation. One for writing + shipping.

Part I · Train the final model (afternoon 1)

1 · Pick your target

You're not training a competitive frontier model. You're training the smallest thing worth publishing. Target:

• ~100M parameters. Big enough to be genuinely useful for chat over your domain; small enough that people can run it on a laptop.
• ~1B tokens of pretraining. Enough to be coherent, small enough to finish in one GPU-day.
• ~20k SFT examples. Enough for identifiable "voice" and helpful behavior.
• ~2k DPO pairs. Enough to visibly nudge preferences.

2 · Architecture snapshot

Freeze one config. Suggested:

Model:  100M-param decoder-only transformer
d_model: 768
n_layers: 12
n_heads: 12
d_ff: 3072 (4x)
vocab: 32000 (SentencePiece BPE, trained on your pretrain corpus)
seq_len: 2048 (with RoPE; YaRN-extendable to 8k later)
activation: SwiGLU
norm: RMSNorm (pre-norm)
positional: RoPE (base 10000)
attention: GQA with 4 KV heads
tie word embeddings: yes

If those choices feel arbitrary, they aren't — every single one traces back to a session. Look them up if any feels fuzzy.

3 · Pretraining, in three commands

Assuming your training scripts from S046–S050:

# Prepare tokenizer + tokenized shards
python scripts/prepare_data.py \
  --input data/pretrain/*.txt \
  --tokenizer sentencepiece \
  --vocab_size 32000 \
  --output data/shards/

The tokenizer step is fast — usually under 10 minutes on a laptop CPU. What matters here is that your vocab_size=32000 and --input glob match whatever you actually collected in S046 (pretraining dataset). If they don't, your training loss will look reasonable but your generations will produce Unicode confetti. Once shards land on disk, kick off the actual pretraining:

# Pretrain
python train.py \
  --config configs/100m.yaml \
  --data data/shards/ \
  --steps 100000 \
  --batch_size 1024 \
  --grad_accum 8 \
  --lr 3e-4 \
  --wandb_project capstone-pretrain
 
# Convert final checkpoint to HF format
python scripts/to_hf.py checkpoints/step_100000.pt output/base-model/

The --grad_accum 8 compensates for whatever batch size your single GPU can actually hold — effective batch stays at 1024, memory footprint stays under 24GB. wandb matters less for the numbers and more for the loss-curve shape you'll show off in the blog post.

Total wall clock on 1× A10G: about 18 hours. On a T4 you can shrink to 50M params or run 8h and stop early.

4 · SFT

Get your instruction dataset ready. A good starter mix (all publicly available):

• 10k samples from databricks/databricks-dolly-15k
• 5k from OpenAssistant/oasst1 (top-scored English pairs)
• 3k from a domain-specific dataset (write your own — 100 careful examples in your voice beat 3k random ones)
• 2k safety examples (refusals, harmful-content rejections, PII refusals) — see S079

Format each as a chat template. Use ChatML — it's a de-facto standard:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

Train:

python sft.py \
  --base output/base-model/ \
  --data data/sft_mix.jsonl \
  --steps 5000 \
  --lr 5e-5 \
  --lora_r 16 \
  --wandb_project capstone-sft

LoRA rank 16 gets you 80% of the quality at 5% of the memory. Use it.

5 · DPO (optional but adds a lot)

Take ~2k preference pairs (prompt, chosen, rejected). Sources: filter oasst1 for pairs with quality-score deltas > 0.3, or generate synthetic pairs by prompting a bigger model.

python dpo.py \
  --base output/sft-model/ \
  --pairs data/dpo_pairs.jsonl \
  --steps 1000 \
  --beta 0.1 \
  --lr 5e-6

Beta 0.1 is a safe default (higher = more aggressive preference push, more risk of collapse).

6 · Evaluate

Run at minimum:

• Perplexity on a held-out slice of pretrain corpus (sanity check).
• MT-Bench (or a simpler 30-question benchmark you write). Compare base-SFT vs base-SFT-DPO.
• Your red-team suite from S079. Log the numbers. All of them.

Record the numbers in a JSON file. You will cite them in your model card and your blog post.

7 · Quantize

# GGUF quantization for llama.cpp compatibility
python convert-hf-to-gguf.py output/dpo-model/ --outfile my-capstone-100m.gguf
./quantize my-capstone-100m.gguf my-capstone-100m-q4_k_m.gguf q4_k_m

Q4_K_M gives you a ~60MB file that runs on a phone. Test that it actually loads and generates before you push.

Part II · Ship it (afternoon 2, morning)

8 · The Hugging Face Hub push

huggingface-cli login   # paste your write token
huggingface-cli repo create your-capstone-100m --type model
 
cd output/dpo-model
git init
git lfs install
git remote add origin https://huggingface.co/YOUR_USERNAME/your-capstone-100m
git lfs track "*.safetensors" "*.gguf" "*.bin"
cp ../my-capstone-100m-q4_k_m.gguf .
git add .
git commit -m "Initial release: 100M-param instruct model"
git push

Also push the GGUF so people can run it in llama.cpp / Ollama without conversion.

9 · The model card (README.md)

This is important. Not marketing. Honest. Template — first the identity, license, and training sections, which are what a downstream user reads before they decide whether to trust your weights:

---
license: apache-2.0
language: en
library_name: transformers
tags:
  - text-generation
  - chat
  - from-scratch
datasets:
  - databricks/databricks-dolly-15k
  - OpenAssistant/oasst1
base_model: none-trained-from-scratch
---
 
# Your Capstone 100M
 
## What it is
A 100M-parameter decoder-only transformer, pretrained from scratch on 1B tokens 
and fine-tuned via SFT + DPO on ~20k instruction-following examples.
 
Built as the capstone project of an 80-session self-study series 
(link to your blog post).
 
## Intended use
- Learning / experimentation.
- Lightweight chat assistant on constrained hardware (works on CPU, phone).
- Explicitly NOT for: medical/legal advice, high-stakes decisions, 
  production use without additional safety layers.
 
## Training
- **Architecture:** 12-layer, d=768, GQA-4, RoPE, SwiGLU, RMSNorm.
- **Pretrain:** 1B tokens (Wikipedia + BookCorpus + subset of C4).
- **SFT:** 20k mixed instruction data.
- **DPO:** 2k preference pairs.
- **Compute:** ~24 GPU-hours on 1× A10G.
 
## Evaluation
| Benchmark | Score |
|---|---|
| Perplexity (held-out) | 12.3 |
| MT-Bench (single-turn, human eval, n=30) | 4.2 / 10 |
| Red-team refusal rate (garak sample) | 71% |

Notice what the top half is doing: it establishes provenance. License, training data, compute budget, honest benchmark numbers — the boring facts that separate a research artifact from a random .bin file on the internet. The bottom half is different in character. It is the section that turns a release from a marketing announcement into an engineering artifact: the failure-mode disclosure, the exact code snippets someone will copy-paste, and the citation you would like people to use. This is the half that protects both you and your users.

## Known failure modes
- Hallucinates factual claims about post-2023 events.
- Refuses harmful English prompts ~71% but drops to ~40% for non-English inputs.
- Susceptible to universal adversarial suffixes (as all open-weight models are).
- Cannot count objects reliably. Poor at arithmetic beyond 3 digits.
 
## How to use
See `example.py` in this repo, or:
` ` `python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("YOUR_USERNAME/your-capstone-100m")
tok   = AutoTokenizer.from_pretrained("YOUR_USERNAME/your-capstone-100m")
# ... standard chat template usage
` ` `
 
Or with llama.cpp:
` ` `bash
llama-cli -m my-capstone-100m-q4_k_m.gguf -p "Hello!" -n 100
` ` `
 
## Citation
` ` `
@misc{your_capstone_100m_2026,
  author = {Your Name},
  title  = {Your Capstone 100M: A From-Scratch 100M-Parameter Instruct Model},
  year   = 2026,
  url    = {https://huggingface.co/YOUR_USERNAME/your-capstone-100m}
}
` ` `

Commit and push. It's live.

10 · Test the "three-line install"

Open a fresh Colab (or any machine that doesn't have your project on it). Verify:

from transformers import pipeline
p = pipeline("text-generation", model="YOUR_USERNAME/your-capstone-100m")
print(p("Hello!", max_new_tokens=50))

If that works with no setup, you shipped a real thing. If it errors, fix and re-push before writing the blog post.

Part III · Write the post (afternoon 2, afternoon)

11 · Blog-post structure that works

Every good "I built an X" post I've read follows roughly:

1. The hook (1 paragraph). One sentence about what you did, one sentence about why it's cool that a single person did it. Concrete metrics in the hook (100M params, 24 GPU-hours, 8 months of study).
2. Why you did this (2 paragraphs). Personal motivation. Don't skip this — it's what makes it a blog post and not a paper. Readers want to know if they can do it too.
3. What you built (3–5 paragraphs, 1 architecture diagram). Model card in prose form.
4. How you built it (4–6 paragraphs). The interesting decisions and trade-offs. Link to the series. Don't rewrite the whole 80 sessions — reference them and pick 3 highlights.
5. What broke (1 paragraph). One or two war stories. Concrete, specific, funny if you can.
6. What it can and cannot do (2 paragraphs, benchmark table, sample outputs — both good and bad). Honesty compounds.
7. What's next (2 paragraphs). Point at your next project so it's clear this isn't the end.
8. How to try it (code block). The three-line install.

Target 1500 words. Half a day of writing.

12 · Blog-post-writing tips (learned the hard way)

• Show, don't tell. "It can write haikus" is weaker than a screenshot of it writing a haiku.
• Include bad outputs. People trust posts that show failure. The reader can then judge for themselves.
• One chart maximum. Loss curve, or eval score progression across training runs. That's it.
• Cite the paper for every architectural choice you can. It signals rigor and helps readers dig deeper.
• Do not claim beat-frontier-model results with a 100M model. Everyone will notice, and it will drown the actual accomplishment.
• Do claim that this is a learning artifact, not a product. That's the honest positioning and it's more interesting anyway.
• End with a specific ask. "If you build something on top of this, tell me. My email is X." Community grows from asks.

13 · Where to post

• Personal blog (dineshblog.com in this case) — canonical.
• Twitter/X thread — link + one killer screenshot per tweet, 5-8 tweets.
• HackerNews Show HN — "Show HN: I trained a 100M-param LLM from scratch in 8 months" — do this if the post is clean.
• r/LocalLLaMA — this community loves small models from scratch. Cross-post the model card.
• HN + Reddit are stochastic. Post at Tuesday 9am ET for best odds. Have a thick skin.
• LinkedIn if you swing that way. Different audience; hits different.

Part IV · Community and next steps

14 · Join the ongoing conversations

• Hugging Face forums / Discord — where people who ship small models hang out.
• r/LocalLLaMA — noisy but the technical density is high.
• The #lm-hackers channel in EleutherAI's Discord — if you're serious.
• Papers with Code + arxiv-sanity — follow the from-scratch and small-model tag.
• Karpathy's Zero-to-Hero Discord — this is who you came from.

15 · What to build next — three concrete projects

Pick one. Not two. One.

Project A: Distill down to 25M. Take your 100M, apply the distillation techniques from S068. Aim for a model that runs on-device (phone, browser via WebGPU). Target 3-month project.

Project B: Agents on top of your model. SFT your model on function-calling data (see S075). Build a specific-purpose agent (personal calendar, code assistant, homework helper). Target 4–6 week project.

Project C: Go multimodal. Take a small vision encoder, apply the LLaVA-projector trick from S074, do the two-stage training. You now have a tiny multimodal assistant. Target 6-week project.

Bonus Project D: Domain fine-tune + serve to actual users. Pick a domain you know well (Dinesh: SQL / Spark / data engineering). Fine-tune your 100M on 5k high-quality domain conversations. Ship a Discord bot or a web UI. Get 10 real users. Learn what actually breaks. Target 2-month project.

Part V · The bigger picture

16 · What you actually know now

You can:

• Explain any modern LLM paper without hand-waving.
• Debug training runs when they NaN.
• Read a model config and predict its FLOPs, memory, and quality ballpark.
• Distinguish real progress in the field from marketing noise.
• Ship your own model end-to-end.

That last one puts you in maybe the top 5% of ML practitioners globally. Not because it's hard (it's just tedious), but because most people stop at "I read the transformer paper." You went past that. Way past.

17 · What you still don't know

Honest list:

• Scaling laws in practice. You built one model at one size. Real intuition for how quality scales with data/params/compute requires building 5–10 models across scales. You don't have that yet. Chinchilla, DeepMind's scaling paper, is a good next read.
• Distributed training at 100+ GPUs. You did single-GPU or maybe DDP on a small node. FSDP, ZeRO-3, sequence parallelism, MoE parallelism — these are their own beasts.
• The bleeding edge. Sparse attention, sub-quadratic attention, state-space models (Mamba), mixture of depths, speculative decoding at scale. The field moves. You'll always be catching up. That's fine.
• Product intuition. Knowing what to build for whom is a separate skill from being able to build it. If your goal is to make a product, most of your remaining leverage is in product / distribution / users, not in model architecture.

18 · One last thing

The most important skill you developed in the last eight months was not being intimidated by any component of the stack. When you read a paper now and it says "we use RoPE with YaRN scaling," you don't skim past — you know what both are, you can implement both, you've debugged both. That non-intimidation compounds. Every future thing you touch — RLHF variants you haven't seen, agent frameworks, new modalities — you now have the ground to stand on while you learn them.

That's the real capstone. Not the model. Not the blog post. Not the HF repo. The fact that no part of the modern DL/LLM stack is a black box to you anymore.

Congratulations. Ship it.

19 · Diagram — the shipping pipeline

🧠 Retention scaffold (the last one)

One-line summary (write it in your own words): _______________________________

Stretch — the only one that matters now:

Publish. Actually publish. Not tomorrow. Today. The blog post can be 1200 words instead of 1500. The chart can be ugly. The Discord bot doesn't need to be pretty. Ship, then iterate. The gap between "almost ready" and "public" is the widest gap in the whole 80-session journey; the person who closes it is the person who gets the compounding.

Spaced review — the whole series

For the last time, look back at:

• S001–S014 (math + NumPy + PyTorch foundations). You started here. Do you still remember why x[:, None] * y[None, :] gives an outer product? If not, it's not because you forgot — it's because you internalized. Prove it: implement one operation in NumPy today.
• S036–S045 (transformers). Draw the transformer block from memory. If you can, you own it.
• S046–S054 (pretraining + data + scaling). The "boring" sessions that make everything else possible. Where your intuition for real-scale ML lives.
• S055–S065 (fine-tuning, SFT, RLHF, DPO). The most under-appreciated modules. The difference between a base model and a useful chat assistant lives entirely here.
• S066–S073 (inference + deployment). Turning a checkpoint into a product.
• S074–S079 (multimodal, agents, RAG, MoE, long context, safety). The frontier — where you're now equipped to keep reading papers as they come out.

Next session (there isn't one — there's a next chapter): whatever you build. Come back to deep-learning-and-llms-from-scratch.mdx in six months and update it with a link to what you shipped. Then close the tab and go do the thing.

Sticky note (keep on your desk forever): attention = softmax(QKᵀ/√d_k) V — the single most important line of math in modern ML. And: the person who ships an ugly v1 beats the person who plans a beautiful v2.

Further reading — places to keep learning

• HuggingFace Daily Papers: https://huggingface.co/papers — curated arxiv, updated daily.
• Sebastian Raschka's Ahead of AI: https://magazine.sebastianraschka.com/
• The Gradient / Import AI (Jack Clark) / Interconnects (Nathan Lambert): newsletters that survive the hype.
• Karpathy's YouTube channel: https://www.youtube.com/@AndrejKarpathy — keep watching the new ones as they land.
• HuggingFace Open LLM Leaderboard: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
• RULER long-context leaderboard: https://github.com/hsiehjackson/RULER
• HarmBench leaderboard: https://www.harmbench.org/
• Papers with Code trending: https://paperswithcode.com/

Closing note

If you got here — really got here, sessions 1 through 80, code that runs, weights that push, a post that ships — you know something about yourself now that most people don't get to find out. You know you can start not knowing anything about a technical field and, given eight months of consistent effort, become someone who ships in it.

That knowledge is transferable. Robotics. Cryptography. Neuroscience. Whatever's next. The pattern is the same: pick a scaffold, don't skip sessions, build the ugly first draft, keep going when it NaNs.

Thanks for coming with me for eighty of these. Now go build the next thing.

— End of series.

Previous: ← DL S079 · Back to: Deep Learning & LLMs From Scratch — series hub