Deep Learning & LLMs From Scratch · 80 Sessions

The pitch: 80 self-paced deep sessions. From matrix multiplication to your own instruction-tuned LLM in ~160 hours. Zero DL knowledge assumed. No calendar — pace it your own way. One or two sessions per week works.

The story

Here's a thing that took me embarrassingly long to internalize: a large language model is a chain of matrix multiplies with a softmax on top. That's it. The magic is in which matrices, how many of them, and what data the numbers inside got shaped by. Once you actually build one — really build one, not pip install transformers build one — the whole field stops feeling like alchemy and starts feeling like a small stack of tricks you already understand.

This series is my attempt to walk that path once, publicly, at a pace that a working engineer with a day job can actually keep up with. It borrows shamelessly from the shoulders of giants: Andrej Karpathy's Zero to Hero, 3Blue1Brown's Essence of Linear Algebra, Sebastian Raschka's LLM book, Umar Jamil's line-by-line PyTorch walkthroughs, and Stanford's CS224N. Where those exist, I point at them. Where they leave a gap — usually the "wait, but why this choice?" gap — I try to fill it.

The end state is concrete. After Session 080 you will have:

• Trained your own 100M-parameter language model from scratch on a rented GPU, with your own tokenizer, your own dataset, and your own training loop.
• Fine-tuned it with SFT + LoRA + DPO to follow instructions.
• Quantized and served it behind an HTTPS endpoint with a KV cache and continuous batching.
• Written it up so someone else can reproduce it.

And along the way — this is the part that actually matters — you will have derived every important equation, hand-computed enough tiny numeric examples that the tensor shapes stop being scary, and rebuilt every core piece (autograd, backprop, self-attention, RoPE, LoRA, quantization) at least once in your own editor.

How to use this series

Rhythm. One or two sessions per week is a healthy pace. Each session is designed as a ~2-hour deep dive: 20 minutes of pre-read videos, 90 minutes of the main text plus code, 10 minutes of the retention scaffold at the end. Do not binge — the whole point is spaced practice. Your future self will thank you.

The study loop for a single session:

1. Pre-read (20 min). Skim the 2–3 videos and 2–3 readings I've hand-picked at the top. All URLs are oembed-verified live. If a video feels too easy, skip it; if it feels too hard, watch it twice.
2. Deep dive (~90 min). Read the session slowly, with a scratchpad. Do every numeric worked example by hand before you scroll to the answer. Type out (do not copy-paste) every code snippet.
3. Recall (10 min). Do the 5 recall questions at the bottom of every session — from memory, no scrolling. If you get fewer than 4/5, do not advance. Come back tomorrow.
4. Sticky note (30 sec). Every session ends with "what to bring back tomorrow": one diagram, one equation, one code snippet. Write them on a sticky note above your monitor. That is your review deck.
5. Advance. Move to the next session only when the recall passes.

When you fall behind (and you will): the modules are ordered so a break is fine. Just re-do the recall questions from the last session you completed before advancing.

Prerequisites

• Python fluency. You can write a for-loop, define a class, write a decorator. If you can't, do the Python module of the 6-month plan first (sessions S005–S014).
• High-school algebra. You remember what a derivative is, or you're willing to relearn in Session 004.
• A computer. Any laptop is fine for Modules M01–M08. Modules M09 onwards use a rented cloud GPU (~$10–$50 total budget for the whole run if you're careful).

You do NOT need: a math degree, an ML class, prior PyTorch experience, prior calculus fluency, or a GPU on your own machine.

The 12 modules at a glance

Module cards

M01 · Math for DL from scratch — sessions 001–006

The math you actually need, re-derived so it feels earned. NumPy warm-up, vectors and matrices as geometry, broadcasting, derivatives and the chain rule, probability and cross-entropy, optimization intuition.

You will build: a set of hand-verified NumPy notebooks that prove every equation the later modules will use.

M02 · Neural nets from scratch in NumPy — sessions 007–013

No PyTorch yet. We build a single neuron, then an MLP, then MNIST, then a tiny autograd engine (micrograd-style). By the end of this module you have a working autograd system you wrote yourself in ~100 lines.

You will build: an MLP that hits >95% on MNIST in pure NumPy, plus a micrograd clone.

M03 · PyTorch fluency — sessions 014–019

Now we adopt PyTorch — but with the earned intuition that nothing here is magic. Tensors, autograd, nn.Module, DataLoaders, the training loop, GPU + mixed precision, and debugging.

You will build: the same MNIST MLP in PyTorch, plus a production-grade training loop with logging, checkpointing, and reproducibility.

M04 · Regularization + optimization deep dive — sessions 020–024

Dropout, batch/layer norm, weight init, schedulers, gradient clipping and accumulation. The knobs that turn "it trains" into "it trains reliably".

You will build: a config-driven training script that you'll reuse for every subsequent model.

M05 · Convnets + vision — sessions 025–030

Convolutions from first principles, then LeNet → ResNet → EfficientNet → ViT. Transfer learning and augmentation strategies that actually work.

You will build: a CIFAR-10 ResNet at >92% + a fine-tuned model on a dataset of your choice.

M06 · RNNs, sequences, embeddings — sessions 031–035

Word2Vec, RNN cells from scratch, LSTM + GRU, seq2seq with teacher forcing, and attention as the fix for the RNN bottleneck. This module exists to earn the transformer.

You will build: a char-level RNN language model + a Bahdanau-attention seq2seq translator.

M07 · Transformers from scratch — sessions 036–043

Self-attention derived, multi-head, positional encodings (sinusoidal → RoPE), the full encoder-decoder, then a GPT-style decoder rebuilt line by line (nanoGPT-style), then trained on Tiny Shakespeare, then a KV cache added, then flash-attention intuition.

You will build: a working ~10M-param GPT on Tiny Shakespeare that generates coherent text.

M08 · Tokenization + data + scaling laws — sessions 044–049

BPE from scratch, SentencePiece, dataset curation (SlimPajama-style), Chinchilla scaling laws, data mixing, quality filters + deduplication.

You will build: a real BPE tokenizer + a curated 1B-token pretraining dataset.

M09 · Pretraining your own foundation model — sessions 050–057

Architecture choices (GPT/Llama/Mistral), DDP/FSDP intuition, renting a GPU, running your first real pretraining job, reading loss curves, checkpointing, common bugs, and scaling math.

You will build: your own 100M-param foundation model, trained end-to-end on rented compute.

M10 · Fine-tuning + alignment — sessions 058–065

SFT, instruction datasets, LoRA + QLoRA from scratch, reward modeling, RLHF/PPO intuition, DPO, and evaluation (MMLU, HumanEval, custom).

You will build: your foundation model → an instruction-following chat model.

M11 · Efficient inference + serving — sessions 066–073

KV cache deep dive, quantization (INT8/INT4/GPTQ/AWQ), pruning + distillation, speculative decoding, continuous batching, vLLM internals, FastAPI + Modal/Runpod deploy, and latency-vs-throughput math.

You will build: your chat model behind a real HTTPS endpoint with streaming SSE.

M12 · Capstone + wildcards — sessions 074–080

Multimodal (CLIP → LLaVA), agents & tool use, RAG from scratch, MoE intuition, long context (RoPE scaling, YaRN, ALiBi), red-team + safety eval, and the CAPSTONE: ship your own instruction-tuned model + write it up publicly.

You will build: the capstone — a model on HuggingFace with your name on it and a blog post explaining every decision.

All 80 sessions

Each session is ~2 hours (pre-read + deep-dive + recall). Start with 001 and go in order — every session lists its prereqs at the top.

M01 · Math for DL — S001 · S002 · S003 · S004 · S005 · S006

M02 · Neural nets in NumPy — S007 · S008 · S009 · S010 · S011 · S012 · S013

M03 · PyTorch fluency — S014 · S015 · S016 · S017 · S018 · S019

M04 · Regularization + optimization — S020 · S021 · S022 · S023 · S024

M05 · Convnets + vision — S025 · S026 · S027 · S028 · S029 · S030

M06 · RNNs, embeddings — S031 · S032 · S033 · S034 · S035

M07 · Transformers from scratch — S036 · S037 · S038 · S039 · S040 · S041 · S042 · S043

M08 · Tokenization + data + scaling — S044 · S045 · S046 · S047 · S048 · S049

M09 · Pretraining your foundation model — S050 · S051 · S052 · S053 · S054 · S055 · S056 · S057

M10 · Fine-tuning + alignment — S058 · S059 · S060 · S061 · S062 · S063 · S064 · S065

M11 · Efficient inference + serving — S066 · S067 · S068 · S069 · S070 · S071 · S072 · S073

M12 · Capstone + wildcards — S074 · S075 · S076 · S077 · S078 · S079 · S080

FAQ

Q: Do I need a GPU? Not for the first eight modules. Everything through Session 049 runs on any laptop (some sessions in Module M07 want ~30 minutes on a GPU, but a Colab free tier is enough). Starting at Session 052 you rent a real GPU for a few hours. Total cloud spend for the whole series, if you're careful, is $10–$50.

Q: How long will this take? At 1 session per week, ~18 months. At 2 sessions per week, ~9 months. The point is not speed — the point is that after Session 080 you actually understand this stuff.

Q: Can I skip ahead? Yes, but not for free. Every session lists its prereq sessions. If you skip a prereq and the current session confuses you, that's your signal to go back.

Q: What if I don't know calculus? Session 004 re-derives everything you need. If it still feels fast, go do Khan Academy Calculus 1 chapters 1–3, then come back. You do not need integrals.

Q: PyTorch or TensorFlow? PyTorch. This is the field's lingua franca in 2026 and the vast majority of open-source LLM code is in it.

Q: How do I know I actually understand? The recall questions. If you can answer 4/5 from memory the day after a session — and re-derive the key equation without looking — you understand it. If you can't, redo the session. This is not a race.

Q: What comes after this? You'll have three obvious next steps: (1) read a real paper a week and reimplement the interesting bit, (2) contribute to an open-source LLM project (vLLM, Transformers, or similar), or (3) go build the thing you actually want to build. This series is the runway, not the destination.

Colophon

This series stands on giants' shoulders. Please go directly support them:

• Andrej Karpathy — Neural Networks: Zero to Hero is the canonical modern DL curriculum. Every session in Module M02 and much of Module M07 owes it a direct debt.
• 3Blue1Brown — the Essence of Linear Algebra and Neural Networks series are the intuition backbone for M01, M02, and M07.
• Umar Jamil — his line-by-line PyTorch walkthroughs of transformer, LLaMA, and Mistral papers are the reference implementation this series mimics.
• Sebastian Raschka — Build a Large Language Model (from Scratch) is the reference for the pretraining pipeline in M09.
• StatQuest and Yannic Kilcher — for making statistical intuition and paper reviews watchable.
• Stanford CS224N / CS231N — the classical curricula this reorganizes and updates.
• HuggingFace — for the tools that make the "actually train it" parts possible.

---

Ready? Start with Session 001 — NumPy Warm-up →