Deep Learning and LLMs: an inspectable learning laboratory

This course takes one question seriously: can you explain and test the computation you are claiming to understand? It begins with a matrix product and develops the full80-session arc: neural models, vision, sequence learning, transformers, tokenization/data, pretraining systems, adaptation/alignment, efficient serving, multimodal models, agents and a reproducibility capstone. A small trained transformer and evidence pipeline are accessible checkpoints, not the ceiling of the course. The emphasis is depth and reproducibility, not a calendar, a parameter-count trophy or a guaranteed cloud budget.

What you can build and how to scale it

• A scalar reverse-mode differentiation engine and a 17-parameter XOR network.
• A cross-correlation operator with verified input and kernel gradients.
• A recurrent-state calculation and a causal attention implementation.
• A complete 688-parameter NumPy decoder, including all parameter gradients.
• A deterministic byte-level BPE tokenizer, a bounded training run and a bigram comparison.
• A low-rank adaptation experiment, an incremental K/V cache and a local retrieval baseline.
• A capstone report with explicit task limits, reproducible tests and a weighted rubric.

The bounded decoder is a tiny transformer, not an assistant. The larger route retains the original100M pretraining, SFT/LoRA/DPO, quantization, streaming HTTPS and publishable report goals as optional staged experiments. Choose authorized local data and compute, prove each small contract first, and publish only measurements you actually make; no100M training, dataset size, endpoint or quality score is conferred by completing the reading. The retrieval lab is an executable lexical/evidence baseline, not a secretly mocked neural RAG demo. You will learn where a separately evaluated pretrained generator belongs without being told that an unrun API call proves anything.

Bring these prerequisites

You should be comfortable with Python functions, loops, dictionaries and basic classes, and with algebraic substitution and sums. You do not need prior PyTorch, a GPU, a cloud account or a paid API. The executable reference uses Python and NumPy only. The small programs were tested on CPU; use their assertions to check your own environment rather than trust a screenshot.

The preserved NumPy lab fences are independently runnable unless they explicitly import tiny_transformer.py; save that module beside those scripts. Broader framework lessons contain dependent snippets or experiment designs and may require separate datasets or GPUs. They are not automatically run by the local CPU checks. Never run cloud/download/serving commands merely because they appear in a lesson.

The progression

Figure text equivalent — labels and relationships

• Node “Arrays, derivatives and probability” (square).
• Node “Optimization and a single neuron” (square).
• Node “Complete neural network and generalization” (square).
• Node “Convolution and sequence state” (square).
• Node “Attention and a complete transformer” (square).
• Node “Tokenizer, data and measured training” (square).
• Node “Adaptation and cache-correct inference” (square).
• Node “Evidence retrieval and capstone” (square).
• “Arrays, derivatives and probability” leads to “Optimization and a single neuron”. Line: normal; ends: arrow point.
• “Optimization and a single neuron” leads to “Complete neural network and generalization”. Line: normal; ends: arrow point.
• “Complete neural network and generalization” leads to “Convolution and sequence state”. Line: normal; ends: arrow point.
• “Convolution and sequence state” leads to “Attention and a complete transformer”. Line: normal; ends: arrow point.
• “Attention and a complete transformer” leads to “Tokenizer, data and measured training”. Line: normal; ends: arrow point.
• “Tokenizer, data and measured training” leads to “Adaptation and cache-correct inference”. Line: normal; ends: arrow point.
• “Adaptation and cache-correct inference” leads to “Evidence retrieval and capstone”. Line: normal; ends: arrow point.

The table above is the tested NumPy spine. It is not the whole curriculum: the full 80-topic progression continues below, including framework, distributed, alignment, serving and multimodal study. Original URL numbers remain stable. Use the explicit links below for the full route. Automatic previous/next navigation may follow a different sequence; use this numbered list for the full course order.

A study loop with a falsifiable exit test

1. Predict: calculate the tiny numeric example before running it. Write the shape beside every array.
2. Execute: run the code and keep the assertions. A printed number without a checked expectation is weak evidence.
3. Perturb: change a batch size, repeated token, target or mask. Predict which invariants should remain true.
4. Explain: answer the exercises without looking. Use the answer sketches to diagnose a gap, not to count a page as read.
5. Record: note what was measured, the environment and what remains untested. A failed experiment is useful if its claim is honest.

For example, if a finite difference disagrees with backpropagation, do not immediately lower the learning rate. The learning rate does not appear in the derivative of the current loss. First inspect the objective, reduction, shapes, mutation order and nonsmooth points.

How sources and verification work

The NumPy documentation defines broadcasting behavior; PyTorch's autograd notes explain reverse-mode mechanics; Attention Is All You Need defines scaled attention and the original transformer architecture. We cite such sources at the claims they support. Our numerical examples are separately executed; a paper citation does not prove a copied implementation is correct.

Read each example’s stated test boundary rather than treating it as certification of every hardware path or security property. Each lab identifies important omissions: no ImageNet score from a convolution example, no natural-language competence from a five-symbol model, no production throughput from a cache equality test.

The advanced progression explicitly includes distributed training, resource planning, PPO/DPO, quantization, multimodal learning, agents, MoE and long-context methods. These are valid conceptual lessons with source-backed contracts. GPU, framework and deployment recipes have separate unexecuted boundaries; a conceptual lesson is not bad content merely because no accelerator experiment was run.

Your destination

The capstone is a replayable experiment with baselines, gradient checks, split evidence, cache parity and failure analysis. It is acceptable for a baseline to win. It is not acceptable to invent results, tune on the test set without disclosure, or call a local toy “production-ready.”

First checkpoint: for $X$ of shape $(4,3)$ and $W$ of shape $(3,2)$, what shape is $XW$? Answer: $(4,2)$; the shared feature dimension is summed out. If you can explain that and are willing to keep asking equally concrete questions, start with the arrays lesson.

The 12 modules at a glance

Math for deep learning — sessions 001–006

Arrays, geometry, broadcasting, differentiation, information and optimization. Exit: derive shapes, gradients and stable losses.

• 001: Arrays, shapes, and the first linear model
• 002: Vectors, Matrices, and the Geometry of Data
• 003: Broadcasting — The Rule That Runs the World
• 004: Derivatives are local measurements; backprop is bookkeeping
• 005: Probability, cross-entropy, and stable logits
• 006: Optimization: steps, momentum, AdamW, and honest experiments

Neural networks from scratch — sessions 007–013

Neurons, activations, MLPs, backpropagation, MNIST design, scalar autograd and vectorization. Exit: gradient-check a model before training.

• 007: One neuron: logistic regression, stable loss, and its boundary
• 008: Activation Functions — sigmoid, tanh, ReLU, GELU, SwiGLU, xIELU
• 009: The Forward Pass: Shapes, XOR and Parameter Counts
• 010: Backprop by Hand: Nine Gradients and a Verified XOR Network
• 011: MLP on MNIST: A Complete Local-Only NumPy Recipe
• 012: Micrograd: Build and Test a Scalar Autograd Engine
• 013: Vectorization: Measure Speed, Shapes and Memory

PyTorch fluency — sessions 014–019

Tensors/autograd, registered modules, datasets/loaders, training loops, mixed precision and debugging. Exit: explain state and execution mode independently of a framework demo.

• 014: PyTorch Tensors and Autograd: Leaves, VJPs and Safe Updates
• 015: nn.Module: Registration, Composition and Checkpoint Contracts
• 016: Datasets, DataLoaders and Collate: Coverage Before Throughput
• 017: The Training Loop, Logging, and Reproducibility
• 018: GPU and Mixed Precision
• 019: Debugging — nan, inf, and Overfitting a Single Batch

Regularization and optimization — sessions 020–024

Dropout, normalization, initialization, schedules, clipping and accumulation. Exit: distinguish changes to the objective, estimator and numerical procedure.

• 020: Dropout and generalization — masks, scaling and controlled experiments
• 021: Batch Norm, Layer Norm, and RMSNorm — the normalization tour
• 022: Weight Initialization — Xavier, He, and μP
• 023: Schedulers — Warmup, Cosine, One-Cycle, and WSD
• 024: Gradient Clipping and Accumulation

Vision — sessions 025–030

Cross-correlation, CNN architectures, CIFAR experiment design, transfer, augmentation and ViT. Exit: compute shapes and test label-preserving assumptions.

• 025: Convolution: local shared operators, shapes and gradients
• 026: CNN architectures: LeNet, AlexNet, residual learning and modern backbones
• 027: CIFAR-10 training: a complete local pipeline and controlled experiments
• 028: Transfer learning: frozen features, fine-tuning and buffer discipline
• 029: Data augmentation: tested mixing, task assumptions and calibration
• 030: Vision Transformers: patch equivalence, position and attention cost

Embeddings and sequences — sessions 031–035

Word2vec, RNNs, LSTM/GRU, seq2seq and additive attention. Exit: trace recurrent derivatives and the encoder-decoder information path.

• 031: Word2Vec: complete skip-gram training, geometry and honest evaluation
• 032: RNNs from scratch: complete BPTT, character generation and structured recurrence
• 033: LSTM and GRU — Gates, State and Gradient Paths
• 034: Seq2Seq — A Complete Length-Aware Encoder–Decoder
• 035: Why Attention — Query-Dependent Source Access

Transformers — sessions 036–043

Attention derivation, heads, positions, blocks, nanoGPT-style construction, text training, KV cache and RoPE/FlashAttention. Exit: distinguish exact algebra, shape correctness and kernel performance.

• 036: Attention is a weighted sum with a precise mask
• 037: Multi-Head Attention — Shapes, Specialization and KV Budgets
• 038: Positional Encodings — Sinusoids, Learned Tables and RoPE
• 039: Transformer Blocks — Full Architectures and an Analytic Decoder
• 040: nanoGPT Rebuilt — A Complete Tested Decoder
• 041: Training a Decoder on Local Text — Tiny Shakespeare Recipe
• 042: KV caching: correct positions, masks, memory and reuse
• 043: RoPE and FlashAttention: relative positions and online softmax

Tokenization, data and scaling — sessions 044–049

BPE, SentencePiece, dataset provenance, scaling fits, domain mixtures and deduplication. Exit: freeze preprocessing and prevent leakage before optimizing a model.

• 044: Byte-pair encoding: merges, Unicode and artifact contracts
• 045: SentencePiece and unigram: lattices, normalization and compatibility
• 046: Pretraining data: auditable extraction, grouping and token shards
• 047: Chinchilla scaling: fit assumptions, budgets and serving cost
• 048: Data mixing: sampling, exposure, proxy ablations and curriculum
• 049: Quality and deduplication: MinHash, LSH and measured decisions

Pretraining systems — sessions 050–057

Architecture choices, DDP/FSDP, resource planning, a 100M experiment design, evaluation, checkpointing, diagnosis and scale accounting. Exit: make a budget and recovery contract without inventing throughput.

• 050: Architecture choices: derive and test a 100M decoder
• 051: Distributed Training Basics — DDP and FSDP
• 052: Renting Your First GPU — Runpod / Modal / Vast.ai Walkthrough
• 053: Training a 100M model: complete local recipe and budget
• 054: Loss curves and evaluation: likelihood, tasks and controls
• 055: Checkpointing and resuming: consistent state and durable publication
• 056: Pretraining bugs: hypotheses, counterexamples and tests
• 057: Scaling from 100M to 1B — A Mental Model

Adaptation and alignment — sessions 058–065

SFT, instruction data, LoRA, QLoRA, reward models, PPO, DPO and evaluation. Exit: derive losses and separate parameter efficiency from improved behavior.

• 058: Supervised fine-tuning: objective, templates and local recipes
• 059: Instruction datasets: provenance, normalization and mixtures
• 060: Adaptation with LoRA: low-rank updates and masked supervision
• 061: QLoRA: NF4 storage, gradients and local integration
• 062: Reward Modeling: Pairwise Likelihood, Data and Failure Modes
• 063: RLHF with PPO — Intuition + Minimal Code
• 064: DPO — RLHF without the RL
• 065: Evaluation and Red-Teaming: Protocols, Uncertainty and Safety

Efficient inference and serving — sessions 066–073

KV memory, quantization, pruning/distillation, speculation, batching, serving-engine internals, streaming APIs and latency. Exit: prove state correctness and specify a benchmark boundary.

• 066: KV Cache Deep Dive: Correctness, Memory and Serving Budgets
• 067: Quantization — INT8, INT4, GPTQ, AWQ
• 068: Pruning and Distillation: Exact Masks and Correct Token Losses
• 069: Speculative Decoding — Acceptance, Residual Sampling and Cost
• 070: Continuous Batching: Scheduling, Cache Admission and Capacity
• 071: Inside vLLM and TGI: Paging, Versioned Schedulers and Operations
• 072: Deploying an LLM API: Bounded FastAPI Streaming and Lifecycle
• 073: Latency vs Throughput — Little's Law, TTFT, ITL, and Load Testing

Multimodal, agents and capstone — sessions 074–080

CLIP/LLaVA, tool controllers, RAG, MoE, context extension, red-teaming and a broad reproducibility capstone. Exit: connect mechanisms to evidence and authorized execution.

• 074: Multimodal — CLIP → LLaVA
• 075: Agents and Tool Use
• 076: RAG as a testable evidence pipeline, not a truth switch
• 077: Mixture of Experts (MoE) Intuition
• 078: Long Context — RoPE Scaling, YaRN, ALiBi
• 079: Red-Team + Safety Eval
• 080: Capstone: a reproducible model-and-evidence laboratory

Advanced study and execution contract

Use Python/linear algebra prerequisites for the early modules; framework lessons then build on the same derivatives and state contracts. Execution is lesson-specific: the NumPy spine and selected NumPy/PyTorch examples include bounded CPU checks. Those checks do not imply TensorFlow, distributed collectives, rented GPUs, pretrained models, paid APIs, clinical use or deployment were run. Treat source-supported theory, pseudocode, dependent snippets and independently executed programs as distinct evidence categories.

No fixed duration, spend, accuracy or model capability is promised. Select a small experiment with a falsifiable acceptance condition, then estimate costs from current quotes and measured throughput. Larger capstones are optional staged work, not achievements conferred by reading a page. Preserve your failed cases as well as successes.

Classical ML and applied LLM branches

The separate classical ML sequence starts with the ML mental model, proceeds through linear/logistic models, regularization, trees, ensembles, evaluation and validation, and joins neural-network and LLM foundations. The applied LLM map connects retrieval, prompting, agents and evaluation. The Kimi architecture study is a pinned-source advanced case study, not a substitute for this progression.

FAQ

Do I need a GPU? Not for the verified NumPy spine. Some framework exercises can be reduced to CPU shape tests; full dataset or distributed experiments require separate resources and permission. Reading a conceptual GPU lesson does not require renting hardware.

How long does it take? Use exit criteria rather than a promised calendar. Spaced recall and changing one assumption in an example are more informative than time spent with a tab open.

Which examples can I run? Start with the tested CPU programs. Each lesson distinguishes runnable examples from pseudocode and references that need additional libraries, data or hardware. A successful syntax check does not mean a training run or deployment was performed.

Module deliverables and branch choices

Each module pairs a mechanism with a concrete artifact. These are study goals, not guaranteed benchmark scores or a fixed160-hour/cloud-price contract.

Pacing, recovery and next steps

A suggested study session is pre-read → hand calculation → code/assertions → perturbation → recall. Twenty minutes of reading,90 minutes of practice and10 minutes of recall were the original pacing example, not a universal completion time. One or two sessions a week can work; adjust to the exit tests and your resources. After a break, reconstruct the last worked example and one failure case before advancing. A4/5 recall score can be a personal cue, not proof of mastery.

Can I skip ahead? Follow prerequisites and return when an equation or state contract is unclear. What if calculus is rusty? Start with the derivative/chain-rule lesson and supplementary calculus practice; advanced expectations and continuous distributions may involve integrals even though many early labs do not. Which framework? The course emphasizes NumPy and PyTorch; that is a teaching choice, not a claim about every lab's framework. What comes next? Reimplement a bounded paper mechanism, contribute a test or documented fix to an open project, or design a domain application with a measured baseline.

Colophon and optional resources

The original course draws inspiration from Karpathy's Zero to Hero,3Blue1Brown's linear-algebra/neural-network series,Umar Jamil's implementation walkthroughs,Sebastian Raschka's from-scratch LLM book,StatQuest,Yannic Kilcher,Stanford CS224N/CS231N and Hugging Face tooling. Use them as complementary explanations; optional links may change, and these examples do not reproduce their full courses. The written80-session map remains self-contained as navigation even if you skip the videos.