Deep Learning Fundamentals — Backprop, CNNs, Optimizers
A deep-dive on Deep Learning Fundamentals — part of a 36-topic evergreen learning series.
Why this session matters
Part of a 36-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.
Session 12 · ML track 🧠
Before Transformers, you need the plumbing: backpropagation and the computation graph, why we need optimizers (SGD → Momentum → Adam → AdamW), CNNs (convolution, pooling, receptive fields), RNNs and their vanishing-gradient problem, and modern regularization (dropout, layer norm, batch norm). This is the foundation the entire LLM stack is built on.
Pre-read (30 min before session)
Watch 1–2 of these before the deep-dive:
- Andrej Karpathy: Neural Nets Zero to Hero
- 3Blue1Brown: Backpropagation
- CNNs Explained — DeepLearningAI
Then skim chapter 4 of Deep Learning (Goodfellow).
Deep-dive (90 min)
1. Backprop from scratch (30 min)
Forward pass → loss. Chain rule → gradient of loss w.r.t. every weight. Computational graph as a DAG. Autograd systems (PyTorch, JAX) just do reverse-mode differentiation over this graph. Concrete example: 2-layer MLP with sigmoid, manually derive gradients.
2. Optimizers (20 min)
SGD (stochastic gradient descent). Momentum (exponential moving average of grads, accelerates). RMSProp (per-parameter learning rate via grad² EMA). Adam (Momentum + RMSProp). AdamW (decouples weight decay from grad — matters for LLMs). Learning rate schedules: cosine, warmup, one-cycle.
3. CNNs (25 min)
Convolution operator. Kernel size, stride, padding, dilation. Pooling (max/avg). Receptive field grows with depth. Modern architectures: ResNet (skip connections solve vanishing grads), EfficientNet (compound scaling).
4. Regularization (15 min)
Dropout (random neuron masking during train). Batch norm vs layer norm (LN is standard in Transformers). Weight decay. Data augmentation. Early stopping.
Reading list
- Deep Learning — Goodfellow, Bengio, Courville (chapters 6-8)
- Andrej Karpathy — Neural Nets Zero to Hero playlist
- Chris Olah — Understanding LSTM Networks / distill.pub articles
Research links
Hands-on drill
Implement a 2-layer MLP in NumPy (no PyTorch). Manually code forward + backward pass for sigmoid activation. Train on XOR. Then port to PyTorch and confirm same loss curve.
Post-session checklist
- Can you explain backprop as chain rule over a DAG in 60 seconds?
- Can you explain why Adam usually converges faster than SGD in 60 seconds?
- Can you explain the vanishing gradient problem and how residual connections help in 60 seconds?
- Did you complete the hands-on drill above?
- Did you write 3 flashcards for tomorrow's recall?
- What's the one thing you'd want to revisit in the next revision session?
What's next
Session 13 continues the series. See the hub page for the full sequence and revision pattern.