S019 · Calculus I — Derivatives & Chain Rule

Module M02: Math Foundations · Session 19 of 130 · Track: Math 📉

What you'll be able to do after this session

• Explain Calculus I — Derivatives & Chain Rule to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S015 — Big-O Notation — Reasoning About Scale

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · 3Blue1Brown — The essence of calculus — the video that makes calculus feel inevitable.
• 🎥 Deep dive (30–60 min) · 3Blue1Brown — Derivative formulas through geometry + Chain rule — visual derivations you'll actually remember.
• 🎥 Hands-on demo (10–20 min) · Andrej Karpathy — Micrograd: Backprop from scratch — first 20 minutes: derivatives + chain rule in code.
• 📖 Canonical article · Khan Academy — Derivatives introduction — free, complete, self-paced.
• 📖 Book chapter / tutorial · Paul's Online Math Notes — Derivatives — the best free calculus reference on the internet.
• 📖 Engineering blog · Colah — Calculus on Computational Graphs: Backpropagation — the clearest explanation of chain rule → backprop, ever.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Rate of change is the heart of learning.

A derivative answers one question: if I nudge the input a tiny bit, how much does the output change? The speedometer in a car is a derivative — it tells you dposition/dtime, how fast the odometer is currently increasing. The slope of a hiking trail is daltitude/ddistance. The steepness of your electricity-bill curve is dcost/dkWh. Once you notice, derivatives are everywhere.

Why do we care in ML? Because "learning" = "reduce the error". To reduce the error you need to know: if I tweak this weight, does the error go up or down, and by how much? That's a derivative of the loss function with respect to each weight. Do that for millions of weights, take a small step in the downhill direction of every one, repeat — congratulations, you've just trained a neural network. The chain rule is what lets you compute those derivatives when your function is composed of many layers of other functions (which is exactly what a neural net is).

The forever-gotcha: the derivative at a point is local — it only tells you the slope right where you are. It doesn't guarantee the direction stays the same one step away. That's why gradient descent uses small steps: trust the local information only a little bit, then re-measure.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The derivative of f(x) at point x is the slope of the tangent line — the limit of (f(x+h) - f(x)) / h as h → 0.

Rules you must know cold (write them on a card and quiz yourself):

Worked example — the chain rule step by step:

Suppose y = (3x + 1)². Rewrite as a composition: outer f(u) = u², inner g(x) = 3x + 1.

• f'(u) = 2u, so f'(g(x)) = 2(3x + 1).
• g'(x) = 3.
• Chain rule: dy/dx = f'(g(x)) · g'(x) = 2(3x + 1) · 3 = 6(3x + 1) = 18x + 6.

Verify by expanding: y = 9x² + 6x + 1, so dy/dx = 18x + 6. ✓

Why "chain" is exactly right: think of a factory assembly line. Wiggle the raw material by 1 mm. The first station amplifies that wiggle by 3. The second station squares whatever comes in, so a small change is amplified by 2·(3x+1). Total amplification = product of amplifications = chain rule.

Neural-net view of chain rule: a network computes loss = L(y_pred, y_true) where y_pred = f₃(f₂(f₁(x))). To find how loss depends on the weights of f₁, you multiply the local derivatives layer by layer, right-to-left. That's backpropagation — nothing more, nothing less. The chain rule is backprop.

Numeric vs symbolic derivatives:

• Numeric: (f(x + 1e-5) - f(x)) / 1e-5. Easy, always works, but rounding errors bite fast.
• Symbolic: apply the rules above to derive an exact formula (SymPy, WolframAlpha).
• Automatic differentiation (autograd): what PyTorch/JAX do. Store the computation graph, apply chain rule mechanically for every parameter. Best of both worlds — exact, general, fast.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

"""Three views of a derivative: numeric, symbolic, and autograd."""
import numpy as np
 
# ---- 1. Numeric derivative (finite differences) --------------------
def numeric_derivative(f, x, h=1e-6):
    """Central difference is more accurate than forward difference."""
    return (f(x + h) - f(x - h)) / (2 * h)
 
f = lambda x: x**3 - 2*x + 1       # true f'(x) = 3x² - 2
for x in [-2, 0, 1, 2]:
    print(f"x={x:+d}  numeric f'(x) = {numeric_derivative(f, x):+.4f}   "
          f"true = {3*x**2 - 2:+d}")
 
# ---- 2. Symbolic derivative with sympy -----------------------------
try:
    import sympy as sp
    x = sp.symbols("x")
    expr = x**3 - 2*x + 1
    deriv = sp.diff(expr, x)
    print(f"\nsympy: d/dx({expr}) = {deriv}")
except ImportError:
    print("(pip install sympy to see symbolic diff)")
 
# ---- 3. Chain rule by hand vs autograd -----------------------------
# y = (3x + 1)^2   →   dy/dx = 18x + 6
# implement as a mini computation graph, compute gradient with chain rule
def forward_and_backward(x):
    # forward
    u = 3*x + 1          # inner
    y = u**2             # outer
    # backward (chain rule): dy/dx = dy/du * du/dx
    dy_du = 2*u
    du_dx = 3
    dy_dx = dy_du * du_dx
    return y, dy_dx
 
for x in [0, 1, 2]:
    y, g = forward_and_backward(x)
    print(f"x={x} → y={y}, dy/dx={g}  (expected {18*x + 6})")
 
# ---- 4. Real autograd with PyTorch (pip install torch) -------------
try:
    import torch
    x = torch.tensor(2.0, requires_grad=True)
    y = (3*x + 1)**2
    y.backward()                          # populates x.grad
    print(f"\ntorch: at x=2, y={y.item()}, dy/dx={x.grad.item()}  (expected 42)")
 
    # deeper composition
    x = torch.tensor(1.5, requires_grad=True)
    y = torch.sin(torch.exp(x**2))
    y.backward()
    # chain: dy/dx = cos(exp(x²)) * exp(x²) * 2x
    import math
    expected = math.cos(math.exp(x.item()**2)) * math.exp(x.item()**2) * 2*x.item()
    print(f"torch: complex fn dy/dx = {x.grad.item():.4f}  (expected {expected:.4f})")
except ImportError:
    print("(pip install torch to see autograd)")

Observe when you run:

1. Numeric derivatives match the analytical 3x² - 2 to ~6 decimal places.
2. Try h = 1e-12 — accuracy gets worse, not better. That's floating-point cancellation biting.
3. SymPy returns the exact symbolic form 3*x**2 - 2 — no numerical error at all.
4. The hand-coded backward pass and PyTorch autograd give identical answers — proving that autograd is literally the chain rule.
5. On the nested sin(exp(x²)) example, autograd handled three chain-rule applications automatically — you'd never want to do this by hand for a million-parameter network.

Try this modification: implement a two-input function f(x, y) = x² + 3xy + y³. Compute the partial derivatives ∂f/∂x and ∂f/∂y with PyTorch (call .backward() and read x.grad, y.grad). Verify against the symbolic answers 2x+3y and 3x+3y².

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Every deep-learning framework — PyTorch, JAX, TensorFlow — is fundamentally an autograd engine: a data structure that records every operation you perform on a tensor and then walks that graph backwards, applying the chain rule to compute gradients. Andrej Karpathy's micrograd (a Python file under 200 lines) implements exactly this and is the best way to internalise what PyTorch is doing.

The chain rule is what enables backpropagation, which is what enables all of deep learning. A GPT-4-scale model has ~1.7 trillion parameters — training it takes trillions of chain-rule applications per second, distributed across thousands of GPUs. Every single weight update is w ← w - η · ∂L/∂w — a derivative, computed by chain rule.

Where derivatives sneak into non-ML production:

• Numerical optimisation — SLSQP, L-BFGS, Adam are all gradient-based. scipy.optimize.minimize needs a gradient (or approximates one).
• Physics simulations — Navier-Stokes, Maxwell, PDE solvers all discretise derivatives.
• Signal processing — a Sobel edge filter is a discrete image derivative.
• Finance — Black-Scholes option pricing uses derivatives of price w.r.t. volatility ("the Greeks").

War stories:

• Vanishing/exploding gradients: in deep nets with sigmoid activations, chain-rule products of many small (or large) derivatives collapse to 0 or ∞. This is why ReLU, batch-norm, and residual connections exist — they keep gradient magnitudes stable across layers. Whole waves of architecture research (2010–2018) were about gradient flow.
• NaN in gradients: usually from log(0), sqrt(negative), or division by zero in the forward pass. Debug with torch.autograd.set_detect_anomaly(True). Once one gradient goes NaN, chain rule spreads it everywhere in one backward pass.
• Numeric-vs-analytic mismatch: if your custom autograd op has a bug, torch.autograd.gradcheck compares against finite differences. Every serious framework has this test in its CI.
• x**2 vs x*x: identical mathematically, but on GPUs x*x can be 2× faster because pow calls a library. Micro-optimisations at Google-brain scale add up to millions of dollars.

At Microsoft on the OneNote Copilot pipeline, our LLM-eval framework treats the aggregate quality score as a function of many knobs (prompt template, retrieval count, temperature). Every deployment tunes those knobs via gradient-free methods (Bayesian optimisation) precisely because we can't compute derivatives through an LLM API — a good reminder that "derivatives exist" is a privilege of white-box systems.

(e) Quiz + exercise · 10 min

1. In one sentence, what does the derivative f'(x) tell you about f at the point x?
2. State the power rule and the chain rule from memory.
3. Compute by hand: d/dx[(2x² + 1)³]. Show the chain-rule application.
4. Why do numeric derivatives (finite differences) get worse accuracy if you make h too small?
5. In one sentence each: what are the tradeoffs between numeric, symbolic, and automatic differentiation?

Stretch (connects to S018 Matrices): the derivative of a scalar function w.r.t. a vector is a gradient (also a vector). The derivative of a vector function w.r.t. a vector is a Jacobian matrix. For y = A·x where A is (3×2) and x is (2,), what is the Jacobian dy/dx? What are its dimensions?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Calculus I — Derivatives & Chain Rule? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S020): Calculus II — Gradients & Gradient Descent from Scratch
• Previous (S018): Linear Algebra II — Matrices, Transforms, Eigenvalues
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.