6-month learning plan · · session 19 / 13019 / 130

mathbeginner 55m read

S019 · Calculus I — Derivatives & Chain Rule

Rate of change is the heart of learning: the derivative as slope, symbolic vs numeric differentiation, and the chain rule that powers every neural net.

📐MathM02 · Math Foundations· Session 019 of 130 90 min

🎯 See a function, know its derivative, and unroll the chain rule for compositions of 3–4 functions — the exact skill backpropagation is built on.

Why this session exists

The universe of machine learning has exactly one universal algorithm: compute the gradient of a loss function and step in the opposite direction. Gradient. Derivative. Slope. If you don't feel derivatives — as the "how does the output wiggle when I wiggle the input" quantity — every ML explanation past this point sounds like magic. It isn't. It's calculus, and the version you need is small enough to fit on one page.

You will be able to

Explain the derivative as ‘instantaneous rate of change’ AND ‘slope of the tangent line’ AND ‘limit of a difference quotient’.
Compute derivatives of polynomials, exponentials, and logs by hand using the power/product/quotient rules.
Apply the chain rule to a composition of 3+ functions symbolically and verify with a numeric check.
Implement forward-mode numeric differentiation in 5 lines of Python and know its limitations.
Recognise the derivative-shaped things in ML: gradient (multi-var derivative), Jacobian (vector-valued), backprop (repeated chain rule).

Prerequisites

S017/S018 · Linear algebra — gradients are vectors; Jacobians are matrices.
High-school precalculus: polynomials, exponents, logs. That's it.

Pre-read · skim before the session (35 min total)

💡 Intuition 17 min

The essence of calculus · Chapter 1

3Blue1Brown

The best 17 minutes of calculus intuition ever recorded. Watch even if you took calc in college.

🔬 Deep dive 18 min

The paradox of the derivative · Chapter 2

3Blue1Brown

Why ‘instantaneous rate’ makes sense despite being the ratio of zero over zero.

🛠️ Hands-on 17 min

Visualizing the chain rule · Chapter 4

3Blue1Brown

Chain rule is 90% of backprop. This is the visualisation that makes it click.

📖 Reading 20 min

Paul's Online Notes · Derivatives

Paul Dawkins · Lamar Univ

The canonical undergraduate-level reference. Skim ‘Derivatives’ + ‘Chain Rule’ sections.

📖 Reading 15 min

CS231n · Backprop, Intuitions

Stanford CS231n

Karpathy's original notes — chain rule with computation graphs, exactly as backprop uses it.

📖 Reading 10 min

Automatic Differentiation

Wikipedia

How PyTorch/JAX actually compute gradients — forward vs reverse mode, in 10 min.

(a) Intuition · 5 min

A derivative is a car's speedometer

🌍 Real world

Your car's odometer tells you total distance travelled — that's the function f(t). Your speedometer tells you how fast that number is changing right this instant — that's the derivative f'(t). If you know your speed at every moment, you can reconstruct where you'll be. That's integration, tomorrow's story.

Now imagine a hill. The steeper the hill, the faster you gain elevation per step forward. That "elevation per step" IS the derivative of the elevation function. Positive: uphill. Negative: downhill. Zero: flat — the top of a peak or bottom of a valley. That's it. Rate of change.

💻 Code world

In code, if y = f(x), the derivative f'(x) tells you: "if I wiggle x by a tiny ε, y wiggles by roughly f'(x)·ε." So if you want y to go DOWN, wiggle x by minus ε·f'(x). Do that repeatedly and you slide downhill in y — you're doing gradient descent, which is 100% of how neural nets learn.

That "how much does the output change per unit change in the input" number is the single most-computed quantity in ML. Every framework spends most of its cycles computing it.

The three equivalent definitions

A derivative is all three of these

Geometric — the slope of the tangent line to y = f(x) at the point (x, f(x)).
Numeric — the limit of (f(x + h) − f(x)) / h as h → 0.
Physical — the instantaneous rate of change of f with respect to x. If f is position, f' is velocity.

A quick history

1665
Newton invents fluxions · Woolsthorpe
Isaac Newton (age 22, isolated by plague) invents calculus to explain planetary motion.
1684
Leibniz publishes derivative notation
Gottfried Leibniz independently discovers calculus. His notation (dy/dx) wins; Newton is furious.
1821
Cauchy formalises limits
Rigorous ε-δ foundations. Calculus stops being ‘magic that works’ and becomes a theorem.
1974
Reverse-mode autodiff · Speelpenning
Paul Werbos (1974) + Bert Speelpenning independently discover what we now call backpropagation.
1986
Backprop popularised · Rumelhart, Hinton, Williams
The Nature paper. Kicks off the connectionist wave. Every deep-learning framework computes chain rule this way.
2018+
JAX · autograd everywhere
Google's JAX makes autodiff a first-class Python transform. PyTorch's autograd, TensorFlow's GradientTape follow.

(b) Visual walkthrough · 15 min

The derivative as a limit — the picture

The rules you actually need

The derivative rulebook (memorise these five)

Power rule: d/dx (xⁿ) = n · xⁿ⁻¹

d/dx(x²) = 2x. d/dx(x⁵) = 5x⁴. d/dx(√x) = 0.5·x⁻⁰·⁵. Covers 80% of school calculus.

power

Exp / log: d/dx(eˣ) = eˣ, d/dx(ln x) = 1/x

eˣ is its own derivative — that's the entire reason it's the natural base. ln is the inverse of exp; its derivative is 1/x.

exp

Trig: d/dx(sin x) = cos x, d/dx(cos x) = -sin x

Cyclic — differentiate four times and you're back where you started. Sign flip on cos.

trig

Sum rule: (f + g)' = f' + g'

Differentiation is linear. Break sums into pieces.

linearity

Product rule: (fg)' = f'g + fg'

Not (fg)' = f'g'. Every calc student gets this wrong once. Memorise it as ‘first-prime-second-plus-first-second-prime’.

product

Chain rule — the star of the show

Identify the composition

y = f(g(x)) means: put x through g first to get u = g(x), then put u through f to get y = f(u).

Differentiate the OUTER function at the INNER value

dy/du = f'(u), evaluated at u = g(x). This is the ‘outer’ contribution.

Multiply by the derivative of the inner

du/dx = g'(x). This is how much u wiggles when x wiggles.

Chain them: dy/dx = f'(g(x)) · g'(x)

The famous formula. For deeper nesting, chain more terms — that's backprop.

Example: y = sin(x²). Outer = sin, inner = x². Then y' = cos(x²) · 2x.

Symbolic vs numeric vs automatic differentiation

Symbolic (SymPy)

Exact algebra

Returns a formula, not a number
Slow — expression trees blow up for deep composition
Great for teaching + verifying
Rarely used in production ML

Numeric (finite differences)

Approximate, uses only function evaluations

(f(x+h) - f(x)) / h with tiny h
Simple, universal — works on ANY function
Precision limited by float epsilon
O(n) function calls for n-variable gradient — too slow for ML

Automatic (PyTorch/JAX)

Exact, reverse-mode

Build a computation graph on the forward pass
Backprop: apply chain rule once per node
Same speed as forward — one gradient for millions of parameters
This is how EVERY modern ML framework does it

Common misconception

✗ What most people think

"A derivative is the slope of a curve — a geometry fact about graphs. Useful in calculus class, but in ML it's just something the autograd library computes for me."

✓ What is actually true

A derivative is a local linear approximation: it answers "if I nudge the input by a tiny amount, how much does the output move, and in which direction?" That reading is what makes it the engine of optimisation, sensitivity analysis, and error propagation — the slope picture is one special case in one dimension.

Why the myth is so sticky

Because the tangent-line image is genuinely correct and it is drawn on the first day, so it becomes the whole concept. It is also sufficient for every exercise you are asked to do by hand. It quietly stops helping the moment there is more than one input: with a million parameters there is no curve to look at, and "slope" has no meaning until you specify a direction. Meanwhile the nudge reading generalises immediately — each partial derivative is "how much does the loss move if I nudge this one parameter, holding the rest fixed", which is exactly what backpropagation computes and exactly what an optimiser consumes. The geometric picture also hides the property that matters most in practice: the approximation is only valid locally, which is the entire reason learning rates exist.

Prove it to yourself

The derivative predicts the nudge — and you can watch the prediction degrade as the step grows:

f  = lambda x: x**3
df = lambda x: 3*x**2
x  = 2.0
for h in [1e-1, 1e-3, 1e-5]:
    actual    = f(x + h) - f(x)
    predicted = df(x) * h
    print(h, actual, predicted, abs(actual - predicted))
# error shrinks like h^2 - the linear model is exact only in the limit

From first principles

Start with the question

Why does the chain rule multiply derivatives? And why does that single fact explain both vanishing gradients and why deep networks were hard to train before residual connections?

1
Near a point, a differentiable function is well approximated by a linear one: a nudge δ in the input produces roughly f'(x)·δ in the output.
forced by · that is the definition of the derivative — the best local linear fit
2
Composing functions means feeding one's output into the next, so the second function receives the first's nudge as its input nudge.
forced by · composition is exactly "the output of g becomes the input of f"
3
Applying the linear approximation twice: an input nudge δ becomes g'(x)·δ, which becomes f'(g(x))·g'(x)·δ.
forced by · each layer applies its own local scaling factor to whatever arrives
4
Therefore the sensitivity of a composition is the product of the per-stage sensitivities — the chain rule is scaling factors accumulating multiplicatively through the chain.
forced by · applying one scale factor after another multiplies them; this is the only thing composition of linear maps can do
5
In a network of L layers, the gradient reaching layer 1 is a product of L factors. If those factors average below 1, the product decays exponentially in L; above 1, it explodes exponentially.
forced by · a product of L numbers is governed by their geometric mean raised to the power L

⇒ Therefore

Therefore vanishing and exploding gradients are not mysterious pathologies of deep learning — they are what multiplication does over long chains, and they were predictable from the chain rule alone.

And note precisely what this predicts, all of which is historically what happened: saturating activations like sigmoid have derivatives bounded well below 1, so deep sigmoid stacks must vanish — hence ReLU, whose derivative is exactly 1 on the active side. It predicts that gradient clipping addresses only the exploding half. And it predicts why a residual connection helps so decisively: y = x + F(x) has derivative 1 + F'(x), so the product always carries a path of 1s through the chain, and the gradient reaching early layers no longer decays with depth. Batch normalisation attacks the same product from the other side by keeping per-layer scales near 1.

Mental modelThe nudge test

For any quantity that depends on any input, ask: if I move this input a little, how much does the output move? That number is the derivative, and its sign tells you which way to move to increase the output. That is the whole of differential calculus as applied to engineering.

In many dimensions, collect one nudge answer per input and you have the gradient — a vector pointing in the direction of steepest increase. Optimisation is then just "compute the nudge answers, step the opposite way, repeat", and the only remaining question is how big a step the local approximation can still be trusted for.

The linear approximation is only valid locally. Step too far and the prediction is wrong — this is exactly what a too-large learning rate is, and why loss diverges rather than merely converging slowly.
Derivative zero means flat, which could be a minimum, a maximum, or a saddle. In high dimensions saddles vastly outnumber true minima, which is why plain gradient descent stalls and why momentum exists.
Non-differentiable points are real and matter: ReLU at 0, absolute value, hard thresholds. Frameworks pick a subgradient by convention, which is why L1 regularisation drives coefficients exactly to zero while L2 only shrinks them.
The chain rule composes multiplicatively, so anything that keeps per-stage factors near 1 — residual paths, normalisation, careful initialisation — is the mechanism that makes depth trainable.

🔔 Fires when you see

Fire this model the moment you see: a loss that diverges to NaN · early layers that barely change during training · a sensitivity or "what-if" question · error bars propagating through a calculation · a hyperparameter you are tuning by hand · requires_grad or .backward().

The tradeoff

You need derivatives of a function in code. Hand-derive them, use finite differences, or use automatic differentiation?

Analytic (hand-derived)

+ you gain exact, fastest at runtime, and no framework dependency; deriving it forces you to understand the function's structure, which frequently reveals simplifications and stability fixes

− you pay error-prone and doesn't scale past small expressions; every change to the forward function silently invalidates the gradient, and a wrong gradient trains quietly to the wrong answer

pick when the function is small and stable, or you need a numerically stable closed form — the reason softmax-cross-entropy is implemented as a fused expression rather than composed naively

Finite differences

+ you gain works on any black box with no access to internals; trivial to implement in a few lines; the standard tool for checking a gradient you got another way

− you pay one or two extra function evaluations per parameter, so it's hopeless above a few dozen; and it faces an unavoidable step-size dilemma — too large gives truncation error, too small gives catastrophic cancellation in floating point

pick when verifying an analytic or custom gradient, or optimising a black box with few parameters where no derivative is available at all

Automatic differentiation

+ you gain exact to machine precision (it applies the chain rule mechanically, it does not approximate), scales to millions of parameters, and reverse mode gets all partials for roughly the cost of one forward pass

− you pay memory to store the forward activations for the backward pass, framework lock-in, and gradients that silently break at detach points, in-place operations, or control flow the tracer didn't capture

pick when any model with more than a handful of parameters — which is why every deep learning framework is fundamentally an autodiff engine with layers attached

What a senior engineer actually does

Use autodiff, and use finite differences to check it whenever you write a custom operation. Gradient checking is cheap insurance against the worst class of ML bug: a wrong gradient produces no error, no warning, and a model that trains to a plausible-looking but incorrect solution.

The asymmetry worth internalising is why reverse mode dominates: forward mode costs one pass per input, reverse mode costs one pass per output. Training has millions of inputs and a single scalar loss, so reverse mode is the obvious winner — and that asymmetry is the entire reason backpropagation, rather than any other differentiation scheme, is what made deep learning computationally feasible.

(c) Hands-on · 25 min

You'll compute derivatives three ways — by hand, numerically, and with SymPy — and verify they agree. Then you'll build a tiny scalar autodiff engine to see how PyTorch works under the hood. Save as calculus_demo.py.

Run it:

uv run --with numpy python calculus_demo.py

Expected output:

What each block does

Anatomy of the code

numeric_deriv · central difference

Uses (f(x+h) − f(x−h)) / (2h) instead of forward difference. Accurate to O(h²) vs O(h) — cancels the leading error term.

numeric

demo_numeric · sanity table

Six functions, three x-values each. Errors should be ~1e-10 or better. If your rule is wrong, this catches it in one row.

verify

demo_chain_rule · by hand + check

Write out the chain rule manually, then verify numerically. This is how researchers double-check gradient math before deploying to PyTorch.

chain

Value class · reverse-mode autodiff

Every arithmetic op returns a new Value that remembers its parents and a local backward function. That's the whole trick.

engine

backward() · topo sort + reverse pass

Build the computation graph in forward order via depth-first traversal, then apply _backward in reverse. This is exactly what PyTorch does internally.

backprop

_backward closures

Each op has a rule for propagating gradients to its inputs: for +, `parent.grad += out.grad`; for ×, use the other input; for x^n, `n · x^(n-1) · out.grad`; for sin, cos(x). Chain rule made mechanical.

rules

Try itFeel why the chain rule matters — build a 3-layer computation

Add a deeper computation and check the gradient:

# y = ((x**2 + 3)*x).sin()  at x = 1.2
x = Value(1.2)
b = x**2 + Value(3)
c = b * x
y = c.sin()
y.backward()
 
# Chain rule by hand: dy/dx = cos(c) · dc/dx
# dc/dx = db/dx * x + b * 1 = 2x * x + b = 2x^2 + (x^2 + 3) = 3x^2 + 3
# So dy/dx = cos((x^2 + 3)*x) * (3x^2 + 3)
import numpy as np
manual = np.cos((1.2**2 + 3) * 1.2) * (3 * 1.2**2 + 3)
print(f"autodiff = {x.grad:.6f}  manual = {manual:.6f}")

Now change Value(3) to Value(10) and predict how x.grad will change before running.

💡 Hint · After running, hand-derive the same gradient using the chain rule three times: dy/dx = dy/dc · dc/db · db/dx. Check that all three factors multiply to what autodiff returned. This is exactly what a deep neural network does per layer.

(d) Production reality · 15 min

War story Andrej Karpathy · micrograd + Tesla Autopilot· 2022OSS teaching tool + billions of gradient calls

🔥 What broke

Every deep-learning engineer eventually meets the ‘I don't really understand what autograd does’ moment — usually when a gradient explodes, a NaN appears mid-training, or a custom layer's gradient is wrong.

Copy-pasting optimizer.zero_grad(); loss.backward(); optimizer.step() without understanding the internals means you can't debug when it breaks.

🧯 The fix

Karpathy's micrograd (100 lines of Python) reimplements the essence of PyTorch's autograd. Once you've read it, PyTorch's C++ source stops being magic. Every ML engineer should read it once. The 45-min ‘Spelled-out intro to neural networks and backpropagation’ video is the best teaching artifact on the subject.

🎓 Lesson to steal

Autograd feels like magic until you build a toy version. Then it's just topological sort + chain rule + closures. Do the 100-line exercise; it pays back every time you debug a gradient.

Post-mortem

War story Common failure · finite-difference gradient in MLresearch-grade slow-down

🔥 What broke

A student implements a custom optimiser and, ‘to be safe’, computes gradients by finite differences instead of autograd. Training that should take 1 hour takes 3 days. For a model with 10M parameters, each optimiser step requires 10M forward passes.

🧯 The fix

Use reverse-mode autograd. For an ℝⁿ → ℝ loss, reverse-mode gives all n partials in ONE backward pass — ~2× the cost of the forward, independent of n. Finite differences would be O(n) forwards. For an even bigger model, the ratio is millions×.

🎓 Lesson to steal

Use reverse-mode autograd for training. Use finite differences ONLY to spot-check gradients (‘gradcheck’ in PyTorch), never as your production path.

War story Common failure · NaN in the gradientevery ML engineer, once

🔥 What broke

A model trains fine for 3 hours, then loss jumps to nan and stays there forever. The culprit: log(0) hidden in a softmax cross-entropy calculation on a batch where one class had probability zero due to floating-point underflow.

🧯 The fix

Use log-sum-exp-stabilised versions of softmax/cross-entropy (built into PyTorch as F.cross_entropy). Add gradient clipping (torch.nn.utils.clip_grad_norm_). Check for NaN with torch.isnan(loss).any() and abort the epoch. Learn to spot the ‘loss is a nice number, gradient is NaN’ pattern early.

🎓 Lesson to steal

Gradients propagate NaN like wildfire — one bad number contaminates everything downstream. Prevention is 100× cheaper than debugging: use library implementations, clip norms, monitor NaNs.

Where this shows up in the rest of the plan

Derivatives are the currency of learning

S020 · Gradient descent from scratch

You'll implement SGD using this session's autodiff engine.

S078 · Regression models

Linear/logistic regression fit by taking derivatives of the loss w.r.t. weights.

S083 · Neural network foundations

Backprop is chain rule applied through every layer, from output back to input.

S090 · Optimisers (Adam, RMSprop, AdamW)

All variants of ‘step in the direction of the gradient, with momentum + adaptive scaling’.

S099 · Reinforcement learning (policy gradients)

Policy gradient IS the derivative of expected reward with respect to policy parameters.

S115 · Transformer training

Softmax + cross-entropy + Adam + autograd, at billion-parameter scale.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal

★ = stretch question

Explain-out-loud test

What does a derivative measure, in one sentence?
State the chain rule and give a 3-line example.
Why does every modern ML framework use reverse-mode autograd?

What comes next

📐MathNext →

S020 · Calculus II — Gradients & Gradient Descent from Scratch

Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.