Calculus I — Derivatives & Chain Rule

Review boundary and corrected contracts

Chain rule applies local sensitivities; products of Jacobians are not guaranteed stable by residual connections. Central differences have O(h²) truncation error under sufficient smoothness but roundoff grows for tiny h. The scalar autodiff graph now resets gradients per backward call and handles exponent zero safely.

The examples below teach the stated input contracts; fragments and deliberately incorrect counterexamples are exercises, not production implementations. Complexity counts elementary operations unless a bit-cost or allocation caveat is stated. Practice times are suggestions, not measured performance or hiring guarantees.

Learning objective: See a function, know its derivative, and unroll the chain rule for compositions of 3–4 functions — the exact skill backpropagation is built on.

Why this session exists

Many trainable differentiable models use gradients of a loss to guide parameter updates. This is not universal to all machine learning—trees and nearest-neighbour methods need no such update. Gradient. Derivative. Slope. If you don't feel derivatives — as the "how does the output wiggle when I wiggle the input" quantity — every ML explanation past this point sounds like magic. It isn't. It's calculus, and the version you need is small enough to fit on one page.

You will be able to

• Explain the derivative as ‘instantaneous rate of change’ AND ‘slope of the tangent line’ AND ‘limit of a difference quotient’.

• Compute derivatives of polynomials, exponentials, and logs by hand using the power/product/quotient rules.

• Apply the chain rule to a composition of 3+ functions symbolically and verify with a numeric check.

• Implement finite-difference numerical differentiation in a few lines of Python and distinguish it from forward-mode automatic differentiation.

• Recognise the derivative-shaped things in ML: gradient (multi-var derivative), Jacobian (vector-valued), backprop (repeated chain rule).

Prerequisites

• S017/S018 · Linear algebra — gradients are vectors; Jacobians are matrices.
• High-school precalculus: polynomials, exponents, logs. That's it.

Optional archived videos

These original learning links are retained as optional resources. Video identity, availability and duration were not reverified; the lesson and governing sources below stand independently of them.

• The essence of calculus · Chapter 1
• The paradox of the derivative · Chapter 2
• Visualizing the chain rule · Chapter 4

Pre-read · skim before the session (35 min total)

Further reading: Paul's Online Notes · Derivatives

Further reading: CS231n · Backprop, Intuitions

Further reading: Automatic Differentiation

(a) Intuition · 5 min

A derivative is a car's speedometer

Real: Your car's odometer tells you total distance travelled — that's the function f(t). Your speedometer tells you how fast that number is changing right this instant — that's the derivative f'(t). Given an initial odometer reading, integrating speed reconstructs distance travelled; position additionally requires velocity/direction. That's integration, tomorrow's story. Now imagine a hill. The steeper the hill, the faster you gain elevation per step forward. That "elevation per step" IS the derivative of the elevation function. Positive: uphill. Negative: downhill. Zero: a horizontal tangent, possibly a peak, valley or neither (x³ at zero). That's it. Rate of change.

 
    In code, if y = f(x), the derivative f'(x) tells you: "if I wiggle x by a tiny ε, y wiggles by roughly f'(x)·ε." So if you want y to go DOWN, wiggle x by minus ε·f'(x). For a nonzero gradient and a sufficiently small positive step, a differentiable function decreases locally along the negative gradient. Repeated updates need suitable step control; a large step can increase the loss.
    That "how much does the output change per unit change in the input" number is a central quantity in differentiable ML. Its computational cost depends on the model and differentiation implementation.
  

The three equivalent definitions

A derivative is all three of these

• Geometric — the slope of the tangent line to y = f(x) at the point (x, f(x)).

• Numeric — the limit of (f(x + h) − f(x)) / h as h → 0.

• Physical — the instantaneous rate of change of f with respect to x. If f is position, f' is velocity.

The single most-abused notation

People write dy/dx, f'(x), ∂f/∂x, D_x f, and ∇f — sometimes all in one paper. They are related but not identical. f'(x) = single-variable derivative. ∂f/∂x = partial derivative (multi-variable, holding others constant). ∇f = gradient (the vector of all partials). Know which one is on the page and you'll never be confused.

A quick history

Newton and Leibniz developed calculus through different notations and motivations; later work on limits sharpened its foundations. The St Andrews calculus history gives context without reducing the story to one discovery date. Reverse-mode differentiation predates today’s neural-network frameworks; Werbos, Speelpenning and the 1986 Rumelhart–Hinton–Williams work belong to distinct contributions, not a single “Speelpenning 1974” event. This lesson makes no priority claim for that archived date or for the claim that PyTorch followed JAX. The operative connection is inspectable today: micrograd’s implementation records local derivative closures and walks the graph backward; JAX’s documentation distinguishes forward and reverse transformations.

(b) From secant slopes to the chain rule · 15 min

Modern frameworks compute derivatives automatically; the missing link is how a slope becomes a reusable calculation. Start with the limit behind a single derivative, then combine derivative rules to follow a change through nested functions.

The derivative as a limit — the picture

Hold the first point on the curve fixed and move the second point towards it. Read the diagram as a sequence of slope measurements: each nonzero separation gives an average rate of change, and the limit asks whether those measurements approach one value.

The last box is the limiting slope, not a calculation made by setting the separation to zero and dividing. At a differentiable point, the secant slopes approach the tangent slope. The rules below package the results of that limiting process for common functions, so you need not reconstruct it for every derivative.

The rules you actually need

The derivative rulebook, with domains

Work over the reals at an interior point of the function’s domain. Differentiability is an assumption, not something a function name guarantees. Polynomial powers hold for every real x; negative integer powers exclude zero; general real powers can be handled on x > 0. ln x needs x > 0, and the displayed square-root derivative needs x > 0. Trigonometric derivatives here use radians.

Name: Power rule: d/dx (xⁿ) = n · xⁿ⁻¹

Desc: d/dx(x²) = 2x. d/dx(x⁵) = 5x⁴. d/dx(√x) = 0.5·x⁻⁰·⁵. Covers 80% of school calculus.

Name: Exp / log: d/dx(eˣ) = eˣ, d/dx(ln x) = 1/x

Desc: eˣ is its own derivative — that's the entire reason it's the natural base. ln is the inverse of exp; its derivative is 1/x.

Name: Trig: d/dx(sin x) = cos x, d/dx(cos x) = -sin x

Desc: Cyclic — differentiate four times and you're back where you started. Sign flip on cos.

Name: Sum rule: (f + g)' = f' + g'

Desc: Differentiation is linear. Break sums into pieces.

Name: Product rule: (fg)' = f'g + fg'

Desc: Not (fg)' = f'g'. Every calc student gets this wrong once. Memorise it as ‘first-prime-second-plus-first-second-prime’.

Name: Quotient rule: (f/g)' = (f'g − fg')/g², where g ≠ 0

Desc: Apply the product and chain rules to f·g⁻¹. For y = (x²+1)/(x−1), x ≠ 1, the derivative is [2x(x−1) − (x²+1)]/(x−1)² = (x²−2x−1)/(x−1)². At x = 2 it is −1. A product example is (x²·eˣ)' = eˣ(2x+x²), not 2x·eˣ alone.

Chain rule — the star of the show

The rulebook handles familiar functions and their sums or products. Nesting is a different operation: a change must pass through the inner function before it reaches the outer one. Follow that intermediate change through the steps below; it explains why both derivatives contribute.

Identify the composition

Desc: y = f(g(x)) means: put x through g first to get u = g(x), then put u through f to get y = f(u).

Differentiate the OUTER function at the INNER value

Desc: dy/du = f'(u), evaluated at u = g(x). This is the ‘outer’ contribution.

Multiply by the derivative of the inner

Desc: du/dx = g'(x). This is how much u wiggles when x wiggles.

Chain them: dy/dx = f'(g(x)) · g'(x)

Desc: The famous formula. For deeper nesting, chain more terms — that's backprop.

Example: y = sin(x²). Outer = sin, inner = x². Then y' = cos(x²) · 2x.

The chain rule requires g differentiable at x and f differentiable at g(x). For three layers, y = exp(sin(x²)) has derivative exp(sin(x²))·cos(x²)·2x. At x = 0 this is 0; no division by an intermediate derivative is needed. For F: ℝⁿ → ℝᵐ, the Jacobian J has shape (m,n), with J[i,j] = ∂Fᵢ/∂xⱼ. If G: ℝᵐ → ℝᵏ, then J of G∘F is J_G(F(x)) J_F(x), shape (k,n). A scalar loss has a gradient with n coordinates; in Euclidean geometry its direction is steepest ascent when it is nonzero.

Symbolic vs numeric vs automatic differentiation

That nested sine example also gives us a way to compare implementations. We can derive a formula, estimate the slope from nearby function values, or record the elementary operations and propagate their local derivatives. The comparison below separates those three routes to the same derivative.

Name: Symbolic (SymPy)

Tagline: Exact algebra

• Returns a formula, not a number

• Slow — expression trees blow up for deep composition

• Great for teaching + verifying

• Rarely used in production ML

Name: Numeric (finite differences)

Tagline: Approximate, uses only function evaluations

• (f(x+h) - f(x)) / h with tiny h

• Needs evaluable nearby inputs and a derivative to approximate; discontinuities, noise or domain boundaries can invalidate it

• Accuracy balances truncation, cancellation/roundoff, function scale and noise

• O(n) function calls for n-variable gradient — too slow for ML

Name: Automatic (PyTorch/JAX)

Tagline: Chain rule through implemented operations

• Build a computation graph on the forward pass

• Backprop: apply chain rule once per node

• Reverse-mode obtains a scalar loss gradient in one reverse sweep, with operation-dependent overhead and saved intermediates

• Frameworks may provide both reverse-mode and forward-mode transformations

Why reverse-mode wins for ML

For a differentiable program ℝⁿ → ℝ, one reverse sweep computes the vector–Jacobian product seeded by 1, hence all n partials. Forward mode computes a Jacobian–vector product per tangent seed; assembling the full gradient from basis seeds needs n such directions. Reverse-mode overhead depends on the primitive operations and saved/recomputed intermediates, not a universal 2–3× factor. For a few inputs and many outputs, forward mode can be preferable.

Myth: "A derivative is the slope of a curve — a geometry fact about graphs. Useful in calculus class, but in ML it's just something the autograd library computes for me."

Truth: A derivative is a local linear approximation: it answers "if I nudge the input by a tiny amount, how much does the output move, and in which direction?" That reading is what makes it the engine of optimisation, sensitivity analysis, and error propagation — the slope picture is one special case in one dimension.

Why: Because the tangent-line image is genuinely correct and it is drawn on the first day, so it becomes the whole concept. It is also sufficient for every exercise you are asked to do by hand. It quietly stops helping the moment there is more than one input: with a million parameters there is no curve to look at, and "slope" has no meaning until you specify a direction. Meanwhile the nudge reading generalises immediately — each partial derivative is "how much does the loss move if I nudge this one parameter, holding the rest fixed", which is exactly what backpropagation computes and exactly what an optimiser consumes. The geometric picture also hides the property that matters most in practice: the approximation is only valid locally, which is the entire reason learning rates exist.

Test: Watch a local linear approximation improve as the step shrinks:

f  = lambda x: x**3
df = lambda x: 3*x**2
x  = 2.0
for h in [1e-1, 1e-3, 1e-5]:
    actual    = f(x + h) - f(x)
    predicted = df(x) * h
    print(h, actual, predicted, abs(actual - predicted))
# error shrinks like h^2 - the linear model is exact only in the limit

In exact arithmetic at x = 2, (2+h)³ − 8 − 12h = 6h²+h³. That algebra establishes the O(h²) local remainder for this polynomial; finite floating-point samples merely illustrate it. Question: Why does the chain rule multiply derivatives? And why does that single fact explain both vanishing gradients and why deep networks were hard to train before residual connections?

Premise: Near a point, a differentiable function is well approximated by a linear one: a nudge δ in the input produces roughly f'(x)·δ in the output.

Because: that is the definition of the derivative — the best local linear fit

Premise: Composing functions means feeding one's output into the next, so the second function receives the first's nudge as its input nudge.

Because: composition is exactly "the output of g becomes the input of f"

Premise: Applying the linear approximation twice: an input nudge δ becomes g'(x)·δ, which becomes f'(g(x))·g'(x)·δ.

Because: each layer applies its own local scaling factor to whatever arrives

Premise: Therefore the sensitivity of a composition is the product of the per-stage sensitivities — the chain rule is scaling factors accumulating multiplicatively through the chain.

Because: applying one scale factor after another multiplies them; this is the only thing composition of linear maps can do

Premise: In a scalar chain, the magnitude is the product of absolute local derivatives. A uniform bound |factor| ≤ c < 1 implies a bound cᴸ; a bound |factor| ≥ c > 1 gives growth at least cᴸ. A signed arithmetic average is insufficient. For nonnegative magnitudes, a uniform arithmetic-mean bound below 1 does upper-bound the product by AM–GM, but an arithmetic mean above 1 alone does not force growth (a zero factor can kill the product). In vector networks, ordered Jacobian products can rotate, amplify or suppress different directions.

Because: for nonzero scalar magnitudes the product is the geometric mean raised to L; matrix behaviour additionally depends on directions, not just an average scalar

Sigmoid derivatives are bounded by 1/4, but layer Jacobians also contain weights. Products of these Jacobians can shrink or grow depending on their singular values and activations. Residual connections provide an identity contribution, not a guarantee against cancellation, vanishing or explosion.

Name: The nudge test

Picture: For any quantity that depends on any input, ask: if I move this input a little, how much does the output move? That number is the derivative, and its sign tells you which way to move to increase the output. That is the whole of differential calculus as applied to engineering.In many dimensions, collect one nudge answer per input and you have the gradient — a vector pointing in the direction of steepest increase. Optimisation is then just "compute the nudge answers, step the opposite way, repeat", and the only remaining question is how big a step the local approximation can still be trusted for.

• The linear approximation is only valid locally. Step too far and the prediction is wrong — this is exactly what a too-large learning rate is, and why loss diverges rather than merely converging slowly.

• Derivative zero means flat, which could be a minimum, a maximum, or a saddle. There is no universal count comparison for arbitrary objectives. At a stationary point, first-order information alone cannot classify the point.

• Non-differentiable points are real and matter: ReLU at 0, absolute value, hard thresholds. At convex kinks such as |x|, a framework may choose a subgradient convention. A hard threshold is discontinuous and has no ordinary derivative at its jump. Exact sparsity from L1 depends on the optimization method (for example a proximal soft-thresholding step), not merely a convention for the gradient at zero.

• Residual paths add an identity Jacobian: for x + F(x), J = I + J_F. If J_F = −I it cancels to zero; if J_F = I it doubles. Initialization, normalization and residual design can help conditioning but guarantee neither stable gradients nor trainability.

Trigger: Fire this model the moment you see: a loss that diverges to NaN · early layers that barely change during training · a sensitivity or "what-if" question · error bars propagating through a calculation · a hyperparameter you are tuning by hand · requires_grad or .backward().

Decision: You need derivatives of a function in code. Hand-derive them, use finite differences, or use automatic differentiation?

Name: Analytic (hand-derived)

Gain: an analytic formula with no required autodiff framework dependency; runtime speed and numerical stability depend on the formula; deriving it forces you to understand the function's structure, which frequently reveals simplifications and stability fixes

Cost: error-prone and doesn't scale past small expressions; every change to the forward function silently invalidates the gradient, and a wrong gradient trains quietly to the wrong answer

Pick: the function is small and stable, or you need a numerically stable closed form — the reason softmax-cross-entropy is implemented as a fused expression rather than composed naively

Name: Finite differences

Gain: uses only black-box evaluations when nearby domain-valid values exist and the function is sufficiently smooth; trivial to implement in a few lines; the standard tool for checking a gradient you got another way

Cost: one or two extra function evaluations per parameter, so full coordinate gradients can be expensive for large n; and it faces an unavoidable step-size dilemma — too large gives truncation error, too small gives catastrophic cancellation in floating point

Pick: verifying an analytic or custom gradient, or optimising a black box with few parameters where no derivative is available at all

Name: Automatic differentiation

Gain: avoids finite-difference truncation by applying local derivative rules, but floating-point arithmetic, unstable formulas and nondifferentiable operations still matter. Reverse mode shares work across all scalar-loss partials

Cost: memory to store the forward activations for the backward pass, framework lock-in, and gradients that silently break at detach points, in-place operations, or control flow the tracer didn't capture

Pick: any model with more than a handful of parameters — which is why every deep learning framework is fundamentally an autodiff engine with layers attached

Verdict: Use autodiff, and use finite differences to check it whenever you write a custom operation. Gradient checking is cheap insurance against the worst class of ML bug: a wrong gradient produces no error, no warning, and a model that trains to a plausible-looking but incorrect solution.The asymmetry worth internalising is why reverse mode dominates: forward mode costs one pass per input, reverse mode costs one pass per output. Training has millions of inputs and a single scalar loss, so reverse mode is the obvious winner — and that asymmetry is the entire reason backpropagation, rather than any other differentiation scheme, is what made deep learning computationally feasible.

(c) Hands-on · 25 min

You'll compute derivatives three ways—by hand, by finite differences, and with a small reverse-mode engine—and compare their results. No SymPy dependency is used. Then you'll build a tiny scalar autodiff engine to see how PyTorch works under the hood. Save as calculus_demo.py.

Contract: finite real values and supported differentiable operations only. Value is a scalar teaching graph, not a tensor engine. Keep .data unchanged between graph construction and backward, because closures read parent data; rebuild the graph after changing inputs. Backward resets reachable gradients on each call, unlike frameworks that accumulate leaf gradients. Its recursive graph walk is bounded by Python’s recursion depth. For powers, x**0 is treated as the constant polynomial 1, including at x=0; non-polynomial powers require a positive base in this implementation. Numeric differences require positive finite h, representably distinct x±h and valid nearby function evaluations.

"""calculus_demo.py — derivatives three ways + a mini autodiff engine."""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Callable
from math import isfinite
 
import numpy as np
 
# ── 1. Numeric derivative via central difference ────────────────────────
def numeric_deriv(f: Callable[[float], float], x: float, h: float = 1e-6) -> float:
    """Central difference; O(h^2) truncation for sufficiently smooth f."""
    if not isfinite(x) or not isfinite(h) or h <= 0:
        raise ValueError("finite x and positive finite h required")
    if not isfinite(x + h) or not isfinite(x - h) or x + h == x or x - h == x:
        raise ValueError("step must give finite distinct sample points")
    return (f(x + h) - f(x - h)) / (2 * h)
 
def demo_numeric() -> None:
    fns = {
        "x**2":     (lambda x: x**2,     lambda x: 2*x),
        "x**3":     (lambda x: x**3,     lambda x: 3*x**2),
        "sin(x)":   (np.sin,             np.cos),
        "exp(x)":   (np.exp,             np.exp),
        "log(x)":   (np.log,             lambda x: 1/x),
        "sin(x**2)":(lambda x: np.sin(x**2), lambda x: np.cos(x**2) * 2*x),
    }
    print(f"{'function':<12} {'x':>4} {'analytic':>12} {'numeric':>12} {'|err|':>10}")
    for name, (f, fp) in fns.items():
        for x in [0.5, 1.0, 2.5]:
            a = fp(x); n = numeric_deriv(f, x)
            print(f"{name:<12} {x:>4.1f} {a:>12.6f} {n:>12.6f} {abs(a-n):>10.2e}")
 
# ── 2. Chain rule by hand + verification ────────────────────────────────
def demo_chain_rule() -> None:
    # y = sin(3x**2 + 1). Let u = 3x**2 + 1, so y = sin(u).
    # dy/dx = cos(u) * du/dx = cos(3x**2 + 1) * 6x
    f  = lambda x: np.sin(3*x**2 + 1)
    fp = lambda x: np.cos(3*x**2 + 1) * 6*x
    for x in [0.3, 1.0, 2.7]:
        a, n = fp(x), numeric_deriv(f, x)
        print(f"x={x}: analytic={a:.6f}  numeric={n:.6f}  err={abs(a-n):.2e}")
 
# ── 3. Mini autodiff engine (à la micrograd) ────────────────────────────
@dataclass
class Value:
    """Scalar wrapper that tracks its own gradient. Reverse-mode autodiff."""
    data: float
    grad: float = 0.0
    _backward: Callable[[], None] = field(default=lambda: None, repr=False)
    _prev: tuple = field(default_factory=tuple, repr=False)
 
    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, _prev=(self, other))
        def _bw():
            self.grad  += 1.0 * out.grad
            other.grad += 1.0 * out.grad
        out._backward = _bw
        return out
 
    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, _prev=(self, other))
        def _bw():
            self.grad  += other.data * out.grad
            other.grad += self.data  * out.grad
        out._backward = _bw
        return out
 
    def __pow__(self, n: float):
        # Real powers: positive base, or a nonnegative integer polynomial.
        if not isinstance(n, (int, float)) or not isfinite(n):
            raise ValueError("finite real exponent required")
        if self.data <= 0 and not (n >= 0 and float(n).is_integer()):
            raise ValueError("this demo requires a positive base for non-polynomial powers")
        out = Value(self.data ** n, _prev=(self,))
        def _bw():
            if n != 0:
                self.grad += n * (self.data ** (n - 1)) * out.grad
        out._backward = _bw
        return out
 
    def sin(self):
        out = Value(np.sin(self.data), _prev=(self,))
        def _bw():
            self.grad += np.cos(self.data) * out.grad
        out._backward = _bw
        return out
 
    __radd__ = __add__
    __rmul__ = __mul__
 
    def backward(self):
        # Build topo order, then apply _backward in reverse
        topo, seen = [], set()
        def build(v: "Value"):
            if id(v) in seen: return
            seen.add(id(v))
            for p in v._prev: build(p)
            topo.append(v)
        build(self)
        for v in topo:
            v.grad = 0.0
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()
 
def demo_autodiff() -> None:
    """Compute d/dx of y = sin(3x^2 + 1) at x = 1.5 using autodiff."""
    x = Value(1.5)
    y = (Value(3) * x**2 + Value(1)).sin()
    y.backward()
    print(f"autodiff  dy/dx = {x.grad:.6f}")
    print(f"analytic  dy/dx = {np.cos(3*1.5**2 + 1) * 6*1.5:.6f}")
 
if __name__ == "__main__":
    print("── NUMERIC vs ANALYTIC ─────────────")
    demo_numeric()
    print("\n── CHAIN RULE ──────────────────────")
    demo_chain_rule()
    print("\n── AUTODIFF ────────────────────────")
    demo_autodiff()

Run it:

uv run --with numpy python calculus_demo.py

Reference output from the bounded CPU run; floating-point error columns may vary by platform:

── NUMERIC vs ANALYTIC ─────────────
function        x     analytic      numeric      |err|
x**2          0.5     1.000000     1.000000   1.29e-11
x**2          1.0     2.000000     2.000000   2.00e-12
x**2          2.5     5.000000     5.000000   6.99e-10
x**3          0.5     0.750000     0.750000   7.50e-13
x**3          1.0     3.000000     3.000000   8.03e-11
x**3          2.5    18.750000    18.750000   2.40e-09
sin(x)        0.5     0.877583     0.877583   7.48e-12
sin(x)        1.0     0.540302     0.540302   2.77e-11
sin(x)        2.5    -0.801144    -0.801144   1.23e-10
exp(x)        0.5     1.648721     1.648721   3.04e-11
exp(x)        1.0     2.718282     2.718282   1.63e-10
exp(x)        2.5    12.182494    12.182494   2.25e-09
log(x)        0.5     2.000000     2.000000   2.00e-12
log(x)        1.0     1.000000     1.000000   2.64e-11
log(x)        2.5     0.400000     0.400000   1.15e-11
sin(x**2)     0.5     0.968912     0.968912   1.29e-11
sin(x**2)     1.0     1.080605     1.080605   7.73e-14
sin(x**2)     2.5     4.997247     4.997247   6.76e-10
 
── CHAIN RULE ──────────────────────
x=0.3: analytic=0.533306  numeric=0.533306  err=6.13e-11
x=1.0: analytic=-3.921862  numeric=-3.921862  err=1.95e-10
x=2.7: analytic=-10.336183  numeric=-10.336183  err=2.42e-09
 
── AUTODIFF ────────────────────────
autodiff  dy/dx = 0.934149
analytic  dy/dx = 0.934149

What each block does

Anatomy of the code

Name: numeric_deriv · central difference

Desc: Uses (f(x+h) − f(x−h)) / (2h) instead of forward difference. For bounded third derivative near x, symmetric Taylor expansions cancel the first-order truncation term, giving O(h²) error in exact arithmetic. Floating-point roundoff can grow roughly as ε/h; decreasing h indefinitely is not more accurate.

Name: demo_numeric · sanity table

Desc: Six functions, three x-values each. Printed errors depend on function, scale and platform; compare with an explicit tolerance for these sample points. Agreement is a check, not a proof, and a bug can agree accidentally at one point.

Name: demo_chain_rule · by hand + check

Desc: Write out the chain rule manually, then verify numerically. This is how researchers double-check gradient math before deploying to PyTorch.

Name: Value class · reverse-mode autodiff

Desc: Every arithmetic op returns a new Value that remembers its parents and a local backward function. That's the whole trick.

Name: backward() · topo sort + reverse pass

Desc: Build the computation graph in forward order via depth-first traversal, then apply _backward in reverse. This teaches the dependency-order principle; it is not a reproduction of PyTorch’s engine, memory management or accumulation policy.

Name: _backward closures

Desc: Each op has a rule for propagating gradients to its inputs: for +, parent.grad += out.grad; for ×, use the other input; for x^n, n · x^(n-1) · out.grad; for sin, cos(x). Chain rule made mechanical.

What to observe

For these smooth moderate inputs, central differences at h=1e−6 should be close under a tolerance such as 1e−7; this is a bounded expectation, not a universal precision guarantee. The chain rule example: sin(3x² + 1)'s derivative is cos(3x² + 1) · 6x. Both the numeric and autodiff paths get the same answer. The small engine differentiates compositions of its supported operations within the stated real-domain and graph-depth limits. Try nesting deeper: ((x**2 * Value(3)).sin() + Value(1)) ** 2. Backward applies the same chain rule, subject to floating-point error and supported domains.

Goal: Feel why the chain rule matters — build a 3-layer computation

Hint: x feeds both b and c directly, so this is a branching graph, not a simple three-link chain. Add both paths: dy/dx = cos(c)·(x·db/dx + b). Multiplying only dy/dc · dc/db · db/dx drops the direct b contribution.

Add a deeper computation and check the gradient:

# y = ((x**2 + 3)*x).sin()  at x = 1.2
x = Value(1.2)
b = x**2 + Value(3)
c = b * x
y = c.sin()
y.backward()
 
# Chain rule by hand: dy/dx = cos(c) · dc/dx
# dc/dx = db/dx * x + b * 1 = 2x * x + b = 2x^2 + (x^2 + 3) = 3x^2 + 3
# So dy/dx = cos((x^2 + 3)*x) * (3x^2 + 3)
import numpy as np
manual = np.cos((1.2**2 + 3) * 1.2) * (3 * 1.2**2 + 3)
print(f"autodiff = {x.grad:.6f}  manual = {manual:.6f}")

With constant 10 instead of 3, the derivative at x = 1.2 is cos((1.2²+10)·1.2)·(3·1.2²+10), approximately 5.697143, versus 4.226989 for constant 3. The gradient is not monotonic in the constant generally, because the cosine factor also changes. Rebuild the graph, do not mutate an existing parent between forward and backward.

(d) Production reality · 15 min

Debug a shared graph before blaming the optimizer

For y = x·x + x at x = 3, the derivative is 2x+1 = 7. Both multiplication edges point to the same object, so both contributions must be accumulated with +=. If a local backward rule overwrites rather than adds, one path disappears. Calling this toy engine’s backward twice should still give 7 because it resets all reachable gradients. This differs from PyTorch-style leaf accumulation and is intentional. A change to .data needs a fresh forward graph.

Use differences as checks, not as derivative proofs

For x³, central differences give 3x²+h² exactly in real arithmetic: halving h quarters the truncation error. But for |x| at zero, a symmetric difference is zero for every h while the left and right derivatives are −1 and +1. A plausible finite-difference answer cannot establish differentiability. Check the domain and compare several step sizes away from kinks. Coordinate central differences need 2n function evaluations for a full n-parameter gradient; a directional check of ∇f·v needs only two evaluations of f(x±hv), but tests only that direction.

Prevent an invalid logarithm rather than clipping a NaN

For finite logits z and target t, cross-entropy is log(Σ exp(zᵢ)) − zₜ. With m = max(z), rewrite it as m + log(Σ exp(zᵢ−m)) − zₜ. At least one shifted exponential is 1, and no shifted argument is positive: this avoids the obvious overflow and a subsequent log(0) from naïve probabilities. For the bounded example below the loss is about 0.313262.

import math
logits = [1000.0, 999.0]
target = 0
m = max(logits)
loss = (m - logits[target]) + math.log(sum(math.exp(z - m) for z in logits))
assert math.isfinite(loss)
assert abs(loss - 0.31326168751822286) < 1e-12

This does not make every floating-point input safe: reject NaN/inf logits, inspect the first nonfinite forward or backward intermediate, and do not update parameters with nonfinite gradients. Gradient clipping can limit a large finite norm; it cannot repair a NaN caused by an invalid operation. Library fused loss functions should be preferred for training; this scalar demonstration is not a PyTorch service or GPU validation.

Where this shows up in the rest of the plan

Root: Derivatives are the currency of learning

Label: S020 · Gradient descent from scratch

Desc: You'll implement SGD using this session's autodiff engine.

Label: Regression models (related topic)

Desc: Linear/logistic regression fit by taking derivatives of the loss w.r.t. weights.

Label: Neural network foundations (related topic)

Desc: Backprop is chain rule applied through every layer, from output back to input.

Label: Optimisers (Adam, RMSprop, AdamW; related topic)

Desc: Use negative-gradient-derived updates for minimization; adaptive scaling, momentum and weight decay change the update rule.

Label: Reinforcement learning (policy gradients; related topic)

Desc: Policy gradient IS the derivative of expected reward with respect to policy parameters.

Label: Transformer training (related topic)

Desc: Softmax + cross-entropy + Adam + autograd, at billion-parameter scale.

(e) Recall + stretch · 10 min

Self-check (answer follows each question).

Recall — click each to reveal

Q: State the definition of the derivative as a limit.

A: f'(x) = lim_(h→0) (f(x+h)−f(x))/h if the finite two-sided limit exists at an interior point of the domain. h is nonzero during the limiting process; one-sided boundary derivatives are a separate convention.

Q: Compute d/dx (3x² + 5x − 7).

A: 6x + 5. Sum rule + power rule, constant vanishes.

Q: State the chain rule for y = f(g(x)).

A: If g is differentiable at x and f at g(x), dy/dx = f'(g(x))·g'(x): outer derivative at the inner value, times the inner derivative.

Q: Compute d/dx (sin(x²)).

A: cos(x²) · 2x. Outer = sin, inner = x². Chain rule: cos(inner) · d/dx(inner).

Q: Compute d/dx (e^(2x)).

A: e^(2x) · 2 = 2·e^(2x). Chain rule with outer = eᵘ (self-derivative), inner = 2x (derivative 2).

Q: What's the difference between forward-mode and reverse-mode automatic differentiation?

A: Forward-mode propagates a chosen input tangent, computing Jv; reverse-mode propagates an output cotangent, computing a vector–Jacobian product. For a scalar loss ℝⁿ→ℝ, a reverse sweep seeded by 1 gives all n partials. Assembling that full gradient by forward basis seeds needs n directions; a single directional derivative does not. Both methods operate within floating-point and differentiability limits.

Q: Why is central-difference numeric differentiation more accurate than forward-difference?

A: With sufficient smoothness (bounded second/third derivatives nearby), forward difference has O(h) truncation error and central difference O(h²). Each uses two evaluations in isolation; a cached f(x) lets forward difference use one additional evaluation. Roundoff, noise, kinks or invalid x−h can erase the central method’s advantage.

Q: Give one situation where you'd use numeric differentiation in real code.

A: Numeric differentiation is useful for checking analytic derivatives and for some black-box optimization workflows. It is costly and sensitive to step size and noise; use analytic or automatic derivatives when suitable.

Q: The derivative of loss with respect to a neural net's weights is called the ______, and gradient descent updates weights by subtracting a scaled ______.

A: Both blanks: gradient. The gradient is a vector (one partial per weight); the update is w ← w − learning_rate · gradient.

Q: Stretch — derive the derivative of the sigmoid function σ(x) = 1 / (1 + e⁻ˣ) and express it in terms of σ(x).

Stretch: true

A: For real x, let u = 1+e⁻ˣ > 0. The reciprocal and chain rules give σ′(x) = −u⁻²·(−e⁻ˣ) = e⁻ˣ/(1+e⁻ˣ)². Factor this as [1/(1+e⁻ˣ)]·[e⁻ˣ/(1+e⁻ˣ)] = σ(x)(1−σ(x)). Reusing σ from the forward pass avoids another exponential, though finite-precision saturation can still round the derivative to zero.

Explain-out-loud test

1. What does a derivative measure, in one sentence?
2. State the chain rule and give a 3-line example.
3. Why is reverse-mode autograd well suited to a scalar loss with many parameters?

Explain-out-loud answer guide

1. At a differentiable point, the derivative is the coefficient of the best local linear change: f(x+h) = f(x)+f′(x)h+o(h). The limit must exist; |x| at zero is a counterexample.

2. Set u = x² and y = sin(u). Then du/dx = 2x and dy/du = cos(u), so dy/dx = cos(x²)·2x. For branching graphs, sum contributions from all paths.

3. For f: ℝⁿ → ℝ, reverse mode propagates one output seed to all n input partials in one reverse sweep. Forward mode is useful for a few tangent directions or few inputs, and both incur finite-precision and domain limitations.

What comes next

Next in sequence: S020 · Gradients and gradient descent. The destination is a navigation link, not part of this review’s acceptance.

Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.

Corrective source notes

• OpenStax, Defining the Derivative and The Chain Rule: existence of the limit and differentiability conditions at each composition point.
• JAX’s JVP/VJP cookbook: Jacobian shapes, input/output directional products and the forward/reverse cost distinction.
• micrograd engine source: accumulation across parent edges and reverse topological traversal. Our toy resets gradients on each call; upstream micrograd’s backward routine does not reset all reachable gradients.
• Python floating-point tutorial: why algebraic equivalence and actual floating-point evaluation must be distinguished.

The displayed scalar programs were checked on bounded CPU inputs against analytic derivatives and independent polynomial-coefficient oracles, including negative controls. They do not certify arbitrary real inputs, deep recursive graphs, JAX/PyTorch runtime behaviour, GPU training or an optimizer’s convergence. The symbolic derivations and domains—not a passing finite difference—establish the mathematical claims.