DL S012 · Micrograd — Build a Tiny Autograd Engine

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 12 / 80 · Module M02 · ~2.5 hours

0 · The story — the 94 lines of Python that made autograd click for a generation

In 2020, in the middle of the pandemic, Andrej Karpathy — then director of AI at Tesla, now one of the most-viewed AI educators alive — sat in his home office and recorded a 2 hour and 25 minute YouTube lecture called "The spelled-out intro to neural networks and backpropagation: building micrograd." Zero slides. Just a Jupyter notebook. He types every line. He gets confused at one point around the topological sort and leaves the confusion in the video. He arrives, minute by minute, at a Value class of about 94 lines of pure Python that implements automatic differentiation for scalar arithmetic — plus a 30-line nn.py neural network module on top.

That video, as of 2025, has ~2 million views. It is — quietly, without corporate sponsorship — the single most influential piece of deep-learning education on the internet. Every serious ML curriculum in 2024–2025 either includes it or is downstream of someone who watched it. Karpathy's follow-up series (makemore, nanoGPT, build GPT from scratch, LLM.c) all descend from the same "just build it, in real time, and read it out loud" style.

Why does micrograd matter more than its 94 lines suggest? Because it is architecturally identical to PyTorch's autograd, JAX's grad, TensorFlow 2's GradientTape, and tinygrad's Function. Same computation graph, same topological sort, same backward closures, same accumulate-with-+= pattern. Every one of those production frameworks is engineering added around the ideas in micrograd: replace scalars with tensors (100× more math per op), replace Python closures with fused CUDA kernels (10,000× faster per op), add distributed all-reduce (thousand-GPU training), add mixed precision (2× throughput), add compile-time optimisation (torch.compile, jax.jit). The IDEAS are all in the 100 lines.

Today you rewrite those lines from scratch. Not copy them. Type them, understand every design choice, extend with a couple of ops you'll need, and train an MLP using your own engine. By the end you'll be able to open PyTorch's autograd/ or tinygrad's Function class and immediately see the four moving parts you're about to build.

Prerequisites

• Session 004 — we built micrograd's ancestor there. This is the full version.
• Session 010 — you've done backprop by hand; today we automate it.

1 · The Value class — the atom

A Value wraps a scalar (a Python float) plus:

• its gradient (a float, accumulates)
• pointers to the Values it was created from (its _prev)
• a _backward function that knows how to distribute its gradient to those parents

class Value:
    def __init__(self, data, _children=(), _op=""):
        self.data      = float(data)
        self.grad      = 0.0
        self._prev     = set(_children)
        self._op       = _op
        self._backward = lambda: None       # default no-op for leaves
 
    def __repr__(self):
        return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"

That's a leaf Value. Creating a = Value(3.0) gives you a node with no children, no op, and a null backward. Its grad starts at 0.

2 · Ops — the _backward closure trick

Here's the crux. Every operation creates a NEW Value (the output). That new Value gets a _backward closure that captures the operands. When backward() runs, this closure is called: it reads self.grad (the output's grad, which downstream ops have accumulated), computes local derivatives, and adds them to the input Values' .grad fields.

2.1 Add

def __add__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), "+")
 
    def _backward():
        # ∂(a+b)/∂a = 1,  ∂(a+b)/∂b = 1
        self.grad  += 1.0 * out.grad
        other.grad += 1.0 * out.grad
    out._backward = _backward
    return out

Study those += operators. They ARE the sum-over-paths rule (S004 §3.1). If a is used at TWO different places downstream, its .grad gets contributions from both, added together. This is why we use += not =.

2.2 Multiply

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), "*")
 
    def _backward():
        # ∂(ab)/∂a = b,  ∂(ab)/∂b = a
        self.grad  += other.data * out.grad
        other.grad += self.data  * out.grad
    out._backward = _backward
    return out

Notice how the closure captures self and other — Python closures make this natural. When _backward runs later during backward(), self.data and other.data are still there (still the forward-pass values). This is exactly the "cache intermediates for backward" pattern from S009.

2.3 Power (for x**2 etc)

def __pow__(self, other):
    assert isinstance(other, (int, float)), "power must be numeric"
    out = Value(self.data ** other, (self,), f"**{other}")
 
    def _backward():
        # ∂(a^n)/∂a = n * a^(n-1)
        self.grad += (other * self.data ** (other - 1)) * out.grad
    out._backward = _backward
    return out

2.4 Convenience — unary negation, subtraction, division

def __neg__(self):       return self * -1
def __sub__(self, other):return self + (-other)
def __radd__(self, other): return self + other        # reverse ops for `2 + Value(3)`
def __rmul__(self, other): return self * other
def __truediv__(self, other): return self * other**-1

Now all four arithmetic operations work — even mixing Value and Python floats.

2.5 tanh — the non-linearity for our neural net

import math
 
def tanh(self):
    t = math.tanh(self.data)
    out = Value(t, (self,), "tanh")
 
    def _backward():
        # ∂tanh(z)/∂z = 1 - tanh(z)^2
        self.grad += (1 - t*t) * out.grad
    out._backward = _backward
    return out

tanh is our activation. We could add ReLU, sigmoid, exp, log — same pattern. Left as exercise.

2.6 exp — bonus for softmax later

def exp(self):
    e = math.exp(self.data)
    out = Value(e, (self,), "exp")
    def _backward():
        self.grad += e * out.grad          # d(exp z)/dz = exp z
    out._backward = _backward
    return out

3 · backward() — the topological walk

Now the driver. Given the loss L (a Value), we want to fill in .grad for every ancestor node. The algorithm:

1. Build a topological order of all Values reachable from L.
2. Set L.grad = 1.0 (the seed).
3. Walk the topological order IN REVERSE, calling each Value's _backward.

def backward(self):
    # 1. topo sort — parents before children (post-order DFS)
    topo, visited = [], set()
    def build(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build(child)
            topo.append(v)
    build(self)
 
    # 2. seed and walk
    self.grad = 1.0
    for v in reversed(topo):
        v._backward()

Why topological order? Because a node's _backward uses self.grad, which should already be fully accumulated by the time we hit it. Topological sort guarantees every downstream node is processed first.

Why reversed? Because build() post-order gives parents-before-children. We want the opposite for backward, so we reverse. (Andrej's original micrograd uses this same pattern.)

4 · Try it on our S010 example

w = Value(0.5)
x = Value(1.0)
b = Value(0.1)
y = Value(1.0)
 
# forward: L = ((w*x + b) - y)^2
z = w*x + b
diff = z - y
L = diff * diff
 
print("forward L =", L.data)       # ((0.5*1 + 0.1) - 1)^2 = (-0.4)^2 = 0.16
L.backward()
print("dw =", w.grad)               # ∂L/∂w = 2*(z-y)*x = 2*(-0.4)*1 = -0.8
print("db =", b.grad)               # ∂L/∂b = 2*(z-y)*1 = -0.8

Output:

forward L = 0.16
dw = -0.8
db = -0.8

Verify by hand or numerically. It's right.

You have written a working autograd engine in ~70 lines of Python.

5 · Build a Neuron, Layer, and MLP on top

Now the fun part — use Value as the building block for a neural network.

import random
 
class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]
        self.b = Value(0.0)
 
    def __call__(self, x):
        # x is a list of Values
        act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
        return act.tanh()
 
    def parameters(self):
        return self.w + [self.b]
 
class Layer:
    def __init__(self, nin, nout):
        self.neurons = [Neuron(nin) for _ in range(nout)]
 
    def __call__(self, x):
        return [n(x) for n in self.neurons]
 
    def parameters(self):
        return [p for n in self.neurons for p in n.parameters()]
 
class MLP:
    def __init__(self, nin, nouts):
        sizes = [nin] + nouts
        self.layers = [Layer(sizes[i], sizes[i+1]) for i in range(len(nouts))]
 
    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x[0] if len(x) == 1 else x
 
    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

Twenty-five more lines. That's a full neural network module.

5.1 Train it on 4 points

Output:

step   0  loss=7.891
step  20  loss=1.234
step  40  loss=0.421
step  60  loss=0.187
...
step 180  loss=0.011

Loss drops from 8 to 0.01 in 200 steps. On micrograd. On four data points. Karpathy's magic, reproduced.

Predict and confirm:

for x, y in zip(Xs, Ys):
    p = model([Value(xi) for xi in x])
    print(f"input {x}  target {y}  pred {p.data:.3f}")
# input [2, 3, -1]   target  1.0  pred  0.98
# input [3, -1, 0.5] target -1.0  pred -0.99
# input [0.5, 1, 1]  target -1.0  pred -0.97
# input [1, 1, -1]   target  1.0  pred  0.99

Nailed it.

6 · How does this differ from PyTorch?

But the MATHEMATICAL CORE is identical. Same graph, same topological sort, same += accumulation, same closure-per-op design. Every time PyTorch does .backward(), it's doing what our backward() does — just faster and with more ops.

7 · Two exercises worth doing right now

8 · The mental checklist for reading any autograd source

Whenever you open any autograd source (PyTorch's autograd/, JAX's grad, tinygrad's Function, TensorFlow 2's GradientTape), you'll see the same four moving parts. Look for them:

1. The Value / Tensor class — data + grad + _prev (parent pointers).
2. Op registration — each op creates a new node and attaches a backward closure / function.
3. backward() driver — topological sort + reversed walk from the loss.
4. Some form of gradient accumulation — += for the multi-variable chain rule (sum over paths, S004 §3.1).

If you see these four, you understand the framework's autograd. Everything else is engineering.

8.1 A 2025 field guide: micrograd → PyTorch → JAX → tinygrad → LLM.c

The modern trend is tracing + JIT-compilation: JAX collects the forward pass into a static graph, differentiates it symbolically, then hands the whole thing to XLA to fuse into kernels. torch.compile (PyTorch 2.0, 2023) does the same via TorchDynamo + AOTAutograd. The mental model you built with micrograd still applies — those systems still walk a graph in reverse and accumulate with += — they just do it in a compiler pass before execution instead of one Python function call at a time.

8.2 Further reading

• Karpathy, micrograd repo — 94 lines, read it.
• Baydin, Pearlmutter, Radul & Siskind, "Automatic Differentiation in Machine Learning: A Survey" (JMLR 2018) — the definitive academic reference. Sections 3 (reverse mode) and 5 (implementation) are what you just built.
• Frostig, Johnson & Leary, "Compiling machine learning programs via high-level tracing" (MLSys 2018) — JAX's design paper. Notice how their trace-then-differentiate is a static-graph version of micrograd's dynamic graph.
• Hotz et al., tinygrad Function source — read after micrograd to see the tensor version.
• Karpathy, llm.c repo (2024) — GPT-2 training in ~2K lines of pure C, backward pass hand-derived for every op. If you loved this session, you'll love that.

9 · War stories

def _backward():
    self.grad  = 1.0 * out.grad
    other.grad = 1.0 * out.grad

10 · Retention scaffold

One-line summary (write it in your own words): _______________________________________________________________

Spaced review: re-implement the Value class from memory in 24 hours (no peeking); revisit §5 (Neuron/Layer/MLP) on day 7 with an added ReLU op.

Next session (S013): micrograd is beautifully clear but painfully slow — one Python call per elementary op. Real neural networks have billions of ops per forward pass; Python would take days. S013 shows how to make micrograd/NumPy-style networks 100× faster using vectorisation (einsum, batched matmul, avoiding Python loops). Last stop before we finally move to PyTorch in M03.

Sticky note (keep on your desk):

• Four moving parts: Value / op-with-closure / topo-sort backward / += accumulation. Every autograd system, everywhere.
• Multiply's _backward: a.grad += b.data * out.grad; b.grad += a.data * out.grad. Memorise.
• The 8-line backward driver: topo DFS post-order → reverse → call every _backward.

Previous: ← DL S011 · MLP on MNIST — All Numpy, No PyTorch · Next: DL S013 · Vectorization and Speed →