Gradients for free. Write the forward maths; the machine hands back the slope for every knob — in one backward pass.
The problem. Gradient descent needs the slope of the loss with respect to every knob. For
one neuron you can write those two derivatives by hand. For a network with millions of knobs across many layers,
you cannot — and you'd have to redo it every time the architecture changes. Autograd (automatic
differentiation) computes them all for you, exactly, from nothing but the forward maths.
ELI5 As the model computes its answer, it jots down every tiny
step it took, like breadcrumbs. To find how much each knob mattered, it just walks the breadcrumbs backwards,
multiplying the little local effects together. You write how to go forward; it figures out
backward on its own.
how it worksA graph, then one walk in reverse
1 · The forward pass secretly builds a graph. Every operation (×, +, sigmoid, −, ²) becomes
a node that remembers its result and which nodes it came from — the boxes above. Each node also knows one thing:
its own local derivative (× knows that ∂(a·b)/∂a = b; sigmoid knows its slope is a(1−a)). That's the only
calculus in the whole system, and it's tiny and local.
2 · The backward pass is one walk of that graph, in reverse. Seed the output with
∂L/∂L = 1, then visit nodes from the loss back toward the knobs. At each node, multiply the gradient arriving
from the right by the node's local derivative (the chain rule) and hand the result to its inputs. Step it
above and read the equation strip: that's every multiplication, in order.
chain rule at a node: grad(input) += grad(output) × ∂(this node)/∂(input)
The += matters: if a knob feeds into two places, the gradients from both paths add up. And the
whole backward walk costs about the same as the forward one — so you get the slope for all knobs, whether
there are 2 or 117 million, for the price of a single pass. That single fact is what makes training large
networks possible at all.
Under the hood (deep): this is reverse-mode automatic differentiation, and it
wins here for a specific reason. Reverse mode gets the derivatives of one output with respect to
many inputs in one pass — which is exactly the shape of training: one loss number, millions of knobs.
(Forward-mode autodiff is the opposite trade and would need one pass per knob.) The alternatives are worse:
numeric differences (nudge each knob, re-run) cost one forward pass per knob and are imprecise;
symbolic differentiation explodes the expression. PyTorch and libtorch are industrial versions of exactly
the boxes above: a tape of operations plus a backward for each. Writing the backward for each op by hand instead
(no tape) is the pure-C path, e.g. Karpathy's llm.c.
historyHistorical perspective
Backprop is famous for being invented several times before anyone noticed. The
reverse-mode trick you just stepped through was first published in 1970 by a Finnish master's student,
Seppo Linnainmaa — as a way to track rounding errors, with no neural network in sight. Paul Werbos
tied it to training neural nets in his 1974 PhD thesis, and it was largely ignored. Only in 1986, when
Rumelhart, Hinton, and Williams published "Learning representations by back-propagating errors" in
Nature, did it catch fire and revive the whole field. The underlying chain rule is far older —
Leibniz was chaining derivatives in 1676 — but doing it automatically, by having a program record
its own operations and replay them backwards, is what separates a calculus exercise from a system that can
train a 117-million-knob model. Today's frameworks (PyTorch, JAX) are industrial descendants of that one 1970
idea.
Carry this out: forward builds a graph of ops that each know one local derivative;
backward walks it once, chaining and summing, to produce the gradient of the loss w.r.t. every knob at once.
Lessons 01–03 turned the knobs; this is where the numbers telling them which way come from. That closes
the foundations — next, the harder question: how do you turn text into numbers a network can eat?
Back to curriculum →