Last lesson you turned knobs by hand. Gradient descent turns them automatically — by always stepping downhill.
knob w—
loss—
gradient (slope)—
step0
The pink tangent is the gradient — the hill's steepness
under the ball. Each step moves the ball opposite the slope, by learning-rate × steepness.
The gradient points the way the loss climbs fastest — uphill — and how steep that climb is.
To reduce the loss you step the opposite way, downhill (that's why the update has a minus sign). It's the
slope of the loss under your current knob: if the slope tilts up-to-the-right, downhill is left, so you decrease
the knob; if it tilts up-to-the-left, you increase it. Do that repeatedly and you slide to the bottom — the knob
setting with the lowest loss. That's training, in one sentence.
update: w ← w − learning-rate × gradient
— take a step downhill; the learning rate is how big a step.
ELI5 You're a ball on a hill in thick fog — you can't see the
valley, but you can feel the slope under your feet. So you step downhill a little, feel again, step again. The
learning rate is your stride length.
Now break it. The learning rate is the one knob that most often ruins training:
Too small (≈0.05): correct, but the ball crawls — real training would take forever.
Just right (≈0.3–0.9): slides smoothly into the valley in a few steps.
Too big (>1): it overshoots the bottom and bounces to the other wall. Past ~2 it
bounces higher every time and flies off to infinity — the loss explodes. Drag the slider up and hit run.
Under the hood (deep): this curve is one knob, so "the gradient" is a single number
you could get with school calculus (for L = ½(w−3)² it's just w−3). A real net has
millions of knobs, so the gradient is a million-number vector — the downhill direction in a
million-dimensional space — and you can't get it by hand. That's exactly what autograd computes for you in
one backward pass (its own subject, in Lesson 04). The update rule above is then applied to every knob at
once. Fancier optimizers (Adam, the most common) give each knob its own step size, adapted from that knob's
gradient history — though they still have a base learning rate you set. But it's still "roll downhill."
historyHistorical perspective
Rolling downhill was written down in 1847 by Augustin-Louis Cauchy, who needed
to solve big systems of equations arising from astronomy — computing the orbits of celestial bodies —
and proposed simply following the gradient down. It stayed a quiet numerical trick for a century. The version
that trains modern nets, stochastic gradient descent (take the step from a small sample of the
data, not all of it), traces to Robbins and Monro's 1951 work on "stochastic approximation." And the
learning-rate headache you just felt is still active research: the optimizer most models use today, Adam
(2014, Kingma & Ba), gives each knob a step size adapted from the gradients' running statistics — far more
forgiving to tune, though you still choose a base learning rate.
Carry this out: gradient = steepest-uphill direction; step = −learning-rate × gradient (i.e. downhill); repeat.
The learning rate is a Goldilocks knob — too big diverges, too small crawls.