The smallest real neural net there is — and the first one that can bend. Give it examples; it learns the function.
The job: a game needs a danger sensor. Feed it one number
about a creature — its size — and it should answer how dangerous the creature is, from
0 (harmless) to 1 (deadly). Small things are safe,
big things aren't. The dots are labelled examples; the neuron has to learn the curve through them.
input = size (x) · output = danger (a). The blue curve is the neuron's function.
loss (mean sq. error)—
gradient ∂loss/∂w—
gradient ∂loss/∂b—
step0
loss over training steps — watch it fall as the curve snaps onto the dots
A neuron computes exactly one function. Take the input x, scale it by a weight
w, add a bias b, and squash the result with a sigmoid:
a = sigmoid( w·x + b )
— two knobs shape one S-curve. That curve is the neuron.
The two knobs move the curve in the two ways that matter: w sets how steep the switch is (and
which way it faces), and b slides the switch left or right — it sets the size at which "safe"
tips over into "dangerous." "Learning the function" just means finding the w, b that thread the
S-curve through the example dots. That's it — this is logistic regression, the simplest classifier there
is, and a single neuron is exactly that.
ELI5 The neuron is a dimmer switch, not an on/off toggle. As a
creature gets bigger, the "danger" dial slides smoothly up from 0 to 1. Training moves where the dial
flips and how sharply, until it agrees with every example you've shown it.
the squashWhat the sigmoid is, and why it's here
The inner part, w·x + b, is just a straight line — it runs off to −∞ and +∞. But a danger
score has to live between 0 and 1, and the world isn't a straight line: past a certain size, "more dangerous"
saturates. The sigmoid fixes both. It takes any number, however huge or negative, and bends it into the
range (0, 1):
sigmoid(z) = 1 / ( 1 + e−z )
— z very negative → ~0; z = 0 → exactly 0.5; z very positive → ~1.
So it turns the unbounded line into a smooth S: a soft threshold you can read as a probability. Two
more reasons it's the classic choice: it's smooth (no sharp corner), so gradient descent always has a
slope to follow; and its own slope has a famously tidy form, sigmoid(z)·(1−sigmoid(z)), which is
the piece that shows up in the gradients below. This "squash" is called the neuron's activation function
— it's the one nonlinear step, and it's the whole reason a stack of neurons can model curves instead of only
straight lines.
the descentTraining it by hand
Everything from the previous two lessons applies unchanged: loss is the mean squared error between the
curve and the dots (Lesson 01), and a train step nudges each knob downhill (Lesson 02):
w −= lr·∂loss/∂w, b −= lr·∂loss/∂b. The only new thing is where the gradients come
from — the chain rule, walked back through the square, the sigmoid, and the line:
∂loss/∂w = mean[ 2(a−y) · a(1−a) · x ] | ∂loss/∂b = mean[ 2(a−y) · a(1−a) ]
— loss←a gives 2(a−y); a←z gives the sigmoid slope a(1−a); z←w gives x, z←b gives 1. Multiply the chain, average over the dots.
Drag w and b to fit the dots by hand and watch the loss and gradients react — then hit
train and let gradient descent turn the knobs for you. As the curve locks onto the data the gradients
fade toward zero: at the bottom of the valley the ground is flat, so there's nothing left to descend.
Under the hood (deep): the two gradient formulas above were derived by hand because
this network is tiny — one neuron, two knobs. That doesn't scale: a real network has millions of knobs threaded
through many layers, and no one derives those by hand. Instead the derivatives are produced mechanically by
automatic differentiation (autograd) — the forward pass records a graph of the operations, and a single
backward walk yields the gradient for every knob at once. That's the only reason the same training loop
scales from 2 knobs to 117 million; Lesson 04 builds it. Curiously, cranking the learning rate up here does
not blow up the way Lesson 02's bowl did: as the S-curve steepens, its ends flatten and their slope
vanishes, which quietly caps how far a step can push. That mercy has a dark side — that same vanishing slope is
exactly why deep sigmoid networks are hard to train, and why modern nets mostly use other activations.
historyHistorical perspective
Two pieces on this page have surprising origins. The sigmoid wasn't born in
neuroscience — the Belgian mathematician Pierre François Verhulst introduced it in 1838 to
model population growth that slows as resources run out, and named it the "logistic" curve in later work
(1845) — which is why fitting one is called logistic regression. The artificial neuron came later: McCulloch and
Pitts gave the first mathematical model in 1943, and in 1958Frank Rosenblatt built the
Mark I Perceptron — a Navy-funded machine whose weights were literal knobs (potentiometers) turned
by electric motors as it learned. The New York Times reported it would soon "walk, talk, see, write, and
reproduce itself." It couldn't: in 1969Minsky and Papert proved a single neuron like this one
can't even learn XOR, and the letdown helped trigger the first "AI winter." The fix — stacking many neurons
into layers — needed the idea in the next lesson.
Carry this out — deep learning on one screen: forward (compute the curve) → loss (score
it against the data) → backward (gradients) → step downhill → repeat. A neuron is a line bent by a sigmoid; a
network is many of them; a GPT is many of them with attention in the middle and billions of knobs.