A model is a machine full of numbers you can turn. Loss is the one number that says how wrong it is.
loss (mean squared error)—
best possible—
Each pink stub is one point's error — the gap between the
line's guess and the real dot. Loss is the average of those gaps, squared. Make it small.
A model is just knobs. This one has two: a slope and an intercept —
together they define a line, and the line is the model's guess of y for any x.
A real GPT is the same idea with 117 million knobs instead of 2. Turning knobs is all "learning" ever is.
ELI5 The dots are the truth. The line is your guess.
Loss is your total wrongness in one number. Learning = wiggle the knobs until wrongness is as small as it goes.
Why one number? Because to improve automatically, the machine needs a single score to push
down — a target it can chase. Two knobs, one score. Everything downstream (the next two lessons, and training a
real net) is just clever ways to lower this one number.
Why squared? The error of a point can be above or below the line (positive or negative).
Squaring makes them all positive (so they can't cancel out) and punishes a big miss far more than a small
one — a miss of 4 costs 16, a miss of 1 costs 1. So the model cares most about its worst mistakes.
loss = mean of ( guess − actual )² =
(1/N) · Σ ( w·x + b − y )²
— "mean squared error", the classic loss for fitting numbers.
Under the hood (deep): "snap to best fit" jumps straight to the exact optimum via
the least-squares formula — a one-shot closed-form solution you get because a linear model with squared
error has a special, neatly-solvable shape. Logistic regression (the sigmoid in Lesson 03) and general
multi-layer networks have no comparable closed form, so they're trained iteratively — you search
for the knob settings instead. That search — nudging every knob a
little bit downhill, over and over — is gradient descent, and it's the entire subject of Lesson 02.
Language models are trained the same way; the loss there is "how surprised was I by the true next token."
historyHistorical perspective
"Minimize the squared error" once helped find a lost planet. On New Year's Day 1801
astronomers spotted the dwarf planet Ceres, tracked it for weeks, then lost it in the Sun's glare. The
24-year-old Carl Friedrich Gauss took the sparse observations, fit an orbit by least squares, and
predicted where it would re-emerge — and it was found there, nearly a year later. Gauss said he'd used the
method since 1795, but Adrien-Marie Legendre published it first in 1805, sparking one of maths' bitterest
priority disputes. Gauss also gave the reason we square the error (rather than cube it): squared error is
exactly the right loss if your measurement noise follows the normal (Gaussian) distribution —
which is why it's everywhere. The name "regression," incidentally, comes from Francis Galton noticing in
the 1880s that tall parents tend to have somewhat shorter children — "regression toward the mean."
Carry this out: model = knobs; loss = one number measuring wrongness; learning =
turning the knobs to shrink the loss. Hold that and the rest of ML is mechanics.