Strip away everything and one job remains: guess the next word. Here's one you can drive.
start from:
Probabilities for the word after
"the" — the top 8 of its next-word beliefs (the full count is in the
readout below; sampling draws from all of them):
vocabulary—
choices after "the"—
That's the entire trick. A language model reads what it has so far and outputs
a probability for every word it knows — a number saying "how likely is this to come next."
Then it picks one, appends it, and repeats. Words fall out one at a time. That loop, at scale,
is ChatGPT.
ELI5 It's autocomplete with an imagination.
Your phone suggests three next words; this suggests all of them at once, each with a confidence,
and then rolls a weighted dice to choose.
The two buttons are the whole personality of the model.
🎯 Greedy always takes the tallest bar — the safest guess. Correct, but repetitive and dull;
it'll often loop the same phrase.
🎲 Sample rolls a dice weighted by the bars — a tall bar is likely, a short bar is possible.
This is where surprise (and usable stories) come from.
Temperature reshapes those bars before the dice roll:
Low (→0.1): tall bars get taller, short ones vanish. Timid, predictable, near-greedy.
High (→1.6): the bars flatten toward equal. Wild, creative, eventually gibberish.
softmax: p(word) = exp(score / T) / Σ exp(score / T)
— the one formula that turns raw scores into probabilities. T is the temperature knob you're dragging.
Under the hood (deep): this toy is a bigram — it counts, in a small
fantasy corpus, how often each word follows each other word, and turns those counts into
score = log(count), then softmax(score / T). It only ever looks at the
single previous word. That's its fatal flaw: it has no memory of the sentence. In "the knight
drew her ___" it only sees "her", not "knight drew". Fixing exactly that — letting the model look
back across the whole context — is the job of attention, the idea GPT is built on and
the subject of Module 2. A real GPT swaps the count table for a trained neural net, but the outer loop
on this page — scores → softmax → sample → repeat — never changes.
historyHistorical perspective
Predicting the next token is over a century old. In 1913 the Russian mathematician
Andrey Markov — inventing what are now called Markov chains — worked through the first 20,000
letters of Pushkin's poem Eugene Onegin by hand, counting how often a vowel followed a consonant. That
tally was the first statistical language model. In 1948Claude Shannon, founding information
theory, played the same game with words: open a book, pick a word, flip to a random spot where it reappears,
take the next word, repeat — generating eerily English-ish text exactly the way the bigram above does.
His 1951 experiment even measured the "entropy of English" by having people guess the next letter — a
human next-token predictor. When n-gram models later powered the first speech recognizers at IBM, Fred Jelinek
reputedly quipped, "Every time I fire a linguist, the recognizer gets better" — the lesson that data
beats hand-written rules, decades early.
Carry this out: an LLM is a next-word probability machine wrapped in a loop.
Everything else — attention, training, billions of parameters — exists only to make those
probabilities smart. The loop stays this simple.