The one idea the Transformer is built on: every token looks back over all the others and decides how much each one matters.
Rows ask the question ("what should I pay attention to?"); columns are
the tokens being looked at. Brighter = more attention. Grey = the future, which a token can't see. Tap a row.
he mixes the earlier tokens in these
proportions — that weighted blend becomes its new, context-aware representation:
Attention lets every position build its own summary of the past. For each token, the model
asks a query ("what am I looking for?"), compares it against a key offered by every earlier token
("what do I have?"), and uses the match scores to take a weighted average of their values ("what I'll
contribute"). Strong match → big weight. The result replaces the token's plain embedding with one that has
pulled in the relevant context — tap "he" and watch it draw mostly from "knight."
ELI5 Every word sends out a little search query, and every
earlier word holds up a label. Words whose labels match the query get listened to more. "He" searches for
"who?", "knight" answers loudest, so "he" ends up meaning "the knight" — the model resolved the pronoun by
looking, not by a rule.
Two details you can feel above. The grid is triangular: a token may only attend to
itself and the past, never the future — that "causal mask" is what makes the model a next-word predictor.
The scores are divided by √d — the "scaled" in scaled dot-product, here √2 — which keeps them sane as
the dimension grows. The extra focus slider is an inverse-temperature control for this demo: low and
attention smears evenly across everything; high and it spikes onto a single word.
Under the hood (deep): here, for clarity, a token's query, key, and value are all just
its embedding. In a real Transformer they're three different learned projections (Q = xWq, K = xWk,
V = xWv), so a token can look for one thing while offering another. The √d in the score is the "scaled" in
scaled dot-product attention — without it, large dimensions make the scores huge and softmax saturates.
Attention is also permutation-blind (it sees a set, not a sequence), so models add positional
information to the embeddings. And it's run several times in parallel — multi-head attention, one head for
"who's the subject," another for "what verb," etc. — which, with the feed-forward network around it, is exactly
the block in Lesson 09. Cost is O(n²): every token compares with every other, which is why context length is
expensive.
historyHistorical perspective
Modern neural attention was introduced in 2014 by Dzmitry Bahdanau, Kyunghyun
Cho and Yoshua Bengio as a fix for machine translation: their recurrent translator kept forgetting the start of long
sentences, so they let the decoder "attend" back to all the input words and softly align to the relevant ones.
It worked beautifully — but was still bolted onto a slow recurrent network. The revolution was the 2017
paper by Vaswani and seven colleagues at Google, defiantly titled "Attention Is All You Need": they threw
out recurrence entirely, letting attention do the sequence-mixing — wrapped with feed-forward layers,
residuals, normalization and positional encodings — the Transformer. It trained far faster (everything
parallel, no step-by-step recurrence) and scaled better
than anything before. Every model in this course, and GPT itself, is a direct descendant of that one paper.
Carry this out: self-attention = for each token, score a query against every earlier
token's key, softmax into weights, and blend their values. It replaces a fixed window with a learned,
content-based look-back — the core move of the Transformer. Next: wrap it with a feed-forward network and stack
it into a real block.