A bigram sees only the previous word — it has amnesia. Watch a prediction stay impossible until the clue slides into view.
Outlined = the one clue that determines the answer.
Bright = what the model can currently see. Slide right to widen the window.
The bigram from Lesson 00 only ever looks at the single previous word. That's fine for
"the → knight," but language is full of clues that sit far from where they're needed: a name introduced a
sentence ago, a pronoun's owner, a topic set at the start. With a window of one, the model literally cannot
reach them — so its guess is a shrug. Widen the window and the moment the clue appears, the answer snaps into
focus.
ELI5 Imagine reading a mystery through a slot that shows only
the last word. You'd never solve it. You need to see the whole page — and, more importantly, know which
earlier words are the ones that matter.
So why not just use a big window? Two problems. First, cost: a fixed "look back N words"
scheme (an n-gram) needs a table that grows impossibly fast as N rises, and it still can't reach past N.
Second, and deeper — not every earlier word matters, and which ones do changes every time. For "whisper
___" the clue is a specific noun six words back; for "she ___" it's whichever character was named; for the
topic sentence it's the subject up front. A fixed window can't learn which of those words matters — an
n-gram just memorises each exact context as its own separate case. What you actually want is a mechanism that
can look at every previous word and learn to weight the relevant ones.
That mechanism is attention
— the subject of the next lesson, and the single idea the whole Transformer is built around. Instead of a fixed
window, each position looks back over the entire context and decides, on the fly, how much to draw from each
earlier token.
historyHistorical perspective
Fixed-window n-gram models ruled language technology for decades — Shannon sketched
them in 1948, and by the 1990s they powered every speech recognizer and the first statistical machine
translation systems at IBM. Their Achilles' heel was exactly this lesson: they cannot see past their window, and
the window can't grow because the table of counts explodes (a 4-gram over a 50k vocabulary has 50,000⁴
possible entries). The field spent years papering over it with smoothing tricks and back-off schemes. Recurrent
neural networks (LSTMs, 1997) finally added a running memory — but squeezed the whole past through one small
hidden state, a bottleneck. The clean answer, letting a model look at all positions at once and weight them,
arrived with attention in the mid-2010s.