← curriculum

06 · Embeddings

An id like 4417 means nothing on its own. An embedding gives every token a place in "meaning-space" — where near = similar.

Tap any word to see its nearest neighbours. Colours = rough categories.

directions carry meaningVector arithmetic

The direction from one word to another is itself meaningful. Take a word, subtract a second, add a third — you land near a fourth. Tap an analogy:

result lands nearest—

Honesty note: these 2-D points are placed by hand for illustration, so the clusters and analogies resolve exactly, by construction. In a real model the coordinates are learned (in hundreds of dimensions), similarity is fuzzier, and analogies like king − man + woman only approximately land on queen.

Token ids are arbitrary labels. After Lesson 05, "sword" might be id 4417 and "blade" id 92 — but 4417 and 92 aren't "far apart" in any meaningful way, and 4417 vs 4418 aren't "close." The raw id carries zero meaning; it's just a row number. Feeding those numbers straight into a network would be like sorting a library by the barcode. We need a representation where similar meanings sit near each other.

An embedding is a learned vector per token. The model keeps a big table — the embedding matrix, one row per vocabulary entry — and each row is a list of numbers: coordinates of that token in a high-dimensional space. Step one of every forward pass is a pure lookup: swap each id for its row. In a real model those coordinates aren't hand-placed; they're knobs, trained by the same gradient descent as everything else, and the model drifts related tokens together because grouping them makes the next word easier to predict. (The map above is placed by hand, purely to show the idea — see the note under it.)

ELI5  Give every word GPS coordinates on a map of meaning. Synonyms end up in the same neighbourhood; "sword," "blade," and "axe" cluster near each other, far from "forest." And whole relationships become directions: the step from "man" to "woman" points the same way as the step from "king" to "queen" — which is why the arithmetic above works.

Two things fall out of this, both visible above. Distance = similarity (tap a word, its neighbours light up). Direction = relationship (the analogy buttons). That a model discovers these on its own, just from being asked to predict text, is one of the small miracles of the field.

Under the hood (deep): this map is a 2-D cartoon. Real embeddings live in hundreds of dimensions — 768 in GPT-2, thousands in large models — and you only ever see them by projecting down (PCA, t-SNE), which is why published word maps look a bit like this. Similarity is usually measured by cosine (angle) rather than straight-line distance. Crucially, the embeddings here are static — one fixed vector per word — but a transformer produces contextual embeddings: the vector for "bank" bends toward "river" or "money" depending on the sentence. Making a token's representation depend on its neighbours is exactly the job of attention, the next module.

historyHistorical perspective

The intuition is older than the maths: in 1957 the linguist J. R. Firth wrote "You shall know a word by the company it keeps" — the distributional hypothesis, that meaning can be read off from context. Statisticians built on it with Latent Semantic Analysis in the 1990s, but the thunderclap came in 2013, when Tomáš Mikolov and colleagues at Google released word2vec and showed that king − man + woman really does land on queen — that arithmetic on learned word vectors captures analogies. Stanford's GloVe followed in 2014. Those static vectors were the bridge from "words as arbitrary symbols" to "words as geometry," and every modern model still begins, at layer zero, with exactly this lookup table.
Carry this out: an embedding turns each token id into a learned vector, so that meaning becomes geometry — near is similar, and directions encode relationships. It's the model's first layer, a lookup table trained like any other weights. But these vectors are still fixed per word; making them depend on context is the one big idea coming next: attention.