← curriculum

05 · Tokenization

A network eats numbers, not letters. Tokenizing is how text becomes a list of integers — and it's a real design choice.

characters in—
tokens out—
chars / token—
vocabulary size—

Every model reads integers. Before a single number-crunching step can happen, text has to be cut into pieces called tokens and each piece swapped for its id (a row number in a fixed vocabulary). Generation runs the other way: the model emits ids, and you look them back up into text. How you cut is the whole game — try the three buttons and watch the counts move.

ELI5  Tokens are LEGO bricks for text. You can build every sentence out of tiny single-letter bricks (lots of them), out of huge whole-word bricks (a giant bin of them, and none for words you've never seen), or out of medium bricks where common chunks are one piece and rare words snap together from smaller pieces. The medium option wins.
schemevocabsequence lengththe catch
characterstiny (~100)very longthe model must learn spelling from scratch; long sequences are expensive
wordshuge (100k+)shortany word not in the vocab is a dead OOV — type "dragonrider" in word mode
subword (BPE)tuneable (~50k)mediumsegmentation can be unintuitive and fragments rare scripts — but the best all-round choice, and the standard

How BPE builds its bricks. Byte-Pair Encoding starts with nothing but individual characters, then repeatedly finds the most frequent adjacent pair in the training text and glues it into a new token — over and over. Frequent letter pairs become chunks, frequent chunks become syllables, and the most common whole words end up as single tokens. The merges this page learned, in order, from its fantasy corpus:

So a common word like the collapses to one token, while a coinage like dragonrider splits into familiar pieces — no word built from the known alphabet is ever truly unknown, because worst case it falls back to single characters. (This toy only knows a–z, so it marks other symbols with ?; real models avoid even that by working from raw bytes — see the note below.)

text → cut into tokens → [ids] → (model) → [ids] → look up → text  — tokenization is the doorway in and out; everything between is just integers.
Under the hood (deep): real GPTs use byte-level BPE — the atoms aren't characters but the 256 raw bytes, so any input (emoji, code, other languages) is representable and there is literally no OOV. GPT-2/3 ship a ~50,257-token vocabulary; the leading space is part of a token (" the" ≠ "the"), which is why spacing matters. Tokenization also explains famous quirks: models miscount the letters in a word because they never see letters, only chunks, and arithmetic is hard partly because numbers tokenize erratically. Google's WordPiece (BERT) and SentencePiece are close cousins of the algorithm above.

historyHistorical perspective

The algorithm powering modern tokenizers began life as data compression. In 1994 Philip Gage published "A New Algorithm for Data Compression" describing byte-pair encoding: replace the commonest pair of bytes with an unused byte, repeat. It sat in compression toolboxes for two decades until, in 2016, Sennrich, Haddow and Birch repurposed it for neural machine translation to handle rare words — "Neural Machine Translation of Rare Words with Subword Units" — and subword tokenization swept the field. It's a rare case of a humble compression hack becoming foundational infrastructure for AI. (The deep reason it works is the same one Shannon found: language is full of predictable, repeated chunks, and a good code spends its short symbols on the frequent ones.)
Carry this out: tokenizing turns text into a list of integer ids from a fixed vocabulary, and back again. Characters give a tiny vocab but long sequences; whole words give short sequences but a huge vocab and unknown-word holes; subword BPE is the compromise that powers real models. Next: those ids are still just arbitrary numbers — Lesson 06 gives each one a meaning.