microllm · curriculum

From "what is a language model" to a tiny GPT that writes game characters — each lesson is a thing you poke at. ELI5 on the maths, deep on the CS.

Module 0 · Intuitions (no jargon, all interactive)

00 What a language model does
It guesses the next word. A live next-word predictor + temperature. live 01 Knobs & loss
A machine full of knobs, and one number that says how wrong it is. Drag two knobs to fit a line. live 02 Rolling downhill (gradient descent)
Training = nudge every knob downhill. Watch a ball find the valley; break it with a big learning rate. live 03 One neuron, by hand
A sigmoid neuron learns a function from examples — loss + gradient descent in the smallest real net. live 04 Autograd
Gradients for free. Step a backward pass through the computation graph and watch the slopes flow. live

Module 1 · Text becomes numbers

05 Tokenization
Type a sentence, watch it shatter into tokens and ids. Characters vs words vs a live-trained BPE. live 06 Embeddings
Every token becomes a point in meaning-space; tap for neighbours, and watch king − man + woman = queen. live

Module 2 · The one big idea — attention

07 Bigram vs. context
Why "last word only" isn't enough — slide the context window and watch a prediction snap into focus. live 08 Self-attention
Each token looks back over the others. The live causal attention matrix — tap a word, see what it reads. live 09 The transformer block
Attention + MLP + residual + layernorm = a GPT layer. Clickable diagram + a parameter calculator. live

Module 3 · Build & train the real thing

10 Assemble the GPT
Stack the blocks. The whole architecture on one screen. soon 11 Train on TinyStories
Watch the loss fall; read what it writes at each checkpoint. soon 12 Sampling — temperature & top-k
The knobs that turn probabilities into prose. soon

Module 4 · Into the game

13 Finetune for characters & quests
Bend the model to your game's format. soon 14 Run it in the browser
Export to ONNX; generate content live inside your WebGL game. soon