Wrap attention in a few standard parts, stack the result N times, and you have a GPT. Here's the whole architecture on one screen.
Data flows top to bottom through one block. Tap any part to see its job.
The green ＋ are residual adds — the skip-connections that make deep stacks trainable.
token vectors in from embeddings (Lesson 06)
│ ┐ skip: copy the input, untouched
LayerNorm
│ ┊
Multi-Head Self-Attention look back & mix context (Lesson 08)
＋ ┘ x ← x + Attention(LN(x))
│ ┐ skip again, from here
LayerNorm
│ ┊
Feed-Forward (MLP) think per-token: two Lesson-03 layers
＋ ┘ x ← x + MLP(LN(x))
token vectors out → into the next block
Tap a component above.
↑ one block · a GPT is this same block repeated
12× (drag layers below)
Attention alone isn't a network — the block is. Self-attention lets tokens share
information, but it's mostly a weighted average; it doesn't do much "thinking" on its own. So each block pairs
it with a small per-token feed-forward network (an MLP — literally a couple of Lesson-03 layers), and
wraps both in two supporting parts: LayerNorm to keep the numbers well-scaled, and residual adds
that let each sub-layer adjust the signal rather than replace it.
ELI5 Attention is "talk to the other words and gather what's
relevant." The feed-forward net is "now go away and think about it by yourself." A block is one round of gather-
then-think; stacking blocks is doing that over and over, each round working with richer ideas than the last.
The residual connections are quietly the most important part. Each sub-layer computes a
small change and adds it to its input rather than overwriting it, so there's a clean "highway" running
straight through the whole stack. That's what lets gradients (Lesson 04) flow back through dozens of layers
without fading to nothing — the practical reason we can train deep models at all.
And that's the entire architecture. Embed the tokens (05–06), add position, run the stack
of identical blocks, and read off a next-token probability at the end (00). GPT-1 is 12 of these blocks; GPT-3
is 96. At this level the recipe barely changes with scale — stack more broadly-similar blocks and widen them
(later generations did tweak details like norm placement and attention patterns) — which the calculator
above turns into a parameter count.
Under the hood (deep): the MLP expands to 4× the width and back (d → 4d → d) — within
each block it holds about twice the weights of attention, and it's the single biggest slice of the whole model
(see the breakdown bar). Modern blocks put the LayerNorm
before each sub-layer ("pre-norm") for stable training, and often swap plain LayerNorm for RMSNorm and
the MLP's activation for GELU/SwiGLU. The block you see is decoder-only (causal mask, no encoder) — the
GPT lineage; the original 2017 Transformer also had an encoder half for translation. Weight tying reuses the
embedding matrix as the output head, which is why the calculator counts it once.
historyHistorical perspective
The block is a stack of borrowed good ideas. Residual connections come from
ResNet (He, Zhang, Ren, Sun, 2015), which won the 2015 ImageNet challenge and made 100-plus-layer
vision networks practical — the skip-connection trick that also rescued deep language models. Layer
normalization is from Ba, Kiros and Hinton (2016). Vaswani et al. assembled these with attention into
the Transformer in 2017; a year later Radford and colleagues at OpenAI kept only the decoder half,
trained it to predict the next token on a heap of books, and called it GPT — Generative Pre-trained
Transformer. GPT-1 (2018) had 117M parameters in 12 blocks; the recipe never fundamentally changed, it just got
bigger — GPT-3 (2020) is the very same block, 175 billion parameters deep.
Carry this out: a transformer block = LayerNorm → self-attention → residual → LayerNorm
→ feed-forward → residual. Attention shares information between tokens; the MLP processes each; residuals and
norms make the stack trainable. A GPT is this block repeated and a next-token head on top — that's the whole
machine. Module 3 next: actually train one.
Back to curriculum →