module

transformer block

The repeating unit of every transformer — attention then feed-forward, each wrapped in a residual connection and normalization. Stack N of them and you have an encoder or a decoder.

Strip any transformer — GPT, BERT, a ViT — down to its skeleton and you find one small circuit repeated over and over: an attention layer, then a feed-forward layer, each wrapped in a residual connection and normalization. That four-part unit is the transformer block. Architectural differences between famous models are mostly outside the block (how many, what masking, what objective); the block itself has barely changed since 2017 — a strong hint that this particular division of labor is doing something right.

How it works

A block receives one vector per position and returns one vector per position — same shape in, same shape out, which is what makes stacking trivial.

1. Attention sublayer — the only place positions interact. Each position queries the others and mixes in what’s relevant: information routing.
2. Feed-forward sublayer — the same two-layer MLP applied to every position independently, no cross-talk: information processing, and where most of the parameters (and, evidence suggests, most stored facts) live.
3. Residuals + normalization around each — the input of each sublayer is added back to its output, so a block computes x + attention(x) then x + ffn(x) rather than replacing x. Modern stacks normalize before each sublayer (pre-LN), which keeps hundred-layer training stable.

The residual view is the most useful mental model: a residual stream flows through the network unchanged by default, and each sublayer reads from it and writes small updates into it. Attention moves information between positions’ streams; the FFN refines each stream in place.

One block, three roles

Stacking N blocks and choosing an attention mask is what differentiates the major architectures:

• Encoder — blocks with unmasked, bidirectional attention; every position sees every other.
• Decoder — blocks with a causal mask (positions only look left), plus, in seq2seq models, a third sublayer that cross-attends to the encoder’s output.
• ViT — exactly the encoder block, fed image patches instead of tokens.

Related concepts

• attention mechanisms — the routing sublayer
• feed-forward networks — the processing sublayer
• residual connections — what makes deep stacks of blocks trainable, and the source of the residual-stream view
• layer normalization — the stabilizer; pre-LN vs post-LN placement is the block’s one big historical revision
• encoder / decoder — what N stacked blocks become, depending on the mask