module

encoder

Reads the whole input at once — bidirectionally — and produces a contextual representation of it. The "understanding" half of the transformer recipe.

An encoder is a stack of identical blocks that turns a raw input sequence into a contextual representation: every position’s vector ends up informed by every other position. Unlike a decoder, it looks at the whole input at once — there is no notion of “past” and “future,” so it can use context from both directions.

How it works

Each encoder block runs two sub-layers: self-attention, which lets every position mix in information from every other position, and a feed-forward network applied identically at each position. Residual connections and layer normalization around each sub-layer keep deep stacks trainable, and positional encodings inject word order, which attention alone cannot see.

Where you’ll see it

Encoder-only stacks power BERT-style models, built to understand text rather than generate it. Paired with a decoder, the encoder is the reading half of encoder–decoder models like the original translation Transformer and T5. A Vision Transformer is an encoder over image patches, and CLIP trains two encoders — one for images, one for text — against each other.

Sources

browse all sources ›