primitive

attention

Lets every position look at every other position and mix in what's relevant. The breakthrough that replaced sequential processing with direct connections between any two positions.

Attention lets a neural network focus on the most relevant parts of its input when producing each part of its output. It replaces the step-by-step processing of recurrent networks with a mechanism that directly connects any two positions in a sequence — in parallel, in a single operation.

How it works

Each token produces three vectors: a query (what am I looking for?), a key (what do I contain?), and a value (what information do I carry?). A token’s query is scored against every key, the scores are normalized with a softmax, and the values are summed with those weights — yielding a representation of each token that has mixed in whatever is relevant from the rest of the sequence. Multi-head attention runs this many times in parallel with different learned projections, so different heads can track different relationships — syntax, meaning, proximity. Because the operation itself is order-blind, it is paired with positional encoding to restore word order.

Attention comes in three flavors. Self-attention lets a sequence attend to itself — every position to every other — and powers the encoder. Masked self-attention restricts each position to what came before it, enforcing the left-to-right generation of the decoder. Cross-attention lets a decoder read an encoder’s output — the original 2014 form, invented to fix the bottleneck in sequence-to-sequence models that squeezed a whole input into one fixed-size vector.

Where you’ll see it

Everywhere. Scaled-up attention across many layers is what gives large language models their capabilities; a Vision Transformer applies it to image patches. Its main cost is memory — every position interacting with every other — which drives variants like attention head sharing and motivates attention-free challengers like state space models.

Sources

browse all sources ›