modifier

attention head sharing (GQA/MLA)

Shares key/value projections across query heads to shrink the KV cache at inference — a modifier on the attention sublayer, not a peer architecture.

Attention head sharing is a modifier on the attention mechanisms sublayer, not a separate architecture family. Standard multi-head attention (MHA) gives every query head its own key and value projections — maximum expressiveness, but maximum memory during generation. At inference, these projections are cached for every past token (the KV cache), growing proportionally with context length and batch size. Head sharing reduces that cache by reusing key/value projections across multiple query heads.

How it works

In standard MHA with H query heads, the cache stores H key vectors and H value vectors per token per layer. The sharing variants cut that number:

Multi-Query Attention (MQA) uses a single shared KV head for all query heads — 1/H the cache, with a small quality penalty.

Grouped-Query Attention (GQA) splits query heads into G groups, each sharing one KV head. This interpolates between MHA and MQA: near-MHA quality with substantially reduced cache. GQA is the production standard — Llama 2/3, Mistral, and Gemma all use it.

Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, takes a different approach: compress all KV information into a single low-rank latent vector per token, then reconstruct full KV heads from it on the fly. Only the latent vector is cached, achieving roughly a 93% reduction in KV cache size versus MHA — and matching or exceeding MHA quality, because compression pressure encourages a more efficient KV representation.

The intuition: diversity among query heads matters more than diversity among key and value heads. Each query asks a different question, but the “facts” it looks up can be shared without much loss.

Where you’ll see it

GQA is the default in virtually every large language model released since 2023. MLA is the frontier variant, used by DeepSeek-V2/V3 alongside Mixture of Experts — MoE reduces computation, MLA reduces memory, the two addressing complementary bottlenecks.

Related concepts

• attention mechanisms — the sublayer this modifier applies to
• mixture-of-experts — a complementary modifier: MoE applies to the feed-forward sublayer
• large language models — GQA is the default for production LLMs