family · architecture

VQ-VAE

An autoencoder with a discrete codebook bottleneck — images become grids of tokens, which is what lets transformers generate them (VQ-VAE, VQGAN, DALL-E 1).

encoder+decoder + reconstruction

▸ open this recipe in the builder

A VAE compresses an image into a continuous latent vector. VQ-VAE makes one pointed change: the latent must be discrete — every position in the compressed representation snaps to its nearest entry in a learned codebook of, say, 8,192 vectors. An image stops being a tensor of floats and becomes a small grid of code indices: 32×32 integers instead of 256×256×3 pixels.

Why force that? Because sequences of discrete symbols are exactly what autoregressive transformers eat. Tokenization did this for text by design; VQ-VAE does it for images by construction — it learns the visual vocabulary. That single move is the bridge that let language-model machinery generate pictures.

How it works

The autoencoder stage. A convolutional encoder maps the image to a grid of latent vectors. Each vector is replaced by its nearest neighbor in the codebook (vector quantization), and a decoder reconstructs the image from the quantized grid. Snapping-to-nearest has no gradient, so training uses the straight-through estimator — gradients skip over the quantization step as if it were the identity — plus two auxiliary terms: a codebook loss pulling code vectors toward the encoder’s outputs, and a commitment loss pulling the encoder toward the codes it uses.

The prior stage. The frozen autoencoder turns every training image into a grid of code indices, and a second model — an autoregressive transformer like a decoder-only LM — learns to generate those grids token by token. Sampling means: generate a plausible index grid, then run it through the decoder to get pixels. This two-stage recipe is DALL-E 1.

VQGAN sharpened the recipe by training the autoencoder with an adversarial loss (a GAN discriminator) alongside reconstruction, so aggressive compression still yields crisp decodes — making the token grids small enough for transformers to model at high resolution.

Where you’ll see it

Discrete visual tokens turned out to be infrastructure, not just a generator: BEiT uses them as targets for masked-image pretraining, audio codecs like SoundStream apply the same quantization idea to sound, and multimodal LLMs that emit images do so by predicting visual tokens. The continuous-latent branch of the same idea — a VAE compressor without quantization — underpins latent diffusion (Stable Diffusion’s “latent space” is exactly this).

Sources

browse all sources ›