family · architecture
masked image modeling (MAE)
BERT's fill-in-the-blank idea applied to pixels — hide most of an image's patches and train a vision transformer to reconstruct them. A simple, scalable way to pretrain vision encoders without labels (MAE, BEiT).
encoder+decoder + masked
▸ open this recipe in the builder
BERT showed that hiding ~15% of a sentence’s tokens and training a model to fill them back in produces representations that transfer to almost any language task. Masked image modeling asks the obvious follow-up: does the same trick work on pixels? The answer is yes — with one striking twist. Language is information-dense, so masking 15% makes a hard puzzle; images are spatially redundant, so MAE has to mask 75% of the patches before reconstruction becomes a task worth learning from. Solving it forces the model to internalize objects, shapes, and scene structure rather than interpolating from neighboring pixels.
The result is one of the simplest recipes for pretraining a vision backbone without labels: no contrastive pairs, no hand-designed augmentations, no tokenizer — just hide most of the image and reconstruct it.
How it works
MAE is an asymmetric autoencoder built from vision transformer parts.
- Patchify and mask. Split the image into patches as a ViT would, then randomly drop 75% of them.
- Encode only what’s visible. The encoder — a full-size ViT — processes just the remaining 25% of patches. Skipping the masked positions entirely is the efficiency trick: the expensive network touches a quarter of the input, making pretraining 3×+ faster.
- Decode the full image. A lightweight decoder receives the encoded visible patches plus shared learnable mask tokens (with positional encodings saying where the holes are) and regresses the raw pixels of the masked patches. The loss is mean squared error on masked positions only.
- Throw the decoder away. After pretraining, only the encoder survives — it becomes the backbone that gets fine-tuned for classification, detection, segmentation, or used as the vision encoder in larger systems.
BEiT, the other founding formulation, masks patches the same way but predicts discrete visual token ids (from a pretrained image tokenizer) instead of raw pixels — literally BERT’s classification-over-vocabulary objective, where MAE swaps in pixel regression and drops the tokenizer dependency.
Relation to other pretraining
- Versus contrastive methods (CLIP-style or SimCLR-style): masked modeling needs no negative pairs and no augmentation engineering, and it scales the way masked language modeling does — more data and bigger ViTs keep helping.
- Versus JEPA: I-JEPA keeps the mask-and-predict structure but moves the target into latent space — predict the representation of hidden patches, not their pixels — arguing that reconstructing exact textures wastes capacity on unpredictable detail. MAE is the pixel-space baseline that JEPA defines itself against.
Where you’ll see it
MAE pretraining (and its video extension, VideoMAE) is a standard recipe for initializing ViT backbones that later get fine-tuned on downstream vision tasks or slotted into vision-language models. When a paper says a vision encoder was “self-supervised pretrained,” masked image modeling is one of the two recipes it usually means — the other being contrastive.
Related concepts
- bert — the masked-prediction objective this family transplants from language
- vision transformer — supplies the patch structure and the encoder backbone
- autoencoders — MAE is literally a masked, asymmetric autoencoder
- encoder / decoder — the asymmetric pair: big encoder kept, small decoder discarded
- JEPA — same masking idea, but predicting latents instead of pixels