TL;DR
- ALIGN trains a dual-encoder model on 1.8 billion raw image–alt-text pairs, using only minimal frequency-based filtering — no human curation, no caption quality checks.
- The key finding: scaling lightly filtered web data can compensate for noise on the evaluated tasks; the paper does not establish that noise is irrelevant for every dataset or model.
- The architecture is deliberately simple: EfficientNet for images, BERT for text, L2-normalized embeddings projected into a shared space, trained with an in-batch InfoNCE contrastive loss.
- The resulting embedding space supports zero-shot image–text retrieval and classification, with performance competitive with CLIP on standard benchmarks, despite the dramatically different data strategy.
Scale over cleaning
Training good vision–language models has traditionally required painstaking data curation: human-written captions (COCO), filtered crawls with heuristics (WIT for CLIP), or expert annotations. Curated datasets top out around 400 million pairs before the effort becomes prohibitive.
ALIGN takes the opposite bet. Google's image search index contains billions of images, each accompanied by an alt-text attribute written by web authors. Most of these alt-texts are useless for visual grounding: filenames like IMG_2043.jpg, marketing copy, or single generic words. But the dataset is enormous. ALIGN applies only the lightest possible filter — removing unsuitable image sizes and aspect ratios, overly frequent images or texts, rare text tokens, and texts outside 3–20 words; evaluation-image near-duplicates are also excluded — and retains 1.8 billion pairs. No human review, no quality scoring.
The empirical question is how much additional data compensates for weaker captions. The answer depends on model capacity and the evaluation task, rather than on a universal noise threshold. The exclusions above are described in paper Section 3.
The curves above are schematic, with invented quality scores and no measured crossover. The actual size-versus-quality comparison in Section 6, Table 10 uses B7 + BERT-base: 3M noisy pairs underperform cleaned CC-3M, while 12M noisy pairs exceed it on the reported MS-COCO retrieval and ImageNet KNN metrics. Larger-scale ablations also show that model capacity affects the benefit from more data.
A simple dual encoder
ALIGN's architecture is a textbook dual encoder. There is nothing structurally novel — the contribution is entirely in the data strategy and scale.
Image encoder. EfficientNet-L2, trained from scratch. Global pooling produces the image embedding, which is L2-normalized; the text projection matches its dimension.
Text encoder. BERT-Large, also trained from scratch, with a 100k wordpiece vocabulary built from the training corpus. The [CLS] token representation is linearly projected and L2-normalized into the same shared space.
Training objective. For a batch of N image–text pairs, the model computes cosine similarities between all N² combinations. The in-batch InfoNCE loss maximizes the N diagonal (matched) similarities and minimizes the N² − N off-diagonal (mismatched) ones, symmetrically in both image→text and text→image directions. A learned temperature scales the logits.
The absence of cross-modal fusion is deliberate. Cross-attention layers (as in later VLMs) produce richer joint representations but require every image–text pair to be encoded together at inference — prohibitively expensive for retrieval at scale. With independent encoders, image and text embeddings can be precomputed and cached separately; retrieval reduces to a dot-product lookup.
For executable normalized dual-encoder loss and retrieval mechanics, see the CLIP implementation. This is a shared mathematical pattern, not the ALIGN training recipe: ALIGN uses EfficientNet + BERT trained from scratch and lightly filtered alt-text, while CLIP uses different encoders and a concept-based data collection strategy (ALIGN Sections 2 and 4.1).
A shared embedding space
The learned embedding space is what makes ALIGN practically useful beyond training. Because both modalities land in the same L2-normalized space, any image embedding can be directly compared with any text embedding via cosine similarity.
Image-to-text retrieval: given a query image, rank all text candidates by cosine similarity and return the top matches.
Text-to-image retrieval: given a text query, rank all image candidates by cosine similarity.
Zero-shot classification: treat each class label (or a templated prompt like a photo of a {class}) as a text query, encode all class texts once, then assign the image to the nearest class text embedding.
On the MS-COCO retrieval benchmark, ALIGN achieves state-of-the-art image-to-text and text-to-image recall at the time of publication, outperforming models trained on orders of magnitude less data. On ImageNet zero-shot classification, Table 4 reports 76.4% for ALIGN and 76.2% for CLIP with prompt ensembling; this is a benchmark-specific comparison. The retrieval scores above are illustrative lookup values, not trained ALIGN outputs.
Why it mattered
ALIGN and CLIP appeared in 2021 as closely related approaches. Together they established the now-standard paradigm for vision–language pre-training: dual encoders, web-scale supervision, contrastive loss. ALIGN's specific contribution to this consensus was demonstrating the data side: lightly filtered data can support strong representations when scaled, under the tested recipes.
The practical consequence was significant. Curation pipelines are bottlenecks. The results show a route that avoids extensive caption curation, while still requiring the stated image/text exclusions, evaluation deduplication, model capacity and substantial training resources. This lowered the barrier to building large-scale multimodal systems and influenced how subsequent models (Florence, BASIC, SigLIP at scale) approached data collection.
ALIGN also demonstrated that EfficientNet, a purely convolutional architecture, could compete with ViT-based models for vision–language representation at the time, though subsequent work has shifted the field toward transformer image encoders.
Related Reading
- CLIP — the concurrent dual-encoder model from OpenAI, trained on a curated 400M-pair dataset; together ALIGN and CLIP defined the paradigm
- SigLIP — replaces InfoNCE with a pairwise sigmoid loss that removes the all-gather requirement, enabling contrastive training at smaller batch sizes
- CoCa — extends the dual-encoder framework with a generative captioning objective, combining contrastive and generative supervision in one model