# ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision > Research note on ALIGN, a dual-encoder model that scales vision-language representation learning using 1.8 billion noisy image-text pairs. - Authors: Hunter Heidenreich - Published: 2021-02-11 - Updated: 2026-03-04 - Paper: ALIGN paper - Paper: PMLR proceedings - Source: Bevaya Labs, --- ## What kind of paper is this? This paper proposes a new method in visual and vision-language representation learning using noisy, large-scale data, leveraging a dual-encoder architecture. The paper mainly presents a big idea with evaluation: a scalable approach to representation learning from noisy, uncurated data. ## What is the motivation? - Motivation: - Visual and vision-language representations still rely heavily on curated, human-labeled datasets (e.g., ImageNet, MSCOCO), which are expensive and time-consuming to create. - There is a need to scale up representation learning using larger datasets that don't require this level of manual curation to support improved downstream task performance and enable zero-shot learning. - Potential Impact: - The approach can enable the training of state-of-the-art models with large but noisy data instead of clean, expensive datasets. - This can reduce the cost and time associated with creating labeled datasets and improve the ability of models to generalize to a wide range of tasks, including zero-shot learning and cross-modal retrieval. ## What is the novelty? - Related Work: - Pre-training on large, curated datasets like ImageNet and OpenImages has become the standard for visual representation learning. - Vision-language pre-training has similarly relied on curated datasets like Conceptual Captions and MSCOCO, which are smaller and require human annotation. - [CLIP](2021_02_clip.md) (Radford et al., 2021) is a closely related model that also uses image-text pairs for visual representation learning. - Contribution: - The key contribution is the ALIGN model, which scales representation learning using a noisy dataset of over 1 billion image alt-text pairs without requiring extensive filtering or cleaning. The paper shows that scale can compensate for noise. - ALIGN achieves state-of-the-art results in image-text retrieval benchmarks and zero-shot image classification. ## What experiments were performed? The paper presents experiments on several tasks, including zero-shot image classification, image-to-text and text-to-image retrieval, and transfer learning to ImageNet and fine-grained classification tasks. - ALIGN achieves state-of-the-art results on Flickr30K and MSCOCO benchmarks, outperforming previous models like CLIP and UNITER. - In zero-shot classification, ALIGN performs comparably to CLIP with 76.4% top-1 accuracy on ImageNet. - ALIGN also shows strong performance on transfer learning tasks and smaller fine-grained classification datasets. ## Dataset - The dataset is constructed using a methodology similar to **Conceptual Captions** (Sharma et al., 2018). - It consists of **1.8 billion image-text pairs**, making it one of the largest available datasets for vision-language pre-training. - The dataset is **noisy** due to minimal filtering; only **frequency-based heuristics** are applied to reduce noise. - **Image-based filtering** includes removing low-resolution and pornographic images, as well as duplicates from evaluation sets. - **Text-based filtering** involves removing alt-texts that are too short, too long, or contain rare or irrelevant tokens. - Unlike Conceptual Captions, the dataset avoids extensive cleaning or curation to prioritize scale, demonstrating that **scale compensates for noise** in training. ## Methodology Highlights - **Dual-Encoder Architecture**: - The model uses **separate encoders** for images and text, aligning their embeddings in a **shared latent space** using contrastive learning. - **Image Encoder**: EfficientNet models (e.g., EfficientNet-L2) are used to encode the visual features. - **Text Encoder**: BERT-based models (e.g., BERT-Large) are employed for text embeddings. - **Contrastive Loss**: - A **normalized softmax contrastive loss** is applied, where matched image-text pairs are pushed together in the embedding space, and non-matched pairs are pushed apart. - Both **image-to-text** and **text-to-image** losses are used, making it analogous to a **classification objective** with text embeddings acting as "label" weights. - **In-batch Negatives**: - The contrastive learning relies on **in-batch negatives**, which are all other image-text pairs within the batch. To enhance this, embeddings from **all computing cores** are concatenated to form larger batches. - **Learned Temperature Parameter**: - The softmax temperature, which scales the logits, is **learned** during training instead of being fixed, allowing it to automatically adjust to improve model performance. - **Training Framework**: - The training batch size is massive (up to **16,384** effective batch size), utilizing **1024 TPUv3 cores**. This is critical for scaling the noisy data and ensuring convergence. - The model is trained with **LAMB optimizer** with weight decay, which allows large-batch training and ensures stability. - **Zero-shot Transfer and Fine-tuning**: - ALIGN is evaluated in **zero-shot settings**, where class names are fed into the text encoder, enabling **zero-shot image classification**. - The model performs exceptionally well in both **zero-shot** and **fine-tuned** transfer to downstream tasks, including **Flickr30K** and **MSCOCO** retrieval benchmarks, and classification tasks like **ImageNet**. ## Training Details - **Optimizer**: LAMB optimizer with weight decay set to **1e-5**. - **Batch Size**: **16,384 effective batch size**, utilizing **1024 TPUv3 cores**. - **Learning Rate**: - Warmup for the first **10,000 steps** to a peak learning rate of **1e-3**. - Learning rate is **linearly decayed** to zero over **1.2M steps** (equivalent to **12 epochs**). - **Gradient Accumulation**: The **in-batch negatives** strategy concatenates embeddings from all cores to ensure large batch sizes and efficient contrastive learning. - **Hardware**: Training is done on **1024 TPUv3 cores**, enabling the model to handle the large-scale dataset and optimize across all cores. - **Precision**: Likely using **mixed precision** for efficiency, though not explicitly mentioned (this is often standard in such large-scale models).