# ALBEF: Align before Fuse: Vision and Language Representation Learning with Momentum Distillation > Research note on ALBEF, a vision-language pre-training framework that aligns image and text representations before fusing them using momentum distillation. - Authors: Hunter Heidenreich - Published: 2021-07-15 - Updated: 2026-03-04 - Paper: ALBEF paper - Paper: NeurIPS proceedings - Code: salesforce/ALBEF - Model: ALBEF (4M) - Model: ALBEF (14M) - Source: Bevaya Labs, --- ## What kind of paper is this? A vision-language representation learning paper proposing a new framework (ALBEF) to improve vision-and-language pre-training. It focuses on introducing new techniques (Align Before Fuse, momentum distillation) and achieving state-of-the-art results on a set of multimodal tasks. This is a small idea with evaluation paper. It introduces several new tricks and techniques to improve vision-and-language pre-training and evaluates them on a variety of tasks. ## What is the motivation? Motivation: The paper addresses the limitations in current Vision-and-Language Pre-training (VLP) methods, including: - Misaligned features: Image features and word tokens are unaligned, making it challenging to learn image-text interactions. - Resource-Intensive Object Detection: Most existing methods require bounding box annotations and high-resolution images, which are computationally expensive. - Noisy Data: Web data used in pre-training is noisy, leading to suboptimal model performance. The authors propose ALBEF to align image and text representations before fusing them using cross-modal attention, making learning more efficient and robust, especially with noisy data. - Potential Impact: ALBEF aims to outperform existing vision-language models on various downstream tasks like image-text retrieval, visual question answering (VQA), and visual reasoning (NLVR2). The model eliminates the need for bounding box annotations and improves inference speed. Its novel method (momentum distillation) also enhances learning from large, uncurated web datasets. ## What is the novelty? Related Work: - Existing VLP methods such as LXMERT, UNITER, OSCAR, etc., use multimodal encoders with object detectors, which have high computational costs and performance limitations when handling noisy web data. - Recent methods like [CLIP](2021_02_clip.md) and [ALIGN](2021_02_align.md) that use separate encoders for image and text with a contrastive loss but struggle with complex image-text interactions. Contribution: - ALBEF introduces the concept of aligning before fusing the image and text features to improve interaction modeling in multimodal tasks. - It proposes momentum distillation (MoD) to improve learning in noisy environments, offering theoretical insights on mutual information maximization in this context. - The model eliminates the need for bounding boxes or high-resolution images and achieves state-of-the-art performance on tasks like image-text retrieval, VQA, and NLVR2. ## What experiments were performed? Theory: - The paper provides a theoretical framework for understanding ALBEF's success, viewing it through the lens of mutual information maximization. Different training tasks (ITC, MLM, MoD) are framed as techniques to generate different views of an image-text pair. Experiments: - ALBEF achieves state-of-the-art results across multiple downstream vision-language tasks (e.g., image-text retrieval, VQA, and NLVR2). For example, ALBEF outperforms previous models like VILLA by 2.37% on VQA and 3.84% on NLVR2. - It also demonstrates faster inference speed and better performance compared to models like CLIP and ALIGN, even when pre-trained on significantly smaller datasets. ## Dataset ALBEF is pre-trained on a combination of image-captioning datasets to cover a wide variety of image-text pairs. The pre-training datasets include: - **Conceptual Captions 3M (CC3M)**: A large-scale dataset with 3 million image-text pairs, focusing on high-diversity web-crawled images and corresponding descriptions. - **Conceptual Captions 12M (CC12M)**: A noisier and larger version of CC3M, containing 12 million image-text pairs, providing a broader but less curated data source. - **SBU Captions**: A dataset containing image-text pairs scraped from Flickr, known for its diverse visual contexts. - **COCO (Common Objects in Context)**: A well-established dataset consisting of images depicting everyday scenes with object annotations and captions. - **Visual Genome**: A dataset rich in image-region annotations and dense captions, adding granular context to the image-text learning process. In total, the training dataset comprises: - **4 million unique images** - **5.1 million image-text pairs** For scaling experiments, the authors also incorporate the **Conceptual Captions 12M (CC12M)** dataset, bringing the total number of images up to **14.1 million**. This larger dataset, while noisier, demonstrates the scalability of ALBEF to web-scale data, showcasing improved performance on downstream tasks. ## Methodology Highlights #### Image-Text Contrastive (ITC) Loss - **Purpose**: Aligns unimodal representations (image and text features) to improve multimodal learning. - **Mechanism**: Enforces that the visual and textual embeddings are jointly predictive of their association, using a contrastive loss that aligns them before they are fused in the multimodal encoder. - **Benefits**: - Makes it easier for the multimodal encoder to fuse aligned features, enhancing cross-modal interactions. - Encourages better understanding of the semantic spaces for both the unimodal encoders (image and text). - Learns a common low-dimensional space for embedding images and texts, facilitating downstream tasks like image-text retrieval. #### Momentum Distillation (MD) - **Purpose**: Helps the model learn from noisy web-based image-text pairs by generating pseudo-targets that provide alternative reasonable outputs. - **Mechanism**: - During training, a **momentum version** of the model is maintained by taking moving averages of its parameters. - This momentum model generates pseudo-targets for the main model, which provides additional supervision beyond the noisy annotations. - Encourages the model to not overfit to web annotations and produce reasonable alternative outputs for tasks like Masked Language Modeling (MLM) and Image-Text Matching (ITM). #### Architecture - **Image Encoder**: A 12-layer **ViT-B/16** (Vision Transformer), pre-trained on ImageNet-1K, with 85.8M parameters. - **Text Encoder**: A 6-layer transformer initialized from the first 6 layers of **BERT-base** (from the BERT-base model). - **Multimodal Encoder**: Another 6-layer transformer initialized from the last 6 layers of **BERT-base**. - **Total Parameters**: Approximately 200M parameters (~85.8M for ViT-B/16 and 123.7M from BERT-base). #### Pre-training Objectives - **Image-Text Contrastive (ITC) Loss**: Aligns the unimodal representations by calculating softmax-normalized similarity scores between paired image and text representations. This loss is minimized to enhance alignment. - **Masked Language Modeling (MLM)**: A standard task where random tokens in the input text are masked, and the model must predict the masked tokens using both image and text context. - **Image-Text Matching (ITM)**: Predicts whether a given image-text pair is correctly matched or not, using the joint embedding from the multimodal encoder. #### Hard Negative Sampling - **Hard Negatives**: For the ITM task, the model mines hard negatives (semantically similar but incorrect pairs) using contrastive similarity, providing a more challenging training signal. - **In-Batch Sampling**: Negatives are sampled from the same mini-batch based on similarity scores, ensuring efficient training without additional computational overhead. #### Model Summary - **Training Setup**: Pre-training is conducted on large-scale web datasets (e.g., Conceptual Captions, SBU Captions) and domain-specific datasets (e.g., COCO, Visual Genome). The model uses random image crops of resolution 256x256, with RandAugment applied during pre-training. - **Optimizer**: The AdamW optimizer is used with a learning rate schedule including warm-up and cosine decay. ## Training Details - **Training Duration**: The model was trained for **30 epochs**. - **Batch Size**: Each training run used a batch size of **512**. - **Hardware**: - The training was conducted on **8 NVIDIA A100 GPUs**. - This configuration provided sufficient compute for both the image and text encoders as well as the multimodal encoder. - **Optimizer**: - The model used the **AdamW** optimizer with a weight decay of **0.02**. - **Learning Rate Schedule**: - The learning rate was **warmed up to 1e-4** within the first **1,000 iterations**. - Following the warm-up phase, a **cosine decay** schedule reduced the learning rate to **1e-5** by the end of training. #### Additional Details - **Data Augmentation**: During pre-training, random image crops of resolution **256x256** were used, along with **RandAugment** (excluding color changes) to increase image variability. - **Fine-Tuning**: For fine-tuning tasks, the image resolution was increased to **384x384**, and positional encodings for image patches were interpolated. - **Momentum Update**: The momentum parameter for updating the momentum model was set to **0.995**, and the size of the queue used for image-text contrastive learning was **65,536**. - **Distillation Weight**: The distillation weight **α** was linearly ramped from 0 to **0.4** within the first epoch, balancing the pseudo-targets with standard supervision.