# VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts > Research note on VLMo, a unified vision-language model using Mixture-of-Modality-Experts to flexibly function as both dual and fusion encoder. - Authors: Hunter Heidenreich - Published: 2021-11-03 - Updated: 2026-03-04 - Paper: VLMo paper - Paper: NeurIPS proceedings - Code: microsoft/unilm (VLMo) - Model: VLMo-Base - Model: VLMo-Base-Plus - Model: VLMo-Large - Source: Bevaya Labs, --- ## What kind of paper is this? It introduces a novel vision-language pre-trained model (VLMo) that unifies both dual and fusion encoders for vision-language tasks. In this way, it's just a small idea with evaluation (the small idea being the unification of dual and fusion encoders). ## What is the motivation? Motivation: The paper addresses the challenge of handling vision-language tasks with existing architectures. Previous models either adopt dual encoders (efficient for retrieval tasks) or fusion encoders (effective for vision-language classification but computationally expensive). There is no existing model that flexibly adapts both methods for different types of tasks while leveraging large-scale single-modality data. Potential impact: The VLMo model can significantly improve the efficiency and accuracy of vision-language tasks across a variety of settings, particularly classification and retrieval tasks. It also demonstrates how stagewise pre-training on single-modality data (image-only, text-only) can enhance multi-modal performance, which can broaden the applicability and scalability of similar models in other multimodal contexts (e.g., speech, video). ## What is the novelty? Related work: The paper builds on previous vision-language pre-training models such as [CLIP](2021_02_clip.md), [ALIGN](2021_02_align.md), and ViLT, which either use dual encoders or fusion encoders for vision-language tasks. These models optimize performance on vision-language retrieval or classification tasks, but they each have limitations: dual encoders perform shallow modality interaction, while fusion encoders are computationally expensive for large-scale retrieval tasks. Contributions: - The paper proposes a unified model (VLMo) that can function as both a dual encoder and fusion encoder, thus combining the advantages of both architectures. - It introduces a Multiway Transformer with modality-specific experts (vision, language, vision-language) to flexibly handle different inputs (image-only, text-only, image-text pairs). - The stagewise pre-training strategy leverages large-scale image-only and text-only data to improve performance on multimodal tasks. - The model achieves state-of-the-art results on both vision-language classification and retrieval tasks. ## What experiments were performed? Experiments: The model is evaluated on a range of vision-language tasks, including: - Visual Question Answering (VQA) - Natural Language for Visual Reasoning (NLVR2) - Image-Text Retrieval on datasets like MSCOCO and Flickr30K - Vision-only tasks, such as image classification on ImageNet and semantic segmentation on ADE20K. Main results: - VLMo achieves state-of-the-art results on vision-language tasks, outperforming models like UNITER, [ALBEF](2021_07_albef.md), and [SimVLM](2021_08_simvlm.md), especially in classification tasks where deeper cross-modal interactions are needed. - VLMo-Large++ demonstrates strong performance when trained on 1B noisy image-text pairs, particularly on VQA and NLVR2, where it sets new benchmark SOTA. - In retrieval tasks, VLMo performs on par with or better than other models, while offering a faster inference speed than fusion-encoder-based models. ## Dataset Pre-training utilized **four image captioning datasets**: - **Conceptual Captions (CC)** - **SBU Captions** - **COCO** - **Visual Genome (VG)** Total: ~**4M images** and **10M image-text pairs**. ## Methodology Highlights #### Architecture - **Vision input**: ViT-style patch embeddings. - **Text input**: WordPiece tokens, embedded with a max sequence length of 40. - **Initial input**: Concatenated text and vision embeddings. - **Mixture of Modality Experts (Multiway Transformer)**: - Shared self-attention across all modalities. - Modality-specific feed-forward (FF) subunits for vision, language, and vision-language tasks. - **Unimodal support**: Uses modality-specific FFN for vision or text inputs. - **Multimodal support**: Uses unimodal FFN at base layers and multimodal FFN at the top layers. - **Architecture type**: Encoder-based with modality flexibility. - **Model sizes**: - **Base**: 12 layers, 768-dimensional hidden state, 12 attention heads, VL fusion in the last 2 layers. - **Large**: 24 layers, 1024-dimensional hidden state, 16 attention heads, VL fusion in the last 3 layers. #### Methods - **Pre-training tasks** (stagewise): - **Contrastive learning** (ViT-style) between image and text representations. - **Masked Language Modeling (MLM)** using whole-word masking. - **Binary matching** classification for image-text pairs. (see: [ALBEF](2021_07_albef.md#image-text-contrastive-itc-loss)) #### Pre-training Stages 1. **Image-only pre-training**: Initializes the vision expert and shared attention modules. 2. **Text-only pre-training**: Initializes the language expert. 3. **Vision-language pre-training**: Learns cross-modal representations. ## Training Details - **30 epochs**, **512 batch size**, using **8xA100 GPUs**. - **Optimizer**: AdamW with `β1 = 0.9`, `β2 = 0.98`, **0.01 weight decay**. - **Learning rate**: Peak at **2e-4** (base model) / **5e-5** (large model). - **Warmup**: **2.5k linear warmup** steps, followed by **linear decay**.