# BEiT-3: Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

> Research note on BEiT-3, a multimodal foundation model that treats images as a foreign language and uses masked data modeling for vision and vision-language tasks.

- Authors: Hunter Heidenreich
- Published: 2022-08-22
- Updated: 2026-03-04
- Paper: BEiT-3 paper <https://arxiv.org/abs/2208.10442>
- Paper: CVPR proceedings <https://openaccess.thecvf.com/content/CVPR2023/html/Wang_Image_as_a_Foreign_Language_BEiT_Pretraining_for_Vision_and_CVPR_2023_paper.html>
- Code: microsoft/unilm (BEiT-3) <https://github.com/microsoft/unilm/tree/master/beit3>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **methodological and empirical paper**.
-   It introduces BEIT-3, a general-purpose multimodal foundation model and
    evaluates it across multiple vision and vision-language tasks.

## What is the motivation?

-   The paper aims to solve the problem of **unified multimodal modeling** by
    treating images as a foreign language (termed "Imglish").
-   The motivation is to simplify pretraining objectives and improve scalability
    for multimodal models.
-   The potential impact lies in improving the generalization and
    transferability of multimodal models across a variety of vision and
    vision-language tasks.

## What is the novelty?

-   The paper builds on the **masked data modeling** paradigm seen in BERT for
    language and BEIT for images.
-   Related work includes **[CLIP](2021_02_clip.md),
    [SimVLM](2021_08_simvlm.md), CoCa**, and others that focus on image-text
    contrastive learning or multi-task pretraining objectives.
-   The main contribution is introducing **BEIT-3**, a model that scales to
    billions of parameters, uses a simplified pretraining task (masked data
    modeling), and shows strong performance across tasks using only public
    datasets.

## What experiments were performed?

-   **Experiments**: BEIT-3 achieves state-of-the-art results on several
    benchmarks, including **COCO, ImageNet, ADE20K**, and **VQA**, outperforming
    models like **CoCa** and **Florence**.
-   **Theory**: The model demonstrates that treating images as a "language"
    enables more efficient and effective masked modeling, with significant
    performance gains over previous models.
-   Results emphasize the scalability and versatility of BEIT-3 across vision
    and vision-language tasks, using a smaller batch size than previous models.

## Dataset

-   **Multimodal Data**: BEIT-3 is pretrained using approximately 15 million
    images and 21 million image-text pairs from public datasets, including:

    -   **CC12M (Conceptual Captions 12M)**
    -   **CC3M (Conceptual Captions 3M)**
    -   **SBU Captions**
    -   **COCO (Common Objects in Context)**
    -   **Visual Genome**

-   **Vision Data**: The model is also pretrained on **ImageNet-21K**, which
    contains **14 million images** for vision tasks.

-   **Text Data**: For the text-only pretraining, BEIT-3 uses a 160GB corpus
    consisting of:
    -   **English Wikipedia**
    -   **BookCorpus**
    -   **OpenWebText**
    -   **CC-News**
    -   **Stories**

These datasets are all publicly accessible, and no private datasets were used
for pretraining, ensuring academic reproducibility. Despite this, BEIT-3
outperforms other models that utilize in-house data.

## Methodology Highlights

-   **Architecture**:

    -   BEIT-3 uses a **Multiway Transformer** architecture, which includes:
        -   A **shared self-attention module** across modalities (vision,
            language, and vision-language).
        -   A **pool of feed-forward network (FFN) experts** for each modality
            (vision, language, and vision-language), allowing the model to
            capture modality-specific information.
    -   The **giant model** has **1.9 billion parameters**, with separate
        parameter allocations for vision, language, and vision-language experts.

-   **Pre-training**:

    -   The model is pre-trained using **masked data modeling objectives**:
        -   **Masked Language Modeling (MLM)**: Similar to BERT, it masks 15% of
            text tokens and learns to recover the original tokens.
        -   **Masked Image Modeling (MIM)**: Masks 40% of image patches and
            reconstructs discrete visual tokens using a pre-trained image
            tokenizer (VQ-KDCLIP).
        -   **Masked Vision-Language Modeling (MVLM)**: Masks tokens or patches
            in image-text pairs, learning to recover them based on the context
            of both modalities.
    -   The use of **masked objectives** (MLM, MIM, MVLM) allows BEIT-3 to be
        pre-trained with **smaller batch sizes** compared to contrastive-based
        methods (e.g., CLIP, CoCa) that require large batches to contrast pairs.
    -   The simplicity of the **mask-then-predict task** enables easier
        implementation of **gradient accumulation**, unlike contrastive methods.

-   **Training Efficiency**:
    -   The model uses a **1024 sequence length** during pre-training.
    -   Smaller batch sizes make BEIT-3 more efficient to scale, while still
        providing strong generalization for downstream tasks.

## Training Details

-   **Training Steps**: BEIT-3 is pre-trained for **1 million steps**.

-   **Batch Size**: The total batch size is **6144 samples**, composed of:

    -   2048 image samples,
    -   2048 text samples,
    -   2048 image-text pairs.

-   **Optimizer**: The model uses the **AdamW** optimizer with the following
    hyperparameters:

    -   **Peak learning rate**: 1e-3,
    -   **Warmup**: Linear warmup for the first **10,000 steps**,
    -   **Learning rate schedule**: Cosine decay after warmup.

-   **Weight Decay**: A **weight decay** of 0.05 is applied.

-   **Training Time & Hardware**: The pretraining process takes approximately
    **2 weeks** and is conducted on **256 A100 40GB GPUs**.