# BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

> Research note on BLIP, a vision-language framework that unifies understanding and generation tasks with a novel dataset bootstrapping method (CapFilt).

- Authors: Hunter Heidenreich
- Published: 2022-01-28
- Updated: 2026-03-04
- Paper: BLIP paper <https://arxiv.org/abs/2201.12086>
- Paper: PMLR proceedings <https://proceedings.mlr.press/v162/li22n.html>
- Code: salesforce/BLIP <https://github.com/salesforce/BLIP>
- Model: BLIP models collection <https://huggingface.co/collections/Salesforce/blip-models-65242f40f1491fbf6a9e9472>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **vision-language pre-training (VLP) framework** paper.
-   It proposes a **unified model** for both understanding-based and
    generation-based tasks in vision-language domains, introducing a new
    architecture and data bootstrapping method.

## What is the motivation?

-   The paper addresses the limitation that existing VLP models excel at either
    **understanding** or **generation** tasks, but not both.
-   It tackles the issue of **noisy web data** commonly used for pre-training,
    proposing a better method to handle it.
-   The potential impact includes **state-of-the-art performance** on a wide
    range of vision-language tasks, improving both understanding and generation
    capabilities while making models **more generalizable** to video-language
    tasks.

## What is the novelty?

-   Existing VLP models are either encoder-based or encoder-decoder-based, and
    they often use **noisy web data** for training.
-   This paper contributes by proposing the **Multimodal Mixture of
    Encoder-Decoder (MED)** architecture, which handles both understanding and
    generation tasks.
-   It introduces **CapFilt**, a new method for dataset bootstrapping by using a
    captioner to generate synthetic captions and a filter to remove noisy
    captions, improving the quality of the training data.

## What experiments were performed?

-   **Experiments** show BLIP achieves **state-of-the-art performance** on tasks
    like image-text retrieval, image captioning, visual question answering, and
    video-language tasks.
-   The results indicate **substantial performance gains**: +2.7% on image-text
    retrieval, +2.8% on captioning, and +1.6% on VQA, with strong generalization
    to video-language tasks in a **zero-shot** manner.

## Dataset

-   **14M image-text pairs** from multiple sources: COCO, Visual Genome,
    Conceptual Captions (CC), CC12M, and SBU Captions.
-   **LAION dataset** (115M images) used with very noisy web text, filtered
    through CapFilt.
-   The dataset is bootstrapped by generating synthetic captions using a
    **captioner** and filtering noisy captions using a **filter**, both
    fine-tuned on high-quality human-annotated data (COCO).

## Methodology Highlights

-   **Multimodal Mixture of Encoder-Decoder (MED)**: A multitask architecture
    with three modes: unimodal encoder, image-grounded text encoder, and
    image-grounded text decoder.
-   **CapFilt**: A novel dataset augmentation strategy combining a **captioner**
    for generating synthetic captions and a **filter** for removing noisy
    captions. Both initialized from pre-trained weights and fine-tuned
    individually.
    -   **Captioner**: Generates one synthetic caption per image using **nucleus
        sampling** for more diverse captions.
    -   **Filter**: Removes noisy image-text pairs using contrastive pairing and
        matching objectives.
-   Pre-training objectives:
    -   **Image-text contrastive (ITC)** loss for vision-language alignment.
    -   **Image-text matching (ITM)** loss for fine-grained multimodal
        representation.
    -   **Language modeling (LM)** loss for image-grounded text generation.

## Training Details

-   **ViT-B/16** and **ViT-L/16** architectures used for image encoding, trained
    on **224x224 resolution** during pre-training, **384x384** during
    fine-tuning.
-   Training conducted on **2 nodes, each with 16 GPUs**.
-   **AdamW optimizer**, trained for **20 epochs**.
    -   Batch sizes: **2880 (ViT-B)** and **2400 (ViT-L)**.
    -   Learning rate: linear warm-up, decayed by 0.85 with peak rates of **3e-4
        (ViT-B)** and **2e-4 (ViT-L)**.
    -   Weight decay: **0.05**.