# BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

> Research note on BLIP-2, a vision-language pre-training method that bridges frozen image encoders and LLMs with a lightweight Querying Transformer (Q-Former).

- Authors: Hunter Heidenreich
- Published: 2023-01-30
- Updated: 2026-03-04
- Paper: BLIP-2 Paper <https://arxiv.org/abs/2301.12597>
- Paper: ICML 2023 Conference Paper <https://proceedings.mlr.press/v202/li23q.html>
- Code: salesforce/LAVIS <https://github.com/salesforce/LAVIS/tree/main/projects/blip2>
- Model: Salesforce/blip2-opt-2.7b <https://huggingface.co/Salesforce/blip2-opt-2.7b>
- Model: Salesforce/blip2-opt-6.7b <https://huggingface.co/Salesforce/blip2-opt-6.7b>
- Model: Salesforce/blip2-flan-t5-xl <https://huggingface.co/Salesforce/blip2-flan-t5-xl>
- Model: Salesforce/blip2-flan-t5-xxl <https://huggingface.co/Salesforce/blip2-flan-t5-xxl>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **vision-language pre-training** paper.
-   It proposes a novel method, **BLIP-2**, for vision-language pre-training by
    using **frozen image encoders** and **frozen large language models (LLMs)**,
    bridging the modality gap with a **lightweight Querying Transformer
    (Q-Former)**.

## What is the motivation?

-   The paper addresses the high **computation cost** of vision-language
    pre-training by leveraging **frozen pre-trained models** instead of full
    end-to-end training.
-   The problem is important because **scaling large multimodal models** has
    become costly, and **BLIP-2** provides a more efficient strategy.
-   The potential impact is that **BLIP-2** can achieve **state-of-the-art
    performance** on various vision-language tasks while being **more
    compute-efficient** than prior methods.

## What is the novelty?

-   The paper builds on work in vision-language models such as **Flamingo**,
    which combine image encoders and large language models, but BLIP-2
    differentiates itself by using **frozen unimodal models**.
-   Existing approaches (e.g., **Flamingo**) rely on an image-to-text generation
    loss but face **modality alignment challenges**.
-   BLIP-2 contributes a **two-stage pre-training** strategy with a **Querying
    Transformer** that efficiently extracts visual features for the LLM without
    requiring end-to-end fine-tuning of the models.

## What experiments were performed?

-   **Experiments**: BLIP-2 achieves **state-of-the-art** performance on tasks
    like **Visual Question Answering (VQAv2)**, **image captioning**, and
    **image-text retrieval**, outperforming models like **Flamingo80B** with
    **fewer trainable parameters**.
-   It excels in **zero-shot** tasks such as **image-to-text generation**,
    demonstrating **visual conversation** and **instruction following**
    capabilities.
-   The method is shown to be both **efficient** (using frozen components) and
    **powerful**, significantly reducing the number of parameters while
    maintaining high performance.

## Dataset

-   Pre-training uses **129M images** from several sources: **COCO**, **Visual
    Genome**, **CC3M**, **CC12M**, **SBU**, and **LAION400M**.
-   Synthetic captions are generated with the **CapFilt** method: **10
    captions** are produced per image using the **BLIP large** model.
-   Captions are ranked based on **image-text similarity** from **CLIP
    ViT-L/14**, and the top 2 captions are retained for training.

## Methodology Highlights

-   **Q-Former**: The core architectural component, with two transformer
    sub-modules sharing self-attention layers.
    -   **Image Transformer**: Cross-attends to frozen image encoder features
        every other block.
    -   **Text Transformer**: Can act as both encoder and decoder, depending on
        the pre-training task.
    -   **Learnable Queries**: 32 queries interact with the frozen image encoder
        and LLM to extract relevant visual features.
-   **Stage 1 (Representation Learning)**: Pre-training without using the LLM.
    -   **Objectives**: Image-Text Contrastive (ITC), Image-Text Matching (ITM),
        Image-Grounded Text Generation (ITG).
-   **Stage 2 (Generative Learning)**: Connects the frozen image encoder and
    Q-Former to a frozen LLM (either decoder-only or encoder-decoder) for
    **vision-to-language generation**.
    -   **Language modeling**: Causal or prefix-based, depending on the LLM.

## Training Details

-   **Stage 1 (Representation Learning)**:
    -   **Steps**: 250k steps.
    -   **Batch Size**: 2320 for ViT-L, 1680 for ViT-g.
-   **Stage 2 (Generative Learning)**:
    -   **Steps**: 80k steps.
    -   **Batch Size**: 1920 for OPT, 1520 for FlanT5.
    -   **Precision**: FP16 for ViTs and OPT, **BF16** for FlanT5.
-   **Optimizer**: **AdamW** with weight decay of **0.05**.
    -   **Learning Rate**: Peak **1e-4** with 2k-step warmup, cosine cooldown to
        a minimum of **5e-5**.
-   **Hardware**: Training done on a single **16xA100 (40GB)** machine.
    -   **Training Time**: ViT-g + FlanT5-XXL takes **6 days** for Stage 1 and
        **3 days** for Stage 2.
-   **Data Augmentation**: Random resized cropping and horizontal flipping.