# Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

> Research note on Qwen-VL, a large-scale vision-language model series with a three-stage training pipeline achieving leading performance in fine-grained visual understanding.

- Authors: Hunter Heidenreich
- Published: 2023-08-24
- Updated: 2026-03-04
- Paper: Qwen-VL Paper <https://arxiv.org/abs/2308.12966>
- Code: QwenLM/Qwen-VL <https://github.com/QwenLM/Qwen-VL>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a research paper that introduces and evaluates a new series of
    large-scale vision-language models (LVLMs), named Qwen-VL.
-   It proposes a novel model for multimodal tasks involving both text and image
    inputs, targeting tasks like image captioning, question answering, and
    fine-grained visual understanding.

## What is the motivation?

-   The motivation is to enhance large language models (LLMs) with the ability
    to process and understand visual information, expanding their utility beyond
    text-based tasks.
-   The impact is significant, as the Qwen-VL models achieve state-of-the-art
    results across various vision-language benchmarks and have multilingual
    capabilities, which could accelerate research and practical applications in
    fields requiring multimodal understanding.

## What is the novelty?

-   Related work includes LVLMs like Flamingo, BLIP-2, and Kosmos-2, which also
    aim to integrate vision and language processing.
-   The main contribution of this paper is the Qwen-VL series, which introduces
    a robust visual receptor, a three-stage training pipeline, and fine-grained
    visual understanding. The models surpass previous open-source LVLMs in tasks
    like image captioning, visual question answering, and object grounding.

## What experiments were performed?

-   **Experiments**: The paper presents results on a wide range of benchmarks,
    including image captioning (Flickr30K, Nocaps), general VQA (VQAv2, OKVQA,
    GQA), and text-oriented visual understanding (TextVQA, DocVQA).
-   The results demonstrate that Qwen-VL and Qwen-VL-Chat models outperform
    other generalist models on several benchmarks, achieving leading performance
    in fine-grained tasks like text-reading and object localization.
-   The paper also shows that Qwen-VL-Chat excels in real-world user behavior
    tests, outperforming similar models in instruction-following tasks.

## Dataset

-   **Stage 1**:
    -   Used a combination of publicly available English and Chinese caption
        datasets totaling 5 billion image-text pairs, cleaned to 1.4 billion.
    -   English data sources: LAION-en, LAION-COCO, DataComp, Coyo, CC12M, CC3M,
        SBU, COCO.
    -   Chinese data includes private in-house data and LAION-zh.
-   **Stage 2**:
    -   Multitask pre-training datasets include:
        -   Captioning data (same as Stage 1 but reduced in size).
        -   Visual Question Answering (VQA) datasets: GQA, VGQA, VQAv2, DVQA,
            OCR-VQA, DocVQA, TextVQA, ChartQA, AI2D.
        -   Grounding and reference tasks: GRIT, Visual Genome, RefCOCO,
            RefCOCO+, RefCOCOg.
        -   OCR data from synthetic English and Chinese datasets (SynthDoG-en,
            Common Crawl PDF & HTML).

## Methodology Highlights

-   **Architecture**:
    -   Base LLM: Qwen-7B, visual encoder: ViT-bigG/14 from OpenCLIP.
    -   Introduced a **VL Adapter**: A cross-attention layer compressing image
        features to 256 fixed-length vector representations.
    -   Special tokens used for bounding boxes (<bbox>), image content (<img>),
        and text references (<ref>) in input/output.
-   **Training Pipeline**:
    -   **Stage 1**: Vision-language alignment with frozen Qwen-7B and unfrozen
        visual encoder and adapter, trained on weakly-labeled image-text pairs.
    -   **Stage 2**: Multitask pre-training with high-resolution image inputs,
        covering tasks like captioning, VQA, grounding, and OCR.
    -   **Stage 3**: Instruction tuning for Qwen-VL-Chat, fine-tuning on
        dialogue tasks involving multi-image comprehension, grounding, and
        multilingual support.

## Training Details

-   **Stage 1**:
    -   Optimizer: AdamW with a cosine learning rate schedule (max lr: 2e-4, min
        lr: 1e-6), 500-step warmup, 5e-2 weight decay.
    -   50k steps with a batch size of 30,720, processing 1.5 billion image-text
        samples (~500 billion image-text tokens).
-   **Stage 2**:
    -   Similar AdamW optimizer setup, with 19k steps and 400-step warmup.
-   **Stage 3**:
    -   Instruction tuning with mixed multi-modal and pure text dialogue data.
    -   Emphasis on multi-image and fine-grained comprehension for dialogue
        models.