# GLIPv2: Unifying Localization and Vision-Language Understanding

> Research note on GLIPv2, an extension of GLIP that unifies localization and vision-language understanding through grounding-based pre-training.

- Authors: Hunter Heidenreich
- Published: 2022-06-12
- Updated: 2026-03-04
- Paper: GLIPv2 paper <https://arxiv.org/abs/2206.05836>
- Paper: NeurIPS proceedings <https://proceedings.neurips.cc/paper_files/paper/2022/hash/ea370419760b421ce12e3082eb2ae1a8-Abstract-Conference.html>
- Paper: OpenReview <https://openreview.net/forum?id=wiBEFdAvl8L>
- Code: microsoft/GLIP <https://github.com/microsoft/GLIP>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This paper introduces GLIPv2, a strong extension of the original
    [GLIP model](2021_12_glip.md) that unifies localization and vision-language
    understanding.
-   It's a simple evaluation paper that focuses on how this model performs a
    variety of tasks, including object detection and phrase grounding, and that
    grounding-based pre-training can improve performance on general
    vision-language tasks.

## What is the motivation?

-   The motivation is to **improve vision-language models** by incorporating
    grounding-based pre-training, which can enhance the model's ability to
    understand visual concepts and their textual descriptions.
-   GLIP already showed promise in object detection and phrase grounding tasks,
    and GLIPv2 aims to **extend these capabilities** to broader vision-language
    tasks, potentially improving performance across various multimodal
    applications.

## What is the novelty?

-   As an extension of GLIP, GLIPv2 builds on the idea of grounding-based
    pre-training to **enhance vision-language understanding**.
-   All other encoder-based models like [CLIP](2021_02_clip.md),
    [ALIGN](2021_02_align.md), and MDETR are relevant, but GLIPv2's focus on
    grounding and localization makes it unique in its approach to
    vision-language tasks.

## What experiments were performed?

-   Models ar evaluated on COCO object detection, LVIS, and other downstream
    tasks. They're also tested on vision-language tasks like VQA.
-   **Results**:
    -   GLIPv2 shows **improved performance** over the original GLIP model in
        object detection and phrase grounding tasks.
    -   The model also demonstrates **strong performance** on vision-language
        tasks like VQA, showcasing the benefits of grounding-based pre-training
        in enhancing multimodal understanding.

## Dataset

-   Pre-training data
    -   Detection: Objects365, COCO, LVIS, OpenImages, VG, ImageNetBoxes
    -   Grounding: GoldG
    -   Caption: Cap4M, CC15M+SBU, Cap4M

## Methodology Highlights

-   Compared with GLIP:
    -   GLIP showed grounded pre-training improves localization. GLIPv2 shows
        that grounded pre-training can improve general vision-language tasks.
    -   GLIPv2 adds a new loss term: inter-image region-word contrastive loss,
        which helps the model learn better visual-semantic alignments. They
        argue this can be viewed as the region-level equivalent of the
        word-level contrastive loss in CLIP.

## Training Details

-   GLIPv2-Tiny
    -   Based on Swin-T
    -   32 GPUs (type not specified) w/ batch size of 64
    -   Different learning rates for language backbone (1e-5) and vision
        backbone (1e-4)
    -   Learning rate reduced 0.1 at fixed points
    -   330,000 steps total
    -   Max token length of 256
    -   MLM is turned off for an additional 300,000 steps
-   GLIPv2-Base
    -   Based on Swin-B
    -   64 GPUs (type not specified) w/ batch size of 64
    -   All params use 1e-4 learning rate
    -   Same learning rate schedule as Tiny
    -   1 million steps total
    -   Max token length of 256
    -   MLM is turned off for an additional 500,000 steps
-   GLIPv2-Huge
    -   Based on CoSwin-Huge
    -   64 GPUs (type not specified) w/ batch size of 64
    -   Most everything is the same as Base, but MLM is not turned off for a
        final stage