# KOSMOS-2: Grounding Multimodal Large Language Models to the World

> Research note on KOSMOS-2, a multimodal LLM that extends KOSMOS-1 with grounding capabilities, linking text spans to image regions via location tokens.

- Authors: Hunter Heidenreich
- Published: 2023-06-26
- Updated: 2026-03-04
- Paper: KOSMOS-2 Paper <https://arxiv.org/abs/2306.14824>
- Code: microsoft/unilm (KOSMOS-2) <https://github.com/microsoft/unilm/tree/master/kosmos-2>
- Model: microsoft/kosmos-2-patch14-224 <https://huggingface.co/microsoft/kosmos-2-patch14-224>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **technical research paper** introducing a new **multimodal large
    language model (MLLM)** called **KOSMOS-2**.
-   It extends previous work (KOSMOS-1) by integrating **grounding
    capabilities**, linking language to visual elements (e.g., bounding boxes).

## What is the motivation?

-   The paper aims to address the challenge of **grounding text in the visual
    world**, which improves multimodal interactions.
-   **Motivation**: Enabling **human-AI interaction** by allowing the model to
    understand image regions directly rather than relying on text descriptions.
-   **Potential impact**: Enhances multimodal AI systems by introducing new
    capabilities like **grounded image captioning** and **referring expression
    comprehension**, contributing towards **artificial general intelligence
    (AGI)**.

## What is the novelty?

-   **Related work**: Builds on MLLMs such as **KOSMOS-1**, **Flamingo**, and
    grounding models like **GLIP**.
-   **Contribution**: Introduces the **grounding capability** to MLLMs by
    linking text spans to image regions via location tokens, constructing the
    large-scale **GRIT dataset** of grounded image-text pairs, and expanding the
    applications of multimodal models to **grounded vision tasks**.

## What experiments were performed?

-   **Theory**: The grounding capability is achieved by representing bounding
    boxes as location tokens integrated into the model's language structure.
-   **Experiments**: KOSMOS-2 is evaluated across multiple tasks, including
    **phrase grounding**, **referring expression comprehension**, and **visual
    question answering**.
    -   Results show **strong performance** in multimodal grounding tasks and
        **competitive performance** on vision-language tasks compared to models
        like **MDETR** and **FIBER**.

## Dataset

-   **GRIT (Grounded Image-Text Pairs)**: Consists of **91M images**, **115M
    text spans**, and **137M bounding boxes**.
-   **Mining process**: Dataset created from captioning datasets (e.g.,
    **LAION-2B**, **COYO-700M**).
-   **Grounding**: Text spans are linked to image regions using **bounding
    boxes**, which are converted into sequences of **location tokens**.
-   The dataset supports **multiple bounding boxes** for a single text phrase,
    enabling precise multimodal grounding.

## Methodology Highlights

-   **Input Representation**: Bounding boxes are represented by the **top-left**
    and **bottom-right points**, which are discretized into **patch indices**.
-   Uses a **Markdown-like format**: Phrases are wrapped in special tokens,
    followed by box tokens that enclose the location tokens.
-   **Multibox support**: Handles multiple bounding boxes linked to a single
    phrase.
-   A **special grounding token** is used to signal when the model should ground
    text to the visual world.
-   **Transition from KOSMOS-1 to KOSMOS-2**: KOSMOS-2 integrates **GRIT data**
    with KOSMOS-1's existing multimodal corpora. KOSMOS-2's weights are
    initialized from KOSMOS-1, with new tokens initialized randomly.
-   **Instruction tuning** includes **LLaVA-Instruct** and newly created
    **GRIT-based instructions**.

## Training Details

-   **Training hyperparameters**:
    -   **Steps**: 60,000, with **375 warmup** steps.
    -   **Learning rate**: Peaks at **2e-4**, with linear decay.
    -   **Optimizer**: AdamW with **β = (0.9, 0.98)**, **weight decay** of 0.01.
    -   **Batch sizes**: Text (93), Image-caption pairs (1117), Grounded
        image-text pairs (1117), Interleaved data (47).
-   **Instruction tuning**:
    -   **Steps**: 10,000, with **375 warmup** steps.
    -   **Learning rate**: **1e-5**.
    -   **Batch sizes**: Text instruction (117), Vision-language instruction
        (351), Grounded image-text pair + grounded instruction (1404), Text
        (30), Interleaved data (15).