# KOSMOS-2.5: A Multimodal Literate Model

> Research note on KOSMOS-2.5, a multimodal literate model for document-level machine reading with spatially-aware text recognition and image-to-markdown generation.

- Authors: Hunter Heidenreich
- Published: 2023-09-20
- Updated: 2026-03-04
- Paper: KOSMOS-2.5 Paper <https://arxiv.org/abs/2309.11419>
- Code: microsoft/unilm (KOSMOS-2.5) <https://github.com/microsoft/unilm/tree/master/kosmos-2.5>
- Model: microsoft/kosmos-2.5 <https://huggingface.co/microsoft/kosmos-2.5>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **research paper** proposing a new multimodal literate model,
    KOSMOS-2.5, for document-level machine reading.
-   It focuses on extending large language models to handle text-intensive
    images through two tasks: spatially-aware text recognition and
    image-to-markdown generation.
-   The paper introduces a pre-trained model and its fine-tuned variant,
    KOSMOS-2.5-CHAT, with applications in document understanding.

## What is the motivation?

-   The problem is enabling multimodal models to handle text-intensive images,
    including document-level reading and structural understanding.
-   Current OCR and multimodal models fail to capture reading order and
    structure comprehensively, which are critical for accurate document
    understanding.
-   The potential impact is significant for Artificial General Intelligence
    (AGI) as it advances models' ability to understand complex text-rich images
    like academic papers, receipts, and web pages.

## What is the novelty?

-   Related work includes OCR-based models (like Tesseract) and multimodal
    models (like GPT-4o and Donut), which either focus on line-level recognition
    or structured parsing in limited domains.
-   Existing approaches fail to provide comprehensive document-level reading,
    especially in diverse domains.
-   The paper contributes KOSMOS-2.5, a model capable of handling document-level
    text recognition and image-to-markdown tasks, and introduces two new
    benchmarks (OCREval, MarkdownEval) for evaluation.

## What experiments were performed?

-   The paper presents a unified Transformer-based framework with a ViT-based
    vision encoder and a language decoder. It introduces a shared architecture
    for handling both spatially-aware text recognition and markdown generation.
-   **Experiments**: KOSMOS-2.5 achieves superior results on the OCREval and
    MarkdownEval benchmarks, outperforming models like GPT-4o and Nougat in
    document-level text recognition and markdown generation tasks.
-   The model excels across diverse document categories and tasks, demonstrating
    impressive performance relative to models with significantly more
    parameters.

## Dataset

-   **Text-intensive image data**:
    -   IIT-CDIP: 27.6M scanned pages.
    -   arXiv papers: 20.9M pages.
    -   PowerPoint slides: 6.2M pages.
    -   General PDFs: 155.2M pages crawled from the web.
    -   Web screenshots: 100M pages.
-   **Markdown data**:
    -   GitHub README files: 2.9M.
    -   DOCX files: 1.1M from the web.
    -   LaTeX papers: 3.7M pages from arXiv.
    -   HTML pages: 6.3M converted to markdown.
-   **Total**: ~300M pages of text-intensive image data, ~14M pages in markdown
    format.
-   Data filtering includes language detection, deduplication, and removing
    malformed data.
-   Uses augmentation techniques such as rotations, blurring, upscaling, and
    downscaling from TrOCR.

## Methodology Highlights

-   **Model architecture**:
    -   Vision Transformer (ViT) encoder combined with a Transformer language
        decoder.
    -   Perceiver resampler compresses visual features into a constant-length
        sequence.
    -   Supports flexible x/y coordinate handling for bounding boxes, compatible
        with Pix2Struct.
    -   Vocab size increased to 108,481 tokens (from ~64k in earlier models).
-   **Core tasks**:
    -   Document-level text recognition, assigning spatial coordinates to text.
    -   Image-to-markdown generation to preserve document structure and style.
    -   Bounding boxes rendered by referencing x/y coordinates for each text
        line.
    -   Markdown output enhances structural sensitivity, especially for tables
        and complex formats.

## Training Details

-   **Training setup**:
    -   Total steps: 200k, with 375 steps warmup.
    -   Batch size: 1024.
    -   Learning rate: 2e-4 with linear decay.
    -   Standard settings for AdamW optimizer: weight decay, betas (0.9, 0.98),
        and epsilon.
-   **Training dataset**:
    -   Total of ~260B tokens.
    -   Two-stage training: first on layout-based data (100k steps), followed by
        combined data (140k steps).