# Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

> Research note on Monkey, a large multimodal model designed for high-resolution image handling and enhanced text description generation.

- Authors: Hunter Heidenreich
- Published: 2023-11-10
- Updated: 2026-03-04
- Paper: Monkey Paper <https://arxiv.org/abs/2311.06607>
- Paper: Monkey CVPR 2024 <https://openaccess.thecvf.com/content/CVPR2024/html/Li_Monkey_Image_Resolution_and_Text_Label_Are_Important_Things_for_CVPR_2024_paper.html>
- Code: Monkey GitHub <https://github.com/Yuliang-Liu/Monkey>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a model proposal paper that introduces "Monkey," a large multimodal
    model (LMM) specifically designed to handle higher resolution images and
    generate enhanced text descriptions.
-   It emphasizes technical improvements in image resolution handling and
    contextual description generation within multimodal frameworks, which are
    critical for applications like image captioning and visual question
    answering (VQA).
-   It validates these improvements through extensive empirical evaluations
    across various vision-language tasks.

## What is the motivation?

-   The paper addresses the limitations of current large multimodal models,
    particularly their struggles with high-resolution inputs and complex scene
    understanding.
-   High-resolution handling is essential because it improves the model's
    ability to capture finer details, crucial for tasks requiring detailed
    image-text association.
-   Monkey's design enables higher-resolution inputs without costly retraining,
    making it a resource-efficient alternative with potential impacts across
    vision-language applications in fields like accessibility, AI-driven content
    creation, and enhanced human-computer interaction.

## What is the novelty?

-   **Related work** includes recent advancements in LMMs, such as Qwen-VL,
    PaLI, and other vision-language models like BLIP-2, which attempt to address
    similar multimodal challenges.
-   Existing models use large-scale datasets like LAION and COYO but often have
    limited success with high-resolution details due to simple captions and
    lower input resolutions.
-   **Contribution**: Monkey introduces a patch-based approach for handling
    higher resolutions (up to 1344×896) and a multi-level description generation
    method that creates richer image-text pairs. This two-fold approach enhances
    contextual understanding in scene-text associations and improves performance
    across various vision-language tasks.

## What experiments were performed?

-   The model theoretically optimizes large input resolution handling by
    dividing images into patches processed independently by a static visual
    encoder, incorporating LoRA adjustments to preserve resolution-specific
    details.
-   **Experiments**: Extensive empirical results on 18 datasets show Monkey
    surpassing other LMMs in tasks like image captioning, general VQA,
    scene-text VQA, and document-oriented VQA.
    -   Monkey achieves significant performance improvements over models like
        GPT4V and Qwen-VL, particularly in dense-text VQA tasks.
    -   Ablation studies indicate the model's success in leveraging high input
        resolution and LoRA modules effectively, showing clear benefits in
        accuracy and computational efficiency over traditional interpolation
        techniques.

## Dataset

-   Uses 1.44 million image-text pairs from publicly available datasets,
    covering tasks like Image Captioning, General VQA, Scene Text-centric VQA,
    and Document-oriented VQA.
-   Includes 427k image-text pairs bootstrapped from CC3M using a multi-level
    description method to enrich captions, improving text-image associations.
-   Example datasets include COCO for general image captions, VizWiz and OKVQA
    for VQA, and DocVQA and ChartQA for document-oriented VQA, among others.

## Methodology Highlights

-   **High-Resolution Handling**: Employs a sliding window attention mechanism
    over image patches and a LoRA-modified ViT encoder to support input
    resolutions up to 1344×896.
-   **Image Patch Strategy**: Divides high-resolution images into uniform
    patches, supplemented by a global image version, with a static resampler
    across all patches and the global image.
-   **Multi-Level Description Generation**: Leverages a multi-model approach
    using segmentation, captioning, and detection models (e.g., BLIP2, PPOCR,
    SAM, ChatGPT) to create detailed, contextual captions.
-   **Architecture Components**:
    -   **Vision Encoder**: ViT-bigG from OpenCLIP for processing patches.
    -   **LLM**: Qwen-VL, interfaced with 256 learnable queries per crop for
        enhanced spatial and contextual understanding.
    -   **LoRA**: Applied with rank 16 for attention and 32 for MLP modules to
        update ViT weights efficiently.

## Training Details

-   **Optimizer**: AdamW with a peak learning rate of 1e-5, cosine decay, and
    100-step warmup.
-   **Batch Size**: 1024 with a weight decay of 0.1 to prevent overfitting.
-   **Compute Requirements**: Training takes 40 A800 GPU days for a single epoch
    on 896×896 images.
-   **Model Size**: Total parameters are 9.8B, combining 7.7B for the LLM, 1.9B
    for ViT, 117M for LoRA, and 90M for the resampler.