# InfMLLM: A Unified Framework for Visual-Language Tasks

> Research note on InfMLLM, a multimodal large language model framework with a novel pool-adapter for vision-language tasks including visual grounding.

- Authors: Hunter Heidenreich
- Published: 2023-11-11
- Updated: 2026-03-04
- Paper: InfMLLM Paper <https://arxiv.org/abs/2311.06791>
- Code: INF-MLLM GitHub <https://github.com/infly-ai/INF-MLLM/tree/main/INF-MLLM1>
- Model: InfMLLM Model Zoo <https://github.com/infly-ai/INF-MLLM/tree/main/INF-MLLM1#model-zoo>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **model development and evaluation paper**.
-   It proposes **InfMLLM**, a multimodal large language model (MLLM) framework
    aimed at improving performance in vision-language tasks.

## What is the motivation?

-   The paper aims to **extend the capabilities of large language models**
    (LLMs) to **vision-language tasks** such as image captioning, visual
    question answering (VQA), and visual grounding.
-   The motivation is to create a **versatile, general-purpose multimodal
    assistant** by training models to handle **various modalities** (e.g.,
    images, text) more effectively.
-   The impact lies in advancing the field of multimodal models, contributing a
    **new framework (InfMLLM)** that achieves state-of-the-art results in
    multiple benchmarks, potentially improving **real-world applications** of
    MLLMs.

## What is the novelty?

-   The paper builds on existing work in **LLMs** like GPT-3, as well as
    **multimodal models** like BLIP-2, Qwen-VL, and LLaVA.
-   Existing approaches such as **Q-Former and Perceiver** struggle with tasks
    that require **spatial relationships** (e.g., visual grounding).
-   This paper contributes a **new visual adapter (pool-adapter)** that
    preserves positional information better, improving performance in tasks like
    visual grounding.
-   The proposed **three-stage training scheme** (alignment pretraining,
    multitask finetuning, and instruction tuning) helps to efficiently finetune
    models for multimodal tasks.

## What experiments were performed?

-   **Experiments**: InfMLLM was evaluated on benchmarks for **VQA, image
    captioning, visual grounding**, and other vision-language tasks.
    -   The model achieves **state-of-the-art or near-SOTA performance** on many
        benchmarks, particularly excelling in **visual grounding** tasks.
    -   Ablation studies show that increasing the number of **visual
        embeddings** improves performance, especially for **visual grounding**.
-   **Key findings**: The introduction of the **pool-adapter** significantly
    enhances the ability to retain positional information, leading to better
    results in tasks requiring spatial reasoning.

## Dataset

-   **Stage 1 (Pretraining)**: Uses **weakly labeled image-text pairs** from
    publicly available datasets such as **CC3M**, **CC12M**, and **LAION-115M**.
-   **Stage 2 (Multitask Finetuning)**: Uses a **uniform mixture** of tasks
    (VQA, captioning, and visual grounding) with datasets like:
    -   **VQA**: VQAv2, OK-VQA, AOK-VQA, GQA, TextVQA, OCR-VQA
    -   **Captioning**: COCO, TextCaps
    -   **Grounding**: RefCOCO, RefCOCO+, RefCOCOg
-   **Stage 3 (Instruction Tuning)**: **Instructional data** from **LLaVA 1.5**
    (665k instruction samples).

## Methodology Highlights

-   **Vision Encoder**: Uses **ViT-g/14 from EVA-CLIP** with the last layer and
    class token discarded for more efficient feature extraction.
-   **Pool-Adapter (VL Connector)**: A novel **pooler adapter**:
    -   Retains **locality of image representations** using local pooling
        operations.
    -   Applies a **2-layer MLP** to project pooled image features into the
        language space.
-   **LLM**: Utilizes **Vicuna-7B** as the core LLM.
-   **Training Scheme**:
    -   **Stage 1**: Pretraining with only the **pool-adapter unfrozen**, using
        captioning data.
    -   **Stage 2**: Multitask finetuning with VQA, captioning, and grounding
        data; **ViT, pool-adapter, and QV matrices** are unfrozen.
    -   **Stage 3**: Instruction tuning with the LLM fully finetuned; **ViT
        remains frozen**.

## Training Details

-   **Optimizer**: **AdamW** with **cosine decay** learning rate scheduler.
-   **Batch Sizes**:
    -   **Stage 1**: 1024
    -   **Stage 2**: 512
    -   **Stage 3**: 128
-   **Learning Rates**:
    -   **Stage 1**: 2e-4
    -   **Stage 2**: 1e-5
    -   **Stage 3**: 2e-5
-   **Steps**: 80k / 40k / 10k (Note: Possible typo, might be 20k for Stage 1).
-   **Image Resolutions**:
    -   **Stage 1**: 224
    -   **Stage 2 & 3**: 448
-   **Compute**:
    -   **Stage 1**: 32×A800 GPUs
    -   **Stage 2**: 32×A800 GPUs
    -   **Stage 3**: 16×A800 GPUs

## Follow-Up: Inf-MLLM2

In a
[tech report](https://github.com/infly-ai/INF-MLLM/blob/main/INF-MLLM2/docs/tech_report.pdf),
Infly.AI has introduced **Inf-MLLM2**, a successor to Inf-MLLM. It lacks a full
paper, but the tech report highlights several changes:

-   Dynamic image resolution handling: The technique of encoding a global image
    (that's sized down) and local patches (that are sized up) to handle varying
    resolutions. This enables the model to support resolutions of upto 1344x1344
    pixels.
-   Enhanced OCR: The model can handle document processing tasks quite well.
    This seems to be driven by changing the data mixture and by handling larger
    image sizes.