# mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

> Research note on mPLUG-Owl2, a multimodal LLM that introduces modality-adaptive modules for improved cross-modality collaboration.

- Authors: Hunter Heidenreich
- Published: 2023-11-07
- Updated: 2026-03-04
- Paper: mPLUG-Owl2 Paper <https://arxiv.org/abs/2311.04257>
- Paper: mPLUG-Owl2 CVPR 2024 <https://openaccess.thecvf.com/content/CVPR2024/html/Ye_mPLUG-Owl2_Revolutionizing_Multi-modal_Large_Language_Model_with_Modality_Collaboration_CVPR_2024_paper.html>
- Code: mPLUG-Owl2 GitHub <https://github.com/X-PLUG/mPLUG-Owl/tree/main/mPLUG-Owl2>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **multi-modal large language model (MLLM) research paper**.
-   It proposes a new architecture (mPLUG-Owl2) focused on **modality
    collaboration**, balancing performance across text and multi-modal tasks.

## What is the motivation?

-   The problem is **modality interference**, where prior models struggle to
    handle multiple modalities without one negatively affecting the other.
-   The importance lies in developing **a unified, generalist model** that can
    excel at both text and multi-modal tasks without fine-tuning for specific
    tasks.
-   The potential impact is a **state-of-the-art model** in multi-modal tasks,
    which could set a new standard for **multi-modal large language models
    (MLLMs)**.

## What is the novelty?

-   Traditional models (e.g., **BLIP-2, MiniGPT-4**) have used cross-modal
    alignment but struggled with modality interference.
-   The contribution of mPLUG-Owl2 is **modality collaboration**, via a
    **modality-adaptive module** that maintains the integrity of both visual and
    text features while sharing parameters for cross-modality interaction.

## What experiments were performed?

-   The paper introduces a **modular network design** using modality-adaptive
    modules to mitigate interference.
-   **Experiments**: The model is evaluated on **8 vision-language benchmarks**
    and achieves **state-of-the-art performance** on most, including
    **Flickr30K, VQAv2**, and others. It also excels in **pure-text tasks**
    (e.g., **MMLU, AGIEval**), indicating that **modality collaboration**
    enhances both text and multi-modal performance.

## Dataset

-   **Pre-training Data**: 400M image-text pairs randomly sampled from **CC3M,
    CC12M, COCO, LAION-EN, COYO, DataComp**.
-   **Instruction Data**:
    -   **Captioning**: TextCaps, COCO.
    -   **Visual Question Answering (VQA)**: VQAv2, OKVQA, OCR-VQA, GQA,
        A-OKVQA.
    -   **Region-aware QA**: RefCOCO, VisualGenome.
    -   **Multi-modal Instruction Data**: LLaVA-instruct-150K.
    -   **Text-only Instruction Data**: ShareGPT-80k, SlimOrca.

## Methodology Highlights

-   **Architecture**:

    -   **Vision Encoder**: ViT-L/14 with a visual abstractor similar to the
        Perceiver Resampler; 6 layers with 64 learnable queries.
    -   **Language Decoder**: LLaMA-2-7B, modified for modality sensitivity in
        **attention layers** (modality-specific layer norms, modality-separated
        key and value matrices).
    -   **Modality-Adaptive Module (MAM)**: Enhances cross-modality interaction
        while preserving unique characteristics of each modality.

-   **Training Stages**:
    -   **Stage 1: Pre-training**:
        -   Language decoder frozen except for **modality-adaptation modules**.
    -   **Stage 2: Joint Instruction Tuning**:
        -   Entire model unfrozen for tuning on both text and multi-modal
            instructions.

## Training Details

-   **Stage 1: Pre-training**:

    -   **Steps**: 42,500 with a batch size of 8,192 (total 348M image-text
        pairs).
    -   **Optimizer**: AdamW with a peak learning rate of 1e-4, 1k warmup steps,
        and cosine decay.
    -   **Layer-wise Learning Rate Decay**: 0.9 applied to vision encoder layers
        to preserve low-level visual features.

-   **Stage 2: Instruction Tuning**:
    -   **Epochs**: 1 with a batch size of 256.
    -   **Learning Rate**: 2e-5 with maintained layer-wise decay.
    -   **Resolution Increase**: Images upsampled from 224x224 to 448x448 for
        better OCR and fine-grained visual tasks.