# mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

> Research note on mPLUG-Owl, a modularized training paradigm for equipping LLMs with multimodal abilities through a two-stage training process.

- Authors: Hunter Heidenreich
- Published: 2023-04-27
- Updated: 2026-03-04
- Paper: mPLUG-Owl Paper <https://arxiv.org/abs/2304.14178>
- Code: X-PLUG/mPLUG-Owl <https://github.com/X-PLUG/mPLUG-Owl/tree/main/mPLUG-Owl>
- Model: MAGAer13/mplug-owl-llama-7b-ft <https://huggingface.co/MAGAer13/mplug-owl-llama-7b-ft>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a paper proposing a **novel modularized training paradigm** for
    equipping large language models (LLMs) with **multimodal abilities**.
-   It presents **mPLUG-Owl**, a system that enhances LLMs' visual understanding
    while maintaining their text generation capabilities through a two-stage
    training process.

## What is the motivation?

-   The paper aims to **address the limitations** of current multimodal LLMs,
    which often struggle with aligning vision and language tasks due to frozen
    visual models or inefficient alignment techniques.
-   The impact lies in providing a framework that **supports multiple
    modalities** (text, vision) concurrently and shows improved **instruction
    understanding, multi-turn conversation, and knowledge reasoning** over
    competing models.

## What is the novelty?

-   Related work includes **Visual ChatGPT**, **MM-REACT**, **BLIP-2**,
    **MiniGPT-4**, and **LLaVA**, all of which attempt to integrate vision and
    language but face issues with efficiency, alignment, or modularity.
-   This paper's contribution is the **modularized training paradigm** that
    enables **better multimodal alignment** and **fine-tuning** via **LoRA**. It
    also introduces **OwlEval**, an evaluation set for visually-related
    instruction tasks.

## What experiments were performed?

-   **Experiments** show that mPLUG-Owl outperforms existing models like
    MiniGPT-4 and LLaVA in **instruction understanding**, **visual reasoning**,
    and **multi-turn dialogue**.
-   The paper provides **quantitative** evidence via **OwlEval** (a custom
    evaluation set), where mPLUG-Owl achieves **higher ratings** across multiple
    visually-related tasks, including **OCR**, **knowledge transfer**, and
    **multi-turn conversations**.

## Dataset

-   **Pre-training datasets**:
    -   **LAION-400M**, **COYO-700M**, **Conceptual Captions**, and **COCO** for
        image-captioning tasks.
    -   Provides diverse and large-scale image-text pairs for teaching the model
        visual knowledge and alignment.
-   **Instruction tuning datasets**:
    -   **LLaVA data** for multimodal instruction tasks.
    -   **Unimodal instructions** sourced from **Alpaca**, **Vicuna**, and
        **Baize**.

## Methodology Highlights

-   **Architecture**:

    -   **Vision encoder**: **ViT-L/14** for visual feature extraction,
        pretrained using CLIP.
    -   **LLM decoder**: **LLaMA-7B**, with an option for LLaMA-2 weights.
    -   **Visual abstractor**: Aggregates visual features using a learnable
        resampler module, providing higher semantic representations.

-   **Training paradigm**:
    -   **Stage 1: Pre-training**:
        -   Frozen LLM, trainable vision encoder (ViT) and abstractor module.
        -   Focuses on aligning visual and textual representations.
    -   **Stage 2: Joint instruction tuning**:
        -   LLM weights trained with **LoRA**, freezing visual components.
        -   Fine-tuned on both unimodal and multimodal instructions.

## Training Details

-   **Stage 1 (Pre-training)**:

    -   **Batch size**: 2.1 million tokens.
    -   **Steps**: 50k updates, consuming ~104 billion tokens.
    -   **Optimizer**: AdamW, learning
    -   **Warmup**: 2k steps, followed by cosine decay for learning rate.
    -   **Image resolution**: 224x224 pixels.

-   **Stage 2 (Instruction tuning)**:
    -   **Batch size**: 256.
    -   **Steps**: 2k updates.
    -   **Learning rate**: 2e-5.