# mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video

> Research note on mPLUG-2, a modularized multi-modal foundation model that addresses modality entanglement across text, image, and video tasks.

- Authors: Hunter Heidenreich
- Published: 2023-02-01
- Updated: 2026-03-04
- Paper: mPLUG-2 Paper <https://arxiv.org/abs/2302.00402>
- Paper: ICML 2023 Conference Paper <https://proceedings.mlr.press/v202/xu23s.html>
- Code: X-PLUG/mPLUG-2 <https://github.com/X-PLUG/mPLUG-2>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **multi-modal pretraining model paper**.
-   It presents a **new modularized framework** (mPLUG-2) that addresses
    multi-modal tasks, unifying text, image, and video understanding and
    generation.
-   The paper solves practical AI problems across multiple domains, leveraging
    both **encoder-based and sequence-to-sequence generation paradigms**.

## What is the motivation?

-   The paper tackles the problem of **modality entanglement** when training
    models on multiple modalities like text, image, and video simultaneously.
-   It highlights the need for **modality collaboration** while preventing
    **interference across modalities** during training.
-   The proposed modularized framework allows for **flexible, efficient
    scaling** across various tasks.
-   The potential impact is significant in **multi-modal AI**, where it achieves
    state-of-the-art or competitive performance on over 30 tasks.

## What is the novelty?

-   The paper builds on **multi-modal foundation models** like CLIP, OFA, and
    BEIT-3, but introduces a more **modularized architecture**.
-   Prior work either focuses on **sequence-to-sequence generation** or
    **encoder-based instance discrimination**, but mPLUG-2 leverages **both
    approaches** through a flexible network.
-   It introduces a **dual-vision encoder** and **universal layers** to handle
    **multi-modal collaboration** and disentanglement.
-   The paper contributes by designing a framework that **selects and combines
    modules** for downstream tasks, improving both **performance** and
    **zero-shot transferability**.

## What experiments were performed?

-   **Empirical study** shows state-of-the-art performance on tasks like
    **MSRVTT video QA** and **video captioning**, achieving new records with
    **smaller model size and data scale**.
-   Achieved **48.0 top-1 accuracy** on MSRVTT video QA and **80.3 CIDEr** for
    video captioning, demonstrating strong results across vision, language, and
    multi-modal benchmarks.
-   The **zero-shot transferability** is also impressive, especially on
    **vision-language** and **video-language tasks**.

## Dataset

-   **Image captioning**: COCO, Conceptual Captions 3M (CC3M), Conceptual
    Captions 12M (CC12M), SBU, Visual Genome.
-   **Video-text**: WebVid-2M.
-   **Text-only**: WikiCorpus (20GB) and cleaned Common Crawl (350GB), mirroring
    the C4 dataset.

## Methodology Highlights

-   **Dual Vision Encoder**: A module that supports both images and videos,
    capturing spatial and temporal features using ViT-B/16 and ViT-L/14 models.
-   **Text Encoder**: BERT-based unimodal encoder, initialized from pre-trained
    checkpoints.
-   **Universal Module**: Aligns vision and language features using text-aware
    vision tokens and vision-aware text tokens, reducing computational
    complexity and aligning modalities in a shared semantic space.
-   **Fusion Module**: Combines self-attention, cross-attention, and
    feed-forward networks (FNNs) to update and refine cross-modal
    representations.
-   **Shared Decoder**: Supports text generation for both uni-modal and
    multi-modal tasks.
-   **Pre-training Losses**:
    -   **Language Loss**: Masked Language Modeling (MLM) for text encoding.
    -   **Image-Text Contrastive (ITC)**: Aligns image and text representations.
    -   **Image-Text Matching (ITM)**: Matches image and text pairs, improving
        multi-modal understanding.
    -   **Instruction-based LM**: Handcrafted instructions for task and modality
        discrimination.

---

## Training Details

-   **Pre-training**: 30 epochs.
-   **Batch size**:
    -   **Base model**: 1024 on 8 NVIDIA A100 GPUs.
    -   **Large model**: 512 on 16 NVIDIA A100 GPUs.
-   **Optimizer**: AdamW with a weight decay of 0.02.
-   **Learning Rate**:
    -   Warmup for 5000 steps, followed by cosine decay.
    -   **Base model**: Peak learning rate of 1e-4.
    -   **Large model**: Peak learning rate of 5e-5.
-   **Data Augmentation**: Random cropping, horizontal flip, and sparse frame
    sampling for videos.