# MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

> Research note on MM1, Apple's multimodal LLM family providing empirical design guidelines for architecture, data, and training configurations.

- Authors: Hunter Heidenreich
- Published: 2024-03-14
- Updated: 2026-03-04
- Paper: MM1 Paper <https://arxiv.org/abs/2403.09611v2>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a research paper focused on building and improving Multimodal Large
    Language Models (MLLMs).
-   It covers empirical evaluations of architectural components and data
    configurations to achieve optimal few-shot performance across benchmarks.

## What is the motivation?

-   The paper aims to address the lack of transparency and comprehensive design
    guidance in developing performant MLLMs.
-   By studying architecture components and pre-training data configurations,
    the authors hope to establish guiding principles that outlast specific model
    implementations.
-   They emphasize the importance of systematic ablation studies to improve
    few-shot and zero-shot multimodal performance.

## What is the novelty?

-   The authors use a series of ablations across different architectures, visual
    encoders, vision-language connectors, and data mixes to determine optimal
    configurations.
-   They pre-train smaller models for initial tests, then scale up to larger
    models to confirm findings, using pre-training data that mixes image-caption
    pairs, interleaved image-text, and text-only documents.
-   For supervised fine-tuning (SFT), they collect diverse vision-language
    datasets and use specialized training recipes, including varying image
    resolutions and model scales up to 64B parameters.

## What experiments were performed?

-   Image resolution and the number of visual tokens significantly impact model
    performance, with the architecture of the vision-language connector having a
    minor role.
-   Interleaved image-text data is crucial for few-shot and text-only
    performance, while caption data enhances zero-shot performance.
-   Synthetic caption data (VeCap) improves few-shot learning, showing a 2–4%
    increase in performance across tasks.
-   Their final model family, MM1, outperforms or matches state-of-the-art
    multimodal models in few-shot learning, maintaining robust language
    understanding capabilities after pre-training.

They benchmark pre-trained MM1 models on captioning and VQA tasks, comparing
their performance to other multimodal models such as Flamingo, IDEFICS, and
Emu2. Supervised fine-tuning (SFT) is evaluated across 12 multimodal benchmarks
to validate few-shot performance gains. The authors include both quantitative
and qualitative evaluations to demonstrate capabilities like counting objects,
performing OCR, and multi-image reasoning. They also employ qualitative examples
to show MM1's capacity for chain-of-thought reasoning and multi-image contextual
understanding.

## What are the outcomes/conclusions?

-   This paper provides empirical design guidelines for developing efficient
    MLLMs, offering a "recipe" for configuring data types, image resolutions,
    and encoder-decoder connections.
-   It introduces MM1, a family of MLLMs that achieve state-of-the-art
    performance in few-shot multimodal tasks and presents a scalable
    pre-training approach for future large multimodal models.
-   The insights on architecture and data configurations establish foundational
    lessons that may influence future multimodal model training, extending the
    body of knowledge on multimodal pre-training.

## Dataset

-   **Pre-training Data Composition**:
    -   45% captioned images
    -   45% image-text interleaved documents
    -   10% text-only data
-   **Special Data Types**:
    -   **Synthetic Data**: Boosts few-shot performance
    -   **Interleaved Data**: Essential for few-shot and text-only performance
    -   **Captioning Data**: Critical for zero-shot performance

## Methodology Highlights

-   **Base Model Architecture**:
    -   Image Encoder: ViT-L/14 and ViT-H, pre-trained on DFN-5B and VeCap-300M
        datasets at high resolutions (336x336 and 378x378).
    -   Vision-Language (VL) Connector: Convolutional abstractor with 144 image
        tokens; tested alternative VL connectors like average pooling and
        attention pooling.
    -   Language Model: 1.2B parameter decoder-only LM for core processing.
-   **Pre-training Ablations**:
    -   **Image Resolution & Model Size**: Key factors impacting performance,
        with higher resolutions and larger models yielding better results.
    -   **Data Mix**: Interleaved image-text data boosts text-only and few-shot
        capabilities; captioning data aids zero-shot.
-   **Supervised Fine-Tuning**:
    -   Uses ~1M SFT samples mined from pre-training data; all model parameters
        unfrozen during fine-tuning.

## Training Details

-   **Batch Size**: 512
-   **Sequence Length**: 4096 tokens, accommodating up to 16 images per sequence
-   **Training Steps**: 200k steps, equating to 100B tokens processed
-   **Resolution Scaling Techniques**:
    -   Positional embedding interpolation and sub-image decomposition for
        higher-resolution adaptation