# SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

> Research note on SPHINX, a multimodal LLM that introduces joint mixing of weights, tasks, and visual embeddings for improved vision-language understanding.

- Authors: Hunter Heidenreich
- Published: 2023-11-13
- Updated: 2026-03-04
- Paper: SPHINX Paper <https://arxiv.org/abs/2311.07575>
- Code: SPHINX GitHub <https://github.com/Alpha-VLLM/LLaMA2-Accessory?tab=readme-ov-file>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **multi-modal model development** paper.
-   It proposes a new architecture, **SPHINX**, for multi-modal large language
    models (MLLMs) that integrates visual and language understanding through a
    novel joint mixing approach.

## What is the motivation?

-   The paper aims to enhance the capabilities of large language models (LLMs)
    to perform **multi-modal tasks** by better integrating **vision and language
    data**.
-   It solves the problem of limited visual instruction-following capacity in
    existing MLLMs by introducing techniques for **visual embedding**, **task
    mixing**, and **model weight integration**.
-   The impact is the development of a **versatile model** that excels across a
    wide range of vision-language tasks, including fine-grained visual
    understanding and **multi-tasking**.

## What is the novelty?

-   Existing work includes multi-modal models like **LLaVA**, **MiniGPT-4**, and
    **BLIP-2**, which often freeze LLMs or use basic visual encoders.
-   SPHINX contributes by **unfreezing LLMs** during pre-training, introducing
    **weight mixing** from real-world and synthetic data, **task mixing** for
    diverse visual tasks, and using **multiple visual encoders** for richer
    embeddings.
-   It also addresses **high-resolution image processing** challenges, offering
    superior performance on various benchmarks.

## What experiments were performed?

-   The main innovation is the joint mixing approach that combines **visual
    embeddings**, **model weights**, and **tasks**, aiming for enhanced
    multi-modal capabilities.
-   **Experiments**: SPHINX achieves **state-of-the-art** results on multiple
    benchmarks (e.g., **MMBench**, **VQAV2**, **RefCOCO**) for tasks like
    **visual question answering** and **human pose estimation**.
-   The model outperforms strong baselines on key tasks, demonstrating
    **improved fine-grained visual perception** and reasoning abilities.

## Dataset

-   **Stage 1 (Pretraining)**:
    -   **LAION-400M** and **LAION-COCO** for vision-language alignment
        (image-caption data).
    -   **RefinedWeb** (text-only dataset) used to maintain text reasoning
        capabilities.
-   **Stage 2 (Finetuning)**:
    -   **VQA**: VQAv2, GQA, OKVQA, A-OKVQA, OCRVQA for visual question
        answering.
    -   **Other tasks**: Multi-object detection, relation reasoning, human pose
        estimation, document layout detection.

## Methodology Highlights

-   **Two-stage training procedure**:
    -   **Stage 1 (Pretraining)**:
        -   Unfrozen LLM trained on real data, followed by fine-tuning on
            synthetic data (e.g., LAION-COCO).
        -   LLM weights from real and synthetic data are averaged (mixed) to
            balance both knowledge domains.
        -   Visual encoders (e.g., CLIP, DINOv2, Q-Former) remain **frozen**
            during this stage.
    -   **Stage 2 (Visual instruction tuning)**:
        -   Mixture of tasks for multi-purpose capabilities, such as VQA,
            multi-object detection, region-level understanding, and text-based
            reasoning.
        -   Each image is encoded as five sub-images (four cropped at 224x224,
            one low-res global), processed independently to improve scaling and
            fine-grained understanding.

## Training Details

-   **Pretraining (Stage 1)**:
    -   Optimizer: **AdamW** with 5e-5 peak learning rate, 2k warmup steps, and
        180k steps with cosine annealing to 5e-6.
    -   Weight decay: **0.1**.
    -   Pretraining time: **125 hours** on 32 A100 GPUs for a 7B model (doubled
        for 13B).
-   **Finetuning (Stage 2)**:
    -   Batch size: **128**.
    -   Similar optimizer settings as pretraining.
    -   Finetuning time: **38 hours** on 16 A100 GPUs for a 13B model.