# LLaVA: Visual Instruction Tuning

> Research note on LLaVA, the first model to use machine-generated multimodal instruction-following data to train a large language-vision model.

- Authors: Hunter Heidenreich
- Published: 2023-04-17
- Updated: 2026-03-04
- Paper: LLaVA Paper <https://arxiv.org/abs/2304.08485>
- Paper: NeurIPS 2023 Conference Paper <https://proceedings.neurips.cc/paper_files/paper/2023/hash/6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html>
- Code: haotian-liu/LLaVA <https://github.com/haotian-liu/LLaVA.git>
- Model: LLaVA Model Zoo <https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **methodology paper** introducing a new approach called visual
    instruction tuning.
-   It describes the first attempt at using machine-generated multimodal
    instruction-following data to train a large language-vision model (LLaVA).

## What is the motivation?

-   The paper addresses the lack of multimodal instruction-following data for
    training general-purpose vision and language assistants.
-   The potential impact includes advancing multimodal AI models by improving
    their zero-shot capabilities on real-world vision-language tasks, which
    could benefit fields like automated assistants and multimodal reasoning.

## What is the novelty?

-   Related work includes instruction-tuning for LLMs in natural language
    processing (e.g., ChatGPT, Alpaca) and existing multimodal models (e.g.,
    CLIP, BLIP-2).
-   This paper contributes by generating vision-language instruction data using
    GPT-4 and introducing LLaVA, a model that connects a vision encoder to an
    LLM for multimodal tasks, and by building new benchmarks for visual
    instruction-following.

## What experiments were performed?

-   **Experiments**: The paper shows that LLaVA performs similarly to multimodal
    GPT-4 on visual reasoning tasks and achieves state-of-the-art accuracy
    (92.53%) on the Science QA dataset when combined with GPT-4.
-   The results highlight LLaVA's effectiveness in multimodal reasoning,
    outperforming existing models like BLIP-2 and OpenFlamingo on a new
    multimodal benchmark (LLaVA-Bench).

## Dataset

-   **Visual Instruction Dataset**: Created using GPT-4 to generate 158K
    multimodal instruction samples.
    -   **58K conversations**, **32K detailed descriptions**, and **77K complex
        reasoning** samples.
-   **Context types**: Captions and bounding boxes used as symbolic image
    representations to prompt GPT for instruction generation.
-   **Source data**: Filtered 595K image-text pairs from CC3M dataset to ensure
    relevant and diverse samples.

## Methodology Highlights

-   **Architecture**: Combines a **frozen ViT-L/14 CLIP encoder** with a
    **decoder-only LLaMA LLM**.
    -   Vision tokens are linearly projected into the language space and
        prepended to text instructions.
-   **Response types**: Three types of GPT-4-generated responses: conversation,
    detailed description, and complex reasoning.
-   **End-to-End Multimodal Model**: The model connects vision and language
    components with minimal architectural changes, focusing on efficient
    training via instruction-tuning.

## Training Details

-   **Two-Stage Training**:
    -   **Stage 1 (Feature Alignment)**: Trains the projection layer to align
        visual features with the LLM's embedding space. All other weights are
        frozen.
        -   Uses 595K image-text pairs from CC3M for this stage.
    -   **Stage 2 (End-to-End Fine-tuning)**: The vision encoder remains frozen,
        while the projection layer and LLM are fine-tuned on the 158K
        instruction-following dataset.