# LLaVA-NeXT: Improved reasoning, OCR, and world knowledge

> Research note on LLaVA-NeXT, an improved large multimodal model with dynamic high-resolution support, enhanced reasoning, and OCR capabilities.

- Authors: Hunter Heidenreich
- Published: 2024-01-30
- Updated: 2026-03-04
- Paper: LLaVA-NeXT Blog Post <https://llava-vl.github.io/blog/2024-01-30-llava-next/>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **system and enhancement** describing LLaVA-NeXT, an improved
    large multimodal model (LMM).
-   It presents design advancements over LLaVA-1.5 in areas such as reasoning,
    OCR, and handling high-resolution images.
-   It also outlines technical improvements and evaluates performance against
    other state-of-the-art (SoTA) models on specific benchmarks.

## What is the motivation?

-   **Motivation**: To enhance the LLaVA model's visual reasoning, OCR, and
    general world knowledge, addressing limitations in existing multimodal
    models, especially for visual detail and zero-shot scenarios.
-   **Impact**: LLaVA-NeXT offers improved performance on multiple benchmarks
    and enables efficient deployment and inference. Its open-source release aims
    to support further research and development in multimodal AI, benefiting
    areas requiring robust visual-linguistic integration.

## What is the novelty?

-   **Related Work**: Builds on LLaVA-1.5, which set a baseline in multimodal
    performance. Other referenced models include Gemini Pro, Qwen-VL-Plus, and
    CogVLM, each contributing distinct multimodal capabilities.
-   **Contribution**: LLaVA-NeXT surpasses prior models in resolution,
    reasoning, and deployment efficiency. It introduces dynamic high-resolution
    support, a refined visual instruction dataset, and compatibility with
    multiple LLM backbones, achieving SoTA performance in both English and
    zero-shot Chinese tasks.

## What experiments were performed?

-   **Experiments**:
    -   LLaVA-NeXT outperforms several benchmarks, matching or surpassing
        proprietary models like Gemini Pro in specific categories.
    -   Zero-shot Chinese performance is notable, achieving SoTA results without
        dedicated Chinese training data.
    -   Model versions (7B, 13B, 34B) achieve varying levels of performance,
        with high-resolution image support significantly improving OCR and
        reasoning capabilities.
-   **Success Metrics**:
    -   The model's success is defined through SoTA performance on a suite of
        multimodal benchmarks, including OCR-specific tasks like TextVQA and
        overall visual reasoning with Math-Vista.
    -   Efficiency metrics are notable, with low training cost achieved on 32
        A100 GPUs over ~1 day.

## Dataset

-   **Image-Text Pairs**: 1.3M visual instruction-tuning samples.
-   **Data Sources**:
    -   **GPT-V Data**: LAION-GPT-V, ShareGPT-4V for enhanced
        instruction-following and visual conversation.
    -   **Specialized Datasets**: Added ChartQA, DVQA, AI2D for improved
        chart/diagram understanding, and DocVQA, SynDog-EN to enhance OCR
        capabilities.
    -   **Data Filtering**: TextCaps removed (overlap with TextVQA); sensitive
        data removed to ensure privacy and safety.
-   **Zero-Shot Capability**: Demonstrates strong zero-shot Chinese language
    support, despite training predominantly on English data.

## Methodology Highlights

-   **Dynamic High Resolution**: Introduced 'AnyRes' technology, supporting
    variable resolutions (up to 4x more pixels), enhancing fine visual detail
    through grid splitting/merging (e.g., 2×2, 3×1 configurations).
-   **Data Mixture**: Improved data diversity through high-quality visual
    instruction samples, designed to capture varied user intents and real-world
    applications.
-   **LLM Backbones**: Models use Vicuna-1.5 (7B, 13B), Mistral-7B, and Yi-34B,
    enabling flexible deployment options and compatibility with larger
    multimodal tasks.

## Training Details

-   **Training Hardware**:
    -   LLaVA-NeXT-7B: 8xA100 GPUs, ~20 hours.
    -   LLaVA-NeXT-13B: 16xA100 GPUs, ~24 hours.
    -   LLaVA-NeXT-34B: 32xA100 GPUs, ~30 hours.
-   **Efficiency**: Minimalist approach maintains data efficiency of LLaVA-1.5
    with low GPU-hour cost, using less than 1M samples in the visual instruction
    phase.
-   **Performance Benchmarking**: Achieves state-of-the-art performance across
    multiple multimodal tasks with optimized training time and compute costs.