# LLaVA-1.5: Improved Baselines with Visual Instruction Tuning > Research note on LLaVA-1.5, a systematic study of design choices for large multimodal models that achieves state-of-the-art performance across 11 benchmarks with minimal data. - Authors: Hunter Heidenreich - Published: 2023-10-05 - Updated: 2026-03-04 - Paper: LLaVA-1.5 Paper - Paper: CVPR 2024 Conference Paper - Code: haotian-liu/LLaVA - Source: Bevaya Labs, --- ## What kind of paper is this? - This is a systematic study and baseline improvement paper. - It investigates design choices for large multimodal models (LMMs) under the LLaVA framework and explores improvements to state-of-the-art models through modifications in architecture and training. ## What is the motivation? - The paper aims to improve the efficiency and performance of LMMs, which are central to creating general-purpose visual-language assistants. - It addresses the problem of balancing data efficiency, model performance, and scaling challenges in LMMs. - The paper's impact lies in its ability to offer a simple, reproducible baseline (LLaVA-1.5) that uses publicly available data and achieves state-of-the-art performance across 11 benchmarks. ## What is the novelty? - Related work includes LLaVA, InstructBLIP, and Qwen-VL, which also explore visual instruction tuning and vision-language model training. These models use various techniques such as resamplers (Qformer) or larger image-text datasets. - The paper's main contribution is the introduction of LLaVA-1.5, which improves upon LLaVA by using an MLP connector and incorporating academic-task-oriented VQA datasets. This achieves state-of-the-art performance with a much smaller training set and computational resources. ## What experiments were performed? - **Experiments**: - LLaVA-1.5 achieves state-of-the-art performance across 11 benchmarks using only 1.2M training samples, compared to competing models trained on much larger datasets. - Scaling to higher resolutions and compositional capabilities further improves results, reducing model hallucination and enhancing generalization across multiple tasks. - Empirical evidence shows that LLaVA-1.5 outperforms larger models like InstructBLIP and IDEFICS on various visual reasoning tasks, while using fewer resources and simpler architecture. ## Dataset - LLaVA-1.5 uses a combination of public datasets for training, totaling 558K image-text pairs. - Datasets include academic task-oriented VQA datasets (e.g., VQA-v2, OKVQA, OCR-based datasets like TextCaps) and region-level perception datasets (Visual Genome, RefCOCO). - GQA and ShareGPT data are incorporated to enhance visual question answering (VQA) and conversational capabilities. - No in-house data is used, ensuring full reproducibility. ## Methodology Highlights - Replaces LLaVA's original linear projection with a more powerful two-layer MLP vision-language connector. - Introduces response format prompting to handle short-form answers better in VQA tasks, improving precision in outputs. - Scales input resolution to 336x336, allowing the model to process higher detail in images, improving accuracy for detailed visual tasks. - The model is scaled to a 13B language model (Vicuna-v1.5), which enhances its performance on benchmarks like MME and MM-Vet. - Demonstrates high data efficiency by reducing training data without significantly compromising performance. ## Training Details - LLaVA-1.5 is trained on 8x A100 GPUs, with the total training time of ~26 hours. - Stage 1 (vision-language alignment pretraining) takes ~6 hours. - Stage 2 (visual instruction tuning) takes ~20 hours. - Efficient training setup with just 1.2M public data points, significantly lower than competitors using much larger datasets.