# LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models

> Research note on LLaVA-Grounding, a model and dataset for grounded visual chat combining conversational LMMs with pixel-level grounding.

- Authors: Hunter Heidenreich
- Published: 2023-12-05
- Updated: 2026-03-04
- Paper: LLaVA-Grounding Paper <https://arxiv.org/abs/2312.02949>
- Paper: LLaVA-Grounding ECCV 2024 <https://link.springer.com/chapter/10.1007/978-3-031-72775-7_2>
- Code: LLaVA-Grounding GitHub <https://github.com/UX-Decoder/LLaVA-Grounding>
- Model: LLaVA-Grounding Model (HuggingFace) <https://huggingface.co/Haozhangcx/llava_grounding_gd_vp>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **model and dataset paper** that introduces a new large multimodal
    model (LLaVA-Grounding) and a specialized dataset (GVC) for **grounded
    visual chat**.
-   It contributes a **new benchmark (Grounding-Bench)** and presents
    improvements in **integrated chat and visual grounding capabilities** in
    LMMs.

## What is the motivation?

-   **Motivation**: Current large multimodal models (LMMs) struggle to perform
    visual grounding and chat together effectively; existing datasets and models
    either focus on one aspect or are limited by the lack of fine-grained,
    grounded data.
-   **Potential Impact**: By combining grounded chat and grounding capabilities,
    this model could enable more effective multimodal interactions across
    applications that require detailed image understanding and conversational
    skills (e.g., AI assistants for complex visual tasks).

## What is the novelty?

-   **Related Work**: Previous LMMs like LLaVA, miniGPT-4, and CogVLM focused on
    visual chat but lacked robust grounding capabilities. Specialized grounding
    models (e.g., Kosmos-2, Shikra) often require distinct prompting and cannot
    handle chat coherently.
-   **Contributions**: The paper introduces:
    -   A dataset of **150K grounded visual chat instances** (GVC).
    -   The **LLaVA-Grounding model**, which integrates a language model with a
        grounding model, supporting pixel-level and object-level grounding in
        visual chat.
    -   A **benchmark (Grounding-Bench)** for evaluating grounded visual chat,
        with new metrics like grounded recall and precision.

## What experiments were performed?

-   The paper proposes a novel method to connect LMM features with a grounding
    model (OpenSeeD) and evaluates it on the new Grounding-Bench, emphasizing
    **object-level and pixel-level grounding** capabilities.
-   **Experiments**:
    -   **On Grounding-Bench**: LLaVA-Grounding outperformed other LMMs in both
        chat and grounding capabilities.
    -   **On classic benchmarks**: Achieved competitive performance on
        RefCOCO/+/g and Flickr30K for phrase grounding and referring expression
        tasks.
    -   **Ablations**: Demonstrated the model's effectiveness across grounding
        types, prompt types (e.g., boxes, clicks), and in handling the trade-off
        between chat and grounding tasks.

## Dataset

-   **Grounded Visual Chat (GVC) Data**: 150K instances labeled with human
    annotations from COCO, grounded in conversational context using GPT-4.
    Ensures noun-phrase references match segmentation masks for accuracy.
-   **Stage 1 Data**: Combines datasets (e.g., RefCOCO, Flickr30K, Visual
    Genome) with alignment pairs from COCO for visual and language alignment
    tasks.
-   **Stage 2 Data**: Instruction set and GVC data without visual prompts,
    leveraging conversational context for grounding-specific tuning.
-   **Stage 3 Data**: Enhanced dataset for visual prompts (boxes, clicks,
    scribbles) with specific instructions for more interactive grounded visual
    chat.

## Methodology Highlights

-   **Pipeline**: Utilizes a three-stage approach to pretrain, align, and tune
    grounding and visual chat tasks.
    -   **Stage 1**: Pretraining aligns visual and language embeddings using a
        projection layer to connect the language model's output with the
        grounding model.
    -   **Stage 2**: Fine-tunes for grounding chat without updating the vision
        and prompt encoders.
    -   **Stage 3**: Adds visual prompting, training only the prompt encoder or
        updating language decoder for marked instances.
-   **Grounding Model**: OpenSeeD model enables both bounding box and
    pixel-level grounding, enhancing the model's ability to perform fine-grained
    visual localization.
-   **Prompt Encoder**: Uses a Semantic SAM model to convert visual prompts into
    language-compatible embeddings, allowing the model to interpret diverse
    prompt types.

## Training Details

-   **Stage 1**: Vision-language (VL) projection and grounding model updated
    with learning rates of `1e-4`, aligning visual encoder and grounding model
    with language model outputs.
-   **Stage 2**: Instruction tuning for grounded responses; prompt encoder and
    CLIP vision encoder are frozen, with other layers fine-tuned at `2e-5` for
    language layers and `1e-4` for grounding.
-   **Stage 3**: Visual prompting (e.g., marks) trained by fine-tuning the
    prompt encoder and projection layers, with additional support for specific
    image regions through Set-of-Mark prompts.
-   **Hyperparameters**: Gradual learning rate decay, weight decay set to `0.0`,
    `bf16` and `tf32` precision for faster, stable training on compatible
    hardware.