# Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

> Research note on Shikra, a multimodal LLM that enables referential dialogue by handling spatial coordinates in natural language without special vocabularies.

- Authors: Hunter Heidenreich
- Published: 2023-06-27
- Updated: 2026-03-04
- Paper: Shikra Paper <https://arxiv.org/abs/2306.15195>
- Code: shikras/shikra <https://github.com/shikras/shikra>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **proposal paper** introducing a new Multimodal Large Language
    Model (MLLM) called Shikra, which enables referential dialogue.
-   It addresses a gap in MLLMs where spatial coordinates in dialogue were
    previously unsupported.

## What is the motivation?

-   The motivation is to introduce **referential dialogue** capabilities into
    MLLMs, allowing users to point to specific areas of an image and ask
    questions, with the model responding in natural language and providing
    relevant spatial coordinates.
-   The impact is significant for **Mixed Reality (XR)**, **robotic
    communication**, and **online shopping**, where such spatial interaction
    could enhance user experience.

## What is the novelty?

-   **Related work**: It builds on existing MLLMs like **Flamingo**, **BLIP-2**,
    and **Mini-GPT4**, which combine vision and language but lack spatial
    referencing capabilities.
-   **Contribution**: Shikra introduces a **unified, simple architecture** that
    handles both input and output of spatial coordinates in natural language
    without extra vocabularies or specialized modules. It enhances the natural
    dialogue capabilities of MLLMs by supporting tasks like REC, PointQA, and
    VQA.

## What experiments were performed?

-   **Theory**: Shikra's architecture integrates a **vision encoder, an
    alignment layer**, and a **LLM**, and uses **natural language numerical
    coordinates** for spatial referencing.
-   **Experiments**: Shikra shows **promising performance** across tasks like
    REC, PointQA, and image captioning. It demonstrates **superior performance**
    compared to other generalist models on REC and PointQA tasks, but still
    falls behind specialist models in some areas.

## Dataset

-   **Shikra-RD**: Custom instruction-tuning dataset generated using **GPT-4**,
    which includes chain-of-thought (CoT) examples with spatial annotations.
-   **Public datasets**: Utilizes **Flickr30K Entities** and **RefCOCO**
    datasets to provide visual data and examples for GPT-4 generation.
-   **Other Vision-Language (VL) datasets**: Includes publicly available
    datasets used in the first stage of training for conventional VL tasks
    (e.g., image captioning, VQA, REC).

## Methodology Highlights

-   **Architecture**: Combines **ViT-L/14** (vision encoder) and a
    **decoder-only LLM** (Vicuna-7B/13B).
-   **Coordinate representation**: Uses **natural language** to represent
    coordinates in the format [x_min, y_min, x_max, y_max] for boxes and [x, y]
    for points.
    -   Multiple points/boxes separated by semicolons.
    -   All coordinates normalized to [0, 1], truncated to **3 decimal places**.
-   **Instruction tuning**: Bootstraps CoT instruction examples from **GPT-4**
    using visual datasets.
-   **No special vocabularies**: Coordinates are represented without introducing
    any special encoders or vocabularies for reference points.

## Training Details

-   **Two-stage training**:
    -   **Stage 1**: Trained on all VL datasets for **100,000 steps** (approx.
        1.5 epochs).
    -   **Stage 2**: Fine-tuned on **Shikra-RD** and **LLaVA-Instruct-150K**
        with increased sampling ratio.
-   **Optimizer**: **AdamW** with frozen visual encoders, no warmup.
-   **Learning rate**: Peak LR of **2e-5**, with **cosine annealing** for
    cooldown.
-   **Compute**: Trained on **8 NVIDIA A100 GPUs**, taking 100 hours for stage 1
    and 20 hours for stage 2.