# VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

> Research note on VisionLLM v2, a generalist multimodal LLM using task-specific decoders and super-linking for hundreds of vision-language tasks.

- Authors: Hunter Heidenreich
- Published: 2024-06-12
- Updated: 2026-03-04
- Paper: VisionLLM v2 Paper <https://arxiv.org/abs/2406.08394>
- Code: VisionLLM v2 GitHub <https://github.com/OpenGVLab/VisionLLM/tree/main/VisionLLMv2>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This paper is a model introduction. They want to demonstrate a single system
    that can do a wide range of vision-language tasks.
-   Not a lot of new ideas, but a lot of training and evaluation for a single
    model across many tasks.

## What is the motivation?

-   The motivation is to have one model that can do any language, vision, or
    vision-language task.
-   The impact is in simplifying the deployment of vision-language models,
    making it easier to use them in a wide range of applications.

## What is the novelty?

-   The most relevant previous work is the original
    [VisionLLM](2023_05_visionllm.md) model.
-   All other decoder-based Vision-Language models are relevant, like
    [Monkey](2023_11_monkey.md).

## What experiments were performed?

-   The model is evaluated on 100+ vision-language tasks, showing strong
    performance across the board.

## Dataset

-   Stage 1: Uses a wide variety of datasets across many tasks: conversation,
    imae captioning, image VQA, OCR, region captioning, region VQA, region
    recognition.
-   Stage 3: Some overlap with Stage 1, but specialized towards object
    detection, instance segmentation, grounded captioning, visual grounding,
    object counting, pose estimation, image generation and editing, and more.
-   Stage 2: Is trained on the combination of Stages 1 and 3.

## Methodology Highlights

-   Model architecture is 4 key components:
    -   An image encoder, and a region encoder
    -   A language decoder / LLM
    -   Task-specific decoders
    -   A module called a ``super-linker''. This appears as a sort of abstractor
        model or a Q-transformer where queries are learned to extract a set of
        features for that specific task in that particular context.
-   A key way that super-linking works is through routing tokens. For a
    particular specialized task, like object detection, the model learns to
    output special route tokens (e.g., `[DET]`, `[SEG]`, `[CAP]`) that guide the
    super-linker to extract the right features for that task AND to route to the
    particular task-specific decoder.

## Training Details

Three stage training process:

-   Stage 1: Essentially [LLaVA](2023_04_llava.md). It included a pre-train and
    fine-tune stage.
-   Stage 2: Specialized training on a subset of tasks. It uses task-specific
    decoders.
-   Stage 3: Keeps every component frozen except task-specific decoders. It
    trains for longer on 128 A100 GPUs.