# CogVLM: Visual Expert for Pretrained Language Models

> Research note on CogVLM, a vision-language model that introduces a trainable visual expert module within attention and feedforward layers for deep fusion of vision and language features.

- Authors: Hunter Heidenreich
- Published: 2023-11-06
- Updated: 2026-03-04
- Paper: CogVLM Paper <https://arxiv.org/abs/2311.03079>
- Paper: OpenReview (ICLR 2024 Submission) <https://openreview.net/forum?id=c72vop46KY>
- Code: THUDM/CogVLM <https://github.com/THUDM/CogVLM>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is an empirical and methodological research paper focused on
    introducing a new vision-language model, CogVLM.
-   It proposes a new approach for integrating visual and linguistic features
    through deep fusion using a "visual expert" module within a pretrained
    language model, rather than using shallow alignment techniques.

## What is the motivation?

-   The motivation is to overcome limitations in shallow alignment methods used
    by prior models, which integrate image features into language models in a
    way that doesn't fully capture the depth of vision-language information.
-   By introducing a trainable module within the attention and feedforward
    layers, CogVLM achieves state-of-the-art performance across multiple
    multimodal benchmarks.
-   The potential impact includes enabling large pretrained language models to
    perform high-quality visual tasks without sacrificing NLP capabilities,
    contributing significantly to both research and industrial applications of
    multimodal AI.

## What is the novelty?

-   Related work includes prior models that use shallow integration techniques,
    like InstructBLIP and MiniGPT-4, which map image features into the language
    model's input embedding space but fall short in performance due to limited
    fusion.
-   CogVLM's main contribution is the development of a trainable "visual expert"
    module within attention and feedforward layers, enabling deep fusion of
    vision and language features.
-   It also includes extensive ablation studies and evaluations to validate this
    approach, resulting in state-of-the-art performance across a variety of
    benchmarks.

## What experiments were performed?

-   **Experiments**: The paper presents CogVLM's superior performance across 17
    cross-modal benchmarks, including image captioning (NoCaps, Flickr30K),
    visual question answering (OKVQA, TextVQA), and visual grounding (RefCOCO).
-   **Performance**: CogVLM outperforms other state-of-the-art models in
    multiple settings, especially for tasks requiring complex vision-language
    interactions, achieving high scores in generalist tasks and grounded VQA.
-   **Ablation Studies**: The authors conduct detailed studies on architectural
    components, such as the impact of using different visual attention masks,
    initialization methods, and visual encoder scales, highlighting the
    importance of deep fusion.

## Dataset

-   **Pre-training Data**:

    -   **LAION-2B** and **COYO-700M** datasets, heavily filtered for quality,
        yielding 1.5B image-text pairs.
    -   **Visual Grounding Dataset**: 40M images with bounding boxes for
        grounded object references, nouns extracted via spaCy, and bounding
        boxes predicted using GLIPv2.

-   **Supervised Fine-Tuning Data**:
    -   **CogVLM-Chat**: VQA datasets (VQAv2, OKVQA, TextVQA, etc.) and
        multi-turn dialogue datasets like LLaVA-Instruct. The VQA data includes
        both concise and detailed responses.
    -   **CogVLM-Grounding**: Grounded captioning, referring expression
        generation (REG), referring expression comprehension (REC), and VQA with
        bounding boxes from sources such as Flickr30K Entities, RefCOCO, and
        Visual7W.

## Methodology Highlights

-   **Architecture**:

    -   **ViT Encoder**: EVA2-CLIP-E (last layer removed); outputs mapped to
        text embedding space using an MLP adapter (two-layer SwiGLU).
    -   **Visual Expert Module**: Added in each layer, separates vision and
        language modalities with unique QKV and FFN layers for vision, while
        sharing layer norms.
    -   **Position Embedding**: RoPE for text tokens, single shared position for
        visual tokens to avoid sequential proximity bias.

-   **Training Approach**:
    -   **First Stage**: Image captioning as a next-token prediction task on
        image-text pairs, for 120K steps with batch size 8192.
    -   **Second Stage**: Referring Expression Comprehension (REC) as VQA-style
        grounding; location answers normalized to 3-digit coordinates, trained
        for 60K steps with batch size 1024.
    -   **Resolution Adjustment**: Increased input resolution to 490x490 for the
        final 30K steps.

## Training Details

-   **CogVLM-Chat**: Uses VQA and multi-turn instruction data for supervised
    fine-tuning.

    -   **Hyperparameters**: 6K steps, learning rate of 1e-5, batch size of
        1024, ViT unfrozen with learning rate set to 1/10 of the full model's
        learning rate.

-   **CogVLM-Grounding**: Fine-tuned on grounded captioning, REG, REC, and
    Grounded VQA.
    -   **Task-Specific Adjustments**: Incorporates bounding box annotations for
        improved localization and grounding performance across diverse datasets.