# CogAgent: A Visual Language Model for GUI Agents

> Research note on CogAgent, a visual language model with high-resolution cross-attention for GUI navigation and general VQA tasks.

- Authors: Hunter Heidenreich
- Published: 2023-12-14
- Updated: 2026-03-04
- Paper: CogAgent Paper <https://arxiv.org/abs/2312.08914>
- Paper: CogAgent CVPR 2024 <https://openaccess.thecvf.com/content/CVPR2024/html/Hong_CogAgent_A_Visual_Language_Model_for_GUI_Agents_CVPR_2024_paper.html>
- Code: CogVLM GitHub <https://github.com/THUDM/CogVLM>
- Model: CogAgent VQA (HuggingFace) <https://huggingface.co/THUDM/cogagent-vqa-hf>
- Model: CogAgent Chat (HuggingFace) <https://huggingface.co/THUDM/cogagent-chat-hf>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a "Big idea" paper that proposes and evaluates a new Visual Language
    Model (VLM), called CogAgent.
-   It specifically aims to advance GUI-based agents using high-resolution image
    encoding and cross-modality for enhanced performance on both general VQA and
    GUI-specific tasks.
-   The paper evaluates CogAgent's versatility across multiple benchmarks,
    positioning it as a state-of-the-art generalist VLM.

## What is the motivation?

-   The motivation stems from the need for large language models (LLMs) to
    interact effectively with GUIs, which is a limitation in current text-only
    models.
-   GUIs are ubiquitous in digital device interactions, and a VLM that can
    navigate these interfaces autonomously could enhance automation across
    various tasks, such as web navigation and mobile app usage.
-   CogAgent's ability to understand and act within GUIs could reduce the need
    for human intervention in routine digital tasks, potentially impacting
    personal productivity and digital accessibility.

## What is the novelty?

-   Existing models like AutoGPT and WebShop have demonstrated limited
    capabilities in combining GUI visual signals with LLMs but often fall short
    in generalist performance or require substantial pre-processing.
-   CogAgent advances these by integrating low- and high-resolution image
    encoders, enabling it to outperform text-extracted methods on GUI and VQA
    tasks.
-   The primary contribution is CogAgent's architecture, which includes a
    high-resolution cross-module for GUIs, allowing the model to process tiny
    text and elements effectively.
-   The authors also created a large, specialized dataset for GUI understanding,
    making CogAgent robust across multiple benchmarks.

## What experiments were performed?

-   The paper introduces a novel high-resolution cross-attention module,
    balancing computational efficiency and high-resolution input processing.
    This module maintains CogAgent's usability across GUI and VQA tasks without
    excessive FLOP increases.
-   **Experiments**: CogAgent is evaluated across several VQA benchmarks and
    GUI-focused datasets (Mind2Web, AITW), demonstrating state-of-the-art
    results, particularly in text-rich VQA and GUI navigation tasks.
-   **Performance**: It surpasses generalist models in benchmarks such as VQAv2,
    OK-VQA, and specific GUI datasets, showing 11.6%–16.5% improvements over
    comparable language-based agents.

## Dataset

-   **Text Recognition**:

    -   **Synthetic Text Data**: Generated 80M synthetically rendered images
        with diverse font styles, colors, orientations, and LAION 2B
        backgrounds.
    -   **OCR on Natural Images**: 18M images with OCR annotations extracted
        from COYO and LAION-2B datasets.
    -   **Academic Documents**: 9M academic-style images processed with similar
        augmentation techniques to the Nougat dataset, featuring text, formulas,
        and tables.

-   **Visual Grounding**:

    -   Collected 40M image-caption pairs from LAION-115M, associating entities
        with bounding boxes.

-   **GUI Imagery**:
    -   **GUI REG** (Referring Expression Generation): Generates HTML code for
        DOM elements based on specified screenshot areas.
    -   **GUI REC** (Referring Expression Comprehension): Creates bounding boxes
        for DOM elements on screenshots.
    -   Built the **CCS400K** dataset from 400k Common Crawl screenshots and
        140M GUI-based REC and REG tasks.

## Methodology Highlights

-   **Architecture**:

    -   Employs **CogVLM-17B** as the base VLM, enhanced with a new
        **high-resolution cross-attention module** for GUI tasks.
    -   Integrates a smaller **EVA2-CLIP-L** image encoder (300M parameters) for
        1120x1120 high-resolution images, retaining key visual details.
    -   High- and low-resolution branches allow for efficient processing;
        high-resolution input captures text and small elements, with
        cross-attention at each decoder layer.

-   **High-Resolution Cross-Module**:
    -   Maintains efficiency by reducing hidden sizes for text features,
        optimizing computation with a lower-dimensional cross-attention setup.
    -   Operates on 14x14 patches, balancing image clarity and computational
        load for GUI screens up to 1120x1120 pixels.

## Training Details

-   **Pre-Training**:

    -   Conducted 60k steps with a batch size of 4,608 and learning rate of
        2e-5.
    -   **Freezing Strategy**: Only the high-resolution cross module was trained
        for the first 20k steps. For the remaining 40k steps, the visual expert
        was unfrozen for further fine-tuning.
    -   Curriculum learning: Pre-trained sequentially, progressing from
        synthetic text recognition tasks to document-style OCR, grounding tasks,
        and finally GUI-based tasks.

-   **Evaluation Benchmarks**:
    -   **VQA Tasks**: Assessed on VQAv2, OK-VQA, TextVQA, OCR-VQA, DocVQA,
        InfoVQA, and ChartQA.
    -   **GUI Interfaces**: Tested on the Mind2Web and AITW datasets, covering
        desktop and Android interface navigation tasks.