# SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

> Research note on SeeClick, a visual GUI agent with GUI grounding pre-training and the ScreenSpot benchmark for cross-platform evaluation.

- Authors: Hunter Heidenreich
- Published: 2024-01-19
- Updated: 2026-03-04
- Paper: SeeClick Paper <https://arxiv.org/abs/2401.10935>
- Code: SeeClick GitHub <https://github.com/njucckevin/SeeClick>
- Model: SeeClick Model (HuggingFace) <https://huggingface.co/cckevinn/SeeClick>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **research paper presenting a new model, benchmark, and evaluation
    framework** for visual GUI agents.
-   It proposes **SeeClick**, a visual agent designed for GUI tasks, and a
    **benchmark called ScreenSpot** for evaluating GUI grounding across multiple
    platforms.

## What is the motivation?

-   The paper addresses the limitations of existing GUI agents that rely on
    structured text, which is often inaccessible and verbose.
-   SeeClick's approach, using **visual GUI grounding** on screenshots, could
    significantly expand the accessibility and efficiency of GUI agents across
    devices.
-   Potential impacts include advancing **GUI automation** for platforms where
    structured text is unavailable, leading to more adaptable, universal agents.

## What is the novelty?

-   **Related work** includes text-based GUI agents (e.g., using HTML), GUI
    agents trained with LLMs, and previous attempts at visual GUI grounding.
-   **Key contributions**:
    -   Developing SeeClick, a **visual GUI agent** that relies solely on
        screenshots, removing dependency on structured text.
    -   Introducing **ScreenSpot**, a benchmark to evaluate GUI grounding
        performance across mobile, desktop, and web GUIs.
    -   Demonstrating that **GUI grounding pre-training** improves agent
        performance on diverse GUI tasks.

## What experiments were performed?

-   The authors establish GUI grounding as a critical capability, allowing LVLMs
    to interpret and locate elements on screen using screenshots.
-   **Experiments**:
    -   **ScreenSpot** results show SeeClick outperforming baseline models in
        GUI grounding, especially on non-text elements.
    -   On **MiniWob**, **AITW**, and **Mind2Web** tasks, SeeClick demonstrates
        superior performance, especially in environments with dynamic layouts or
        varied interfaces.
    -   **Overall**, results confirm that GUI grounding significantly boosts
        task success, validating the model's approach and pre-training method.

## Dataset

-   **SeeClick Dataset**: Comprised of ~300k web screenshots, extracting visible
    text content and "title" attributes as grounding cues.
-   **Mobile Data**: Utilizes RICO dataset's widget captioning and screen
    summarization, along with auto-generated RICO SCA data.
-   **General Data**: Incorporates instruction-following data collected via
    GPT-4 from the LLaVA dataset for maintaining general vision-language
    capabilities.
-   **ScreenSpot Benchmark**: A curated, human-annotated dataset for GUI
    grounding across mobile, desktop, and web interfaces.

## Methodology Highlights

-   **Model Architecture**: Built on QwenVL, a vision-language model with an
    integrated Vision Transformer (ViT).
-   **GUI Grounding Pre-training**: Trained to locate elements in GUI
    screenshots by predicting action locations with floating-point coordinates
    (e.g., `click (0.12, 0.54)`).
-   **Data Curation**: Automates collection of GUI grounding data from web,
    mobile, and general datasets, creating diverse and context-rich training
    samples.
-   **Task Adaptation**: SeeClick was tested and fine-tuned for three downstream
    tasks: MiniWob (simplified web tasks), AITW (Android interactions), and
    Mind2Web (web navigation tasks).

## Training Details

-   **Training Process**: Conducted over 10k steps (1 epoch), leveraging the
    LoRA technique to fine-tune both ViT and the language model.
-   **Optimizer**: Used AdamW with a learning rate of 3e-5 and a global batch
    size of 64.
-   **Hardware**: Training completed in approximately 24 hours on 8 NVIDIA A100
    GPUs.