# ScreenAI: A Vision-Language Model for UI and Infographics Understanding

> Research note on ScreenAI, a Google vision-language model for UI and infographics understanding combining PaLI and pix2struct architectures.

- Authors: Hunter Heidenreich
- Published: 2024-02-07
- Updated: 2026-03-04
- Paper: ScreenAI Paper <https://arxiv.org/abs/2402.04615>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **small idea with evaluation** paper.
-   It introduces **ScreenAI**, a vision-language model for understanding UIs
    and infographics, and evaluates its performance through ablation studies and
    comparisons with state-of-the-art (SoTA) models.

## What is the motivation?

-   The paper addresses the challenge of understanding complex screen UIs and
    infographics, which share visual language but are difficult to model due to
    their diverse designs.
-   The impact is significant for fields requiring automated UI understanding,
    such as **UI navigation, infographics comprehension, and screen-based
    question-answering (QA)**.

## What is the novelty?

-   Existing works in **screen-based UI models** focus on narrow tasks like icon
    detection, whereas **generalist foundation models** excel in broad
    multimodal tasks.
-   The paper contributes **ScreenAI**, which extends **[PaLI](2022_09_pali.md)
    and pix2struct** architectures, introduces a **screen annotation task**, and
    automates dataset generation using **LLMs**. It also releases three new
    datasets and achieves state-of-the-art results in several tasks.

## What experiments were performed?

-   The paper introduces a novel architecture that combines
    **[PaLI](2022_09_pali.md)'s multimodal encoder-decoder** with **pix2struct's
    adaptive patching** mechanism.
-   **Experiments**:
    -   ScreenAI achieves **new SoTA** on several benchmarks like **Multipage
        DocVQA, WebSRC, MoTIF**, and excels in tasks like **ChartQA** and
        **InfographicVQA**.
    -   The model's performance improves with larger sizes, and ablation studies
        confirm the benefits of the **pix2struct patching strategy** and using
        **LLM-generated data** during pre-training.

## Dataset

-   **353M autogenerated screen schemas**: Includes OCR, object detection,
    classification, and image/icon captioning.
-   **38.6M Q&A samples**: Added to address challenges with arithmetic,
    counting, and understanding complex graphics.
-   **15.9M navigation samples**: Instructions like "click BOX" for UI
    navigation.
-   **13.2M screen summarizations**: Summaries of what's shown on the screen.
-   **3M chart-to-table translation samples**: Converting graphical data to
    tabular formats.
-   **1M other data**: From PaLI, used for additional text and Q&A tasks.

## Methodology Highlights

-   **Screen Schema**: UI type + short caption or OCR text + bounding box (e.g.,
    BUTTON 800 100 900 150); supports nested elements (e.g., TEXT in a BUTTON).
-   **Aspect-ratio preserving patching**: For flexible handling of various
    screen shapes and resolutions, drawn from **pix2struct**.
-   **Data generation**: Combines object detection, OCR, and image captioning
    for generating pretraining data; supplemented with LLM-enhanced Q&A and
    summarization.
-   **Task-specific fine-tuning**: Conducted on human-annotated data for tasks
    like Q&A, navigation, and summarization.

## Training Details

-   **Pre-training**: Uses a mixture of self-supervised learning on
    autogenerated datasets (e.g., screen schemas) and LLM-generated data.
-   **Fine-tuning**: On specific tasks like Q&A, navigation, and summarization,
    using both human-annotated and machine-generated data.
-   **Model sizes**: Trained models include 670M, 2B, and 5B parameters; the 5B
    model starts from PaLI-3 multimodal checkpoints.
-   **Freezing the ViT encoder**: During fine-tuning, the **ViT encoder** is
    frozen, focusing training on the language model (e.g., mT5 or UL2 depending
    on size).