# Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

> Research note on Pix2Struct, a pretrained image-to-text model that uses screenshot parsing as a pretraining objective for visually-situated language understanding.

- Authors: Hunter Heidenreich
- Published: 2022-10-07
- Updated: 2026-03-04
- Paper: Pix2Struct paper <https://arxiv.org/abs/2210.03347>
- Paper: PMLR proceedings <https://proceedings.mlr.press/v202/lee23g.html>
- Code: google-research/pix2struct <https://github.com/google-research/pix2struct>
- Model: google/pix2struct-base <https://huggingface.co/google/pix2struct-base>
- Model: google/pix2struct-large <https://huggingface.co/google/pix2struct-large>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **pretrained model proposal** paper.
-   It introduces Pix2Struct, a pretrained image-to-text model designed for
    visually-situated language understanding.
-   The paper focuses on a general-purpose model capable of handling multiple
    tasks across various domains.

## What is the motivation?

-   The paper addresses the need for a **unified approach** to visually-situated
    language understanding, where text and visuals are closely intertwined, such
    as in web pages, documents, and UIs.
-   The problem is important because previous work has been fragmented, relying
    on **domain-specific pipelines** with limited data/model sharing.
-   The potential impact includes enabling more **general-purpose models**
    capable of handling diverse visual language tasks without relying on
    domain-specific pipelines.

## What is the novelty?

-   Related work includes models like **Donut** (OCR-focused for document
    understanding), **VisionTaPas**, and **GIT2** for image captioning and
    visual language tasks.
-   Existing approaches often rely on task-specific engineering or external
    systems like OCR, which limits adaptability.
-   Pix2Struct's main contribution is **screenshot parsing** as a pretraining
    objective, which subsumes pretraining signals like OCR and image captioning
    into a more general framework. It integrates visual and language inputs
    holistically, leading to improved performance in **low-resource domains**
    like UIs and illustrations.

## What experiments were performed?

-   **Experiments**: Pix2Struct outperforms previous **visual-only models**
    (like Donut) in eight out of nine tasks across four domains: documents,
    illustrations, UIs, and natural images.
-   It achieves **state-of-the-art** results in six out of the nine benchmarks,
    excelling in **low-resource tasks** such as UI and illustration
    understanding.
-   The model lags in tasks with **high-resource domains** like document
    understanding, where OCR pipelines still dominate, but shows promise for
    future scaling.

## Dataset

-   **80M screenshots** collected from URLs in the **C4 dataset**.
-   Screenshots taken at **1024x1024 resolution**.
-   Pretraining includes a **reading curriculum** stage using **BooksCorpus**,
    where text is rendered as images with random fonts and colors.

## Methodology Highlights

-   **Screenshot parsing** pretraining: Condenses HTML DOM tree, retaining only
    visible elements for modeling in output.
-   **Masked span-based modeling**: 50% of the text is masked during pretraining
    for better context modeling.
-   **Flexible aspect-ratio ViT**: Scales input images to extract the maximal
    number of fixed-size patches, enabling the model to handle **extreme aspect
    ratios** and **variable sequence lengths**.
-   **ViT-encoder + text-based decoder** architecture: Integrates vision and
    language by rendering language prompts (e.g., VQA questions) directly onto
    the input image, eliminating the need for cross-attention mechanisms.

## Training Details

-   **Pretraining (Reading stage)**: 30K steps, **128 patch input length**.
-   **Pretraining (Screenshot stage)**:
    -   **Base**: 270K steps, 2048 batch size, 64 TPUs.
    -   **Large**: 170K steps, 1024 batch size, 128 TPUs.
-   **Optimizer**: **Adafactor** with linear warmup (1K steps) to 0.01 learning
    rate, followed by cosine decay.
-   **Input sequence length**: **2048 patches**; decoder sequence limited to
    **128 tokens**, targets limited to **1024 characters**.