# Pix2Act: From Pixels to UI Actions

> Research note on Pix2Act, a pixel-based agent that learns to follow instructions via graphical user interfaces without relying on structured representations.

- Authors: Hunter Heidenreich
- Published: 2023-06-01
- Updated: 2026-03-04
- Paper: Pix2Act Paper <https://arxiv.org/abs/2306.00245>
- Paper: NeurIPS 2023 Conference Paper <https://proceedings.neurips.cc/paper_files/paper/2023/hash/6c52a8a4fadc9129c6e1d1745f2dfd0f-Abstract-Conference.html>
- Code: google-deepmind/pix2act <https://github.com/google-deepmind/pix2act>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **systems and benchmark paper**.
-   It proposes a **new model (PIX2ACT)** and demonstrates its capabilities on
    GUI-based tasks, benchmarking it against existing methods.

## What is the motivation?

-   The paper addresses the problem of automating tasks through **graphical user
    interfaces (GUIs)** without relying on structured representations like
    **HTML or DOM trees**.
-   The motivation stems from limitations of prior approaches that require
    structured data, which is often unavailable or misaligned with visual
    elements in GUIs.
-   The potential impact is significant for **digital agents**,
    **accessibility**, and **automation**, especially in environments where
    structured data is inaccessible or incomplete.

## What is the novelty?

-   Prior work has focused on using **structured representations** (like HTML,
    DOM) for GUI-based tasks, which limits applicability when those
    representations aren't available.
-   This paper's main contribution is developing a **pixel-based agent**
    (PIX2ACT) that interacts with GUIs using **screenshots and generic actions**
    (mouse/keyboard), without relying on structured inputs.
-   The paper also shows **PIX2ACT** can outperform human crowdworkers on the
    **MiniWob++ benchmark** using only pixel inputs.

## What experiments were performed?

-   **Theory**: The paper builds on **PIX2STRUCT** for pixel-based pre-training
    and incorporates **Monte Carlo Tree Search (MCTS)** for policy improvement.
-   **Experiments**:
    -   **MiniWob++ benchmark**: PIX2ACT outperforms human crowdworkers and
        improves task scores compared to previous models that do not access DOM
        information.
    -   **WebShop benchmark**: PIX2ACT establishes a baseline, but there remains
        a gap compared to models using HTML inputs.
    -   **Ablation studies** show that **pre-training** on screenshots is
        critical for success, especially for tasks where structured data is
        unavailable.

## Dataset

-   The paper adapts two benchmarks for GUI-based tasks:
    -   **MiniWob++**: A set of over 100 web-based tasks requiring interaction
        through visual elements. The authors support 59 tasks using pixel-based
        inputs and mouse/keyboard actions.
    -   **WebShop**: A shopping environment with over 1.1 million products,
        where the task is to find and purchase items based on human-authored
        instructions.
-   Human demonstrations were used for both benchmarks, with **81% conversion
    success** from the MiniWob++ demonstrations into the new action format.
-   Tasks involve **natural language instructions** and various types of
    interactions like clicking, dragging, and scrolling.

## Methodology Highlights

-   The proposed **PIX2ACT model** relies solely on **pixel-based inputs**
    (screenshots) and uses **generic mouse/keyboard actions** to interact with
    GUIs.
-   Builds on **PIX2STRUCT**, a **Transformer-based image-to-text model**,
    pre-trained to map screenshots to structured HTML representations.
-   **Monte Carlo Tree Search (MCTS)** is used to improve policy decisions by
    exploring action sequences in the environment.
-   Uses **behavioral cloning** with human demonstrations and tree search to
    iteratively improve the model's performance.

## Training Details

-   **Optimizer**: Uses **Adafactor** with a learning rate of 0.01.
-   **MiniWob++**: Trained for **26K steps** with a batch size of 512. Policy
    improvement using tree search generated **826K episodes**.
-   **WebShop**: Finetuned on **MiniWob++** before 10K additional steps on
    WebShop-specific data with a batch size of 256.
-   **Input lengths**: Sequence lengths of **512 tokens** for MiniWob++ and
    **4096 tokens** for WebShop due to text-heavy data.