# PaLI: A Jointly-Scaled Multilingual Language-Image Model

> Research note on PaLI, a multilingual language-image model that jointly scales vision and language components to advance multimodal tasks.

- Authors: Hunter Heidenreich
- Published: 2022-09-14
- Updated: 2026-03-04
- Paper: PaLI paper <https://arxiv.org/abs/2209.06794>
- Paper: ICLR OpenReview <https://openreview.net/forum?id=mWVoBz4W0u>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **large-scale model introduction and performance evaluation**
    paper.
-   It introduces the PaLI model, which jointly scales language and vision
    components to perform multimodal tasks across multiple languages.

## What is the motivation?

-   The paper seeks to address the challenge of joint vision-language modeling
    at scale, using both modalities more effectively.
-   It emphasizes the importance of balancing scaling between vision and
    language models, claiming this approach improves accuracy across tasks like
    captioning and visual question-answering.
-   The potential impact is providing a scalable, multilingual, multimodal model
    that advances the state-of-the-art in vision-language tasks.

## What is the novelty?

-   Related works include models like SimVLM, CoCa, Flamingo, and Vision
    Transformers (ViTs), which focus on vision-language tasks but often
    prioritize language scaling over vision scaling.
-   The paper contributes by:
    -   Introducing PaLI, which balances parameter scaling between the vision
        and language components.
    -   Training a massive multilingual dataset (WebLI), which supports over 100
        languages, and achieves SOTA performance in various vision-language
        tasks.

## What experiments were performed?

-   **Theory**: The paper demonstrates that scaling vision models (ViT-e)
    provides substantial performance gains when compared to smaller vision
    backbones, especially in joint tasks.
-   **Experiments**:
    -   PaLI outperforms previous SOTA models in benchmarks such as COCO
        Captioning, VQAv2, and OKVQA.
    -   The 17B-parameter version achieves better results than models with
        significantly skewed parameter distributions between vision and language
        components.

## Dataset

-   **WebLI dataset**: 10 billion images, tens of billions of image-text pairs.
-   **Multilingual**: Supports over 100 languages.
-   **Image-text pairs**: Filtered for quality using cross-modal similarity
    scoring, top 10% retained (~1 billion examples).
-   **Closed-source**: Built using internal tools, limited details on
    construction.
-   **OCR data**: Includes 29 billion image-OCR pairs.

## Methodology Highlights

-   **Pre-training tasks**: 8 diverse tasks designed to enhance multimodal
    capabilities.
    -   Span corruption on text-only data (similar to masked language modeling).
    -   Split-captioning task where captions are split and completed by the
        model.
    -   CC3M captioning in 35 languages.
    -   OCR task using WebLI OCR-text data.
    -   English and multilingual VQA and VQG.
    -   English-only object-aware VQA.
    -   Object detection task similar to pix2seq.
-   **Architecture**:
    -   Pre-trained vision (ViT-G, ViT-e) and language (mT5-L, mT5-XXL) models.
    -   Standard ViT encoder feeding into a fusion encoder with a text decoder.
    -   ViT weights are frozen during training, updating only the language
        model.

## Training Details

-   **Hardware**: Largest model trained on 1024 TPUv4 chips for 7 days.
-   **High-resolution phase**: An additional 3 days on 512 TPUv4 chips.
-   **Optimizer**: Adafactor, with a peak learning rate of 5e-3.
-   **Data size**: Pre-training uses 260TB of data.
-   **Training objective**: Standard softmax cross-entropy with teacher forcing.