# PaLI-X: On Scaling up a Multilingual Vision and Language Model

> Research note on PaLI-X, a scaled multilingual vision and language model achieving new results across more than 25 benchmarks.

- Authors: Hunter Heidenreich
- Published: 2023-05-29
- Updated: 2026-03-04
- Paper: PaLI-X Paper <https://arxiv.org/abs/2305.18565>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **multilingual vision and language model paper**.
-   It focuses on scaling up the PaLI-X model for vision-language tasks and
    evaluates it across various benchmarks.
-   The paper proposes improvements in model architecture, training mixtures,
    and evaluation methods for complex tasks like image captioning, VQA, and
    few-shot learning.

## What is the motivation?

-   **Motivation**: To investigate the impact of scaling vision-language models
    (PaLI-X) in both size and task complexity, inspired by similar scaling
    efforts in language models.
-   **Importance**: Vision-language models play a critical role in tasks like
    image captioning and VQA, which are essential for applications in AI-driven
    document understanding, multimodal learning, and interaction systems.
-   **Impact**: The paper aims to push the state-of-the-art (SoTA) across a wide
    range of vision-language benchmarks and demonstrate emerging capabilities,
    including multilingual object detection and complex counting.

## What is the novelty?

-   **Related Work**: Builds on models like PaLI, Flamingo, and ViT (vision
    transformers). It also leverages approaches in few-shot learning, multimodal
    models, and scaling trends in both language and vision models.
-   **Contribution**:
    -   Demonstrates that scaling both the vision and language components leads
        to significant performance improvements.
    -   Introduces a novel training mixture that combines self-supervision and
        full supervision.
    -   Establishes new SoTA results across more than 25 benchmarks, showcasing
        advances in tasks such as image captioning, VQA, and document
        understanding.

## What experiments were performed?

-   **Experiments**:
    -   Fine-tuning experiments on diverse benchmarks (e.g., COCO, VQAv2,
        NoCaps).
    -   Few-shot learning tasks that evaluate multilingual captioning and
        question-answering.
    -   Performance on video-based tasks and object detection tasks.
-   **Results**:
    -   PaLI-X outperforms previous models, achieving new SoTA results in tasks
        like image captioning, VQA, and complex counting.
    -   Significant improvements in document and infographic understanding.
    -   Few-shot learning capabilities demonstrated strong multilingual
        performance.
-   **Emerging capabilities**: PaLI-X exhibits abilities in multilingual object
    detection and complex counting that were not explicitly part of the training
    set.

## Dataset

-   **Image captioning**: COCO, Conceptual Captions 3M (CC3M), Conceptual
    Captions 12M (CC12M), SBU, Visual Genome.
-   **Video-text**: WebVid-2M.
-   **Text-only**: WikiCorpus (20GB) and cleaned Common Crawl (350GB), mirroring
    the C4 dataset.

## Methodology Highlights

-   **Dual Vision Encoder**: A module that supports both images and videos,
    capturing spatial and temporal features using ViT-B/16 and ViT-L/14 models.
-   **Text Encoder**: BERT-based unimodal encoder, initialized from pre-trained
    checkpoints.
-   **Universal Module**: Aligns vision and language features using text-aware
    vision tokens and vision-aware text tokens, reducing computational
    complexity and aligning modalities in a shared semantic space.
-   **Fusion Module**: Combines self-attention, cross-attention, and
    feed-forward networks (FNNs) to update and refine cross-modal
    representations.
-   **Shared Decoder**: Supports text generation for both uni-modal and
    multi-modal tasks.
-   **Pre-training Losses**:
    -   **Language Loss**: Masked Language Modeling (MLM) for text encoding.
    -   **Image-Text Contrastive (ITC)**: Aligns image and text representations.
    -   **Image-Text Matching (ITM)**: Matches image and text pairs, improving
        multi-modal understanding.
    -   **Instruction-based LM**: Handcrafted instructions for task and modality
        discrimination.

## Training Details

-   **Pre-training**: 30 epochs.
-   **Batch size**:
    -   **Base model**: 1024 on 8 NVIDIA A100 GPUs.
    -   **Large model**: 512 on 16 NVIDIA A100 GPUs.
-   **Optimizer**: AdamW with a weight decay of 0.02.
-   **Learning Rate**:
    -   Warmup for 5000 steps, followed by cosine decay.
    -   **Base model**: Peak learning rate of 1e-4.
    -   **Large model**: Peak learning rate of 5e-5.
-   **Data Augmentation**: Random cropping, horizontal flip, and sparse frame
    sampling for videos.