# PaLI-3: Vision Language Models: Smaller, Faster, Stronger > Research note on PaLI-3, a 5B parameter vision-language model that demonstrates smaller models with contrastive pretraining can match or outperform much larger models. - Authors: Hunter Heidenreich - Published: 2023-10-13 - Updated: 2026-03-04 - Paper: PaLI-3 Paper - Paper: OpenReview (ICLR 2024 Submission) - Source: Bevaya Labs, --- ## What kind of paper is this? - This is a **comparison and performance evaluation** paper. - It focuses on contrasting two pretraining methods (classification vs. contrastive) for Vision Transformers (ViT) in the context of vision-language models (VLMs). - It also introduces a new model (PaLI-3) and assesses its state-of-the-art performance on several benchmarks. ## What is the motivation? - The motivation is to create a smaller, faster, and more efficient vision-language model, PaLI-3, that can compete with much larger models in terms of performance. - The problem addressed is scaling VLMs while maintaining or improving performance on key benchmarks like localization and visually-situated text understanding. - The impact lies in demonstrating that models do not need to be extremely large to be state-of-the-art, potentially encouraging more research into efficient VLMs. ## What is the novelty? - The paper builds upon previous works like PaLI, PaLI-X, and others that pretrain vision encoders either with classification or contrastive objectives. - Existing approaches show promise in scaling but rely on larger models. This paper's contribution is showing that a 5B parameter model (PaLI-3) with contrastive pretraining can match or outperform larger models. - It also introduces a 2B SigLIP-based image encoder and shows superior performance in multimodal tasks. ## What experiments were performed? - **Experiments**: PaLI-3 achieves new state-of-the-art results on several benchmarks including RefCOCO, TextVQA, and multilingual cross-modal retrieval, all while being smaller in size. - Results show that contrastive pretraining (SigLIP) is particularly effective for tasks requiring visually-situated text understanding and localization. ## Dataset - Uses a mixture of web-scale image-text datasets for pretraining, including WebLI, CC3M-35L, and others. - Additional datasets for multimodal tasks include RefCOCO (for referring expression segmentation) and VQA datasets (TextVQA, OCRVQA, etc.). - WebLI dataset was filtered for quality control using a model-based approach, retaining around 40% of image-text pairs. - Document understanding tasks were enhanced with datasets containing dense text images such as posters and PDFs across 100+ languages. ## Methodology Highlights - **Pretraining Objectives**: Contrastive pretraining (SigLIP) for vision encoder, focusing on image-text alignment over noisy web-scale data. UL2 pretraining for the text encoder-decoder. - **Architecture**: 2B parameter ViT-G/14 vision encoder paired with a 3B UL2 encoder-decoder. - **Multimodal Training**: Vision encoder frozen in the early stages, later fine-tuned at increasing resolutions. - **Ablations**: Comparison between classification-pretrained (JFT) and contrastive-pretrained (SigLIP) vision encoders reveals significant improvements in visually-situated text understanding and localization tasks. ## Training Details - **Stage 0 (Unimodal Pretraining)**: SigLIP contrastive objective applied to the vision encoder on image-text data. UL2 model pretraining as per original UL2 framework. - **Stage 1 (Multimodal Training)**: Vision encoder frozen, trained across a mixture of captioning, VQA, and object detection tasks at 224x224 resolution. - **Stage 2 (Resolution Increase)**: Gradual increase in image resolution during fine-tuning (812x812 and 1064x1064) with the vision encoder unfrozen, improving detailed understanding.