# CoCa: Contrastive Captioners are Image-Text Foundation Models

> Research note on CoCa, an image-text foundation model that unifies contrastive learning and image captioning in a single encoder-decoder architecture.

- Authors: Hunter Heidenreich
- Published: 2022-05-04
- Updated: 2026-03-04
- Paper: CoCa paper <https://arxiv.org/abs/2205.01917>
- Paper: TMLR OpenReview <https://openreview.net/forum?id=Ee277P3AYC>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **model proposal and evaluation** paper.
-   It proposes a new image-text foundation model called **Contrastive Captioner
    (CoCa)**, unifying contrastive learning and image captioning.

## What is the motivation?

-   The paper aims to improve **image-text foundation models** by merging
    contrastive learning and captioning, building on methods like
    [CLIP](2021_02_clip.md) and [SimVLM](2021_08_simvlm.md).
-   The impact lies in creating a single, **unified model** that performs well
    across multiple tasks (e.g., image classification, image captioning,
    crossmodal retrieval) with **zero-shot transfer** or minimal task-specific
    adaptation.

## What is the novelty?

-   Related works include models like **[CLIP](2021_02_clip.md)**,
    **[ALIGN](2021_02_align.md)**, and **[SimVLM](2021_08_simvlm.md)**, which
    focus on contrastive learning and generative tasks, respectively.
-   Prior methods either excelled at crossmodal alignment (contrastive) or
    multimodal understanding (captioning), but none effectively combined both.
-   CoCa contributes by **combining contrastive and captioning losses** in a
    minimalist encoder-decoder architecture that **subsumes capabilities** from
    both contrastive models (e.g., [CLIP](2021_02_clip.md)) and captioning
    models (e.g., [SimVLM](2021_08_simvlm.md)).

## What experiments were performed?

-   **Experiments**: CoCa achieves **state-of-the-art results** in a variety of
    tasks:
    -   **Zero-shot ImageNet classification** (86.3% accuracy).
    -   **Crossmodal retrieval** on MSCOCO and Flickr30k.
    -   **Multimodal understanding** (e.g., VQA, SNLI-VE) and **image
        captioning** (e.g., MSCOCO, NoCaps).

## Dataset

-   **JFT-3B**: Internal, closed dataset used for image classification
    pretraining.
-   **ALIGN**: Noisy image-text dataset used for contrastive pretraining.
-   Strict **deduplication** applied to filter near-domain examples for fair
    evaluation.

## Methodology Highlights

-   **Architecture**: Encoder-decoder model with an image encoder and text
    decoder.
    -   **Unimodal and multimodal decoders**: Omits cross-attention in earlier
        decoder layers to produce unimodal text representations and multimodal
        image-text representations.
    -   **Attentional pooling layer**: Learned atop the vision encoder for
        task-specific pooling.
-   **Unified approach**: Combines **contrastive** and **captioning** losses in
    a single architecture.
-   **Model sizes**: 383M, 787M, 2.1B parameters across different model
    variants.
-   Pretrained at **288x288** image resolution; additional **1 epoch** at
    **576x576**.

## Training Details

-   **Batch size**: 65,536 image-text pairs (half from JFT, half from ALIGN).
-   **Training steps**: 500k steps (5 epochs on JFT, 10 epochs on ALIGN).
-   **Loss weighting**: Captioning loss weighted twice as much as contrastive
    loss.
-   **Optimizer**: Adafactor with weight decay set to 0.01.
-   **Learning rate schedule**: Linear warmup for the first 2% of steps to a max
    LR of 8e-4, then linearly decayed.
-   **Hardware**: Training took 5 days on **2,048 TPUv4 chips**.