# Flamingo: a Visual Language Model for Few-Shot Learning

> Research note on Flamingo, a visual language model that enables few-shot learning on multimodal tasks by bridging pretrained vision and language models.

- Authors: Hunter Heidenreich
- Published: 2022-04-29
- Updated: 2026-03-04
- Paper: Flamingo paper <https://arxiv.org/abs/2204.14198>
- Paper: NeurIPS proceedings <https://proceedings.neurips.cc/paper_files/paper/2022/hash/960a172bc7fbf0177ccccbb411a7d800-Abstract-Conference.html>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a **small idea with evaluation** paper, proposing a new Visual
    Language Model (VLM), Flamingo, for few-shot learning.
-   It introduces architectural innovations to integrate pre-trained vision and
    language models and evaluates performance across several benchmarks.

## What is the motivation?

-   The paper addresses the challenge of adapting models to new vision-language
    tasks with minimal annotated examples.
-   The problem is significant because existing approaches require substantial
    task-specific fine-tuning, which is time-consuming and data-intensive.
-   By solving this, the model enables rapid adaptation to new tasks in
    data-scarce regimes, making it impactful for multimodal AI applications.

## What is the novelty?

-   The paper builds on the recent advances in multimodal models and large
    language models (LLMs) like GPT-3 for few-shot learning.
-   Existing vision-language models focus on classification or retrieval tasks,
    but Flamingo extends this capability to open-ended tasks like captioning and
    visual question-answering.
-   The main contribution is a novel architecture that combines vision and
    language models with minimal task-specific training, outperforming existing
    methods with fewer examples.

## What experiments were performed?

-   **Experiments**: Flamingo is evaluated on 16 multimodal tasks, including
    COCO, VQAv2, and VATEX, outperforming the fine-tuned state of the art in 6
    tasks using only a few examples.
-   The model is trained on large-scale multimodal data using a mixture of
    image-text pairs, interleaved text and image sequences, and video-text
    pairs. The architecture leverages pretrained vision and language models with
    newly added cross-attention layers.
-   Results show that Flamingo achieves strong performance in both few-shot and
    fine-tuned settings, scaling well with model size and number of shots.

## Dataset

-   **M3W**:
    -   Extracted text and images from 43M webpages (closed-source).
    -   Mixture of text-image pairs with interleaved HTML content.
-   **ALIGN**:
    -   Large-scale dataset of 1.8B images paired with alt-text descriptions.
    -   Noisy due to poor alignment between images and text.
-   **LTIP (Long Text & Image Pairs)**:
    -   312M image-text pairs (closed-source), with more descriptive captions
        than ALIGN.
    -   Average 20.5 tokens per caption.
-   **VTP (Video & Text Pairs)**:
    -   27M short video-text pairs (closed-source).
    -   Average video length: 22 seconds.

## Methodology Highlights

-   **Vision Encoder**:
    -   Uses **NFNet-F6** model (Normalizer-free ResNet).
-   **Perceiver Resampler**:
    -   Summarizes visual input into 64 outputs.
    -   Flexible input size; supports sampled video frames.
    -   Multi-layer architecture with 6 layers, 1536 internal dimension, 16
        attention heads, and Squared ReLU activation.
    -   194M parameters in total.
-   **Text Decoder**:
    -   Frozen decoder with gated cross-attention mechanism.
    -   Gating uses scalar with **tanh** activation; initialized to stabilize
        training by starting with zero-weight cross-attention.
    -   Decoder cross-attends only to the most recent visual token.
-   **Model Sizes**:
    -   **Flamingo-3B**: 1.4B frozen LM + 1.2B learnable params.
    -   **Flamingo-9B**: 7B frozen LM + 1.6B learnable params.
    -   **Flamingo-80B**: 70B frozen LM + 10B learnable params.

## Training Details

-   **Optimizer**:
    -   **AdamW** with gradient norm clipping (1.0) and 0.1 weight decay (none
        on Perceiver Resampler).
-   **Learning Rate**:
    -   Peak learning rate of 1e-4, with 5,000 warmup steps.
-   **Training Duration**:
    -   Trained for 500k steps.
-   **Hardware**:
    -   Flamingo-80B model trained on **1536 TPUv4** chips for 15 days.