# FLAVA: A Foundational Language And Vision Alignment Model > Research note on FLAVA, a unified vision and language model that handles unimodal and multimodal tasks simultaneously using publicly available data. - Authors: Hunter Heidenreich - Published: 2021-12-09 - Updated: 2026-03-04 - Paper: FLAVA paper - Paper: CVPR proceedings - Project: FLAVA project site - Code: facebookresearch/multimodal (FLAVA) - Model: facebook/flava-full - Source: Bevaya Labs, --- ## What kind of paper is this? - This is a small idea with comparison paper. - It presents FLAVA, a unified vision and language model, and evaluates it on a broad set of tasks across vision, language, and multimodal domains. ## What is the motivation? - The paper aims to create a "foundational" model that can handle vision, language, and multimodal tasks simultaneously. - The problem addressed is that many models excel in either cross-modal or multimodal tasks but not both, limiting their generality. - The impact is the potential to improve performance across diverse tasks with a single model, using less data than competing models like [CLIP](2021_02_clip.md) and [ALIGN](2021_02_align.md). ## What is the novelty? - The paper builds on work from [CLIP](2021_02_clip.md), [ALIGN](2021_02_align.md), and other contrastive and fusion-based vision-language models. - It addresses the limitations of contrastive methods that struggle with multimodal tasks and fusion models that ignore unimodal performance. - The key contribution is the development of a model that jointly handles unimodal (vision or language) and multimodal tasks, along with using publicly available datasets for pretraining. ## What experiments were performed? - **Theory**: The paper introduces new multimodal objectives like global contrastive loss, masked multimodal modeling (MMM), and image-text matching (ITM), alongside unimodal losses for vision (MIM) and language (MLM). - **Experiments**: FLAVA achieves strong results across 35 tasks, outperforming similar models like [CLIP](2021_02_clip.md) in many multimodal and language tasks, and remains competitive in vision tasks, despite using smaller datasets for pretraining. - Results show FLAVA's strength in diverse domains, achieving superior "macro" average scores across vision, language, and multimodal benchmarks. ## Dataset #### Unimodal Data - **Unimodal Text**: - CCNews - BookCorpus - **Unimodal Image**: - ImageNet-1k #### Multimodal Data (Image-Text Pairs) The total number of image-text pairs used in FLAVA's pretraining is 70 million, derived from the following datasets: - **COCO**: 0.9M image-text pairs - **SBU Captions**: 1.0M image-text pairs - **Localized Narratives**: 1.9M image-text pairs - **Conceptual Captions**: 3.1M image-text pairs - **Visual Genome**: 5.4M image-text pairs - **Wikipedia Image Text**: 4.8M image-text pairs - **Conceptual Captions 12M**: 11.0M image-text pairs - **Red Caps**: 11.6M image-text pairs - **YFCC100M**: Filtered to 30.3M image-text pairs (after applying filters for English captions and removing captions with fewer than two words) ## Methodology Highlights The methodology used in the FLAVA paper revolves around a unified architecture and several pretraining objectives designed for multimodal, vision, and language tasks. #### Architecture - **Image Encoder**: A ViT-style encoder (Vision Transformer), specifically using the **ViT-B/16** configuration (B = base model, 16 = patch size). - **Text Encoder**: Mirrors the **ViT architecture** but uses different (non-shared) parameters from the vision encoder. This allows the model to handle text input in the same structural manner as images, but with distinct learned representations. - **Multimodal Encoder**: Also ViT-style, but introduces new parameters to fuse the outputs from the image and text encoders. This is crucial for the model's capability to reason about multimodal inputs. - **Overall Model Size**: The full model consists of three ViT-B/16 blocks (one for images, one for text, and one for multimodal), totaling approximately **450M parameters**. #### Multimodal Pre-Training Objectives 1. **Global Contrastive Loss**: A contrastive learning objective similar to that used in **CLIP**, where image-text pairs are brought closer in representation space, and mismatched pairs are pushed apart. This objective helps the model align visual and textual representations effectively. 2. **Masked Multimodal Modeling (MMM)**: This involves masking patches from images and tokens from text, and then reconstructing them. A **dVAE codebook** is used for the image encoder to tokenize image patches in a manner similar to tokenizing text. This joint masked modeling improves both vision and language understanding simultaneously. 3. **Image-Text Matching (ITM)**: A binary classification task where the model predicts whether a given image and text pair are correctly matched or not. This helps the model better understand paired multimodal data. #### Unimodal Pre-Training Objectives 1. **Masked Image Modeling (MIM)**: Applied to unimodal image datasets, this task involves masking random patches of the image and reconstructing them, similar to **BEiT**. It helps the model improve vision-only performance. 2. **Masked Language Modeling (MLM)**: A standard pretraining objective where the model masks 15% of text tokens and predicts them, similar to **BERT**. This boosts the model's language understanding capabilities when trained on large text corpora. ## Training Details #### Multimodal Pre-Training - **Batch Size**: 8192 - **Learning Rate**: Peak learning rate of 1e-3 with a warm-up of 10,000 steps - **Optimizer**: AdamW - **Weight Decay**: 0.1 - **Training Steps**: 150,000 #### Text Pre-Training - **Training Steps**: 125,000 - **Batch Size**: 2048 - **Learning Rate**: 5e-4 #### Vision Pre-Training - **Method**: DINO-based pretraining on unimodal vision data (e.g., ImageNet-1k)