# CLIP: Learning Transferable Visual Models From Natural Language Supervision > Research note on CLIP, a contrastive learning framework that trains visual models from natural language supervision for zero-shot transfer. - Authors: Hunter Heidenreich - Published: 2021-02-26 - Updated: 2026-03-04 - Paper: CLIP paper - Paper: PMLR proceedings - Code: openai/CLIP - Model: CLIP ViT-B/32 - Model: CLIP ViT-B/16 - Model: CLIP ViT-L/14 - Model: CLIP ViT-L/14@336px - Source: Bevaya Labs, --- ## What kind of paper is this? - This paper introduces a new method, and a new idea for how to train multimodal models. It is a method paper but also a bit of a big idea paper. - It focuses on image-text pairs for zero-shot learning, combining contrastive learning approaches with large-scale pre-training. ## What is the motivation? - Problem: The current state-of-the-art (SOTA) computer vision systems require large, labeled datasets (e.g., ImageNet) to perform well. This reliance on supervised learning limits their generality and applicability to new tasks, as new labeled data is often needed. - Importance: Learning directly from raw text about images could remove this limitation, enabling broader, more flexible, and more scalable visual learning. - Impact: The potential for zero-shot transfer of visual models is significant. The authors show that their model, CLIP, can match SOTA supervised models (like ResNet50) without using any labeled training examples, enabling a broader set of tasks like OCR, action recognition, and more. ## What is the novelty? - Related work: Previous work in image-caption learning (Joulin et al., VirTex, ConVIRT) demonstrated the potential of text-based supervision for learning visual representations but lacked scalability and efficiency compared to fully supervised models. - Contribution: The paper introduces CLIP, a contrastive learning framework that scales efficiently to large datasets of 400 million image-text pairs. The contribution lies in the contrastive pre-training approach and the demonstration of its ability to perform zero-shot transfer to a wide variety of downstream tasks. ## What experiments were performed? Experiments: They validate this by testing the model on over 30 computer vision datasets. The results show that CLIP outperforms supervised models on many tasks in a zero-shot setting and is more robust to natural distribution shifts than models like ResNet50. - They achieve 76.2% accuracy on ImageNet zero-shot, matching ResNet50 trained on labeled data. - CLIP outperforms supervised models in tasks like action recognition and fine-grained object classification, though it underperforms on more specialized tasks like traffic sign recognition and satellite imagery. ### What are the implications of the paper? (Broader Impact) - Technological implications: The ability to create visual classifiers without specific training data could revolutionize areas like autonomous systems, medical imaging, surveillance, and content moderation. It offers a more scalable and adaptable approach to visual understanding. - Stakeholders: Researchers and engineers developing AI systems will benefit from the flexibility of this approach. However, communities affected by the deployment of AI (e.g., for surveillance) may experience increased privacy concerns. - Positive impact: The potential to enable broad transferability of visual models without relying on labeled datasets could drastically reduce the time and cost of developing AI for new domains. - Negative consequences: There are concerns about bias and fairness, as the model may reinforce existing social biases present in the training data. Additionally, the ease of use for surveillance tasks raises privacy concerns. ## Dataset - WebImageText (WIT) - 400M text-image pairs collected from the internet. - 500k queries generated using common words from Wikipedia, high pointwise mutual information bigrams, and WordNet synsets. - Restricted to 20k image-text pairs per query to maintain class balance. - Filtered to exclude low-quality pairs (e.g., automatically generated filenames or non-descriptive text). - Covers a broad range of visual concepts and is comparable in size to the WebText dataset used for GPT-2. - **Availability:** Proprietary / Non-public. ### Method - Contrastive Language-Image Pre-training (CLIP) - **Objective**: Predict which text is paired with which image in a batch of `N` image-text pairs, generating `N^2` possible pairings (`N` true, `N^2 - N` false). - **Contrastive Loss**: Uses cosine similarity between image and text embeddings. A symmetric cross-entropy loss is applied to maximize similarity for correct pairs and minimize it for incorrect ones. - **Multimodal Embedding Space**: - **Image Encoder**: ResNet or Vision Transformer (ViT). - **Text Encoder**: Transformer-based model. - Both are linearly projected into a shared embedding space and **L2 normalized**. - **Note:** This is a dual-encoder formulation in contrast to fusion encoder formulations that mix all modalities as equal tokens. - **Temperature Scaling**: A learned temperature parameter controls the sharpness of the similarity distribution. - **Data Augmentation**: Random square cropping for images; simplified text processing (typically single sentences). ### Architecture - **Text Encoder**: - **Transformer-based**, BERT-style. - **63M parameters**. - Configuration: **12 layers**, **512 dimensions**, and **8 attention heads**. - **Lower-cased vocabulary** of 50k tokens. - **Maximum sequence length**: 76. - The text encoder's **width is scaled** to match the size of the vision model for different configurations. - **Vision Encoders**: - **ResNet-style** (with slight modifications to pooling): - **ResNet-50**, **ResNet-101**, and scaled variants: - **RN50x4** (4x wider), **RN50x16**, **RN50x64**. - ResNet improvements include **ResNetD** modifications (He et al., 2019) and **antialiased blur pooling** (Zhang, 2019). - All ResNet weights are available in the [GitHub repo](https://github.com/openai/CLIP). - **ViT-style** (Vision Transformer): - **ViT-B/32** (151M parameters): [Hugging Face Model](https://huggingface.co/openai/clip-vit-base-patch32). - **ViT-B/16** (150M parameters): [Hugging Face Model](https://huggingface.co/openai/clip-vit-base-patch16). - **ViT-L/14** (428M parameters): [Hugging Face Model](https://huggingface.co/openai/clip-vit-large-patch14). - **ViT-L/14@336px** (428M parameters): Trained at higher 336px resolution for improved performance. [Hugging Face Model](https://huggingface.co/openai/clip-vit-large-patch14-336). ### Training - **Optimizer**: AdamW. - Weight decay of 0.1. - Learning rate set via **cosine learning rate schedule**. - **Epochs**: 32. - **Batch size**: 32k images per batch. - **Techniques**: - **Gradient checkpointing** to reduce memory usage. - **Mixed precision** and **half precision (FP16)** for faster training and lower memory consumption. - **Distributed training** across multiple GPUs (e.g., trained on up to 592 V100 GPUs for the largest model). ### Hardware and Training Times - **Largest ResNet (RN50x64)**: - **18 days** on **592 V100 GPUs**. - **Largest ViT (ViT-L/14)**: - **12 days** on **256 V100 GPUs**. - **Compute efficiency**: ViT models demonstrated higher compute efficiency compared to ResNet models in this setup. ### Benchmarks: - **Zero-shot image classification** on over **30 datasets** spanning a wide variety of tasks. - Includes standard datasets such as **ImageNet**, **CIFAR-10**, **CIFAR-100**, and **PascalVOC**. - **Fine-grained classification** tasks (e.g., **Stanford Cars**, **FGVC Aircraft**, **Flowers102**). - **Specialized tasks** such as **OCR**, **geo-localization**, and **action recognition**. - **Robustness benchmarks**: Evaluated on **distribution shifts** using datasets like **ImageNetV2**, **ImageNet-R**, **ImageNet-A**, and **ObjectNet**. - Achieved competitive performance with **fully supervised models** (e.g., ResNet50) in a zero-shot setting without using labeled training data. - **Zero-shot CLIP outperforms** supervised models on tasks like **OCR** and **action recognition**, while underperforming on **specialized tasks** such as satellite imagery (e.g., **EuroSAT**), traffic sign recognition (e.g., **GTSRB**), and **object counting** (e.g., **CLEVRCounts**). - Additional analysis includes **few-shot performance** comparison and general **robustness to natural distribution shifts**.