# KOSMOS-1: Language Is Not All You Need

> Research note on KOSMOS-1, a multimodal large language model that aligns perception with language models for tasks like image captioning, VQA, and nonverbal reasoning.

- Authors: Hunter Heidenreich
- Published: 2023-02-27
- Updated: 2026-03-04
- Paper: KOSMOS-1 Paper <https://arxiv.org/abs/2302.14045>
- Paper: NeurIPS 2023 Conference Paper <https://proceedings.neurips.cc/paper_files/paper/2023/hash/e425b75bac5742a008d643826428787c-Abstract-Conference.html>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## What kind of paper is this?

-   This is a research paper introducing a **new model**: KOSMOS-1, a multimodal
    large language model (MLLM).
-   It proposes aligning perception (vision) with language models to handle
    multimodal tasks like image captioning, visual question answering, and
    nonverbal reasoning.

## What is the motivation?

-   The paper addresses the **limitations** of current large language models
    (LLMs) in handling **multimodal data** like images and audio.
-   The goal is to move toward **artificial general intelligence** by aligning
    multimodal perception (e.g., vision) with language.
-   The impact lies in enabling **new tasks**, such as robotics, document
    intelligence, and zero-shot multimodal reasoning, by training a model that
    can perceive, reason, and generate across modalities.

## What is the novelty?

-   Related work includes other multimodal models like **Flamingo** and language
    models like **METALM**, which treat language models as general-purpose
    interfaces.
-   Existing approaches are insufficient because they do not natively align
    perception with LLMs or handle **instruction following** and **in-context
    learning** across modalities.
-   This paper contributes KOSMOS-1, which integrates **multimodal input** into
    a **Transformer-based model**, allowing it to learn and follow instructions
    across both language and vision tasks.

## What experiments were performed?

-   KOSMOS-1 extends LLMs to multimodal tasks, allowing them to perceive and
    understand input from different modalities.
-   **Experimental results**:
    -   KOSMOS-1 outperforms comparable models (e.g., Flamingo) in zero-shot and
        few-shot settings across tasks like **image captioning** and **visual
        question answering**.
    -   It demonstrates the ability to perform **nonverbal reasoning** on IQ
        tests and handle **OCR-free** tasks.
    -   KOSMOS-1 also shows **cross-modal transfer** capabilities, transferring
        knowledge between modalities (e.g., from language to vision).

## Dataset

-   **Unimodal text**: Includes **The Pile**, **Common Crawl**, **CC
    snapshots**, **CC-Stories**, and **RealNews**.
-   **Exclusions**: Data from **GitHub**, **arXiv**, **Stack Exchange**, and
    **PubMed Central** are excluded.
-   **Image-caption pairs**: Constructed from **English LAION-2B**,
    **LAION-400M**, **COYO-700M**, and **Conceptual Captions** datasets.
-   **Interleaved multimodal data**: Created by extracting and **interleaving
    text and images** from Common Crawl web pages (approx. 71M web pages).
-   **Instruction tuning data**: Includes **Unnatural Instructions** and
    **FLANv2** (random 54k samples).

## Methodology Highlights

-   **Architecture**: Uses **ViT-L/14** as the vision encoder with **Resampler**
    (attentive pooling); ViT is kept **frozen** except for the last layer.
-   **Input representation**: Encoded image tokens wrapped with special
    `<image>` and `</image>` tags.
-   **Pretraining task**: Standard **next-token prediction** for text,
    image-caption, and interleaved multimodal data.
-   **Position encoding**: **xPos** (relative position encoding) is used for
    long-context modeling in the **Magneto** Transformer variant.
-   **Multimodal chain-of-thought**: In some tasks, the model generates an
    intermediate rationale to improve reasoning performance.

## Training Details

-   **Pretraining setup**:
    -   **Steps**: 300k total with 375 warmup steps.
    -   **Batch sizes**: 256 for text, 6144 for image-caption pairs, and 128 for
        interleaved data.
    -   **Learning rate**: Max of 2e-4 with linear decay.
    -   **Optimizer**: **AdamW** with `eps=1e-6`, `beta=(0.9, 0.98)`, and weight
        decay of 0.01.
-   **Language-only instruction tuning**:
    -   **Steps**: 10k with 375 warmup steps.
    -   **Batch sizes**: 256 for instructions, 32 for text, 768 for
        image-caption pairs, 16 for interleaved data.
    -   **Learning rate**: Max of 2e-5.