# VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks > Research note on VisionLLM, a framework that unifies vision and language tasks using an LLM-based open-ended task decoder with language-guided image tokenization. - Authors: Hunter Heidenreich - Published: 2023-05-18 - Updated: 2026-03-04 - Paper: VisionLLM Paper - Paper: NeurIPS 2023 Conference Paper - Code: OpenGVLab/VisionLLM - Source: Bevaya Labs, --- ## What kind of paper is this? - This is a **big idea** paper proposing a new model and framework, VisionLLM, that unifies vision and language tasks. - It introduces a large language model-based framework for vision-centric tasks, providing flexibility beyond current vision foundation models (VFMs). ## What is the motivation? - The motivation is to bridge the gap between large language models' (LLMs) open-ended task capabilities and the pre-defined nature of vision foundation models (VFMs). - The paper aims to unlock flexible vision task customization using language instructions, leading to a unified approach to handling diverse tasks like object detection, segmentation, and image captioning. - Its potential impact includes setting a new baseline for generalist vision-language models and making it easier to manage vision-centric tasks through a unified framework. ## What is the novelty? - Related work includes large language models like GPT-4, visual-language models like Flamingo, and vision generalist models such as OFA and Pix2Seq. - Existing approaches are limited to pre-defined tasks or visual prompt tuning that doesn't fully integrate LLMs. - This paper's contribution is VisionLLM, which aligns vision tasks with LLM methodologies and introduces components like language-guided image tokenization and an LLM-based open-ended task decoder. ## What experiments were performed? - **Theory**: VisionLLM redefines how vision tasks are formulated, using language instructions to structure outputs, allowing for customizable task handling. - **Experiments**: It achieves over 60% mAP on COCO for object detection, performs well on other tasks like visual grounding and image captioning, and shows the ability to customize both task descriptions and output formats. - The model's generalist nature allows for strong performance on tasks that usually require separate models. ## Dataset - **COCO2017**: Used for object detection and instance segmentation tasks. - **RefCOCO/RefCOCOg/RefCOCO+**: Employed for visual grounding tasks. - **LLaVA-Instruct**: Synthetic dataset used for instruction fine-tuning, connecting vision and language. ## Methodology Highlights - **Unified LLM-based Decoder**: Applies to both vision-only and vision-language tasks, allowing flexible task customization using language instructions. - **Language-Guided Image Tokenizer**: Extracts multi-scale visual features (via ResNet/InternImage-H) and updates them using cross-attended language features from a BERT-like encoder. - **Special Tokens**: Augments the LLM with position and class tokens for object localization and classification. - **Training Stages**: Two-stage training process: - Stage 1: Freezes the LLM while training the visual backbone. - Stage 2: Freezes the visual backbone and fine-tunes the LLM with LoRA-based updates. ## Training Details - **50 epochs** with **AdamW** optimizer and **cosine annealing** learning rate schedule. - Peak learning rate: **2e-4**. - Hardware: **4x8 NVIDIA A100s**, processing one sample per GPU.