# TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document > Research note on TextMonkey, an OCR-free multimodal model for document understanding with shifted window attention and token resampling. - Authors: Hunter Heidenreich - Published: 2024-03-07 - Updated: 2026-03-04 - Paper: TextMonkey Paper - Code: TextMonkey GitHub - Source: Bevaya Labs, --- ## What kind of paper is this? - This is a **large multimodal model (LMM) development paper** with a specific focus on document understanding, without relying on OCR. - It presents a new model, **TextMonkey**, optimized for text-centric tasks such as document analysis and scene text spotting. ## What is the motivation? - **Motivation**: Existing OCR-dependent models create limitations in document processing by adding complexity and often misalign text and visual context, leading to error accumulation. - **Impact**: TextMonkey aims to streamline document analysis by integrating cross-modal understanding without OCR, potentially setting new standards in document processing for both industry and academic applications. ## What is the novelty? - **Related Work**: Traditional approaches rely heavily on OCR to extract text, but OCR introduces errors and extra engineering needs, especially in complex document layouts. - **Contribution**: TextMonkey eliminates OCR reliance, proposing a novel attention mechanism (Shifted Window Attention) for cross-window context. Additionally, it introduces token compression strategies to handle high-resolution images and complex tasks like text grounding and structured text extraction. ## What experiments were performed? - The paper's theoretical contributions include Shifted Window Attention with zero initialization to stabilize training and token resampling to reduce redundancy. - **Experiments**: TextMonkey outperforms several benchmarks across 12 tasks, achieving a 5.2% improvement in scene text-centric tasks, a 6.9% increase in document-oriented tasks, and a 2.8% rise in key information extraction. It also sets a new standard on OCRBench with a score of 561, surpassing prior large multimodal models. ## Dataset - **Scene Text**: Datasets include COCOText, OCRText, HierText, TextVQA, and MLT. These support model training for scene text understanding, including text spotting and VQA. - **Document-Based VQA**: Training uses IIT-CDIP, DocVQA, ChartQA, InfoVQA, DeepForm, KLC (Kleister Charity), and WikiTableQuestions (WTQ), focusing on document question answering, table understanding, and information retrieval. - **Evaluations**: - **Scene Text VQA**: Benchmarks include STVQA, TextVQA, and OCRVQA. - **Document-Oriented VQA**: Benchmarks such as DocVQA, InfoVQA, and ChartQA assess the model's document comprehension. - **Key Information Extraction (KIE)**: Benchmarks like FUNSD, SROIE, and POIE test the model's ability to extract structured information. ## Methodology Highlights - **Shifted Cross-Window Attention**: Introduces cross-window attention at intervals in the Transformer layers, enabling the model to capture long-range dependencies across image patches. - **Zero Initialization for Cross-Window Attention**: Inspired by LoRA, this stabilizes early training by gradually introducing cross-window connections. - **Token Resampling for Efficiency**: Redundant visual tokens are filtered by their cosine similarity, retaining only the top tokens to reduce token count. The remaining tokens attend to discarded tokens to capture essential information. ## Training Details - **Optimizer**: AdamW with cosine decay scheduling. - **Learning Rate and Schedule**: Starts at 1e-5 with a warmup over 150 steps, cooling down to 5e-6. - **Batch Size**: 128 per training step. - **Training Time**: 12 days on A800 GPUs for a single epoch. - **Regularization**: Applies weight decay of 0.1 to prevent overfitting.