# dots.ocr: Unified Autoregressive Document Parsing > Analysis of dots.ocr, a 2.9B ViT-LLM that jointly performs layout detection, text recognition, and reading order as a single autoregressive sequence - Authors: Hunter Heidenreich - Published: 2025-12-01 - Updated: 2026-03-04 - Paper: dots.ocr Paper - Code: dots.ocr Code - Model: dots.ocr Models - Source: Bevaya Labs, --- ## TL;DR Li et al. introduce dots.ocr, a $\\sim$2.9B parameter ViT-LLM model that jointly performs layout detection, text recognition, and reading order prediction as a single autoregressive sequence for document pages. The model is trained using a three-stage multilingual data engine that combines teacher-student synthetic generation, large-scale auto-labeling, and human-corrected hard examples, and it reports competitive results on OmniDocBench (English and Chinese), a new 126-language benchmark (XDocParse), and olmOCR Bench. * * * ## What kind of paper is this? **Dominant:** $\\Psi\_{\\text{Method}}$ (unified VLM architecture and task formulation that treats document parsing as one sequence generation problem). **Secondary:** $\\Psi_{\\text{Resource}}$ (introduces XDocParse, a 126-language benchmark for end-to-end document parsing, and a large internal synthetic training corpus, though only the benchmark is intended for release); $\\Psi_{\\text{Evaluation}}$ (proposes a confidence-free two-stage F1 metric for layout detection and positions dots.ocr as a baseline on multiple public benchmarks). * * * ## What is the motivation? - **Fragmented pipelines.** Existing document systems often separate layout detection, OCR, and structure or reading order, which introduces error propagation and loses cross-task synergies. - **General VLMs are not ideal.** General-purpose VLMs can do high-level QA and summarization but struggle with precise localization, dense text, and large-scale throughput, mostly due to architecture and cost. - **Multilingual coverage is weak.** Training and evaluation datasets are heavily skewed toward English and a few high-resource languages; most languages have little labeled data. - **Data bottleneck.** Curating fully annotated, multilingual document corpora with layout and structure labels is expensive, so an end-to-end approach needs a different data strategy. * * * ## What is the novelty? ### Unified task formulation Document parsing is cast as one autoregressive sequence over semantic blocks. Each block is a triple $(B_k, c_k, t_k)$ where: - $B_k$: bounding box coordinates - $c_k$: block category (title, header, paragraph, table, figure, list, etc.) - $t_k$: textual content (plain text or LaTeX-style structured text for tables or formulas) Blocks are ordered according to reading order, so a single generation pass must jointly solve detection, recognition, and ordering. ### Model design ViT-LLM architecture with: - A **1.2B parameter vision encoder** trained from scratch on documents, designed for up to about **11M input pixels** (high-resolution pages). - A **1.7B parameter language decoder**, initialized from the Qwen2.5 1.5B base model with tied embeddings. Encoder training objective is set up to capture both fine-grained text and higher-level layout. ### Three-stage multilingual data engine 1. **Stage 1: Teacher-guided multilingual synthesis.** - Use Qwen2.5-VL-72B as a teacher VLM. - Given labeled English documents and their structural representation, the teacher produces layout-preserving renderings in target languages, which are then rendered to images, forming parallel multilingual seeds. - Distill this capability to a smaller Qwen2.5-VL-7B student model that can generate such documents much more cheaply. 2. **Stage 2: Curated large-scale auto-labeling.** - Apply the 7B student to a large pool of internal PDFs, selected via stratified sampling on layout complexity, language rarity, and domain. - Over-sample low-resource languages and complex layouts (multi-column, heavy tables, scientific diagrams) to offset dataset bias. - Student predictions convert millions of PDFs into structured training data. 3. **Stage 3: Human-in-the-loop targeted correction.** - Run the pre-trained dots.ocr model over diverse documents. - Use Qwen2.5-VL-7B Instruct as an oracle to audit outputs, flagging localization errors, incorrect types, omissions, and hallucinations by checking crops or masked regions. - Human annotators correct these high-confidence failure cases, creating a focused dataset of more than 15k samples that specifically target weaknesses. - This dataset is used for final supervised fine-tuning. ### New benchmark and metric - **XDocParse:** Real-world documents in 126 languages, used purely for evaluation of end-to-end parsing (Overall Edit, Text Edit, Table metrics, reading order). - **Layout detection metric:** A two-stage F1-based metric that first does one-to-one matching between predicted and ground truth boxes using the Hungarian algorithm, then clusters remaining boxes into super-boxes to handle one-to-many or many-to-many relationships. Supports category-aware and category-agnostic modes and avoids confidence scores, which autoregressive models do not naturally output. * * * ## What experiments were performed? ### OmniDocBench (English and Chinese) Primary end-to-end evaluation. Uses the benchmark from Ouyang et al. with rich annotations for text, tables, formulas, and reading order. Metrics: Overall Edit, Text Edit, Formula Edit, TableTEDS, Table Edit, Reading Order Edit. Baselines: - **Pipeline tools:** MinerU v1 and v2, Docling, PPStruct, Pix2Text, OpenParse, etc. - **Specialized OCR or document VLMs:** MonkeyOCR-pro-3B, OCRFlux, Dolphin, Mistral OCR, SmolDocling, GOT-OCR, olmOCR, Nougat, etc. - **General VLMs:** GPT-4o, Gemini-2.5-Pro, Qwen2-VL-72B, Qwen2.5-VL-72B, doubao models. ### XDocParse New benchmark constructed by the authors with documents in 126 languages. Same metric family as OmniDocBench. Baselines include MonkeyOCR-3B, doubao-1.5 and 1.6 variants, and Gemini-2.5-Pro. ### olmOCR Bench (supplementary) End-to-end evaluation on olmOCR Bench subsets: ArXiv, Old Scans, Math Tables, Headers and Footers, Multi-column, Long Tiny Text, and Base. Compared against tools like Marker, MinerU, Mistral OCR, GPT-4o, Gemini Flash 2, Qwen2-based models, Nanonets OCR, and olmOCR itself. ### Ablation studies 1. **Synergy of joint task learning.** Train variants that remove one component: - **M-Det:** no detection targets. - **M-Rec:** no recognition targets. - **M-RO:** no supervised reading order (replace with heuristic horizontal, vertical, or random ordering). Evaluate Overall Edit, Reading Order Edit, and detection F1. 2. **Unified versus specialist paradigms.** - **U $\\rightarrow$ U:** joint training and joint inference on all tasks (full dots.ocr). - **U $\\rightarrow$ S:** joint training, but specialize at inference to one task. - **S $\\rightarrow$ S:** train and evaluate on a single task only. 3. **Data engine ablations.** Remove one data pillar at a time: - **D-Multilingual:** no multilingual synthetic data. - **D-Structured:** no structured-heavy data (tables, formulas). - **D-Correction:** no targeted correction set. Evaluate impacts on Overall Edit, Reading Order Edit, and detection F1. ### Qualitative analyses Visual comparisons showing how removing recognition or reading order supervision leads to fragmented boxes, incorrect grouping of tables, and broken reading sequences. Examples of dots.ocr as a data engine: grounding-enhanced OCR, natural and scientific figure caption pairs, text inpainting masks, and next-page pairs for long-context modeling. * * * ## What are the outcomes and limitations? ### Outcomes **OmniDocBench performance:** - Overall Edit: **0.125 (EN)** and **0.160 (ZH)**, better than all listed baselines. For example, MonkeyOCR-pro-3B is at 0.138 (EN) and 0.206 (ZH), and Gemini-2.5-Pro is at 0.148 (EN) and 0.212 (ZH). - Text Edit: **0.032 (EN)** and **0.066 (ZH)**, lower than other methods. - TableTEDS: **88.6 (EN)** and **89.0 (ZH)**, at or above other models. - Reading Order Edit: **0.040 (EN)** and **0.067 (ZH)**, best among reported methods. **XDocParse performance:** - Overall Edit **0.177**, compared to **0.251** for Gemini-2.5-Pro and about **0.291--0.299** for doubao variants. - Text Edit **0.075**, roughly half of Gemini-2.5-Pro (0.163). **olmOCR Bench:** - Overall score **79.1% $\\pm$ 1.0**, higher than MonkeyOCR-pro-3B (75.8%) and olmOCR v0.1.75 anchored configuration (75.5%). **Evidence for task synergy:** - Removing detection supervision increases Reading Order Edit; removing recognition supervision slightly improves raw detection F1 but harms end-to-end metrics, suggesting recognition acts as a semantic regularizer. - Degrading reading order supervision (heuristic or random) harms both reading order and detection F1, indicating that sequence structure guides visual learning. **Unified paradigm vs specialists:** - Jointly trained and inferred model (U $\\rightarrow$ U) yields better recognition and reading order scores than both unified training with specialist inference (U $\\rightarrow$ S) and specialist-only (S $\\rightarrow$ S) models, while detection F1 stays similar across configurations. **Data engine impact:** - Dropping targeted correction (D-Correction) hurts detection F1 the most (from 0.849 to 0.788), showing that the small curated correction set is critical for high-quality localization. - Removing multilingual or structured data degrades Overall Edit and reading order, with structured data particularly important for Chinese and complex layouts. ### Limitations and open questions - **Data transparency.** The training corpus relies heavily on internal documents and synthetic data generated from proprietary VLMs; the paper does not quantify total document counts, tokens, or language distribution in detail. - **Compute and hardware.** Parameter counts are reported, but there are no explicit numbers for GPU types, training duration, or energy cost, which makes reproducibility at scale harder to assess. - **Benchmark scope.** XDocParse is multilingual, but the paper does not break down performance by language family or resource level, so it is unclear how well the model handles individual low-resource languages versus the aggregate. - **Real-world deployment.** The paper focuses on benchmark accuracy; there is little discussion of throughput, latency, memory usage, or robustness to noisy scans and edge cases in production. - **Data engine as VLM pretraining source.** The authors sketch how dots.ocr could serve as a data engine for future VLM pretraining (e.g., next-page prediction, inpainting, grounded supervision) but do not present experiments that actually use this data to improve a downstream VLM. * * * ## Model ### Architecture **Vision encoder:** - 1.2B parameters. - ViT-style encoder optimized for documents, trained from scratch rather than fine-tuning a natural-image model. - Supports native resolutions up to about 11M pixels, which allows full-page inputs without aggressive downsampling. **Language model decoder:** - Based on Qwen2.5 1.5B base, modified with tied word embeddings, resulting in 1.7B parameters. - Acts as an autoregressive decoder over a tokenized representation of bounding boxes, categories, and text content. **Vision-language connection:** - Adopts the standard ViT-LLM pattern: image tokens from the encoder are inserted or projected into the language model context; the decoder then generates the structured output sequence. **Output format:** - Bounding boxes are encoded numerically as tokens. - Categories are generated from a fixed vocabulary (e.g., title, header, paragraph, list, table, picture, formula, footer; see right panel of Figure 2). - Text content is emitted as standard subword tokens, with LaTeX-like markup for tables and formulas to capture internal structure. * * * ## Data ### Training data From the three-stage engine (exact sizes not always given): 1. **Seed multilingual structured data.** - English documents with annotations are transformed by the teacher VLM into multilingual, layout-preserving versions. - Covers many languages beyond English and Chinese. 2. **Curated large-scale auto-labeled data.** - Millions of internal PDFs sampled using heuristics for: - Layout complexity: number of columns, table density, presence of images. - Linguistic rarity: prioritizing low-resource languages. - Domain diversity: including scientific and niche domains. - 7B student VLM converts these into structured training examples. 3. **Targeted correction set.** - Over 15k samples, collected by running dots.ocr, auditing with an oracle VLM, and manually correcting errors. 4. **Supervised fine-tuning subset.** - The main supervised fine-tuning stage uses about 300k diverse samples, though it is not fully clear how these intersect with the stages above. ### Evaluation data - **OmniDocBench** (English and Chinese) for end-to-end parsing. - **XDocParse** which the authors curate from real-world multilingual documents (126 languages). - **olmOCR Bench** for additional validation on scientific and scanned documents. * * * ## Algorithms / Training **Unified training objective:** - Autoregressive next-token prediction over sequences representing bounding boxes, types, text, and separators between blocks. - Training implicitly couples detection, recognition, and reading order. **Pretraining and finetuning:** - Vision encoder trained from scratch jointly with the LM decoder under the unified objective. - Supervised finetuning on about 300k curated samples after large-scale pretraining. **Optimization:** - Optimizer: AdamW. - Peak learning rate: $5 \\times 10^{-5}$, with cosine decay schedule. - Other hyperparameters (batch size, warmup schedule, gradient clipping) are not detailed in the main text. **Teacher-student procedures:** - Teacher VLM (Qwen2.5-VL-72B) generates multilingual structured documents from English seeds; the 7B student is fine-tuned to imitate this behavior. - Oracle VLM (Qwen2.5-VL-7B Instruct) is used during Stage 3 to identify potential errors in model outputs; those are then corrected by humans and fed back as high-signal supervision. * * * ## Evaluation ### Metrics - **Edit distances:** Overall Edit summarizes combined errors across layout, text, and structure. Text, Formula, and Table Edit focus on components. - **Table structure:** TableTEDS for comparing predicted vs ground truth table structures. - **Reading order:** Reading Order Edit measures discrepancy between predicted and target reading sequences. - **Layout detection F1:** Two-stage matching and clustering metric described in Algorithm 1 of the supplement. ### Key results **OmniDocBench (EN / ZH):** - Overall Edit: 0.125 / 0.160 - Text Edit: 0.032 / 0.066 - Reading Order Edit: 0.040 / 0.067 **XDocParse:** - Overall Edit: 0.177 - Text Edit: 0.075 - TableTEDS: 79.2 - Reading Order Edit: 0.152 **olmOCR Bench:** - Overall score: 79.1% $\\pm$ 1.0, with strong performance on ArXiv, Multi-column, Long Tiny Text, and Base subsets; slightly behind MonkeyOCR-3B on some Old Scans categories but ahead overall. * * * ## Hardware / Production - The paper does not provide detailed hardware specs, training time, or energy use. - Parameter counts are modest compared to frontier VLMs (about 2.9B total), which suggests that training and inference are more affordable than very large VLMs, but still substantial. - There is no discussion of deployment details such as batching strategies, throughput on standard GPUs, or latency for single-page vs multi-page documents. * * * _Note: This analysis follows the Bevaya research note guidelines and classification taxonomy. For academic or production use, consult the original paper and verify claims through independent evaluation._