# MonkeyOCR: Structure-Recognition-Relation for Document Parsing > Analysis of MonkeyOCR, a 3B-parameter document parsing system using a Structure-Recognition-Relation triplet paradigm with YOLO detection and unified LMM - Authors: Hunter Heidenreich - Published: 2025-06-15 - Updated: 2026-03-04 - Paper: MonkeyOCR Paper - Code: MonkeyOCR Code - Model: MonkeyOCR-pro-3B Model - Source: Bevaya Labs, --- ## TL;DR MonkeyOCR is a 3B-parameter document parsing system built on a **Structure-Recognition-Relation (SRR)** triplet paradigm: layout detection (Where), block-level recognition (What), and reading-order prediction (How). Trained on **MonkeyDoc**, a bilingual dataset with 3.9M block-level instances across 10+ document types, it achieves Edit distance **0.140 (EN)** and **0.297 (ZH)** on OmniDocBench, outperforming MinerU, Marker, and general VLMs like Qwen2.5-VL-7B. Multi-page inference reaches **0.84 pages/sec** on A800 hardware, 29% faster than MinerU (0.65 pages/sec). * * * ## What kind of paper is this? **Dominant:** $\Psi_{\text{Method}}$ (introduces SRR decomposition, unified LMM with type-aware prompts, block-level relation model, and end-to-end training procedure). **Secondary:** $\Psi_{\text{Resource}}$ (MonkeyDoc dataset with 3.9M instances, bilingual coverage, and 3-stage construction pipeline is a substantial contribution). **Also present:** $\Psi_{\text{Evaluation}}$ (large-scale benchmark comparison on OmniDocBench with 981 pages, but no new metric or evaluation protocol). Rough superposition: `0.55 Method + 0.35 Resource + 0.10 Evaluation`. ## What is the motivation? **Pipeline tool error propagation:** Traditional multi-stage toolchains (e.g., MinerU-style: layout detection $\rightarrow$ crop $\rightarrow$ separate recognizers for text/table/formula) accumulate errors across stages. The paper illustrates formula cropping mistakes causing hallucinated superscripts when recognition operates on incorrect regions. **End-to-end efficiency constraints:** Full-page VLM inference faces quadratic attention cost and slow throughput on dense, high-resolution documents (e.g., academic papers with hundreds of text lines). General VLMs like Qwen2.5-VL-7B achieve only 0.12 pages/sec on multi-page workloads. **Goal:** Design a decomposition that maintains accuracy while improving scalability and throughput for real-world document parsing at scale. ## What is the novelty? **SRR triplet paradigm:** Explicitly decomposes document parsing into three questions answered by specialized components: - **Structure (Where is it):** YOLO-based layout detector identifies regions and types. - **Recognition (What is it):** Unified Large Multimodal Model (LMM) processes cropped blocks with type-specific prompts (e.g., "Recognize the text in this formula block"). - **Relation (How is it organized):** Block-level reading-order model predicts sequence over detected regions to reassemble content. > **Contrast to pipeline tools:** SRR avoids multi-tool error propagation by using a unified LMM for all content types (text, tables, formulas, code) rather than separate specialized recognizers. This tighter integration shares representations while maintaining modularity for efficient parallelization. > **Contrast to end-to-end VLMs:** SRR operates on block-level crops rather than full pages, enabling parallel processing of detected regions and avoiding the $O(n^2)$ attention cost of full-page high-resolution inference. **Type-aware block recognition:** A single 3B LMM handles all content types by conditioning on element type via prompts. This contrasts with pipeline tools that route different element types (text, table, formula) to specialized models, and with general VLMs that process full pages without type-specific guidance. **MonkeyDoc dataset:** 3.9M block-level instances spanning layout detection, reading-order prediction, and recognition (text, tables, formulas, code blocks). Bilingual (Chinese + English), multi-domain (books, slides, financial reports, arXiv papers, etc.), built via a 3-stage pipeline: 1. **Structure Detection:** Harmonizes layout annotations from M6Doc, DocLayNet, D4LA, CDLA into 11 classes; adds 300k+ Chinese pages with auto-annotation and manual correction (41k samples). 2. **Content Recognition:** Crops 1.9M elements from layout annotations; refines PubTabNet (470k tables); adds UniMER-1M formulas; synthesizes 526k Chinese tables/formulas; extracts 36k arXiv LaTeX samples. 3. **Relation Prediction:** Refines DocGenome reading-order labels (951k samples); adds 154k manually annotated Chinese samples; auto-annotates 78k samples via PPOCR + LayoutReader. ## What experiments were performed? **Primary benchmark:** OmniDocBench (981 PDF pages, 9 document types, 4 layout styles, 3 language categories: English, Chinese, mixed). **Baselines:** - **Pipeline tools:** MinerU, Marker, Nougat, PaddleOCR, Mathpix. - **Expert VLMs:** GOT-OCR, Mistral OCR, Docling. - **General VLMs:** GPT-4o, Gemini 2.5 Pro, Qwen2.5-VL-7B, InternVL3-8B, Idefics3-8B-Llama3. **Evaluation tasks:** - **Overall parsing:** Edit distance on full document recovery (text + structure + formulas + tables). - **Text recognition:** Edit distance on plain text blocks. - **Formula recognition:** CDM (Character Detection Match) and ExpRate (expression-level exact match). - **Table recognition:** TEDS (Tree Edit Distance-based Similarity). - **Reading order:** Sequence accuracy over detected blocks. **Document type breakdown:** Separate results for books, slides, financial reports, research papers, exams, magazines, textbooks, newspapers, handwriting. **Throughput comparison:** Single-page and multi-page inference speed (pages/sec) on A800 GPUs for MonkeyOCR-3B, MinerU, Qwen2.5-VL-7B. **Ablation (implicit):** The paper reports a Chinese-specialized variant (**MonkeyOCR\***) trained with additional Chinese data, showing improved Chinese edit distance but no detailed ablation of SRR components. ## What are the outcomes/limitations? **Outcomes:** - **Overall end-to-end parsing on OmniDocBench:** MonkeyOCR-3B achieves Edit distance **0.140 (EN)** and **0.297 (ZH)**, leading among open-source models. Gemini 2.5 Pro achieves slightly better Chinese results (0.265 ZH) but comparable English (0.141 EN). - **Formula recognition gains:** +15.0% average improvement over MinerU on CDM and ExpRate metrics. - **Table recognition gains:** +8.6% average improvement over MinerU on TEDS. - **Multi-page throughput:** 0.84 pages/sec vs. 0.65 pages/sec (MinerU) and 0.12 pages/sec (Qwen2.5-VL-7B), a 29% improvement over MinerU. - **Single-page throughput:** 0.24 pages/sec vs. 0.28 pages/sec (MinerU), slightly slower on single-page workloads. - **Document type breakdown:** Best on books, slides, financial reports; competitive on research papers, textbooks, magazines. **Limitations and open questions:** - **Chinese gap remains:** Even with MonkeyOCR\* (Chinese-specialized variant), Gemini 2.5 Pro achieves better Chinese edit distance (0.265 vs. 0.297 for base MonkeyOCR, value for MonkeyOCR\* not clearly stated). The paper does not explain whether this gap stems from dataset coverage, model capacity, or linguistic features. - **Single-page throughput tradeoff:** MonkeyOCR is 14% slower than MinerU on single-page inference (0.24 vs. 0.28 pages/sec), suggesting the SRR overhead (detection + relation prediction) may not amortize on short documents. - **Missing ablations:** The paper does not isolate the contribution of each SRR component (layout quality, recognition accuracy, reading-order correctness). It is unclear whether gains come primarily from the unified LMM, the type-aware prompts, or the relation model. - **Reproducibility gaps:** Code and models are not yet released. The paper provides high-level architecture descriptions but omits: - LMM backbone architecture (transformer specs, vision encoder details, fusion mechanism). - Prompt templates for type-aware recognition (exact text for "Recognize the text in this formula block"). - Loss functions and training objectives per stage (layout loss, recognition loss, relation loss). - Hyperparameters beyond optimizer/schedule (batch size per stage, gradient accumulation, warmup steps). - **Deployment cost:** Training requires 53 hours on 32 A800 GPUs. The paper does not provide cost-per-page estimates for inference or compare resource efficiency to MinerU/olmOCR (which report $176/M pages on L40S for olmOCR). - **Generalization beyond MonkeyDoc domains:** OmniDocBench covers 9 document types, but real-world parsing spans invoices, forms, receipts, technical diagrams, and other specialized layouts not represented in the 10+ domains listed for MonkeyDoc. Cross-domain generalization is not evaluated. * * * ## Model ### High-level architecture MonkeyOCR decomposes document parsing into a **Structure-Recognition-Relation (SRR)** pipeline with three specialized components: 1. **Structure detection:** YOLO-based layout detector outputs bounding boxes and element types for each region. 2. **Recognition:** Unified Large Multimodal Model (LMM) processes cropped blocks with type-specific prompts. 3. **Relation prediction:** Block-level reading-order model predicts sequence over detected regions. Final output reassembles recognized content in predicted reading order. ### Structure detection (YOLO-based) **Architecture:** YOLO object detector adapted for document layout analysis. **Output:** Bounding boxes (x, y, width, height) + element type (11 classes). **Element types (harmonized from MinerU conventions):** - Text blocks - Titles/headings - Tables - Figures - Formulas - Code blocks - Captions - Footnotes - Headers/footers - Page numbers - Other **Training data:** Aggregates and harmonizes layout annotations from M6Doc, DocLayNet, D4LA, CDLA. Cleaning rules include: - Remove nested boxes (keep largest region). - Filter low-information boxes (area < 35% of page). **Chinese supplementation:** 300k+ collected pages, pre-annotated, post-processed to 28k high-quality samples, plus 13k manually corrected samples. ### Recognition (unified LMM with type-aware prompts) **Model:** 3B-parameter Large Multimodal Model (LMM). Architectural details (backbone, vision encoder, fusion mechanism) not specified in the paper. **Input:** Cropped image region from structure detection + type-specific text prompt. **Prompt format (examples, exact templates not provided):** - Text block: "Recognize the text in this text block" - Formula: "Recognize the text in this formula block" - Table: "Recognize the table structure and content" - Code: "Recognize the code in this code block" **Output:** Structured text (plain text for text blocks, LaTeX for formulas, HTML for tables, etc.). **Key design choice:** A single unified model handles all content types via prompt conditioning, rather than routing to specialized models (e.g., separate formula recognizer, table structure recognizer). This enables shared representations and parallel processing of cropped regions. ### Relation prediction (reading-order model) **Task:** Predict a total ordering over detected blocks to reassemble content in reading sequence. **Input:** Detected block positions (bounding boxes) + types from structure detection. **Output:** Permutation over block indices (e.g., [3, 1, 5, 2, 4] for 5 blocks). **Architecture:** Not specified in the paper. Likely a transformer or graph neural network operating on block embeddings (position + type features). **Training data:** 951k samples from DocGenome (refined reading-order labels), 154k manually annotated Chinese samples, 78k auto-annotated samples (PPOCR + LayoutReader). ## Data ### MonkeyDoc overview **Size:** 3.9M block-level instances. **Coverage:** 10+ document types (books, slides, financial reports, research papers, exams, magazines, textbooks, newspapers, handwriting, contracts). **Languages:** Chinese + English (bilingual). **Tasks supported:** - Layout detection (structure) - Reading-order prediction (relation) - Text recognition - Table recognition - Formula recognition - Code block recognition **Construction pipeline:** 3-stage process mirroring SRR paradigm (Structure Detection $\rightarrow$ Content Recognition $\rightarrow$ Relation Prediction). ### Structure Detection data construction **Base datasets:** M6Doc, DocLayNet, D4LA, CDLA. **Harmonization:** Converts original label sets to 11 unified classes following MinerU conventions. **Cleaning rules:** - Remove nested boxes: When multiple boxes overlap, keep only the largest region. - Filter low-information boxes: Remove boxes with area < 35% of page (likely page margins, headers, footers with minimal content). **Chinese supplementation:** - Collect 300k+ Chinese pages from web/internal sources. - Pre-annotate with existing layout models. - Post-process and quality-filter to 28k high-quality samples. - Add 13k manually corrected Chinese samples. **Total structure detection samples:** Not explicitly stated (aggregated from multiple datasets). ### Content Recognition data construction **Element cropping:** Crop 1.9M blocks from layout annotations (text blocks, figures, captions, etc.). **Partial element labeling:** Use Gemini 2.5 Pro to annotate cropped regions without ground-truth OCR labels. **Table recognition:** - Refine PubTabNet (original 568k tables) with quality checks. - Final table dataset: 470k high-quality samples. **Formula recognition:** - Use UniMER-1M (multi-source formula dataset: arXiv, textbooks, exams). **Synthesized Chinese tables/formulas:** - Generate 526k additional Chinese samples via synthesis pipeline (details not provided). **arXiv LaTeX extraction:** - Extract 36k academic paper samples with LaTeX source for tables and formulas. **Total recognition samples:** 1.9M (elements) + 470k (tables) + UniMER-1M (formulas) + 526k (Chinese synthesis) + 36k (arXiv) $\approx$ 3M+ instances. ### Relation Prediction data construction **DocGenome refinement:** - Original DocGenome dataset provides reading-order annotations. - Refine labels for consistency and accuracy. - Final: 951k samples with reading-order sequences. **Manual Chinese reading-order annotation:** - Annotate 154k Chinese document samples with reading order. **Auto-annotation pipeline:** - Run PPOCR line recognition on document images. - Apply LayoutReader reading-order model to predicted text lines. - Filter and validate auto-annotations. - Final: 78k additional samples. **Total relation prediction samples:** 951k + 154k + 78k = 1.183M samples. ## Algorithms / Training ### Optimizer and schedule **Optimizer:** AdamW. **Learning rate:** 2e-5. **Schedule:** Cosine annealing (details on warmup steps, min LR not provided). **Batch size:** 64 (likely global batch size across GPUs, per-device batch size not specified). ### Training procedure (inferred) The paper does not provide detailed training objectives or multi-stage training procedures. Likely approach based on SRR decomposition: 1. **Structure detection (YOLO):** Standard object detection loss (bounding box regression + classification). 2. **Recognition (LMM):** Autoregressive language modeling loss on recognition targets (text, LaTeX, HTML), conditioned on cropped image + type prompt. 3. **Relation prediction:** Sequence prediction loss (e.g., cross-entropy over permutation or pairwise ordering loss). **End-to-end training:** The paper does not clarify whether SRR components are trained jointly or sequentially. ### Training hardware and duration **Hardware:** 32 A800 GPUs (80GB VRAM each). **Duration:** 53 hours for full 3B model training. **Estimated compute:** 32 GPUs $\times$ 53 hours = 1,696 GPU-hours on A800 hardware. ## Evaluation ### OmniDocBench overall parsing (Edit distance) Edit distance measures character-level differences between predicted and ground-truth full document text (including structure markers, formula LaTeX, table HTML, etc.). Lower is better. | Method | EN Edit $\downarrow$ | ZH Edit $\downarrow$ | | -------------------- | -------------------: | -------------------: | | **MonkeyOCR-3B** | **0.140** | **0.297** | | Gemini 2.5 Pro | 0.141 | 0.265 | | GPT-4o | 0.215 | 0.398 | | Qwen2.5-VL-7B | 0.278 | 0.425 | | InternVL3-8B | 0.312 | 0.467 | | MinerU | 0.333 | 0.350 | | Marker | 0.418 | 0.512 | | GOT-OCR | 0.389 | 0.521 | MonkeyOCR-3B achieves best English results and competitive Chinese results (trailing only Gemini 2.5 Pro). The Chinese-specialized variant MonkeyOCR\* improves Chinese scores but exact values are not provided in the available text. ### Formula recognition (CDM and ExpRate) **CDM (Character Detection Match):** Character-level F1 score on LaTeX output. **ExpRate:** Expression-level exact match (entire formula correct). The paper reports **+15.0% average improvement** over MinerU on formula recognition but does not provide absolute CDM/ExpRate values or a structured comparison table. ### Table recognition (TEDS) **TEDS (Tree Edit Distance-based Similarity):** Measures structural similarity between predicted and ground-truth table HTML. Higher is better (range 0--1). The paper reports **+8.6% average improvement** over MinerU on TEDS but does not provide absolute values or a structured comparison table. ### Reading-order accuracy The paper does not report standalone reading-order metrics (e.g., Kendall's Tau, pairwise ordering accuracy). Reading-order quality is implicitly reflected in overall edit distance. ### Throughput comparison | Method | Single-page (pages/sec) | Multi-page (pages/sec) | | -------------- | ----------------------: | ---------------------: | | MonkeyOCR-3B | 0.24 | 0.84 | | MinerU | 0.28 | 0.65 | | Qwen2.5-VL-7B | 0.12 | 0.12 | **Multi-page inference:** MonkeyOCR achieves **29% higher throughput** than MinerU (0.84 vs. 0.65 pages/sec), likely due to block-level parallelism. **Single-page inference:** MonkeyOCR is **14% slower** than MinerU (0.24 vs. 0.28 pages/sec), suggesting SRR overhead (detection + relation prediction) may not amortize on short documents. ### Document type breakdown The paper provides per-document-type results (books, slides, financial reports, research papers, etc.) but numeric values are not included in the available text. Key qualitative findings: - **Best performance:** Books, slides, financial reports. - **Competitive performance:** Research papers, textbooks, magazines. - **Challenging:** Handwriting (likely due to limited handwriting data in MonkeyDoc). ## Hardware / Production ### Training infrastructure **Hardware:** 32 NVIDIA A800 GPUs (80GB VRAM each). **Duration:** 53 hours for full 3B model training. **Estimated cost:** At typical cloud pricing ($3--4/hour per A800), training cost is approximately $5,000--$7,000. ### Inference infrastructure **Single-GPU inference:** Runs on RTX 3090 (24GB VRAM) via LMDeploy (quantization/optimization toolkit). **Throughput:** - **Multi-page:** 0.84 pages/sec (A800 hardware, reported in paper). - **Single-page:** 0.24 pages/sec (A800 hardware, reported in paper). **Cost-per-page estimates:** Not provided in the paper. ### Deployment options **LMDeploy integration:** The paper mentions RTX 3090 deployment via LMDeploy, suggesting support for: - INT8/INT4 quantization for reduced memory footprint. - TensorRT optimization for faster inference. - Multi-GPU serving for high-throughput workloads. **Serving infrastructure:** Not discussed in the paper. Likely requires custom FastAPI/Triton setup (similar to PaddleOCR 3.0) for production deployment. ## Implementation sketch ### Python API (inferred from paper description) The paper does not provide code examples. Based on SRR decomposition, a likely API structure: ```python from monkeyocr import MonkeyOCR # Initialize parser parser = MonkeyOCR(model="MonkeyOCR-3B") # Single-page parsing result = parser.parse("document.pdf", page=0) # Access SRR outputs layout = result.structure # Bounding boxes + types blocks = result.recognition # Recognized content per block reading_order = result.relation # Block sequence # Reassembled document text (in reading order) full_text = result.text full_markdown = result.to_markdown() full_html = result.to_html() # Multi-page parsing with parallelism results = parser.parse_batch(["doc1.pdf", "doc2.pdf"], batch_size=4) ``` ### Type-aware recognition prompt format (inferred) The paper mentions "type-specific prompts" but does not provide templates. Likely format: ```python # Text block prompt = "Recognize the text in this text block" # Formula prompt = "Recognize the text in this formula block" # Table prompt = "Recognize the table structure and content" # Code block prompt = "Recognize the code in this code block" ``` * * * ## Notes and open questions ### Observations **SRR paradigm tradeoffs:** The Structure-Recognition-Relation decomposition achieves strong accuracy (0.140 EN edit distance) while improving multi-page throughput over pipeline tools (29% faster than MinerU). However, single-page inference is 14% slower, suggesting the detection + relation overhead may not amortize on short documents. This positions MonkeyOCR for batch processing workloads (e.g., large-scale document archives) rather than interactive single-page parsing. **Chinese gap persists:** Despite a bilingual MonkeyDoc dataset (3.9M instances) and a Chinese-specialized variant (MonkeyOCR\*), Gemini 2.5 Pro achieves better Chinese edit distance (0.265 vs. 0.297 for base MonkeyOCR). This gap may stem from: - Dataset coverage: MonkeyDoc's Chinese samples (41k structure, 154k relation, 526k synthesis) may underrepresent linguistic complexity (classical Chinese, technical terminology, etc.). - Model capacity: A 3B LMM may lack capacity for dense Chinese character vocabularies (tens of thousands of characters vs. ~26 letters in English). - Formula/table challenges: Chinese documents often mix CJK characters with formulas and tables, requiring robust multimodal understanding. **Unified LMM design:** Using a single 3B model for all content types (text, tables, formulas, code) with type-aware prompts contrasts with pipeline tools that route to specialized recognizers. This design choice enables shared representations and parallel processing but may sacrifice per-type accuracy (e.g., a dedicated formula recognizer might outperform a general LMM prompted for formulas). The paper does not ablate this tradeoff. ### Open questions **SRR component ablations:** What is the contribution of each stage (structure, recognition, relation) to overall accuracy? Specifically: - **Structure quality:** How much does layout detection accuracy (mAP, IoU) correlate with end-to-end edit distance? - **Recognition quality:** If we assume perfect layout detection (oracle boxes + types), how much does edit distance improve? - **Relation quality:** If we assume perfect recognition but random reading order, how much does edit distance degrade? Without these ablations, it is unclear whether MonkeyOCR's gains come primarily from better layout detection, superior LMM recognition, or improved reading-order modeling. **Prompt engineering details:** The paper mentions "type-specific prompts" but does not provide templates or ablate prompt design. Key questions: - What is the exact prompt format? (e.g., "Recognize the text in this [TYPE] block" vs. more detailed instructions?) - Does the LMM receive only the prompt + cropped image, or additional context (e.g., neighboring blocks, document metadata)? - How sensitive are results to prompt wording? (e.g., "Recognize" vs. "Extract" vs. "Transcribe") **Loss functions and training objectives:** The paper specifies optimizer (AdamW) and learning rate (2e-5) but omits: - Structure detection loss (likely standard YOLO loss, but hyperparameters?). - Recognition loss (autoregressive language modeling? CTC? Seq2seq with teacher forcing?). - Relation prediction loss (cross-entropy over permutations? Pairwise ordering loss? Pointer network?). - Multi-task balancing: Are structure, recognition, and relation trained jointly or sequentially? If jointly, how are losses weighted? **Model availability:** The 3B-parameter model is publicly available under Apache 2.0 license at [echo840/MonkeyOCR-pro-3B](https://huggingface.co/echo840/MonkeyOCR-pro-3B), enabling independent validation of the reported OmniDocBench results (0.140 EN, 0.297 ZH). Key reproducibility gaps remain: - LMM architecture: Backbone (LLaMA, Qwen, etc.)? Vision encoder (CLIP, SigLIP, etc.)? Fusion mechanism (cross-attention, prefix tuning, etc.)? - Training data licenses: MonkeyDoc aggregates M6Doc, DocLayNet, D4LA, CDLA, PubTabNet, UniMER-1M. Are all components commercially usable? - Evaluation protocol: OmniDocBench preprocessing (PDF to images? Resolution? Page cropping?), postprocessing (text normalization? LaTeX cleanup?), and metric implementation (edit distance algorithm, TEDS version). **Deployment cost and efficiency:** The paper reports throughput (0.84 pages/sec multi-page) but not cost-per-page. Key questions for production planning: - What is the dollar cost per million pages on A800/H100/L40S hardware? - How does MonkeyOCR compare to olmOCR ($176/M pages on L40S) or MinerU2.5 (1.224 pages/sec on A100)? - Can LMDeploy quantization (INT8/INT4) maintain accuracy while reducing cost? **Generalization beyond OmniDocBench domains:** MonkeyDoc covers 10+ document types, but real-world parsing spans invoices, receipts, forms, technical diagrams, and domain-specific layouts (legal, medical, government). The paper does not evaluate zero-shot transfer to unseen document types or provide guidance on fine-tuning for new domains.