# OmniDocBench: Benchmarking Diverse PDF Document Parsing > A benchmark for end-to-end document parsing evaluation across 9 document types, with an adjacency search matching algorithm and multi-level metrics for text, tables, formulas, and reading order. - Authors: Bevaya AI Team - Published: 2024-12-10 - Updated: 2026-03-15 - Paper: Preprint (arXiv) - Paper: Published (CVPR 2025) - Benchmark: OmniDocBench (Code & Evaluation Kit) - Dataset: OmniDocBench (Dataset) - Source: Bevaya Labs, --- ## TL;DR OmniDocBench is a document parsing benchmark with 981 annotated PDF pages across 9 document types (academic papers, textbooks, newspapers, handwritten notes, etc.), 19 layout categories, and 15 attribute labels. It introduces a multi-level evaluation pipeline with an Adjacency Search Match algorithm that addresses paragraph boundary mismatches between model predictions and ground truth. Evaluations of pipeline tools and VLMs reveal that pipelines excel on structured layouts while VLMs generalize better to degraded or unconventional documents. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$ (Benchmark dataset). The headline contribution is the dataset itself: 981 pages with comprehensive annotations (bounding boxes, text, LaTeX formulas, HTML/LaTeX tables, reading order, and fine-grained attributes) spanning 9 document types. The paper's main novelty is the reusable evaluation infrastructure rather than a new model or algorithm. **Secondary:** $\Psi\_{\text{Evaluation}}$ (Novel evaluation pipeline). The Adjacency Search Match algorithm and multi-level evaluation protocol (end-to-end, component-level, attribute-level) represent a meaningful measurement contribution that addresses the "granularity mismatch" problem in document parsing evaluation. ## What is the motivation? The authors identify three gaps in existing document parsing benchmarks: 1. **Task misalignment**: Benchmarks like PubLayNet and DocLayNet measure layout detection via `mAP` (bounding box accuracy). RAG systems and LLM data pipelines need valid Markdown or text output, not just correctly drawn boxes. 2. **Narrow document diversity**: Most evaluation sets focus on clean academic papers (e.g., arXiv). Real-world pipelines encounter noisy scans, multi-column newsletters, handwritten notes, exam papers, and financial reports. 3. **Granularity mismatch**: End-to-end VLMs produce continuous text output, while ground truth annotations are stored as separate blocks. Standard text similarity metrics (edit distance, BLEU) fail when prediction paragraph boundaries do not align with ground truth boundaries, penalizing correct content unfairly. ## What is the novelty? OmniDocBench introduces two main contributions: ### Hierarchical Taxonomy The annotation schema covers: - **Block-Level (15 categories)**: Text paragraphs, headings, tables, figures, charts, display formulas, lists, code blocks, headers, footers, page numbers, TOC, index, captions, and footnotes. - **Span-Level (4 categories)**: Text lines, inline formulas, subscripts/superscripts, and footnote markers (nested within block-level annotations). - **Attributes (15 labels)**: 6 page-level attributes (PDF type, layout type, language, watermark, fuzzy scan, colored background) and 9 bbox-level attributes (text language, text background, text rotation, table language, table frame type, merged cells, formulas in tables, colorful tables, rotated tables). ### Adjacency Search Match To handle the paragraph boundary mismatch between model output and ground truth, the authors propose a dynamic matching algorithm. The process operates in two phases: 1. Compute a matrix of Normalized Edit Distance (NED) similarity between all prediction and ground truth blocks: $$ \text{NED}(T\_{\text{pred}}, T\_{\text{gt}}) = 1 - \frac{\text{EditDistance}(T\_{\text{pred}}, T\_{\text{gt}})}{\max(|T\_{\text{pred}}|, |T\_{\text{gt}}|)} $$ Pairs exceeding a similarity threshold are matched directly. 2. For unmatched blocks, apply fuzzy matching to detect substring relationships, then iteratively merge adjacent ground truth (or prediction) blocks until NED stops improving. This approach penalizes fragmentation less severely than strict one-to-one matching while still enforcing semantic correctness. The final reported metrics use Edit Distance (lower is better) rather than the NED similarity score. ## What experiments were performed? ### Data Construction The dataset was curated through a human-in-the-loop pipeline: 1. **Sourcing**: Over 200k PDFs from Common Crawl, Google, Baidu search engines, and internal data. 2. **Visual Clustering**: ResNet-50 features extracted from pages, clustered with Faiss, and 6,000 visually diverse pages sampled from 10 cluster centers. 3. **Annotation**: - **Pre-annotation**: LayoutLMv3 (layout detection), PaddleOCR (text), UniMERNet (formulas), GPT-4o (tables). - **Correction**: Human annotators refined bounding boxes, reading order, and character-level content. - **Expert Review**: Three researchers reviewed unrenderable formulas and tables using CDM rendering techniques. 4. **Final Selection**: 981 pages balanced across 9 document types. ### Evaluation Protocol Models are evaluated at three levels: - **End-to-End**: Full page Markdown text similarity using Normalized Edit Distance. - **Component-Level**: Text (Edit Distance), Formula (BLEU and CDM, i.e., Character Detection Matching), Table (TEDS, i.e., Tree-Edit-Distance-based Similarity), Reading Order (Edit Distance). - **Attribute-Level**: Performance breakdown by language, rotation, handwriting, background color, table frame type, and other attributes. ### Models Evaluated Three categories of methods: - **Pipeline Tools**: MinerU (v0.9.3), Marker (v1.2.3), Mathpix. - **Expert VLMs**: GOT-OCR 2.0, Nougat. - **General VLMs**: GPT-4o, Qwen2-VL-72B, InternVL2-76B. ## What are the outcomes/conclusions? Key findings from the benchmark: 1. **Pipeline tools outperform VLMs on standard documents.** MinerU achieves the best overall Edit Distance on English text (0.061) and reading order. Pipeline tools benefit from explicit layout segmentation that preserves reading order better than autoregressive generation. 2. **VLMs generalize better to unconventional formats.** GPT-4o and Qwen2-VL perform more robustly on handwritten notes, slides, and visually degraded pages (fuzzy scans, watermarks, colored backgrounds). Their broader pretraining data helps handle long-tail document types that pipeline tools were not specifically trained for. 3. **VLMs struggle with high-density layouts.** On newspapers and dense multi-column pages, VLMs frequently miss content or hallucinate due to input resolution and token length limitations. Pipeline tools, which process each region independently, maintain higher accuracy on these layouts. 4. **Chinese lags behind English across all methods.** Nearly all models show lower accuracy on Chinese pages compared to English, likely reflecting training data imbalances. 5. **Component-level results reveal specialization.** For table recognition, OCR-based models (RapidTable) lead overall. For formula recognition, GPT-4o achieves the highest CDM score (86.8%), while UniMERNet leads on Normalized Edit Distance. DocLayout-YOLO dominates layout detection (47.38 mAP) and powers MinerU's strong end-to-end performance. ### Limitations - The paper does not report hardware specifications, training costs, or statistical variance (error bars, multiple runs) for the benchmark evaluations. - The dataset focuses on single-page parsing; multi-page document understanding (e.g., cross-page tables) is not addressed. - Inline formulas are converted to Unicode for evaluation rather than being assessed in LaTeX format, which limits evaluation granularity for mathematical content. ## Mapping to Matter vs. Meaning Framework How does OmniDocBench's hierarchical taxonomy map to our [Matter vs. Meaning framework](../matter-vs-meaning/)? ### Block-Level Categories (15) | OmniDocBench Class | Visual Primitive | Logical Role | Notes | | :--- | :--- | :--- | :--- | | **Text paragraph** | `Text` | `Body` | Standard prose content. | | **Heading** | `Text` | `Title` / `SectionHeader` | No distinction between document title and section headers. | | **Table** | `Table` | *(primitive only)* | Tabular regions evaluated via TEDS. | | **Figure** | `Image` | `Figure` | Photographs, illustrations. | | **Chart** | `Image` | `Chart` | Data visualizations (bar, line, pie). Notably separated from Figure. | | **Display formula** | `Formula` | `DisplayEquation` | Block-level math, evaluated via CDM and BLEU. | | **List** | `Text` | `ListItem` | Annotated as container blocks, not individual items. | | **Code block** | `Text` | `Code` | Computer source code. | | **Header** | `Text` | `PageHeader` | Running headers. | | **Footer** | `Text` | `PageFooter` | Running footers. | | **Page number** | `Text` | `PageNumber` | Explicit page indices. | | **TOC** | `Table` | `TOC` | Table of contents. Treated as a grid structure. | | **Index** | `Table` | `Index` | Back-of-book index. | | **Caption** | `Text` | `Caption` | Descriptions of figures, tables, charts. | | **Footnote** | `Text` | `Footnote` | Bottom-of-page notes. | ### Span-Level Categories (4) | OmniDocBench Class | Visual Primitive | Logical Role | Notes | | :--- | :--- | :--- | :--- | | **Text line** | `Text` | *(sub-element)* | Line-level segmentation within blocks. | | **Inline formula** | `Formula` | `InlineEquation` | Math embedded within text. Converted to Unicode for evaluation. | | **Subscript/Superscript** | `Text` | *(sub-element)* | Positional text variants. | | **Footnote marker** | `Text` | *(sub-element)* | Reference marks linking text to footnotes. | ### Attribute Labels (15) OmniDocBench defines 6 page-level and 9 bbox-level attributes. These partially map to the M-vs-M Attributes layer: `text rotation` maps to `orientation`, `text background` maps to visual context. Most OmniDocBench attributes (language, frame type, fuzzy scan) are metadata for evaluation stratification rather than structural annotations. **Coverage gaps:** No explicit classes for `Author`, `Affiliation`, `Abstract`, `BibEntry`, `Blockquote`, or any form primitives (`Field`, `Selection`, `Checkbox`). The `Heading` class does not distinguish between document title and section headers. ## Reproducibility ### Data - **Dataset**: 981 annotated PDF pages across 9 types (academic papers, textbooks, books, slides, exam papers, financial reports, magazines, newspapers, handwritten notes). Available on [Hugging Face](https://huggingface.co/datasets/opendatalab/OmniDocBench) under a research-only (non-commercial) license. - **Annotations**: Over 20,000 block-level and 70,000 span-level annotations with bounding boxes, text content, LaTeX formulas, HTML/LaTeX tables, reading order, and attribute labels. - **Sourcing pipeline**: PDFs collected from Common Crawl, Google, Baidu, and internal data. Visual clustering with ResNet-50 + Faiss for diversity selection. ### Evaluation - **Code**: Full evaluation toolkit (extraction, matching, metric calculation) released on [GitHub](https://github.com/opendatalab/OmniDocBench) under Apache-2.0. - **Metrics**: Edit Distance (text, formulas, tables, reading order), CDM (Character Detection Matching for formulas), BLEU (formulas), TEDS (tables). - **Baselines**: Pipeline tools (MinerU, Marker, Mathpix), expert VLMs (GOT-OCR, Nougat), general VLMs (GPT-4o, Qwen2-VL-72B, InternVL2-76B). Specific model versions documented (e.g., MinerU v0.9.3, Marker v1.2.3). ### Models - No new model is proposed. The benchmark evaluates existing pipeline tools and VLMs. ### Algorithms - The Adjacency Search Match algorithm is implemented in the open-source evaluation toolkit. No model training is involved. ### Hardware - The paper does not report specific hardware configurations for the benchmark evaluations. Running the VLM baselines (e.g., Qwen2-VL-72B, InternVL2-76B) requires significant GPU resources or API access. The evaluation scripts themselves are lightweight text comparison tools. ## BibTeX ```bibtex @inproceedings{ouyang2024omnidocbench, title={OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations}, author={Ouyang, Linke and Qu, Yuan and Zhou, Hongbin and Zhu, Jiawei and Zhang, Rui and Lin, Qunshu and Wang, Bin and Zhao, Zhiyuan and Jiang, Man and Zhao, Xiaomeng and Shi, Jin and Wu, Fan and Chu, Pei and Liu, Minghao and Li, Zhenxiang and Xu, Chao and Zhang, Bo and Shi, Botian and Tu, Zhongying and He, Conghui}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pages={24838--24848}, year={2025} } ```