# OmniDocBench: A Diverse Benchmark for End-to-End Document Parsing > OmniDocBench is a document parsing benchmark covering nine document types with comprehensive layout, text, formula, and table annotations, supporting end-to-end and task-specific evaluation of both pipeline-based tools and vision-language models. - Authors: Hunter Heidenreich - Published: 2024-12-10 - Updated: 2026-03-30 - Paper: Preprint (arXiv) - Code: OmniDocBench (code and evaluation framework) - Dataset: OmniDocBench (dataset on Hugging Face) - Source: Bevaya Labs, --- ## TL;DR OmniDocBench is a document parsing benchmark from Shanghai AI Laboratory that covers 981 pages across nine document types (academic papers, books, textbooks, magazines, slides, financial reports, newspapers, handwritten notes, and exam papers), with human-verified annotations for layout detection, text, formulas, and tables, including dual HTML+LaTeX table annotations for TEDS-style evaluation. The authors evaluate three classes of systems: pipeline tools, expert VLMs, and general VLMs. Pipeline-based tools generally lead on standard document types while general VLMs generalize better to uncommon formats and degrade less under visual noise. No single method dominates across all nine document types or both language settings (English and Chinese). ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Evaluation}}$ The primary contribution is a new benchmark and evaluation methodology for document parsing. The dataset design, annotation protocol, multi-level evaluation pipeline, and systematic comparison of methods are all in service of a measurement goal: determining how well current document parsing systems handle diverse real-world document types. **Secondary:** $\Psi\_{\text{Resource}}$ The benchmark itself, including 981 annotated pages, over 100,000 annotations, dual HTML+LaTeX table annotations, and a publicly released evaluation codebase, is a reusable community resource. ## What is the motivation? Document parsing, the task of extracting structured, machine-readable content from PDFs, underpins data pipelines for LLM pretraining and retrieval-augmented generation systems. Two main paradigms have emerged: pipeline-based approaches (layout detection followed by specialized OCR, formula, and table recognition modules) and end-to-end vision-language models. Despite progress on both fronts, fair and comprehensive comparison between systems has been difficult because existing benchmarks carry several well-documented shortcomings. Document diversity is narrow: most benchmarks focus on academic papers from arXiv, overlooking document types common in real workflows such as textbooks, financial reports, handwritten notes, newspapers, and slides. Evaluation metrics are inconsistent or superficial: benchmarks relying solely on edit distance or BLEU do not account for the syntactic variety of valid LaTeX or HTML table representations, leading to systematic under-scoring of structurally correct but syntactically non-identical predictions. Evaluations are also typically flat, reporting one aggregate score without breakdown by document type, content category, language, or layout complexity. The combination of narrow document coverage and weak metrics makes it difficult to identify where particular systems actually fail, or to determine whether a reported improvement generalizes beyond the training distribution. ## What is the novelty? OmniDocBench addresses these gaps with three main contributions. **Diverse evaluation corpus.** The dataset spans nine document types: academic papers (129 pages), slides/PPT-to-PDF (133 pages), financial reports (81 pages), colorful textbooks (96 pages), exam papers (114 pages), magazines (97 pages), handwritten and typed notes (116 pages), newspapers (111 pages), and books (104 pages). The corpus covers three language settings: English (290 pages), Simplified Chinese (612 pages), and mixed (79 pages), with multiple layout types and three classes of visual degradations (fuzzy scans, watermarks, and colorful backgrounds). **Comprehensive annotations.** Each page carries layout bounding boxes for 19 region categories with reading order and caption-footnote affiliation annotations, nine bounding-box-level attribute labels (three text attributes: language, background, rotation; six table attributes: language, frame type, merged cells, colorful background, formula content, rotation), and content recognition annotations in plain text for paragraphs and titles, LaTeX for formulas, and dual HTML plus LaTeX for tables. The dual HTML+LaTeX annotation for tables is a direct response to the known problem that TEDS-style metrics penalize syntactically different but semantically equivalent table representations. **Multi-level evaluation pipeline.** OmniDocBench supports three evaluation granularities: 1. End-to-end evaluation: full-page Markdown prediction versus per-element ground truth 2. Task-specific evaluation: isolated assessment of layout detection (mAP), OCR (normalized edit distance), table recognition (TEDS), and formula recognition (CDM + ExpRate) 3. Attribute-based evaluation: performance broken down by document type, language, background, rotation, table frame type, and special conditions A key technical contribution is the Adjacency Search Match algorithm, which addresses the paragraph-splitting problem that arises when different parsers apply different line-break and merge conventions. The algorithm computes a matrix of normalized edit distances between ground truth and predicted paragraph sequences, applies fuzzy matching for unmatched pairs, then iteratively merges adjacent paragraphs in either direction until similarity no longer improves. The CDM (Character-level Differential Metric) is applied for formula evaluation. For tables, TEDS is computed against both HTML and LaTeX annotations and the better of the two scores is retained, reducing the syntactic-equivalence penalty. ## What experiments were performed? **Systems evaluated.** The authors evaluate three categories of methods: - Pipeline tools: MinerU v0.9.3, Marker v1.2.3, Mathpix (commercial) - Expert VLMs: GOT-OCR 2.0, Nougat 0.1.0-base (350M parameters) - General VLMs: GPT-4o (2024-08-06), Qwen2-VL-72B, InternVL2-Llama3-76B For general VLMs, deterministic decoding is applied (`do_sample=False`) with per-model `max_token` settings (32,000 for Qwen2-VL-72B; 4,096 for InternVL2; default for GPT-4o). **End-to-end evaluation.** Each model produces a full-page Markdown prediction. The pipeline extracts structured elements, matches them to ground-truth elements via the adjacency search algorithm, and computes per-category scores for text, formulas, tables, and reading order, reported separately for English and Chinese pages. Selected end-to-end table TEDS results: | Method | Table TEDS EN | Table TEDS ZH | | :--- | :---: | :---: | | MinerU | 78.6 | 62.1 | | Mathpix | ~75 | ~63 | | Qwen2-VL-72B | 73.2 | 75.1 | | GPT-4o | 71.1 | 58.0 | | InternVL2-76B | 60.9 | 58.5 | | GOT-OCR | 51.7 | 46.2 | | Nougat | 36.2 | 0.3 | **Layout detection.** DocLayout-YOLO achieves a mean average precision of 47.38 averaged across nine document types, substantially ahead of LayoutLMv3 (28.84) and DIT-L (26.90). Performance drops markedly on handwritten notes and slides across all models evaluated. **Table recognition (component-level).** Six models are evaluated on an isolated table subset using TEDS broken down by language, frame type, and special conditions. OCR-based models (RapidTable: 82.5 overall, PaddleOCR: 73.6) outperform expert VLMs (StructEqTable: 75.8, GOT-OCR: 74.9) and general VLMs (Qwen2-VL-7B: 71.0, InternVL2-8B: 71.5). Table rotation degrades all models substantially, with StructEqTable showing the best relative robustness on rotated tables. **OCR (component-level).** PaddleOCR achieves the lowest normalized edit distance on the OCR subset across standard and complex-background conditions. GPT-4o achieves the lowest English edit distance (0.020) but is substantially weaker on Chinese (0.224). Rotation to 270 degrees is challenging for most systems. **Formula recognition.** GPT-4o leads on CDM (86.8) and ExpRate@CDM (65.5); UniMERNet achieves the best normalized edit distance (0.238). Mathpix reports high CDM (86.6) but low ExpRate (2.8), consistent with high character-level precision combined with occasional punctuation omissions. ## What are the outcomes/conclusions? Pipeline tools (MinerU, Mathpix) generally lead on structured content tasks for standard document types such as academic papers and financial reports. Their strength comes from specialized downstream models and layout segmentation preprocessing. General VLMs (Qwen2-VL-72B, GPT-4o) generalize better to uncommon or visually unusual document types such as handwritten notes, slides, and exam papers, and degrade less under visual degradations. The authors attribute this to broader training data covering long-tail scenarios. VLMs struggle with high-density documents like newspapers, where limitations in input resolution and context length cause content truncation. The failure modes differ qualitatively from pipeline failures: VLMs tend to miss content on dense pages or produce hallucinated content on hard-to-read ones, while pipeline tools tend to misclassify layout regions on uncommon formats. All systems perform worse on Chinese than English pages. All systems degrade substantially on rotated text and rotated tables. No single method achieves strong performance across all nine document types, both language settings, and all content categories simultaneously. This finding suggests that OmniDocBench captures a meaningfully broader evaluation space than prior benchmarks focused narrowly on academic papers. Limitations acknowledged by the authors include that the dataset is evaluation-only and not suitable for training, the Chinese-language subset represents a specific subset of Chinese document styles, and some annotation categories such as code blocks are represented by too few examples for reliable evaluation. ## Reproducibility ### Models The evaluated systems range widely in openness. MinerU and Marker are open-source pipeline tools. Nougat (350M) and GOT-OCR 2.0 are open expert VLMs with released weights. Qwen2-VL-72B and InternVL2-Llama3-76B are open general VLMs with released weights. Mathpix and GPT-4o are commercial API services with no public weights. ### Algorithms For pipeline tools, default settings are used throughout. For general VLMs, `do_sample=False` is set for reproducibility. Nougat uses its 0.1.0-base checkpoint; GOT-OCR uses its "format OCR" mode to produce structured output. The Adjacency Search Match algorithm applies normalized edit distance with a similarity threshold, followed by iterative paragraph merging until similarity ceases to improve. ### Data The 981 PDF pages were collected from Common Crawl, Google and Baidu search, and internal data at Shanghai AI Laboratory. Pages were selected by clustering visual features from 6,000 candidate pages using ResNet-50 features and Faiss at 10 cluster centers, with manual balancing across nine document types and page attributes. Annotation used a three-stage pipeline: automatic pre-annotation with LayoutLMv3 (layout), PaddleOCR (text), UniMERNet (formulas), and GPT-4o (tables); manual annotator correction with rendering verification using Tables Generator and latexlive; and expert quality inspection using CDM rendering to flag unrenderable elements. The evaluation code is available on GitHub under Apache-2.0. The dataset on Hugging Face carries no formal SPDX license; a copyright statement on the dataset page explicitly restricts use to research purposes and prohibits commercial use. The dataset files include an annotation JSON (`OmniDocBench.json`), a corresponding images directory, and a PDF version of the evaluation pages added in December 2024. ### Evaluation Primary metrics: normalized edit distance for text and reading order (lower is better); CDM and ExpRate@CDM for formulas (higher is better); TEDS for tables (higher is better, computed against both HTML and LaTeX annotations with the better score taken); mAP for layout detection. Inline formulas are converted to Unicode for cross-model fairness before text evaluation. Ignore handling is applied to reduce noise from inter-system differences in header and footer retention conventions and in caption placement conventions. Statistical details (number of runs, seeds, confidence intervals) are not reported; single-run results are presented for all methods. ### Hardware Hardware requirements are not reported. The evaluation mixes API calls to commercial services (Mathpix, GPT-4o) with local inference for open-weight models. No GPU hours, memory requirements, or cost estimates are provided. ## BibTeX ```bibtex @inproceedings{ouyang2025omnidocbench, title={OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations}, author={Ouyang, Linke and Qu, Yuan and Zhou, Hongbin and Zhu, Jiawei and Zhang, Rui and Lin, Qunshu and Wang, Bin and Zhao, Zhiyuan and Jiang, Man and Zhao, Xiaomeng and Shi, Jin and Wu, Fan and Chu, Pei and Liu, Minghao and Li, Zhenxiang and Xu, Chao and Zhang, Bo and Shi, Botian and Tu, Zhongying and He, Conghui}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, year={2025} } ```