# OmniDocBench: A Diverse, Multi-Level Benchmark for Document Parsing Evaluation > OmniDocBench introduces a 981-page benchmark spanning nine document types with comprehensive annotations for layout, OCR, table recognition, formula parsing, and reading order evaluation, enabling fair multi-level comparison of pipeline and VLM-based document parsers. - Authors: Hunter Heidenreich - Published: 2024-12-10 - Updated: 2026-04-02 - Benchmark: OmniDocBench - Source: Bevaya Labs, --- ## TL;DR OmniDocBench is an evaluation benchmark covering 981 annotated PDF pages from nine distinct document types, including handwritten notes and newspapers. It supports end-to-end, task-specific, and attribute-level evaluation of document parsing systems across OCR, layout detection, table recognition, formula parsing, and reading order. Evaluation of pipeline tools and vision-language models on the benchmark surfaces consistent strengths and weaknesses that narrower prior benchmarks could not reveal. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: the primary contribution is the benchmark dataset itself: 981 carefully curated and annotated pages, along with the evaluation framework and leaderboard infrastructure. The paper's center of gravity is the curation pipeline, annotation schema, and diversity of the resource. **Secondary:** $\Psi\_{\text{Evaluation}}$: the evaluation methodology section introduces non-trivial measurement contributions, including an Adjacency Search Match algorithm designed to handle paragraph-splitting inconsistencies across models, format-specific metrics (CDM for formulas, TEDS for tables), and a principled reading order evaluation protocol. These are genuine measurement contributions beyond standard edit distance. ## What is the motivation? Document parsing (converting unstructured PDFs into machine-readable formats) is a foundational step for large language model pre-training and retrieval-augmented generation (RAG) pipelines. Despite substantial progress in both pipeline-based systems (layout detection + OCR + formula/table recognition modules) and end-to-end vision-language models (VLMs), the field lacked a benchmark capable of fairly comparing these two paradigms. Existing benchmarks had three key gaps. First, document diversity was narrow: most datasets focused exclusively on academic papers from arXiv, ignoring textbooks, financial reports, exam papers, handwritten notes, newspapers, and magazines that appear routinely in real deployments. Second, evaluation metrics were inconsistent: generic string similarity metrics such as edit distance and BLEU treat LaTeX formula representations as arbitrary strings, penalizing syntactically different but semantically equivalent expressions. Third, evaluations were not fine-grained: benchmarks typically reported a single aggregate score, making it impossible to identify whether failures were concentrated in a particular document type, content category, or visual attribute. These gaps made it difficult to diagnose model weaknesses or choose the right system for a given document domain. ## What is the novelty? OmniDocBench contributes three interconnected components. **Benchmark dataset.** The dataset contains 981 pages sampled from over 200,000 initial PDFs sourced from Common Crawl and search engines. Pages were clustered using ResNet-50 visual features and Faiss to maximize visual diversity, then manually labeled for page-level attributes. The final dataset covers nine document types: academic papers, books, colorful textbooks, exam papers, magazines, notes, newspapers, slides, and financial reports. Each page carries six page-level attributes (type, layout, language, fuzzy scan, watermark, colorful background), and each annotation block carries up to nine attribute labels covering text language, background color, text rotation, table frame type, merged cells, formula presence, and table rotation. Annotations span two granularity levels. Block-level annotations cover 19 region categories (titles, text paragraphs, figures, tables, formulas, headers, footers, code blocks, references, and more), with over 20,000 bounding boxes. Reading order labels are provided for all blocks except headers, footers, and page notes, totaling over 16,000 ordered annotations. Span-level annotations cover inline formulas (4,009, in LaTeX), equation-ignore spans, text spans, and footnote markers, totaling over 70,000 entries. **Evaluation pipeline.** The evaluation framework addresses three practical challenges that undermine naive metric computation. First, different models apply different standards for headers, footers, and caption placement; OmniDocBench handles these through ignore rules so that incidental differences do not dominate scores. Second, paragraph boundaries vary across models; the Adjacency Search Match algorithm resolves this by constructing a normalized edit distance matrix between predicted and ground-truth paragraphs and iteratively merging adjacent paragraphs until the best-possible alignment is found. Third, formula and table outputs are format-diverse; the benchmark applies the CDM metric (Character-level Distinction Metric) to formulas and the Tree Edit Distance-based Similarity (TEDS) metric to tables, rather than treating these as plain strings. For reading order, after matching predicted text blocks to ground-truth blocks, the benchmark derives an ordering over matched pairs and computes normalized edit distance between the predicted sequence and the ground-truth sequence. Tables, images, and deliberately ignored blocks are excluded from this calculation. Formally, the reading order score uses normalized edit distance: $$\text{ROD}\_{\text{Edit}} = \frac{\text{EditDistance}(\hat{s},\ s)}{\max(|\hat{s}|,\ |s|)}$$ where $\hat{s}$ is the predicted block sequence and $s$ is the ground-truth sequence, after adjacency-match alignment. Lower values indicate better ordering. The overall end-to-end normalized edit distance aggregates across text, formula, and table elements: $$\text{Overall}^{\text{Edit}} = \frac{\sum\_{i} \text{EditDistance}(\hat{t}\_{i},\ t\_{i})}{\sum\_{i} \max(|\hat{t}\_{i}|,\ |t\_{i}|)}$$ **Comprehensive model comparison.** The benchmark is applied to three categories of document parsing systems: pipeline tools (MinerU v0.9.3, Marker v1.2.3, Mathpix), expert VLMs trained specifically for document parsing (GOT-OCR 2.0, Nougat 0.1.0-base), and general-purpose VLMs (GPT-4o, Qwen2-VL-72B, InternVL2-Llama3-76B). ## What experiments were performed? Evaluation spans four levels. **End-to-end evaluation** measures full-page parsing accuracy across seven dimensions: text (normalized edit distance $\downarrow$), formula edit distance $\downarrow$, formula CDM $\uparrow$, table TEDS $\uparrow$, table edit distance $\downarrow$, reading order edit distance $\downarrow$, and overall edit distance $\downarrow$. Results are broken down by language (English vs. Chinese), document type (nine categories), layout type, and visual degradation (fuzzy scan, watermark, colorful background). **Layout detection evaluation** uses mean Average Precision (mAP) on a layout-annotated subset, comparing DIT-L, LayoutLMv3, DocLayout-YOLO, SwinDocSegmenter, GraphKD, and DOCX-Chain across the nine page types. **Table recognition evaluation** uses TEDS on a table-annotated subset, stratified by language, frame type (full frame, omission line, three-line, no frame), and special conditions (merged cells, colorful background, formula content, rotation). **OCR evaluation** uses normalized edit distance on a text-annotated subset, stratified by language, background color, and text rotation (normal, 90°, 270°, horizontal). Models evaluated include PaddleOCR, Tesseract, Surya, GOT-OCR, Mathpix, Qwen2-VL-72B, InternVL2-76B, and GPT-4o. **Formula recognition evaluation** uses CDM, ExpRate@CDM, BLEU, and normalized edit distance on a formula subset. Models include GOT-OCR, Mathpix, Pix2Tex, UniMERNet-B, GPT-4o, InternVL2-76B, and Qwen2-VL-72B. For general VLMs, `do_sample=False` was set to ensure reproducibility. Qwen2-VL-72B used `max_token=32000`; InternVL2-Llama3-76B used `max_token=4096`. Pipeline tools and expert VLMs used default settings. ## What are the outcomes/conclusions? **Pipeline tools vs. VLMs.** Pipeline tools, especially MinerU and Mathpix, achieve lower end-to-end edit distance on common document types such as academic papers and financial reports. General VLMs generalize better to unconventional formats (e.g., notes, exam papers, slides) and show greater robustness under visual degradations such as fuzzy scans and colored backgrounds, which the authors attribute to their broader training data. **Layout detection.** DocLayout-YOLO significantly outperforms other layout detectors, particularly on diverse formats. Other methods struggle with slides and notes due to limited training data diversity. **Reading order.** All models degrade substantially on multi-column and complex layouts. MinerU maintains the most consistent reading order overall, but its performance drops on handwritten single-column pages. VLMs tend to merge multi-column text, compounding both recognition and ordering errors. **Newspapers.** Most VLMs fail on newspaper pages due to high text density and resolution constraints on input tokens. Pipeline tools, which apply layout segmentation before recognition, handle this case substantially better. **Rotated text and tables.** Rotation is a consistent failure mode across all model types; table rotation in particular renders almost all models ineffective. **Formula recognition.** GPT-4o, Mathpix, and UniMERNet-B achieve CDM scores of 86.8, 86.6, and 85.0 respectively. Mathpix has high character-level precision but occasionally omits punctuation, reducing its exact-match rate. **Limitations.** The benchmark is annotated for English and Chinese only, leaving multilingual generalization untested. The 981-page scale, while carefully curated, remains relatively small compared to training sets. The evaluation framework relies on matched block sequences, so models that produce highly fragmented or reorganized output may be penalized more than their semantic accuracy warrants. ## Reproducibility ### Models OmniDocBench does not introduce new model weights. It evaluates existing systems: MinerU v0.9.3, Marker v1.2.3, Mathpix (commercial API), Nougat 0.1.0-base (350M parameters), GOT-OCR 2.0, GPT-4o (2024-08-06 version), Qwen2-VL-72B, and InternVL2-Llama3-76B (76B parameters). Layout detection models evaluated include DIT-L (361.6M), LayoutLMv3 (138.4M), DocLayout-YOLO (19.6M), SwinDocSegmenter (Swin-L, 223M), and GraphKD (ResNet-101, 44.5M). ### Algorithms No novel training algorithm is introduced. The evaluation pipeline includes: regex-based element extraction (LaTeX tables, HTML tables, display formulas, markdown tables, code blocks), Unicode normalization for inline formulas, Adjacency Search Match for paragraph alignment (normalized edit distance matrix with a similarity threshold, followed by iterative fuzzy merging), and metric calculation via normalized edit distance, CDM, and TEDS. The annotation pipeline used LayoutLMv3 for pre-annotation of layout, PaddleOCR for text, UniMERNet for formulas, and GPT-4o for tables, followed by human correction and expert quality inspection using CDM's rendering technique to identify unrenderable formula annotations. ### Data The benchmark was constructed from over 200,000 PDFs sourced from Common Crawl and Google/Baidu search engines. ResNet-50 visual features were extracted and clustered with Faiss (10 cluster centers, 6,000 candidate pages sampled), then manually filtered to 981 balanced pages. The dataset contains English (290 pages), Simplified Chinese (612 pages), and mixed (79 pages) pages. The dataset and evaluation code are available at https://github.com/opendatalab/OmniDocBench under an Apache-2.0 license. ### Evaluation Metrics used: normalized edit distance (text, formula, reading order, overall), CDM (formula recognition, higher is better), TEDS (table recognition, higher is better), and mAP (layout detection). The reading order metric applies normalized edit distance over the matched ordered block sequence, excluding tables, images, and ignored elements. No statistical significance tests or confidence intervals are reported; results are single-run evaluations. Baseline comparisons are generally fair within each modality, though the pipeline tool versions are pinned while VLMs are evaluated at their publicly available weights. ### Hardware The paper does not report GPU specifications, training compute, or inference latency for the evaluated systems. No cost estimates are provided. The evaluation framework itself is lightweight (regex extraction, edit distance computation), but inference costs for large VLMs (e.g., Qwen2-VL-72B, InternVL2-76B) would require substantial GPU memory. ## BibTeX ```bibtex @inproceedings{ouyang2025omnidocbench, title={OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations}, author={Ouyang, Linke and Qu, Yuan and Zhou, Hongbin and Zhu, Jiawei and Zhang, Rui and Lin, Qunshu and Wang, Bin and Zhao, Zhiyuan and Jiang, Man and Zhao, Xiaomeng and Shi, Jin and Wu, Fan and Chu, Pei and Liu, Minghao and Li, Zhenxiang and Xu, Chao and Zhang, Bo and Shi, Botian and Tu, Zhongying and He, Conghui}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, year={2025} } ```