# READoc: A Unified Benchmark for Realistic Document Structured Extraction > READoc reframes document structured extraction as an end-to-end task converting raw PDFs to structured Markdown, exposing gaps in pipeline tools, expert models, and VLMs across 3,576 real-world documents. - Authors: Hunter Heidenreich - Published: 2026-04-02 - Paper: Preprint (arXiv) - Benchmark: READoc Dataset and Evaluation Suite - Source: Bevaya Labs, --- ## TL;DR READoc introduces a benchmark that defines Document Structured Extraction (DSE) as a single end-to-end task: given a raw PDF, produce structured Markdown. The benchmark comprises 3,576 documents drawn from arXiv, GitHub, and Zenodo, and comes with a three-module Evaluation S$^3$uite (Standardization, Segmentation, Scoring) that measures text, heading, formula, table, and reading-order quality in a unified way. Evaluation of 16 systems reveals large gaps between current tools and the holistic DSE objective. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$ The primary contribution is the READoc dataset and evaluation suite, both released as reusable community infrastructure. The benchmark itself, its construction pipeline, and the S$^3$uite are the headline deliverables. **Secondary:** $\Psi\_{\text{Evaluation}}$ The paper also contributes a measurement framework with novel metrics and protocols (KTDS for reading order, TEDS for tables and ToC trees, EDS for text), and the empirical evaluation exposes systematic failure modes that could not be identified with prior fragmented benchmarks. ## What is the motivation? Document Structured Extraction sits at the foundation of knowledge-base construction, corpus creation, and retrieval-augmented generation pipelines. Despite the large number of DSE systems that have emerged from both the academic and open-source communities, a rigorous unified evaluation framework has been absent. Prior benchmarks fragment DSE into independent subtasks: layout analysis, OCR, reading-order detection, table recognition, formula conversion, and table-of-contents extraction each have their own datasets with distinct formats, annotation conventions, and data sources. This fragmentation creates two practical problems. First, it is impossible to determine how well a system performs at the realistic end-to-end task that downstream users actually care about. Second, because each subtask dataset focuses on a narrow slice, it does not reveal interactions and failure modes that only appear when subtasks are chained together (for example, a reading-order error that corrupts an otherwise accurate formula extraction). The authors also note that many existing benchmarks use synthetic or curated data, rather than real-world documents in the wild, which limits their ecological validity. ## What is the novelty? The central contribution is the framing of DSE as a single PDF-to-Markdown conversion task and the construction of a benchmark that operationalizes this framing at scale. **Task definition.** READoc takes a complete multi-page PDF as input and expects structured Markdown as output. The Markdown variant supports LaTeX syntax for tables and formulas, following the convention introduced by Nougat. This formulation is well-defined, practical (Markdown is directly consumable by LLMs and RAG pipelines), and challenging. **Dataset construction.** The 3,576-document corpus is assembled from three heterogeneous sources, each stressing different DSE capabilities: - *READoc-arXiv* (1,009 documents): academic papers with complex multi-column layouts, formulas, and tables. Ground truth Markdown is derived from LaTeX sources via LaTeXML and a modified Nougat conversion pipeline. - *READoc-GitHub* (1,224 documents): README files converted to PDF via Pandoc and Eisvogel. These test heading detection and ToC construction under varied and often unmarked heading styles. - *READoc-Zenodo* (1,343 documents): DOCX and HTML documents from Zenodo's open repository, covering 27 languages, diverse types (theses, reports, posters, books), and long documents (average 14.93 pages). Ground truth is generated via Markitdown and Pandoc. **Evaluation S$^3$uite.** The suite operates in three sequential stages. The Standardization module normalizes formula boundaries, heading markers, table syntax, and image/link references to make model outputs comparable regardless of formatting idiosyncrasies. The Segmentation module partitions both prediction and reference into four semantic unit types: headings, embedded formulas, isolated formulas, tables, and plain text. The Scoring module then applies four metrics: - Edit Distance Similarity (EDS) for text and heading concatenation: $$EDS(A, B) = 1 - \frac{ED(A, B)}{\max(|A|, |B|)}$$ - Tree Edit Distance Similarity (TEDS) for ToC and table structure: $$TEDS(T\_1, T\_2) = 1 - \frac{TED(T\_1, T\_2)}{\max(|T\_1|, |T\_2|)}$$ - Kendall's Tau Distance Similarity (KTDS) for reading order, computed at both block and token granularity: $$KTDS(X, Y) = 1 - \frac{2 \cdot K\_d}{n(n-1)}$$ where $K\_d$ is the number of discordant pairs and $n$ is the total number of ranked items. - Vocabulary F1 for plain text. Reading-order quality is thus measured implicitly within the holistic task rather than as a separate subtask with a dedicated annotation layer. ## What experiments were performed? The paper evaluates 16 systems across four categories on READoc-arXiv and READoc-GitHub (and selectively on READoc-Zenodo): **Baselines:** PyMuPDF4LLM (PDF bytecode parsing), Tesseract (OCR-based), and MarkItDown (multi-format converter). These represent simple, widely applicable approaches without specialized document-understanding capabilities. **Pipeline tools:** Marker, MinerU, Pix2Text, and Docling. These integrate engineering pipelines with specialized deep learning submodels for distinct DSE subtasks. **Expert visual models:** Nougat-small and Nougat-base (end-to-end academic document parsing), and GOT-OCR 2.0. These are Transformer-based models trained directly for document-to-text conversion. **Vision-Language Models:** DeepSeek-VL-7B-Chat, MiniCPM-Llama3-V2.5, LLaVA-1.6-Vicuna-13B, InternVL-Chat-V1.5, and GPT-4o-mini. VLMs process page images one at a time, with concatenated output. The paper also performs several fine-grained analyses: - Impact of document length and depth, showing that pipeline tools and expert models degrade with increasing heading depth, while VLMs degrade with increasing document length. - Impact of layout complexity (single-column vs. multi-column), using pdfplumber classification. - Multi-page paradigm exploration: GPT-4o-mini is given all page images simultaneously for short documents to test global context effects. - Efficiency comparison measuring per-document processing time on a sample of 50 documents. ## What are the outcomes/conclusions? The evaluation on READoc-arXiv (Table 3 in the paper) surfaces several findings: 1. **Pipeline tools outperform baselines, but fall well short of expert models on specialized subtasks.** MinerU achieves the highest average score (73.07) among pipeline tools, but expert models Nougat-small and Nougat-base reach averages of 81.38 and 81.42 respectively, driven by strong formula and heading conversion. 2. **VLMs lag considerably on specialized content.** Even GPT-4o-mini, the best-performing VLM, achieves only 57.98 average on READoc-arXiv. Formula conversion scores are particularly low (31.77 for embedded, 18.65 for isolated), reflecting the difficulty of recognizing and rendering mathematical notation from page images. 3. **Reading order detection is relatively tractable.** The heuristic-based Tesseract baseline scores 96.70 block-level KTDS on READoc-arXiv, suggesting that most systems maintain correct sequential ordering even when specialized content conversion is poor. READoc's holistic framing still surfaces confused reading order in multi-column and complex layouts, as documented in the case studies. 4. **Single-page paradigm limits global structure recovery.** Processing pages independently prevents models from constructing accurate cross-page ToC trees. A multi-page experiment with GPT-4o-mini shows significant ToC improvement but degrades local fine-grained capabilities (tables, formulas), indicating an unresolved tension. 5. **Efficiency gaps are large.** Inference times range from 23.86 seconds per document (Marker, 1x Titan RTX) to over 1,182 seconds (InternVL-Chat-V1.5, 2x A100). VLMs are orders of magnitude slower than pipeline tools, which matters for large-scale corpus construction or real-time RAG applications. **Limitations acknowledged by the authors:** The automated PDF-Markdown construction pipeline introduces noise that is difficult to eliminate entirely, and the Standardization module cannot cover all format variations produced by diverse DSE systems. ## Reproducibility ### Models READoc does not introduce a new model. All evaluated systems are publicly available third-party tools referenced with their original publications. Architecture details and parameter counts are not reported by this paper; readers should consult each system's source. The evaluation pipeline itself is released alongside the benchmark dataset. ### Algorithms The S$^3$uite is implemented in Python. Segmentation is performed via regular expressions. EDS uses standard Levenshtein edit distance. TEDS uses a maximum bipartite matching algorithm for table alignment before computing tree edit distance. KTDS counts discordant pairs directly from ranked lists constructed from block-level and token-level co-occurrence positions. For VLMs, generation uses top-p = 0.8, top-k = 100, temperature = 0.7, do\_sample = True, and repetition\_penalty = 1.05. No training is performed. ### Data - **READoc-arXiv:** 1,009 documents from arXiv (1996-2024). LaTeX sources converted via LaTeXML then a modified Nougat HTML-to-Markdown script. English only. Covers 6 document types and 8 disciplines. - **READoc-GitHub:** 1,224 README files (2008-2024) with at least 500 stars, English, no HTML syntax, no tables or formulas. PDF generated via Pandoc + Eisvogel template. - **READoc-Zenodo:** 1,343 documents (910 DOCX + 433 HTML) from 2014-2024. DOCX converted via Microsoft Markitdown then python-docx; HTML via Pandoc then Chromium. 27 languages. - Ground truth Markdown is generated automatically; some noise remains despite filtering. No human annotation. - The dataset is released for research use; per-document source licenses inherit from arXiv (CC-BY or equivalent), GitHub (varied open source), and Zenodo (varied open access). ### Evaluation - **Metrics:** EDS (text, heading, formula concatenation), Vocabulary F1 (plain text), TEDS (ToC tree, table structure), KTDS at block level and token level (reading order). - **Benchmarks:** Three subsets of READoc: arXiv, GitHub, Zenodo. Main results reported on arXiv and GitHub. Zenodo used in supplementary analyses. - **Baselines:** 16 systems covering four paradigm classes; no custom fine-tuning is performed. - **Statistical rigor:** The paper does not report standard deviations or confidence intervals. Efficiency measurements are averaged over 50 sampled documents from READoc-arXiv. - **Scope limitations:** READoc-arXiv ground truth derives from LaTeX source, which may not capture all rendering nuances of the compiled PDF. READoc-GitHub and READoc-Zenodo are constructed from original source files, but automated conversion chains may introduce artifacts. ### Hardware - Nougat-small and Nougat-base: evaluated on 1x Titan RTX (24 GB). - MiniCPM-Llama3-V2.5: evaluated on 1x A100 (80 GB). - InternVL-Chat-V1.5: evaluated on 2x A100 (80 GB). - Marker, MinerU, Pix2Text: GPU type and VRAM not specified in detail. - No training is performed; all reported times are inference throughput. ## BibTeX ```bibtex @inproceedings{li2025readoc, title = {{READOC}: A Unified Benchmark for Realistic Document Structured Extraction}, author = {Zichao Li and Aizier Abulaiti and Yaojie Lu and Xuanang Chen and Jia Zheng and Hongyu Lin and Xianpei Han and Shanshan Jiang and Bin Dong and Le Sun}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2025}, year = {2025} } ```