# Benchmarking Table Extraction from Heterogeneous Scientific Documents > An end-to-end table extraction benchmark covering three datasets, including two new heterogeneous collections (Table-arXiv and Table-BRGM), plus formally justified metrics that expose how models fail when table styles diverge from their training distribution. - Authors: Hunter Heidenreich - Published: 2025-11-24 - Updated: 2026-03-30 - Benchmark: Table Extraction Benchmark (Code and Datasets) - Paper: Preprint (arXiv) - Source: Bevaya Labs, --- ## TL;DR This paper proposes an end-to-end benchmark for table extraction (TE) from scientific PDF documents. It introduces two new heterogeneous datasets (Table-arXiv: 36k samples from arXiv LaTeX sources; Table-BRGM: 124 tables from French geological reports), alongside a formally justified set of metrics that jointly evaluate table detection and structure recognition. Experiments across nine methods show that models trained on homogeneous collections like PubTables-1M suffer substantial performance drops on heterogeneous data, and that table extraction remains an unsolved problem. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Evaluation}}$. The central contribution is a new benchmark framework: two new datasets, a formal metric design covering TD, TSR, and end-to-end TE, and a rigorous evaluation protocol that captures model uncertainty and cascading errors. The paper does not introduce a new model or training algorithm. **Secondary:** $\Psi\_{\text{Resource}}$. Table-arXiv and Table-BRGM are released as community resources. Their heterogeneity (diverse LaTeX typesetting, Word-processed geological reports, multilingual content) makes them useful for training and evaluation beyond the paper's own experiments. ## What is the motivation? Table extraction from PDFs is a prerequisite for many downstream tasks: question answering over tables, data lake integration, and automated document analysis pipelines. Despite a large number of competing tools, choosing among them is difficult because existing evaluations have two systematic limitations. First, prior benchmarks assess table detection (TD) and table structure recognition (TSR) as independent problems, each given a clean pre-cropped input. In practice, a TE pipeline chains these steps: the TSR model receives whatever the TD model detected. Errors in detection degrade structure recognition, but this cascade is invisible when the two tasks are scored separately. Second, the dominant large-scale dataset, PubTables-1M, draws exclusively from PubMed Central biomedical articles produced with professional publishing systems. Models trained on this distribution learn typesetting conventions that do not transfer to LaTeX-compiled preprints from other domains, Word-processed reports, or documents in languages other than English. Benchmarks constructed from only this source inflate reported scores and mask generalization failures. The authors address both limitations: they design an end-to-end evaluation protocol and construct heterogeneous test data that stress-tests out-of-distribution behavior. ## What is the novelty? ### Formal end-to-end metrics The paper introduces new metrics that propagate table detection quality into the TSR score. For a model whose predictions include bounding boxes and optionally a confidence score $\tau$, the TE precision and recall are defined as: $$ P\_{\text{TSR}} = \frac{1}{|\mathcal{P}|} \sum\_{\delta \in \mathcal{P}} \mathbb{1}\_{\delta > \lambda} \, B^{\text{TSR}}(\delta) \qquad R\_{\text{TSR}} = \frac{1}{|\mathcal{G}|} \sum\_{\delta \in \mathcal{G}} \mathbb{1}\_{\delta > \lambda} \, B^{\text{TSR}}(\delta) $$ where $\delta$ is the Jaccard index between a detected table and its ground-truth match, $\lambda$ is the matching threshold, and $B^{\text{TSR}}(\delta)$ is the TSR score (GriTS or TEDS) for that detected table. Missed detections receive $B^{\text{TSR}} = 0$, so a TD failure directly penalizes the end-to-end score. For probabilistic models that output a confidence score $\tau\_p$ per prediction, the paper additionally derives expected precision and recall by treating the Jaccard threshold as a random variable with a probability density function. Under the uniform density $S\_0$, the expected value of the indicator $\mathbb{1}\_{\lambda \geq \tau}$ for a prediction with Jaccard index $\delta$ is $2\delta$, providing a smooth, principled summary that avoids dependence on a single threshold choice. Calibration of probabilistic models is measured with the object detection version of Expected Calibration Error (D-ECE): $$ \text{D-ECE} = \sum\_{i=1}^{M} \frac{|\mathcal{P}\_i|}{|\mathcal{P}|} \left| \text{prec}\_i - \text{conf}\_i \right| $$ where predictions are binned by confidence and the gap between average precision and average confidence within each bin is accumulated. ### Two new heterogeneous datasets **Table-arXiv** is built from 2,443 LaTeX source files downloaded from arXiv, spanning the last 20 years and all arXiv subject areas. TD annotations are generated by instrumenting the LaTeX `tabular` environment to inject anchors that survive PDF compilation. TSR annotations (with cell content) are produced by converting the LaTeX source to HTML using LaTeXML. The result is 36,869 pages with 6,308 annotated tables, all from documents compiled with LaTeX and covering mathematics, physics, astrophysics, and other domains where PubTables-1M has no representation. **Table-BRGM** is a small, manually annotated domain-specific set from geological survey reports published by BRGM, France's national geological survey institution. It contains 6 PDFs, 499 pages, and 124 tables in English and French. Most documents were produced with Word processing software rather than LaTeX. Tables include large multi-column structures, borderless layouts, empty cells, and merged cells, representing challenges absent from PubTables-1M. Both datasets include negative pages (pages with no tables). Prior benchmarks consisting only of positive instances inflate precision scores because models are never penalized for predicting tables on table-free pages. The new datasets expose this gap by including pages where false positives can occur. ### Content-based table detection evaluation For models like the LVLM (GPT-4o mini) that produce HTML tables without bounding boxes, the paper introduces a content-Jaccard metric based on character-level 2-grams. Given multisets $\mu(P)$ and $\mu(G)$ of character bigrams from predicted and ground-truth table cells: $$ \text{Jaccard}\_c(P, G) = \frac{|\mu(P) \cap \mu(G)|}{|\mu(P) \cup \mu(G)|} $$ This allows all methods to be evaluated for TD even when spatial coordinates are unavailable, using a content-based notion of whether a table was found. ## What experiments were performed? The benchmark evaluates nine end-to-end TE methods across three datasets. **Baseline methods (no confidence scores):** PDFPlumber, PyMuPDF, Camelot (rule-based Python libraries); Grobid (ML-based CRF extraction pipeline); and GPT-4o mini used as a zero-shot LVLM given a page image and a prompt to return HTML tables. **Probabilistic methods (output confidence scores):** Docling (RT-DETR for TD + TableFormer for TSR + EasyOCR for content); TATR-extract (Table Transformer detector + Table Transformer structure model, with a modified inference loop that retains low-confidence table predictions before NMS); XY+TATR-extract (XY-cut page segmentation prepended to TATR-extract to handle pages with multiple tables); and VGT+TATR-structure (Vision Grid Transformer for TD, which also uses text token grid information, combined with the same TATR-structure model). Metrics reported include AP and F1 for TD, GriTS-Topology, GriTS-Content, and TEDS for TSR and end-to-end TE, and D-ECE for calibration of probabilistic models. Inference speed is measured as CPU seconds per image on 4 CPUs with 2 concurrent tasks. ## What are the outcomes/conclusions? **No single method dominates across datasets.** Docling achieves the best overall TD F1 scores across all three datasets (F1 of 0.99 on PubTables, 0.89 on Table-arXiv, 0.90 on Table-BRGM), benefiting from its RT-DETR model trained on the diverse DocLayNet dataset. Among probabilistic models, TATR achieves near-perfect AP on PubTables (1.00) but drops to 0.84 AP on Table-BRGM, and VGT produces better-calibrated confidence scores on heterogeneous data while achieving lower raw AP than TATR on PubTables. **Rule-based tools collapse on heterogeneous data.** Camelot achieves F1 of 0.88 on Table-BRGM (where tables often have explicit grid lines) but only 0.25 on PubTables and 0.33 on Table-arXiv. PyMuPDF follows a similar pattern. These tools depend on visual lines and alignment cues that are absent from the LaTeX-typeset documents in PubTables and Table-arXiv. **PubTables-1M-specialized models do not generalize.** TATR is trained on PubTables-1M and achieves near-perfect scores on the PubTables test set. On Table-arXiv and Table-BRGM, its scores drop substantially. Its confidence scores also become poorly calibrated: on Table-arXiv, predictions with 80-90% confidence scores correspond to fewer than 2% true positives on average, making the scores essentially uninformative. **End-to-end metrics are more pessimistic than subtask metrics.** Once TD errors are propagated into TSR evaluation, all scores decrease. GriTS-Topology scores remain relatively high (tables that are detected tend to have their row/column structure recognized correctly), but GriTS-Content and TEDS drop more, because token extraction (via PDFAlto) misses stylized text like mathematical formulas, limiting content accuracy for TATR-structure-based models. **The LVLM hallucinates cell content.** GPT-4o mini in zero-shot mode replaces capital-I characters with the digit 1, reformats numbers, fills in empty cells with fabricated values, and changes numeric values. Its content TD is accordingly low on Table-arXiv. It also achieves only modest F1 scores on TD across all datasets, consistent with prior reports that GPT-4o mini struggles with precise spatial localization. **Inference speed differs by an order of magnitude.** Rule-based tools and Grobid require roughly 0.4-1.2 seconds per image. Docling, TATR, and XY require 9-13 seconds. VGT is substantially slower at 128 seconds per image due to its grid transformer pipeline. The authors conclude that table extraction is not a solved problem. Heterogeneous data exposes model fragility that homogeneous benchmarks conceal, and the cascade from TD to TSR means that strong subtask numbers do not guarantee strong end-to-end quality. ## Reproducibility ### Models No new model weights are introduced. The benchmark uses existing off-the-shelf models: - **TATR-detect and TATR-structure:** Table Transformer models from Microsoft, trained on PubTables-1M. Available via the [table-transformer](https://github.com/microsoft/table-transformer) repository and HuggingFace. - **VGT:** Vision Grid Transformer, fine-tuned on PubLayNet with DiT-base ViT weights. Available via the [VGT repository](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/VGT). Uses BERT tokenizer with LayoutLM word embeddings. - **Docling:** Available as a Python library. Uses RT-DETR (trained on DocLayNet) for TD and TableFormer for TSR. - **Grobid:** Available as an open-source library. - **GPT-4o mini:** Accessed via OpenAI API in zero-shot mode with the prompt shown in Appendix A.2 of the paper. ### Algorithms The benchmark evaluation code is available at the GitLab repository listed in the artifacts. The authors state that their pipelines were built starting from the [microsoft/table-transformer](https://github.com/microsoft/table-transformer) repository. AP is computed with Scikit-learn's `average_precision_score` (area under the full precision-recall curve, following the sklearn convention rather than the Pascal VOC interpolation). No training is performed. Key pipeline parameters: - TATR-extract retains all predictions with confidence above 0.05 for "table" or "table rotated" (modifying the original inference which kept only the top-1 label), then applies NMS. - XY+TATR-extract runs XY-cut recursive page segmentation (counting black pixels along both axes) before TATR-detect; each chunk retains only the top 2 predictions. - Table padding for TATR-structure crops: $\delta\_{\text{pxs}} = 100$ pixels added around detected tables; $\delta\_{\text{pxd}} = 10$ pixels padding for XY-cut sub-images. - Content-Jaccard matching threshold for LVLM TD evaluation: $\iota = 0.5$. - Camelot uses the "lattice" mode (demarcated cell lines), which outperformed other modes in this evaluation. ### Data **Table-arXiv:** 2,443 LaTeX source files from arXiv, sampled from all subject domains over 20 years. TD annotations are generated automatically by instrumenting LaTeX compilation; TSR annotations are produced by LaTeXML. Contains 36,869 pages, 5,214 positive pages, 6,308 annotated tables. All documents are LaTeX-compiled. **Table-BRGM:** 6 PDF documents from BRGM geological reports (French and English). 499 pages, 91 positive pages, 124 tables. Manually annotated; an end-to-end model was used as a starting point and outputs were manually corrected. Mix of Word-processed (67%) and LaTeX (33%) documents. **PubTables-Test:** Subset of the PubTables-1M test set for which PDF files could be retrieved from PubMed Central (Open Access subset). 23,175 PDFs, 46,942 pages (all positive), 55,990 annotated tables. Ground-truth HTML markup was generated from the original dataset annotations. Note that not all PubMed PDFs were available; the subset used here covers only papers with valid retrievable PDFs. The PubTables-1M original annotations are available at the [table-transformer repository](https://github.com/microsoft/table-transformer); the reconstructed HTML ground truth and the PDF subset used are provided via the benchmark GitLab repository. Datasets are available at https://gitlab.inria.fr/msoric/table-extraction-benchmark under MIT license. ### Evaluation Metrics used: - **TD:** Precision, Recall, F1, and AP at IoU threshold $\lambda = 0.5$; expected Precision/Recall (integral over $\lambda$ weighted by density $S\_{0.5}$); D-ECE for calibration. - **TSR:** GriTS-Topology, GriTS-Content, TEDS computed on HTML markup normalized to only ``, ``, `
` tags (header/body distinctions stripped for comparability). - **End-to-end TE:** $P\_{\text{TSR}}$ and $R\_{\text{TSR}}$ with each of the three TSR scores; AP versions for probabilistic models and F1 versions for baseline models. TSR evaluation is conditioned on TD: only true-positive detections (IoU $\geq 0.5$) are passed to the structure model. Missed tables receive a score of 0 in the end-to-end aggregation. This means TSR scores cannot be compared directly across models because each model evaluates on a different subset of detected tables. The benchmark does not report statistical significance tests or error bars. Each model is run once. LVLM results carry a contamination caveat: GPT-4o mini may have been trained on benchmark datasets. ### Hardware Inference is measured on CPU (4 CPUs per task, 2 tasks concurrently). No GPU-hours or training hardware are relevant. Approximate inference times per image: PDFPlumber 0.35s, PyMuPDF 0.41s, Grobid 0.78s, Camelot 1.18s, LVLM 8.88s (API), XY+TATR 9.32s, Docling 11.36s, TATR 13.33s, VGT 127.98s. ## BibTeX ```bibtex @article{soric2025benchmarking, title={Benchmarking Table Extraction from Heterogeneous Scientific Documents}, author={Soric, Marijan and Manolescu, Ioana and Gracianne, C\'{e}cile and Senellart, Pierre}, journal={arXiv preprint arXiv:2511.16134}, year={2025} } ```