# Benchmarking PDF Parsers on Table Extraction with LLM-based Semantic Evaluation > A benchmarking study of 21 PDF parsers on table extraction using synthetically generated documents and LLM-as-a-judge evaluation, showing that LLM-based metrics substantially outperform rule-based metrics in agreement with human judgment. - Authors: Hunter Heidenreich - Published: 2026-03-30 - Paper: Preprint (arXiv) - Code: pdf-parse-bench (benchmark pipeline and parser configs) - Dataset: pdf-parse-bench (benchmark dataset on Hugging Face) - Code: table-metric-study (metric evaluation and human study) - Source: Bevaya Labs, --- ## TL;DR Horn and Keuper evaluate 21 contemporary PDF parsers on table extraction using 100 synthetically generated documents with exact LaTeX ground truth. Their central finding is that LLM-as-a-judge evaluation correlates substantially better with human judgment (Pearson $r = 0.93$ for Gemini-3-Flash-Preview) than rule-based metrics such as TEDS ($r = 0.68$) or GriTS ($r = 0.70$). Among the 21 parsers tested, the Gemini 3 family achieves the highest scores, while rule-based tools lag far behind all learning-based approaches, and parser choice can mean the difference between near-perfect and nearly unusable table extraction. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Evaluation}}$ The paper's primary contribution is a new evaluation methodology for table extraction. It proposes LLM-as-a-judge as a replacement for rule-based metrics, validates that methodology against over 1,500 human ratings, and applies it to benchmark 21 parsers. The headline results concern measurement reliability and parser rankings, not a new model architecture. **Secondary:** $\Psi\_{\text{Systematic}}$ The work also surveys the landscape of existing table extraction metrics and document-level benchmarks, systematically characterizing the failure modes of TEDS, GriTS, and SCORE with concrete examples from real parser outputs. ## What is the motivation? Accurate table extraction from PDFs is increasingly important for scientific data mining, language model pretraining, and retrieval-augmented generation pipelines. Yet the dominant evaluation metrics, TEDS and GriTS, operate purely at the syntactic level. Both metrics compare structure and cell content as strings, which means they penalize semantically harmless representational differences (format conversions, symbol encoding variants, value equivalences like "85.0%" vs. "85%") while being nearly insensitive to small but meaning-altering errors (a lost decimal point, a sign flip). This creates a measurement gap: a parser that produces structurally different but semantically equivalent tables is unfairly penalized, while one that preserves structure but corrupts a handful of cell values receives an inflated score. Existing document-level benchmarks such as OmniDocBench and READoc adopt the same metrics without addressing this flaw. Separately, most benchmarks rely on manually annotated ground truth, which is expensive and limits scale and reproducibility. ## What is the novelty? The paper makes three interconnected contributions. **LLM-based semantic evaluation.** The authors propose using an LLM as a judge to evaluate table extraction quality. Given a ground truth table and its parsed counterpart, the model assigns scores on a 0 to 10 scale for content accuracy and structural preservation, defined as whether every cell value can be unambiguously mapped to its row and column headers. This framing captures semantic equivalence that string-level comparison cannot. **Synthetic benchmark with exact ground truth.** Rather than relying on manual annotation, the authors construct 100 benchmark pages by embedding real LaTeX tables sourced from arXiv papers (published December 2025, to avoid overlap with parser training data) into synthetically generated PDFs. Each page samples a random layout configuration (document class, font, margins, column layout) and fills available space with filler text and tables, compiled with `pdflatex`. The LaTeX source serves as exact ground truth at zero annotation cost. Tables are classified into three structural complexity tiers by an LLM classifier: simple (regular grid), moderate (limited cell merging), and complex (multi-dimensional merging, nested structures). **LLM-based table matching pipeline.** Because parsers produce tables in diverse formats (HTML, Markdown, LaTeX, plain text) and may split, merge, or reorder content, aligning parser output to ground truth tables is non-trivial. The authors address this with a Gemini-3-Flash-Preview matching step that identifies and extracts the parsed counterpart of each ground truth table from the full parser output, followed by rule-based post-validation to correct minor LLM artifacts. ## What experiments were performed? **Meta-evaluation of metrics.** To compare automated metrics against human judgment, the authors collected 1,554 human ratings across 518 ground truth/parser output table pairs, sampled from diverse parsers, output formats, and table complexities. Three independent evaluators each rated all 518 pairs on a 0 to 10 scale. Inter-annotator agreement is reported as Krippendorff's $\alpha = 0.77$ (interval), with average pairwise Pearson correlation $r = 0.85$ and a human ceiling of $r = 0.89$ (leave-one-out). To help surface subtle discrepancies, evaluators were shown LLM-generated hints identifying potential differences, though final scores remained entirely human decisions. Four LLM judges were evaluated: DeepSeek-v3.2, GPT-5-mini, Gemini-3-Flash-Preview, and Claude Opus 4.6. All rule-based metrics (TEDS, GriTS variants, SCORE variants) were rescaled to a 0 to 10 range for comparability. Correlations with mean human scores across 518 pairs: | Metric | Type | Pearson $r$ | Spearman $\rho$ | Kendall $\tau$ | |:---|:---|:---|:---|:---| | TEDS | Rule-based | 0.684 | 0.717 | 0.557 | | GriTS-Avg | Rule-based | 0.698 | 0.763 | 0.604 | | SCORE-Avg | Rule-based | 0.637 | 0.684 | 0.539 | | DeepSeek-v3.2 | LLM | 0.802 | 0.827 | 0.713 | | GPT-5-mini | LLM | 0.888 | 0.827 | 0.739 | | Gemini-3-Flash-Preview | LLM | 0.927 | 0.889 | 0.799 | | Claude Opus 4.6$^\dagger$ | LLM | 0.939 | 0.890 | 0.804 | $^\dagger$ Also used to generate error hints shown to evaluators; correlation may be inflated. The authors note that Claude Opus 4.6 also generated the error hints shown to human evaluators, which may inflate its correlation. Gemini-3-Flash-Preview ($r = 0.927$) had no role in the annotation process, making it the recommended judge for the subsequent parser benchmark. **Parser benchmark.** Using the validated Gemini-3-Flash-Preview judge, the authors evaluated 21 parsers on 451 tables from the 100 synthetic pages, reporting mean LLM score (0 to 10), per-complexity breakdowns, TEDS for comparison, and inference cost or wall-clock time on an NVIDIA RTX 4090. The parser set spans a broad spectrum: specialized OCR VLMs (LightOnOCR-2-1B, dots.ocr, MonkeyOCR-3B, GOT-OCR2.0, Chandra, olmOCR-2-7B, Nanonets-OCR-s, DeepSeek-OCR, MinerU2.5), commercial API services (Mathpix, Mistral OCR 3), general-purpose multimodal models prompted for Markdown output (Gemini 3 Pro, Gemini 3 Flash, Gemini 2.5 Flash, Qwen3-VL-235B, GLM-4.5V, GPT-5 mini, GPT-5 nano, Claude Sonnet 4.6), and rule-based tools (PyMuPDF4LLM, GROBID). ## What are the outcomes/conclusions? **LLM evaluation vs. rule-based metrics.** All rule-based metrics cluster in a narrow band of $r = 0.56$ to $0.70$ with human judgment. Even the weakest LLM judge (DeepSeek-v3.2, $r = 0.80$) surpasses the best rule-based metric. The authors argue this confirms that semantic assessment captures dimensions of table quality that string-level comparison systematically misses. TEDS scores for the 21 parsers cluster within 22% of its scale (0.66 to 0.88), while LLM scores span 38% (5.75 to 9.55), suggesting the former paints a misleadingly uniform picture of parser quality. **Parser rankings.** Overall LLM scores range from 2.10 (GROBID) to 9.55 (Gemini 3 Pro). Selected results: | Parser | LLM Score | Simple | Moderate | Complex | Cost/Time | |:---|:---:|:---:|:---:|:---:|:---| | Gemini 3 Pro | 9.55 | 9.58 | 9.57 | 9.49 | $10.00 API | | Gemini 3 Flash | 9.50 | 9.53 | 9.38 | 9.61 | $0.57 API | | LightOnOCR-2-1B | 9.08 | 9.41 | 8.90 | 8.91 | 30 min GPU | | Mistral OCR 3 | 8.89 | 8.92 | 8.69 | 9.07 | $0.20 API | | dots.ocr | 8.73 | 9.01 | 8.43 | 8.76 | 20 min GPU | | MinerU2.5 | 6.49 | 7.07 | 6.03 | 6.35 | free-tier API | | PyMuPDF4LLM | 5.25 | 6.78 | 4.86 | 3.91 | 30 s CPU | | GROBID | 2.10 | 2.27 | 1.94 | 2.09 | 2 min CPU | The top two systems are general-purpose multimodal models (Gemini 3), not dedicated OCR tools, suggesting broad visual-linguistic capabilities transfer well to table extraction. However, LightOnOCR-2-1B achieves 9.08 with only 1B parameters, running on a single consumer GPU within 30 minutes, narrowing the gap between proprietary APIs and self-hosted pipelines for privacy-constrained deployments. **Complexity sensitivity.** Table complexity affects parsers unevenly. Gemini 3 Flash actually scores slightly higher on complex tables than on simple ones, while GLM-4.5V drops 2.19 points from simple to complex, Qwen3-VL-235B drops 1.56 points, and Mathpix drops 1.55 points. The authors identify handling multi-dimensional cell merging as a key differentiator. **Score distributions.** Per-parser histograms reveal bimodal failure patterns that mean scores obscure. Top parsers (Gemini 3, LightOnOCR) concentrate more than 70% of tables at score 10. Claude Sonnet 4.6 and olmOCR-2-7B show strongly bimodal distributions: they frequently omit tables entirely (score 0) but extract near-perfectly when they succeed. Mid-tier parsers such as GPT-5 mini and Gemini 2.5 Flash produce broad distributions centered around scores 5 to 8, indicating pervasive partial errors. The authors note that, depending on the application, a missed table may be preferable to a corrupted one, making bimodal parsers more useful than uniformly mediocre ones in certain contexts. **Limitations.** The authors acknowledge several scope restrictions. Synthetic PDFs do not capture the full diversity of real-world documents (scanned pages, non-standard layouts). The table source is exclusively arXiv, which may bias toward scientific formats and leave financial or medical table styles unrepresented. LLM-as-a-judge is not infallible and depends on proprietary models, though evaluation costs are described as modest (approximately $0.20 to score all 451 tables, roughly $1 per parser for a full benchmark run). The authors do not report statistical significance for parser score differences, which may matter for closely ranked systems. ## Reproducibility ### Models The paper does not introduce a new model. The judge model used for the parser benchmark is Gemini-3-Flash-Preview (Google DeepMind), accessed via API. The matching pipeline also uses Gemini-3-Flash-Preview. For the human evaluation study, Claude Opus 4.6 was used to generate discrepancy hints shown to raters. No model weights are released as part of this work. ### Algorithms The benchmark pipeline proceeds as follows: 1. LaTeX tables are scraped from arXiv papers (December 2025), cleaned, and classified into simple/moderate/complex tiers by an LLM classifier. 2. Synthetic PDF pages are generated by sampling random layout configurations and filling pages with filler text and tables, compiled with `pdflatex`. 3. Each of the 21 parsers processes the 100 PDF pages. 4. A Gemini-3-Flash-Preview matching step identifies the parsed counterpart for each ground truth table in the parser output, followed by rule-based post-validation. 5. Matched pairs are scored by the LLM judge on a 0 to 10 scale for content accuracy and structural preservation. The evaluation pipeline, parser configurations, exact prompts, and software versions used to produce the leaderboard are available in the `pdf-parse-bench` repository. The metric meta-evaluation code and human study data are in the `table-metric-study` repository. ### Data The benchmark consists of 100 synthetically generated PDF pages containing 451 tables derived from arXiv papers published in December 2025. Ground truth is the original LaTeX source, requiring no manual annotation. The dataset is released as part of the `pdf-parse-bench` repository and on Hugging Face (`piushorn/pdf-parse-bench`) under the MIT license. For the human evaluation study, 518 ground truth/parser output table pairs were rated by three independent evaluators, yielding 1,554 total ratings. The human study data is available in the `table-metric-study` repository. ### Evaluation The primary metric is the LLM judge score (0 to 10) using Gemini-3-Flash-Preview, selected over Claude Opus 4.6 for cost efficiency and lack of confounding with the human annotation process. TEDS (0 to 1 scale) is reported alongside for comparison purposes. Rule-based baselines evaluated in the meta-study: TEDS, GriTS$\_{\text{Top}}$, GriTS$\_{\text{Con}}$, GriTS-Avg, SCORE Index, SCORE Content, SCORE-Avg. LLM judges evaluated in the meta-study: DeepSeek-v3.2, GPT-5-mini, Gemini-3-Flash-Preview, Claude Opus 4.6. Human reference scores were collected from three evaluators rating 518 pairs, with Krippendorff's $\alpha = 0.77$ and average pairwise Pearson $r = 0.85$. ### Hardware Parser benchmarking was run on a single NVIDIA RTX 4090 for GPU-based models, or via API for cloud-hosted models. API pricing in USD is reported at the time of writing. No training hardware is required as the benchmark evaluates existing parsers. ## BibTeX ```bibtex @article{horn2026benchmarking, title={Benchmarking PDF Parsers on Table Extraction with LLM-based Semantic Evaluation}, author={Horn, Pius and Keuper, Janis}, journal={arXiv preprint arXiv:2603.18652}, year={2026} } ```