# PubTables-1M > A comprehensive review of PubTables-1M, a large-scale dataset for table extraction with canonicalized annotations to address ground truth inconsistency. - Authors: Bevaya AI Team - Published: 2021-09-30 - Updated: 2026-03-12 - Paper: Paper - Code: Table Transformer - Dataset: PubTables-1M - Model: Detection Model - Model: Structure Model - Source: Bevaya Labs, --- ## TL;DR PubTables-1M is a large-scale table extraction dataset containing nearly one million tables from PubMed Central Open Access articles. The dataset introduces richer annotations (including blank cells, rows, columns, and projected row headers) and a canonicalization procedure to address oversegmentation in markup-derived ground truth, substantially improving both training and evaluation reliability for table structure recognition. --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$ (the dataset, annotation pipeline, and quality verification procedures constitute the primary contribution). **Secondary:** $\Psi\_{\text{Evaluation}}$ (demonstrates how canonicalization reduces ground truth ambiguity and changes evaluation conclusions), $\Psi\_{\text{Method}}$ (DETR-based baseline models for table detection and structure recognition). ## What is the motivation? Table extraction requires inferring logical structure from visual presentation; modern deep learning approaches depend on large-scale, unambiguous ground truth. Prior markup-derived datasets exhibit three key limitations: - **Incomplete spatial annotations:** lack explicit row/column/cell bounding boxes; omit blank cells entirely - **Missing header information:** do not label row headers or projected headers, limiting functional analysis capabilities - **Oversegmentation artifacts:** spanning header cells are frequently split into multiple grid cells in source markup, creating contradictory or non-unique ground truth that degrades both training and evaluation ## What is the novelty? - **Scale and coverage:** nearly 1M tables supporting all three subtasks (table detection, table structure recognition, functional analysis) - **Richer annotation schema:** explicit bounding boxes for rows, columns, and cells (including blank cells); projected row header labels - **Canonicalization procedure:** data is processed via automated rule-based merging of oversegmented header cells to produce unique, unambiguous structure interpretations. The algorithm operates in two phases: (1) inferring header regions (column headers, projected row headers, and first-column row headers) and (2) merging adjacent cells under structural constraints derived from the Wang model's hierarchical header-tree assumption. - **Automated quality control:** verification pipeline with measurable consistency checks (text alignment quality, cell-word assignment, overlap detection) ## What experiments were performed? The authors trained object detection baselines (Faster R-CNN and DETR) for table detection and joint table structure recognition with functional analysis. Key experimental comparisons: - **Canonicalization ablation:** training DETR on canonicalized vs. non-canonical annotations (DETR-NC), evaluated against both test label variants - **Noise isolation:** testing the same model (DETR-NC) against canonical vs. non-canonical test labels to measure evaluation noise independent of training data effects - **Metrics:** table detection uses standard object detection metrics (AP/AP50/AP75/AR); table structure recognition reports $\text{Acc}\_{\text{Cont}}$ (exact table content match), adjacent cell content F-score ($\text{Adj}\_{\text{Cont}}$), and **GriTS** (Grid Table Similarity). **GriTS formulation:** $$ \text{GriTS}\_f(\mathbf{A}, \mathbf{B}) = \frac{2 \cdot \sum\_{i,j} f(\tilde{\mathbf{A}}\_{i,j}, \tilde{\mathbf{B}}\_{i,j})}{|\mathbf{A}| + |\mathbf{B}|} $$ where $\mathbf{A}$ and $\mathbf{B}$ are the ground truth and predicted table matrices, $\tilde{\mathbf{A}}$ and $\tilde{\mathbf{B}}$ are their most similar substructures (a selection of $m$ rows and $n$ columns), and $f$ is a similarity function. Three variants measure different aspects: $\text{GriTS}\_{\text{Top}}$ for cell topology, $\text{GriTS}\_{\text{Cont}}$ for cell content, and $\text{GriTS}\_{\text{Loc}}$ for cell location. ## What are the outcomes/conclusions? **Outcomes** Canonicalized ground truth materially changes experimental conclusions. For complex tables, exact-match table content accuracy ($\text{Acc}\_{\text{Cont}}$) jumps from 0.5360 (DETR-NC evaluated on non-canonical test labels) to 0.6944 (DETR evaluated on canonical test labels). This comparison conflates two effects (improved training data and cleaner test labels), but the authors disentangle them by also evaluating DETR-NC against canonical test labels. In that isolated comparison, DETR-NC scores 0.9349 on simple tables with canonical test labels vs. 0.8678 with non-canonical test labels, demonstrating that canonicalization alone reduces evaluation noise. DETR achieves strong performance under the object detection framing without task-specific customizations (test AP 0.966 for table detection, 0.912 for joint structure recognition and functional analysis). **Limitations** - **Domain specificity:** canonicalization rules are designed around PMCOA-derived annotation characteristics; generalization to other domains may require new assumptions. The authors explicitly note this in the paper. - **Single-page scope:** multi-page tables are excluded. - **Partial header inference:** full row-header structure is outside scope; coverage limited to projected row headers and first-column structure. - **No cross-dataset evaluation:** all experiments use PubTables-1M only; the paper does not test whether models trained on PubTables-1M generalize to other table extraction benchmarks. --- ## Reproducibility ### Models - Both DETR and Faster R-CNN use **ResNet-18** backbones pretrained on ImageNet with early layers frozen. - DETR architecture: 6 encoder layers, 6 decoder layers. TD model uses 15 object queries; TSR+FA model uses 125 object queries (slightly above the maximum object count in training data). - Pre-trained weights for both detection and structure recognition models are publicly released under MIT license. ### Algorithms - All models trained for **20 epochs** on a single NVIDIA Tesla V100 GPU. - Learning rate selection: one short experiment comparing initial learning rates of 0.0002, 0.0001, and 0.00005; best validation performance after one epoch determined the choice. - TSR+FA model: initial learning rate of 0.00005, no-object class weight of 0.4. - Both models: learning rate drop of 1, gamma of 0.9. - Standard data augmentations (random cropping, resizing). No custom components, losses, or training procedures. - TD input: PDF pages rendered as images with maximum length of 1000 pixels. - TSR+FA input: table region cropped from page image with 30-pixel padding on all sides. ### Data - Source: PubMed Central Open Access (PMCOA) scientific articles. - Alignment: Needleman-Wunsch algorithm for character-level matching between XML markup and PDF-rendered text. - Splits: 80/10/10 at document level to prevent leakage. - **947,642** tables total for TSR; 460,589 pages for TD. - Quality filters discard tables with overlapping rows/columns, high edit distance (threshold 0.05), low word-cell overlap (threshold 0.9), or extreme outlier object counts (>100). Less than 0.1% of tables are discarded as outliers. - Licensing: CDLA-Permissive-2.0 applies to Microsoft's annotations. Underlying PMCOA articles have mixed licenses (CC0, CC BY, CC BY-NC, etc.); per-article verification is needed for commercial use. ### Evaluation - **TD metrics:** AP, AP50, AP75, AR (standard COCO-style object detection). - **TSR metrics:** $\text{Acc}\_{\text{Cont}}$ (exact table content match), $\text{Adj}\_{\text{Cont}}$ (adjacent cell content F-score from Gobel et al.), $\text{GriTS}\_{\text{Top}}$, $\text{GriTS}\_{\text{Cont}}$, $\text{GriTS}\_{\text{Loc}}$. - Post-prediction, bounding boxes are retroactively tightened (removing dilation padding) before scoring, so that training-time dilation does not unfairly penalize location metrics. - Baselines compared: DETR vs. Faster R-CNN (both on canonical data), plus DETR-NC (on non-canonical data) for the canonicalization ablation. - No error bars, significance tests, or multi-run reporting. Single training run per configuration. - No cross-dataset evaluation or comparison with prior published results on other benchmarks. ### Hardware - **Training:** Single NVIDIA Tesla V100 GPU (reported in Appendix 9.1). - Training time and GPU-hours are not reported. - Inference latency and throughput are not reported. --- ## Reported Results (Test Set) **Table Detection (TD):** | Model | AP | AP50 | AP75 | AR | |-------|-----|------|------|-----| | DETR | 0.966 | 0.995 | 0.988 | 0.981 | | Faster R-CNN | 0.825 | 0.985 | 0.927 | 0.866 | **Joint TSR+FA (Object Detection):** | Model | AP | AP50 | AP75 | AR | |-------|-----|------|------|-----| | DETR | 0.912 | 0.971 | 0.948 | 0.942 | | Faster R-CNN | 0.722 | 0.815 | 0.785 | 0.762 | **Table Structure Recognition (Canonical Test, All Tables):** | Model | $\text{Acc}\_{\text{Cont}}$ | $\text{GriTS}\_{\text{Top}}$ | $\text{GriTS}\_{\text{Cont}}$ | $\text{GriTS}\_{\text{Loc}}$ | |-------|--------|--------|--------|--------| | DETR | 0.8138 | 0.9845 | 0.9846 | 0.9781 | | DETR-NC | 0.5851 | 0.9576 | 0.9588 | 0.9449 | | Faster R-CNN | 0.1039 | 0.8616 | 0.8538 | 0.7211 | --- ## Mapping to Unified Taxonomy PubTables-1M covers **both Table Detection (TD)** and **Table Structure Recognition (TSR)**. ### Detection (Layout Level) For the Layout Analysis pipeline, we only care about the Detection classes: | PubTables-1M Class | Visual Primitive | Logical Role | Notes | | :--- | :--- | :--- | :--- | | **Table** | `Table` | `Table` | Core bounding box. | ### Structure (TSR Level) Inside the detected table, it provides detailed structure annotations (rows, columns, headers) similar to PubTabNet but with bounding boxes for cells. These are not part of the page-level Layout Taxonomy but are inputs for the **OTSL** pipeline. --- ## BibTeX ```bibtex @inproceedings{smock2022pubtables, title={PubTables-1M: Towards comprehensive table extraction from unstructured documents}, author={Smock, Brandon and Pesala, Rohith and Abraham, Robin}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pages={4634--4642}, year={2022} } ```