# PubTabNet: Image-based table recognition > Large-scale dataset (568K tables) with HTML annotations from scientific papers, encoder-dual-decoder architecture for structure+content recognition, and tree-edit-distance metric (TEDS) for evaluation - Authors: Hunter Heidenreich - Published: 2019-11-20 - Updated: 2026-03-12 - Paper: Preprint (arXiv) - Paper: Published (ECCV 2020) - Code: Code (GitHub) - Dataset: Data (Hugging Face) - Source: Bevaya Labs, --- ## TL;DR PubTabNet introduces a large-scale table recognition dataset with 568K table images and HTML ground truth extracted from PubMed Central articles. The paper proposes an encoder-dual-decoder (EDD) architecture that separates structure prediction from cell content generation, achieving 88.3% TEDS (All) versus 78.6% for the single-decoder WYGIWYS baseline. The work also introduces TEDS (tree-edit-distance-based similarity) to address known failure modes of adjacency-based evaluation. ## What kind of paper is this? **Dominant: $\Psi\_{\text{Resource}}$** The primary contribution is the PubTabNet dataset: an automated annotation pipeline, curation methodology, and evaluation tooling for 568K table images with HTML ground truth. **Secondary: $\Psi\_{\text{Method}}$** Introduces the encoder-dual-decoder (EDD) architecture for table recognition. **Secondary: $\Psi\_{\text{Evaluation}}$** Proposes the TEDS metric for measuring table structure similarity via tree-edit distance. The dataset release and benchmark establish the paper's resource contribution, while the architectural and evaluation innovations provide methodological advances. ## What is the motivation? Table recognition requires reconstructing both layout structure (rows, columns, spanning cells) and cell content from images. Progress has been limited by specific gaps: 1. **Lack of large-scale training data**: Existing datasets either provide only structure annotations without cell content or have insufficient scale for deep learning. 2. **Inadequate evaluation metrics**: Adjacency-based metrics under-react to structural errors (misaligned row/column boundaries) and over-react to minor content variations. The authors address both gaps with a dataset construction pipeline, a model architecture suited to the task decomposition, and an evaluation metric aligned with structural similarity. ## What is the novelty? ### Dataset: PubTabNet **Scale and diversity:** - 568,192 table images with HTML ground truth - Sourced from 6,000+ journals in PubMed Central Open Access Subset - Includes cell-type annotations (header vs. body cells) **Auto-annotation pipeline:** 1. Match XML-structured tables from PMCOA articles to page-rendered PDF representations 2. Render table images at 72 PPI and generate HTML ground truth from XML 3. Validate quality via TF-IDF cosine similarity between PDF-extracted text and XML cell text (threshold 0.9), with an additional constraint that text lengths differ by less than 10% 4. Curate for learnability: remove tables with cells spanning more than 10 rows/columns, characters occurring fewer than 50 times, or math/inline-formula nodes; normalize HTML by stripping non-visual attributes and unifying header cell definitions **Design choices:** - HTML as target format enables web integration and natural tree representation for TEDS - Header/body cell distinction supports models that need to differentiate cell roles - Filtering for consistent structure vocabulary balances diversity with learnability ### Model: Encoder-Dual-Decoder (EDD) **Architecture design:** The EDD model decomposes table recognition into two coupled generation tasks: 1. **Structure decoder**: Predicts HTML structural tokens (tags, span attributes) via attention-based LSTM 2. **Cell decoder**: Generates cell content tokens (character-level) when the structure decoder opens a new cell; it receives the structure decoder's hidden state to attend to the correct cell region **Tokenization:** - **Structural tokens:** HTML tags (``, ``, etc.) plus colspan/rowspan attributes (vocabulary size: 32) - **Cell tokens:** Character-level with inline HTML tags (e.g., ``, ``) treated as single tokens (vocabulary size: 281) **Training objective:** $$\mathcal{L} = \lambda \mathcal{L}\_s + (1-\lambda) \mathcal{L}\_c$$ where $\mathcal{L}\_s$ is structure cross-entropy, $\mathcal{L}\_c$ is cell content cross-entropy, and $\lambda \in [0, 1]$ balances the two losses. **Architectural rationale:** Separating structure from content allows the model to focus on layout grammar independently of vocabulary, particularly beneficial for complex spanning patterns. Unlike other dual-decoder architectures where the decoders are independent, EDD's cell decoder is triggered by the structure decoder and receives its hidden state, ensuring a one-to-one match between cells and content sequences. ### Evaluation: TEDS Metric **Definition:** Tables are represented as HTML trees (root $\rightarrow$ thead/tbody $\rightarrow$ tr $\rightarrow$ td, with attributes colspan/rowspan/content). TEDS computes normalized tree-edit distance: $$\text{TEDS}(T\_a, T\_b) = 1 - \frac{\text{EditDist}(T\_a, T\_b)}{\max(|T\_a|, |T\_b|)}$$ **Edit costs:** - Insertion/deletion: cost 1 - Substitution of non-td nodes: cost 1 - Substitution of td nodes: cost 1 if colspan or rowspan differs; otherwise, the normalized Levenshtein distance ($\in [0, 1]$) between cell content strings **Validation:** Perturbation experiments demonstrate that TEDS responds proportionally to structural errors (e.g., row/column misalignment) while the adjacency relation metric under-reacts. At 90% cell-shift perturbation, adjacency F1 remains near 80% while TEDS drops by 60%. Conversely, for cell content perturbations, the adjacency metric over-reacts (dropping over 70% at the 10% perturbation level) while TEDS decreases linearly from 90% to 40% across perturbation levels 10% to 90%. ## What experiments were performed? ### Architecture Specifications **Encoder:** - ResNet-18 CNN backbone - Five encoder variants tested, differing in stride and shared vs. independent final CNN layers - Best variant (EDD-S1S1) uses stride-1 final layers and independent final convolutional layers for structure/cell decoders **Decoders:** - Single-layer LSTMs with hidden dimensions 256 (structure) and 512 (cell) - Soft attention with hidden layer size 256 - Embedding dimensions: 16 (structural tokens) and 80 (cell tokens) - Structure decoder operates autoregressively; cell decoder is invoked when structure decoder emits cell-opening tags **Inference:** - Beam search with beam width 3 - Structure and cell predictions are synchronized: cell decoder must complete before structure decoder continues ### Data Construction Details **Source:** PubMed Central Open Access Subset scientific articles with both PDF renderings and XML markup. **Pipeline steps:** 1. **Matching:** Align XML table elements to page-rendered table regions using the algorithm from Zhong et al. (PubLayNet) 2. **Rendering:** Generate table images from PDFs at 72 PPI 3. **HTML generation:** Convert XML markup to HTML with normalized tag vocabulary 4. **Quality validation:** Compute TF-IDF cosine similarity (PDF-extracted text vs. XML cell text); threshold at 0.9 with length difference below 10% 5. **Curation:** Remove tables with rare structures (cells spanning $>$10 rows/columns, characters with $<$50 occurrences), math/inline-formula nodes, or multiple merged tables **Curation rationale:** The authors filter for consistent structure patterns to improve learnability, trading exhaustive diversity for training stability. ### Scale and Splits | Split | Original | Balanced (used for eval) | |-------|----------|--------------------------| | Train | 548,592 | 548,592 | | Val | 8,910 | 10,000 (5K spanning + 5K non-spanning) | | Test | 10,690 | 10,000 (5K spanning + 5K non-spanning) | **Training constraint:** GPU memory limits require filtering the training set to 399K samples satisfying: - Image dimensions $\leq$ 512 $\times$ 512 pixels - Structural tokens $\leq$ 300 - Longest cell $\leq$ 100 tokens Validation and test sets are **not** subject to these constraints, ensuring evaluation on full complexity. **Balanced splits rationale:** Raw dev/test distributions are skewed toward simple non-spanning tables. Balanced subsets ensure adequate representation of complex spanning structures. ### Training Setup **Hardware:** Two NVIDIA V100 GPUs, approximately 16 days training time. **Preprocessing:** - Training images rescaled to 448 $\times$ 448 pixels for batching, with per-channel z-score normalization **Optimization:** - Two-stage training: 1. **Structure pretraining:** $\lambda = 1$ (structure-only loss), batch size 10, learning rate 0.001 for 10 epochs then 0.0001 for 3 epochs 2. **Joint training:** $\lambda = 0.5$ (balanced structure + content), batch size 8, learning rate 0.001 for 10 epochs then 0.0001 for 2 epochs - Adam optimizer ### Baselines **Off-the-shelf extraction tools (PDF input):** - Tabula, Traprange, Camelot, PDFPlumber (document parsing libraries that require text-based PDF) - Adobe Acrobat Pro (tested with both PDF and high-resolution 300 PPI image input) **Model baselines:** - WYGIWYS: Single-decoder image-to-markup architecture (Deng et al.), trained on PubTabNet - TIES: Graph neural network model evaluated on synthetic data ### PubTabNet Test Results | Input | Method | Simple TEDS (%) | Complex TEDS (%) | All TEDS (%) | |-------|--------|-----------------|-------------------|--------------| | PDF | Tabula | 78.0 | 57.8 | 67.9 | | PDF | Traprange | 60.8 | 49.9 | 55.4 | | PDF | Camelot | 80.0 | 66.0 | 73.0 | | PDF | PDFPlumber | 44.9 | 35.9 | 40.4 | | PDF | Acrobat Pro | 68.9 | 61.8 | 65.3 | | Image | Acrobat Pro | 53.8 | 53.5 | 53.7 | | Image | WYGIWYS | 81.7 | 75.5 | 78.6 | | Image | **EDD-S1S1** | **91.2** | **85.4** | **88.3** | ### Generalization to Synthetic Data **Setup:** 500K synthetic tables (420K/40K/40K train/val/test split) used to compare with TIES baseline, which lacks sufficient real-data training labels. **Results (four complexity levels C1/C2/C3/C4):** | Model | Avg TEDS | Exact Match | |-------|----------|-------------| | TIES | N/A | 96.9 / 94.7 / 52.9 / 68.5 | | EDD | 99.8 / 99.8 / 99.7 / 99.7 | 99.7 / 99.9 / 97.2 / 98.0 | **Evaluation note:** TIES comparison uses adjacency-based exact match without checking cell content recognition errors. For fairness, EDD's cell recognition errors are ignored in this comparison, measuring only structural correctness. ### Ablations **Encoder variants:** - Tested feature-map resolution (stride-1 vs. stride-2) and shared vs. independent final CNN layers - EDD-S1S1 (stride-1, independent layers) selected via validation performance as the best configuration ## What are the outcomes/conclusions? **Key observations:** - EDD achieves +9.7 TEDS improvement over the WYGIWYS single-decoder baseline (88.3% vs. 78.6%) - The advantage is more pronounced on complex (spanning) tables: +9.9 TEDS (85.4% vs. 75.5%) compared to +9.5 on simple tables (91.2% vs. 81.7%) - Camelot is the best off-the-shelf tool at 73.0% All TEDS - Adobe Acrobat Pro's performance drops substantially when using image input (53.7%) compared to PDF input (65.3%), illustrating the difficulty of image-only table recognition - On synthetic data, EDD achieves near-perfect TEDS ($>$99.7%) across all complexity categories, with no significant degradation on complex structures (unlike TIES) **Error analysis:** Both EDD and WYGIWYS show performance degradation as table size increases (in width, height, structural token count, or longest cell length). The authors attribute this primarily to aggressive image downsampling for batching and suggest grouping tables by size with different rescaling factors. ### Limitations **Missing spatial information:** PubTabNet does not include cell bounding box coordinates. The authors note this limits integration with detection-based pipelines and plan to add spatial annotations in future releases (PubTabNet 2.0.0, released July 2020, added bounding boxes for non-empty cells). **Detection not included:** EDD assumes pre-cropped table images. End-to-end document processing requires coupling with a separate table detection model. **Scale sensitivity:** Performance degrades on large tables. The authors suggest batching by table size to reduce aggressive downsampling, which loses fine-grained spatial detail. **Training subset constraint:** GPU memory limitations require filtering to 399K samples (approximately 73% of the full training set) with size/token constraints. Full-scale training might improve performance but requires larger memory or architectural modifications. **License complexity:** While annotations are permissive (CDLA-Permissive-1.0), underlying PMCOA images have mixed per-article licenses. Commercial users must audit article-level terms. **No code or weights released:** Due to legal constraints, IBM does not release the EDD model code or pretrained weights. Replication requires reimplementing the architecture from the paper's description. ## Reproducibility ### Models - ResNet-18 encoder with modified final layers (EDD-S1S1: stride-1, independent layers for each decoder) - Structure decoder: single-layer LSTM, hidden size 256, embedding size 16 - Cell decoder: single-layer LSTM, hidden size 512, embedding size 80 - Attention hidden layer size: 256 for both decoders - **No pretrained weights released** due to legal constraints ### Algorithms - Two-stage training: structure pretraining ($\lambda = 1$) followed by joint training ($\lambda = 0.5$) - Adam optimizer; stage 1: batch 10, LR 0.001 (10 epochs) then 0.0001 (3 epochs); stage 2: batch 8, LR 0.001 (10 epochs) then 0.0001 (2 epochs) - Beam search decoding with beam width 3 - Training images rescaled to 448 $\times$ 448, per-channel z-score normalization ### Data - 568K table images from PubMed Central Open Access Subset - Training subset filtered to 399K (image $\leq$ 512 $\times$ 512, structure tokens $\leq$ 300, longest cell $\leq$ 100) - Balanced val/test: 10K each (5K spanning + 5K non-spanning) - Annotations: CDLA-Permissive-1.0; images: per-article PMCOA licenses - Test set ground truth withheld for ICDAR competition ### Evaluation - Primary metric: TEDS (tree-edit-distance-based similarity), reported as mean across test samples - Results broken down by Simple (non-spanning) and Complex (spanning) tables - Baselines include both PDF-input tools and image-input models - TIES comparison on synthetic data uses structure-only exact match for fairness ### Hardware - 2x NVIDIA V100 GPUs - Approximately 16 days total training time - No inference latency or throughput figures reported ## BibTeX ```bibtex @inproceedings{zhong2020image, title={Image-based table recognition: data, model, and evaluation}, author={Zhong, Xu and ShafieiBavani, Elaheh and Yepes, Antonio Jimeno}, booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XII 16}, pages={564--580}, year={2020}, organization={Springer} } ```