# TableBank: Table Benchmark for Image-based Table Detection and Recognition > Large-scale weakly supervised dataset of 417K+ tables for detection and structure recognition built from Word and LaTeX sources. - Authors: Hunter Heidenreich - Published: 2019-03-05 - Updated: 2026-03-12 - Paper: Paper - Code: Code - Dataset: Data (HuggingFace) - Model: Model Zoo - Source: Bevaya Labs, --- ## TL;DR TableBank is a large-scale, weakly supervised dataset of 417K+ labeled tables for image-based table detection and structure recognition. The authors generate high-quality annotations automatically from Word (Office XML) and LaTeX source code, sidestepping the need for manual labeling. Baseline Faster R-CNN and image-to-text models demonstrate that scale matters but cross-domain generalization (Word vs. LaTeX) remains a challenge. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$ : Introduces a large-scale benchmark dataset (TableBank) with an automated label generation pipeline and baseline models. **Secondary:** $\Psi\_{\text{Evaluation}}$ : Provides baseline systems and cross-domain evaluation (Word vs. LaTeX vs. mixed). --- ## What is the motivation? Table detection and structure recognition are central tasks in document analysis because tables encode structured information across diverse layouts. Traditional heuristics fail to generalize, while deep learning methods require more labeled data than effectively exists in human-annotated sets (which are typically small, e.g., ~1k tables). Manual annotation is prohibitively expensive at scale. The key insight is that many documents (Word, LaTeX) contain explicit table markup in their source code. This allows for scalable, high-quality weak supervision without human labeling. --- ## What is the novelty? The core innovation is the **automatic generation of weak labels** from source documents to create a dataset orders of magnitude larger than prior work. ### 1. Weak Supervision Pipeline The authors developed a method to extract bounding boxes and structure labels by manipulating source code: - **Word:** Modifying Office XML tags (``) to render tables with distinct borders. - **LaTeX:** Wrapping table environments in `fcolorbox` with distinct colors. - **Label Extraction:** Recovering bounding boxes by pixel-level differencing between the "marked" rendered page and the specific table color. ### 2. Dataset Scale This pipeline produced **TableBank**, utilizing documents crawled from the web and arXiv: - **Detection:** 417,234 labeled tables (163k Word, 253k LaTeX). - **Structure Recognition:** 145,463 instances. - **Validation:** Manual spot-checks of 1,000 samples found only 5 erroneous bounding boxes. ### 3. Structure Recognition Formulation The paper formulates structure recognition as an image-to-text task, predicting an **HTML-like tag sequence** (e.g., ``, ``, ``, ``, ``) rather than just coordinates. The vocabulary is deliberately small (12 tokens), keeping the output space tractable. --- ## What experiments were performed? ### Baseline Models **Table Detection:** - **Architecture:** Faster R-CNN with ResNeXt-101 and ResNeXt-152 backbones (ImageNet pretrained). - **Training:** Detectron framework (Caffe2), 4×P100 GPUs, standard synchronous SGD. **Structure Recognition:** - **Architecture:** Encoder-decoder image-to-text model with attention (OpenNMT). - **Output:** Sequence of layout tags implies the table structure. Cell content is recognized separately via OCR and filled into the predicted structure heuristically. ### Evaluation Metrics The authors assess detection using area-based metrics rather than standard object detection mAP. Precision and Recall are computed via pixel-area overlap aggregation across documents (following Gilani et al., 2017): $$ \begin{aligned} \text{Precision} &= \frac{\text{Area of Ground Truth } \cap \text{ Detected}}{\text{Area of Detected Tables}} \\\\ \text{Recall} &= \frac{\text{Area of Ground Truth } \cap \text{ Detected}}{\text{Area of Ground Truth Tables}} \\\\ \text{F1} &= 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \end{aligned} $$ Structure recognition is evaluated using **4-gram BLEU** on the generated tag sequence, hypothesizing that n-gram overlap correlates with structural correctness. --- ## What are the outcomes/conclusions? ### Key Findings 1. **Scale matters:** Deep learning baselines trained on TableBank outperform traditional methods and models trained on smaller datasets (like ICDAR 2013). 2. **Domain gaps exist:** Models trained on Word documents perform poorly on LaTeX (and vice versa). Mixed training (Word+LaTeX) successfully improves cross-domain robustness. 3. **Sequence length limits:** The structure recognition model struggles with complex tables. Exact-match rates drop significantly as the length of the tag sequence increases. ### Limitations - **Metric Selection:** The area-based precision/recall metrics are less standard than COCO-style mAP, potentially obscuring object-level errors like merging or splitting adjacent tables. - **Cross-Domain Generalization:** Despite mixed training, the gap between document types remains a challenge for structure recognition. - **Task Scope:** The structure recognition task is limited to layout tags; it does not solve end-to-end cell text extraction within the same model. --- ## Reproducibility ### Models - **Table Detection:** Faster R-CNN with ResNeXt-101 and ResNeXt-152 backbones, pretrained on ImageNet. Implemented via the Detectron framework (Caffe2). Confidence threshold set to 90% at inference. - **Table Structure Recognition:** Encoder-decoder image-to-text model from OpenNMT. Output vocabulary is 12 tokens (``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``). ### Algorithms - **Detection training:** Synchronous SGD with a mini-batch size of 16 images. Other hyperparameters use Detectron defaults. - **Structure recognition training:** Learning rate of 0.1, batch size of 24. Other hyperparameters use OpenNMT defaults. ### Data - **Word documents:** Crawled from the internet in `.docx` format. Multi-language (English, Chinese, Japanese, Arabic, etc.). - **LaTeX documents:** Sourced from arXiv bulk data access (2014 to 2018). Primarily English. - **Detection split:** 415,234 training images; 2,000 sampled from each domain (Word, LaTeX) for validation (1,000) and test (1,000). - **Structure recognition split:** 144,463 training instances; 500 each for validation and test per domain. - **License Note:** The GitHub repository `LICENSE` file is Apache 2.0, but the README explicitly states "Our data can only be used for research purpose" and "Please DO NOT re-distribute our data." We recommend adhering to the stricter README terms for the dataset itself. ### Evaluation - **Detection metric:** Area-based Precision/Recall/F1 (following Gilani et al., 2017), aggregated across documents by pixel-area overlap. This is not standard COCO-style mAP; it may obscure object-level errors such as merging or splitting adjacent tables. - **Structure recognition metric:** 4-gram BLEU on generated tag sequences against a single reference. - **ICDAR 2013 cross-evaluation:** TableBank models also evaluated on the ICDAR 2013 table competition dataset. ### Hardware - 4$\times$ NVIDIA P100 GPUs for both detection and structure recognition baselines. - No GPU-hours, inference latency, or cost estimates reported. ## Mapping to Unified Taxonomy TableBank is a specialized dataset focused entirely on a single Primitive: **Table**. It does not concern itself with text, figures, or hierarchy. | TableBank Class | Visual Primitive | Logical Role | Notes | | :--- | :--- | :--- | :--- | | **Table** | `Table` | `Table` | The only class. Includes diverse layouts (APA, grid, borderless). | ## BibTeX ```bibtex @inproceedings{li-etal-2020-tablebank, title = "TableBank: Table Benchmark for Image-based Table Detection and Recognition", author = "Li, Minghao and Cui, Lei and Huang, Shaohan and Wei, Furu and Zhou, Ming and Li, Zhoujun", booktitle = "Proceedings of the 12th Language Resources and Evaluation Conference", month = may, year = "2020", address = "Marseille, France", publisher = "European Language Resources Association", url = "https://aclanthology.org/2020.lrec-1.236", pages = "1918--1925" } ```