# TabRecSet: A Large-Scale Bilingual Dataset for End-to-End Table Recognition in the Wild > TabRecSet is a large-scale bilingual (English/Chinese) dataset of 38,177 table images sourced from wild scenarios, providing complete polygon annotations for table detection, structure recognition, and content recognition simultaneously. - Authors: Hunter Heidenreich - Published: 2023-03-27 - Updated: 2026-03-30 - Paper: Preprint (arXiv) - Paper: Published (Scientific Data 2023) - Dataset: TabRecSet Dataset - Code: TabRecSet Code and TableMe Tool - Source: Bevaya Labs, --- ## TL;DR TabRecSet is a large-scale, bilingual (English and Chinese) dataset of 38,177 table images collected from multiple real-world scenarios, providing polygon-based spatial annotations for all three table recognition sub-tasks: table detection (TD), table structure recognition (TSR), and table content recognition (TCR). The authors also release TableMe, a custom annotation tool that assisted in building the dataset, and they benchmark several existing models on the new data. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: The primary contribution is a new dataset. The paper describes a careful collection, cleaning, and annotation pipeline to produce a reusable benchmark with specific properties (bilingual, wild scenarios, polygon annotations, end-to-end coverage) not found in any prior dataset. **Secondary:** $\Psi\_{\text{Evaluation}}$: The paper includes a technical validation section that trains or fine-tunes several published models on TabRecSet and reports baseline performance across all three sub-tasks, establishing initial benchmark numbers. ## What is the motivation? Table recognition (TR) breaks down into three interdependent tasks: table detection (TD) to locate tables in an image, table structure recognition (TSR) to recover the spatial and logical cell layout, and table content recognition (TCR) to extract the text within each cell. Tackling all three simultaneously (end-to-end TR) is a practical goal for document automation, yet no existing dataset supports it well. Prior datasets have one or more of three shortcomings. First, many restrict annotations to a single sub-task: WTW (14.5k images, 2021) covers TSR and TCR but not TD; PubTabNet (568k images) covers TSR and TCR but not TD; PubTables-1M covers all three but comes entirely from scanned, axis-aligned PDF documents with no real-world distortion. Second, large-scale datasets published before 2019 rely on programmatically generated annotations from structured sources (LaTeX, PDFs) rather than human annotation of real images, which limits background and layout diversity. Third, most datasets use bounding boxes or quadrilaterals for spatial annotation, which cannot accurately represent distorted or curved table cells common in camera-captured documents. The gap is a human-annotated, multi-scenario, bilingual dataset that covers TD, TSR, and TCR with polygon-level spatial precision for irregular tables. ## What is the novelty? TabRecSet addresses the three identified gaps in a single release: **Scale and bilingual coverage.** The dataset contains 32,072 images and 38,177 table instances: 15,542 images with 20,415 English tables and 16,530 images with 17,762 Chinese tables. An additional 21,228 generated images provide border-incomplete (three-line and no-line) variants. At the time of publication this was approximately 2.6 times larger than the prior largest multi-scenario dataset (WTW at 14,500 images). **Wild scenarios.** Images are collected through reverse-image search engines using seed images of real tables, then filtered manually. This produces samples from documents, Excel spreadsheets, exam papers, financial invoices, ingredient labels, and books, both scanned and camera-captured. As a result, many tables exhibit rotation, inclination, and perspective or warp distortion. **Polygon spatial annotation.** Where most datasets use bounding boxes or quadrilaterals for cell boundaries, TabRecSet annotates both the outer table body and individual cells using polygons. This provides greater precision for distorted cells, and the polygon annotations are shared between the all-line tables and their generated border-incomplete counterparts so that the invisible borders remain annotatable. **Complete end-to-end annotation.** Each sample carries three aligned annotation types stored in a LabelMe-based JSON format: - Table body polygon for TD - Per-cell polygon plus logical properties (row, column, rowspan, colspan) for TSR - Per-cell text content (including hand-written text, with `#` for indistinguishable characters) for TCR The cell label string encodes all four logical properties and text as `----`. **Automated annotation assistance.** The authors develop a TSR auto-annotation algorithm that rectifies irregular tables using Thin-Plate Spline (TPS) transformation to remove curvature distortion and Affine transformation to remove rotation, then computes logical cell properties by analyzing corner-point proximity on the rectified image. This algorithm achieves approximately 80% accuracy, leaving annotators to correct only the remaining 20% manually. Table body polygons are auto-generated from cell polygons using morphological closure and contour tracing. **Border-incomplete generation.** Three-line and no-line table variants are generated by pixel-level edge erasure: the program identifies pixel runs on cell borders from polygon annotations and replaces them with the median background color of a surrounding kernel, effectively removing the visible line without altering the underlying annotation. **TableMe annotation tool.** The team releases a custom annotation tool built on LabelMe that adds table-specific features: group assignment for multi-table images, batch logical property assignment for a row or column of cells at once, and an interactive digital-table view that renders the current annotation as a live spreadsheet. Annotators can type cell content directly into the rendered cells, and structure errors become immediately visible. ## What experiments were performed? The technical validation section trains or fine-tunes four published models on an 80/20 train/test split of TabRecSet and reports results on the held-out test set. No validation set is reported separately. **Topology structure and content recognition (TSR- + TCR).** EDD (the sequence-based HTML prediction model from PubTabNet) is evaluated in two settings: trained on PubTabNet and evaluated zero-shot on TabRecSet, and fine-tuned on the TabRecSet training set. The metric is Tree-Edit-Distance-based Similarity (TEDS), computed either on the HTML structure alone (TEDS-S) or on structure and content jointly (TEDS-All). **Spatial and topological structure recognition (TSR).** TableMaster (sequence-based, predicts HTML and bounding boxes) and TGRNet (graph-based, predicts a table graph) are tested. TableMaster is evaluated with PubTabNet pre-training followed by TabRecSet fine-tuning; the metric is TEDS-S for topology and bounding-box precision for spatial detection. TGRNet is trained from scratch on TabRecSet and evaluated with logical-property classification accuracy and cell detection precision. **Table detection (TD).** CDeC-Net is trained on the TabRecSet training set and evaluated with Average Precision (AP) for table segmentation. Results are summarized in Table 6 of the paper. Selected numbers: | Model | Condition | Metric | Value | |-------|-----------|--------|-------| | EDD | PubTabNet only, zero-shot on TabRecSet | TEDS-S | 72.34% | | EDD | Fine-tuned on TabRecSet | TEDS-S | 90.68% | | EDD | Fine-tuned on TabRecSet | TEDS-All | 70.70% | | EDD | Trained from scratch on TabRecSet | TEDS-S | 51.75% | | TableMaster | Fine-tuned on TabRecSet | TEDS-S | 93.13% | | TableMaster | Fine-tuned on TabRecSet | Cell precision | 11.00% | | TGRNet | Trained from scratch | TSR(-) Acc. | 65.66% | | TGRNet | Trained from scratch | Cell precision | 74.82% | | CDeC-Net | Trained from scratch | AP-Table | 92.80% | The large gap between TEDS-S (93.13%) and cell precision (11.00%) for TableMaster illustrates that regression-based spatial localization fails on distorted tables even when topology is recovered correctly. ## What are the outcomes/conclusions? TabRecSet establishes a benchmark for end-to-end table recognition in multi-scenario, real-world conditions. The results from the validation experiments suggest the following: Fine-tuning on TabRecSet substantially improves model performance relative to zero-shot transfer. EDD gains 18 TEDS-S points (72.34% to 90.68%) after fine-tuning, which indicates that wild-scenario diversity is genuinely different from the scanned-document distribution of PubTabNet. Training from scratch on TabRecSet yields limited performance for sequence-based models. EDD's TEDS-S of 51.75% and TableMaster's TEDS-S of 16.61% without pre-training suggest that the dataset's structural complexity and visual diversity make it harder than prior benchmarks for methods that have not been pre-trained on a simpler distribution first. Spatial cell localization on distorted tables remains an open problem. TableMaster's cell precision of 11.00% after fine-tuning (compared to 93.13% TEDS-S for topology alone) makes clear that recovering the spatial locations of cells in irregular tables is the harder sub-task, and that existing regression-based approaches are insufficient for this. Graph-based TSR models handle the polygon-annotated data reasonably well. TGRNet achieves 74.82% cell precision and 65.66% logical property accuracy when trained from scratch, suggesting that the dataset is usable for this class of methods. The authors do not report an end-to-end TR result because, at the time of writing, no published model chains TD, TSR, and TCR together. The dataset is intended to enable that line of research. **Limitations.** The border-incomplete tables are generated from all-line tables rather than collected from natural sources, so the distribution of these variants may not fully reflect organic three-line or no-line tables found in documents. The data collection relies on Creative Commons-licensed images from web search, and the geographic and domain balance is not fully characterized. Annotation quality is validated by cross-checking and one round of proofreading, but inter-annotator agreement statistics are not reported. ## Reproducibility ### Models No model weights are released as part of this paper; the four validated models (EDD, TableMaster, TGRNet, CDeC-Net) are third-party published systems. The paper reports that training was performed using the official code and pre-trained checkpoints for each model, but links to those checkpoints or exact configuration files are not included in the paper or repository. ### Algorithms The TSR auto-annotation algorithm uses TPS transformation (via a TPS interpolation library) and OpenCV's `findContour` API. The border-incomplete generation algorithm is purely image-based, using a median-filter kernel to erase border pixels. Python source code for both algorithms is available in the GitHub repository. No optimizer, learning rate, or training schedule is reported for the baselines since those use the original training procedures from their respective papers. ### Data - **Total size:** 32,072 images, 38,177 tables (original all-line subset). An additional 21,228 generated images add three-line and no-line variants: 5,113 English (6,728 tables) and 5,501 Chinese (5,911 tables). - **English subset (original):** 15,542 images, 20,415 tables. - **Chinese subset (original):** 16,530 images, 17,762 tables. - **Scenarios:** scanned documents, camera-captured documents, Excel spreadsheets, exam papers, financial invoices, ingredient labels. - **Table types:** all-line (border-complete), three-line, and no-line (border-incomplete); regular and irregular (rotated, distorted, nested, over/under-exposed, hand-written). - **Annotation format:** LabelMe JSON. TD annotation in table-wise JSON files; TSR + TCR annotation in cell-wise JSON files. Images are JPG. All-line originals and their generated border-incomplete counterparts share the same filename and annotation files. - **Public availability:** Dataset hosted on Figshare ([https://doi.org/10.6084/m9.figshare.20647788](https://doi.org/10.6084/m9.figshare.20647788)), licensed CC-BY-SA 4.0. The DOI resolves correctly (verified). The GitHub repository ([https://github.com/MaxKinny/TabRecSet](https://github.com/MaxKinny/TabRecSet)) is independently licensed CC-BY-SA 4.0 (verified). - **Splits:** There is no fixed official train/val/test split in the release. The 80% training / 20% test division used in the paper's technical validation experiments is the authors' own partition for those experiments only. The Usage Notes section of the paper recommends that users mix and re-divide the data themselves according to their task. - **Data collection:** images sourced from web search (Google, Baidu) using Creative Commons license filters or manual source verification; watermarked, privacy-sensitive, and duplicate images removed; samples from datasets without derivative licenses excluded; approximately 30% of raw downloads were filtered out in data cleaning. - **Annotation process:** five qualified annotators (including some authors) were involved. Human annotation was performed with TableMe, assisted by the TSR auto-annotation algorithm (approximately 80% automation rate for logical properties). At each of the four annotation steps, annotators exchanged sub-datasets and cross-checked each other's work. One final proofreading pass by a designated checker reviewed all samples for remaining dirty images and incorrect annotations; wrongly generated border-incomplete tables were deleted rather than corrected. Inter-annotator agreement statistics are not reported. ### Evaluation - **TEDS / TEDS-S:** Tree-Edit-Distance-based Similarity comparing predicted HTML sequences to ground-truth. TEDS-S ignores cell text content; TEDS-All includes it. - **TSR(-) Acc.:** Classification accuracy of logical properties (row, column, rowspan, colspan) per cell. - **P-Cell:** Bounding-box precision for detected cells, following standard object detection precision as defined in Liu et al. (2020, ref. 26 in the paper). The IoU threshold used for cell matching is not specified in the paper. - **AP-Table:** Average Precision for table detection/segmentation, also following ref. 26. The IoU threshold is not specified in the paper. - Baselines are not evaluated under identical compute budgets; some are fine-tuned from PubTabNet pre-trained weights while others are trained from scratch. - No error bars, confidence intervals, or multiple-run statistics are reported. ### Hardware Training hardware, GPU counts, and training duration are not reported. The paper does not include inference latency or memory requirements for any of the evaluated models. ## BibTeX {{< bibtex >}} @article{yang2023large, title={A large-scale dataset for end-to-end table recognition in the wild}, author={Yang, Fan and Hu, Lei and Liu, Xinwu and Huang, Shuangping and Gu, Zhenghui}, journal={Scientific Data}, volume={10}, number={1}, pages={110}, year={2023}, publisher={Nature Publishing Group UK London} } {{< /bibtex >}}