# WikiDT: Visual-based Table Recognition and Question Answering Dataset > WikiDT is a large-scale TableVQA dataset built from Wikipedia pages, providing 70,652 QA pairs, 159,905 table annotations, and hierarchical intermediate labels for table detection, structure recognition, and retrieval. - Authors: Hunter Heidenreich - Published: 2024-07-01 - Updated: 2026-03-26 - Paper: Published (ICDAR 2024) - Dataset: WikiDT - Source: Bevaya Labs, --- ## TL;DR WikiDT is a large-scale document TableVQA dataset built from Wikipedia pages. It contains 16,887 full-page images (54,032 sub-pages after pagination), 159,905 table annotations, and 70,652 question-answer pairs. A key design feature is the layered set of intermediate labels covering table detection (TD), table structure recognition (TSR), table retrieval, and SQL-form queries, enabling sub-task diagnosis and modular training. Baseline experiments with T5, LaTr, and TAPAS show that all tasks remain substantially unsolved, with the best overall TableVQA accuracy reaching only 45.23%. --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: the primary contribution is the dataset itself, including its automated curation pipeline, multi-level annotation schema, and release on HuggingFace. **Secondary:** $\Psi\_{\text{Evaluation}}$: baseline experiments across four sub-tasks establish difficulty benchmarks and surface model failure modes, particularly around multi-step reasoning and long table contexts. --- ## What is the motivation? Intelligent document processing is a growing commercial area, with the global IDP market projected to grow from $1.1B in 2022 to $5.2B by 2027. Table extraction and table-based QA sit at the center of many real-world workflows: insurance claim parsing, income report processing, and financial data extraction all require systems that can locate, recognize, and reason over tabular content. Existing document VQA datasets fall short in several ways: - **Extractive-only questions:** Most benchmarks emphasize directly span-extractable answers. Fewer than 2% of InfographicVQA samples require genuine reasoning or synthesis. - **Single-table context:** Datasets like DUE (DocumentUnderstanding Benchmark) crop images to contain only the target table, making table recognition trivial and the task less realistic. - **Missing intermediate labels:** Models trained on end-to-end supervision cannot be easily diagnosed; it is unclear whether errors stem from table detection, retrieval, or the QA component itself. - **Scale and diversity limitations:** Prior table recognition benchmarks (ICDAR-2013 at 450 samples, ICDAR-2019 at 3.2k) are small, and large ones (PubTables-1M from scientific PDFs) come from a narrow visual domain. WikiDT targets all of these gaps by providing full-page web screenshots with diverse, multi-table layouts, complex reasoning questions, and a full chain of intermediate annotations. --- ## What is the novelty? ### Dataset scale and composition WikiDT is assembled from Wikipedia pages referenced by three existing TableQA datasets: WikiTableQuestions, TabFact, and WikiSQL. The dataset provides: - 16,887 full-page images rendered as web screenshots at 1600px width - 54,032 sub-page images after a dynamic-height pagination step - 159,905 table annotations (web-derived ground truth + AWS Textract output) - 70,652 QA pairs with table retrieval labels - ~49,000 SQL annotations linking questions to executable queries The table content spans a wide domain: sports records, historical events, political data, geography, and more. ### Hierarchical annotation schema Each QA sample is paired with a full chain of intermediate labels: 1. **Table detection (TD):** Bounding boxes for every visible table in the sub-page image (WikiDT-detection, 54,032 images). 2. **Table structure recognition (TSR):** Row, column, cell (including merged cells), and header bounding boxes extracted from HTML rendering, stored in Pascal VOC format (WikiDT-structure, 159,905 table crops). 3. **Table retrieval:** For each question, the label identifies which table on the page contains the answer. 4. **TableVQA:** End-to-end QA requiring multi-step reasoning over the full-page image with OCR assistance. 5. **SQL annotation:** Executable SQL queries covering ~70% of samples, derived from WikiSQL and WikiTableQuestions. ### Automated, heuristic-verified construction The pipeline renders Wikipedia pages with Puppeteer, extracts table bounding boxes directly from HTML DOM elements, and connects them back to existing QA pairs. Because web content drifted after original dataset creation, the authors recover historical page versions from Wikipedia's edit history and re-execute WikiSQL queries against the updated table content to regenerate correct answers. Manual quality inspections supplement the automated checks. ### Challenging table characteristics Compared to PDF-derived datasets like PubTables-1M, WikiDT tables appear at arbitrary positions and widths across the page (the bounding box distribution is nearly uniform rather than concentrated at a few column positions). Many pages contain 10 or more tables per sub-page (mean 3.4, max over 40), and tables can span beyond a single paginated segment. --- ## What experiments were performed? ### Sub-task formulations The paper evaluates four tasks: | Task | Input | Output | | :--- | :--- | :--- | | Table Detection | sub-page image | table bounding boxes | | Table Structure Recognition | table crop image | row/column/cell bounding boxes | | Table Retrieval | tables + question | target table identifier | | TableVQA | full-page image + question | answer string(s) | ### TableVQA Three models were evaluated using denotation accuracy (exact match, ignoring number formatting): | Model | Single Answer (%) | Multi-Answer (%) | Overall (%) | | :--- | :--- | :--- | :--- | | T5 (text-only, spatial-blind) | 32.74 | 1.30 | 31.67 | | LaTr (layout-aware visual) | 35.29 | 0.0 | 34.08 | | TAPAS (modularized, with ground-truth retrieval) | 46.24 | 6.86 | 45.23 | TAPAS receives table structure information from Textract and ground-truth retrieval labels, making it the most favorable setup. All models struggle badly on multi-entry answers. An ablation comparing Textract table annotations vs. web (ground truth) table annotations shows the improvement from better table recognition is modest for monolithic models (T5, LaTr), whereas model size has a much larger impact. For TAPAS, table recognition quality matters more. ### Table Extraction Pre-trained models from other domains transfer poorly to WikiDT: - TableNet (trained on Marmot) and CascadeTabNet (trained on ICDAR-19) show substantially degraded F1 compared to their home benchmarks, confirming domain shift. - DETR pre-trained on PubTables-1M achieves AP$\_{50}$ well below the WikiDT-trained DETR, illustrating that PubTables-1M's narrow PDF layout distribution does not generalize to web pages. At IoU=0.5, Textract achieves 0.901 precision, 0.558 recall, and F1=0.689 on the WikiDT test set, leaving considerable room for improvement especially on recall. ### Table Retrieval BM25 and a dense retrieval model (TAPAS backbone with a learned $R\_{CLS}$ token) are compared on Mean Reciprocal Rank (MRR): | Method | Web Tables (MRR) | Textract Tables (MRR) | | :--- | :--- | :--- | | BM25 | 0.382 | 0.389 | | Dense Retrieval | 0.587 | 0.524 | Dense retrieval substantially outperforms BM25. Table recognition quality has only a marginal effect on retrieval. --- ## What are the outcomes/conclusions? WikiDT surfaces challenges that remain unsolved by models performing well on other VQA benchmarks. The key findings are: - All four evaluated sub-tasks have substantial performance headroom, with the best overall TableVQA accuracy at 45.23%. - Multi-answer questions are particularly difficult; TAPAS achieves only 6.86% denotation accuracy on them. - Multi-step reasoning (e.g., "count in one group, count in another, then compare") consistently defeats TAPAS, which can handle at most one aggregation operation. - Web-page table layouts are substantially more diverse than PDF-derived layouts, causing significant domain transfer failures for models trained on existing benchmarks. - Improving table recognition quality helps modularized systems more than end-to-end models; for end-to-end models, scaling model capacity has a larger effect. The paper notes geographic and cultural bias in the dataset toward content from the United States and Canada, and acknowledges that Wikipedia's inherent misinformation and outdated data carry over into WikiDT. --- ## Reproducibility ### Models - **T5** (base and large): standard text-to-text transformer; spatially blind; inputs are linearized table tokens in row-column order. - **LaTr** (Layout-aware Transformer): pre-trained on scene-text and document VQA tasks; takes image patches, OCR tokens with layout coordinates, and question as input; generates answer autoregressively. - **TAPAS**: pre-trained table parser; takes flattened table, column names, and question; predicts aggregation operator and cell selection; cannot compose more than one aggregation. - **DETR**: transformer-based object detector trained on WikiDT for table detection and structure recognition. All models fine-tuned from publicly released pre-trained checkpoints on Hugging Face, except LaTr (fine-tuned by the original LaTr authors). ### Algorithms - TAPAS and T5 fine-tuned using standard Hugging Face training procedures; hyperparameters not explicitly stated in the paper. - Dense retrieval model trained with weighted binary cross-entropy to address label imbalance; each background sample receives weight $1 / |target\_sample| / |background\_sample|$ to counteract the class imbalance. - DETR trained from random initialization on WikiDT; configuration identical to the PubTables-1M-pre-trained version. ### Data - **Source:** Wikipedia pages from WikiTableQuestions, TabFact, and WikiSQL URL lists. - **Rendering:** Puppeteer at 1600px width; historical page versions recovered via Wikipedia edit history for TabFact URLs. - **Pagination:** Dynamic-height segmentation using blank-line detection (window W=1200px, H=10px); segments to near-1:1 aspect ratio. - **Table annotation:** Extracted directly from HTML DOM via Puppeteer; filtered to visible elements with at least two rows and two columns; nested outer tables removed. - **OCR:** AWS Textract applied to sub-page images; footnotes indicate annotations predate merged-cell support added in March 2022. - **QA generation:** WikiTableQuestions answers reused as-is; WikiSQL answers regenerated by executing translated SQL on the retrieved web table (due to content drift). - **SQL annotation:** WikiSQL queries translated and validated; coverage approximately 70% of QA samples. - **Dataset availability:** Released on HuggingFace at [AmazonScience/WikiDT](https://huggingface.co/datasets/AmazonScience/WikiDT); license not stated in the paper. - **Data split:** Train/dev/test splits follow standard practice; dev and test sets exclude pages with only one table for the retrieval task. ### Evaluation - **TableVQA metric:** Denotation accuracy. A prediction is correct if it matches the ground truth in number of entries and each entry can be non-repeatedly matched (format-insensitive, e.g., "1000" matches "1,000"). - **Table extraction metrics:** IoU-based AP (AP$\_{50}$, AP$\_{75}$, AP$\_{95}$) and AR following MS-COCO convention; pixel-level precision/recall/F1 for TableNet (mask outputs). - **Table retrieval metric:** Mean Reciprocal Rank (MRR) on the subset of samples with multiple table candidates. - Multi-answer denotation accuracy is evaluated strictly: all entries must match, regardless of order. - No error bars or significance tests reported; single-run results. ### Hardware - Training hardware, GPU configuration, and compute time are not reported in the paper. - AWS Textract was used as an external service for OCR and table extraction annotations. --- ## BibTeX ```bibtex @inproceedings{shi2024wikidt, title={WikiDT: Visual-based Table Recognition and Question Answering Dataset}, author={Shi, Hui and Xie, Yusheng and Goncalves, Luis and Gao, Sicun and Zhao, Jishen}, booktitle={International Conference on Document Analysis and Recognition (ICDAR)}, pages={401--418}, year={2024}, publisher={Springer}, doi={10.1007/978-3-031-70533-5_24} } ```