# TableNet: End-to-End Joint Table Detection and Structure Recognition > TableNet is a multi-task encoder-decoder architecture built on VGG-19 that jointly segments table and column regions in scanned document images, followed by rule-based row extraction for full cell-level data extraction. - Authors: Hunter Heidenreich - Published: 2020-01-06 - Updated: 2026-03-24 - Paper: Preprint (arXiv) - Paper: Published (ICDAR 2019) - Dataset: Marmot Extended - Source: Bevaya Labs, --- ## TL;DR TableNet proposes a multi-task encoder-decoder architecture based on VGG-19 that simultaneously predicts table and column regions in scanned document images through two separate decoder branches trained jointly. Rule-based post-processing derives row boundaries from OCR word positions and the predicted column masks. Experiments on ICDAR 2013 show results broadly comparable to DeepDeSRT. The authors also release Marmot Extended, a column-annotated version of the Marmot dataset. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The central contribution is the TableNet architecture, a multi-task FCN that couples table detection and column segmentation within a single shared encoder. The paper focuses on architectural design choices, training schedules, and baseline comparisons on ICDAR 2013. **Secondary:** $\Psi\_{\text{Resource}}$: The authors manually annotated column bounding boxes for 509 English documents from the Marmot dataset and release this augmented split (Marmot Extended) to the community. ## What is the motivation? Most approaches to extracting structured data from scanned documents split the pipeline into two independent problems: table detection (locating the table region) and table structure recognition (segmenting rows and columns within it). Prior deep learning methods such as DeepDeSRT [Schreiber et al., 2017] train separate models on separate datasets for each sub-problem. This independence is at odds with the intuition that tables and their columns share spatial context: column regions are subsets of table regions, so evidence for columns should reinforce the detection of the enclosing table and vice versa. The authors argue that coupling both tasks in a single model can improve accuracy through shared low-level features and mutually reinforcing gradients. A secondary motivation is data scarcity. Few large, publicly available datasets provide both table detection and structure annotations. The largest at the time, Marmot, had only table-level bounding boxes, not column-level annotations, limiting the ability to train structure-aware models directly. ## What is the novelty? TableNet adapts the Fully Convolutional Network (FCN) architecture from Long et al. (2014) to a dual-task setting. The encoder is the convolutional stack of VGG-19 (conv1 through pool5), pretrained on ImageNet. The fully connected layers of VGG-19 are discarded and replaced with a pair of $1 \times 1$ convolutional layers (conv6) followed by dropout (rate 0.8). After this shared bottleneck, two independent decoder branches diverge: - **Table branch**: a $1 \times 1$ conv layer (conv7\_table) followed by fractionally strided (transposed) convolutions that upsample the feature map back to the original image resolution, with skip connections from pool4 and pool3. - **Column branch**: a $1 \times 1$ conv layer (conv7\_column) with ReLU and dropout, then another $1 \times 1$ conv (conv8\_column), followed by the same upsampling scheme with pool4 and pool3 skip connections. Each branch produces a binary pixel-wise prediction mask. The joint training objective is: $$\mathcal{L} = \mathcal{L}\_{\text{table}} + \mathcal{L}\_{\text{col}}$$ where both $\mathcal{L}\_{\text{table}}$ and $\mathcal{L}\_{\text{col}}$ are pixel-wise binary cross-entropy losses over the table and column segmentation masks respectively. The shared encoder receives gradient updates from both branches simultaneously. An optional semantic feature augmentation step color-codes word bounding boxes by inferred data type (e.g., numeric strings get one color, alphabetic strings another). These color-coded overlays are pixel-wise added to the original image before feeding to the network. The authors report that this provides modest performance gains. Row boundaries are not predicted by the model. Instead, after column masks are extracted, a rule-based system uses OCR word positions (from Tesseract) and three heuristics: Radon-transform detection of horizontal line demarcations, multi-line row detection based on column fill patterns, and single-line assignment when no demarcations are present. ## What experiments were performed? **Datasets used:** - *Training:* 509 English documents from the Marmot dataset, with manually added column annotations (Marmot Extended). - *Evaluation:* ICDAR 2013 table competition dataset, tested on 34 documents (the same split used by DeepDeSRT). - *Fine-tuning (Experiment 3):* The Marmot-trained model is fine-tuned on the ICDAR 2013 train split to match DeepDeSRT's experimental setup. **Baselines:** DeepDeSRT [Schreiber et al., ICDAR 2017] and Tran et al. [2015]. **Metrics:** - *Table detection:* Precision, Recall, and F1 based on completeness and purity over character-level sub-objects. - *Table structure recognition and data extraction:* Precision, Recall, and F1 over cell adjacency relations (horizontal and vertical neighbors), with normalized cell text. **Training schedule:** Initial 500 iterations train table and column branches in a 2:1 ratio (batch size 2); once both losses are comparable, training switches to a 1:1 ratio for 5,000 total iterations. Optimizer: Adam ($\beta\_1 = 0.9$, $\beta\_2 = 0.999$, $\varepsilon = 10^{-8}$), learning rate $10^{-4}$. Threshold for pixel-wise prediction at inference: 0.99. ## What are the outcomes/conclusions? Results on ICDAR 2013 are as follows. **Table detection (Table I):** | Model | Recall | Precision | F1 | |---|---|---|---| | TableNet + Sem. (fine-tuned) | 0.9628 | 0.9697 | 0.9662 | | TableNet + Sem. | 0.9621 | 0.9547 | 0.9583 | | TableNet | 0.9501 | 0.9547 | 0.9547 | | DeepDeSRT | 0.9615 | 0.9740 | 0.9677 | | Tran et al. | 0.9636 | 0.9521 | 0.9578 | **Table structure recognition and data extraction (Table II):** | Model | Recall | Precision | F1 | |---|---|---|---| | TableNet + Sem. (fine-tuned) | 0.9001 | 0.9307 | 0.9151 | | TableNet + Sem. | 0.8994 | 0.9255 | 0.9122 | | TableNet | 0.8987 | 0.9215 | 0.9098 | | DeepDeSRT | 0.8736 | 0.9593 | 0.9144 | TableNet with semantic features and fine-tuning achieves F1 of 0.9662 on table detection and 0.9151 on structure recognition, compared to DeepDeSRT's 0.9677 and 0.9144 respectively. The authors characterize the results as "comparable" rather than clearly superior. The semantic feature augmentation provides a small, consistent improvement across all variants. Fine-tuning on ICDAR domain data provides a further modest boost. Inference time is reported as 0.3765 seconds per document image on the test hardware; no comparison to DeepDeSRT is possible since that model was not publicly released. The authors acknowledge that the row detection step is entirely rule-based, and plan to add a third model branch for row segmentation in future work. They also note that more abstract semantic types (e.g., currency, city, country) might further improve performance. Notable limitations not fully discussed by the authors: (1) the evaluation covers only ICDAR 2013, a relatively small benchmark of 34 test images; (2) the comparison with DeepDeSRT is on a dataset (ICDAR) that DeepDeSRT was explicitly trained on, while TableNet is fine-tuned on it, making the comparison partially asymmetric; (3) the precision of the VGG-19 base means the model is not lightweight; inference latency and memory requirements at training time are not fully detailed. ## Reproducibility ### Models - **Architecture:** VGG-19 encoder (conv1 through pool5) plus two $1 \times 1$ conv layers (conv6 with dropout), two decoder branches (table and column), each using transposed convolutions with pool4/pool3 skip connections. No parameter count reported. - **Pretrained weights:** VGG-19 initialized from ImageNet pretraining (via standard sources, e.g., Keras/TensorFlow model zoo). - **Initialization/freezing:** The paper does not state whether the VGG-19 encoder weights are frozen or updated during joint training. The standard FCN practice is to fine-tune the full network end-to-end, but this is not confirmed in the paper. - **Released weights:** No model weights are released. ### Algorithms - **Optimizer:** Adam; $\beta\_1 = 0.9$, $\beta\_2 = 0.999$, $\varepsilon = 10^{-8}$; learning rate $10^{-4}$. - **Batch size:** 2. - **Training schedule:** 500 initial iterations at 2:1 (table:column) branch ratio, then 1:1 ratio to 5000 total iterations for Marmot pre-training; 3000 fine-tuning iterations at 1:1 on ICDAR 2013. - **Loss functions:** Pixel-wise binary cross-entropy for each branch ($\mathcal{L}\_{\text{table}}$ and $\mathcal{L}\_{\text{col}}$), summed with equal weight. No auxiliary losses. - **Training tricks:** No warmup, gradient clipping, mixed precision, EMA, or curriculum learning are mentioned. - **Preprocessing:** Images resized to $1024 \times 1024$; histogram equalization applied; Tesseract OCR used to extract word patches; optional color-coding by data type (pixel-wise added to original image). - **Inference threshold:** 0.99 for pixel-wise table/column prediction. - **Framework:** TensorFlow (version not specified). - **Code:** Not publicly released. ### Data - **Training:** 509 English documents from Marmot Extended (column annotations added by the authors). The Marmot dataset contains 1,016 total documents (Chinese and English); only the English portion with manually added annotations was used. - **Validation split:** No held-out validation set is mentioned; the paper uses Marmot Extended for training and ICDAR 2013 for evaluation only. - **Evaluation:** ICDAR 2013 table competition dataset; 34-image test split matching DeepDeSRT's setup. - **Marmot Extended availability:** Released via Google Drive (see frontmatter artifacts); license not specified. - **Annotation process:** Column bounding boxes were added manually by the authors for the Marmot English split; no inter-annotator agreement or quality details are reported. ### Evaluation - **Table detection metric:** Completeness/purity-based Precision, Recall, F1 over character-level sub-objects within each document region (standard ICDAR 2013 protocol). - **Structure recognition metric:** Precision, Recall, F1 over cell adjacency relations with normalized text content. - **Statistical rigor:** No error bars, confidence intervals, or multi-seed runs are reported. Results are averaged across documents but individual variances are not disclosed. - **Limitations acknowledged:** Row detection is rule-based, not learned. Fine-tuning comparison with DeepDeSRT is noted as not fully controlled (different original training sets). DeepDeSRT weights not publicly available for runtime comparison. - **Liberties taken:** Evaluation is restricted to ICDAR 2013 (34 test images), a small benchmark that limits generalizability claims. The model requires Tesseract OCR as an external dependency for both semantic augmentation and row extraction, meaning end-to-end performance depends on OCR quality, which is not ablated. ### Hardware - **Training hardware:** Intel Xeon Silver CPU (32 cores), 128 GB RAM, Tesla V100-PCIE GPU (reported as 6 GB GPU memory, though this may reflect available/allocated memory rather than total VRAM). - **Total GPU-hours:** Not reported. - **Inference time:** ~0.3765 seconds per document image. - **Cost/energy:** Not reported. - **Local deployment:** Feasible in principle given the VGG-19 base, but model weights are not released and memory requirements at training time are not fully characterized. ## BibTeX ```bibtex @inproceedings{paliwal2019tablenet, title={{TableNet}: Deep Learning Model for End-to-end Table Detection and Tabular Data Extraction from Scanned Document Images}, author={Paliwal, Shubham and D, Vishwanath and Rahul, Rohit and Sharma, Monika and Vig, Lovekesh}, booktitle={2019 International Conference on Document Analysis and Recognition (ICDAR)}, year={2019}, organization={IEEE}, doi={10.1109/ICDAR.2019.00029} } ```