# TabStruct-Net: Top-Down and Bottom-Up Cues for Table Structure Recognition > TabStruct-Net jointly trains a cell detection network (Mask R-CNN with alignment loss) and a graph-based structure recognition network (DGCNN + LSTM) to predict cell bounding boxes and row/column adjacency matrices from table images alone, without any PDF metadata or OCR input. - Authors: Hunter Heidenreich - Published: 2020-10-09 - Updated: 2026-03-24 - Paper: Preprint (arXiv) - Paper: Published (ECCV 2020) - Code: TabStructNet (code) - Source: Bevaya Labs, --- ## TL;DR TabStruct-Net (ECCV 2020) is an end-to-end trainable architecture for table structure recognition that combines a modified Mask R-CNN cell detector with a graph neural network (DGCNN + LSTM) to simultaneously predict cell bounding boxes and row/column adjacency matrices. It introduces an alignment loss that enforces grid-like spatial constraints between detected cells, operates directly on table images without PDF metadata or OCR input, and was evaluated across eight benchmarks. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: the headline contribution is the end-to-end trainable architecture (TabStruct-Net) combining a modified cell detection network and a graph-based structure recognition network, supported by ablation studies. **Secondary:** $\Psi\_{\text{Evaluation}}$: the paper provides a cross-dataset comparison study that unifies results from previously published methods under two evaluation setups (image-only vs. image + ground-truth boxes), which has independent value for understanding the field at the time of publication. ## What is the motivation? At the time of this work, most table structure recognition systems relied on either PDF metadata (character bounding boxes, font information) or OCR outputs to extract layout features before inferring structure. These dependencies create generalization problems: PDF metadata is unavailable for scanned documents, and OCR errors compound for complex or dense tables. Methods that operated on images directly (e.g., image-to-markup models) struggled with large, complex tables. The authors target tables with complex layouts: spanning cells, dense content, no ruling lines, and multi-line cells. The central question is whether visual cues alone, without any textual or metadata input, can support accurate structure recognition. ## What is the novelty? TabStruct-Net treats TSR as a two-stage process operating on image features only: 1. **Top-down cell detection**: Locate every cell as a bounding box. 2. **Bottom-up structure recognition**: Given the detected cells, predict pairwise row and column adjacency relationships to reconstruct the grid. The architecture introduces three specific contributions over the Mask R-CNN + DGCNN baseline: **Modified Feature Pyramid Network.** The standard FPN is extended with both a top-down pathway (propagating high-level semantics to low-resolution feature maps) and a bottom-up pathway that feeds low-level visual features upward. Dilated convolutions replace standard $3 \times 3$ convolutions to expand the receptive field and better capture row- and column-spanning structures. **Alignment loss.** An auxiliary loss is added to the cell detector to explicitly penalize misalignment of cells that share the same row or column. For cells $c\_i$ and $c\_j$ in the same starting row $r \in SR$, the alignment penalty is: $$L\_{align} = \begin{aligned} &\sum\_{r \in SR} \sum\_{c\_i, c\_j \in r} \|y1\_{c\_i} - y1\_{c\_j}\|\_2^2 \\\\ +\, &\sum\_{r \in ER} \sum\_{c\_i, c\_j \in r} \|y2\_{c\_i} - y2\_{c\_j}\|\_2^2 \\\\ +\, &\sum\_{c \in SC} \sum\_{c\_i, c\_j \in c} \|x1\_{c\_i} - x1\_{c\_j}\|\_2^2 \\\\ +\, &\sum\_{c \in EC} \sum\_{c\_i, c\_j \in c} \|x2\_{c\_i} - x2\_{c\_j}\|\_2^2 \end{aligned}$$ where $SR$, $ER$, $SC$, $EC$ denote sets of cells sharing the same starting row, ending row, starting column, and ending column, respectively. This loss acts as a geometric regularizer that makes predicted boxes easier to post-process. **LSTM visual features for structure recognition.** The DGCNN graph network receives per-cell features extracted from the P2 layer of the FPN. Rather than using only the centroid feature (as in the original DGCNN paper), horizontal and vertical line scans of each cell region are fed through LSTMs (depth 128) to produce summary vectors. This captures content distribution across the full height and width of the cell. The overall training loss combines detection losses with the structure recognition loss: $$L = L\_{box} + L\_{cls} + L\_{mask} + L\_{align} + L\_{gnn}$$ where $L\_{box}$, $L\_{cls}$, and $L\_{mask}$ are the standard Mask R-CNN losses; $L\_{align}$ is the alignment regularizer; and $L\_{gnn}$ is the cross-entropy loss from the row/column adjacency classifiers. Structure recognition is formulated as predicting two adjacency matrices $M\_{row}, M\_{col} \in \mathbb{R}^{N\_{cells} \times N\_{cells}}$, where $M\_{row\_{i,j}} = 1$ if cells $i$ and $j$ belong to the same row, and similarly for columns. Pairs are classified by concatenating interaction features (from the DGCNN edge convolutions over 40 nearest neighbors) with the difference between bounding box coordinates. ## What experiments were performed? **Datasets.** Eight datasets are used: SciTSR (12K/3K train/test), SciTSR-COMP (12K/716), ICDAR-2013 (158 test, no official train), ICDAR-2013-partial (124/34), ICDAR-2019 cTDaR archival (600/150), UNLV-partial (446/112), TableBank (145K/1K), and PubTabNet (339K/114K). **Two evaluation setups** are used throughout: - **Setup A (S-A)**: image-only input. Cell detection runs normally. - **Setup B (S-B)**: image + ground-truth cell bounding boxes. The detection component is bypassed; GT boxes are fed to the structure recognition network. This isolates structure recognition quality from detection errors. **Baselines.** DeepDeSRT, TableNet, GraphTSR (the DGCNN paper), SPLERGE, Bi-directional GRU, and Image-to-Text are compared where applicable. **Metrics.** Precision, recall, and F1 at IoU $\geq 0.6$ for physical structure; TEDS and BLEU for logical structure (TableBank, PubTabNet). CIDER and ROUGE are also reported for XML-level evaluation. **Implementation.** ResNet-101 backbone pretrained on MS-COCO, input images scaled to $1536 \times 1536$, batch size 1, TITAN X GPU (12 GB), SGD with lr = 0.001, momentum = 0.9, weight decay = 0.0001. Two-stage training: first stage uses 2,014 anchors and 512 ROIs; second stage uses 3,072 anchors and 2,048 ROIs. ## What are the outcomes/conclusions? **Physical structure (Setup A).** On ICDAR-2013, TabStruct-Net achieves F1 = 0.906, outperforming DeepDeSRT by approximately 27.5 percentage points; DeepDeSRT cannot handle spanning cells because it infers cells purely from row/column intersection. Under equivalent training conditions (no dataset-specific heuristics), TabStruct-Net outperforms SPLERGE by roughly 3 percentage points on ICDAR-2013. On SciTSR, F1 = 0.920; on SciTSR-COMP (the harder split with complex structures), F1 = 0.895. On ICDAR-2019 cTDaR archival, F1 = 0.804 when trained on SciTSR + ICDAR-2019. The gap between S-A and S-B (e.g., F1 0.906 vs. 0.981 on ICDAR-2013) confirms that cell detection quality remains a bottleneck: when GT boxes are given, structure recognition F1 exceeds 0.98 on most benchmarks. **Logical structure.** On TableBank, BLEU = 0.914 (Word) and 0.937 (LaTeX). On PubTabNet, TEDS = 0.901 (training on SciTSR only). **Ablation.** Adding the top-down/bottom-up pathways and alignment loss together improves cell detection F1 by 2.3 points and structure recognition F1 by 2.4 points over the plain Mask R-CNN + DGCNN baseline. Adding LSTM-based visual features contributes a further 2.1 points. End-to-end joint training contributes approximately 1.5 points over training the two networks separately. **Failure mode.** The model fails on tables with many contiguous empty cells: empty regions lack visual gradients that the cell detector relies on, leading to missed detections and cascading structure errors. **Limitations not acknowledged.** The comparison table in S-A vs. S-B reveals that detection errors dominate; no analysis is provided of how detection precision affects downstream TSR quality at the system level. Training is done on SciTSR (scientific PDFs) and transferred to other domains; domain shift to archival or financial tables is observed but not analyzed in depth. The use of Tesseract OCR for post-processing content extraction introduces a dependency the architecture nominally avoids. Inference latency and throughput are not reported. ## Reproducibility ### Models - **Cell detection**: ResNet-101 backbone (pretrained on MS-COCO), Mask R-CNN framework with extended FPN (top-down + bottom-up pathways). Dilated convolutions: RPN uses $2 \times 2$ filter with dilation 2; FPN uses filter-7 dilated convolution blocks. - **Structure recognition**: DGCNN with edge convolution over 40 nearest neighbors, dense layer size 64 with ReLU; LSTM depth 128 for visual feature extraction. Features from P2 layer of the shared FPN. - **Weights**: The code repository is available at https://github.com/sachinraja13/TabStructNet. The repository carries an MIT license inherited from Matterport's Mask R-CNN base; no separate license covers the TabStructNet additions by the authors. Pretrained model weights are not released. - **Parameter count**: Not reported. ### Algorithms - SGD optimizer; learning rate 0.001, momentum 0.9, weight decay 0.0001. - Two-stage training: stage 1 (2,014 anchors, 512 ROIs) then stage 2 (3,072 anchors, 2,048 ROIs). - Anchor scales: 0.5, 1, 2; anchor box sizes: 8, 16, 32, 64, 128. - Monte Carlo sampling during training for class balancing in the adjacency classifiers; all pairs enumerated at test time. - IoU threshold of 0.5 to filter predicted cells for structure recognition training. - Maximum 2,400 vertices per graph. - Ground-truth bounding boxes for datasets with content-level annotations (e.g., ICDAR-2013) are preprocessed to enforce row/column alignment before computing alignment loss. ### Data - Primary training set: SciTSR (12K train split), plus ICDAR-2019 for the archival evaluation. - Marmot Extended (1,016 training images) is listed in the dataset table but specific results are not reported in the main paper. - All datasets used (SciTSR, ICDAR-2013, ICDAR-2019, UNLV, TableBank, PubTabNet) are publicly available with varying licenses; SciTSR is MIT, TableBank is research-only. - No data augmentation details are reported. ### Evaluation - IoU threshold: 0.6 for confusion matrix computation (consistent across all datasets). - Evaluation excludes empty cells (consistent with ICDAR-2013 practice). - TEDS metric used as defined in the PubTabNet paper; BLEU as in the TableBank paper. - No error bars, confidence intervals, or multiple-run statistics are reported. - The two-setup evaluation (S-A vs. S-B) provides a useful isolation of detection vs. structure recognition, but direct comparisons against methods that use additional inputs (GT boxes, PDF metadata) are not always fair; the paper acknowledges this. ### Hardware - Training: 1x NVIDIA TITAN X GPU (12 GB VRAM), batch size 1. - No training time or total compute hours reported. - Inference speed not reported. ## BibTeX ```bibtex @inproceedings{raja2020tabstructnet, author = {Sachin Raja and Ajoy Mondal and C. V. Jawahar}, editor = {Andrea Vedaldi and Horst Bischof and Thomas Brox and Jan-Michael Frahm}, title = {Table Structure Recognition Using Top-Down and Bottom-Up Cues}, booktitle = {Computer Vision -- ECCV 2020, Part XXVIII}, series = {Lecture Notes in Computer Science}, pages = {70--86}, publisher = {Springer}, year = {2020}, doi = {10.1007/978-3-030-58604-1_5}, } ```