# GTE: Global Table Extractor and the FinTabNet Dataset > A research note on the GTE framework for joint table detection and cell structure recognition, and the FinTabNet dataset of 113k financial tables from SEC annual reports. - Authors: Hunter Heidenreich - Published: 2020-05-01 - Updated: 2026-03-16 - Paper: Preprint (arXiv) - Paper: Published (WACV 2021) - Dataset: FinTabNet Dataset (FinTabNet.c, canonicalized) - Source: Bevaya Labs, --- ## TL;DR GTE is a vision-guided framework for joint table detection and cell structure recognition that introduces two innovations: a constraint loss coupling cell detection to table detection during training, and a hierarchical cell network that classifies table style before routing to a specialized cell detector. The paper also introduces FinTabNet, a dataset of ~113k financial tables from S&P 500 annual reports with HTML annotations, filling the gap between scientific-paper datasets (PubTabNet) and real-world financial documents. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$ The headline contribution is the GTE framework: the constraint-guided table detector, the hierarchical cell detection network, and the cluster-based cell-to-structure algorithm. The bulk of the paper is devoted to architecture design, ablations, and competitive-baseline comparisons on ICDAR 2013 and 2019. **Secondary:** $\Psi\_{\text{Resource}}$ The paper introduces FinTabNet, a large-scale dataset of financial tables from SEC 10-K filings with HTML cell-level annotations, as a complement to the scientific-paper-focused PubTabNet. ## What is the motivation? Table extraction from PDF and image documents requires solving two coupled problems: locating table regions on a page, and recovering the logical cell grid within each table. Prior approaches treat these as independent tasks. The authors identify two specific weaknesses in prior detection-based work: 1. **Coupling failure:** Table and cell detection are trained independently, ignoring the natural geometric constraint that cells must lie inside tables and tables must contain cells. 2. **Style blindness:** Object detection models attend to local regions and miss the global ruling-line style of a table, causing performance degradation when mixing tables with and without graphical lines in training. Additionally, existing large-scale TSR datasets (PubTabNet, TableBank) draw exclusively from scientific articles and are not representative of financial documents, which have denser and more complex multi-level header structures. ## What is the novelty? ### GTE-Table: Constraint-guided detection The table detector is a RetinaNet with a ResNet-50-FPN backbone. The key innovation is a **constraint loss** (CL) that explicitly enforces the table-cell spatial relationship during training. Two boolean operators over predicted cell masks form the building blocks. Let $\text{SLC}(M(B), b)$ denote the area of overlap between cell mask $M(B)$ and box $b$, and $A(b)$ denote the area of $b$. Then: $$ C(b\_{in}, b\_{out}) = \bigl[\text{SLC}(M(B\_{cells}), b\_{out}) - \text{SLC}(M(B\_{cells}), b\_{in})\bigr] < \alpha \cdot \bigl[A(b\_{out}) - A(b\_{in})\bigr] $$ $$ D(b\_{in}, b\_{out}) = \bigl[\text{SLC}(M(B\_{cells}), b\_{out}) - \text{SLC}(M(B\_{cells}), b\_{in})\bigr] > 0 $$ $C$ is true when the cell coverage in the annular region between $b\_{in}$ and $b\_{out}$ falls below a fraction $\alpha$; $D$ is true when any cells exist in that region. The penalty indicator for a candidate table box $b\_{tbl}$ combines four geometric checks: $$ I(b\_{tbl}) = C(\mathbf{0}, b\_{tbl}) \;\lor\; C(S(b\_{tbl}, \mu\_1), b\_{tbl}) \;\lor\; D(S(b\_{tbl}, \mu\_2), S(b\_{tbl}, \mu\_3)) \;\lor\; C(U(b\_{tbl}, \mu\_4), b\_{tbl}) $$ where $S(b, \mu)$ expands box $b$ by $\mu$ pixels on all sides and $U(b, \mu)$ expands only the bottom edge. The constraint loss is: $$ \mathcal{L}\_{\text{CL}} = \sum\_{b\_{tbl} \in B\_{tbl}} I(b\_{tbl}) P(b\_{tbl}) + \gamma\_1 (1 - I(b\_{tbl}))(1 - P(b\_{tbl})) $$ where $P(\cdot)$ is the detection probability and $\gamma\_1 = 0.1$ down-weights the false-negative penalty. One channel of the table detector input is also replaced with a mask from the cell detector output to directly propagate cell evidence into table training. At inference, standard non-maximum suppression is replaced with a **Constraint Coefficient** (CCoef) ranking. For each candidate box, CCoef measures cells just outside minus cells just inside: $$ \text{CCoef}(b\_{tbl}) = \text{SLC}(M(B\_{cell}), S(b\_{tbl}, \mu\_5)) - \text{SLC}(M(B\_{cell}), b\_{tbl}) - \gamma\_2 \cdot \bigl[\text{SLC}(M(B\_{cell}), b\_{tbl}) - \text{SLC}(M(B\_{cell}), S(b\_{tbl}, \mu\_6))\bigr] $$ For overlapping pairs where detection probabilities are close ($|P(b\_i) - P(b\_j)| < \epsilon$), the box with the higher CCoef (more cells outside, fewer inside) is discarded. ### GTE-Cell: Hierarchical style-aware detection A three-stage hierarchy: 1. **Attribute network:** A ResNet-50 binary classifier determines whether a table has vertical graphical ruling lines. This routes the table to one of two specialized cell detectors. 2. **Specialized cell detectors:** Two RetinaNet models trained with different augmentation schemes. The "graphical lines" model trains on original images plus augmentations that add full boundaries at every row/column. The "no lines" model additionally trains on augmentations that erase existing graphical lines. 3. **Cluster-based structure recovery:** Cell bounding boxes are converted to a logical grid via K-means clustering on cell coordinates to identify row and column locations, followed by heuristic post-processing (text-line alignment, spanning cell expansion based on text capitalization, leftover text box assignment). ### FinTabNet dataset FinTabNet uses the same automated annotation pipeline as PubTabNet: pixel-level matching between PDF-rendered pages and XML markup from source documents. The source is annual reports of S&P 500 companies (SEC 10-K filings), chosen to represent complex financial table structures with multi-level headers, spanning cells, and dense numeric content that are absent from scientific paper datasets. The annotation includes pixel-coordinate cell bounding boxes, logical row/column span, and cell text content as HTML. At the time of submission (arXiv v2, December 2020), release was pending legal review; the dataset was subsequently published on IBM's Data Asset eXchange. ## What experiments were performed? ### Datasets - **ICDAR 2013:** 96/156 train/test tables from EU and US government reports. Primary benchmark. - **ICDAR 2019:** Modern and archival document tracks. - **PubTabNet (val):** Scientific tables; used to compare TEDS without access to the test set. - **FinTabNet:** Financial tables from annual reports; models tested with and without fine-tuning. ### Metrics - **Table detection (ICDAR 2013):** Character-level Recall, Precision, F1 (official competition script); also Purity (Pu) and Completeness (Cpt) counts, defined over a set $N$ of test documents as: $$ Pu = \sum\_{n \in N} [\text{Rec}(n)], \qquad Cpt = \sum\_{n \in N} [\text{Prec}(n)] $$ - **Cell structure recognition (ICDAR 2013):** Adjacency relation Precision, Recall, F1. - **TSR (PubTabNet / FinTabNet):** TEDS (Tree-Edit-Distance Similarity, introduced in PubTabNet, arXiv:1911.10683), computed on the validation set PDF images (not the withheld test set). GTE modifies the evaluation script to ignore bold/italic styling tags. ### Training setup - RetinaNet with ResNet-50-FPN, pretrained on MS COCO. - Table network pretrained on TableBank + PubTabNet boundaries, then fine-tuned on ICDAR 2013 train. - Cell network pretrained on PubTabNet, then fine-tuned on ICDAR 2013 train. - Table input resolution: $900 \times 643$; cell input resolution: $965 \times 1350$ (higher resolution needed for small, dense cells). - Attribute (style) classifier: ResNet-50 with binary head, pretrained on SD-Tables dataset attributes, fine-tuned on ICDAR 2013. ### Key results **Table detection on ICDAR 2013:** | Method | Input | Recall | Precision | F1 | |:---|:---|:---:|:---:|:---:| | FineReader | PDF | 99.71 | 97.29 | 98.48 | | Nurminen | PDF | 90.77 | 92.10 | 91.43 | | TableBank | Image | / | / | 96.25 | | GTE | Image | **99.77** | **98.97** | **99.31** | **Cell structure on ICDAR 2013 (without GT border):** | Method | Recall | Precision | F1 | |:---|:---:|:---:|:---:| | Nurminen | 80.78 | 86.93 | 83.74 | | GTE | **92.72** | **94.41** | **93.50** | **PubTabNet and FinTabNet (Table F1 and TEDS, validation set):** | Dataset | Method | Finetuned? | Table F1 | TEDS | |:---|:---|:---:|:---:|:---:| | PubTabNet | GTE | Y | N/A | 93.01 | | FinTabNet | Det-Base | N | 81.17 | 41.57 | | FinTabNet | GTE | N | 89.97 | 87.14 | | FinTabNet | GTE | Y | **95.29** | **91.02** | The Table F1 values show that GTE improves table detection on FinTabNet even without fine-tuning (89.97 vs. 81.17), with a further gain after fine-tuning. TEDS without fine-tuning (87.14) already far exceeds the detection-base structure result (41.57). ## What are the outcomes/conclusions? The authors report a measurable improvement of approximately 3.6 F1 points from the constraint-guided training loop over the version without inter-network information (GTE-Table-Sep). The hierarchical cell style classifier improves cell structure F1 over mixed-style training. On ICDAR 2013, GTE achieves the highest F1 for both table detection (99.31) and cell structure recognition (93.50 without GT border, 96.24 with GT border) compared to published methods at the time of submission. On FinTabNet, the large gap between the detection-base (41.57 TEDS) and GTE without fine-tuning (87.14 TEDS) results suggest that the cell detection architecture matters significantly in the complex financial document domain. ### Limitations - **Anchor-based cell detection:** Very wide cells with extreme aspect ratios and very long lines of text are often detected with boxes that are too short. The anchor configuration partially compensates, but the fundamental limitation of fixed-aspect anchors remains. - **Style classifier errors:** The attribute network achieves 78.84% accuracy on ICDAR 2013 (123/156 tables). Failures occur on very small tables or tables with header-only vertical graphical lines, which are genuinely ambiguous in the training data. The paper uses a fallback strategy based on sampling standard deviation to mitigate this. - **Cluster algorithm sensitivity:** The post-processing structure recovery relies on heuristics (capitalization-based merge, text-line alignment) that may fail on languages or formatting conventions not represented in training. - **PubTabNet TEDS comparability:** GTE reports TEDS of 93.01 on PubTabNet validation using PDF images and a modified evaluation script that ignores bold/italic styling tags. This is not directly comparable to the original EDD score of 88.38, which used rasterized table images and includes styling in the metric. - **FinTabNet statistics not in the paper:** The paper does not report table counts or split sizes. The commonly cited ~113k figure comes from the IBM Data Asset eXchange release metadata, not from the paper itself. The original IBM distribution channel has since been decommissioned. ## Reproducibility ### Models - **GTE-Table:** RetinaNet with ResNet-50-FPN. Modified anchor configuration: aspect ratios {0.1, 0.25} added per feature map for wide tables. Input: $900 \times 643$. - **GTE-Cell:** RetinaNet with ResNet-50-FPN. Uses pyramid levels P3 and P5 (P4 skipped). Anchor sizes {0.5, 0.7, 1, 1.2, 1.6} times standard. Input: $965 \times 1350$. - **Attribute classifier:** ResNet-50 with binary linear head. - No model weights are publicly released. ### Algorithms - All detection models initialized from MS COCO pretrained weights. - Constraint loss hyperparameters: $\mu\_1 = -5$, $\mu\_2 = 5$, $\mu\_3 = 10$, $\mu\_4 = -10$, $\alpha = 1/8$, $\gamma\_1 = 0.1$. - Inference ranking hyperparameters: $\mu\_5 = -20$, $\mu\_6 = \{0.25(x\_2 - x\_1),\, 0.25(y\_2 - y\_1)\}$, $\gamma\_2 = 0.1$, $\epsilon = 0.1$, $\delta = 25$. - At test time, each page is run at multiple zoom scales to improve detection of unusually small or large tables. - K-means cluster algorithm for structure recovery; full pseudocode in supplementary material (Algorithm 1). - Augmentation schemes: "no lines" erases existing graphical lines; "full boundaries" adds lines at median inter-cell positions. - Optimizer, learning rate, batch size, and number of training steps are not reported in the paper. ### Data - **FinTabNet:** ~113k tables from annual reports of S&P 500 companies (SEC 10-K filings), per the original IBM release metadata (not stated in the paper). Annotations use the same automated pipeline as PubTabNet (PDF-XML pixel matching). HTML annotation format. License: CDLA-Permissive-1.0. The original IBM Data Asset eXchange distribution channel is no longer available (404 as of early 2026). The closest accessible version is FinTabNet.c (a canonicalized variant by Brandon Smock, used by the Microsoft Table Transformer project) at `https://huggingface.co/datasets/bsmock/FinTabNet.c`. Exact split sizes not reported in the paper. - **PubTabNet:** Used for pretraining and cross-domain evaluation. Accessed via the original IBM release. - **TableBank:** Used for table detection pretraining only (no cell annotations available). - **ICDAR 2013:** 96/156 train/test; primary fine-tuning target. ### Evaluation - ICDAR 2013 official evaluation script (character-level metrics). - TEDS (Tree-Edit-Distance Similarity, Zhong et al., arXiv:1911.10683) computed on validation PDF images, not the official test set. GTE modifies the evaluation script to ignore bold/italic styling tags, making results not directly comparable to the original PubTabNet EDD baseline (88.38 TEDS). - No error bars, significance tests, or multi-run statistics reported. - FinTabNet results reported with and without fine-tuning, but without per-split breakdown. ### Hardware - Hardware specifications, GPU types, and training time are not reported. ## BibTeX ```bibtex @inproceedings{zheng2021global, title={Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context}, author={Zheng, Xinyi and Burdick, Douglas and Popa, Lucian and Zhong, Xu and Wang, Nancy Xin Ru}, booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, year={2021} } ```