# LGPMA: Complicated Table Structure Recognition with Local and Global Pyramid Mask Alignment > LGPMA combines local and global pyramid soft-mask supervision on a Mask-RCNN backbone to produce accurately aligned cell bounding boxes for table structure recognition, with a three-step recovery pipeline that explicitly handles empty cell detection and merging. - Authors: Hunter Heidenreich - Published: 2026-03-24 - Paper: Preprint (arXiv) - Paper: Published (ICDAR 2021) - Code: LGPMA (DAVAR-Lab-OCR) - Source: Bevaya Labs, --- ## TL;DR LGPMA proposes a Mask-RCNN-based framework that simultaneously learns local (RoI-level) and global (full-image) pyramid soft-mask predictions to produce accurately aligned cell bounding boxes for table structure recognition. A mask re-scoring module fuses the two prediction levels, and a three-step post-processing pipeline uses global segmentation cues to locate and merge empty cells. The method achieves 96.7 TEDS-Struct on the PubTabNet validation set and 98.8 F1 on SciTSR. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The paper introduces LGPMA, an architecture with a specific training recipe combining dual pyramid mask supervision across local and global feature maps, a multi-task loss, and a structured inference pipeline. The center of gravity is the model design, demonstrated through ablation studies that attribute gains to each module. **Secondary:** none significant. ## What is the motivation? Table structure recognition requires recovering the full cell grid from a cropped table image, including logical row/column indices and spanning relations. Two broad families of methods face complementary weaknesses: - **Global-object-based methods** predict row/column separators or segmentation masks, but struggle with cells that span multiple rows or columns and with multi-line text that confuses line-cutting decisions. - **Local-object-based methods** detect text regions as nodes and recover cell relations via heuristic rules or GNNs. Because these methods anchor predictions to visible text, they cannot reliably identify *empty cells*, which have no text and are visually ambiguous with neighboring cross-span cells. Empty cell handling directly affects downstream editable document conversion. The authors observe that if one could obtain *aligned bounding boxes* (boxes expanded from the text region to the full cell extent, including empty padding) for every cell, the table grid becomes recoverable by simple coordinate overlap. The challenge is that aligned bounding boxes are difficult to regress because cell boundaries typically have no visible texture to guide the detector. ## What is the novelty? The core idea is *soft pyramid mask supervision* applied at two spatial granularities simultaneously, combined with a re-scoring procedure that fuses local and global predictions. **Pyramid mask label.** For a cell proposal of height $H$ and width $W$, the text region corners are $(x\_1, y\_1)$ (top-left) and $(x\_2, y\_2)$ (bottom-right). Each pixel $(h, w)$ in the proposal is assigned a soft label in $[0, 1]$ that is maximal at the text midpoint and decreases linearly toward the cell boundary: $$ t\_h^{(w,h)} = \begin{cases} w/x\_{mid} & w \le x\_{mid} \\\\ \frac{W-w}{W-x\_{mid}} & w > x\_{mid} \end{cases}, \quad t\_v^{(w,h)} = \begin{cases} h/y\_{mid} & h \le y\_{mid} \\\\ \frac{H-h}{H-y\_{mid}} & h > y\_{mid} \end{cases} $$ where $x\_{mid} = (x\_1 + x\_2)/2$ and $y\_{mid} = (y\_1 + y\_2)/2$. Because every pixel in the proposal participates in predicting the boundary, the network can push the predicted boundary beyond the initial proposal rectangle. **Local Pyramid Mask Alignment (LPMA).** Within each RoI-aligned feature map, the mask head learns two tasks jointly: a binary text region segmentation and horizontal/vertical pyramid label regression. Local features provide reliable texture cues near the text content but have a limited receptive field. **Global Pyramid Mask Alignment (GPMA).** A parallel branch operates on the full feature map and learns two tasks: (1) a global binary segmentation covering all cells including empty ones (empty cell ground-truth boundaries are derived from the maximum height/width of non-empty cells in the same row/column), and (2) global pyramid label regression for non-empty cells. The global feature captures long-range spatial context but lacks fine-grained texture precision. **Pyramid mask re-scoring.** At inference, for each proposed aligned bounding box with text region midpoint $(x\_{mid}, y\_{mid})$ and overlap with the global segmentation map $P\_o$, the local predictions $F^{(L)}$ and global predictions $F^{(G)}$ are combined with distance-weighted coefficients: $$ F(x) = \begin{cases} \frac{x-x\_1}{x\_{mid}-x\_1} F\_{hor}^{(L)} + \frac{x\_{mid}-x}{x\_{mid}-x\_1} F\_{hor}^{(G)} & x\_1 \le x \le x\_{mid} \\\\ \frac{x-x\_2}{x\_{mid}-x\_2} F\_{hor}^{(L)} + \frac{x\_{mid}-x}{x\_{mid}-x\_2} F\_{hor}^{(G)} & x\_{mid} < x \le x\_2 \end{cases} $$ The fused pyramid map for each side of the box is used to fit a plane by least squares: $$ \min \sum\_{y\_i=y\_1}^{y\_2} \sum\_{x\_i=x\_{mid}}^{x\_2} (ax\_i + by\_i + c - F(x\_i, y\_i))^2 $$ The zero-crossing of the fitted plane gives the refined boundary coordinate: $$ x\_{refine} = - \frac{1}{y\_2 - y\_1 + 1} \sum\_{y\_i=y\_1}^{y\_2} \frac{b y\_i + c}{a} $$ All four boundaries (left, right, top, bottom) are refined analogously. **Table structure recovery pipeline.** Three sequential steps follow bounding box refinement: (1) *cell matching* by coordinate midpoint overlap in horizontal and vertical directions, (2) *empty cell searching* using the Bron-Kerbosch maximum clique algorithm on the matching graph to identify row/column cliques and find vacant positions, and (3) *empty cell merging* guided by the fraction of GPMA segmentation pixels predicted as foreground in the region between adjacent empty cells; neighbor cells are merged when the ratio exceeds a threshold. **Multi-task loss.** Training optimizes: $$ \mathcal{L} = \mathcal{L}\_{rpn} + \lambda\_1(\mathcal{L}\_{cls} + \mathcal{L}\_{box}) + \lambda\_2(\mathcal{L}\_{mask} + \mathcal{L}\_{LPMA}) + \lambda\_3(\mathcal{L}\_{seg} + \mathcal{L}\_{GPMA}) $$ where $\mathcal{L}\_{seg}$ uses the Dice coefficient loss and $\mathcal{L}\_{LPMA}$, $\mathcal{L}\_{GPMA}$ use pixel-wise L1 loss. The weights $\lambda\_1 = \lambda\_2 = \lambda\_3 = 1$ were set empirically. ## What experiments were performed? **Datasets:** - ICDAR 2013: 98 train / 156 test samples from PDF-extracted government reports. - SciTSR: 12,000 train / 3,000 test scientific literature tables; SciTSR-COMP is a harder subset of 2,885 train / 716 test. - PubTabNet: 500,777 train / 9,115 val / 9,138 test scientific tables. **Baselines:** DeepDeSRT, Split, DeepTabStR, Siddiqui et al., ReS2TIM, GTE, GraphTSR, and TabStruct-Net on ICDAR 2013 and SciTSR; EDD, TabStruct-Net, and GTE on PubTabNet. **Metrics:** Adjacency F1 (micro-averaged precision/recall of neighboring cell-pair relations) for ICDAR 2013 and SciTSR; TEDS and TEDS-Struct for PubTabNet. OCR for PubTabNet TEDS uses a separate attention-based recognizer (Lee and Osindero, CVPR 2016). **Ablations (Table 3, 60k train / 1k val from PubTabNet):** Incremental addition of LPMA, GPMA, and Alignment Loss (AL from TabStruct-Net), measured on text region detection, aligned bounding box detection (IoU threshold 0.7), and TEDS-Struct. The combination LPMA+GPMA without AL achieves the best aligned bounding box Hmean (84.95) and TEDS-Struct (95.53). Adding AL on top of LPMA+GPMA degrades aligned bounding box Hmean by 0.27 pp, suggesting the two supervision signals conflict. **Ablations (Table 4):** Three empty cell merging strategies compared on empty-cell detection F1, all-cell detection F1, and TEDS-Struct. The LGPMA-guided strategy achieves 70.21% empty-cell Hmean vs. 59.40% for minimum-cells and 14.70% for maximum-cells. When non-empty bounding box ground truth is provided (ideal upper bound), LGPMA recovery achieves 97.47% empty-cell Hmean and TEDS-Struct 99.77. ## What are the outcomes/conclusions? **Main results on held-out test/validation sets:** | Benchmark | Metric | Prior best | LGPMA | | :--- | :--- | :--- | :--- | | ICDAR 2013 (SciTSR train) | F1 | 0.935 (GTE) | 0.953 | | ICDAR 2013 (fine-tuned) | F1 | 0.935 (GTE) | 0.979 | | SciTSR | F1 | 0.953 (GraphTSR) | 0.988 | | SciTSR-COMP | F1 | 0.955 (GraphTSR) | 0.980 | | PubTabNet val | TEDS | 93.0 (GTE) | 94.6 | | PubTabNet val | TEDS-Struct | n/a | 96.7 | The paper received the Best Industry Paper Award at ICDAR 2021. **Key findings:** - Pyramid soft supervision allows aligned bounding boxes to exceed initial proposal bounds, helping particularly for cross-span cells where the proposal often underestimates the true cell extent. - GPMA's global receptive field and LPMA's local texture detail are complementary; the re-scoring fusion outperforms either branch alone. - The empty cell merging strategy using GPMA visual guidance substantially outperforms naive minimum or maximum strategies on empty-cell detection metrics. - When non-empty bounding boxes are provided as ground truth, the recovery pipeline achieves near-perfect TEDS-Struct (99.77), indicating the structural recovery logic itself is nearly lossless and that detection accuracy is the primary bottleneck. - Alignment Loss (AL, from TabStruct-Net) degrades aligned bounding box detection by 3.1 pp Hmean when combined with LGPMA's pyramid supervision, suggesting the approaches encode redundant or conflicting gradients. **Limitations:** - The method is designed for axis-aligned, print-format tables without rotation or perspective distortion. Wild or photographed tables are out of scope. - Empty cell merging uses a fixed pixel-ratio threshold on GPMA segmentation output, which may be sensitive to heavily bordered or stylized tables. - No inference latency or throughput figures are reported. - OCR is handled by a separately trained recognizer; the full pipeline is not end-to-end. ## Reproducibility ### Models - Backbone: ResNet-50 with Feature Pyramid Network (FPN), initialized from MS-COCO pretrained weights. - LPMA: six anchor ratios [1/20, 1/10, 1/5, 1/2, 1, 2] to handle tall and wide cell shapes; RCNN NMS IoU threshold 0.1 at test time. - GPMA: full-image segmentation branch with pyramid label regression; aligned bounding box ground-truth shrunk by 5% before generating targets to prevent overlap. - Code is released in the DAVAR-Lab-OCR repository under Apache-2.0 (the repo-level license applies to the lgpma subdirectory; no subdirectory-specific license file exists). - No pretrained model weights are publicly released by the authors. Reproduction requires training from scratch using the released code and configuration files. ### Algorithms - Optimizer: SGD, momentum 0.9, weight decay $1 \times 10^{-4}$. - Batch size: 4. - Learning rate: $1 \times 10^{-2}$, divided by 10 every 5 epochs. - Training duration: 12 epochs for SciTSR and PubTabNet; 25 epochs for ICDAR 2013 fine-tuning. - Input scale augmentation: random scale of longer side to [480, 1080] during training; fixed 768 at test time. - Loss weights: $\lambda\_1 = \lambda\_2 = \lambda\_3 = 1$ (empirical; no ablation over these values reported). - Plane fitting for boundary refinement uses least squares; iterative refinement is optional following the Pyramid Mask Text Detector procedure. - Empty cell merging threshold (the pixel-ratio cutoff for deciding whether to merge two adjacent empty cells via GPMA segmentation output) is not reported in the paper and is not exposed in the released configuration files based on available documentation. ### Data - Primary training for SciTSR and ICDAR 2013 experiments: SciTSR training split (12,000 images). - ICDAR 2013 fine-tuning: 98-image training split, after SciTSR pretraining. - PubTabNet experiments: full 500,777-image training split. - Ablation experiments: 60,000 randomly sampled PubTabNet training images. - ICDAR 2013: publicly available; hosted via ICDAR competition infrastructure; no explicit open license but widely used for academic research. - SciTSR: publicly available on GitHub (https://github.com/Academic-Integrity-ML/SciTSR); no explicit open-source license stated by the authors. - PubTabNet: publicly available on IBM's GitHub (https://github.com/ibm-aur-nlp/PubTabNet) under CDLA-Permissive-1.0. ### Evaluation - Adjacency F1: micro-averaged correctness of neighboring cell-pair relations; standard for ICDAR 2013 and SciTSR benchmarks. - TEDS / TEDS-Struct: tree edit distance similarity on full HTML (with and without cell content); introduced with PubTabNet. - No error bars, confidence intervals, or multi-run seeds reported. - Training data conditions vary across baselines (some use private pretraining data); LGPMA uses only public datasets. - Full TEDS reproduction requires an OCR component: the authors use an attention-based recognizer (Lee and Osindero, CVPR 2016) trained separately. No checkpoint or training recipe for this recognizer is provided by the LGPMA authors; reproducing the TEDS (full) score requires sourcing or training a compatible OCR model independently. TEDS-Struct (structure only) is reproducible without OCR. ### Hardware - Training: 8 $\times$ Tesla V100 (32 GB each), PyTorch. - No GPU-hours, per-sample latency, or throughput figures reported in the paper. - The ResNet-50 + FPN + Mask-RCNN architecture is standard and deployable on a single modern GPU for inference. ## BibTeX ```bibtex @inproceedings{qiao2021lgpma, title = {LGPMA: Complicated Table Structure Recognition with Local and Global Pyramid Mask Alignment}, author = {Qiao, Liang and Li, Zaisheng and Cheng, Zhanzhan and Zhang, Peng and Pu, Shiliang and Niu, Yi and Ren, Wenqi and Tan, Wenming and Wu, Fei}, booktitle = {Document Analysis and Recognition -- ICDAR 2021}, pages = {99--114}, year = {2021}, publisher = {Springer International Publishing}, series = {Lecture Notes in Computer Science}, volume = {12821} } ```