# GridFormer: Table Structure Recognition via Grid Prediction > GridFormer proposes an M×N grid representation for tables and a DETR-style two-stream decoder for single-shot TSR across wired, wireless, oriented, and distorted tables. - Authors: Hunter Heidenreich - Published: 2026-03-23 - Paper: Preprint (arXiv) - Paper: Published (ACM MM 2023) - Source: Bevaya Labs, --- ## TL;DR GridFormer frames table structure recognition as predicting the vertices and edges of an $M \times N$ grid. A Deformable DETR backbone with parallel row and column decoders predicts this grid in a single forward pass, covering wired, wireless, oriented, and distorted tables. The method matches or improves on prior results across five benchmarks without requiring OCR annotations. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The paper's headline contribution is a novel architecture for TSR: a new table representation (the $M \times N$ vertex-edge grid) paired with a DETR-style two-stream decoder. The bulk of the paper describes the model design, loss functions, and ablation studies validating each module choice. **Secondary:** none. The paper uses existing public benchmarks and does not release a dataset, metric, or code repository. ## What is the motivation? Table structure recognition (TSR) must handle tables that vary widely in structure, line style, and image quality. Prior methods each handle this diversity poorly in at least one dimension: - **Split-and-merge methods** (region-based) use a two-stage pipeline: a segmentation model divides the table into an over-split grid of regions, then a merge model combines spanning cells. The two-stage design complicates end-to-end training and degrades on wireless or geometrically distorted tables. - **Graph-based methods** model cells or text lines as graph nodes and predict pairwise relationships. They depend on OCR annotations at inference time, creating a hard coupling to text detection quality. - **Cell-based methods** detect individual cells via object detection. They work well on wired tables but struggle with wireless tables where cell boundaries are implicit. - **Markup language-based methods** generate HTML sequences autoregressively. They suffer from attention drift on long sequences and scale poorly to large tables. None of these approaches cleanly handles all four challenging scenarios (wireless, oriented, distorted, multi-spanning-cell) within a single-stage, OCR-free pipeline. GridFormer is designed to close that gap. ## What is the novelty? **Grid representation.** The central observation is that any table can be expressed as an $M \times N$ grid of vertices and edges. Given a table with $r$ rows and $c$ columns, the corresponding grid has $(r+1) \times (c+1)$ vertices and two sets of edges (rightward and downward). A vertex is "positive" if it coincides with a cell corner; an edge is "positive" if it lies on a cell boundary. Both sets are binary-classified, and each positive vertex stores its physical $(x, y)$ coordinate. This representation is: - **Unified:** wired and wireless tables differ only in which edges are positive; the representation itself does not change. - **Compact:** no autoregressive decoding; the grid is a fixed-size tensor for a given dataset. - **Geometry-aware:** physical coordinates are stored directly, enabling localization for distorted or rotated tables. **Two-stream decoder.** GridFormer adapts Deformable DETR with two parallel transformer decoders. One decoder processes row queries $Q\_{\text{row}} \in \mathbb{R}^{M \times d}$ and predicts $y$-axis coordinates; the other processes column queries $Q\_{\text{col}} \in \mathbb{R}^{N \times d}$ and predicts $x$-axis coordinates. Decoupling the axes simplifies vertex localization because within a single row, the $y$ range is narrow while $x$ varies widely, and vice versa. **Query selection module.** Rather than using learnable reference point embeddings (as in standard DETR), the query selection module generates image-conditioned reference points. Row proposals are placed at fixed $x = W/4$ positions evenly distributed along $y$; column proposals at fixed $y = H/4$ evenly along $x$. The top-80 scoring proposals (no NMS) initialize the decoder reference points, and L1 supervision is applied to these points during training. **Three prediction heads.** Given row embeddings $Z\_{\text{row}}$ and column embeddings $Z\_{\text{col}}$: 1. **Classification head:** predicts positive/negative probability for each row and column via Focal loss. $$L\_{\text{cls}} = \text{Focal}(\hat{p}\_{\text{row}}, p\_{\text{row}}) + \text{Focal}(\hat{p}\_{\text{col}}, p\_{\text{col}})$$ 2. **Position head:** predicts normalized $y$-coordinates from row embeddings and $x$-coordinates from column embeddings via L1 loss, with an auxiliary cell-level GIoU loss on reconstructed cell bounding boxes. $$L\_{\text{coord}} = L1(\hat{r}\_{\text{row}}, r\_{\text{row}}) + L1(\hat{r}\_{\text{col}}, r\_{\text{col}}) + \gamma\_1 L\_{\text{iou}}(\hat{g}, g)$$ 3. **Edge head:** at each grid vertex, concatenates the corresponding row/column query embedding (global context) with a locally sampled CNN feature (local context), then applies binary Focal classification for rightward and downward edges separately. $$L\_{\text{edge}} = \text{Focal}(\hat{e}\_{\text{row}}, e\_{\text{row}}) + \text{Focal}(\hat{e}\_{\text{col}}, e\_{\text{col}})$$ The total loss is: $$L = \lambda\_1 L\_{\text{cls}} + \lambda\_2 L\_{\text{coord}} + \lambda\_3 L\_{\text{ref}} + \lambda\_4 L\_{\text{edge}}$$ with $\lambda\_1 = 1$, $\lambda\_2 = \lambda\_3 = \lambda\_4 = 5$, $\gamma\_1 = 0.1$. Auxiliary losses from all six decoder layers are also used. **Table reconstruction.** After decoding, rows and columns with score $> \tau\_1 = 0.5$ are kept and sorted by reference point coordinates. Edges with score $> \tau\_2 = 0.4$ are kept. A breadth-first search on the resulting grid groups adjacent positive vertices into cells. ## What experiments were performed? GridFormer is evaluated on five benchmarks spanning both regular (PDF-derived) and scene (wild-image) tables. **Datasets:** - **SciTSR:** 12k train / 3k test; scientific PDFs; evaluated with cell adjacency F1. - **PubTabNet:** 500k train / 9k val; scientific PDFs; evaluated with TEDS and TEDS-Struct on the validation set (test annotations are unreleased). - **FinTabNet:** 92k train / 10k val; financial tables; evaluated with TEDS-Struct on the validation set. - **WTW:** 11k train / 3.6k test; wild photos with deformation; evaluated with cell adjacency precision/recall/F1. - **TAL (TAL\_OCR\_TABLE):** 12.3k train / 3k test (custom split from the competition set); wild scene tables; also evaluated on TAL\_rotated ($\pm 30^\circ$) and TAL\_curved (geometric distortion variants). Metrics: TEDS-Struct and cell F1 at IoU 0.6. **Baselines compared:** EDD, TabStruct-Net, GTE, SEM, LGPMA, FLAG-Net, NCGM, TableFormer, TSRFormer, TRUST, VAST (PubTabNet/FinTabNet); GraphTSR, RobustTabNet (SciTSR); Cycle-CenterNet, TSRFormer, NCGM (WTW); SPLERGE, TableMaster (TAL). **Implementation:** ResNet-50 backbone; 6 deformable encoder layers; 6 decoder layers per stream; AdamW, LR 2e-4; 8 V100 GPUs, batch size 24; multi-scale training with short side in {384, 416, 448, 480, 512}; long side capped at 640. **Ablations (on WTW):** - Removing the query selection module (random reference points): F1 drops from 94.1% to 86.3% (-7.8%). - Replacing the two-stream decoder with a single decoder: F1 drops from 94.1% to 70.2% (-23.9%), suggesting that axis decoupling matters considerably. - Using only query embeddings for edge classification (no local visual features): F1 drops to 62.2%. - Using only local visual features for edge classification (no query embeddings): F1 drops to 88.7%. - Both combined: 94.1% (best), suggesting the value of global-plus-local feature fusion. ## What are the outcomes/conclusions? **Regular tables (PDF-derived):** | Dataset | Metric | GridFormer | Nearest competitor | |:---|:---|:---|:---| | PubTabNet (val) | TEDS | 95.84% | VAST: 96.31% | | PubTabNet (val) | TEDS-Struct | 97.0% | TSRFormer: 97.5% | | FinTabNet (val) | TEDS-Struct | 98.63% | VAST: 98.63% | | SciTSR (test) | F1 (excl. empty) | 99.3% | NCGM/TSRFormer: 99.6% | On PubTabNet and SciTSR, GridFormer is competitive but not the best-reported number. On FinTabNet, it ties VAST at 98.63% (an improvement of 1.8 points over TableFormer). **Scene tables (wild images):** | Dataset | Metric | GridFormer | Nearest competitor | |:---|:---|:---|:---| | WTW (test) | F1 | 94.1% | NCGM: 94.1% | | TAL (test) | TEDS-Struct | 99.4% | TableMaster: 98.8% | | TAL (test) | Cell F1 | 98.9% | TableMaster: 80.8% | | TAL\_rotated | Cell F1 | 92.9% | TableMaster: 27.8% | | TAL\_curved | Cell F1 | 96.8% | TableMaster: 64.5% | The gap between methods is largest on rotated and curved tables. TableMaster's autoregressive decoder drops substantially under geometric distortion, from 27.8% (TAL\_rotated) and 64.5% (TAL\_curved) versus GridFormer at 92.9% and 96.8% respectively. **Limitations:** - No code or weights are released, limiting reproducibility. - The TAL evaluation uses a custom train/test split from the competition set rather than an established evaluation protocol; comparison with future methods may be inconsistent. - TEDS on PubTabNet requires external OCR (the authors use PSENet + MASTER); the quality of these OCR results directly affects reported TEDS scores, and the paper does not ablate OCR quality. - The grid size $M \times N$ is matched to the largest table in each dataset. For datasets with very large tables, this increases query count and memory usage proportionally. - The method does not output cell text content; it is a structure-only recognizer. Downstream content extraction requires a separate OCR stage. ## Reproducibility ### Models - **Backbone:** ResNet-50 (ImageNet pre-trained; standard initialization). - **Encoder:** 6 deformable transformer encoder layers from Deformable DETR; 256-channel features; multi-scale features from ResNet stages 3-5. - **Decoder:** two parallel deformable transformer decoders, each with 6 layers; hidden dimension 256. - **Query counts:** $M$ and $N$ are set to the maximum row and column counts in each dataset (e.g., 50 rows and 50 columns for TAL). Ablations show modest sensitivity to this choice. - No pre-trained TSR checkpoints, model weights, or code repository are released. ### Algorithms - **Optimizer:** AdamW, initial LR 2e-4. No learning rate schedule (warmup, decay policy, or total training steps/epochs) is reported in the paper. - **Training hardware:** 8 NVIDIA V100 GPUs; total batch size 24. - **Multi-scale training:** short side randomly sampled from {384, 416, 448, 480, 512}; long side capped at 640. - **Inference resizing:** long side fixed at 640 while preserving aspect ratio. - **Score thresholds:** $\tau\_1 = 0.5$ (row/column classification), $\tau\_2 = 0.4$ (edge classification). - **Loss weights:** $\lambda\_1 = 1$, $\lambda\_2 = \lambda\_3 = \lambda\_4 = 5$, $\gamma\_1 = 0.1$. - **Bipartite matching:** Hungarian algorithm used to assign ground-truth rows/columns to predicted queries, following DETR. ### Data - All training data come from existing public datasets (SciTSR, PubTabNet, FinTabNet, WTW, TAL). No new data is collected. - TAL uses a custom 12,285/3,000 train/test split from the original competition training set; split file names are stated to be released but no repository is given. - TAL\_rotated and TAL\_curved are derived from TAL via data augmentation (random rotation $\pm 30^\circ$; geometric distortion); these variants are not available as standalone downloads. - Label generation for datasets with text-line level bounding box annotations (e.g., SciTSR, PubTabNet) requires a multi-step coordinate extension procedure described in the appendix; this preprocessing is non-trivial but documented in the paper. ### Evaluation - **TEDS / TEDS-Struct:** standard metrics from PubTabNet; TEDS-Struct ignores cell content and compares HTML structure trees only. - **Cell adjacency F1:** used for SciTSR and WTW; measures whether pairs of horizontally or vertically adjacent cells are correctly identified. - **Cell F1 at IoU 0.6:** localization metric used for TAL; measures how well predicted cell bounding boxes overlap with ground truth. - PubTabNet TEDS results depend on external OCR (PSENet + MASTER); the paper does not provide TEDS-Struct-only comparisons that would isolate the structure predictor from OCR errors. - No error bars, significance tests, or repeated runs are reported. ### Hardware - **Training:** 8 V100 GPUs; total batch size 24. Total GPU-hours and wall-clock time are not reported. - **Inference:** long side 640; no inference latency or throughput numbers are reported. - **Deployment:** no discussion of CPU-only or low-memory inference feasibility. ## BibTeX ```bibtex @inproceedings{lyu2023gridformer, title={GridFormer: Towards Accurate Table Structure Recognition via Grid Prediction}, author={Lyu, Pengyuan and Ma, Weihong and Wang, Hongyi and Yu, Yuechen and Zhang, Chengquan and Yao, Kun and Xue, Yang and Wang, Jingdong}, booktitle={Proceedings of the 31st ACM International Conference on Multimedia}, year={2023}, doi={10.1145/3581783.3611961} } ```