# TSRFormer: Table Structure Recognition with Transformers

> A research note on TSRFormer, an ACM Multimedia 2022 approach to table structure recognition that reformulates separation line detection as a regression problem using a two-stage DETR decoder, enabling robust recognition of distorted and curved tables.

- Authors: Bevaya AI Team
- Published: 2022-08-09
- Updated: 2026-03-18
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2208.04921>
- Paper: Published (ACM Multimedia 2022) <https://doi.org/10.1145/3503161.3548038>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

TSRFormer reformulates table separation line detection as a direct regression problem rather than a pixel segmentation task. The core module, SepRETR, uses a two-stage DETR-like decoder: a reference point detector finds one anchor per separator, then a transformer decoder regresses the full curvilinear line from that anchor. On SciTSR, PubTabNet, and WTW, TSRFormer achieves results competitive with prior work; on a proprietary in-house dataset with distorted and curved tables it outperforms a re-implemented SPLERGE baseline by more than 11 F1 points.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$ The headline contribution is the SepRETR architecture and its two companion improvements (prior-enhanced bipartite matching, high-resolution cross-attention sampling). The paper is centered on the model design and its ablation, with benchmark comparisons as supporting evidence.

**Secondary:** None. No new dataset or benchmark is released; the in-house dataset is used for evaluation but not distributed.

## What is the motivation?

Table structure recognition (TSR) must recover which rows and columns partition a table and identify spanning cells that cross those boundaries. Prior split-and-merge methods predict row and column separators via pixel-level segmentation, then apply post-processing to convert probability masks into line coordinates. This pipeline introduces heuristic mask-to-line steps that are fragile when tables are distorted, curved, or contain large empty regions that disrupt the feature representation.

At the same time, the DETR family of object detectors showed that set prediction with bipartite matching could eliminate NMS and anchor engineering, but DETR converges slowly and struggles to leverage high-resolution feature maps due to the quadratic cost of global attention.

The authors aim to address both problems at once: replace segmentation with direct regression and adapt two-stage DETR to the geometry of line prediction, where the location of a separator provides a strong spatial prior.

## What is the novelty?

### SepRETR: Separator REgression TRansformer

TSRFormer's split module runs two parallel SepRETR branches (one for rows, one for columns) on a shared convolutional feature map $P\_2$ from a ResNet-18-FPN backbone.

**Feature enhancement.** A series of downsampling blocks reduces $P\_2$ to a compact feature map $P'\_2 \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{32} \times C}$. Two spatial CNN (SCNN) modules then propagate context across the width, producing a context-enhanced map $E\_{\text{row}}$ where each position integrates information from both sides of the image.

**Reference point detection.** At a fixed column $x\_r = \lfloor W/4 \rfloor$ of the upsampled map $E''\_{\text{row}}$, a sigmoid classifier scores each pixel as a candidate reference point. After 7×1 max-pool NMS, the top-100 points above a threshold of 0.05 become object queries for the DETR decoder.

**Separation line regression.** Each row separator is represented by three parallel curvilinear lines (top boundary, center, bottom boundary), each parameterized as $K = 15$ points at fixed x-coordinates. The decoder attends to a downsampled feature map $C'\_{\text{row}} \in \mathbb{R}^{H \times K \times C'}$ formed by stacking the $K$ columns of $E''\_{\text{row}}$. A 3-layer transformer decoder produces per-query embeddings; two feedforward networks then classify each query and regress its 3K y-coordinates.

### Prior-enhanced bipartite matching

Standard DETR bipartite matching is unstable during early training because any query can be assigned to any ground-truth object. The authors observe that detected reference points consistently fall within the top and bottom boundaries of their corresponding separator across training epochs. They exploit this by constructing a cost matrix where the cost between reference point $i$ and ground-truth separator $k$ is the distance to the GT reference point if the reference point lies within the separator's boundaries, and $\infty$ otherwise. The Hungarian algorithm then produces a stable matching, substantially accelerating convergence.

### Loss functions

Reference point detection uses a focal-loss variant with Gaussian-augmented targets:

$$
L\_{\text{ref}}^{\text{row}} = -\frac{1}{N\_r} \sum\_{i=1}^{H} \begin{cases} (1-p\_i)^\alpha \log(p\_i), & p\_i^\ast=1 \\\\ (1-p\_i^\ast)^\beta p\_i^\alpha \log(1-p\_i), & \text{otherwise} \end{cases}
$$

where $\alpha=2$, $\beta=4$, and the Gaussian soft label $p\_i^\ast$ within separator $k$ is:

$$
p\_i^\ast = \exp\!\left(-\frac{(i - y\_k)^2}{2\sigma\_k^2}\right), \quad \sigma\_k = \sqrt{\frac{w\_k^2}{2\ln(10)}}
$$

with $w\_k$ the separator thickness, so that $p\_i^\ast \geq 0.1$ within the separator region.

Separation line regression uses a combined focal and L1 loss after bipartite matching $\hat{\sigma}$:

$$
L\_{\text{line}}^{\text{row}} = \sum\_{i=1}^{Q} \left[ L\_{\text{cls}}(c\_i, c\_{\hat{\sigma}(i)}^\ast) + \mathbf{1}\_{\\{c\_i \neq \emptyset\\}} L\_{\text{reg}}(l\_i, l\_{\hat{\sigma}(i)}^\ast) \right]
$$

An auxiliary binary cross-entropy segmentation loss penalizes per-pixel separator predictions:

$$
L\_{\text{aux}}^{\text{row}} = \frac{1}{|S\_{\text{row}}|} \sum\_{(x,y) \in S\_{\text{row}}} \text{BCE}(M\_{\text{row}}(x,y), M\_{\text{row}}^\ast(x,y))
$$

where $S\_{\text{row}}$ is the set of sampled pixels and $M\_{\text{row}}^\ast(x,y)$ is 1 only if the pixel falls within a row separator. The cell merging module uses a corresponding BCE loss over sampled cell pairs:

$$
L\_{\text{merge}} = \frac{1}{|S\_{\text{rel}}|} \sum\_{i \in S\_{\text{rel}}} \text{BCE}(P\_i, P\_i^\ast)
$$

The full training objective is:

$$
L = \lambda(L\_{\text{ref}}^{\text{row}} + L\_{\text{ref}}^{\text{col}}) + L\_{\text{aux}}^{\text{row}} + L\_{\text{aux}}^{\text{col}} + L\_{\text{line}}^{\text{row}} + L\_{\text{line}}^{\text{col}} + L\_{\text{merge}}
$$

with $\lambda = 0.2$.

### Cell merging module

After intersecting row and column separators to produce a regular cell grid, a relation network recovers spanning cells. RoI Align extracts $7 \times 7$ features for each cell from $P\_2$; a two-layer MLP maps them to 512-d vectors. Three feature enhancement blocks aggregate row-level, column-level, and local context. For each adjacent cell pair, the concatenated features plus an 18-d spatial compatibility feature are fed to a binary MLP classifier.

## What experiments were performed?

### Datasets

- **SciTSR**: 12k train / 3k test; axis-aligned tables from scientific PDFs. A harder subset, SciTSR-COMP (716 tables), is evaluated separately.
- **PubTabNet**: ~500k train, 9k validation; axis-aligned tables from scientific articles. The structure-only TEDS-Struct metric is used to avoid confounds from OCR engine differences.
- **WTW**: ~11k train / ~3.6k test; bordered tables photographed in wild scenes, including skewed and curved instances.
- **In-house dataset**: 40,590 train / 1,053 test; heterogeneous scanned and camera-captured documents including scientific, financial, and invoice content. Evaluation uses the cTDaR TrackB metric at IoU 0.9.

### Baselines

On public benchmarks, TSRFormer is compared against TabStruct-Net, GraphTSR, LGPMA, and FLAG-Net (SciTSR); EDD, GTE, LGPMA, FLAG-Net (PubTabNet); and Cycle-CenterNet (WTW). For in-house evaluation, a re-implemented SPLERGE with the same backbone is the primary comparison.

### Implementation

Backbone: ResNet-18-FPN; $P\_2$ channel width set to 64. Optimizer: AdamW with initial LR $10^{-4}$, polynomial decay (power 0.9). Batch size 16 on 8 Nvidia Tesla V100 GPUs. Staged training: reference point detection and auxiliary segmentation for $N$ epochs, then regression added for $N$ epochs, then cell merging added for $N$ epochs ($N = 12$ for PubTabNet, $N = 20$ for others).

## What are the outcomes/conclusions?

**SciTSR:** TSRFormer reaches 99.4% F1 on the full test set and 98.9% on SciTSR-COMP, competitive with FLAG-Net (99.5% / 98.5%).

**PubTabNet:** 97.5% TEDS-Struct, 0.8 points above LGPMA (the ICDAR 2021 competition winner at 96.7%).

**WTW:** 93.4% F1, 1.0 point above Cycle-CenterNet (92.4%), which was designed specifically for this wild-scene scenario.

**In-house (distorted/borderless tables):** TSRFormer reaches 95.2% F1 vs. 83.8% for the re-implemented SPLERGE baseline, a gap of more than 11 points. The ablation attributes this gain primarily to replacing segmentation with regression (line prediction without heuristic mask-to-line steps) and to the cell merging module.

**Ablation highlights:** The prior-enhanced matching strategy matches the convergence quality of the original DETR strategy at 40 epochs while needing only 20 epochs. Using sampled high-resolution features $C\_{\text{row}}, C\_{\text{col}}$ in cross-attention adds 0.5 F1 over using the full encoder features.

**Limitations acknowledged:** Model weights and code are not released. The in-house dataset is not publicly available, so the largest performance claims cannot be externally verified. Evaluation on WTW covers bordered tables only; generalization to borderless wild-scene tables is not demonstrated. No latency or throughput numbers are reported.

## Reproducibility

| Resource | Type | License | Link |
|:---|:---|:---|:---|
| Preprint (arXiv) | Paper | arXiv-nonexclusive-v1.0 | [arXiv 2208.04921](https://arxiv.org/abs/2208.04921) |
| Published (ACM MM 2022) | Paper | Unknown | [DOI 10.1145/3503161.3548038](https://doi.org/10.1145/3503161.3548038) |
| Model weights | Model | N/A | Not released |
| Training code | Code | N/A | Not released |
| SciTSR | Dataset | Unknown | [GitHub](https://github.com/Academic-Hammer/SciTSR) |
| PubTabNet | Dataset | CDLA-Permissive-1.0 | [GitHub](https://github.com/ibm-aur-nlp/PubTabNet) |
| WTW | Dataset | Unknown | [GitHub](https://github.com/wangwen-whu/WTW-Dataset) |

### Models

- ResNet-18-FPN backbone; $P\_2$ feature dimension 64. ResNet-18 weights initialized from ImageNet pretraining.
- SepRETR decoder: 3 transformer decoder layers; query dimension 256; 16 attention heads; FFN dimension 1024.
- $E'\_{\text{row}}/E'\_{\text{col}}$ channel dimension: 256.
- Cell merging: 2-layer MLP (512 nodes), relation network with 3 feature enhancement blocks, binary classifier MLP (2 hidden layers, 512 nodes).
- Model weights are **not publicly released**.

### Algorithms

- Optimizer: AdamW; $\beta = (0.9, 0.999)$; $\epsilon = 10^{-8}$; weight decay $5 \times 10^{-4}$.
- Learning rate: $10^{-4}$ with polynomial decay (power 0.9).
- Data augmentation: shorter side randomly rescaled to one of $\{416, 512, 608, 704, 800\}$ (aspect ratio preserved) for all datasets except WTW, where both sides are resized to 1024.
- Synchronized BatchNorm during training.
- Training on 8 V100 GPUs, batch size 16; staged training ($N$ epochs per stage, $N=12$ or $N=20$).
- Reference points: top-100 per image, score threshold 0.05, 7×1 max-pool NMS.
- $K = 15$ regression points per line; $x\_r = \lfloor W/4 \rfloor$ for rows; $y\_r = \lfloor H/4 \rfloor$ for columns.
- Cell merging: 1024 cells sampled per mini-batch (1024 positive + 1024 negative for segmentation; 64 hard positive + 64 hard negative for merging, selected by OHEM).
- $\lambda = 0.2$ (loss balance for reference point detection).

### Data

- SciTSR: publicly available; 12k/3k train/test split. License: see dataset page.
- PubTabNet: publicly available; ~500k/9k/9k train/val/test (test annotations not released). License: CDLA-Permissive-1.0.
- WTW: publicly available; ~11k/3.6k train/test.
- In-house dataset: proprietary; 40,590 train / 1,053 test; not released.
- Ground-truth separation lines for SciTSR and PubTabNet are generated from cell-box annotations following the procedure in RobusTabNet [arXiv:2203.09056].

### Evaluation

- SciTSR: cell adjacency relationship metric (F1). Two variants: with and without empty cells.
- PubTabNet: TEDS-Struct (structure-only TEDS, ignoring OCR). Validation set only (test annotations unreleased).
- WTW: cell adjacency F1 at IoU 0.6.
- In-house: cTDaR TrackB metric at IoU 0.9 using GT text boxes.
- No error bars or multi-run statistics are reported.
- SPLERGE baseline is re-implemented by the authors with the same backbone for a fair comparison; results may differ from the original published SPLERGE.

### Hardware

- Training: 8 Nvidia Tesla V100 GPUs.
- GPU-hours and memory requirements are not reported.
- Inference resolution: longer side rescaled to 1024 pixels (aspect ratio preserved) for SciTSR, PubTabNet, and in-house; both sides to 1024 for WTW.
- No inference latency or throughput numbers reported.

## BibTeX

```bibtex
@inproceedings{lin2022tsrformer,
  title={TSRFormer: Table Structure Recognition with Transformers},
  author={Lin, Weihong and Sun, Zheng and Ma, Chixiang and Li, Mingze and Wang, Jiawei and Sun, Lei and Huo, Qiang},
  booktitle={Proceedings of the 30th ACM International Conference on Multimedia},
  year={2022}
}
```