# SEMv3: A Fast and Robust Approach to Table Separation Line Detection

> A research note on SEMv3, an IJCAI 2024 table structure recognizer that replaces segmentation-based line detection with keypoint offset regression, achieving competitive accuracy on distorted and wireless tables with lower computational cost.

- Authors: Bevaya AI Team
- Published: 2024-05-20
- Updated: 2026-03-18
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2405.11862>
- Paper: Published (IJCAI 2024) <https://www.ijcai.org/proceedings/2024/132>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

SEMv3 is the third iteration of the SEM (Split, Embed and Merge) family for table structure recognition. The split stage replaces instance segmentation with a Keypoint Offset Regression (KOR) module that predicts offsets from fixed horizontal proposals to the true line keypoints, removing the need for mask-to-line post-processing. The merge stage introduces a four-class merge action map (Stay, Left, Upward, X) that describes the cell grid structure in O(NM) space rather than the O(N²M²) needed by prior merge-map approaches. On the iFLYTAB, WTW, and ICDAR-2019 cTDaR Historical benchmarks the combined system outperforms SEMv2 and is competitive with LORE.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$ The headline contribution is the KOR split module and the merge action formulation. The paper is organized around model design, ablation studies on iFLYTAB, and comparisons on four benchmarks.

**Secondary:** None. No new dataset or benchmark is released. iFLYTAB is introduced in the SEMv2 paper (Zhang et al., 2024); SEMv3 evaluates on it but does not release it.

## What is the motivation?

The split-and-merge paradigm for TSR detects row and column separation lines, intersects them to form a grid, and then merges over-split grid cells to recover spanning cells. The split stage has historically used semantic segmentation to predict a per-pixel separator mask, but this requires a heuristic mask-to-line post-processing step. Segmentation-based methods also struggle with wireless tables (no visible borders) and geometrically deformed tables, because the predicted masks become noisy and the post-processing amplifies errors.

Instance-segmentation approaches (e.g., SEMv2) partially address this by predicting a per-line mask through dynamic convolution, but the position-insensitivity of instance convolution kernels can reduce robustness, and the inference cost grows linearly with the number of lines.

A separate weakness is in the merge stage. Predicting a full merge map for each grid cell (as in SEMv2) requires quadratic space and computation in the grid dimensions, limiting scalability.

SEMv3 addresses both: replace segmentation with direct offset regression for the split stage, and replace the per-grid merge map with a compact action classification for the merge stage.

## What is the novelty?

### Keypoint Offset Regression (KOR) Split Module

The backbone is ResNet-34 with FPN, producing four feature levels $\mathbf{P}\_2, \mathbf{P}\_3, \mathbf{P}\_4, \mathbf{P}\_5$ that are fused into a single map $\mathbf{F} \in \mathbb{R}^{H/4 \times W/4 \times C}$.

**Separation line representation.** The $i$-th row separation line is represented by a sequence of keypoints $\\{k\_{ij}^{\text{row}} \mid j = 0, \ldots, N\_k^{\text{row}} - 1\\}$. The number of keypoints is determined by image width $W$ and a fixed sampling step $t$:

$$
N\_k^{\text{row}} = \left\lceil \frac{W}{t} \right\rceil
$$

The x-coordinates of keypoints are fixed ($x\_{ij} = j \times t$), so the line is fully characterized by the y-coordinates of its keypoints. Rather than predicting absolute y-coordinates, KOR predicts the offset $\delta\_{ij}^{\text{row}}$ from a horizontal proposal to the true keypoint position. Proposals share the same x-coordinates as keypoints and have y-coordinates matching the starting point of their line.

**Feature enhancement.** Two separate feature enhancement modules (with non-shared parameters) process $\mathbf{F}$ using SCNN for row and column context propagation, producing $\mathbf{F}\_{sd}^{\text{row}}$ for starting point detection and $\mathbf{F}\_{lr}^{\text{row}}$ for line regression.

**Starting point detection.** Row-wise average pooling and a softmax over $\mathbf{F}\_{sd}^{\text{row}}$ produce a probability vector $\mathbf{P}^{\text{row}}$ for whether each row contains a separator starting point. NMS removes duplicate predictions.

**Keypoint offset head.** Proposal features $\mathbf{K}' \in \mathbb{R}^{C \times N\_k^{\text{row}}}$ are sampled from $\mathbf{F}\_{lr}^{\text{row}}$ at proposal positions. A line-level representation is computed by averaging proposal features within the same line:

$$
\mathbf{S}\_i^{\text{row}} = \frac{1}{N\_k^{\text{row}}} \sum\_{j=1}^{N\_k^{\text{row}}} \mathbf{K}'\_{ij}
$$

The per-keypoint feature is then formed by channel-wise concatenation of the proposal feature and the line representation:

$$
\mathbf{K}\_{ij} = \text{concat}(\mathbf{K}'\_{ij},\; \mathbf{S}\_i^{\text{row}})
$$

A $1 \times 3$ convolution over $\mathbf{K}$ predicts the offset $\delta\_{ij}^{\text{row}}$, allowing each keypoint to consider its neighbors for a smoother predicted line.

### Merge Action Map

After intersecting row and column separators to produce a grid $\mathbf{B} \in \mathbb{R}^{M \times N \times 8}$ (quadrilateral boxes), grid features are extracted via RoI Align and absolute position embedding:

$$
\mathbf{E}' = \text{RoiAlign}(\mathbf{F}, \mathbf{B}) + \text{PE}(\mathbf{B})
$$

A row/column self-attention layer produces the enhanced grid representation $\mathbf{E}$.

Each grid cell is assigned one of four merge actions:
- **S (Stay):** upper-left cell of a spanning cell; do not merge in either direction.
- **L (Left):** merge with the cell to the left.
- **U (Upward):** merge with the cell above.
- **X:** merge both left and upward.

A convolutional auxiliary branch classifies starting grids; the final merge action is predicted from the concatenation of the starting-grid feature and $\mathbf{E}$. This formulation requires only one merge action map of size $M \times N$ per image, reducing the space complexity from $O(N^2 M^2)$ (per-grid merge maps) to $O(NM)$.

### Loss Functions

**Starting point detection** uses binary cross-entropy:

$$
\mathcal{L}\_{sp}^{\text{row}} = \frac{1}{H/2} \sum\_{i=1}^{H/2} \mathcal{L}\_{\text{BCE}}(\hat{\mathbf{P}}\_i^{\text{row}}, \mathbf{P}\_i^{\text{row}})
$$

**Keypoint offset regression** uses L2 loss:

$$
\mathcal{L}\_\delta^{\text{row}} = \frac{1}{N^{\text{row}} N\_k^{\text{row}}} \sum\_{i=1}^{N^{\text{row}}} \sum\_{j=0}^{N\_k^{\text{row}}-1} \left\| \hat{\delta}\_{ij}^{\text{row}} - \delta\_{ij}^{\text{row}} \right\|\_2
$$

**Starting grid classification** and **merge action prediction** both use focal loss:

$$
\mathcal{L}\_{\text{sg}} = \frac{1}{NM} \sum\_{i=1}^{N} \sum\_{j=1}^{M} L\_{\text{focal}}(\hat{\mathbf{P}}\_{ij}^{\text{sg}}, \mathbf{P}\_{ij}^{\text{sg}})
$$

$$
\mathcal{L}\_{\text{ma}} = \frac{1}{NM} \sum\_{a} \sum\_{i=1}^{N} \sum\_{j=1}^{M} L\_{\text{focal}}(\hat{\mathbf{P}}\_{ij}^{a}, \mathbf{P}\_{ij}^{a})
$$

**Overall loss:**

$$
\mathcal{L} = \mathcal{L}\_{sp}^{\text{row}} + \mathcal{L}\_{sp}^{\text{col}} + \mathcal{L}\_\delta^{\text{row}} + \mathcal{L}\_\delta^{\text{col}} + \mathcal{L}\_{\text{ma}} + \mathcal{L}\_{\text{sg}}
$$

## What experiments were performed?

### Datasets

- **ICDAR-2019 cTDaR Historical**: 600 train / 150 test; archival historical documents with sparse borders and geometric deformation. Metric: cell adjacency F1 at IoU 0.6.
- **WTW**: 14,581 wired tables from real business scenarios, including seven hard cases (inclined, curved, occluded, extreme aspect ratio, etc.). Metric: cell adjacency F1 at IoU 0.6.
- **iFLYTAB** (introduced in SEMv2): 12,103 train / 5,188 test; four subsets: Wired-Digital (WDD), Wired-Camera-Capture (WDC), Wireless-Digital (WLD), Wireless-Camera-Capture (WLC). Cell adjacency F1 and grid detection F1-G (IoU 0.9) are both reported.
- **SciTSR-COMP** and **PubTabNet**: axis-aligned digital tables for broader comparison. SciTSR-COMP uses cell adjacency F1; PubTabNet uses TEDS and TEDS-Struct.

### Baselines

ICDAR-2019 cTDaR Historical: TabStructNet, FLAGNet, NCGM, SEMv2, LORE. WTW: Cycle-CenterNet, NCGM, TSRFormer-DQ, SEMv2, LORE. iFLYTAB: SEM, SEMv2. SciTSR/PubTabNet: RobusTabNet, TSRFormer, LORE, TSRFormer-DQ, SEMv2.

### Ablation

Table 5 in the paper crosses two split options (IS from SEMv2, KOR) with two merge options (MP from SEMv2, MA from SEMv3) on iFLYTAB's four subsets:
- T1: IS + MP; T2: KOR + MP; T3: IS + MA; T4: KOR + MA.
Grid detection F1-G on wireless subsets (WLC, WLD) rises from roughly 18-24% (IS) to 37-59% (KOR) under both merge settings, confirming that the split quality improvement is primarily attributable to KOR.

### Implementation

ResNet-34 + FPN backbone; feature channel $C = 256$; grid feature channel $C\_g = 512$; sampling step $t = 32$. Trained end-to-end for 100 epochs. Optimizer: Adam, initial LR $10^{-4}$ decayed to $10^{-6}$ via cosine annealing. Four Nvidia Tesla V100 GPUs (24 GB each); PyTorch 1.7.1.

## What are the outcomes/conclusions?

**ICDAR-2019 cTDaR Historical:** 89.3% F1, ahead of LORE (88.3%) and SEMv2 (85.9%).

**WTW:** 95.1% F1, tied with LORE* (95.1%), but SEMv3 uses the stricter IoU threshold of 0.6 vs. LORE's 0.5. The authors note this makes the comparison favorable to SEMv3 relative to LORE.

**iFLYTAB:** 94.4% F1 overall, 0.9 points above SEMv2 (93.5%). On wireless subsets the grid F1-G gain is substantially larger (18-24% for IS to 37-59% for KOR).

**SciTSR-COMP:** 99.0% F1, comparable to TSRFormer (98.9%) and TSRFormer-DQ (98.8%).

**PubTabNet:** 97.3% TEDS, 97.5% TEDS-Struct. TEDS-Struct is on par with TSRFormer-DQ and SEMv2; TEDS is slightly below LORE (98.1%).

**Inference speed:** KOR's cost is nearly constant as the number of rows and columns grows because it predicts a single shared offset map, while IS cost grows linearly with the number of lines. In the paper's timing comparison on SciTSR at 512×512, KOR is significantly faster than IS for tables with many rows/columns.

**Limitations acknowledged:** Performance on SciTSR is noted as potentially confounded because the test set has been "manually rectified in various ways." On PubTabNet the method is slightly below LORE (a region-based method). No code or trained weights are publicly released. The iFLYTAB dataset used for ablation is from the SEMv2 paper and is not freely downloadable.

## Reproducibility

| Resource | Type | License | Link |
|:---|:---|:---|:---|
| SEMv3 paper (preprint) | Paper | arXiv-nonexclusive-v1.0 | [arXiv:2405.11862](https://arxiv.org/abs/2405.11862) |
| SEMv3 code | Code | Not released | N/A |
| SEMv3 model weights | Model | Not released | N/A |
| iFLYTAB (released with SEMv2) | Dataset | Unknown | See [SEMv2 (arXiv:2405.11757)](https://arxiv.org/abs/2405.11757) |
| WTW | Dataset | Unknown | [GitHub](https://github.com/wangwen-whu/WTW-Dataset) |
| ICDAR-2019 cTDaR Historical | Dataset | Unknown | [ICDAR 2019 competition page](https://cndplab-founder.github.io/cTDaR2019/) |
| SciTSR | Dataset | MIT | [GitHub](https://github.com/Academic-Hammer/SciTSR) |
| PubTabNet | Dataset | CDLA-Permissive-1.0 | [IBM Research / GitHub](https://github.com/ibm-aur-nlp/PubTabNet) |

### Models

- Backbone: ResNet-34 pretrained on ImageNet, with FPN producing four feature levels; fused into $\mathbf{F} \in \mathbb{R}^{H/4 \times W/4 \times 256}$.
- Feature enhancement: SCNN with 4 downsampling blocks; two separate modules per branch (row and column), each with non-shared parameters.
- KOR offset head: $1 \times 3$ convolution on concatenated keypoint features.
- Merge module: RoI Align + positional embedding; row/column self-attention; convolutional starting-grid head; merge action classification head.
- Model weights are **not publicly released**.

### Algorithms

- Optimizer: Adam, $\beta\_1 = 0.9$, $\beta\_2 = 0.999$ (defaults assumed; not specified in paper).
- Learning rate: $10^{-4}$ with cosine annealing to $10^{-6}$.
- Training: 100 epochs end-to-end.
- Sampling step $t = 32$; grid feature channel $C\_g = 512$.
- During training, ground-truth grid coordinates are used for RoI Align in the merge module (teacher-forcing).
- NMS applied to starting point predictions (algorithm matches SEMv2).

### Data

- [ICDAR-2019 cTDaR Historical](https://cndplab-founder.github.io/cTDaR2019/): 600 train / 150 test; publicly available from the ICDAR 2019 competition page. License unknown.
- [WTW](https://github.com/wangwen-whu/WTW-Dataset): 14,581 images; publicly available on GitHub. License unknown.
- iFLYTAB: 12,103 train / 5,188 test; introduced in the SEMv2 paper ([arXiv:2405.11757](https://arxiv.org/abs/2405.11757)) by iFLYTEK Research. Not independently released with SEMv3; public download availability is unclear. License unknown.
- [SciTSR](https://github.com/Academic-Hammer/SciTSR): 12,000 train / 3,000 test (SciTSR-COMP subset: 716 complex tables); publicly available under MIT license.
- [PubTabNet](https://github.com/ibm-aur-nlp/PubTabNet): approximately 500k train / 9k validation; publicly available under CDLA-Permissive-1.0.

### Evaluation

- Cell adjacency F1 at IoU 0.6 for ICDAR-2019 cTDaR, WTW, and iFLYTAB.
- Grid detection F1-G at IoU 0.9 for iFLYTAB (measures split stage independently).
- TEDS and TEDS-Struct for PubTabNet.
- Cell adjacency F1 for SciTSR-COMP.
- No error bars, confidence intervals, or multi-seed runs reported.
- LORE on WTW uses IoU 0.5 vs. SEMv3's IoU 0.6; direct comparison is not fair and the paper acknowledges this.
- SciTSR rectification caveat acknowledged but not quantified.

### Hardware

- Training: 4 Nvidia Tesla V100 GPUs, 24 GB each.
- Framework: PyTorch 1.7.1.
- Inference timing reported on SciTSR at 512×512; no absolute latency figures (only relative IS vs. KOR comparison from a chart).
- GPU-hours, memory at inference, and cost are not reported.

## BibTeX

```bibtex
@inproceedings{qin2024semv3,
  title={SEMv3: A Fast and Robust Approach to Table Separation Line Detection},
  author={Qin, Chunxia and Zhang, Zhenrong and Hu, Pengfei and Liu, Chenyu and Ma, Jiefeng and Du, Jun},
  booktitle={Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence},
  pages={1191--1199},
  year={2024},
  url={https://www.ijcai.org/proceedings/2024/132}
}
```