# SepFormer: Coarse-to-Fine Separator Regression for Table Structure Recognition

> A research note on SepFormer, an ICDAR 2025 table structure recognition model that uses an RT-DETR backbone with dual two-stage decoder branches to regress row and column separators in a coarse-to-fine manner, achieving 25.6 FPS on average with competitive accuracy on SciTSR, PubTabNet, WTW, and iFLYTAB.

- Authors: Bevaya AI Team
- Published: 2026-03-27
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2506.21920>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

SepFormer is a real-time table structure recognition model that directly regresses row and column separator lines using a DETR-style architecture. It combines the RT-DETR backbone (ResNet-34 + hybrid encoder) with dual decoder branches, each organized as two sequential transformer decoder stacks: a coarse decoder that predicts straight single-line separators, and a fine decoder that refines P evenly sampled points from each coarse line into a line-strip. This coarse-to-fine decomposition avoids segmentation masks entirely and eliminates the ROIAlign-based merge stage found in split-and-merge systems. On four benchmarks (SciTSR-COMP, PubTabNet, WTW, iFLYTAB), SepFormer runs at 25.6 FPS on average while staying within 0.6-1.2% of the top non-real-time methods on each dataset.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$ The main contribution is the architecture: a coarse-to-fine dual decoder design for direct separator regression. The paper's structure centers on model design, ablation studies, and comparisons against SOTA on four benchmarks.

**Secondary:** None. No new dataset or benchmark is released.

## What is the motivation?

Table structure recognition (TSR) aims to reconstruct the logical layout of tables in document images as machine-readable representations. The two dominant paradigms for the split-and-merge family are segmentation-based methods (which predict pixel masks and apply post-processing to extract separator lines) and regression-based methods (which directly output line coordinates).

Segmentation-based methods such as SEMv2 and RTSR require high-resolution feature maps and additional post-processing steps to convert masks to lines. Regression-based approaches such as TSRFormer and its DQ-DETR variant already eliminate the segmentation stage, but retain a merge module relying on ROIAlign for the second stage, adding both computational cost and a dependency on the quality of the first stage.

SepFormer's goal is to collapse the split-and-merge pipeline into a single forward pass. By framing TSR purely as separator line regression using a DETR architecture, the method removes both segmentation masks and the ROIAlign merge module, targeting practical real-time deployment (the authors note their system is deployed in a commercial product).

## What is the novelty?

### Architecture overview

SepFormer builds on the RT-DETR backbone: a ResNet-34 encoder extracts multi-scale feature maps $C\_2, C\_3, C\_4$ at strides 8, 16, and 32; a hybrid encoder (one transformer encoder layer on $C\_4$ plus a cross-scale fusion module) produces enhanced feature maps $\\{M\_1, M\_2, M\_3\\}$. These are flattened and concatenated into a single sequence $M \in \mathbb{R}^{S \times C}$ with $C = 256$ channels, which is then fed into two parallel decoder branches for rows and columns respectively.

### Coarse-to-fine decoder

Each branch contains two sequential stages (Fig. 2 in the paper).

**Query selection.** A detection head operating on the full encoder sequence scores $S$ candidate line proposals and selects the top $K = 300$ positions (by classification confidence) as initial reference points for the coarse decoder. Unlike the box proposals in Deformable DETR, SepFormer uses line proposals: row proposals take the form $\\{x\_i, y\_i, x\_i + 2^{l-1} \times s, y\_i\\}$ and column proposals $\\{x\_i, y\_i, x\_i, y\_i + 2^{l-1} \times s\\}$.

**Coarse decoder.** Three deformable transformer decoder layers refine the selected queries iteratively. The output for the $k$-th query is a single-line separator $l^k = \\{(x\_1^k, y\_1^k), (x\_2^k, y\_2^k)\\}$ and a classification score $c^k$.

**Sampling.** For each accepted coarse separator, $P = 16$ reference points are evenly interpolated between its two endpoints:

$$
\text{Sampling} = \left\\{ \left(1 - \frac{t}{P}\right)(x\_1, y\_1) + \frac{t}{P}(x\_2, y\_2) \;\middle|\; t = 1, \ldots, P \right\\}
$$

**Fine decoder.** Three more deformable transformer decoder layers refine the sampled reference points, producing a line-strip $ls^k = \\{(x\_i^k, y\_i^k) \mid i = 1, \ldots, P\\}$. The fine stage reuses the same content queries and encoder memory; only the reference points change.

### Loss function

The training objective combines four terms per branch. Hungarian matching uses only single-line ($L\_1$) distance for pairing predictions with ground-truth lines:

$$
\mathcal{L}\_{\text{match}} = \sum\_{i=1}^{N\_{\text{row}}} \left( \lambda\_{\text{coord}} \|l\_{gt}^i - l^{\sigma(i)}\| + \lambda\_{\text{cls}} \, c^{\sigma(i)} \right)
$$

After matching, the full loss for the row branch is:

$$
\mathcal{L}^{\text{row}} = \lambda\_1 \mathcal{L}\_{\text{cls}}^{\text{row}} + \lambda\_2 \mathcal{L}\_{\text{angle}}^{\text{row}} + \lambda\_3 \mathcal{L}\_{\text{line}}^{\text{row}} + \lambda\_4 \mathcal{L}\_{\text{linestrip}}^{\text{row}}
$$

The angle loss uses cosine similarity between the predicted and ground-truth line direction vectors, with an additional penalty term $\|c\_{gt}^n\| \times 4$ in the denominator to increase sensitivity for short separators:

$$
\mathcal{L}\_{\text{angle}}^{\text{row}} = \frac{1}{N\_{\text{row}}} \sum\_{n=1}^{N\_{\text{row}}} \left( 1 - \frac{\cos(c\_{gt}^n, c\_n)}{\|c\_{gt}^n\| \times 4} \right)
$$

The coordinate losses $\mathcal{L}\_{\text{line}}$ and $\mathcal{L}\_{\text{linestrip}}$ are $L\_1$ distances on single-line and line-strip coordinates respectively. The total loss sums row and column branches:

$$
\mathcal{L}^{\text{sep}} = \mathcal{L}^{\text{row}} + \mathcal{L}^{\text{col}}
$$

Loss coefficients are set to $\lambda\_1 = 1, \lambda\_2 = 1, \lambda\_3 = 3, \lambda\_4 = 1$; matching coefficients to $\lambda\_{\text{cls}} = 2, \lambda\_{\text{coord}} = 3$.

## What experiments were performed?

### Datasets

- **SciTSR-COMP**: 2,885 train / 716 test complex PDF tables. Metric: cell adjacency relationship F1.
- **PubTabNet**: 500,777 train / 9,115 validation / 9,138 test (validation set used, as the test labels are not released). Metric: TEDS-Struct.
- **WTW**: 10,970 train / 3,611 test; real-world wired tables including inclined, curved, and occluded examples. Metric: cell adjacency F1.
- **iFLYTAB**: 12,104 train / 5,187 test; wired and wireless tables from digital documents and camera capture. Metric: cell adjacency F1. Ablations also run on this dataset.

### Baselines

Real-time comparison: RTSR (segmentation-based, 34.4 FPS average). Non-real-time comparisons on each benchmark include TSRFormer w/ DQ-DETR, SEMv2, SEMv3, GrabTab, LORE++, GridFormer, and TRUST among others.

### Ablation studies

Table 5 in the paper evaluates two design choices on iFLYTAB: (1) two-stage vs. one-stage decoders and (2) single-line vs. line-strip matching. A one-stage 3-layer decoder achieves 90.8% F1; one-stage 6-layer achieves 89.7% (worse, suggesting overfitting at depth). The two-stage 3-layer decoder with single-line matching reaches 93.8%, a gain of 3.0% over the one-stage 3-layer baseline. Line-strip matching degrades performance in all configurations, indicating that the coarser $L\_1$ single-line distance provides better optimization signal for the Hungarian algorithm.

Table 6 in the paper shows the angle loss contributes a marginal +0.2% F1 on iFLYTAB but improves consistency for short separators.

## What are the outcomes/conclusions?

**Speed:** SepFormer averages 25.6 FPS across all four test datasets (RTX 3060), versus RTSR's 34.4 FPS. It is the second real-time-capable method reported and the only regression-based real-time TSR system in the comparison.

**Accuracy on SciTSR-COMP:** 98.6% F1 (99.0 P / 98.2 R), within 0.7% of the best non-real-time method LORE++ (99.3%) and above RTSR (98.3%).

**Accuracy on PubTabNet:** 96.8% TEDS-Struct, 1.1 points above RTSR (95.7%) and within 0.7% of TSRFormer w/ DQ-DETR and SEMv3 (both at 97.5%).

**Accuracy on WTW:** 93.9% F1 (93.7 P / 94.2 R), above RTSR (92.9%) and within 1.2% of SEMv3 / GrabTab / LORE++ (all at 95.1%).

**Accuracy on iFLYTAB:** 93.8% F1 (94.6 P / 93.2 R), above RTSR (91.1%) and within 0.6% of SEMv3 (94.4%).

**Limitations:** The authors identify three failure modes: (1) heavily warped images where text lines exhibit varying degrees of warping within the same image; (2) two adjacent separators that are collapsed into a single predicted separator (attributed to insufficient discrimination in the $L\_1$ single-line matching criterion); (3) tables with very high row or column density where over- or under-detection occurs. No code or model weights are publicly released.

## Reproducibility

### Models

- Backbone: ResNet-34 pretrained on ImageNet. Three feature levels $C\_2$, $C\_3$, $C\_4$ at strides 8, 16, 32.
- Hybrid encoder: one transformer encoder layer (on $C\_4$ only) plus cross-scale fusion; $C = 256$ channels.
- Dual decoder branches: query selection + 3 coarse deformable decoder layers + 3 fine deformable decoder layers per branch. 8 attention heads, 4 sampling points per deformable attention.
- $K\_{\text{row}} = K\_{\text{col}} = 300$ top queries selected.
- Model weights are not publicly released.

### Algorithms

- Optimizer: not specified (implicitly AdamW or Adam for DETR-style models, but not stated).
- Learning rate: $3 \times 10^{-5}$; cosine annealing over 100 epochs (20 epochs for PubTabNet).
- Input resolution: longer side scaled to one of $\\{864, 896, 928, 960\\}$ during training (aspect ratio preserved); 896 at inference.
- Confidence thresholds: $\tau\_{\text{row}} = \tau\_{\text{col}} = 0.95$.
- $P = 16$ sampling points per separator.

### Data

- [SciTSR](https://github.com/Academic-Hammer/SciTSR): MIT license. 12,000 train / 3,000 test; SciTSR-COMP subset used (2,885 train / 716 test).
- [PubTabNet](https://github.com/ibm-aur-nlp/PubTabNet): CDLA-Permissive-1.0. ~500k train; validation set (9,115) used for evaluation.
- [WTW](https://github.com/wangwen-whu/WTW-Dataset): License unknown. 10,970 train / 3,611 test.
- [iFLYTAB](https://arxiv.org/abs/2405.11757): Introduced with SEMv2. License unknown. 12,104 train / 5,187 test.

### Evaluation

- Cell adjacency relationship F1 for SciTSR-COMP, WTW, and iFLYTAB.
- TEDS-Struct for PubTabNet (test labels unreleased; validation set used).
- No error bars, confidence intervals, or multi-seed results reported.
- FPS measured on RTX 3060; training on a single NVIDIA V100 32 GB.
- RTSR FPS figures taken from the prior paper [31]; direct head-to-head hardware comparison is approximate.

### Hardware

- Training: single NVIDIA V100 32 GB.
- Inference: NVIDIA RTX 3060.
- GPU-hours, total compute cost, and memory at inference are not reported.

## BibTeX

```bibtex
@inproceedings{nguyen2025sepformer,
  title={SepFormer: Coarse-to-fine Separator Regression Network for Table Structure Recognition},
  author={Nguyen, Nam Quan and Pham, Xuan Phong and Tran, Tuan-Anh},
  booktitle={Proceedings of the International Conference on Document Analysis and Recognition (ICDAR)},
  year={2025}
}
```