# TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism

> A research note on TFLOP, an IJCAI 2024 table structure recognition framework from Upstage that replaces bounding box prediction and matching with a direct layout pointer mechanism, eliminating post-processing misalignment issues in image-to-text TSR pipelines.

- Authors: Bevaya AI Team
- Published: 2025-01-21
- Updated: 2026-03-18
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2501.11800>
- Paper: Published (IJCAI 2024) <https://doi.org/10.24963/ijcai.2024/105>
- Code: TFLOP Code <https://github.com/UpstageAI/TFLOP>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

TFLOP replaces the conventional dual-decoder TSR pipeline (predict cell bounding boxes, then match them to OCR text regions) with a layout pointer mechanism: text region bounding boxes are encoded as context from the start, and the decoder directly associates each predicted table data tag with the corresponding text region. An additional span-aware contrastive supervision improves recognition of tables with row and column spans. On PubTabNet, FinTabNet, and SynthTabNet, TFLOP achieves the highest reported TEDS scores at the time of submission, and the authors demonstrate practical utility on watermarked and non-English industrial documents.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$ The headline contribution is the TFLOP architecture: the layout pointer mechanism and span-aware contrastive supervision. The paper devotes most pages to the model design, ablations, and benchmark comparisons.

**Secondary:** None. No dataset is released (the Korean evaluation set is small and kept in the supplementary; the watermark dataset is synthesized from FinTabNet and not distributed).

## What is the motivation?

Recent image-to-text TSR systems follow a dual-decoder pattern: one decoder generates HTML or OTSL structure tokens, and a second decoder predicts cell bounding boxes conditioned on those tokens. The bounding boxes are then matched to text regions obtained from an OCR engine or PDF parser to produce the complete table.

The matching step requires carefully calibrated post-processing heuristics to pair predicted bounding boxes with the correct text regions. When a predicted box does not precisely align with the true cell boundary, the matched text may be wrong or incomplete, degrading the final table structure. The gap between TEDS-Struct (structure only) and TEDS (structure + content) scores in prior work is a direct indicator of this misalignment problem.

TFLOP's premise is that text region bounding boxes are already available (from OCR or PDF parsing) before inference. Rather than predicting those boxes and then matching them, the model can treat the available boxes as input context and directly point to them from the generated structure tokens, skipping the matching stage entirely.

## What is the novelty?

### Overall Architecture

TFLOP comprises four modules operating on an input table image and its associated text region bounding boxes:

1. **Image Encoder (Swin Transformer):** Produces visual patch features $\{z\_i\}\_{i=1}^{P} \in \mathbb{R}^{d}$.
2. **Layout Encoder (MLP):** Embeds each text region bounding box combined with $2 \times 2$ RoI Align features from $\{z\_i\}$ into layout embeddings $\{l\_j\}\_{j=1}^{B} \in \mathbb{R}^{d}$.
3. **Logical Structure Decoder (BART):** Auto-regressively generates a sequence of $T$ table tags $\{y\_k\}\_{k=1}^{T} \in \mathbb{R}^{v}$, conditioned on both visual features (cross-attention) and layout embeddings (context prompt). Tags are in OTSL format, which has a 1-to-1 mapping with HTML.
4. **Layout Pointer:** Associates the decoder's last hidden states with the text region bounding boxes to resolve which text belongs to which cell.

Feature dimension $d = 1{,}024$; input resolution $768 \times 768$; output sequence length $N = 1{,}376$.

### Layout Pointer Mechanism

The decoder's final hidden states $\{h\_i\}\_{i=1}^{N}$ are split into bounding box features $\{b\_j\}\_{j=1}^{B}$ and tag features $\{t\_k\}\_{k=1}^{T}$. Both are linearly projected:

$$
b\_j = \text{proj}\_b(b\_j), \quad t\_k = \text{proj}\_t(t\_k)
$$

For bounding boxes that are not empty, the pointer loss is an InfoNCE-style objective over all data tag positions in set $D$:

$$
L\_{\text{ptr}} = -\frac{1}{B} \sum\_{j=1}^{B} \log \frac{\exp(b\_j \cdot t\_{k=j} / \tau)}{\sum\_{k' \in D} \exp(b\_j \cdot t\_{k'} / \tau)}
$$

where $k=j$ is the index of the data tag corresponding to the $j$-th bounding box and $\tau = 0.1$ is the temperature.

Empty data cells (no corresponding bounding box) receive a separate BCE loss using a special learnable embedding $b\_0$:

$$
L\_{\text{ptr}}^{\text{empty}} = \frac{1}{|D|} \sum\_{k' \in D} \text{BCE}\!\left(\sigma(b\_0 \cdot t\_{k'}),\; I(k')\right)
$$

where $I(k')$ is 1 if data tag $k'$ has no associated bounding box.

### Span-aware Contrastive Supervision

To improve recognition of tables with row or column spans, TFLOP applies contrastive supervision across bounding box embeddings. For the $j$-th bounding box projected to $\hat{b}\_j = \text{proj}\_s(b\_j)$:

$$
L\_{\text{contr},j} = -\sum\_{p \in P(j)} \frac{c\_p(j)}{|P(j)|} \log \frac{\exp(\hat{b}\_j \cdot \hat{b}\_p / \tau)}{\sum\_{a \in A(j)} \exp(\hat{b}\_j \cdot \hat{b}\_a / \tau)}
$$

where $A(j)$ is all bounding boxes except $j$, and $P(j)$ is the set of positive samples (same row or column as $j$). The span coefficient $c\_p(j)$ weighs each positive pair by the degree of span overlap:

$$
c\_p(j) = \frac{(\text{overlap}(p, j))^2}{\text{span}(p) \cdot \text{span}(j)}
$$

$\text{span}()$ is the row or column span count; $\text{overlap}(p,j)$ is the number of rows or columns shared between $p$ and $j$. This formulation assigns higher weight to cells that overlap more completely, and lower weight to partially overlapping cells in multi-span scenarios.

### Overall Loss

The training objective combines five terms:

$$
L = \lambda\_1 L\_{\text{cls}} + \lambda\_2 L\_{\text{ptr}} + \lambda\_3 L\_{\text{ptr}}^{\text{empty}} + \lambda\_4 L\_{\text{contr}}^{\text{row}} + \lambda\_5 L\_{\text{contr}}^{\text{col}}
$$

where $\lambda\_1 = \lambda\_2 = \lambda\_3 = 1$ and $\lambda\_4 = \lambda\_5 = 0.5$. $L\_{\text{cls}}$ is the cross-entropy loss for tag classification.

## What experiments were performed?

### Datasets

- **PubTabNet**: 500,777 train / 9,115 validation / 9,064 test. Tables from scientific articles; cell-level annotations available for train and val. For the test set, text regions were obtained using PSENet + MASTER (same OCR pipeline as prior work).
- **FinTabNet**: 112,887 tables from financial reports; cell-level annotations provided via PDF parsing (no OCR noise).
- **SynthTabNet**: 600,000 synthetic tables with diverse styles; cell-level annotations provided.

### Baselines

PubTabNet: TableMaster, LGPMA, TableFormer, VAST, RobusTabNet, DRCC. FinTabNet and SynthTabNet: TableFormer, GridFormer, VAST, DRCC.

### Ablation (Table 3)

Four configurations on PubTabNet test TEDS (%):

| Config | Simple | Complex | All |
|:---|:---:|:---:|:---:|
| TFLOP\_BASE | 97.92 | 94.85 | 96.42 |
| + Image ROI (I) | +0.04 | +0.14 | +0.08 |
| + I + Uniform contrastive (U) | +0.01 | +0.12 | +0.06 |
| + I + Span-aware contrastive (S) = TFLOP\_FULL | **+0.14** | **+0.35** | **+0.24** |

The gain from span-aware over uniform contrastive is concentrated in complex tables (+0.35 vs +0.12 for complex), supporting the claimed benefit for spanning cells.

### Industrial Experiments

**Watermark TSR:** A watermarked FinTabNet set was synthesized by inpainting 20 candidate texts per image (20% probability per box). TFLOP extended with a 2-layer MLP filter (Layout Filter) achieves 99.54% TEDS-S / 99.41% TEDS vs. TableMaster's 82.18% / 72.83%.

**Cross-lingual TSR:** 30 manually annotated Korean financial report tables (15 simple, 15 complex) and 175 QA pairs. TFLOP trained on English PubTabNet generalizes to Korean: 95.76% / 89.41% simple/complex TEDS vs. GPT-4V 79.43% / 68.39% and TableMaster 89.96% / 83.94%.

### Implementation

- Image encoder: Swin Transformer.
- Structure decoder: BART (configured similarly to Donut).
- Input resolution: $768 \times 768$.
- Output sequence length: $N = 1{,}376$; bounding box context length $B = 640$ for PubTabNet/FinTabNet, $864$ for SynthTabNet.
- Optimizer: AdamW (assumed; not stated explicitly), learning rate $8 \times 10^{-5}$ with cosine scheduling over 250K steps.
- Hardware: 4 Nvidia A100 GPUs.

## What are the outcomes/conclusions?

**PubTabNet (val):** TFLOP\_FULL reaches 98.3% TEDS-S and 98.0% TEDS, ahead of DRCC (98.9% TEDS-S, 97.8% TEDS) on TEDS-S but better on TEDS. Note: DRCC reports no TEDS on the test set.

**PubTabNet (test):** 98.38% TEDS-S, 96.66% TEDS. The TEDS-Struct to TEDS gap for TFLOP\_FULL (0.11 on FinTabNet) is substantially smaller than for prior dual-decoder methods (0.42 for comparison), directly reflecting the reduction in bounding box misalignment.

**FinTabNet:** 99.56% TEDS-S, 99.45% TEDS, ahead of VAST (98.63% TEDS-S, 98.21% TEDS).

**SynthTabNet:** 99.42% TEDS-S, 99.40% TEDS, ahead of DRCC (98.70% TEDS-S).

**Limitations acknowledged:** The Korean TSR evaluation uses only 30 tables (15 simple, 15 complex), which is too small for reliable statistical conclusions. The watermark dataset is synthesized; performance on natural watermarks may differ. TFLOP requires text region bounding boxes as input, which means it is not a fully end-to-end system and depends on an upstream OCR pipeline. For datasets without cell-level annotations (like PubTabNet test), OCR quality affects downstream TEDS scores.

**Unacknowledged gaps:** No comparison with non-English training baselines; the Korean experiment only tests zero-shot transfer. No latency or throughput measurements. The optimizer is not stated in the main paper. No error bars or multi-seed runs.

## Reproducibility

| Resource | Type | License | Link |
|:---|:---|:---|:---|
| Preprint (arXiv) | Paper | CC-BY-4.0 | [arXiv 2501.11800](https://arxiv.org/abs/2501.11800) |
| Published (IJCAI 2024) | Paper | Unknown | [DOI 10.24963/ijcai.2024/105](https://doi.org/10.24963/ijcai.2024/105) |
| TFLOP Code | Code | CC-BY-NC-4.0 | [GitHub](https://github.com/UpstageAI/TFLOP) |
| Model weights | Model | Unknown | Not confirmed released |
| PubTabNet | Dataset | CDLA-Permissive-1.0 | [GitHub](https://github.com/ibm-aur-nlp/PubTabNet) |
| FinTabNet | Dataset | CDLA-Permissive-1.0 | [IBM Research](https://developer.ibm.com/exchanges/data/all/fintabnet/) |
| SynthTabNet | Dataset | CDLA-Permissive-1.0 | [GitHub](https://github.com/IBM/SynthTabNet) |

### Models

- Image encoder: Swin Transformer (configuration matches Donut; exact size not specified in paper).
- Structure decoder: BART architecture; feature dimension $d = 1{,}024$.
- Layout encoder: MLP with $2 \times 2$ RoI Align; output dimension $d = 1{,}024$.
- Layout pointer: two linear projection heads for $b\_j$ and $t\_k$; one additional projection $\text{proj}\_s$ for contrastive.
- Watermark extension: 2-layer MLP (Layout Filter) with BCE and sigmoid.
- Model weights: the GitHub repository exists (license: CC-BY-NC-4.0); weight availability should be checked directly.

### Algorithms

- Learning rate: $8 \times 10^{-5}$ with cosine scheduling.
- Training steps: 250K.
- Hardware: 4 Nvidia A100 GPUs.
- Input resolution: $768 \times 768$.
- $N = 1{,}376$ (max sequence length); $B = 640$ (PubTabNet/FinTabNet), $B = 864$ (SynthTabNet).
- Temperature $\tau = 0.1$ for both pointer and contrastive losses.
- Loss weights: $\lambda\_1 = \lambda\_2 = \lambda\_3 = 1$; $\lambda\_4 = \lambda\_5 = 0.5$.
- OTSL tokenization used (from Lysak et al., 2023; arXiv:2305.03393).
- Optimizer not explicitly stated in the paper text.

### Data

- PubTabNet: 500,777 train / 9,115 val / 9,064 test; license CDLA-Permissive-1.0. Test set has no cell-level annotations; OCR (PSENet + MASTER) used.
- FinTabNet: 112,887 tables; PDF-parsed cell annotations; license CDLA-Permissive-1.0.
- SynthTabNet: 600,000 synthetic tables; cell-level annotations; license CDLA-Permissive-1.0.
- Korean tables: 30 manually annotated tables + 175 QA pairs; not released.
- Watermark dataset: synthesized from FinTabNet (not released); synthesis procedure described in Appendix B.

### Evaluation

- TEDS and TEDS-Struct as primary metrics (tree-edit-distance similarity on HTML trees).
- PubTabNet: both val and test splits reported.
- Ablation on PubTabNet test TEDS.
- Watermark evaluation: custom synthesized dataset.
- Korean evaluation: 30 tables; tiny set, no confidence intervals.
- No error bars, significance tests, or multi-seed runs.

### Hardware

- Training: 4 Nvidia A100 GPUs.
- Training time: 250K steps; approximate GPU-hours not reported.
- Inference latency and memory requirements not reported.
- No deployment or cost estimates.

## BibTeX

```bibtex
@inproceedings{khang2024tflop,
  title={TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism},
  author={Khang, Minsoo and Hong, Teakgyu},
  booktitle={Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence},
  pages={947--955},
  year={2024},
  doi={10.24963/ijcai.2024/105}
}
```