# TableFormer and SynthTabNet: Transformer-Based Table Structure Recognition with a Synthetic Dataset

> A research note on TableFormer, a transformer-based end-to-end model for table structure recognition and cell bounding box prediction, and SynthTabNet, the 600k synthetic table dataset introduced alongside it.

- Authors: Bevaya AI Team
- Published: 2022-03-02
- Updated: 2026-03-17
- Dataset: SynthTabNet Dataset <https://github.com/IBM/SynthTabNet>
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2203.01017>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

TableFormer is an end-to-end transformer-based model that simultaneously predicts table HTML structure and per-cell bounding boxes from a table image. Alongside the model, the authors release SynthTabNet, a 600k synthetic table dataset with diverse styles and complexity. On PubTabNet, TableFormer reaches 96.75% TEDS overall, compared to 89.9% for EDD, the previous best-reported result on that benchmark.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$ The headline contribution is the TableFormer architecture, which replaces LSTM decoders with transformer decoders and introduces a dedicated cell bounding-box decoder. The bulk of the paper is devoted to describing the model, training procedure, and benchmark comparisons.

**Secondary:** $\Psi\_{\text{Resource}}$ The paper also releases SynthTabNet, a 600k synthetic dataset created to address distribution imbalances in existing benchmarks. The dataset is a meaningful contribution in its own right and is described in detail across several pages.

## What is the motivation?

Extracting structured data from table images is a prerequisite for many downstream applications: search engines, knowledge graphs, and document automation pipelines. The problem is hard because tables vary widely in shape, size, line style, header complexity, and cell content.

At the time of this work, the leading approaches for table structure recognition fell into three families: image-to-text networks (including the LSTM-based EDD model from PubTabNet), graph neural networks, and hybrid deep-learning/rule-based pipelines. Each had significant weaknesses. Image-to-text methods relied on implicit OCR decoders trained only on the languages present in the training set, making them language-dependent. Graph-based methods required explicit text-cell positions as input and reported lower accuracy than image-to-text approaches. Hybrid methods were not end-to-end and required hand-written rules for new table types.

A secondary motivation is the observation that existing public datasets (PubTabNet, FinTabNet, TableBank) are skewed toward simple tables with few rows and columns, and toward specific visual styles. A more diverse training set is needed to improve generalization.

## What is the novelty?

### TableFormer Architecture

TableFormer takes a table image as input and outputs two things simultaneously: a sequence of HTML structure tags and a bounding box for each data cell. The model has three main components.

**CNN Backbone:** A ResNet-18 (modified by removing the final linear and pooling layers and adding an adaptive pooling layer of size $28 \times 28$) encodes the input image into a feature map. The input is resized to $448 \times 448$ pixels.

**Structure Decoder:** A standard transformer encoder-decoder architecture. The encoder has 2 layers; the decoder has 4 layers. Both use a feature size of 512, a feed-forward dimension of 1024, and 4 attention heads. The decoder autoregressively predicts HTML tags one token at a time. Replacing the LSTM with a transformer decoder is the primary upgrade over the EDD architecture from PubTabNet.

**Cell BBox Decoder:** Inspired by DETR, this component reuses the hidden state produced by the Structure Decoder for each `<td>` or `<` tag (the opening token of a spanning cell) as an object query. A cross-attention network combines this hidden state with the CNN feature map to produce a cell-specific feature. An MLP (3 layers, ReLU activations) then regresses normalized bounding box coordinates. A linear layer classifies each predicted box as empty or non-empty.

Because the HTML tag sequence is naturally ordered, the predicted bounding boxes are already in one-to-one correspondence with the table cells, eliminating the need for the Hungarian matching algorithm used in DETR.

**Loss function:** TableFormer is trained with a multi-task loss:

$$
\begin{aligned}
l\_{\text{box}} &= \lambda\_{\text{iou}} \, l\_{\text{iou}} + \lambda\_{l1} \, l\_{1} \\\\
l &= \lambda \, l\_{s} + (1 - \lambda) \, l\_{\text{box}}
\end{aligned}
$$

where $l\_{s}$ is the cross-entropy loss for the structure decoder, $l\_{\text{iou}}$ is the generalized IoU loss, $l\_{1}$ is the L1 bounding box regression loss, and $\lambda \in [0,1]$ balances the two objectives.

### SynthTabNet

To address distribution skew in existing datasets, the authors generate SynthTabNet: 600k PNG table images synthesized from four themed subsets of 150k each (FinTabNet style, marketing style, PubTabNet style, and sparse/low-density). Each subset targets a different visual style and complexity profile.

The generation pipeline controls four independent axes: table structure (header rows, body rows, columns, row/column span ratios), appearance (hand-designed CSS styling templates), content (curated term lists from PubTabNet and FinTabNet, plus random text), and rendering (a web browser engine produces exact bounding boxes per cell). Every cell, including empty ones, receives a bounding box. All splits follow an 80%/10%/10% train/test/val division.

## What experiments were performed?

### Datasets

The authors train and evaluate on three public benchmarks: PubTabNet (509k tables, scientific papers), FinTabNet (112k tables, financial documents), and TableBank (145k tables, mixed documents). They also report baseline results on SynthTabNet itself.

A data preparation step unifies all datasets into PubTabNet annotation format at 72 dpi PNG, filters out extreme-sized tables, and regenerates missing bounding boxes using a grid-inference procedure (required for 48% of simple and 69% of complex PubTabNet tables, and 68% and 98% respectively for FinTabNet).

### Metrics

**TEDS (Tree-Edit-Distance-Based Similarity)**, introduced with PubTabNet, measures structural similarity between the predicted and ground-truth HTML trees:

$$
\text{TEDS}(T\_{a}, T\_{b}) = 1 - \frac{\text{EditDist}(T\_{a}, T\_{b})}{\max(|T\_{a}|, |T\_{b}|)}
$$

where $T\_{a}$ and $T\_{b}$ are the table representations as HTML trees, EditDist is the tree-edit distance, and $|T|$ is the number of nodes.

Cell bounding box quality is measured with PASCAL VOC mAP on the non-empty cell class.

### Baselines

Structure: EDD (the encoder-dual-decoder model from the PubTabNet paper) and GTE (Global Table Extractor). Cell detection: EDD augmented with TableFormer's Cell BBox Decoder. Cell content (structure + content via PDF extraction): Tabula, Traprange, Camelot, Acrobat Pro, and EDD.

### Training Details

Three Adam optimizers are used (one per component). Starting learning rate 0.001 for 12 epochs (batch size 24), then 0.0001 for 12 more epochs or convergence (batch size 18). $\lambda = 0.5$. Dropout rate 0.5. Input constraints: images up to $1024 \times 1024$ pixels, structure token sequences up to 512 tokens.

## What are the outcomes/conclusions?

**Structure accuracy:** On PubTabNet, TableFormer achieves 98.5% TEDS on simple tables and 95.0% on complex tables (96.75% overall), compared to 91.1%/88.7% (89.9% overall) for EDD and 93.01% overall for GTE. On FinTabNet, 97.5%/96.0% (96.8% overall) vs. 88.4%/92.08% (90.6%) for EDD. On TableBank, 89.6% vs. 86.0% for EDD. On SynthTabNet, 96.9%/95.7% (96.7%).

**Cell bounding box:** TableFormer achieves 82.1% mAP on PubTabNet (86.8% with post-processing), vs. 79.2% (82.7%) for EDD augmented with the same bbox decoder.

**Structure with content (PDF extraction):** 95.4%/90.1% (93.6% overall) on PubTabNet, vs. 91.2%/85.4% (88.3%) for EDD and 80.0%/66.0% (73.0%) for Camelot.

**Language generalization:** Although trained only on English tables, TableFormer can extract Japanese-language tables by combining its bounding box predictions with native PDF text cells. This is a direct consequence of removing the language-dependent OCR decoder.

**Acknowledged limitations:** Performance degrades on tables that occupy a large portion of the page, because these are aggressively downsampled to 448 pixels during preprocessing, producing indistinguishable features. The authors suggest a separate model with a larger input resolution for such cases. No error bars or significance tests are reported. The comparison to commercial systems (Acrobat Pro, Tabula) conflates table structure recognition quality with content extraction mechanism: those tools differ from TableFormer in both dimensions, so the content-with-structure evaluation (Table 4) is not a controlled comparison.

**Gaps not acknowledged in the paper:** The paper contains no component-level ablations. There is no experiment isolating the effect of replacing the LSTM decoder with a transformer decoder separately from adding the Cell BBox Decoder, so the attribution of the performance gains to individual design choices remains unclear. Additionally, the training data mixture used for each benchmark evaluation is not explicitly stated, making it difficult to know whether results reflect single-dataset training or combined training including SynthTabNet.

## Reproducibility

| Resource | Type | License | Link |
|:---|:---|:---|:---|
| SynthTabNet Dataset | Dataset | CDLA-Permissive-1.0 | [GitHub](https://github.com/IBM/SynthTabNet) |
| Preprint | Paper | CC-BY-4.0 | [arXiv 2203.01017](https://arxiv.org/abs/2203.01017) |
| Model weights | Model | N/A | Not released |
| Training code | Code | N/A | Not released |

### Models

- ResNet-18 CNN backbone (pretrained weights from torchvision), modified for table images.
- Transformer encoder: 2 layers, feature size 512, FFN 1024, 4 heads.
- Transformer decoder: 4 layers, same dimensions.
- Cell BBox Decoder: cross-attention module + 3-layer MLP + linear classification head.
- Model weights are **not publicly released** in the paper or the SynthTabNet repository. The SynthTabNet GitHub only provides the dataset and a usage notebook.

### Algorithms

- Optimizer: Adam (3 separate instances: one per sub-network).
- Learning rate: 0.001 for 12 epochs, then 0.0001 for 12 epochs or convergence.
- Batch sizes: 24 (warm phase), 18 (fine-tuning phase).
- Dropout: 0.5 on all dropout layers.
- $\lambda = 0.5$ (balances structure vs. bbox loss).
- Inference optimization: single forward pass through encoder + caching of decoded token features for autoregressive decoding.
- Post-processing (PDF documents): 9-step pipeline to align predicted bounding boxes with PDF text cells via IOU matching, column alignment, and orphan cell recovery.

### Data

- **Training:** Combination of PubTabNet (509k), FinTabNet (112k), and TableBank (145k), preprocessed to a unified PNG format at 72 dpi. Combined-Tabnet (PubTabNet + FinTabNet) is ~400k; Combined (all three) is ~500k. SynthTabNet adds 600k synthetic tables.
- **SynthTabNet:** 600k PNG images, 4 subsets of 150k (Part 1: FinTabNet style, Part 2: marketing style, Part 3: PubTabNet style, Part 4: sparse). All splits are 80%/10%/10%. Total download size approximately 37 GB. License: CDLA-Permissive-1.0.
- **Annotation:** For PubTabNet and FinTabNet, missing bounding boxes are computed via a grid-inference procedure. Tables with non-strict HTML structures (rows of unequal column counts) are discarded.
- **Augmented dataset:** The paper mentions releasing the preprocessed combination of PubTabNet/FinTabNet/TableBank with generated bounding boxes for reproducibility, but no persistent download link is provided.

### Evaluation

- TEDS (structure-only and structure+content variants). No nonstandard variants used.
- PASCAL VOC mAP for cell bounding box evaluation (content cells only).
- No error bars, confidence intervals, or multi-seed runs are reported.
- The content evaluation (Table 4) notes that ground-truth HTML and extracted PDF text sometimes differ in whitespace and unicode representation, potentially deflating scores slightly.

### Hardware

- Framework: PyTorch and Torchvision.
- Training hardware: not specified in the paper.
- GPU-hours, memory requirements, and inference latency are not reported.
- No cost estimates provided.

## BibTeX

```bibtex
@inproceedings{nassar2022tableformer,
  title={TableFormer: Table Structure Understanding with Transformers},
  author={Nassar, Ahmed and Livathinos, Nikolaos and Lysak, Maksym and Staar, Peter},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}
```