# UniTable: Towards a Unified Framework for Table Structure Recognition

> A research note on UniTable, a training framework that unifies table recognition tasks under a single language modeling objective and self-supervised visual pretraining, achieving SOTA on four of five major TR benchmarks.

- Authors: Bevaya AI Team
- Published: 2024-03-07
- Updated: 2026-03-18
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2403.04822>
- Code: UniTable Code <https://github.com/poloclub/unitable>
- Demo: UniTable Demo <https://poloclub.github.io/magic-table/>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

UniTable introduces two complementary ideas for table recognition (TR). First, a self-supervised pretraining (SSP) stage trains the visual encoder on up to 2 million unannotated tabular images via VQ-VAE masked image modeling, providing structured visual representations before any supervised data is seen. Second, all three TR tasks (table structure, cell bounding boxes, and cell content) are unified under a single language modeling objective, with a shared encoder and task decoder that operates on raw pixel inputs alone, without requiring an external PDF or OCR system. On SynthTabNet, removing SSP drops accuracy by up to 14 percentage points, and UniTable achieves SOTA on four of five major TR benchmarks.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$ The primary contribution is the UniTable training framework: the SSP stage with VQ-VAE masked image modeling and the unified language modeling objective for all TR tasks. Ablations, scaling experiments, and cross-architecture validation are all in service of establishing the framework's design choices.

**Secondary:** None. No new dataset is released; the paper evaluates on existing public benchmarks.

## What is the motivation?

Table recognition in document processing involves three interrelated subtasks: predicting the table structure (as an HTML or similar token sequence), localizing cell bounding boxes, and reading cell content. Prior work treated these as separate problems with task-specific decoders and often required external inputs such as PDF text layers or separate OCR systems. The result is fragmented pipelines that are harder to train, maintain, and generalize.

A second problem is initialization quality. Supervised TR datasets range from roughly 100k to 500k annotated examples, but unlabeled tables are abundant in documents. Standard supervised training from random or ImageNet-pretrained weights does not leverage this readily available unlabeled data, and performance on diverse table styles (colorful backgrounds, sparse content, financial tables) is correspondingly inconsistent.

UniTable addresses both problems together: a unified training objective eliminates task-specific decoder design, and SSP on unannotated tabular images provides a domain-adapted initialization before any supervised training begins.

## What is the novelty?

### Overall Architecture

UniTable has two main components: a visual encoder and a shared task decoder.

**Visual encoder:** A linear projection Transformer (ViT-style). Given input image $I \in \mathbb{R}^{H \times W \times C}$, it is divided into $N = HW/P^2$ non-overlapping patches of size $P \times P$. With $P = 16$ and $I \in \mathbb{R}^{448 \times 448 \times 3}$, the sequence length is $28 \times 28 = 784$ patch tokens.

**Task decoder:** A 4-layer Transformer decoder shared across all TR tasks. At inference, the decoder generates token sequences autoregressively using greedy decoding. Maximum sequence lengths are 512 for table structure, 1024 for cell bounding boxes, and 200 for cell content.

Two model sizes are reported:

| Variant | Params | Encoder layers | Heads | Hidden size |
|:---|:---:|:---:|:---:|:---:|
| UniTable-base | 30M | 4 | 8 | 512 |
| UniTable-large | 125M | 12 | 12 | 768 |

### Self-Supervised Pretraining (SSP)

Before supervised finetuning, the visual encoder is pretrained via masked image modeling using a VQ-VAE visual codebook.

**Visual codebook.** A VQ-VAE is trained to represent any tabular image as a sequence of discrete tokens drawn from a codebook $Z \in \mathbb{R}^{K \times D}$. The encoder $q\_\phi(z \mid I)$ maps the image to discrete tokens $z$; the decoder $p\_\psi(I \mid z)$ reconstructs the image. The VQ-VAE training objective is:

$$
\max\_{\phi, \psi} \; \mathbb{E}\_{z \sim q\_\phi(z \mid I)}\bigl[\log p\_\psi(I \mid z)\bigr]
$$

Because sampling discrete tokens is not differentiable, Gumbel-Softmax reparameterization is used during training. Two codebook sizes are evaluated: $K = 8{,}192$ for a 1M-image VQ-VAE and $K = 16{,}384$ for a 2M-image VQ-VAE.

**Pretraining objective.** Approximately 40% of patch tokens are randomly masked. The visual encoder is trained to predict the VQ-VAE codebook index of each masked patch, maximizing the log-likelihood of the masked visual tokens given the unmasked context. This is analogous to masked language modeling but applied to table images.

### Unified Finetuning Objective

During supervised finetuning, all three TR tasks are formulated as next-token prediction over vocabulary sequences. The table structure task predicts HTML tokens; the cell bbox task predicts coordinate tokens; and the cell content task predicts text tokens. A single cross-entropy loss applies across all tasks without any task-specific auxiliary objectives or detection heads. The input is always a raw pixel image; no external PDF or OCR output is required.

Prior pipelines used a detection head (Faster R-CNN or DETR) for bounding boxes alongside a separate sequence decoder for structure, and relied on PDF-extracted text for cell content. UniTable replaces all of these with a single language modeling objective over raw pixel inputs.

### Architecture Generality

The unified language modeling objective also works with a hybrid CNN-Transformer encoder (replacing the linear projection with ResNet-18), yielding results roughly comparable to the linear projection model pretrained on 1M images. The authors recommend the linear projection variant for three reasons: (1) it benefits from SSP, (2) it aligns with modern VLM architectures that share the same design, and (3) the hybrid variant does not surpass SSP-2M linear projection despite its larger parameter count.

## What experiments were performed?

### Datasets

- **ICDAR 2019 B2 Modern (IC19B2M)**: 100 modern-subset tables; evaluation by weighted average F1 (WAvg. F1) over IoU thresholds $\\{0.6, 0.7, 0.8, 0.9\\}$:

$$
\text{WAvg. F1} = \frac{\sum\_{i=1}^{4} \text{IoU}\_{i} \times \text{F1@IoU}\_{i}}{\sum\_{i=1}^{4} \text{IoU}\_{i}}
$$

- **PubTabNet**: 509k scientific article tables; three metrics: COCO $\text{AP}\_{50}$ (cell bbox), S-TEDS (structure), and TEDS (structure + content).
- **FinTabNet**: 113k S&P 500 financial report tables; evaluated on S-TEDS.
- **SynthTabNet**: 600k synthetic tables across four style subsets (Finance, PubTabNet, Marketing, Sparse); $\text{AP}\_{50}$ and S-TEDS.
- **PubTables-1M**: 947k tables with word-level bbox annotations; evaluated on COCO $\text{AP}\_{50}$ and $\text{AP}\_{75}$.

### Baselines

EDD, TableFormer, VAST, GTE, DRCC, and a DETR baseline from the PubTables-1M authors. General-purpose VLMs (GPT-4V, LLaVA-v1.6 in multiple sizes) are also evaluated on SynthTabNet in Appendix A.

### SSP Ablation (Table 2)

S-TEDS on all four SynthTabNet subsets (base and large models):

| Configuration | Finance (Base/Large) | Marketing (Base/Large) |
|:---|:---:|:---:|
| No SSP | 88.95 / 90.75 | 68.05 / 70.60 |
| SSP 1M | 98.73 / 99.56 | 95.14 / 99.05 |
| SSP 2M | 99.41 / **99.58** | 98.35 / **99.08** |

SSP 1M improves over no SSP by an average of 14.40 percentage points (pp); SSP 2M adds a further 0.74 pp. Without SSP, scaling from base to large barely helps; with SSP, large models consistently outperform base by an average of 1.51 pp.

### Implementation

- All models trained with AdamW, cosine LR schedule with linear warmup, 24 epochs.
- Teacher forcing during training; greedy decoding at inference.
- VQ-VAE pretrained on 1M (PubTabNet + SynthTabNet) or 2M (+ PubTables-1M + TableBank) unannotated tabular images.
- Finetuning: table structure trained on PubTabNet + FinTabNet + SynthTabNet; cell bbox on PubTabNet + SynthTabNet; cell content on PubTabNet + SynthTabNet + PubTables-1M.

## What are the outcomes/conclusions?

**IC19B2M:** UniTable improves WAvg. F1 over the prior best result held by GTE; the authors report the margin as large (see Table 1 of the paper). GTE was the previous leader on this benchmark.

**PubTabNet:** UniTable-base and UniTable-large both surpass VAST on $\text{AP}\_{50}$ by more than 3 pp. VAST required a task-specific visual-alignment auxiliary loss; UniTable achieves this with the unified language modeling objective alone.

**FinTabNet:** New SOTA on S-TEDS; performance scales from base to large, confirming the framework benefits from additional model capacity.

**SynthTabNet:** SOTA on both $\text{AP}\_{50}$ and S-TEDS; SSP is the largest single contributor to performance on the more challenging Marketing and Sparse subsets.

**PubTables-1M:** Results are competitive with the DETR baseline from the dataset creators, but the authors identify previously unacknowledged annotation inconsistencies that may inflate DETR's reported score (detailed in Appendix F.1).

**VLM comparison (Appendix A):** LLaVA-v1.6 variants score between 32% and 50% S-TEDS on SynthTabNet; GPT-4V reaches approximately 64-69% on the Finance and PubTabNet subsets. UniTable-large with SSP 2M scores above 99% on three of four subsets, illustrating the gap between specialist TR models and general VLMs on structured recognition tasks.

**Limitations acknowledged:** The paper is a preprint and has not undergone peer review at the time of this writing. PubTables-1M evaluation is complicated by discovered annotation issues. No latency or throughput measurements are reported. The cross-dataset finetuning setup (combining multiple training sets) complicates direct comparison with baselines trained on single datasets. Learning rate, batch size, and exact hardware are not stated in the main paper.

**Unacknowledged gaps:** No multi-seed error bars or confidence intervals. No comparison with methods that use OCR-augmented inputs on a common benchmark. The VQ-VAE training details (optimizer, LR, hardware, training time) are not disclosed.

## Reproducibility

| Resource | Type | License | Link |
|:---|:---|:---|:---|
| Preprint (arXiv) | Paper | CC-BY-4.0 | [arXiv 2403.04822](https://arxiv.org/abs/2403.04822) |
| UniTable Code | Code | MIT | [GitHub](https://github.com/poloclub/unitable) |
| UniTable Demo | Demo | MIT | [magic-table](https://poloclub.github.io/magic-table/) |
| PubTabNet | Dataset | CDLA-Permissive-1.0 | [GitHub](https://github.com/ibm-aur-nlp/PubTabNet) |
| FinTabNet | Dataset | CDLA-Permissive-1.0 | [IBM Research](https://developer.ibm.com/exchanges/data/all/fintabnet/) |
| SynthTabNet | Dataset | CDLA-Permissive-1.0 | [GitHub](https://github.com/IBM/SynthTabNet) |
| PubTables-1M | Dataset | MIT | [GitHub](https://github.com/microsoft/table-transformer) |

### Models

- Visual encoder: linear projection Transformer (ViT-style), patch size $P = 16$, input resolution $448 \times 448$.
- Task decoder: 4 Transformer decoder layers, shared across all three TR tasks.
- UniTable-base: 30M parameters; 4 encoder layers, 8 attention heads, hidden size 512.
- UniTable-large: 125M parameters; 12 encoder layers, 12 attention heads, hidden size 768.
- VQ-VAE codebook: $K = 8{,}192$ entries for 1M pretraining; $K = 16{,}384$ for 2M.
- Model weights: the GitHub repository exists; availability of pretrained checkpoints should be verified directly. A finetuned demo model is accessible via the HuggingFace-hosted public API.

### Algorithms

- Optimizer: AdamW.
- LR schedule: cosine with linear warmup (learning rate value not stated in main paper).
- Training duration: 24 epochs for all supervised finetuning experiments.
- Teacher forcing during training; greedy decoding at inference.
- Masking ratio for SSP: approximately 40%.
- Gumbel-Softmax reparameterization for VQ-VAE token sampling.
- VQ-VAE training details (LR, batch size, hardware, epochs) are not reported.

### Data

- VQ-VAE 1M pretraining: PubTabNet + SynthTabNet (unannotated tabular images).
- VQ-VAE 2M pretraining: + PubTables-1M + TableBank.
- Supervised finetuning data is task-dependent (see main results section).
- All five evaluation datasets are publicly available; licenses vary (see table above).
- TableBank license: see the original repository; not confirmed in this note.

### Evaluation

- CAR F1 at multiple IoU thresholds (IC19B2M); COCO $\text{AP}\_{50}$/$\text{AP}\_{75}$ (cell bbox); S-TEDS and TEDS (structure and content).
- WAvg. F1 is a weighted average over four IoU thresholds as defined in the ICDAR 2019 competition.
- No error bars, significance tests, or multi-seed runs reported.
- Baselines are from published papers; not all use the same training data as UniTable, so comparisons may not reflect equal supervision.

### Hardware

- Training hardware, GPU count, and training time are not reported in the main paper.
- Inference hardware and latency are not reported.
- No cost estimates or deployment considerations are provided.

## BibTeX

```bibtex
@article{peng2024unitable,
  title={UniTable: Towards a Unified Framework for Table Recognition via Self-Supervised Pretraining},
  author={Peng, ShengYun and Chakravarthy, Aishwarya and Lee, Seongmin and Wang, Xiaojing and Balasubramaniyan, Rajarajeswari and Chau, Duen Horng},
  journal={arXiv preprint arXiv:2403.04822},
  year={2024}
}
```