# ClusterTabNet: Supervised Clustering for Table Detection and Structure Recognition > ClusterTabNet frames table detection and structure recognition as supervised clustering over OCR word boxes, predicting an adjacency matrix via a compact transformer encoder with binary cross-entropy loss. - Authors: Hunter Heidenreich - Published: 2024-02-12 - Updated: 2026-03-23 - Paper: Preprint (arXiv) - Paper: Published (ICDAR 2024) - Code: ClusterTabNet Code - Source: Bevaya Labs, --- ## TL;DR ClusterTabNet reframes table detection and table structure recognition as a supervised clustering problem over OCR word tokens. A compact transformer encoder predicts adjacency matrices indicating which word pairs belong to the same table, row, column, cell, or header. The resulting model has roughly 5M parameters (excluding embeddings), achieves detection accuracy on par with or better than DETR on PubTables-1M, and is inherently robust to rotated or skewed documents. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The paper proposes a new architecture and training formulation for TD and TSR. The primary contribution is the supervised clustering objective applied to transformer encoders, with an ablation study over model variants and comparisons against DETR and Faster R-CNN baselines. **Secondary:** $\Psi\_{\text{Evaluation}}$: The paper includes careful discussion of metric comparability, notes reproducibility challenges with the DETR baseline checkpoint versus retrained weights, and cross-dataset evaluation on PubTables-1M, FinTabNet, PubTabNet, and ICDAR-2019 cTDaR. ## What is the motivation? Standard image-based table detection and structure recognition approaches treat these tasks as object detection problems, applying models like DETR or Faster R-CNN to raw document images. This has several practical drawbacks. Large vision models are expensive to run, often requiring high-resolution inputs. They must implicitly re-learn OCR to identify text cues like "Table" captions. And bounding-box outputs fail gracefully on rotated or skewed scans where axis-aligned boxes do not represent the actual table region. Multi-modal approaches inspired by LayoutLM take OCR word boxes as input but are primarily designed for classification tasks rather than clustering tasks like table structure recovery. The authors observe that both detection (grouping words into tables) and structure recognition (partitioning words into rows, columns, cells, and headers) are fundamentally clustering problems: there is no intrinsic feature distinguishing one table from another on the same page, so supervised clustering (learning cluster membership from labeled examples) is the right abstraction. ## What is the novelty? **Supervised clustering via adjacency matrices.** Given $n$ words on a page produced by an OCR engine, ClusterTabNet predicts a binary $n \times n$ adjacency matrix per clustering target. Position $(i, j)$ equals 1 if and only if words $i$ and $j$ belong to the same cluster (table, cell, row, column, or header). Clusters correspond to connected components of the resulting graph, and connected-components post-processing enforces transitivity to correct isolated missed edges. **Symmetric vs. directed adjacency for spanning cells.** For table membership and cell membership, the adjacency relation is symmetric. For rows, columns, and headers, the paper uses directed (asymmetric) adjacency to represent spanning cells: a word in a spanning cell points to all words in the rows it spans, but those words do not point back to it, enabling clean recovery of the spanning structure. **Architecture.** The model uses a standard 4-layer transformer encoder with $d\_{\text{model}} = 256$ and $d\_{\text{ff}} = 1024$. Each OCR word is represented by summing: - Learned embeddings of its normalized word content (vocabulary of 30,015 entries) - Learned embeddings of its bounding-box coordinates, quantized to integers in $[0, 1023]$ Optionally, a small ResNet-like CNN processes a $32 \times 32$ image crop around the word bounding box (4 blocks with 16, 32, 48, 80 channels), producing a $d\_{\text{model}}$-dimensional vector added to the token embedding. Each output head splits the transformer output of shape $(N, L, D)$ into two tensors $Q$ and $K$ of shape $(N, L, D/2)$, and computes: $$\hat{A} = \sigma(QK^T)$$ where $\sigma$ is the sigmoid function. The training loss is elementwise binary cross-entropy between $\hat{A}$ and the ground-truth adjacency matrix, summed over unpadded positions: $$\mathcal{L} = -\sum\_{i,j} \left[ A\_{ij} \log \hat{A}\_{ij} + (1 - A\_{ij}) \log(1 - \hat{A}\_{ij}) \right]$$ **Inference post-processing.** To enforce symmetry and recover missed edges, the predicted matrix is symmetrized before thresholding: $$\frac{1}{2}\left(\sigma(QK^T) + \sigma(KQ^T)\right) \ge k$$ The threshold $k$ is selected on a validation set (empirically $k \approx 0.7$ maximizes Dice score). Connected components are then computed on the thresholded graph. For spanning cells, a second pass identifies weak (one-directional) connections above a fixed 0.5 threshold. **Word normalization for language robustness.** Word contents are normalized to a 4-character alphabet (uppercase $\to$ 'A', lowercase $\to$ 'a', digits $\to$ '1', other $\to$ ',') before vocabulary lookup, making the dictionary compact and language-agnostic without subword tokenization. ## What experiments were performed? **Datasets.** Training uses PubTables-1M (primary; provides ground-truth word coordinates, avoiding OCR noise), FinTabNet, PubTabNet, and SynthTabNet. The model is evaluated on PubTables-1M, FinTabNet, PubTabNet, and ICDAR-2019 cTDaR (modern track only). Test sets are capped at 10,000 documents for runtime. **Training details.** Adam optimizer; learning rate $10^{-4}$ for $100 \times 5000$ steps then $10^{-5}$ for $100 \times 5000$ steps; batch size 8; maximum sequence length 1,000 words. **Metrics.** Dice score on adjacency matrices is used for validation and threshold selection. Standard COCO-style $\text{AP}@[\text{IoU}=0.50]$ and $\text{AP}@[0.50:0.95]$ are computed for final evaluation, with a caveat: ground-truth boxes are shrunk to the tight bounding box around words before scoring, making the numbers not directly comparable to published DETR results. **Baseline.** The Table Transformer DETR from PubTables-1M (Smock et al., CVPR 2022) is the main comparison target, evaluated both with the authors' published GitHub checkpoint and with a model retrained by the ClusterTabNet authors following the same code, revealing a reproducibility gap (AP 0.948 checkpoint vs. 0.888 retrained for TSR). **Ablation.** Variants explore the number of transformer layers (2, 4, 8), $d\_{\text{model}}$ (128, 256, 512), output head dimension $C\_{\text{out}}$ (300, 1000), and the presence or absence of image patches. ## What are the outcomes/conclusions? **Table detection.** On PubTables-1M, ClusterTabNet with image patches achieves AP 0.989 and AR 0.994, outperforming both the published DETR checkpoint (AP 0.966, AR 0.981) and the retrained DETR (AP 0.949). **Table structure recognition (4 classes: cells, rows, columns, headers).** On PubTables-1M, ClusterTabNet with image patches reaches AP 0.931, AP50 0.972, AR 0.952. The published DETR checkpoint achieves AP 0.948 (best), and the retrained DETR reaches AP 0.888. Image patches contribute roughly +0.04 AP versus the text-only variant. **Cross-dataset TSR** (single model trained on all four datasets): AP 0.924 (PubTables-1M), 0.792 (FinTabNet), 0.921 (PubTabNet), 0.746 (ICDAR-2019). **Ablation findings.** More transformer layers or larger hidden dimension provide negligible gains beyond the 4-layer, $d\_{\text{model}}=256$ baseline. The smallest tested variant (2-layer, $d\_{\text{model}}=128$, 2M non-embedding parameters) still achieves AP 0.912, while the full model uses 5M non-embedding parameters versus roughly 29M for DETR. **Key limitations.** The primary limitation of the AP metric comparison is the word-shrinking normalization applied to ground-truth boxes, which makes numbers incomparable with published DETR results. The paper does not evaluate on TEDS, the standard metric for PubTabNet and FinTabNet, so broader comparison to other TSR methods is difficult. Spanning cell recovery relies on a fixed hyperparameter (0.5 threshold) due to data sparsity. The model requires OCR output at inference time, coupling it to an upstream OCR engine whose choices (padding, orientation) affect results. ## Reproducibility ### Models - Architecture: 4-layer transformer encoder, $d\_{\text{model}} = 256$, $d\_{\text{ff}} = 1024$, $C\_{\text{out}} = 300$; optional 4-block ResNet CNN for image patches; approximately 15M total parameters (with patches), 5M excluding embeddings. - Code is released at [https://github.com/SAP-samples/clustertabnet](https://github.com/SAP-samples/clustertabnet) under Apache-2.0. A pretrained checkpoint (`model_weights/table_recognition.pth`, ~29 MB) is distributed in the repository and is covered by the same Apache-2.0 license. - No separate model card is provided; the README references the checkpoint directly for inference. ### Algorithms - Adam optimizer; learning rate $10^{-4}$ for 100 epochs of 5,000 steps, then $10^{-5}$ for 100 more epochs. - Batch size 8; maximum sequence length 1,000 words. - Threshold $k \approx 0.7$ selected via Dice score on validation set. - No gradient clipping, mixed precision, or other training tricks reported. ### Data - PubTables-1M (training + evaluation): ~947k table instances; word coordinates provided directly. - FinTabNet, PubTabNet, SynthTabNet: used for training; OCR run with an internal SAP engine (not public). - ICDAR-2019 cTDaR (modern track): evaluation only. - Data preprocessing: JSON format; FinTabNet header labels masked during training. - Test sets capped at 10,000 documents for runtime; authors report negligible accuracy difference between capped and full sets. ### Evaluation - $\text{AP}@[0.50:0.95]$, $\text{AP}@50$, and AR via torchvision CocoEvaluator. - Ground-truth boxes shrunk to tight word-bounding-box before scoring; results are **not** directly comparable to published detection-based TSR numbers. - Spanning cells and row headers excluded from 4-class comparison with DETR. - Dice score used only for validation and threshold selection, not as a final reported metric. - No error bars, significance tests, or multiple training run statistics reported. ### Hardware - DETR baseline was retrained for 1 week on a single V100 GPU for reference. - No explicit hardware specification given for ClusterTabNet training. - Inference hardware: not reported; model is compact (~15M parameters) and OCR-based, suggesting CPU-feasible inference is plausible. ## BibTeX ```bibtex @inproceedings{polewczyk2024clustertabnet, title={ClusterTabNet: Supervised clustering method for table detection and table structure recognition}, author={Polewczyk, Marek and Spinaci, Marco}, booktitle={International Conference on Document Analysis and Recognition}, year={2024} } ```