# ComFinTab: A Compound Financial Table Dataset for End-to-End Table Understanding > ComFinTab introduces a 10k-image benchmark of compound financial tables with rich annotations for both table structure recognition and table understanding tasks, paired with the CTUNet multi-modal framework. - Authors: Hunter Heidenreich - Published: 2026-04-01 - Paper: Published (ACM MM 2022) - Dataset: ComFinTab Dataset - Code: CTUNet (DAVAR-Lab-OCR) - Source: Bevaya Labs, --- ## TL;DR ComFinTab is a 10,000-image benchmark of financial document tables, of which over 70% are compound tables: tables that integrate multiple basic sub-tables with complex, non-uniform header structures. Alongside the dataset, the authors propose CTUNet, a multi-modal graph-based framework that jointly handles table structure recognition and a new unified "table item extraction" task defined over the dataset. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: The headline contribution is the ComFinTab dataset and its associated annotation scheme. The paper devotes substantial space to collection methodology, annotation pipeline, and the formal task definition that the dataset enables. No existing public dataset provided compound-table coverage with both structure and understanding annotations for image-based tables. **Secondary:** $\Psi\_{\text{Method}}$: CTUNet is a non-trivial multi-modal framework combining Faster R-CNN cell detection, RoI-Align visual features, BERT text features, positional embeddings, and a graph attention network for joint cell classification and relation linking. It serves primarily as a strong baseline to validate the new task formulation rather than as a standalone architectural contribution. ## What is the motivation? Real-world financial documents routinely contain tables that go beyond the three basic forms (relational, entity, matrix) described in prior taxonomies. These "compound tables" integrate multiple basic sub-tables into a single grid, so cells in the same row or column may belong to entirely different semantic groups and cannot be correctly linked by simple header-chasing heuristics. Existing image-based datasets such as PubTabNet (568k), FinTabNet (113k), and SciTSR (15k) contain fewer than 2% compound tables. Digital-format table understanding datasets (WebSheet, SAUS, DeEX, Spider, WikiSQL) use spreadsheet or database representations that carry color, font, and structural metadata unavailable in scanned images. The result is a gap: no public benchmark tests visual table understanding on the challenging compound tables that appear routinely in financial filings. The paper identifies a secondary motivation: the proliferation of incompatible task formulations across prior table understanding work (table type classification, cell type classification, column type identification, entity linking, Table QA). The authors argue these tasks can largely be unified into a single "table item extraction" formulation, and use ComFinTab to instantiate and evaluate that formulation. ## What is the novelty? **Dataset.** ComFinTab contains 10,000 table images from public annual reports of companies listed on the Shanghai and Shenzhen Stock Exchanges: 6,000 Chinese-language and 4,000 English-language tables. Over 70% are compound tables. Annotations cover cell bounding boxes, row and column indices, text bounding boxes, text content, cell type (TH, LH, DA, OT), and cell linking (which data cells are linked to which header cells). This combination of structure and understanding annotations in an image-based, majority-compound-table dataset is the first of its kind. **Task formulation.** The paper proposes a "table item extraction" task. Each data (DA) cell is the root of a tree: the left subtree stores the left-header (LH) hierarchy and the right subtree stores the top-header (TH) hierarchy, with virtual header nodes added when a sub-table lacks one axis of headers. This representation encodes cell type and cell linking jointly, and can losslessly express the outputs of prior cell-type classification and cell-linking tasks for basic tables. **Evaluation metric.** The authors introduce Tree-F1-Score, which extends standard F1 with Tree-Edit-Distance-Based Similarity (TEDS) as a soft matching credit. Formally: $$ \begin{aligned} \text{Tree-R} &= \frac{\sum\_{t\_i \in T\_G} \text{TEDS}(t\_i, t\_i')}{|T\_G|} \\\\ \text{Tree-P} &= \frac{\sum\_{t\_i \in T\_P} \text{TEDS}(t\_i, t\_i')}{|T\_P|} \end{aligned} $$ $$ \text{Tree-F1} = \frac{2 \times \text{Tree-R} \times \text{Tree-P}}{\text{Tree-R} + \text{Tree-P}} $$ where $T\_P$ and $T\_G$ are predicted and ground-truth tree sets, and $t'\_i$ is the predicted tree sharing the same root DA cell as $t\_i$. Items with no corresponding root in the counterpart set score TEDS = 0. **CTUNet.** The framework fuses three feature modalities per detected cell: - **Position feature:** $PE\_i = \text{embedding}(b\_i)$ from the bounding box coordinates $b\_i = (x\_1, y\_1, x\_2, y\_2)$, $PE\_i \in \mathbb{R}^{d\_F}$. - **Visual feature:** RoI-Align crop from the Faster R-CNN feature pyramid, followed by convolutions and a linear projection, $V\_i \in \mathbb{R}^{d\_F}$. - **Textual feature:** BERT (or BERT-Chinese) encoding of cell text content, $T\_i \in \mathbb{R}^{d\_F}$. The combined cell representation is: $$F\_i = \text{LayerNorm}(PE\_i + V\_i + T\_i)$$ A graph attention network (GAT) with a masked self-attention mechanism propagates features within each cell's structural neighborhood (all cells in the same row or column). Enhanced node features $F'\_i$ are computed as: $$ \begin{aligned} e\_{ij} &= \text{LeakyReLU}(w^T [WF\_i \| WF\_j]) \\\\ \alpha\_{ij} &= \frac{\exp(e\_{ij})}{\sum\_{k \in N\_i} \exp(e\_{ik})} \\\\ F'\_i &= \sigma\!\left(\sum\_{j \in N\_i} \alpha\_{ij} W F\_j\right) \end{aligned} $$ Edge features are formed by pairwise concatenation $E\_{i,j} = [F'\_i \| F'\_j]$. Three branches are trained simultaneously: node classification (cell type, 4 classes), row relation linking (sigmoid binary), and column relation linking (sigmoid binary). The combined loss is: $$L = L\_{\text{det}} + \lambda\_1 L\_{\text{node}} + \lambda\_2 (L\_{\text{rlink}} + L\_{\text{clink}})$$ where $L\_{\text{det}}$ is the Faster R-CNN detection loss and the remaining terms are cross-entropy losses. ## What experiments were performed? The paper evaluates on ComFinTab-Chinese (4,500 train / 1,500 test) and ComFinTab-English (3,200 train / 800 test), split at company level to prevent format leakage between splits. Because no prior method directly addresses the table item extraction task, the authors construct two comparison settings: 1. **Cell Classification + Rules.** TUTA (a table-pretrained language model) and LayoutLMv2 (a multi-modal document model) are fine-tuned for cell type classification, then handcrafted adjacency rules link headers to data cells. TUTA is English-only; LayoutLMv2 is tested on both splits. 2. **End-to-End Relation Extraction.** The task is recast as triple extraction: ('DA 1', 'Left-Linking', 'LH 1') etc. AGCCN (an NLP relation extraction GCN) is adapted and augmented with the same multi-modal features used in CTUNet. All baselines and CTUNet use the same OCR source: PaddleOCR for Chinese, Tesseract for English. A ground-truth oracle variant (GT) is also reported to isolate table understanding performance from OCR errors. Table structure recognition quality (TEDS) is reported as a reference: CTUNet achieves 98.99 on English and 98.83 on Chinese, reflecting that bordered financial tables are relatively straightforward structurally once cell detection is accurate. Ablations test (a) the contribution of each modality (position, visual, semantic) and (b) neighborhood definitions in the GAT (none, 1-step, all-pairs, row/column). The row/column neighborhood setting outperforms all others on both splits. ## What are the outcomes/conclusions? On the table item extraction task (Tree-F1), CTUNet substantially outperforms rule-based approaches and modestly outperforms the end-to-end AGCCN baseline: | Method | English Tree-F1 | Chinese Tree-F1 | |:---|:---:|:---:| | TUTA + rule | 72.91 | 71.69 | | LayoutLMv2 + rule | 72.05 | 84.14 | | AGCCN | 84.67 | 86.98 | | CTUNet (ours) | **89.20** | **86.98** | | CTUNet (GT) | **90.37** | **88.90** | Rule-based methods struggle specifically on compound tables: cell classification accuracy is adequate (91-92%), but heuristic header-to-data matching fails when the same row or column spans multiple logically independent sub-tables. Ablation results confirm that all three modalities contribute. Removing position features causes the largest drop (Tree-F1 falls to roughly 31-38%), reflecting the centrality of spatial layout in table reasoning. Visual and textual features each add 5-6 points independently; together they further improve over any single-modality baseline. The GAT neighborhood ablation shows that using row/column neighbors (the structural prior) outperforms both 1-step spatial neighbors and all-pairs connections, suggesting that long-range row/column context is genuinely useful and that the structural graph provides meaningful inductive bias. **Limitations noted or apparent:** The model relies on an offline OCR engine, so OCR errors propagate into the understanding stage (the GT oracle consistently scores 1-2 points higher). The dataset is restricted to bordered, horizontally-oriented tables extracted from PDF annual reports, which is a narrower distribution than general document tables. The annotation pipeline used heuristic pseudo-labels for cell types (subsequently manually corrected), and heuristic linking rules (also manually cleaned), leaving some residual annotation noise. The authors do not report inter-annotator agreement statistics. ## Reproducibility ### Models CTUNet uses a ResNet-50 backbone with an FPN neck, instantiated within a Faster R-CNN cell detector. RoI-Align crops from the FPN map feed a small CNN stack followed by a linear projection to produce visual features. BERT-base (multilingual or Chinese variant) produces textual features. A single-layer GAT with masked self-attention constitutes the relational graph construction module. The paper does not specify the feature dimension $d\_F$ explicitly; the re-implementation in DAVAR-Lab-OCR targets the same architecture. Pre-trained model weights (gated download; access codes in the DAVAR-Lab-OCR README) are provided separately for Chinese and English variants; re-implementation numbers differ slightly from paper results (Chinese: 93.59 vs. 91.78 Cell-F1; English: 92.75 vs. 92.27). ### Algorithms Training uses SGD with momentum 0.9, weight decay $1 \times 10^{-4}$, initial learning rate $1 \times 10^{-3}$, divided by 10 every 20 epochs, batch size 4. OCR is handled offline by PaddleOCR (Chinese) or Tesseract (English) and is not part of the end-to-end training graph. Loss weights $\lambda\_1$ and $\lambda\_2$ are not stated explicitly in the paper. All experiments run on 8 Tesla V100 GPUs. ### Data ComFinTab contains 10,000 table images from public Chinese-listed company annual reports (Shanghai and Shenzhen Stock Exchanges). Split by language: 6,000 Chinese, 4,000 English. Train/test splits are 4,500/1,500 (Chinese) and 3,200/800 (English), partitioned at the company level. Annotation pipeline: (1) bordered table detection and cell-line crossing to extract cells; (2) LGPMA cell-matching strategy for row/column indices; (3) PDFPlumber for text locations and content; (4) color-based auto-labeling for cell types, refined by a text classifier for pseudo-labels, then manual correction; (5) heuristic linking rules, manually cleaned. The dataset is publicly available via a gated application process (application form required; email to the corresponding author). License: CC-BY-NC-SA-4.0 (non-commercial only). See [ComFinTab dataset page](https://davar-lab.github.io/dataset/comfintab.html). ### Evaluation Primary metric: Tree-F1-Score (TEDS-weighted precision and recall over tree representations of table items). Secondary metric: Macro-F1 for cell type classification. Table structure quality is also reported via TEDS (the standard TSR metric from PubTabNet). All baselines and CTUNet use the same OCR pipelines. Ground-truth oracle variants are reported to isolate understanding from OCR quality. No error bars, confidence intervals, or multi-seed averages are reported. ### Hardware All experiments use 8 Tesla V100 GPUs. Total training time and GPU-hours are not reported. ## BibTeX ```bibtex @inproceedings{li2022comfintab, author = {Zaisheng Li and Pengfei Li and Shiliang Pu and Yi Li and Zhanzhan Cheng and Xi Li and Qiao Liang and Yi Niu}, title = {End-to-End Compound Table Understanding with Multi-Modal Modeling}, booktitle = {Proceedings of the 30th ACM International Conference on Multimedia (MM '22)}, year = {2022}, pages = {4112--4121}, doi = {10.1145/3503161.3547885}, publisher = {ACM} } ```