# MTL-TabNet: End-to-End Multi-Task Learning for Image-Based Table Recognition > MTL-TabNet frames table recognition as a multi-task learning problem, with a shared encoder-decoder backbone feeding three task-specific decoders for structure, cell detection, and cell content recognition simultaneously. - Authors: Hunter Heidenreich - Published: 2026-03-23 - Paper: Preprint (arXiv) - Paper: Published (VISAPP 2023) - Code: MTL-TabNet Code - Model: MTL-TabNet Weights (PubTabNet) - Model: MTL-TabNet Weights (FinTabNet) - Source: Bevaya Labs, --- ## TL;DR MTL-TabNet proposes a single encoder-decoder model that jointly learns table structure recognition, cell bounding box detection, and cell content recognition as three coupled tasks. On PubTabNet and FinTabNet, it reported improvements over TableFormer on all three sub-tasks, achieving competitive results with the top entries in the ICDAR 2021 competition without ensembling or additional annotations. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The paper's core contribution is the architecture: a multi-task learning network that jointly handles three aspects of table recognition through shared and task-specific decoder components. The validation is comparison tables against strong baselines with ablation logic implicit in the sub-task decomposition. **Secondary:** None significant. No new dataset, metric, or theoretical derivation is introduced. ## What is the motivation? Most table recognition pipelines at the time of publication were non-end-to-end: a separate system would handle table structure recognition, and a separate OCR system would handle cell content recognition. These two stages were trained independently, so errors from one stage could not be corrected by the other, and the systems could not share visual representations. Prior end-to-end attempts (EDD, IM2TEX) existed but lagged substantially behind non-end-to-end methods. The authors argue that multi-task learning is a natural fit for table recognition because structure, cell layout, and cell content are interdependent. ## What is the novelty? The core architectural contribution is a five-component design: 1. **Shared Encoder:** ResNet-31 with Multi-Aspect Global Context Attention (GCAttention) after each residual block, producing a spatial feature grid that is unrolled column-by-column and passed through positional encoding. 2. **Shared Decoder:** Two standard Transformer decoder layers that consume the encoder output and a right-shifted sequence of structural HTML tokens, producing a context vector for each predicted cell token. 3. **Structure Decoder:** One Transformer decoder layer predicting an HTML tag sequence for the table structure. Cells are tokenized at the HTML tag level; spanning cells are decomposed into `` tokens. 4. **Cell-BBox Decoder:** Triggered whenever the structure decoder emits a cell-start token. Takes the corresponding shared-decoder output and predicts four bounding box coordinates via a linear layer and sigmoid activation. 5. **Cell-Content Decoder:** Similarly triggered per cell. Autoregressively generates the text content of that cell at character level, attending to both the shared encoder output and the per-cell shared decoder representation. The overall loss is: $$ \mathcal{L} = \lambda\_{1}\mathcal{L}\_{\text{struc.}} + \lambda\_{2}\mathcal{L}\_{\text{cont.}} + \lambda\_{3}\mathcal{L}\_{\text{bbox}} $$ where $\mathcal{L}\_{\text{struc.}}$ and $\mathcal{L}\_{\text{cont.}}$ are cross-entropy losses and $\mathcal{L}\_{\text{bbox}}$ is an L1 regression loss. All three $\lambda$ values are set to 1. The key design choice is that the shared decoder processes the structural token sequence, giving all three task-specific decoders a contextualized cell-level representation without requiring three separate encoder passes. ## What experiments were performed? **Datasets:** - **PubTabNet:** 568k table images from PMCOA scientific papers, annotated with HTML structure, cell bounding boxes, and cell text. Split used: 500,777 train / 9,115 validation (dev phase) / 9,064 final evaluation. - **FinTabNet:** 112k complex financial tables from S&P 500 annual reports. 81 / 9.5 / 9.5 train/val/test split. **Metrics:** - TEDS-Struct: Tree Edit Distance Similarity considering only structural HTML tokens (not cell content). - TEDS: Full Tree Edit Distance Similarity including cell content. - mAP (PASCAL VOC) for cell bounding box detection on PubTabNet. **Baselines compared:** - EDD (PubTabNet baseline) - GTE and GTE (fine-tuned) for FinTabNet - LGPMA (required additional text-line bounding box annotations) - TableFormer (with and without PDF-based post-processing) - SEM (3rd place, ICDAR 2021) - VCGroup / TableMASTER (2nd place, ICDAR 2021, uses model ensembles) - Davar-Lab-OCR (1st place, ICDAR 2021, uses additional annotation + ensembles) ## What are the outcomes/conclusions? **Table Structure Recognition (TEDS-Struct):** - FinTabNet: 98.79% All (vs. 96.80% TableFormer, +2.0%) - PubTabNet val: 97.88% All (vs. 96.75% TableFormer, +1.1%) **Cell Detection (mAP on PubTabNet val):** - 88.93% (vs. TableFormer 82.10%, +6.8%; vs. TableFormer+PP 86.80%, +2.1%) **Full Table Recognition (TEDS on PubTabNet val):** - 96.67% All (vs. TableFormer 93.60%, vs. SEM 93.70%, vs. VCGroup without ensemble 96.26%) **ICDAR 2021 Final Evaluation (PubTabNet final set):** - 96.17% TEDS All, approximately 4th place, without ensembles or additional training data. - Davar-Lab-OCR (1st): 96.36%; VCGroup (2nd): 96.32%; XM (3rd): 96.27%. The multi-task design appears to benefit cell detection in particular, likely because the structure decoder provides explicit cell token triggers, focusing the bbox decoder on exactly one cell at a time rather than running box detection over the whole table image. The content decoder benefits from this same focus mechanism. **Limitations the authors do not discuss:** The architecture is autoregressive in the structure decoder, which means inference time scales with the number of cells in a table. Large tables with many cells could be slow. The paper does not report inference speed or latency. The cell-content decoder also autoregressively generates characters per cell, which may compound slow inference. There is no evaluation on wild or camera-captured table images (e.g., WTW dataset), so robustness outside clean document scans is unknown. The paper also lacks a formal ablation study. No experiment isolates the contribution of individual decoders (for example, training without the cell-content loss, or replacing the shared decoder with per-task decoders). The gain over TableFormer is attributed to the MTL formulation, but the relative contributions of the stronger backbone (GCAttention), the shared decoder design, and multi-task gradient flow are not disentangled. Additionally, the shared decoder uses teacher forcing during training (conditioned on gold HTML tags) but must consume its own predicted output at inference. This train/inference mismatch is a known source of error accumulation in autoregressive sequence models and is not acknowledged. ## Reproducibility ### Models - **Backbone:** ResNet-31 with GCAttention (Multi-Aspect Global Context Attention from MASTER, Lu et al. 2021) after each residual block. - **Input resolution:** 480 $\times$ 480 pixels; CNN output feature map: 60 $\times$ 60. - **Shared decoder:** $N = 2$ Transformer decoder layers; hidden size 512, FFN size 2048, 8 attention heads. - **Structure decoder:** 1 Transformer decoder layer; same hidden/FFN/head config. - **Cell-BBox decoder:** 1 Transformer decoder layer + linear + sigmoid. - **Cell-Content decoder:** 1 Transformer decoder layer + linear + softmax. - **Max sequence lengths:** 500 structural tokens; 150 character tokens per cell. - **Weights:** Pretrained checkpoints for both PubTabNet and FinTabNet are released on Google Drive (see frontmatter artifacts). The repository and weights are licensed Apache-2.0 (inherited from the MMOCR codebase). Note: the repository README instructs using `master_decoder_old20220923.py` instead of the default `master_decoder.py` when loading the released checkpoints. ### Algorithms - **Optimizer:** SGD (implied; the paper says "stochastic gradient descent algorithms" without specifying Adam vs. SGD). - **Learning rate:** 0.001 for the first 12 epochs; then 0.0001 for 8 more epochs or until convergence. - **Batch size:** 4 per GPU (2 GPUs; effective batch size 8). - **Loss:** Weighted sum of cross-entropy (structure), cross-entropy (content), and L1 (bounding box), with $\lambda\_{1} = \lambda\_{2} = \lambda\_{3} = 1$. - **Framework:** PyTorch + MMCV; specifically MMOCR-0.2.0, MMDetection-2.11.0, mmcv-full-1.3.4. - **Data augmentation:** Not described. ### Data - **PubTabNet:** Publicly available (CDLA-Permissive-1.0 for annotations; underlying PMCOA images have per-article mixed licenses). - **FinTabNet:** Published by IBM; CDLA-Permissive-1.0 for annotations; underlying S&P 500 report images are from copyrighted filings. The original IBM distribution is no longer easily accessible; the canonicalized FinTabNet.c version is available on HuggingFace. - Both datasets provide structure annotations in HTML, cell bounding boxes, and cell text per non-empty cell. ### Evaluation - TEDS and TEDS-Struct as defined in Zhong et al. 2020. - Simple vs. Complex splits: Simple = no spanning cells; Complex = tables with multi-row or multi-column cells. - Cell detection evaluated with PASCAL VOC mAP protocol on PubTabNet. - No error bars, significance tests, or multi-run statistics reported. - Comparison with ICDAR 2021 competition entries is fair only in the sense that the same final evaluation set was used; competition entries may have used different training data augmentation or proprietary preprocessing. ### Hardware - **Training:** 2 $\times$ NVIDIA A100 80 GB GPUs. - **Training time:** Not reported. - **Inference speed:** Not reported. - **Deployment:** The autoregressive structure and content decoders suggest non-trivial inference latency on large tables; no benchmarks provided. ## BibTeX ```bibtex @inproceedings{ly2023mtltabnet, title={An End-to-End Multi-Task Learning Model for Image-based Table Recognition}, author={Nam Tuan Ly and Atsuhiro Takasu}, booktitle={Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5: VISAPP}, pages={626--634}, year={2023}, organization={SCITEPRESS}, doi={10.5220/0011685000003417} } ```