# MuTabNet: Multi-Cell Decoder and Mutual Learning for Table Structure and Character Recognition > MuTabNet improves end-to-end table recognition by introducing a multi-cell content decoder that attends to neighboring cells and a bidirectional mutual learning mechanism for the HTML structure decoder. - Authors: Hunter Heidenreich - Published: 2024-04-19 - Updated: 2026-03-26 - Paper: Preprint (arXiv) - Code: MuTabNet Code - Source: Bevaya Labs, --- ## TL;DR MuTabNet is an end-to-end table recognition model that improves upon prior Transformer-based approaches through two mechanisms: a multi-cell decoder that reads cell contents sequentially across multiple cells while attending to neighbors, and bidirectional mutual learning that trains the structure decoder in both left-to-right and right-to-left directions simultaneously. On FinTabNet and PubTabNet, MuTabNet achieves results that exceed or match all prior end-to-end and OCR-assisted baselines. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: the paper's headline contribution is an architectural modification to an existing end-to-end table recognition pipeline. The two proposed components (multi-cell decoder and bidirectional mutual learning) are architectural additions not present in prior work, and the paper is organized around ablation tables and SOTA comparisons that validate those additions. **Secondary:** $\Psi\_{\text{Evaluation}}$: the paper carefully measures performance across multiple table-length subsets (250+, 500+, 600+, 700+ tokens) to isolate how well the method handles long tables, and it explores sensitivity to local attention window sizes. ## What is the motivation? Extracting structured table content from document images is an important preprocessing step for LLM-based knowledge retrieval, but it remains difficult. Two classes of approaches exist: those that rely on an external OCR system to recognize cell text (which achieves high accuracy but depends on a separate pipeline) and end-to-end approaches that jointly recognize structure and content. Recent end-to-end work (Ly and Takasu, ICDAR 2023) showed that joint learning can reach accuracy comparable to OCR-assisted models, and introduced local attention to handle tables with hundreds of cells. However, both the HTML structure decoder and the cell content decoder in these models have a directional limitation: the HTML decoder reads tokens strictly left to right, and each cell's content is decoded independently without any context from neighboring cells. These design choices forgo potentially useful context, particularly for long or complex tables where surrounding cells can disambiguate ambiguous content. ## What is the novelty? MuTabNet makes two architectural additions to the prior Ly-Takasu end-to-end framework: **Multi-cell decoder.** Instead of decoding each cell's content independently, the proposed cell decoder reads all cell contents as a single sequence, separated by a special SEP token. The decoder receives a concatenation of character embeddings and HTML features extracted from the HTML decoder output, giving it access to structural context for each cell. Local attention (window size $w = 300$) handles the resulting long sequences. This means cell $n$ can attend to the decoded content of cells $1, \ldots, n-1$ within the attention window, allowing contextual correction of ambiguous characters. **Bidirectional mutual learning.** Inspired by deep mutual learning (Zhang et al., CVPR 2018), the HTML decoder is trained to predict structure in both left-to-right (LtoR) and right-to-left (RtoL) directions. To avoid doubling the parameter count, both directions share a single decoder; a one-hot vector specifying direction is appended to the embedded HTML tokens. The loss for the LtoR decoder combines standard cross-entropy against the ground truth with a KL divergence term that aligns its output distribution to the RtoL decoder's predictions: $$ \mathcal{L} = -\frac{1}{N}\sum\_{n=1}^{N} p(\vec{x}\_n) \log q(\vec{x}\_n) + \frac{1}{N}\sum\_{n=1}^{N} q(\overleftarrow{x}\_n) \log \frac{q(\overleftarrow{x}\_n)}{q(\vec{x}\_n)}. $$ The symmetric loss for the RtoL decoder reverses the roles. This regularizes the structure decoder to be consistent regardless of reading direction. The encoder is a TableResNetExtra backbone (26 convolutional layers, three Global Context Attention blocks) that encodes a $520 \times 520$ input to $65 \times 65$ feature maps with 512 channels, followed by 2D positional encoding and flattening for cross-attention. ## What experiments were performed? **Datasets.** Two large public benchmarks are used: - **FinTabNet** (112k tables from S&P 500 annual reports; evaluation on the 10,656-image validation set following prior convention) - **PubTabNet** (568k tables from PubMed Central; validation set and the held-out ICDAR 2021 competition evaluation set) - **PubTabNet250** (subset of PubTabNet with 250+ HTML tokens; used for ablations to reduce training time from ~179 hours to ~45 hours) **Baselines.** Comparisons include EDD, GTE, TableFormer, VAST (an OCR-assisted model), and the direct predecessor series from Ly and Takasu (weakly supervised, multi-task, and local-attention variants). **Metric.** Tree Edit Distance Similarity (TEDS), defined as: $$ \text{TEDS}(T\_a, T\_b) = 1 - \frac{\text{EditDist}(T\_a, T\_b)}{\max(|T\_a|, |T\_b|)}, $$ computed in two variants: structural TEDS (excluding cell text) and total TEDS (including cell text). Results are broken out by simple vs. complex tables on PubTabNet. **Ablation.** Table 5 isolates the contribution of local attention (LA), multi-cell decoder (MC), and bidirectional mutual learning (BML) across four token-length subsets. Table 6 sweeps local attention window sizes (100, 200, 300, 400, 500) for the cell decoder. **Training.** Four NVIDIA V100 GPUs, batch size 8, Ranger optimizer, learning rate 0.001 for 25 epochs then stepped down to 0.0001 and 0.00001. No data augmentation, no ensemble, no early stopping. ## What are the outcomes/conclusions? **FinTabNet.** MuTabNet achieves 98.87% structural TEDS (best among all models) and 97.69% total TEDS. The total TEDS is below VAST's 98.21%, which the authors attribute to VAST's use of an external OCR system for cell content. Among end-to-end models, MuTabNet leads on both metrics. **PubTabNet validation.** MuTabNet scores 98.16% (simple), 95.53% (complex), and 96.87% (total) TEDS, outperforming all prior methods including those using external OCR. On the ICDAR 2021 evaluation set, MuTabNet achieves 98.01% (simple), 94.98% (complex), and 96.53% (total) TEDS; the authors report this as the highest single-model score among submissions without ensemble methods or additional annotation. **Ablation findings.** Adding the MC decoder to the LA baseline produces a large jump in total TEDS across all table lengths. BML then adds a smaller but consistent improvement, especially visible in total TEDS scores (e.g., 95.02% to 95.81% at the 250+ token threshold). The effect of BML on structural TEDS alone is modest, suggesting it primarily helps cell content recognition by improving the implicit structural representations passed to the cell decoder. A window size of 300 is optimal for the cell decoder in most cases; very long tables (500+ tokens) slightly prefer a smaller window of 100. **Limitations the authors acknowledge.** The paper notes that future work should extend toward understanding the semantic meaning of table contents, not just their structure and character-level text. There is no discussion of inference latency per table, runtime memory, or behavior on non-English or handwritten tables. ## Reproducibility ### Models - Architecture: TableResNetExtra encoder (26 conv layers, 3 GCA blocks); HTML decoder (1 embedding layer + 3 local-attention blocks + 2 output layers); cell decoder (1 embedding layer + 1 local-attention block + output layer). All attention blocks use 8 heads and 512 channels. - No pretrained weights are released; no model card is provided. - Input size: $520 \times 520$ pixels. Feature map: $65 \times 65 \times 512$. ### Algorithms - Optimizer: Ranger (a combination of RAdam and LookAhead) - Learning rate schedule: 0.001 for epochs 1-25, 0.0001 for epochs 26-28, 0.00001 for epochs 29-30 - Batch size: 8 across 4 GPUs - Local attention window: 300 for both HTML and cell decoders (default) - Maximum HTML token sequence length: 800; maximum cell content sequence length: 8000 - Decoding: greedy search - No data augmentation; no early stopping; no ensembling ### Data - FinTabNet: publicly available, extracted from S&P 500 annual reports. License: see the GTE paper (Zheng et al., WACV 2021). - PubTabNet: publicly available from IBM Research, derived from PubMed Central Open Access. License: Community Data License Agreement - Permissive (CDLA-Permissive). - PubTabNet250: a filtered subset created by Ly and Takasu; not separately distributed; reproducible by filtering PubTabNet for tables with 250+ HTML tokens. ### Evaluation - TEDS metric computed on complete HTML trees (including cell text) and on structure-only trees. - PubTabNet simple/complex split follows Zhong et al. (ECCV 2020): simple means no merged cells. - FinTabNet evaluation set follows prior convention (using the labeled "validation" split of 10,656 images). - No error bars or significance tests are reported; single training run per configuration. - Ablations use PubTabNet250 (a subset), so absolute scores there are not directly comparable to full-dataset results. ### Hardware - Training: 4 NVIDIA V100 GPUs - Full PubTabNet training time: approximately 179 hours (as implied by the comparison; ablations use PubTabNet250 at approximately 45 hours per model) - Inference: 3.78 hours on 4 GPUs for the FinTabNet evaluation set; 3.23 hours for PubTabNet validation; 3.13 hours for PubTabNet evaluation. Per-table inference speed is not reported. - No code repository is released with the paper. ## BibTeX ```bibtex @inproceedings{kawakatsu2024mutabnet, title={Multi-Cell Decoder and Mutual Learning for Table Structure and Character Recognition}, author={Kawakatsu, Takaya}, booktitle={International Conference on Document Analysis and Recognition (ICDAR)}, year={2024} } ```