# ICDAR 2021 Competition on Scientific Table Image Recognition to LaTeX > A competition report introducing two benchmark tasks and datasets for converting scientific table images to LaTeX code, covering both structure and content reconstruction. - Authors: Hunter Heidenreich - Published: 2021-05-30 - Updated: 2026-03-24 - Paper: Preprint (arXiv) - Paper: Published (ICDAR 2021) - Benchmark: Competition Platform (CodaLab) - Source: Bevaya Labs, --- ## TL;DR ICDAR 2021 introduces a competition and two new datasets for converting scientific table images to LaTeX code, splitting the problem into separate structure recognition (TSRD: 46K samples) and content recognition (TCRD: 38K samples) subtasks. The winning VCGroup method achieves 74% exact match on structure and 55% on content, beating prior baselines by 5 and 12 percentage points respectively. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: The primary contribution is the competition infrastructure: two purpose-built datasets (TSRD and TCRD) with ground-truth LaTeX annotations, a CodaLab evaluation platform, and a publicly accessible post-challenge leaderboard. **Secondary:** $\Psi\_{\text{Evaluation}}$: The paper proposes a suite of fine-grained metrics tailored to LaTeX sequence evaluation, covering both coarse structural accuracy (row/column counts) and fine-grained token-category accuracy (alphanumeric, LaTeX tokens, LaTeX symbols, non-LaTeX symbols). ## What is the motivation? Scientific documents rely heavily on tables to convey experimental comparisons and structured data. Extracting that information accurately requires both understanding the spatial structure of cells and recognizing complex content including mathematical symbols and LaTeX macros. Existing datasets for this problem are small (e.g., ICDAR 2013 had only 150 table instances) or contain simple tables that do not represent real-world complexity. Prior work by Deng et al. used word-level tokenization without fine-grained task decomposition, resulting in poor recognition of the full LaTeX output. There was no dedicated benchmark that cleanly separated structure recognition from content recognition or that handled the diversity of LaTeX table libraries (booktabs, multirow, amsmath, etc.). ## What is the novelty? The paper introduces two complementary subtask formulations: **Task 1: Table Structure Reconstruction (TSR)** asks systems to generate the LaTeX skeleton of a table, using a placeholder token `CELL` for cell content. The target vocabulary includes alignment specifiers, column separators, spanning commands, and line commands: $$\mathcal{V}\_{\text{TSR}} = \\{ \&, 0\text{--}9, \texttt{CELL}, \backslash\backslash, \texttt{\\hline}, \texttt{\\multirow}, \texttt{\\multicolumn}, \texttt{\\toprule}, \ldots \\}$$ **Task 2: Table Content Reconstruction (TCR)** asks systems to generate the full alphanumeric and mathematical content of each cell, including LaTeX math tokens and symbols. Separating these two goals is the core structural novelty: structure and content are independently evaluated, enabling cleaner diagnosis of model capabilities and failure modes. The paper also introduces a suite of evaluation metrics. For both tasks, the primary metric is Exact Match (EM), which requires the predicted token sequence to exactly match the ground truth: $$\text{EM}(y^\ast, y) = \mathbf{1}[y\_1^\ast = y\_1 \wedge \cdots \wedge y\_{n\_2}^\ast = y\_{n\_2}]$$ A relaxed variant, EM@95%, accepts a prediction if at least 95% of the ground-truth tokens appear as a contiguous subsequence in the prediction (no insertions or deletions allowed within the matched span): $$\text{EM@95\%}: \exists\, i,j \text{ s.t. } y\_i^\ast = y\_j, \ldots, y\_{i+l}^\ast = y\_{j+l},\ l \geq 0.95 \times n\_2$$ For TSR, Row Accuracy (RA) and Column Accuracy (CA) measure whether the predicted row and column counts match. For TCR, token-category accuracies partition the output into four groups: alphanumeric (AN), LaTeX tokens (LT), LaTeX symbols (LS), and non-LaTeX symbols (NLS), measuring exact match within each category independently. ## What experiments were performed? The competition ran from October 2020 to March 2021 on the CodaLab platform with three phases: validation, testing (hidden leaderboard), and post-evaluation. Three teams participated in the TSR test phase; two in the TCR test phase. **Datasets:** | Dataset | Max Seq. Len. | Total Samples | Train | Val | Test | |:--------|:-------------:|:-------------:|:-----:|:---:|:----:| | TSRD | 250 | 46,141 | 43,138 | 800 | 2,203 | | TCRD | 500 | 37,917 | 35,500 | 500 | 1,917 | Tables were sourced from scientific documents; ground truth LaTeX was extracted from original source files. **Baselines:** - **CNN-Baseline**: Eight convolutional layers with interleaved max-pooling, followed by bidirectional LSTM encoder with trainable positional embeddings per row, and an LSTM decoder with visual attention (based on Deng et al., 2017). - **Transformer-Baseline**: ResNet-101 image encoder with a Transformer decoder, adapted from a scene text recognition architecture (Feng et al., 2020). **Participant method (VCGroup):** Based on the MASTER architecture (originally for scene text recognition), which uses a multi-aspect non-local block integrated into a CNN backbone to capture global spatial context. Specific techniques used by VCGroup include: - Ranger optimizer (combining RAdam, LookAhead, and gradient centralization) - Data augmentation: shear, affine, perspective, contrast, brightness, saturation - Synchronized Batch Normalization for multi-GPU training - Feature concatenation of the last two transformer decoder layers with linear projection - Model ensemble (voting and bagging) - For TCR only: multi-resolution inputs (400x400 to 600x400) and pre-training on 58K samples from a larger external table-to-LaTeX dataset ## What are the outcomes/conclusions? **TSR results:** | Method | EM | EM@95% | RA | CA | |:-------|:--:|:------:|:--:|:--:| | VCGroup | 0.74 | 0.88 | 0.95 | 0.89 | | Transformer-Baseline | 0.69 | 0.85 | 0.93 | 0.86 | | CNN-Baseline | 0.66 | 0.79 | 0.92 | 0.86 | | Format* | 0.57 | 0.80 | 0.91 | 0.87 | | asda* | 0.50 | 0.75 | 0.90 | 0.86 | **TCR results:** | Method | EM | EM@95% | AN | LT | LS | NLS | |:-------|:--:|:------:|:--:|:--:|:--:|:---:| | VCGroup | 0.55 | 0.74 | 0.85 | 0.75 | 0.96 | 0.62 | | Transformer-Baseline | 0.43 | 0.64 | 0.74 | 0.67 | 0.94 | 0.54 | | CNN-Baseline | 0.41 | 0.67 | 0.76 | 0.67 | 0.94 | 0.48 | | Format* | 0.00 | 0.52 | 0.67 | 0.54 | 0.92 | 0.35 | *Method description unavailable for rows marked with an asterisk. VCGroup's strongest gains appear in longer sequences, where the Transformer-Baseline degrades more sharply. The gap between RA/CA and EM reveals that row and column count errors are less common than token-level errors; exact sequence matching is harder than structural coarse accuracy. Common failure modes across all methods include: incorrect handling of multi-row and multi-column spanning cells, confusion between `\multicolumn` and `\multirow`, and inconsistent dollar sign (`$`) prediction around numeric tokens. Both baselines tend to truncate long sequences prematurely. The competition received 101 registrations but only a handful of test-phase submissions (three teams in TSR, two in TCR), which the authors attribute to the difficulty of the tasks. The datasets and platform remain available for post-challenge submissions. ## Reproducibility ### Models - **CNN-Baseline**: Eight conv layers + five max-pool layers, bidirectional LSTM encoder, LSTM decoder with visual attention. Architecture based on Deng et al. (2017); no weights released as part of this competition report. - **Transformer-Baseline**: ResNet-101 encoder + Transformer decoder. Based on Feng et al. (2020); no weights released. - **VCGroup (MASTER-based)**: Multi-aspect non-local block fused into CNN backbone + Transformer decoder. Full implementation details in He et al. (2021), arXiv:2105.07395. No weights publicly released with this report. ### Algorithms - VCGroup used the Ranger optimizer (RAdam + LookAhead + gradient centralization). - Data augmentation: shear, affine, perspective, contrast, brightness, saturation transforms. - Synchronized BatchNorm used for multi-GPU training. - Feature concatenation of last two decoder layers with linear projection replaces single final layer output. - Model ensemble via voting and bagging. - Pre-training on a trimmed external dataset (58K of 450K samples) for the TCR task. - No training hyperparameters (learning rate, batch size, number of epochs) are reported in this competition overview paper. ### Data - TSRD: 46,141 image-text pairs, max sequence length 250 tokens. Train/val/test splits: 43,138 / 800 / 2,203. - TCRD: 37,917 image-text pairs, max sequence length 500 tokens. Train/val/test splits: 35,500 / 500 / 1,917. - Data sourced from scientific papers; ground-truth LaTeX extracted from original source files. - Available through the CodaLab competition platform (https://competitions.codalab.org/competitions/26979). No explicit standalone download link or Hugging Face release mentioned. - License for the datasets is not explicitly stated in the paper beyond the arXiv CC-BY-4.0 license of the report itself. ### Evaluation - Primary metric: Exact Match (EM): full sequence equality. - Secondary metric: EM@95%: at least 95% of ground-truth tokens matched as a contiguous subsequence (no insertions/deletions within the span). - TSR-specific: Row Accuracy (RA) and Column Accuracy (CA): count-level structural accuracy. - TCR-specific: token-category accuracies (AN, LT, LS, NLS). - No error bars, significance tests, or multiple-run averaging are reported. The competition format uses single submissions evaluated on fixed test sets. - No contamination analysis of the train/test split is reported. ### Hardware - No hardware details are reported for the baseline methods in this competition overview paper. - VCGroup used multi-GPU training (SyncBN implies multiple GPUs), but GPU type, count, and training time are not reported here. Full hardware details may appear in He et al. (2021). ## BibTeX ```bibtex @inproceedings{kayal2021icdar, title={{ICDAR} 2021 Competition on Scientific Table Image Recognition to {LaTeX}}, author={Kayal, Pratik and Anand, Mrinal and Desai, Harsh and Singh, Mayank}, booktitle={Document Analysis and Recognition -- ICDAR 2021}, year={2021}, publisher={Springer}, doi={10.1007/978-3-030-86337-1_50} } ```