# Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models

> A research note on Table2LaTeX-RL, a NeurIPS 2025 paper that builds a 1.2M-pair arXiv-derived training corpus and fine-tunes multimodal language models with a dual-reward GRPO variant (VSGRPO) combining structural and visual fidelity signals for table image-to-LaTeX generation.

- Authors: Hunter Heidenreich
- Published: 2026-03-30
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2509.17589>
- Code: Table2LaTeX-RL Code <https://github.com/newLLing/Table2LaTeX-RL>
- Model: Table2LaTeX-RL Model Weights (Qwen2.5-VL-3B-VSGRPO) <https://huggingface.co/LLLHHH/Table2Latex-RL>
- Dataset: Table2LaTeX-RL Evaluation Dataset <https://huggingface.co/datasets/LLLHHH/Table2LaTeX-RL>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

Table2LaTeX-RL constructs a large-scale corpus of approximately 1.2 million table image-to-LaTeX pairs extracted from arXiv source files, then fine-tunes multimodal large language models in two stages: supervised fine-tuning on the full corpus followed by reinforcement fine-tuning via VSGRPO, a dual-reward extension of GRPO that combines a TEDS-Structure reward with a CW-SSIM reward computed on rendered output images. The resulting Qwen2.5-VL-3B model reaches TEDS-Structure of 0.9218 on complex tables, the first reported result above 0.9 on that subset, outperforming models with more than 20 times as many parameters.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$ The primary contribution is the VSGRPO training framework: a visual-in-the-loop reinforcement learning strategy that renders generated LaTeX to PNG and computes CW-SSIM directly as an optimization signal alongside a structure-level TEDS-Structure reward. The paper devotes most of its space to method design, ablations on reward components, and comparative benchmarks.

**Secondary:** $\Psi\_{\text{Resource}}$ The 1.2M table image-to-LaTeX dataset scraped from arXiv, together with the complexity-stratified test split covering simple, medium, and complex tables, represents a meaningful data contribution to a domain that previously lacked large-scale public training corpora.

## What is the motivation?

Scientific documents depend on LaTeX tables to communicate experimental results, and digitizing or reusing those tables requires converting scanned or rendered table images back into source code. Most prior table structure recognition work targets HTML representations, which do not support the full expressiveness of LaTeX: nested multirow/multicolumn headers, mathematical content in cells, and typographic control. The handful of systems that do target LaTeX (LATTE, LaTeXNet, Nougat) either rely on from-scratch training with limited data or do not address the combined difficulty of large, deeply nested structures.

Two intertwined gaps motivate the work. First, no large-scale, publicly available dataset of table image-to-LaTeX pairs existed prior to this work. Second, standard evaluation metrics are poorly suited to the task. TEDS and BLEU compare LaTeX code at the token level, which penalizes semantically equivalent but syntactically different code (e.g., `{}` wrapper groups that do not affect rendering, or `\textbf` vs `{\bf}`). Visual similarity metrics like CW-SSIM capture rendered appearance but ignore structural correctness. Neither alone gives a reliable picture of generation quality.

## What is the novelty?

The paper makes three interrelated contributions.

**Large-scale table corpus.** The authors scrape arXiv LaTeX source files for papers published between October 2017 and April 2023, extract table environments using regular expressions, and clean the code by removing references, color commands, and other non-structural elements. The result is 1,209,986 table-LaTeX pairs. Tables are classified into three complexity tiers based on cell count and the presence of `\multirow`/`\multicolumn` commands: complex (over 160 cells), medium (100-160 cells with 2 or more `\multirow`/`\multicolumn` commands), and simple (all others). In the training set, simple tables account for approximately 94%, with medium and complex each at roughly 3%.

**Hybrid evaluation protocol.** The authors propose combining TEDS-Structure (measuring structural tree edit distance, ignoring cell content) with a modified CW-SSIM tuned for binary, high-contrast table images. The CW-SSIM variant applies a one-level Haar wavelet decomposition and computes SSIM per sub-band:

$$
\text{CW-SSIM}(X, Y) = \frac{1}{4} \sum\_{i \in \\{A,H,V,D\\}} \text{SSIM}(c^i\_X, c^i\_Y)
$$

where $A$, $H$, $V$, $D$ denote the approximation, horizontal, vertical, and diagonal sub-bands respectively. TEDS-Structure is defined as:

$$
\text{TEDS-Structure} = 1 - \frac{\text{TED}\_{\text{structure}}}{\max(|\mathcal{T}\_{\text{pred}}|, |\mathcal{T}\_{\text{gt}}|)}
$$

**VSGRPO dual-reward reinforcement fine-tuning.** Standard GRPO optimizes text generation quality using correctness-based rewards without a separate value network. VSGRPO extends this to include a visual fidelity reward. For each training example, the model generates $N$ candidate LaTeX outputs. Each candidate is compiled to an image; if compilation fails the reward is 0. Successful compilations are scored by CW-SSIM against the ground-truth rendered image, yielding a binary reward at threshold 0.6. In parallel, each candidate is converted to HTML and scored by TEDS-Structure against the ground truth, yielding a second binary reward at threshold 0.9. The combined reward drives the GRPO objective:

$$
J\_{\text{RFT}}(\theta) = \mathbb{E}\left[ \frac{1}{N} \sum\_{i=1}^{N} \min\!\left( \frac{\pi\_\theta(o\_i|q)}{\pi\_{\theta\_{\text{old}}}(o\_i|q)} A\_i,\ \text{clip}\!\left(\frac{\pi\_\theta(o\_i|q)}{\pi\_{\theta\_{\text{old}}}(o\_i|q)}, 1-\varepsilon, 1+\varepsilon\right) A\_i \right) - \beta D\_{\text{KL}}(\pi\_\theta \| \pi\_{\text{ref}}) \right]
$$

where $\varepsilon = 0.2$ and $\beta = 0.02$, and the group advantage $A\_i$ is the z-score of reward $r\_i$ within the group of $N$ samples:

$$
A\_i = \frac{r\_i - \text{mean}(\\{r\_1, \ldots, r\_N\\})}{\text{std}(\\{r\_1, \ldots, r\_N\\})}
$$

Because LaTeX rendering is non-differentiable, this visual reward signal can only be incorporated via the reward mechanism in RL, not during gradient-based supervised training. That is the core motivation for using GRPO rather than extending SFT.

## What experiments were performed?

**Training data.** SFT uses all 1,209,986 table-LaTeX pairs. VSGRPO fine-tunes on a curated subset of 5,936 complex-tier tables whose ground-truth LaTeX is under 3,000 characters, chosen to balance structural difficulty and rendering feasibility.

**Test data.** A separate crawl of arXiv papers from January to November 2024 yields 101,469 entries; from these, 496 simple, 354 medium, and 361 complex tables are sampled as the test set. An external dataset from the LATTE paper serves as an additional generalization benchmark.

**Baselines.** Commercial: Mathpix. General-purpose MLLMs: GPT-4o, Qwen2.5-VL-72B, InternVL2.5-78B. Task-specific: Nougat, LATTE.

**Metrics.** CW-SSIM and compile ratio measure rendered image quality. TEDS-Structure and TEDS measure structural and content alignment at the source level. Human preference evaluation covers 200 tables (50/50/100 by complexity), with majority-vote preference across four systems.

**Ablations.** The paper ablates (1) RL training data selection (simple-only vs. mixed vs. complex-only), (2) reward components (TEDS-Structure alone, CW-SSIM alone, both combined), and (3) the necessity of SFT initialization before VSGRPO.

**Hardware.** SFT for Nougat runs on 4 nodes of 8xA100 GPUs. InternVL2-1B SFT uses 4 nodes; VSGRPO uses 2 nodes. Qwen2.5-VL-3B SFT and VSGRPO each use 2 nodes.

## What are the outcomes/conclusions?

**Main benchmark results.** On the authors' own test set, Qwen2.5-VL-3B-VSGRPO achieves CW-SSIM scores of 0.8186 (simple), 0.7236 (medium), and 0.6145 (complex). On complex tables, the CW-SSIM gain over the SFT baseline is +0.034, and the compile ratio reaches 0.9917, matching or exceeding all baselines including Mathpix. On TEDS and TEDS-Structure, Qwen2.5-VL-3B-VSGRPO reaches 0.8673 TEDS and 0.9218 TEDS-Structure on complex tables. The reported TEDS gap over the next-best model is 0.1225 on complex tables. Intern2.5-VL-78B, a model more than 25 times larger, shows a TEDS collapse on complex tables (0.3379) while Qwen2.5-VL-3B-VSGRPO remains robust.

**External benchmark.** On the LATTE external dataset (predominantly simple tables), Qwen2.5-VL-3B-VSGRPO achieves CW-SSIM of 0.8225 and TEDS-Structure of 0.9461, outperforming LATTE itself (0.7615 / 0.9445) despite LATTE being a specialist system fine-tuned for this format.

**Human evaluation.** In the 200-table preference study, Qwen2.5-VL-3B-VSGRPO received the highest vote counts across all complexity levels: 42 vs. 29 for the SFT-only baseline on simple tables, 37 vs. 28 on medium, and 70 vs. 56 on complex.

**Ablation findings.** Training VSGRPO on complex tables only outperforms both simple-only and mixed-data variants across all metrics. Both reward components contribute independently; their combination achieves the highest performance on every metric. Skipping SFT and applying VSGRPO directly to the pre-trained base model results in substantially lower scores (CW-SSIM 0.4695 vs. 0.6145 on complex), confirming that a strong initialization is required.

**Limitations.** The authors acknowledge that the visual rendering step in VSGRPO training is computationally expensive. Each candidate output must be compiled to PDF and converted to PNG for CW-SSIM scoring, creating a training bottleneck even with multi-threading. This overhead restricted the VSGRPO training set to 5,936 tables. Additionally, the TEDS metric remains sensitive to syntactically irrelevant LaTeX differences (e.g., empty groups, different bold commands), which the paper illustrates with qualitative examples but does not fully resolve.

## Reproducibility

### Models

Two model families are reported. InternVL2-1B is fine-tuned via the VLM-R1 framework. Qwen2.5-VL-3B is fine-tuned via the ms-swift infrastructure. Both use full-parameter fine-tuning for SFT. Architecture details (layer count, attention heads) are not described in the paper; they are inherited from the pre-trained base models (Qwen2.5-VL-3B-Instruct from HuggingFace). During VSGRPO, the ViT visual encoder is frozen (`--freeze_vit true`, `--freeze_parameters visual`), so only the language decoder weights are updated in the RL phase. Trained weights for Qwen2.5-VL-3B-VSGRPO are publicly released at `https://huggingface.co/LLLHHH/Table2Latex-RL` (Apache-2.0, BF16 safetensors format, approximately 4B parameters).

### Algorithms

SFT trains for one epoch with a maximum output length of 4,096 tokens. InternVL2-1B SFT uses batch size 4 with gradient accumulation steps of 2; Qwen2.5-VL-3B SFT uses batch size 4 with gradient accumulation steps of 2 on 2 nodes. For VSGRPO, InternVL2-1B uses 8 sampled generations per input, batch size 8, gradient accumulation 2, and 2 nodes. Qwen2.5-VL-3B VSGRPO uses 4 generations per input, batch size 4, gradient accumulation 2, and 2 nodes, for 1 epoch. Reward thresholds are 0.6 for CW-SSIM and 0.9 for TEDS-Structure. KL penalty $\beta = 0.02$; PPO clipping $\varepsilon = 0.2$.

The VSGRPO training script (published in the repository at `examples/train/grpo/plugin/run_external_rm.sh`) reveals additional hyperparameters not stated in the paper: learning rate 1e-5, warmup ratio 0.05, bfloat16 mixed precision, training-time sampling temperature 0.9, maximum completion length 3,000 tokens, maximum image resolution 524,288 pixels (`--max_pixels`), and 8 GPUs per node via `torchrun`. The optimizer is not named explicitly in the script but ms-swift defaults to AdamW. Testing uses maximum output length 8,192, batch size 1, temperature 0. Training requires a Docker environment; the README specifies a ModelScope image with CUDA 12.6, PyTorch 2.7.1, vLLM 0.10.1, and ms-swift 3.8.1.

### Data

Training data covers arXiv papers from October 2017 to April 2023 (1,209,986 table entries). Test data covers arXiv papers from January to November 2024 (496 simple, 354 medium, 361 complex tables). VSGRPO uses 5,936 complex tables with ground-truth LaTeX under 3,000 characters. The full 1.2M SFT training corpus is not directly downloadable from the repository; instead, the authors provide `arxiv_papers_get.py` and a `table.ipynb` Kaggle notebook to re-crawl arXiv source files and reconstruct the dataset. The evaluation split (1,211 images, Apache-2.0) is released at `https://huggingface.co/datasets/LLLHHH/Table2LaTeX-RL`. The underlying arXiv source files are available under arXiv's open-access policy, though the license status of individual paper LaTeX sources varies by author. The code repository is Apache-2.0.

### Evaluation

CW-SSIM is computed using a Python implementation with Haar wavelet decomposition (provided as `cw_ssim.ipynb` in the repository). TEDS-Structure and TEDS use tree edit distance over structural HTML representations parsed from the LaTeX. All testing is conducted inside a `texlive-full` Docker environment to ensure consistent LaTeX rendering. No error bars, significance tests, or seed sensitivity analysis are reported; the NeurIPS checklist explicitly states "No" for statistical significance on the grounds that "running a complete experiment takes one week" and compute resources were limited. Human evaluation uses majority voting by 5 graduate students in computer science on 200 tables (50 simple, 50 medium, 100 complex), with anonymized and randomly shuffled model outputs.

### Hardware

SFT for Nougat: 4 nodes of 8xA100 GPUs (32 A100s total). SFT for InternVL2-1B: 4 nodes of 8 GPUs; for Qwen2.5-VL-3B: 2 nodes of 8 GPUs (16 GPUs total). VSGRPO for both models: 2 nodes of 8 GPUs (16 GPUs total); the training script confirms 8 processes per node via `--nproc_per_node=8`. The GPU type for all non-Nougat experiments is not stated in the paper (only A100 is explicitly named for Nougat). Total GPU-hours, cost estimates, and inference latency figures are not reported. The NeurIPS checklist self-reports "Yes" on compute resources but the paper body only enumerates node counts. The rendering bottleneck in VSGRPO (PDF compilation per candidate per training step) is noted qualitatively but not measured; the authors state it was the reason for limiting VSGRPO training to 5,936 tables.

## BibTeX

```bibtex
@inproceedings{ling2025table2latexrl,
  title={Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models},
  author={Ling, Jun and Qi, Yao and Huang, Tao and Zhou, Shibo and Huang, Yanqin and Yang, Jiang and Song, Ziqi and Zhou, Ying and Yang, Yang and Shen, Heng Tao and Wang, Peng},
  booktitle={Advances in Neural Information Processing Systems},
  year={2025}
}
```