# MMSci: Benchmarking and Improving Multimodal Scientific Table Understanding > MMSci introduces a three-part dataset suite for scientific table understanding: 52K TSR pre-training samples, 12K instruction-tuning samples with chain-of-thought reasoning, and a 3,114-sample benchmark that tests numerical reasoning over scientific tables. - Authors: Hunter Heidenreich - Published: 2026-03-30 - Dataset: MMSci Dataset (GitHub) - Paper: Preprint (arXiv) - Source: Bevaya Labs, --- ## TL;DR MMSci is a dataset suite built from the SciGen corpus of scientific tables, comprising MMSci-Pre (52K image-to-HTML pairs for table structure learning), MMSci-Ins (12K chain-of-thought instruction samples), and MMSci-Eval (3,114 benchmark samples requiring numerical reasoning). The authors find that 52K domain-specific scientific tables outperform 150K general-domain tables on both in-distribution and held-out evaluation, suggesting that data source matters more than raw volume for this task. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: The primary contribution is the construction and release of the three MMSci datasets (pre-training, instruction tuning, and evaluation), along with the construction methodology, quality control procedures, and reasoning-type taxonomy. **Secondary:** $\Psi\_{\text{Evaluation}}$: MMSci-Eval serves as a purpose-built benchmark for numerical reasoning over scientific tables, and a substantial portion of the paper is devoted to comparing models on this benchmark and analyzing cross-modal consistency. ## What is the motivation? Large language models have shown growing capability on table understanding tasks such as table question answering (TQA), table fact verification (TFV), and table-to-text generation (T2T). Most table-oriented LLMs, however, require converting tables into sequential text formats such as HTML strings, which can discard structural and positional information. Multimodal large language models (MLLMs) sidestep this by processing table images directly, but existing MLLM-based approaches suffer from at least three practical problems: 1. Fixed input image resolutions constrain applicability to tables of varying sizes. 2. Training corpora largely consist of general-domain tables (Wikipedia, financial reports, government documents) and do not cover the dense numerical content typical of scientific publications. 3. Models trained on these general corpora show substantial performance degradation on scientific tables that require arithmetic and statistical reasoning. Prior datasets such as TAT-QA and FinQA address numerical reasoning in the financial domain, and SciGen introduces scientific table-to-text generation, but no existing resource combines multimodal (image-based) input with comprehensive numerical reasoning evaluation across multiple task types for scientific tables. ## What is the novelty? The primary novelty is the MMSci dataset suite itself. The authors build all three components by sourcing raw tabular data from the SciGen dataset (Moosavi et al., 2021), which pairs scientific tables from computer science papers with textual descriptions that naturally invoke arithmetic reasoning. **MMSci-Pre** contains 52K image-to-HTML pairs. The authors convert SciGen's LaTeX-formatted tables to HTML using a rendering pipeline (imgkit) and store the resulting table images alongside their HTML ground truth. The pre-training objective is standard sequence generation: given a table image, produce the corresponding HTML string. This stage develops table structure perception within the MLLM. **MMSci-Ins** contains 12K instruction-tuning samples spanning TQA, TFV, and T2T. For each table, GPT-4o is prompted with both the table image and its SciGen description to generate a question-answer pair, a claim-and-verdict pair, and an extended table-to-text description, each accompanied by explicit chain-of-thought reasoning steps. To improve quality, self-consistency voting (Wang et al., 2023) is applied: multiple reasoning paths are sampled and a majority-vote mechanism selects the final output. The construction prompt is made available in the paper (Table 7). The dataset is balanced: each table contributes exactly one TQA, one TFV, and one T2T sample. **MMSci-Eval** contains 3,114 test samples derived from SciGen's test set using the same GPT-4o generation pipeline, with an additional round of human verification covering all samples. The benchmark covers eight reasoning types: | Reasoning Type | Proportion | |---|---| | Addition | 21.1% | | Subtraction | 15.3% | | Max/Min | 15.7% | | Division | 14.2% | | Comparison | 13.7% | | Ranking | 9.6% | | Look-up | 8.9% | | Domain Knowledge Calculation | 1.5% | On the modeling side, the authors implement dynamic input resolution on two base architectures (Qwen2-VL-7B-Instruct and LLaVA-NeXT-7B), addressing the fixed-resolution limitation of prior table MLLMs. Qwen2-VL achieves this through 2D-RoPE positional encodings; LLaVA-NeXT splits images into grids and encodes them independently. ## What experiments were performed? **Training procedure.** Both models are fine-tuned in two stages on 4 × A100 80 GB GPUs using LoRA (rank 64, sequence length 4096). In the first stage (table structure learning), models are trained for one epoch on image-to-HTML data with varying combinations of MMSci-Pre (52K) and MMTab-Pre (150K from Table-LLaVA). In the second stage (visual instruction tuning), models are fine-tuned for 4 epochs on MMSci-Ins (12K), freezing the visual encoder and updating only the MLP connector and LLM weights. **Evaluation.** The held-in evaluation uses MMSci-Eval's TQA, TFV, and T2T splits. Held-out evaluation uses the MMTab-Eval benchmark (from Table-LLaVA), covering TABMWP, WTQ, HiTab, TAT-QA, FeTaQA (TQA), TabFact, InfoTabs (TFV), and HiTab\_T2T, Rotowire, WikiBIO (T2T). Accuracy is used for TQA and TFV; BLEU is used for T2T. **Baselines.** The authors compare against a range of MLLMs: GPT-4V, InternVL-2-76B, Qwen-2-VL-72B-Instruct, LLaVA-NeXT (72B/34B/13B/7B), Table-LLaVA (13B/7B), Pixtral-12B, Llama-3.2-11B-Vision-Instruct, MiniCPM-V-2.6-8B, and InternVL-2-8B. **Ablations.** Three ablation axes are evaluated: (1) pre-training data source (MMSci-Pre 52K vs. MMTab-Pre 150K vs. combined MM-Pre 202K vs. no pre-training), (2) inclusion of explicit reasoning steps in instruction tuning, and (3) instruction data scaling from 3K to 12K samples. **Representational alignment analysis.** The authors also measure language-vision alignment using kernel similarity metrics (CKA, CKNNA, SVCCA, and several KNN-based measures) on ImageNet, Wikipedia captions, and the MMSci T2T task. Qwen2-VL-7B-Instruct scores highest across most measures, which the authors correlate with its stronger task performance. ## What are the outcomes/conclusions? **Data quality over quantity.** Across both model architectures, training on MMSci-Pre (52K scientific domain) produces comparable or better results than training on MMTab-Pre (150K general domain), particularly on MMSci-Eval. For Qwen2-VL-7B-Instruct, MMSci-Pre (52K) + MMSci-Ins achieves 41.13% TQA accuracy and 72.92% TFV accuracy, versus 40.75% TQA and 72.73% TFV for MMTab-Pre (150K) + MMSci-Ins. Combining both (MM-Pre, 202K) pushes results to 42.10% TQA and 73.98% TFV. **Reasoning steps help consistently.** Ablations show that including chain-of-thought reasoning steps during instruction tuning improves TQA by roughly 6-7 percentage points and TFV by 8-10 percentage points across all pre-training configurations. The benefit is consistent regardless of whether 3K or 12K instruction samples are used, and it generalizes to held-out datasets like TABMWP and TAT-QA. **Generalization to held-out data.** Despite not training on any MMTab-Ins data, the MMSci-tuned Qwen2-VL models achieve 21.15% average TQA accuracy and 41.63% average TFV accuracy on the MMTab held-out benchmark, which is competitive with models explicitly trained on general-domain table data. **Limitations.** The authors note three main limitations. First, the framework focuses on numerical tables; qualitative or methodology tables are not well covered. Second, complex statistical analyses and domain-specific mathematical notation (e.g., subscripted variables, special symbols) may still challenge the models. Third, very large tables with dense information remain difficult to process due to computational constraints and potential information loss during visual encoding. GPT-4V continues to lead most subtasks, suggesting that the fine-tuned 7B models leave substantial room for improvement. ## Reproducibility ### Models Both fine-tuned model variants build on publicly available checkpoints: - Qwen2-VL-7B-Instruct (ViT + MLP connector + Qwen2-7B-Instruct backbone) - LLaVA-NeXT-7B (CLIP encoder + MLP connector + Vicuna-7B backbone) The paper does not report releasing fine-tuned model weights. The base checkpoints are available via their respective repositories under permissive licenses. ### Algorithms - LoRA fine-tuning: rank 64, applied to both stages - Sequence length: 4096 tokens - Stage 1 (table structure learning): 1 epoch; LLaVA-NeXT updates only the MLP connector; Qwen2-VL updates the full model - Stage 2 (instruction tuning): 4 epochs; visual encoder frozen; MLP and LLM weights updated - Training time (4 × A100 80 GB): LLaVA-NeXT table structure learning takes approximately 15 hours for MMTab-Pre (150K), 3 hours for MMSci-Pre (52K), and 20 hours combined (one epoch); Qwen2-VL-7B takes approximately 15 hours, 8 hours, and 19 hours for the same three configurations respectively; instruction tuning takes approximately 1 hour for 4 epochs on 12K samples for both models - Optimizer, learning rate, and warmup schedule are not reported in the paper - Self-consistency voting used during dataset generation; exact number of sampled reasoning paths is not stated ### Data - MMSci-Pre: 52K image-to-HTML pairs derived from SciGen training and development splits; HTML rendered via the imgkit Python package - MMSci-Ins: 12K samples from SciGen training split; generated by GPT-4o with self-consistency voting; 40% manually verified (94.36% accuracy after regeneration of failures) - MMSci-Eval: 3,114 samples from SciGen test split; all samples manually verified (95.25% accuracy) - The dataset JSON files and a zip of MMSci-Pre are publicly available at the GitHub repository; image data is referenced to HuggingFace but exact access path is not detailed in the paper - The underlying SciGen dataset (Moosavi et al., 2021) is publicly available; its license should be consulted before use ### Evaluation - TQA and TFV: accuracy (exact match for TQA answers formatted as JSON) - T2T: BLEU score - No error bars or significance tests are reported; single-run results - Held-out datasets (TABMWP, TAT-QA, TabFact, WTQ, etc.) are evaluated zero-shot relative to MMSci training distribution ### Hardware - Training: 4 × NVIDIA A100 80 GB GPUs - Inference hardware not explicitly specified - No cloud cost estimates provided ## BibTeX ```bibtex @article{yang2025mmsci, title={Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning}, author={Bohao Yang and Yingji Zhang and Dong Liu and André Freitas and Chenghua Lin}, journal={arXiv preprint arXiv:2501.13042}, year={2025} } ```