# SciGraphQA: Notes > Large-scale synthetic multi-turn QA dataset for scientific graphs extracted from ArXiv papers - Authors: Hunter Heidenreich - Published: 2023-08-01 - Updated: 2026-03-04 - Code: SciGraphQA Code - Dataset: SciGraphQA Dataset - Source: Bevaya Labs, --- **TL;DR:** SciGraphQA is a large synthetic **multi-turn** QA dataset grounded in **real scientific graphs** extracted from ArXiv papers: the images are real, but the dialogues (questions/answers) are generated (Palm-2) from text context around each figure. The authors report current MLLMs perform poorly zero-shot on this benchmark (low CIDEr), but **prompt augmentation** with DePlot-extracted tables and **fine-tuning** a LLaVA-13B baseline materially improves scores. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Resource}}$ 0.65 - **Secondary:** $\\Psi_{\\text{Evaluation}}$ 0.20, $\\Psi_{\\text{Method}}$ 0.15 The primary contribution is a **large-scale dataset/benchmark** (SciGraphQA) with construction pipeline + release framing. The paper also evaluates multiple MLLMs on the dataset and introduces a baseline fine-tuning recipe plus a prompt-augmentation technique (DePlot tables). * * * ## What is the motivation? Scientific papers communicate key results via **graphs/figures** that require interpretation in context (abstract + surrounding paragraph), often via **interactive multi-turn** explanation in real life (reading groups, presentations). Existing chart/graph VQA datasets either rely heavily on **synthetic charts** and/or **template questions**, or are much smaller in scale; the authors aim to scale "ChartQA-style" real-chart VQA to the academic graph domain. They also position "dialogue QA" as more useful than "caption prediction" for graphs because captions can be short/underspecified and not very helpful to a user. * * * ## What is the novelty? **Dataset construction (core novelty):** - Uses ~**290,000** CS/ML ArXiv papers (2010--2020) and extracts figures (PDFFigures 2.0 is referenced) - Focuses on **graphs** (not all figure types) and generates **multi-turn QA dialogues** about each graph using **Palm-2** prompted with text context: title, abstract, figure caption, and the first paragraph mentioning the figure; OCR text is also included in the context input - Scale: **295K** multi-turn samples and **657K** QA pairs/turns (multi-turn, average **2.23** turns per sample) **Prompt augmentation idea (secondary novelty):** - Prepend the question with a **serialized data table** extracted from the chart using **DePlot** (plot-to-table model) to mitigate weak OCR/text-reading in MLLMs (Figure 4 shows the "table then serialize" concept with newline token `<0x0A>`) * * * ## What experiments were performed? **Dataset quality checks:** - GPT-4 is used as a judge to rate QA matching quality on a **3K** test subset; authors report average **8.7/10**, with distribution heavily skewed high (Figure 2, p.6) **Zero-shot MLLM evaluation:** - Zero-shot evaluation on **3K** test samples with NLP overlap metrics: **BLEU-4, ROUGE, CIDEr** - Models discussed include LLaVA variants, mPLUG-owl, BLIP-2, OpenFlamingo, and DePlot+LLM pipelines **Fine-tuning:** - Fine-tune **LLaVA-13B** on SciGraphQA: (1) 1 epoch on full dataset, then (2) additional fine-tuning on a **30K** DePlot-augmented subset - Study effect of training set size on fine-tuning performance (Figure 5, p.9) * * * ## What are the outcomes/limitations? **Outcomes (as reported):** - Baseline zero-shot scores are very low; prompt augmentation with DePlot tables improves some models; fine-tuning improves further (Table 2, p.8; Figure 5, p.9) - Authors argue this indicates: (a) scientific graphs are out-of-distribution and hard for current MLLMs, and (b) external "chart-to-table" extraction can meaningfully help **Limitations / caveats:** - **Synthetic answers**: images are real graphs, but QA text is model-generated (Palm-2). So evaluation is against synthetic "ground truth," and overlap metrics may reward stylistic similarity rather than correctness - **Filtering heuristic**: they drop QA turns not containing any of: `graph/diagram/figure/chart/axis/plot/table/image/visual/illustrat` (string match), which can remove legitimate conceptual follow-ups - **Metric choice**: they mostly report BLEU/ROUGE/CIDEr for reproducibility; they discuss GPT-4 evaluation issues (inconsistency with chain-of-thought prompting; resource limits; nondeterminism) - **Internal inconsistency to note**: the abstract text claims LLaVA-13B is "most performant" zero-shot at CIDEr 0.08, but Table 2 lists **OpenFlamingo v2-7B** at **CIDEr 0.12**, higher than 0.08. Also the paragraph mentions prompting **GPT-3.5**, while Table 2 labels **DePlot+GPT-3**. (This may be a reporting mismatch, but it is present in the paper text/tables) # Reproducibility Details * * * ## Model **Baselines evaluated:** - MLLMs: BLIP2-2.7B; mPLUG-owl-7B; LLaVA-7B; LLaVA-13B; OpenFlamingo v2-7B - "Expert system" style: DePlot + (mPLUG-owl / LLaVA-13B / GPT-3.x) **Vision and language backbones (as described):** - Evaluations generally use a **fixed CLIP/ViT** vision encoder; mPLUG-owl is noted as an exception (encoder unfrozen in its own training, per related-work discussion) - Fine-tuning: uses a LLaVA checkpoint with **LLaMA-2-13B-chat** as base model **Prompt augmentation (DePlot):** - DePlot produces a text table with chart title/legends and **interpolated values**; they serialize it into a single string and prepend to the question prompt (Figure 4, p.8) * * * ## Data **Source corpus:** - ~**290,000** ArXiv papers (CS or stat.ML), **2010--2020** - Figures extracted (mentions PDFFigures 2.0); of ~**2.1M** figures, top categories include tables (23.6%), graphs (19.2%), flowcharts (8.5%); graphs chosen as focus **SciGraphQA dataset stats:** - Generation target: multi-turn dialogues about graphs using Palm-2, with context: title, abstract, caption, first paragraph referencing the figure, OCR text; in-context examples used in prompting (Figure 1, p.5; Appendix prompt on p.11--12) - Pipeline yields **350K** initial samples; after filtering, **295K** multi-turn entries - Total size: **59.1M tokens** (byte-wise BPE). Avg turn count **2.23**; **111K** samples have 3+ turns - Per-turn verbosity: avg **143 chars** per question and **775 chars** per answer; avg **39 tokens** per question and **164 tokens** per answer; tokens per sample **199 ± 98** **Quality rating:** - GPT-4 judge on 3K test subset: average **8.7/10**; authors report 86% are rated 8.5+; small tail of low ratings (Figure 2, p.6) **Dataset availability:** - Publicly available via GitHub repository and Hugging Face datasets - GitHub repo contains multiple dataset artifacts: 295K training set and 3K test set, sizes "excluding images" - Licensing note from repo README: "data, code and checkpoint is intended and licensed for research use only" with uses restricted to those that follow Palm-2, LLaMA and GPT-4 license agreements - Practical guidance: treat as publicly accessible for research-only use; commercial applications should seek clarification from authors * * * ## Algorithms / Training **Question filtering:** - Removes questions deemed "not graph-related" via keyword list: `graph, diagram, figure, chart, axis, plot, table, image, visual, illustrat` **Fine-tuning recipe (LLaVA-13B baseline):** - Two-step: (1) 1 epoch on full 295K dataset; (2) fine-tune on 30K subset with DePlot-augmented prompts - Optimization: cosine LR schedule, **warmup ratio 3%**, learning rate **$5 \\times 10^{-6}$** (explicitly lower than prior LLaVA instruction-tuning LR) - PEFT: **LoRA rank 64**, **LoRA dropout 0.05** * * * ## Evaluation **Metrics and setup:** - Test set: **3K** (image, question, answer) samples; compute **BLEU-4, ROUGE, CIDEr** between generated and reference answers **Key result table (Table 2, p.8):** | Model | Finetuned on SciGraphQA | DePlot table in prompt | CIDEr | BLEU-4 | ROUGE | | ---------------------------- | ----------------------: | ---------------------: | ----: | -----: | ----: | | BLIP2-2.7B | No | No | 0.007 | 0.003 | 0.10 | | DePlot + mPLUG-owl-7B | No | Yes | 0.037 | 0.058 | 0.22 | | mPLUG-owl-7B | No | No | 0.040 | 0.062 | 0.22 | | LLaVA-7B | No | No | 0.048 | 0.070 | 0.18 | | LLaVA-13B | No | No | 0.080 | 0.070 | 0.23 | | OpenFlamingo v2-7B | No | No | 0.120 | 0.081 | 0.22 | | DePlot + GPT-3 (label) | No | Yes | 0.130 | 0.098 | 0.226 | | DePlot + LLaVA-13B | No | Yes | 0.153 | 0.106 | 0.273 | | DePlot + SciGraphQA-baseline | Yes | Yes | 0.268 | 0.123 | 0.31 | **Additional evaluation notes:** - OpenFlamingo: authors attribute underperformance to not reproducing the retrieval-based in-context selection (RICE); random 3/6/9-shot in-context examples did not help in their trials - Dataset-size scaling: performance improves with more training data; biggest gains in the first ~50% of dataset; authors speculate LoRA (vs full fine-tuning) may limit gains from 50% to 100% (Figure 5, p.9) * * * ## Hardware / Production - Fine-tuning run uses **DeepSpeed ZeRO-2** with **4× A100-80GB** GPUs (Azure) - Batch sizing: per-device batch size **16**, gradient accumulation **2**, effective global batch size **128**