# SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials > Chart-to-table extraction method using simple chart distillation, row-column rendering, and human-oriented prompting to improve both table extraction and chart QA performance - Authors: Hunter Heidenreich - Published: 2024-05-22 - Updated: 2026-03-04 - Code: SIMPLOT Code - Source: Bevaya Labs, --- ## TL;DR SIMPLOT improves **chart-to-table extraction** (and downstream chart QA) by (1) training on **essential-only "simple charts"** to avoid chart noise, (2) **distilling** an encoder so original charts map into the simple-chart representation space, (3) adding **row/column rendering** supervision, and (4) prompting an LMM with a **human-oriented chart instruction** so it uses both the **extracted table + original image** at inference. Reported gains include better table extraction on ChartQA (RDF1) and higher QA relaxed accuracy than Deplot in their table-plus-LMM setting. * * * ## What kind of paper is this? **Dominant:** $\\Psi\_{\\text{Method}}$ - Main contribution is an end-to-end **method/training recipe** (preprocess + 2-phase training + inference prompting) to improve chart-to-table extraction and chart QA. **Secondary:** $\\Psi\_{\\text{Evaluation}}$ - Proposes a **modified metric** (Relative Distance, RD) derived from RMS to better evaluate chart-to-table extraction when header strings differ but reasoning is unaffected. * * * ## What is the motivation? - Prior "chart-to-table then LLM reasoning" approaches (e.g., Deplot) struggle because charts mix **essential** and **irrelevant** information, and this "noise" degrades table extraction (examples include missed units and inclusion of irrelevant text like attribution). - Table-only reasoning also misses **purely visual questions** that require chart geometry or visual attributes (e.g., "third bar from the top", "what year does the orange line represent?"). * * * ## What is the novelty? ### 1) Essential-only training signal via "simple chart generation" - Uses existing datasets' chart-to-CSV table annotations to **re-plot** a simplified chart (Matplotlib) from the ground-truth table, excluding irrelevant chart artifacts; training on these "simple charts" improves extraction. ### 2) Row-column rendering (explicit supervision) - Renders **row/column header info onto the image** during training (since ground-truth table exists). - For inference, since ground-truth is unavailable, an LMM is used to extract row/column text from the chart and render it onto the image (Appendix B with an illustrative example). ### 3) Two-phase distillation to map original charts into the "simple chart" representation space - **Phase 1:** train a teacher encoder + table decoder using essential-only (simple chart) pairs. - **Phase 2:** train a student encoder on original charts with a **triplet loss** (anchor = original chart; positive = simple chart; negative = value-shuffled simple chart) plus a **table cross-entropy loss**, to generate accurate tables from original charts. ### 4) Human-oriented chart instruction prompt (for LMM reasoning using image + table) - A chart-specific prompt that encodes stepwise "how humans interpret charts" (universal rules + chart-type-specific rules for bar/line/pie). It is used so the LMM can resolve questions that require aligning the predicted table with chart positions and visual cues. * * * ## What experiments were performed? ### Benchmarks / tasks - **ChartQA**: pie/bar/line charts; QA includes human-authored + LLM-augmented questions; evaluation reports Relaxed Accuracy on 2,500 test questions. - **PlotQA**: dot line/line/bar; large synthetic QA set; they sample 10% of images for training/inference to reduce cost. - Additional evaluations discussed: **OpenCQA** (open-ended chart QA; BLEU) and **MultiChartQA** (comparative/sequential multi-chart reasoning). ### Comparisons / baselines (high level) - Vision-language pretraining baselines and supervised chart models are compared for ChartQA QA. Table-based reasoning baselines include **Deplot** and **Unichart (table extraction variant)**, with GPT-4V used for reasoning in the "extract table then LMM" setting for fairness. ### Key ablations - Impact of (a) row-column rendering and (b) distillation from simple charts on table extraction, plus (c) the human-oriented prompt on QA. - Effect of using **image + table** vs table-only for Unichart/Deplot reasoning and for failure cases where both methods have low table extraction quality. - Negative-sample (triplet) benefit is evaluated (small but positive). * * * ## What are the outcomes and limitations? ### Outcomes (reported) - **ChartQA table extraction (RDF1):** SIMPLOT reported higher overall RDF1 than Deplot and UniChart in their setup. - **ChartQA QA (RA):** In the "table + LMM" category, SIMPLOT reports substantially higher overall relaxed accuracy than Deplot in their table-plus-LMM comparison table. - **Ablations:** row/column rendering and simple-chart distillation each improve table extraction; adding the human-oriented prompt improves QA. - **Harder questions:** when questions require referencing multiple rows/columns, SIMPLOT's reported gap vs a "Deplot + image + prompt" baseline increases, supporting the claim that table extraction quality matters more under harder compositional queries. ### Limitations / failure modes (reported) - Expected reduced performance on unseen OOD charts; they argue one round of table-extraction training can adapt to OOD better than per-task training, but still flag OOD as future work. - Practical error cases: row/column extraction can split entities due to line breaks; LMM can fail at fine-grained color identification (example: confusing orange vs red). - Misuse risk: high-performing chart interpretation could be used to mislead; authors recommend critical verification. * * * ## Model ### Backbone and components - Built on **Deplot-style** chart-to-table: an **image encoder** and a **text/table decoder** (they follow Deplot and use **ViT** encoders). - Naming: chart encoder $E_{\\text{chart}}$ and table decoder $D_{\\text{table}}$; Phase 1 produces a **teacher** encoder $E^{\\text{teacher}}_{\\text{chart}}$; Phase 2 trains a **student** encoder $E^{\\text{student}}_{\\text{chart}}$. ### Tokenization / table linearization details - Table sequence uses `|` to separate cells and `<0x0A>` as line break, following Deplot. * * * ## Data ### Training inputs created by preprocessing - **Simple charts:** generated by re-plotting from ground-truth CSV tables using Matplotlib; intended to contain only "essential" information needed for reasoning. - **Row-column rendering:** in training, rows/columns from the ground-truth table are rendered onto the paired chart image; in inference, an LMM extracts the row/column strings for rendering. ### Dataset stats (as reported) - ChartQA split counts are provided (train/val/test counts by chart type; 2,500 QA pairs in test). - PlotQA is very large; they sample 10% of PlotQA images stratified by type for training/inference, and use one QA pair per image in their reduced setting. ### Data release - **Not explicitly released.** The authors **do not describe releasing a new SIMPLOT dataset** or any additional data artifacts; instead, they use **existing public datasets** (ChartQA and PlotQA) and note those are "publicly accessible for research purposes." - They describe generating "simple charts" offline from the datasets' **existing CSV tables**, which suggests their pipeline can _produce derived artifacts_, but they **don't state they publish those derived data outputs**. * * * ## Algorithms / Training ### Preprocessing stage (offline) 1. **Simple Chart Generation:** re-render chart from CSV table using Matplotlib. 2. **Row-Column Rendering:** render row/column header strings onto images; inference uses an LMM to extract these headers. ### Training stage: chart-to-table extraction **Phase 1 (teacher):** fine-tune Deplot backbone on (simple chart, table) pairs. - Output: teacher encoder representations focus on essential info; $D\_{\\text{table}}$ and an FC layer are later frozen. **Phase 2 (student):** train $E^{\\text{student}}\_{\\text{chart}}$ on original charts to match teacher-space outputs. - Define original chart $A$, simplified positive $P$, negative $N$ (shuffled values). - Representations: - $z_a = D_{\\text{table}}(E^{\\text{student}}\_{\\text{chart}}(A))$ - $z_p = D_{\\text{table}}(E^{\\text{teacher}}\_{\\text{chart}}(P))$ - $z_n = D_{\\text{table}}(E^{\\text{teacher}}\_{\\text{chart}}(N))$ - Triplet loss: $$ L\_{\\text{triplet}}(A,P,N)=\\max{d(z_a,z_p)-d(z_a,z_n)+m,0} $$ with $d$ as $\\ell_2$ distance and margin $m$. - Table generation cross-entropy $L_{\\text{table}}$ over the linearized table tokens, and final loss: $$ L_{\\text{final}}=\\lambda L_{\\text{triplet}}+(1-\\lambda)L_{\\text{table}} $$ with $\\lambda=0.1$ reported. ### Inference stage: reasoning - LMM receives **(original chart image + extracted table)**, plus the **human-oriented chart instruction** prompt so it can answer both numeric table queries and visual-attribute queries. * * * ## Evaluation ### Metrics - **Chart QA:** Relaxed Accuracy (RA) on 2,500 ChartQA test questions. - **Chart-to-table extraction:** they discuss RMS and introduce **Relative Distance (RD)** with RDF1 as the harmonic mean of precision/recall using numeric relative distance after minimal-cost matching over header strings. - **Open-ended QA:** BLEU for OpenCQA comparisons. ### Notable reported result patterns (qualitative) - Table-based reasoning (table + LMM) generally outperforms image-only supervised models on ChartQA in their comparison, and SIMPLOT is reported best within their table-based group. - Adding the **image** at inference helps table-based methods answer questions that need positional/visual interpretation, and the prompt further improves this. * * * ## Hardware / Production - GPU: **NVIDIA RTX A6000**. - Training time per epoch: Phase 1 about **2 hours/epoch**, Phase 2 about **4 hours/epoch**; they train Phase 1 for **7 epochs** and Phase 2 for **9 epochs**. - Parameter count: **374M parameters** (SIMPLOT model); GPT-4 parameter count is unknown (as noted). - LMM prompting: temperature **0.1** is mentioned.