# DePlot: One-shot Visual Language Reasoning by Plot-to-Table Translation > DePlot decomposes chart QA into plot-to-table translation via an image-to-text model plus LLM reasoning, achieving 29.4% gains on ChartQA human queries with one-shot prompting - Authors: Hunter Heidenreich - Published: 2023-07-01 - Updated: 2026-03-04 - Code: DePlot Code - Model: DePlot Model - Source: Bevaya Labs, --- ## TL;DR DePlot decomposes chart question answering into (1) **plot-to-table translation** (image $\\rightarrow$ linearized table) and (2) **LLM reasoning** over the translated table using one-shot prompts. They train an image-to-text Transformer (initialized from MatCha) for plot-to-table, introduce a table-matching metric (**RMSF1**) meant to be more structure-aware than prior number-set matching, and report large gains on **human-written ChartQA** when combining DePlot with LLM prompting. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Method}}$. The headline contribution is a new **modality conversion module** (DEPLOT) plus a plug-and-play pipeline to use LLMs for chart QA with one-shot supervision, with extensive benchmark results. - **Secondary:** $\\Psi\_{\\text{Evaluation}}$. They propose and validate a new metric (RMSF1) and run a human-correlation study to argue it better reflects extraction quality than RNSS. - **Secondary:** $\\Psi\_{\\text{Resource}}$ (weaker). They standardize task formats/metrics for plot-to-table and train on a combined corpus, but the core novelty is not a new dataset release. (These labels follow the provided taxonomy.) * * * ## What is the motivation? - End-to-end chart QA systems need **large task-specific finetuning sets** and still struggle on **complex human-written questions** (example given: MatCha at 38.2% on ChartQA human). - LLMs have strong **few-shot reasoning**, but the missing piece is getting chart content into a form LLMs can reliably use without bespoke multimodal training. - Prior chart extraction systems are often **pipeline-based** (OCR + rules + detection) and chart-type-specific, with inconsistent evaluation metrics. * * * ## What is the novelty? - **Two-stage decomposition:** 1. Convert chart image to a **linearized table** (markdown-style table text). 2. Use an off-the-shelf LLM with **one-shot prompting** to answer questions from the table. - **DEPLOT model:** an end-to-end image-to-text Transformer finetuned specifically for plot-to-table translation, intended to be chart-type-agnostic (line, dot, bar, pie). - **RMSF1 metric:** a table similarity metric that (a) accounts for row and column headers plus values, (b) supports approximate matching for numeric/text errors, (c) exposes precision vs. recall, and (d) is invariant to row/column permutations and transposition. - **Prompting stack:** evaluates Chain-of-Thought, Self-Consistency, and Program-of-Thought prompting (including executing generated Python for arithmetic). * * * ## What experiments were performed? - **Plot-to-table evaluation:** PlotQA plot-to-table reconstruction compared against ChartOCR (pipeline), PaLI-17B finetuned variants, MatCha off-the-shelf, and DePlot (new finetune). Metrics: RNSS and RMSF1. - **Downstream QA:** ChartQA (augmented + human) and PlotQA (v1 + v2) QA accuracy with 5% numeric tolerance; compares fully supervised models vs. DePlot+LLM one-shot variants. - **Metric validation:** human study on 50 plot-table pairs with 6 annotators; correlates RNSS vs. RMSF1 with human ratings. - **OOD check:** annotate 10 TaTa charts (excluding choropleths) to estimate generalization; report average RMSF1 and qualitative failure modes. * * * ## What are the outcomes and limitations? **Key outcomes** - **ChartQA human:** best reported DePlot+LLM setup reaches **67.6%**, compared with MatCha **38.2%** (reported as +29.4 absolute points). - **ChartQA augmented:** DePlot+FlanPaLM+Codex PoT SC reported at **91.0%**, comparable to strong supervised baselines (MatCha 90.2). - **PlotQA QA:** DePlot+LLM underperforms MatCha on synthetic PlotQA (example: 66.6 vs. 91.5 average in their table), despite doing well on ChartQA human. - **Metric:** RMSF1 correlates better with human judgments than RNSS (Pearson's $r$ 0.87 vs 0.46; Spearman's $\\rho$ 0.96 vs 0.84). **Limitations and failure modes** - **Loss of visual attributes:** plot-to-table drops information like **color**, which breaks questions referencing "gray bar", "red line", etc. They show a concrete ChartQA failure where the LLM answers "Yes" for the wrong reason because the table has no color encoding. - **Synthetic-query bias:** they argue PlotQA's templatic synthetic questions can be exploited by supervised finetuning, which one-shot prompting cannot leverage. - **OOD robustness unclear:** in a small TaTa sample, they observe distraction by adjacent text and trouble with arrow-linked labels; they mention cropping helps in that setting. - **Scope:** they note DEPLOT is not suited to visual language without a clear latent textual structure (example: some textbook figures). * * * # Reproducibility Details ## Model ### DEPLOT architecture and representation - **Model type:** image-to-text encoder-decoder Transformer, trained autoregressively to emit a table left-to-right. - **Initialization:** starts from MatCha weights and continues finetuning specifically for plot-to-table conversion. - **Table format:** markdown-like linearization, using `|` separators between cells and newline between rows. - **Parameter count:** DEPLOT 282M parameters (LLMs used are much larger: FlanPaLM 540B; GPT-3 and Codex reported around 175B). ### DEPLOT + LLM interface - Output table is appended with the question and a **one-shot demonstration**; they evaluate natural-language reasoning (CoT) and code-generation reasoning (PoT) plus self-consistency. Figure 1 illustrates the overall flow. * * * ## Data ### Plot-to-table finetuning corpus Three sources mixed **1:1:1** (only training splits used to reduce leakage into downstream eval): - Synthetic plots from Liu et al. (MatCha pretraining pipeline): **270K** plot-table pairs. - ChartQA plot-table pairs: **22K**. - PlotQA plot-table pairs: **224K**. ### QA benchmarks and test sizes - ChartQA: human and machine (augmented) sets, each reported as **1,250 QA pairs** with **625** (human) and **987** (machine) tables. - PlotQA: v1 and v2 each with **33K tables**; QA pairs reported as **1.2M** (v1) and **4.3M** (v2). * * * ## Algorithms and training ### Training - **Training steps:** 10k steps. - **Max sequence length:** 512 tokens (they note MatCha used 192; they extend to fit longer tables). - **Inference temperature for DePlot:** 0 (deterministic decoding). ### Prompting and decoding for LLMs - **LLM prompting temperature:** 0.4 in their experiments. - **Self-consistency:** majority vote across 10 samples (they also combine CoT and PoT samples in a joint voting scheme for best ChartQA results). - **Program-of-Thought:** prompt Codex to emit Python snippets, then execute to compute the final answer; motivated by more reliable arithmetic. - Prompts are shown in the appendix figures (Figure 3: CoT-style table reasoning; Figure 4: Python-code variant). * * * ## Evaluation ### Plot-to-table metrics **RNSS (baseline)** - Treats predicted and target tables as **unordered sets of numbers** and matches them with a relative-distance cost, then normalizes by max set size. They define relative distance: $$ D(p, t)=\\min\\left(1, \\frac{|p-t|}{|t|}\\right) $$ and compute a minimal-cost matching matrix $X$ over predicted set $P$ and target set $T$, with: $$ RNSS = 1 - \\frac{\\sum_i\\sum_j X_{ij} D(p_i, t_j)}{\\max(N, M)} $$ as reported in the paper. **RMSF1 (proposed)** - Represents a table as an unordered set of (row header, column header, value) triples. - Matches entries by header similarity (normalized Levenshtein with thresholding) and value similarity (relative error with thresholding), then computes precision/recall and $F1$. - Handles transposition by scoring both table orientations and taking the better score. The core idea is that similarity is high only when both header keys and values align; unlike RNSS, this penalizes "right numbers, wrong structure". ### Plot-to-table results On PlotQA plot-to-table reconstruction (their Table 4): - ChartOCR: RNSS 81.0, RMSF1 60.1 - PaLI-17B (224 res): RNSS 77.2, RMSF1 24.8 - PaLI-17B (588 res): RNSS 90.5, RMSF1 74.9 - MatCha: RNSS 95.4, RMSF1 92.3 - **DePlot:** RNSS 97.1, RMSF1 94.2 The PaLI 224 vs 588 contrast is used to argue input resolution matters for chart extraction. ### Downstream QA results Key rows from their main results table (Table 5): - MatCha: ChartQA aug 90.2, human 38.2, avg 64.2; PlotQA avg 91.5 - DePlot+FlanPaLM+Codex PoT SC: ChartQA aug 91.0, human 67.6, avg 79.3; PlotQA avg 66.6 - They also report intermediate variants (GPT-3 CoT/SC, FlanPaLM CoT/SC, Codex PoT SC) showing the effect of prompting and tool-use. ### Error analysis highlights - "Visual attribute" questions (color, shape, orientation) are a recurring failure because the table encoding omits those attributes. Their Table 7 example centers on not being able to identify "highest value of the gray bar" after translation. - Another failure mode: imperfect alignment between plotted points and x-axis labels can cause incorrect table reconstruction (their Table 11 example). * * * ## Hardware and production - **Training hardware:** 64 TPUv3 on GCP. - **Training time:** roughly 5 hours for DEPLOT. * * * # Release & Licensing Notes ## What "other (synthetic) data" DePlot uses - For **plot-to-table training**, DePlot trains on a 1:1:1 mix of: 1. **synthetic data generated by Liu et al. (2023a)**, 2. **synthetic data generated by Methani et al. (2020)** (the same synthetic source used in **PlotQA**), and 3. **real-world data crawled by Masry et al. (2022)** (the same source used in **ChartQA**, sourced from Statista / Pew Research / Our World in Data / OECD). - In their **Table 2** training stats, they explicitly list **"synthetic (by us)" = 270K** alongside ChartQA and PlotQA portions. - They also note they use **only training-set charts** from ChartQA/PlotQA for training (to avoid leakage). ## Was that synthetic data shared publicly, or just code/concept? ### DePlot authors' _explicit_ release statement - The DePlot paper explicitly points to **"Code and models"** (a repo link in footnote), but **does not explicitly say** "we release the 270K synthetic plot--table pairs" as a downloadable dataset. - Their ethics statement says training/eval data are **synthetic via rules or publicly available web data with permissive licenses**---but that's about the **inputs' provenance**, not a clear statement that they redistribute the full synthetic corpus. **Best-supported takeaway:** from the paper text alone, it looks like **they publicly shared code + model artifacts**, and they **describe** the synthetic data + sources, but **they don't clearly claim a public release of the "synthetic (by us) 270K" dataset itself**. ## If shared, what licenses apply? ### ChartQA - **Availability:** ChartQA dataset is distributed via the repo and also via a Hugging Face dataset. - **License:** Repo lists **GPL-3.0**. ### PlotQA - **License breakdown (explicitly stated):** - **Dataset:** **CC-BY-4.0** - **Models + code:** **MIT** ### DePlot (code/models) - **Paper statement:** "Code and models" are provided via the google-research repo path referenced in the paper. - **Model (Hugging Face):** **Apache-2.0** - **google-research repository (where DePlot code lives):** repo license is **Apache-2.0** ## Questions answered 1. **Was the (synthetic) data shared publicly?** - **ChartQA / PlotQA:** yes, they're publicly distributed. - **DePlot's "synthetic (by us) 270K":** the paper does **not** clearly state that this synthetic corpus is released as a dataset download; it only explicitly calls out **code + models**. 2. **Or just the code?** - For DePlot: **code + models are explicitly shared**. 3. **Or just the concept?** - DePlot also fully describes the idea + training mixture, but the _explicit "released artifact"_ callout is **code + models**. 4. **If anything was shared, what license was attached to the data/code?** - **ChartQA:** GPL-3.0 - **PlotQA:** Dataset CC-BY-4.0; Models/code MIT - **DePlot:** Code in google-research under Apache-2.0; HF model card shows Apache-2.0 ## Open point (what I couldn't verify from the paper text alone) - **Exact license (if any) for DePlot's newly-generated "synthetic (by us) 270K" plot--table pairs** isn't stated in the paper sections we have---so I can't responsibly claim a data license for that specific synthetic corpus without checking the repo/docs that accompany the code release.