# Chart-to-Text: Notes

> Benchmark for generating natural-language summaries of charts

- Authors: Hunter Heidenreich
- Published: 2022-03-01
- Updated: 2026-03-04
- Paper: Chart-to-Text Paper <https://aclanthology.org/2022.acl-long.277/>
- Code: Chart-to-Text Code & Data <https://github.com/vis-nlp/Chart-to-text>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

The paper introduces **Chart-to-Text**, a benchmark for generating natural-language summaries of charts, with **two datasets** (Statista and Pew) totaling **44,096 charts** across multiple chart types and topics, plus baselines spanning **image captioning**, **table-to-text**, and **OCR-to-text** settings. Results suggest pretrained seq2seq models (BART/T5) are strongest, but **hallucinations**, **factual errors**, and **trend/pattern reasoning** remain key failure modes, especially when the underlying data table is unavailable.

* * *

## What kind of paper is this?

-   **Dominant:** $\\Psi\_{\\text{Resource}}$

    -   Primary contribution is a **large-scale benchmark** (two datasets, construction + analysis) and public release framing.
-   **Secondary:** $\\Psi\_{\\text{Evaluation}}$

    -   Strong baseline suite + automatic metrics + human evaluation + qualitative error analysis.
-   **Tertiary:** $\\Psi\_{\\text{Method}}$

    -   Some method engineering (OCR pipelines, chart-text role classifier, ChartOCR extension), but not the main novelty.

* * *

## What is the motivation?

-   Charts are widely used for communicating quantitative information, but extracting key insights can require substantial cognitive/perceptual effort; summaries can help readers, authors, accessibility use cases, and retrieval/indexing.
-   Prior work is limited by:

    -   **Small datasets**, narrow chart coverage (often just bar/line), and single-source collections.
    -   **Template-heavy systems** that describe how to read charts rather than synthesizing insights.
    -   Lack of baselines leveraging modern large-scale pretraining for generation.

* * *

## What is the novelty?

-   **Benchmark scale + breadth:** two sources, many topics, multiple chart types, and two task settings:

    1.  **Table-available** chart-to-text: input includes chart image + underlying table + metadata.
    2.  **Image-only** chart-to-text: underlying table unavailable, requiring extraction from chart images (OCR-based).
-   **Pew pipeline for chart-summary alignment** when pages contain many paragraphs and charts:

    -   OCR extraction, chart-text role classification, candidate paragraph heuristics, and crowd labeling for relevance.
-   **Baseline sweep** spanning:

    -   image captioning (vision-only),
    -   data-to-text (table-to-text),
    -   OCR+text hybrids.

* * *

## What experiments were performed?

-   **Dataset analysis:** chart-type distribution, linguistic stats (tokens/sentences), semantic content categories.
-   **Automatic evaluation:** BLEU, CIDEr, BLEURT, Content Selection (CS), and perplexity using GPT-2 Medium.
-   **Baseline comparisons** across Statista and Pew, with variants that use:

    -   ground-truth tables (TAB-\*),
    -   OCR text (OCR-\*),
    -   automatically extracted tables (TAB_OCR-\*).
-   **Human evaluation:** pairwise comparisons among TAB-T5, OCR-T5, and gold summaries on factual correctness, coherence, fluency.
-   **Qualitative error analysis** on sampled outputs to categorize failures (hallucination, factual errors, reasoning over trends).

* * *

## What are the outcomes/limitations?

-   **Best-performing family:** pretrained seq2seq (TAB-T5 / TAB-BART when tables exist; OCR-T5 on Pew), but performance drops sharply in Pew (image-only, diverse styles, missing tables).
-   **Persistent issues:** hallucinations and factual errors, especially in OCR-based settings where value-to-label association is brittle, plus difficulty describing complex trends/patterns that humans perceive easily.
-   **Measurement gap:** good fluency and reasonable overlap metrics can coexist with factual mistakes; human evaluation highlights factual correctness deficits relative to gold summaries.

* * *

# Reproducibility Details

* * *

## Data

### Sources and scale

-   **Statista**

    -   Crawled **34,810** publicly accessible webpages (Dec 2020), yielding **34,811 charts**.
    -   Collected: chart screenshot, downloaded data table (when available), title, axis labels, and human-written description text.
-   **Pew Research**

    -   Scraped **3,999** publicly accessible pages (Jan 2021), yielding **9,285 charts**.
    -   Underlying data tables are usually unavailable (only **143 charts** had tables).
    -   Collected: chart image, surrounding paragraphs, and alt text (if present).
-   **Total:** **44,096 charts** across both datasets.

### Chart complexity definition

-   **Statista:** "simple" charts have data tables with **two columns**; "complex" charts have **$\\ge 3$ columns** (e.g., grouped/stacked bars, multiple-line charts).
-   **Pew:** complexity labeled **manually** because tables are generally missing.

### Summary selection and annotation

-   **Statista summary selection:** used the first part of the webpage text (from chart icon to next heading) as the summary; remaining text often contains background.
-   **Statista x-axis label completion:**

    -   Many charts lacked explicit x-axis labels.
    -   Used regex heuristics over cell values to detect common entity types; remaining missing labels handled via Wikidata-based entity typing, then **manual annotation** when labels were too generic.

### Pew: 3-stage alignment pipeline (Figure 2)

**(i) Data extraction from chart images**

-   OCR text extracted using **CRAFT**.
-   Extracted bounding boxes and geometric features; trained **gradient boosting classifiers** to categorize recognized text into: **title**, **axis labels**, **legends**, **data labels**.
-   Separate classifier per chart type.
-   Manual labels: **319 examples** (171 bar, 68 line, 80 pie) split 8:1:1 into train/val/test.
-   Reported performance: **95.0% precision overall**, **97.6% precision for title classification** on test.
-   Title choice: if alt text exists, take the longer of (alt text, OCR-title); otherwise use OCR-title.

**(ii) Identification of candidate paragraphs**

-   Candidate set: paragraph adjacent to the chart plus five before and five after (max 11).

-   Heuristic relevance score:

    -   Sentence relevance:
        $$s_i = 0.58 l_i + 1.4 n_i - 0.5 u_i$$
        where $l_i$ is lexical matches, $n_i$ numerical matches (excluding years), $u_i$ numerical tokens in sentence not in chart.
    -   Content score:
        $$\\text{content}=\\frac{1}{1+\\exp(0.3(-\\max_i(s_i)+1.7))}$$
    -   Proximity score (distance $dist \\in [-5,5]$):
        $$\\text{proximity}=0.4\\exp(-0.1|dist|^2)+0.6$$
    -   Paragraph relevance:
        $$rel = \\text{content} \\times \\text{proximity}$$
    -   Paragraph considered relevant if constraints hold, including $rel > 0.72$, lexical matches sum $>3$, numerical/year matches $>0$, and no extra numerical tokens ($\\sum u_i = 0$). (Figure 7)

-   Heuristic evaluation (random sample): **recall 21.1%**, **precision 100%**, chosen to prioritize precision.

**(iii) Selection of relevant paragraphs via crowdsourcing**

-   Annotated **5,478 charts** and **13,237 paragraphs**.
-   Two annotators per chart; agreement used when both label irrelevant vs relevant; disagreements (**2,888 paragraphs**) resolved internally.
-   Reported overall agreement: **78.2%**.

### Splits and descriptive stats

-   Train/val/test split: **70% / 15% / 15%**.
-   Chart types (Table 1): bar charts dominate both datasets; line charts second; Pew includes additional types (area/scatter).
-   Linguistic stats (Table 2): Pew summaries are ~2× longer than Statista by characters/tokens/sentences; complex charts tend to have longer summaries.
-   Semantic content analysis (Table 3): "statistical/comparative" content most common; Pew has more "perceptual/cognitive" sentences than Statista.

* * *

## Model

### Task formulation

-   Dataset instance: $\\langle C, T, M, S \\rangle$ where

    -   $C$ chart image,
    -   $T$ data table (when available),
    -   $M=(C_{title}, C_{type}, C\_{labels})$ metadata,
    -   $S$ reference summary.
-   Two input settings:

    -   table-available: $X=\\langle C, T, M\\rangle$
    -   image-only: $X=\\langle C, M\\rangle$ (must recover info from chart image)

### Baseline families (Section 4)

**1) Image captioning (vision-only)**

-   Show, Attend, and Tell style: **ResNet50** encoder + **uni-directional LSTM** decoder.
-   ResNet50 pretrained via **Barlow Twins** self-supervision (separate pretraining per dataset) because ImageNet-pretrained ResNet transferred poorly to charts.

**2) Data-to-text (table-to-text)**

-   **Chart2text** (adapted transformer, with auxiliary content selection objective, plus templating strategy to reduce hallucination).
-   **Field-Infusing** model: LSTM encodes cell values, concatenated with row index + column heading embeddings, then a Transformer encoder-decoder generates text.
-   **BART / T5**: flatten table row-by-row; input includes title + table content; T5 uses prefix `"translate Chart to Text:"` to mimic pretraining style.

**3) Vision+text hybrids (OCR-to-text)**

-   Use **CRAFT** OCR to extract chart text, then feed to text generation models (Chart2text, Field-Infuse, BART, T5).
-   OCR-T5 has a variant that injects bounding-box positional embeddings (inspired by Tan and Bansal style spatial encoding).

* * *

## Algorithms / Training

### Training setup (Appendix A.3)

-   Hardware: CPU **Intel Xeon Gold 6240 @ 2.60GHz**, GPU **4× NVIDIA GTX 2080 Ti**.
-   Training time note: T5 fine-tuning is reported as the most expensive, ~**16--20 hours** on 4 GPUs.

### Model-specific training details

-   **Chart2text:** 1 encoder layer, 6 decoder layers, dropout 0.1, **80 epochs**, batch size 6; beam size 4 at inference.
-   **Field-Infusing:** 10 epochs, dropout 0.1, batch size 1.
-   **BART-Base:** ~140M params, 6 layers; fine-tune **500K iterations**, batch size 4, LR 0.0005; validate every 2,000 iters; beam size 4 at inference.
-   **T5-Base:** ~220M params, 12-layer encoder-decoder; fine-tune **500K iterations**, batch size 4, LR 0.0005; validate every 2,000 iters; beam size 4 at inference.

* * *

## Evaluation

### Automatic metrics (Section 5.1)

-   **BLEU** and **CIDEr**: n-gram overlap (CIDEr TF-IDF weighted).
-   **BLEURT-base-128**: learned metric for grammaticality/semantic similarity (sentence-level averaged).
-   **Content Selection (CS):** overlap in selected records relative to gold (sentence-level averaged).
-   **Perplexity (PPL):** computed with **GPT-2 Medium** for fluency proxy.

### Main quantitative results (Table 4)

-   **Statista (table-available is strongest setting):**

    -   Best BLEU among reported baselines: **TAB-T5 37.01**, **TAB-BART 36.36**.
    -   OCR-only variants are slightly worse (example: **OCR-T5 35.29**) but still competitive, with generally lower PPL.
    -   Image captioning baseline has relatively low CS even if PPL is low.
-   **Pew (mostly no tables, more diverse styles):**

    -   Best BLEU among reported baselines: **OCR-T5 10.49** (OCR-T5\* 10.42).
    -   Vision-only image captioning collapses (BLEU ~4).

### Human evaluation (Section 5.2, Table 5)

-   Setup: **150** Statista charts; **4** internal annotators (native English); **450** pairwise comparisons:

    -   TAB-T5 vs OCR-T5
    -   Gold vs TAB-T5
    -   Gold vs OCR-T5
-   Criteria: factual correctness, coherence, fluency.
-   Agreement on a subset (excluding ties): **74.3%**.
-   Outcome: **TAB-T5 beats OCR-T5** strongly on factual correctness (and also coherence/fluency), while **gold summaries** still win more often than either model, especially on factual correctness/coherence.

### Error analysis themes (Section 5.3)

-   **Perceptual and reasoning failures:** models struggle with trends/relationships that are visually salient but not trivial from raw extracted text (examples shown in Figure 4).
-   **Hallucinations:** fluent but irrelevant tokens/statements.
-   **Factual errors:** especially OCR-based, due to missing data labels or mis-association between values and entities (Figure 4 example where follower counts swap entities).
-   **Computer vision constraints:** charts often omit explicit numeric labels, and OCR alone does not recover mark-to-label alignment.
-   **Proposed direction (as an aspiration):** richer representations such as semantic graphs encoding numerical/logical relations among chart objects.

* * *

## Hardware / Production

-   Reported training environment: Xeon Gold 6240 CPU, **4× GTX 2080 Ti**, with T5 fine-tuning taking **~16--20 hours** in their setup.
-   No serving/latency benchmarks; evaluation is offline.

* * *

## Appendix-specific implementation detail: automatic table extraction (Appendix A.5)

-   They extend **ChartOCR** to recover fully-structured tables:

    -   Keypoint detection for chart elements and marks; extend detector to include textual labels and legend marks.
    -   OCR (CRAFT) recognizes x-axis/legend labels; associate values to nearest labels and series by color; estimate scale using y-axis labels.
-   Reported automatic extraction accuracy: **77.31%** (used to create TAB_OCR-\* model inputs).

* * *

## Notes on ethics and misuse (brief)

-   Dataset collection constrained to publicly available charts with publication rights considerations (Statista free studies; Pew attribution/terms).
-   AMT compensation targeted to minimum wage rates; per-chart payments 0.10--0.15 USD depending on candidate paragraphs.
-   They explicitly call out a misuse risk: fluent outputs with hallucinations/factual errors could misinform if published uncorrected.

* * *

# Data Availability & Licensing Notes

## TL;DR

Yes: the benchmark datasets are **publicly available online** (GitHub), and the repository is licensed under **GNU GPL v3**. But: the charts come from **third-party sites** (Statista, Pew), and your rights to redistribute/reuse those chart images can be constrained by **their Terms of Use**, even if the repo itself is GPL.

* * *

## Questions answered

### 1) Is there data publicly available?

**Yes.** The paper explicitly says the "code and benchmark datasets are publicly available" (via their GitHub).

### 2) Online? Where?

**Yes (online).** The stated primary location is the public GitHub repository [vis-nlp/Chart-to-text](https://github.com/vis-nlp/Chart-to-text). (There are also public mirrors/derivatives in the ecosystem, but GitHub is the canonical source per the paper.)

### 3) With what licensing?

**Repository license:** The repo includes a **GNU General Public License v3.0** license file.

**Important nuance (data vs. code):**

-   GPL v3 clearly applies to "software and other kinds of works" released under it.
-   However, **the dataset contains/derives from third-party chart content**, and that content may carry **separate legal/contractual restrictions** (see below).

* * *

## Third-party terms that affect what you can do with the data

### Statista (as used by the dataset)

The authors say they use **only "free studies"** from Statista and cite Statista's Terms: free content comes with "publication rights" for academic purposes, but _paid content does not_---and they claim to use only the free portion.

The Statista Terms document explicitly discusses redistribution limits and conditions, including that some redistribution is only allowed for **"free material"** and typically requires leaving materials unchanged and referencing Statista. It also prohibits "crawlers/spiders" (relevant because the dataset was constructed via crawling).

### Pew Research Center

Pew's Terms grant a license to use Pew content **with attribution**, and include a specific requirement to "provide proper attribution... in accordance with the citation below." They also restrict reuse of content attributed to _another_ party that is not Pew.

* * *

## Practical implications (what this means for you)

-   If you just need a **benchmark to reproduce research results**, the GitHub release being public + GPL'd is straightforward.
-   If you plan to **redistribute the dataset**, host it, or use it in a **commercial product**, you should treat the **third-party chart/image rights** (Statista/Pew) as potentially more restrictive than the repo's GPL label.
-   If your use-case needs clean licensing, you may prefer datasets built from **explicitly open-licensed charts** (or datasets whose creators confirm redistribution rights unambiguously).