# ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries > Large-scale chart summarization benchmark with 84,363 charts featuring both short system-generated and long human-written summaries, revealing that models produce fluent text but struggle with factual correctness - Authors: Hunter Heidenreich - Published: 2023-04-25 - Updated: 2026-03-04 - Paper: ChartSumm Paper - Code: ChartSumm Code - Dataset: ChartSumm Dataset - Source: Bevaya Labs, --- ## TL;DR ChartSumm is a chart-to-text benchmark dataset with 84,363 chart samples, each with chart images, metadata, and paired summaries spanning short system-generated summaries (Knoema) and longer descriptive human-written summaries (Statista). The authors benchmark T5 and BART variants and report that while models can produce fluent summaries, they often fail on factual correctness, trend description, and exhibit hallucination. * * * ## What kind of paper is this? $\\Psi_{\\text{Resource}}$ 0.70, $\\Psi_{\\text{Evaluation}}$ 0.30 **Dominant: Resource (0.70)** --- The headline contribution is a new large-scale benchmark dataset for chart summarization with defined splits, dual summary regimes (short vs. long), and documented chart types/topics. The authors explicitly release the dataset and code. **Secondary: Evaluation (0.30)** --- The paper includes baseline benchmarking with T5 and BART variants, cross-dataset generalization comparisons (Chart-To-Text), manual error analysis of 100 sampled generations, and multilingual exploration (Bengali). However, it does not introduce novel evaluation protocols or metrics. * * * ## Key Questions ### 1. What is the motivation? Chart summarization benefits visually impaired users and improves information retrieval by converting chart/tabular insights into natural language. The field is data-constrained: prior datasets are limited in size, coverage, or summary quality/type (e.g., captioning-only or template-generated descriptions). ChartSumm aims to address this by providing a larger, more varied benchmark with both short and long summaries from diverse chart types and topics. ### 2. What is the novelty? **Dataset scale and dual-summary regimes:** - **84,363 charts** with images, metadata, and summaries from two sources: - **Knoema:** 43,179 charts with short descriptions generated by its "digital data assistant" (Yodatai), primarily year-indexed line charts - **Statista:** 41,184 charts with human-written descriptive summaries; includes multiple chart types with "simple vs. complex" categorization **Test set design for summary length:** - **test-k:** From Knoema, featuring "precise and well structured" shorter summaries - **test-s:** From Statista, featuring longer descriptive summaries **Multilingual exploration:** - Bengali expansion via machine translation with human-translated test set and mT5 baseline **Key distinction from prior work:** The dual-summary regime allows evaluation of models across different summary styles (short/precise vs. long/descriptive), which prior datasets did not systematically address. ### 3. What experiments were performed? **Baselines:** - T5-Base - BART-Base, BART-Large-CNN, BART-Large-XSUM **Training regimes:** - Fine-tune on ChartSumm full dataset, ChartSumm-K (Knoema subset), and ChartSumm-S (Statista subset) - Compare against models fine-tuned on Chart-To-Text (Kantharaj et al., 2022) - Evaluate on multiple test sets including Chart-To-Text Statista test split **Metrics:** - BLEU, BLEURT (base-128), CIDEr - Perplexity (using pretrained GPT-2) - Content Selection (CS; Wiseman et al., 2017) **Error analysis:** - Manual review of 100 sampled generations, categorizing common failures: factual errors (wrong numbers/units), wrong trend descriptions, uninformative summaries, hallucinated irrelevant facts ### 4. What are the outcomes? **Main results (as reported):** - BART-Large variants tend to lead on BLEURT, CIDEr, and CS metrics - T5 tends to achieve best perplexity across tests - Models fine-tuned on ChartSumm-S reportedly outperform Chart-To-Text trained baselines even on the Chart-To-Text test set, suggesting stronger generalization - Models trained on Chart-To-Text transfer poorly to ChartSumm-K (short/precise style), indicating Chart-To-Text is less suited for structured summaries **Limitations and failure modes:** - **Critical issue:** Even fluent outputs often contain wrong facts/units (e.g., "million" vs. "billion"), incorrect trend interpretation, and hallucinated attributes not grounded in chart metadata - **Evaluation gap:** Primarily automatic-metric-driven; qualitative error analysis reveals that BLEU/BLEURT scores do not correlate well with factual correctness - **Generalization concerns:** The authors acknowledge that models trained on one summary style (short vs. long) do not transfer well to the other > **Contrast to UniChart:** While UniChart (2023) also addresses chart-to-text tasks, ChartSumm focuses on providing a benchmark with dual summary styles and explicit error categorization, whereas UniChart emphasizes pretraining on diverse chart tasks (table extraction, reasoning, QA, summarization) with GPT-distilled summaries. ### 5. What are good follow-ups? **Concrete experiment: Factual consistency as a primary metric** Establish a factual consistency evaluation protocol (e.g., using structured claim extraction + verification against chart data tables) and re-rank model outputs by factual correctness rather than fluency-oriented metrics. This directly addresses the paper's core finding that automatic metrics miss critical factual errors. **Motivation:** The manual error analysis reveals that the highest-scoring outputs by BLEU/BLEURT often contain factually incorrect numbers or trends. A follow-up that prioritizes factual grounding would better align evaluation with the downstream task requirements (especially for accessibility applications where accuracy is critical). * * * ## Reproducibility Details ### Model **Task formulation:** - Table-to-text summarization using chart metadata - Input: chart metadata (title + data table + labels) - Table is flattened row-wise and concatenated with caption/title using separator tokens - T5-style prompting uses prefix: "Summarize chart: " **Baselines:** - T5-Base - BART-Base - BART-Large-CNN - BART-Large-XSUM ### Data **Sources and composition:** ChartSumm contains 84,363 charts from two sources: 1. **Knoema (43,179 charts):** - Crawled ~110,000 statistics, filtered to publicly available sources - Charts are year-based line charts - Summaries: short descriptions from Knoema's digital assistant (Yodatai) 2. **Statista (41,184 charts):** - Crawled ~750,000 pages - Charts categorized as simple vs. complex based on column count - Summaries: descriptive human-written text **Chart type distribution (Statista):** - Bar: 64.70% - Line: 33.76% - Pie: 1.54% **Topic coverage (via LDA):** - Economy & Politics: 21.60% - Society & Science: 13.03% - Internet & Media: 11.43% - Public life & Health: 10.42% - Sports & Entertainment: 9.14% - Consumer Goods: 7.71% - Retail & Trade: 5.35% - Education: 5.32% **Preprocessing:** - Tokenize title/caption, remove whitespace/newlines via stemming - Normalize numeric entities - Missing x-axis labels assigned heuristically (Year/Month/Day/Quarter/Country/City/Area) - NER-based types for companies, social media, etc. - Chart type classification uses ChartReader **Splits (80/10/10):** | Split | Knoema | Statista | Total | | ---------- | ------ | -------- | ------ | | Train | 34,503 | 32,985 | 67,488 | | Validation | 4,338 | 4,101 | 8,439 | | Test | 4,338 | 4,098 | 8,436 | | **Total** | 43,179 | 41,184 | 84,363 | **Overlap with Chart-To-Text:** - Overlap defined as >90% token similarity in captions - Reported overlaps (Statista samples): 5,338 total (4,144 simple, 1,194 complex) **Dataset statistics:** | Source subset | Avg cell count | Avg summary length (tokens/chars) | Avg title length (tokens/chars) | | ------------------ | -------------- | --------------------------------- | ------------------------------- | | Knoema | 55.44 | 34.76 / 207.69 | 8.86 / 57.18 | | Statista (simple) | 13.31 | 46.96 / 288.68 | 9.58 / 63.08 | | Statista (complex) | 37.95 | 55.54 / 340.19 | 10.54 / 67.08 | ### Algorithms / Training **Fine-tuning setup (English baselines):** - Epochs: 3 - Batch size: 8 - Initial learning rate: $1 \\times 10^{-6}$ - Optimizer: AdamW - Loss: Cross-entropy - Implementation: HuggingFace Transformers - Compute: Google Colab (specific GPU type not specified) **Bengali experiment (multilingual):** - Translation: - Train/validation: machine-translated using NLLB - Test: human-translated by undergraduate students proficient in English and Bengali - Model: mT5 pretrained on multilingual XL-SUM - Fine-tuning: - Epochs: 4 - Batch size: 8 - Initial learning rate: $1 \\times 10^{-6}$ - Optimizer: AdamW - Loss: Cross-entropy - Evaluation: BLEU only (BLEURT/CIDEr not used due to lack of Bengali-specific models) ### Evaluation **Metrics (English):** - BLEU (n-gram overlap) - BLEURT (base-128) - CIDEr - Perplexity (via GPT-2) - Content Selection (CS; Wiseman et al., 2017) **Manual error analysis (100 samples):** Categories identified: - Wrong facts/units (e.g., "million cubic meters" vs. gold "billion cubic meters") - Incorrect trend interpretation - Uninformative outputs (restates framing without key numbers/trends) - Hallucinated unrelated details (e.g., company headquarters not in chart/metadata) **Key finding:** Automatic metrics (BLEU, BLEURT) do not reliably correlate with factual correctness. High-scoring outputs can contain critical factual errors. ### Hardware / Production - English experiments conducted in Google Colab - Detailed hardware specs (GPU type, wall-clock training time) not specified in paper * * * ## Reproducibility Assessment - **Dataset available:** Released via GitHub + Google Drive with documented structure - **License unclear:** No LICENSE file in repository; reuse rights not explicitly granted - **Baseline code:** Available in GitHub repository - **Training hyperparameters:** Fully specified (epochs, batch size, LR, optimizer) - **Hardware details:** Limited (only "Google Colab" mentioned; no GPU type or training time) - **Evaluation protocol:** Metrics clearly documented; error analysis methodology described - **Preprocessing details:** High-level description provided; exact heuristics for label assignment may require code inspection