# ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization

> ChartThinker combines chart parsing (OCR + DePlot), context-enhanced CoT generation, and retrieval to improve chart summarization; introduces Chart-Sum-QA dataset (595K charts, 8.17M QA pairs)

- Authors: Hunter Heidenreich
- Published: 2024-04-01
- Updated: 2026-03-04
- Code: ChartThinker Code <https://github.com/Notonion/ChartThinker>
- Dataset: Chart-Sum-QA Dataset <https://huggingface.co/datasets/ChartThinker/Chart-Sum-QA>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

ChartThinker targets chart summarization failure modes in VLMs: (1) poor chart-text **matching degree** (omissions, hallucinated values) and (2) **reasoning errors** about the chart's intended message. It introduces (a) a large chart-caption + instruction QA dataset (Chart-Sum-QA) and (b) a method that combines chart parsing (OCR + DePlot), chain-of-thought (CoT) style staged generation, and a small retrieval library for in-context examples.

* * *

## What kind of paper is this?

**Dominant:** $\\Psi_{\\text{Method}}$ (new end-to-end approach: chart parsing + context-enhanced CoT generation + retrieval integration; extensive ablations).  
**Secondary:** $\\Psi_{\\text{Resource}}$ (release of Chart-Sum-QA: 595,955 chart-caption pairs + 8.17M instruction QA pairs).  
(Using the taxonomy defined in the provided guidelines.)

* * *

## What is the motivation?

The authors argue that large vision-language models often fail at chart summarization because:

-   **Insufficient matching degree:** summaries omit small-but-critical numbers/text, or fabricate chart content due to pretraining priors.
-   **Reasoning errors:** models misread trends/patterns or infer incorrect "takeaways" from complex charts.

Figure 1 illustrates both error types via examples comparing LLaMA-Adapter-v2 and MiniGPT-4 outputs to a chart, showing both hallucinations and incorrect inferences.

* * *

## What is the novelty?

1.  **Chart-Sum-QA dataset (scale + supervision breadth)**

    -   **595,955 chart-caption pairs** (pretraining) and **8,170,000 instruction question-answer pairs** (fine-tuning).
    -   Covers chart types: **scatter, line, bar, pie** (per Table 1).

2.  **Context-Enhanced CoT Generator (retrieval + staged "thoughts")**

    -   The model generates multiple **thought steps** (example steps: identify chart type; understand legend/axes; observe trends/proportions), and **for each thought** retrieves top-$k$ chart-text exemplars from a **small retrieval library** to use as context examples, then consolidates outputs into a final summary (overview shown in Figure 2).
    -   Retrieval uses **cosine similarity** between encoded chart features and stored example features, then uses an **order-based weighting** where example weight is proportional to $1/i$ for rank $i$ (Equations 3--4), and conditions generation on a weighted context function (Equation 5).

3.  **Chart parsing module that fuses OCR + DePlot outputs**

    -   OCR extracts textual/numeric strings; DePlot converts chart to a table (text-number alignment), and the module outputs:
        -   **text-number pairs** and
        -   **other text** (title/notes/etc.).
    -   The workflow is depicted in Figure 3.

* * *

## What experiments were performed?

### Benchmarks and baselines

They compare against two baseline groups:

-   **Classic (text-generation) transformer baselines** with OCR-augmented inputs: T5, Chart2text, Field-Infuse, BART, plus their OCR-ChartThinker variant (Table 2).
-   **Large VLM baselines** (encoder-decoder style): LLaMA-Adapter-v2, MiniGPT-4, mPLUG-Owl, LLaVA, plus ChartThinker (Table 3).

### Metrics

Automatic evaluation uses:

-   **BLEU**, **BLEURT (base-128)**, **CIDEr**, **Content Selection (CS)**, and **perplexity** using **GPT-2 Medium**; plus a normalized aggregate score **$S\_{\\text{norm}}$** over five indicators.

Human evaluation:

-   **200 generated summaries**, scored by 3 evaluators on **Matching Degree** and **Reasoning Correctness** (1--5).

### Ablations (Table 4)

Ablations remove:

-   chart parsing module,
-   context retrieval ("No Context-Enhanced"),
-   CoT ("No CoT"),
-   entire context-enhanced CoT generator,
-   fine-tuning on caption dataset,
-   fine-tuning on instruction dataset.

* * *

## What are the outcomes/limitations?

### Key results (as reported)

-   OCR-ChartThinker vs OCR baselines (Table 2): reported best BLEU (11.81), best CIDEr (2.21), best PPL (9.23), best $S_{\text{norm}}$ (0.948), though not best CS (32.72% vs OCR-T5 40.87%).
-   ChartThinker vs VLM baselines (Table 3): reported best across listed metrics among those VLM baselines (BLEU 5.82, CIDEr 1.58, CS 21.68%, PPL 11.43).
-   Human evaluation (Table 5): ChartThinker rated highest on Matching Degree (4.32) and Reasoning Correctness (4.27) among compared models in their study.

### Limitations / caveats (from the paper + inferred risks)

**Evaluation metric concerns:**

-   CS (Content Selection) metric shows counterintuitive behavior: the authors argue it can penalize longer correct summaries that include additional valid detail beyond the reference (Table 6 example). This raises questions about whether CS is a reliable primary metric.
-   Heavy reliance on automatic metrics (BLEU, CIDEr, BLEURT) may not capture summary quality aspects like clarity, coherence, or usefulness to end users.
-   Human evaluation is limited to 200 samples with 3 evaluators; inter-rater agreement is not reported.

**Data generation pipeline concerns:**

-   A large portion of instruction QA (exact ratio not specified) is generated using ChatGPT-4 from human-written summaries, which may introduce:
    -   Style biases toward GPT-4's generation patterns
    -   Potential data leakage if GPT-4 was trained on overlapping sources
    -   Limited diversity in question types and reasoning patterns
-   The paper mentions "a subset" is manually validated but does not specify validation coverage, criteria, or pass rates.
-   Prompt templates and filtering heuristics for QA generation are not fully detailed.

**Retrieval library limitations:**

-   Library contains only 1,000 chart-text pairs split across 4 stages (250 each).
-   Coverage of unusual chart designs, rare chart types, or domain-specific visualizations may be insufficient.
-   The paper does not analyze retrieval failure modes or cases where no good match exists.

**Architectural and scope constraints:**

-   Chart parsing module combines OCR + DePlot but may inherit limitations from both (e.g., DePlot's known weakness with color-dependent questions).
-   Context-enhanced CoT is stage-specific, but the paper does not justify the choice of 4 stages or explore alternative decompositions.
-   Generalization to chart types beyond scatter, line, bar, and pie is not evaluated.

**Reproducibility gaps:**

-   The paper provides LoRA hyperparameters but omits:
    -   Full hardware specifications (GPU type, memory, count)
    -   Wall-clock training time
    -   Inference throughput (tokens/sec, images/sec)
    -   Total training cost or energy consumption
-   Dataset mixing ratios and sampling strategies for the multi-source training corpus are not detailed.

* * *

## Research questions (explicitly answered)

The paper poses three RQs; below are the answers supported by the reported experiments:

-   **RQ1: Can answer reasoning benefit from introducing a chain of thought?**  
    Yes, per Table 4: removing CoT ("No CoT") reduces BLEU (5.10 vs 5.82) and lowers human score (3.92 vs 4.25), indicating CoT contributes to both automatic metrics and judged quality.

-   **RQ2: How can context retrieval and chain of thought effectively interact with each other?**  
    The reported synergy claim is supported by ablations: removing context enhancement reduces performance ("No Context-Enhanced": BLEU 5.45 vs 5.82; human 4.11 vs 4.25), and removing the entire combined module drops more ("No Context-Enhanced CoT Generator": BLEU 4.59; human 3.85). The method's interaction mechanism is "retrieve exemplars per thought step" (Figure 2).

-   **RQ3: How does instruction fine-tuning improve the chart-summary matching degree?**  
    Table 4 shows dropping instruction fine-tuning degrades BLEU (4.52 vs 5.82) and BLEURT (--0.63 vs --0.45), consistent with the claim that directive QA tuning improves chart-specific grounding.

* * *

## Model

### High-level architecture (from Figure 2)

Inputs: **chart image + prompt**. The pipeline includes:

-   **Image encoder** (CLIP-based encoder) producing chart feature vector $V$.
-   **Text encoder** for the prompt producing token sequence/features $K$ (paper text labels it "encoder" but describes generation with a decoder; see below).
-   **Chart parsing module** produces underlying data features by merging OCR + DePlot outputs and concatenating with prompt features.
-   **Context-Enhanced CoT Generator** generates a sequence of "thoughts"; for each thought it retrieves top-$k$ example chart-text pairs and uses them as in-context examples, then integrates outputs into the final summary.

### Generator backbone

-   Paper states the final generation is done with **Idefics** and also mentions a **LLaMA2 decoder** (the description mixes terms: "text encoder" vs "decoder"); the operational takeaway is: a VLM generator (Idefics) plus LoRA is used for conditioned generation.

### Retrieval library design (Appendix B summary in-text)

-   **1,000 chart-text pairs**, split into **4 stages** of **250** each:
    1.  chart type,
    2.  chart overview (title-driven),
    3.  axes meanings,
    4.  numerical trend description.

* * *

## Data

### Chart-Sum-QA composition (Table 1)

-   Aggregates multiple existing chart datasets (Autochart, Linecap, DVQA, PlotQA, Chart-to-text, FigureQA), yielding:
    -   **595,955 images** and
    -   **8,170,000 QA pairs** total (their constructed dataset).

### Construction steps (Section 3.1)

1.  **Data collection** from six datasets.
2.  **Preprocessing**: resize, standardize formats, clean titles/descriptions, ensure alignment.
3.  **Generate QA pairs**: create ~400k QA from summaries (Chart-to-text, Autochart, Linecap), merge with other QA datasets, filter, finalize 8.17M; use ChatGPT-4 to generate questions from human-written summaries; manually validate a subset.
4.  **Splits**: 80% train, 10% validation, 10% test.

* * *

## Algorithms / Training

### Chart parsing module (Figure 3)

-   **OCR output:** text strings and numbers (without positions).
-   **DePlot output:** a table of text-number correspondences (can be affected by irrelevant chart elements; may have numeric extraction errors).
-   **Fusion output:** separate **text-number pairs** (key aligned entities) and **other text**.

### Retrieval + weighting

-   Similarity: cosine similarity between input chart feature vector $V$ and example feature $T_i$ (Eq. 3).
-   Context weighting: rank-based weight $1/i$; aggregated weighting function $W\_\\ell(\\cdot)$ conditions generation at token $\\ell$ (Eq. 4--5).

### Fine-tuning implementation (LoRA)

-   LoRA rank **16**, applied to **qproj/kproj/vproj**.
-   LoRA scaling factor **$\\alpha = 32$**, dropout **0.05**.
-   Optimizer: **paged_adamw_8bit**, learning rate **$2\\times 10^{-4}$**.
-   Gradient accumulation: **8**; eval every **20** steps.

* * *

## Evaluation

### Automatic metrics

-   BLEU, BLEURT-base-128, CIDEr, CS, PPL (GPT-2 Medium), plus normalized aggregate score $S\_{\\text{norm}}$.

### Human evaluation protocol

-   200 summaries; 3 annotators; criteria:
    -   Matching Degree (data fidelity: minimal omissions/fabrications)
    -   Reasoning Correctness (intended message inferred correctly)
-   Rated 1--5; randomized order to reduce bias.

* * *

## Hardware / Production

Not fully specified. The paper provides the optimizer choice and LoRA settings, but does not give:

-   GPU type/count, training duration, batch size (beyond gradient accumulation), or inference throughput.

* * *

## Dataset release + license

### Do they share their dataset online?

**Yes.** The paper explicitly says their "dataset and codes are publicly accessible" and that they "release our dataset and codes at OpenChartThinker." A public repo associated with the project also points to a Hugging Face dataset upload ([GitHub: Notonion/ChartThinker](https://github.com/Notonion/ChartThinker)).

### Under what license?

On Hugging Face, the **ChartThinker/Chart-Sum-QA** dataset is labeled **MIT** ("License: mit"). ([HuggingFace: ChartThinker/Chart-Sum-QA](https://huggingface.co/datasets/ChartThinker/Chart-Sum-QA))

### Detail to be aware of

In the dataset construction section, the authors note the **source datasets** they aggregate are under various licenses (examples given: **CC BY-NC-SA 4.0, MIT, GPL3.0**). So, even though the HF card says **MIT**, it's worth double-checking provenance/constraints for any subset you plan to use (especially commercial use).