# ChartQA: Notes > Benchmark for question answering about charts with visual and logical reasoning - Authors: Hunter Heidenreich - Published: 2022-05-01 - Updated: 2026-03-04 - Paper: ChartQA Paper - Code: ChartQA Code & Data - Source: Bevaya Labs, --- **TL;DR:** ChartQA is a **benchmark for question answering over real-world charts** requiring **visual references** (color, position, size) and **multi-step logical or arithmetic reasoning**. It contains **9,608 human-written questions** and **23,111 machine-generated questions** over **20,882 charts** from Statista, Pew Research, OWID, and OECD. The paper evaluates a pipeline combining **chart data extraction** (extended ChartOCR) with **TableQA-style transformers** augmented with image features (VL-T5, VisionTaPas), showing that **VisionTaPas achieves 61.12% accuracy** on human questions with gold tables but drops to **44.33% with extracted tables**. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Resource}}$ 0.60 - **Secondary:** $\\Psi_{\\text{Method}}$ 0.25, $\\Psi_{\\text{Evaluation}}$ 0.15 The primary contribution is a **new large-scale benchmark** with real-world charts and human-authored questions. The paper also introduces models (VisionTaPas) and provides detailed evaluation, but these serve to establish baselines for the benchmark. * * * ## What is the motivation? Existing chart question answering datasets and models under-serve: - **Complex reasoning questions** requiring multiple operations (difference, sum, max, etc.) - Questions that **refer to visual attributes** (e.g., "orange line", "rightmost bar") - **Human-authored language** with natural variation, not small template sets - **Real-world chart styles** with diverse layouts and visual complexity Prior datasets (DVQA, PlotQA, FigureQA) are largely **synthetic** and/or **template-based**, limiting their ability to evaluate systems on realistic chart understanding tasks. * * * ## What is the novelty? **Benchmark contributions:** - **ChartQA-H (human-authored)**: 9,608 questions collected via Amazon Mechanical Turk, with workers explicitly asked to write **compositional** and **visual** questions requiring multi-step reasoning or visual references - **ChartQA-M (machine-generated)**: 23,111 questions generated from human-written Statista chart summaries using a two-stage T5 pipeline (answer extraction + answer-aware question generation), then filtered for answerability - **Real-world chart diversity**: 20,882 charts from four sources (Statista, Pew, OWID, OECD) covering multiple domains and chart types - **Open-vocabulary answers**: Unlike prior work with closed-vocabulary or template-based answers, ChartQA requires generating free-form text or numeric answers **Modeling contributions:** - **VisionTaPas**: extends TaPas with a **cross-modality encoder** that fuses ViT image features with TaPas table encodings via cross-attention blocks - **Operation extension**: adds **SUBTRACT** and **DIVIDE** operations to TaPas (which originally supported SUM, COUNT, AVERAGE) to handle difference and ratio questions - **Extended ChartOCR**: adapts ChartOCR to output fully-structured data tables (not just mark values) by adding text recognition with CRAFT and associating values to labels using positional and color information * * * ## What experiments were performed? **Dataset construction:** - **ChartQA-H annotation**: AMT workers write 2 questions + answers per chart; second annotator answers the same questions; manual resolution on disagreements - **ChartQA-M generation**: fine-tune T5 on SQuAD, apply to Statista summaries; filter questions whose answers don't appear in chart data; manual check of 1,250 pairs found **86.64% valid** **Models evaluated:** - **T5**: flatten table + question into text sequence; generate answer - **TaPas**: table encoder with row/column embeddings; heads for aggregation + cell selection - **VL-T5**: T5 with visual mark features from Mask R-CNN (36 objects) - **VisionTaPas**: TaPas extended with ViT image encoder and cross-modality fusion via cross-attention **Experimental conditions:** - **Gold tables**: use ground-truth data tables extracted from chart sources - **Extracted tables**: use tables predicted by their extended ChartOCR pipeline **Ablations:** - Impact of operation extension (SUBTRACT + DIVIDE) on TaPas and VisionTaPas - Comparison of gold vs. extracted tables to isolate extraction quality impact * * * ## What are the outcomes/limitations? **Main results (relaxed accuracy: non-numeric exact match, numeric within 5% relative error):** | Model | ChartQA-H (Gold) | ChartQA-H (Extracted) | ChartQA-M (Gold) | ChartQA-M (Extracted) | | --------------- | ---------------- | --------------------- | ---------------- | --------------------- | | T5 | 55.88 | --- | 64.32 | --- | | TaPas | 58.29 | 42.61 | 70.77 | 55.82 | | VL-T5 | 53.93 | --- | 65.28 | --- | | **VisionTaPas** | **61.12** | **44.33** | **74.47** | **59.88** | **Key findings:** - **VisionTaPas outperforms all baselines** on both human and machine-generated questions when using gold tables - **Accuracy drops significantly with extracted tables**: VisionTaPas goes from 61.12% to 44.33% on ChartQA-H, indicating **data extraction is a major bottleneck** - **Operation extension matters**: adding SUBTRACT + DIVIDE improves VisionTaPas from 65.19% to 74.47% on ChartQA-M (gold table) - **Data extraction accuracy**: overall extraction accuracy on ChartQA reported as **83.85%** using a normalized distance metric with linear assignment **Question type distribution (sample of 300 human questions):** | Type | Percentage | | --------------------------- | ---------- | | Compositional | 43.0% | | Visual | 10.7% | | Both visual + compositional | 33.3% | | Data retrieval | 13.0% | 76.33% of questions require compositional reasoning or combine visual + compositional reasoning. **Limitations called out by the authors:** - **Extraction brittleness**: modular pipeline breaks when table extraction fails; motivates end-to-end approaches - **Separate representations**: current approach combines table and vision "separately then combine"; authors propose **semantic graph representations** to better capture chart structure - **Nested reasoning difficulty**: even with correct extraction, models struggle with multi-step computations (e.g., subtraction then sum over multiple years) - **Extraction metric limitations**: their adapted ChartOCR metric ignores noisy chart text (tick labels) and needs refinement # Reproducibility Details * * * ## Data **Chart sources:** - **Statista**: crawled publicly available charts - **Pew Research**: crawled charts (images only, no underlying tables available) - **Our World in Data (OWID)**: crawled charts with underlying data - **OECD**: crawled charts with underlying data **What they store:** - Chart images - Underlying data tables (when available) - Metadata: title, type, source - SVG files (when available, used to extract bounding boxes for training extraction models) - Text descriptions (for machine-generated question synthesis) **Dataset splits:** | Split | ChartQA-H | ChartQA-M | | ----- | ---------------------- | ------------------------ | | Train | 3,699 charts (7,398 Q) | 15,474 charts (20,901 Q) | | Val | 480 charts (960 Q) | 680 charts (960 Q) | | Test | 625 charts (1,250 Q) | 987 charts (1,250 Q) | **Chart type distribution (Statista-M subset, Table 3):** - Bar: 15,223 - Line: 1,768 - Pie: 150 **Human annotation procedure (ChartQA-H):** 1. AMT workers write **2 questions + answers** per chart, focusing on compositional and visual questions 2. Second annotator answers the same questions independently 3. Manual resolution on disagreements 4. **Agreement metrics**: exact-match 61.04%; manual check on 500 samples yields 78.55% when accounting for typos/lexical variation 5. **Compensation**: $0.6 per task (estimated 3--5 minutes), relative to US minimum wage ($7.25/hour at time of study) **Machine augmentation procedure (ChartQA-M):** 1. Fine-tune T5 on SQuAD 2. Apply to Statista chart summaries: - **Answer extraction model**: generate candidate answers from summary - **Answer-aware question generation model**: conditioned on (answer + summary) 3. **Filter**: remove questions whose answer is not found in chart data table 4. **Quality check**: manual verification of 1,250 generated pairs found **86.64% valid** 5. Test set manually cleaned to ensure quality * * * ## Model ### VisionTaPas Architecture Extends TaPas with cross-modal fusion: - **ViT encoder**: processes image patches (standard Vision Transformer) - **TaPas encoder**: processes question + flattened table with row/column embeddings - **Cross-modality encoder**: 4 blocks, each containing: - Bidirectional cross-attention (image ↔ table) - Self-attention layers - Feed-forward layers - Residual connections - **TaPas heads**: aggregation head (selects operation: SUM, COUNT, AVERAGE, SUBTRACT, DIVIDE) and cell selection head ### VL-T5 Visual Features - Train **Mask R-CNN** (ResNet-101 backbone) to detect chart marks - Object categories: xAxisTitle, yAxisTitle, legend, label, bar, pie, line, etc. - Pad object features to **36 objects** (fixed size) - Feed visual features alongside text into T5 ### Operation Head Extension - Original TaPas supports: SUM, COUNT, AVERAGE - **Extension adds**: SUBTRACT, DIVIDE - **Supervision**: heuristic rules for selecting operand cells - **Noise level**: manual check of 100 questions found **24% noisy labels** ### Data Extraction Pipeline Extends ChartOCR to output structured tables: 1. **Value detection**: detect numeric values on chart (ChartOCR baseline) 2. **Text recognition**: add CRAFT detector to recognize axis labels, legend text, tick labels 3. **Value-to-label association**: use positional information (x/y coordinates) and color matching to link values to semantic labels 4. **Table construction**: assemble into structured rows and columns * * * ## Training All training on **1 Tesla P100 GPU**. | Model | Epochs | Time | Batch Size | Learning Rate | | ----------- | ------ | --------- | ---------- | ------------- | | TaPas | 30 | ~10 hours | 32 | 5e-5 | | VisionTaPas | 20 | ~30 hours | 16 | 5e-5 | | T5 | 20 | ~11 hours | 8 | 5e-4 | | VL-T5 | 20 | ~15 hours | 16 | 5e-4 | All models fine-tuned from pretrained checkpoints (TaPas-Base, T5-Base, VL-T5-Base). * * * ## Evaluation **QA metric (relaxed accuracy):** - **Non-numeric answers**: exact string match after normalization (lowercase, strip whitespace, remove articles) - **Numeric answers**: correct if within **5% relative error** of ground truth **Data extraction metric (adapted from ChartOCR):** - Compute cost based on normalized distance between ground-truth and predicted values - Solve minimum-cost linear assignment between gt and predictions - Overall score: 1 minus average normalized cost - **Limitation noted by authors**: metric ignores noisy chart text (tick labels, axis titles); better metrics needed **Error analysis (by question type):** The authors perform manual error analysis on a sample of failures, identifying: - **Extraction errors**: incorrect or missing values in extracted table - **Reasoning errors**: model fails to perform correct operation sequence even with correct extraction - **Visual grounding errors**: model cannot resolve visual references (e.g., "rightmost bar") * * * ## Ethical Considerations **Data sourcing:** - Used publicly available charts under source terms (Statista, Pew, OECD, OWID) - Authors state they complied with each source's terms of service - AMT annotators anonymized **Annotator compensation:** - Estimated 3--5 minutes per task - Paid $0.6 per task - Authors frame relative to US minimum wage at time ($7.25/hour) **Potential for misuse:** - Authors explicitly note models could be abused to mislead about chart content or manipulate chart-based arguments - Recommend caution in deployment for public-facing applications without human oversight * * * ## Practical Takeaways **If we were to use ChartQA:** - **Best for**: evaluating chart QA systems on real-world language with explicit visual references and multi-step reasoning - **ChartQA-H is the gold standard**: human-authored questions with natural language variation, typos, synonyms - **ChartQA-M is larger**: useful for pretraining or augmentation, but slightly lower quality (86.64% valid) - **Extraction quality is critical**: performance drops ~17 absolute points when using extracted vs. gold tables, so any deployment would need robust extraction - **Operation space matters**: if your questions involve arithmetic (differences, ratios), extending operation heads significantly improves TaPas-style models (+9 points for VisionTaPas on ChartQA-M) **Debugging strategy:** - Separate extraction errors from reasoning errors - The authors' results show extraction accounts for roughly half the error gap - Test with gold tables first to isolate model reasoning capabilities **Chart types:** - Benchmark heavily skewed toward **bar charts** (>88% of Statista-M) - Line and pie charts underrepresented - Results may not generalize to other chart types (scatter, heatmap, etc.) **License considerations:** - GNU GPL-3.0 applies to code and dataset files - Underlying charts from third-party sources have their own terms - For commercial use, review source terms (Statista especially restrictive)