# PlotQA: Notes > Large-scale dataset for question answering over scientific plots with open-vocabulary, real-valued answers - Authors: Hunter Heidenreich - Published: 2019-09-01 - Updated: 2026-03-04 - Paper: PlotQA Paper - Code: PlotQA Code - Dataset: PlotQA Dataset - Project: PlotQA Project Page - Source: Bevaya Labs, --- ## TL;DR PlotQA is a large-scale dataset for question answering over scientific plots that targets real-world variability and open-vocabulary, real-valued answers that often are not explicitly written in the image. The paper reports that common VQA baselines fail badly on these OOV questions, and proposes a hybrid system: classification for simple fixed-vocabulary questions, and a perception-to-table-to-semantic-parsing pipeline for harder reasoning questions. --- ## What kind of paper is this? * **Dominant:** $\Psi_{\text{Resource}}$ (dataset + annotations at scale are the headline contribution: 224,377 plots and 28.9M QA pairs; also bounding boxes for plot elements). * **Secondary:** $\Psi_{\text{Method}}$ (hybrid model + staged pipeline for OOV numeric reasoning). * **Secondary:** $\Psi_{\text{Evaluation}}$ (analysis showing model failures on OOV; pipeline module breakdown for VED/OCR/SIE). (Informal coefficients: $\approx 0.45\,\Psi_{\text{Resource}} + 0.35\,\Psi_{\text{Method}} + 0.20\,\Psi_{\text{Evaluation}}$.) --- ## What is the motivation? * Existing plot-QA datasets (e.g., FigureQA, DVQA) are synthetic and often assume answers are either from a small fixed vocabulary or directly extractable text from the image. * Many realistic plot questions require **numeric reasoning** over floating point values where the answer is **not** a visible string in the chart and is **not** in a small vocabulary (example shown: averaging multiple bars to get 51.67). * PlotQA aims to close this gap by using real-world data sources and question templates derived from crowd-sourced questions, with heavy emphasis on open-vocabulary / real-valued answers. --- ## What is the novelty? ### Dataset design * **Scale:** 224,377 plots; 28,952,641 question-answer pairs. * **Plot types:** bar plots, line plots, scatter (dot-line) plots. * **Real-world data sourcing:** crawled from sources such as World Bank Open Data, Open Government Data, and Global Terrorism Database; 841 indicator variables and 160 entity types (countries, cities, etc.). Data spans 1960–2016 with values from 0 up to $3.50\times 10^{15}$. * **Visual variability knobs:** grid lines on/off, font size, tick label notation (scientific-E vs standard), line style (solid/dashed/dotted/dash-dot), marker style (asterisk/circle/diamond/square/triangle/inverted triangle), legend positions (multiple placements), and colors chosen from 73 colors. X-axis discrete elements vary from 2–12; legend entries from 1–4. * **Annotations:** bounding boxes for title, legend box, legend names and markers, axis titles, axis ticks, bars, lines, etc., intended to support supervised perception submodules. ### Question/answer taxonomy * Questions partitioned into a $3\times 3$ grid: * Question type: **Structural Understanding**, **Data Retrieval**, **Reasoning** * Answer type: **Yes/No**, **Fixed Vocabulary**, **Open Vocabulary (OOV)** * Key empirical point: a large majority of questions are open-vocabulary for data retrieval and reasoning (no open-vocab structural questions). ### Model contribution (system approach) * Hybrid approach: 1. A **binary question classifier** routes questions to either: * **QA-as-classification** (predict from a small top-$k$ answer vocab) * **Multi-stage pipeline** for OOV / reasoning answers: VED $\rightarrow$ OCR $\rightarrow$ semi-structured table extraction $\rightarrow$ table QA via semantic parsing. --- ## What experiments were performed? ### Dataset construction and statistics * Crowd-sourcing stage: sample 1,400 plots; 5 workers per plot; 7,000 total questions collected on Mechanical Turk; workers encouraged to ask complex reasoning questions; pay $0.1 per question. * Template stage: manually distilled into 74 templates, then instantiated at scale using a semi-automated process with in-house paraphrasing for natural phrasing. * Split: Train 157,070 images (20,249,479 QA), Val 33,650 (4,360,648 QA), Test 33,657 (4,342,514 QA). ### Baselines compared on PlotQA * IMG-only (VGG19 image embedding $\rightarrow$ fixed-vocab classifier) * QUES-only (LSTM question embedding $\rightarrow$ fixed-vocab classifier) * SAN, BAN, LoRRA (models designed around fixed vocab and/or OCR-copy constraints) * Proposed hybrid model ### Additional evaluation on DVQA * Reports comparisons against SAN and SANDY-OCR and shows improved accuracy on DVQA test and DVQA test-novel. ### Module-level diagnostics (pipeline analysis) * VED accuracy measured by AP at IoU thresholds (0.5, 0.75, 0.9) by element class and overall mAP. * OCR evaluated in oracle mode (GT boxes) and pipeline mode (predicted VED boxes). * SIE evaluated via F1 on extracted table tuples $\{row, col, value\}$. --- ## What are the outcomes/limitations? ### Main quantitative outcomes * PlotQA accuracy (reported): * IMG-only 4.84%, QUES-only 5.35% * SAN 7.76% * BAN 0.01%, LoRRA 0.02% * **Hybrid model 22.52%** * DVQA accuracy (reported): * SAN 32.1% (test), 30.98% (test-novel) * SANDY-OCR 45.77% (test), 45.81% (test-novel) * **Hybrid model 57.99% (test), 59.54% (test-novel)** ### Human performance estimate * Human accuracy: 80.47% on 5,860 questions over 160 test images, using the same metric (with numeric tolerance). Main human errors attributed to numeric precision difficulties. ### Key limitations and failure drivers (as analyzed in the paper) * The staged pipeline is bottlenecked by **tight localization** requirements for plot element detection: while mAP@0.5 is high, performance drops sharply at IoU 0.9 for several classes (e.g., dotlines and title). Small box misalignment can cause large numeric value errors downstream. * OCR is relatively robust to VED noise (oracle 97.06% vs pipeline 93.10%), suggesting VED/SIE alignment issues are more critical than raw OCR quality in this setup. * Semi-structured extraction quality is imperfect even when VED mAP@0.5 looks strong: reported SIE table extraction F1 is 0.68, attributed to imperfectly tight boxes and mis-associations (example given where IoU 0.58 box changes inferred bar value from 680 to 760). --- ## Model ### High-level architecture (hybrid routing) * Input: plot image + question text. * Router: binary question classifier predicts whether the answer is in a small fixed vocabulary (simple) vs OOV / needs reasoning (complex). * If simple: QA-as-classification predicts from top-$k$ vocabulary (softmax over answer classes). * If complex: staged pipeline: 1. **VED:** detect bounding boxes for plot elements and classify them 2. **OCR:** read text in detected boxes (titles, tick labels, legend labels) 3. **SIE:** construct a semi-structured table from detections + OCR + geometric/color heuristics 4. **Table QA:** answer the question by semantic parsing over the table-as-KG ### Visual Elements Detection (VED) * Data-bearing elements treated as 10 classes (title, axis labels, tick labels, legend markers/names, bars/lines, etc.). * Implementation choice: Faster R-CNN with Feature Pyramid Network (FPN) selected after comparing multiple detection methods (Fast R-CNN, YOLO, SSD, Mask R-CNN cited as candidates). ### OCR * Crop each detected textual element to its bounding box, grayscale, resize/deskew, then run OCR. * Paper states a pretrained OCR module performs well on machine-written English text in these plots. ### Semi-Structured Information Extraction (SIE) * Target table: rows are x-axis tick values; columns are legend entries; cell $(i,j)$ stores value for tick $i$ and legend $j$. * Heuristics described: * Legend name $\leftrightarrow$ legend marker/color: nearest bounding box association * Tick label $\leftrightarrow$ tick mark: nearest bounding box association * Bar $\rightarrow$ x-tick: closest tick label box to bar box * Bar $\rightarrow$ legend series: dominant bar color matched to legend marker color * Bar value: infer bar height from box geometry; find y-tick labels immediately above/below and interpolate. ### Table Question Answering * Table converted to knowledge graph; question mapped to candidate logical forms using compositional semantic parsing; logical forms ranked via log-linear model; top logical form executed to produce the answer. * Motivation: supports numeric reasoning without forcing answers into a small classification vocabulary. --- ## Data ### Sources and scope * Crawled structured data from multiple public sources (World Bank Open Data, Open Government Data, Global Terrorism Database, etc.). * 841 indicator variables and 160 unique entities; years 1960–2016 (not all variables cover all years). Values include integers, floats, percentages, and linear-scale values; range reported as 0 to $3.50\times 10^{15}$. ### Plot generation * 224,377 plots produced by combining indicators and entities. Plot-level randomization over visual style settings described earlier (grid/font/notation/line/marker/legend position/colors). * Provides bounding box labels for many plot components to support supervised perception training. ### QA generation * 7,000 crowd questions collected first, then abstracted into 74 templates, then instantiated with paraphrasing to avoid unnatural template fills. Total QA pairs: 28,952,641. ### Where to download (public) * **Paper shortlink:** the paper states the dataset (and crowd-sourced questions) can be downloaded from **bit.ly/PlotQA**. * **Official GitHub repo:** the PlotQA repository includes a "download the dataset" link. ([GitHub](https://github.com/NiteshMethani/PlotQA)) * **Official project site:** the PlotQA webpage provides direct download links (Google Drive) for **Plot Images**, **Annotations**, and **QA Pairs**. ([NLP at IIT Madras](https://iitmnlp.github.io/PlotQA/)) ### Licensing * The **datasets** (i.e., PlotQA data) are released under **CC-BY-4.0**. ([GitHub](https://github.com/NiteshMethani/PlotQA)) * The **models & code** are released under the **MIT license**. ([GitHub](https://github.com/NiteshMethani/PlotQA)) --- ## Algorithms / Training ### Baseline SAN training details (as reported) * Image features: last pooling layer of VGG19. * Question features: last hidden state of LSTM. * Both mapped to 1024-d via FC, combined and passed through $\tanh$. * Optimizer: Adam; learning rate 0.0003; batch size 128; 25,000 iterations. ### Question router (binary classifier) * 50-d word embeddings $\rightarrow$ LSTM with 128 hidden units $\rightarrow$ projection to 256-d $\rightarrow$ binary output layer. * Training: 10 epochs with RMSProp; learning rate 0.001; validation accuracy 87.3%. * Label generation for router training: for each question, label is 1 if QA-as-classification performs better than multi-stage pipeline, else 0. ### VED training * Faster R-CNN + FPN trained using PlotQA bounding boxes. * Batch size 32; 200,000 steps; RMSProp learning rate 0.004. ### Table QA training * Trained using questions from PlotQA paired with corresponding ground-truth tables. --- ## Evaluation ### Metrics * Primary metric: accuracy. * Textual answers require exact match. * Numeric answers treated as correct if within 5% of ground truth (to avoid strict float exact-match). ### Reported PlotQA results * Large gap between proposed model (22.52%) and human (80.47%). * Baselines that assume fixed vocab or OCR-copy answers perform very poorly, consistent with the dataset's emphasis on OOV numeric reasoning. ### Pipeline diagnostics (reported) * VED: overall mAP is strong at IoU 0.5 (96.43%) but degrades at stricter IoU 0.9 (72.29% overall), with especially sharp drops for some classes (e.g., title, dotline). * OCR: 97.06% (oracle boxes) vs 93.10% (after VED boxes). * SIE: table extraction F1 = 0.68; argued to be sensitive to tight localization and alignment, motivating improved VED for structured images. --- ## Hardware / Production * The paper provides training-step counts and batch sizes for key modules (SAN and VED) and optimizers/learning rates, but does **not** report concrete hardware specs (GPU type/count), wall-clock training time, or throughput/latency.