# DVQA: Notes > Large synthetic benchmark for bar-chart question answering with 300K charts and 3.5M QA pairs - Authors: Hunter Heidenreich - Published: 2018-04-01 - Updated: 2026-03-04 - Paper: DVQA Paper - Dataset: DVQA Dataset - Code: DVQA Code - Source: Bevaya Labs, --- ## TL;DR DVQA introduces a large synthetic benchmark for bar-chart question answering (300,000 charts; 3,487,194 QA pairs) designed to break standard VQA assumptions, especially fixed vocabularies and chart-specific text in questions/answers. Standard VQA baselines perform well mainly on "structure" questions but fail on chart-specific answers; the paper proposes two stronger baselines (MOM and SANDY) that explicitly handle chart text via OCR-driven mechanisms. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Resource}}$ - The headline contribution is the **DVQA dataset** (scale, splits, controlled variability, question templates, bias reduction). - **Secondary:** $\\Psi_{\\text{Method}} + \\Psi_{\\text{Evaluation}}$ - Two model variants aimed at OOV/chart-specific text (MOM and SANDY) plus extensive baseline comparisons and breakdowns (including chart-specific subsets). * * * ## What is the motivation? - Bar charts are common in scientific papers, web articles, and reports, but are **not machine-interpretable** by typical vision pipelines; even small appearance changes can invalidate heuristics. - DVQA is positioned as both: - a practical capability: **querying chart repositories** for numeric/semantic info, and - a stress test for multi-step attention, measurement, and reasoning that typical VQA does not cover. - The paper emphasizes three DVQA-specific challenges relative to natural-image VQA: 1. fixed vocabularies break on **chart-specific labels/answers**, 2. chart text tokens are **arbitrary and context-dependent**, and 3. charts are **brittle**: small style changes can completely alter meaning. * * * ## What is the novelty? ### Dataset novelty (DVQA) - **Scale:** 300,000 bar-chart images; **3,487,194** question-answer pairs. - **Three question families:** structure understanding, data retrieval, reasoning (template-generated but with large style/data variation). - **Two test regimes:** - **Test-Familiar:** only labels seen in training - **Test-Novel:** labels drawn from a disjoint set to force **OOV generalization**. - **Style diversity via Matplotlib:** variation in number of bars/groups, gridlines, color/width/spacing/orientation/texture, label/legend orientation and location. - **Bias controls:** randomization to decorrelate style/color/labels and downsampling to balance certain yes/no questions. ### Model novelty (two "strong baselines" for chart-specific text) - **MOM (Multi-Output Model):** pairs a conventional classifier with an OCR-based answer generator and chooses between them via a learned branch selector. - **SANDY (SAN with DYnamic encoding):** introduces a **dynamic local dictionary** built from OCR-detected text boxes to encode chart-specific words in questions and to output chart-specific answer tokens. * * * ## What experiments were performed? - Evaluated multiple baselines (YES, IMG, QUES, IMG+QUES, SAN-VQA) vs. **MOM** and **SANDY** on: - **Test-Familiar** and **Test-Novel** - breakdown by **Structure / Data / Reasoning** - targeted subsets: **chart-specific questions** and **chart-specific answers**. - Metric: exact string match for correctness; additionally reports **MOM (±1)** allowing edit distance $\\le 1$ to quantify near-miss OCR/string errors. - Additional transfer check (limited): manually annotated **500+ structure** questions on real bar charts scraped from the internet; SAN-based models reached ~59% without fine-tuning, reported as ~15% absolute gain over QUES. * * * ## What are the outcomes/limitations? ### Key outcomes - Standard VQA-style models do well on structure but struggle on chart-specific semantics: - **SAN-VQA overall:** 36.04 (Familiar), 36.14 (Novel) - **SANDY (Oracle) overall:** 56.48 (Familiar), 56.62 (Novel) - The biggest failure mode for fixed-vocab VQA is **chart-specific answers**: - SAN-VQA chart-specific answers: 0.10 (Familiar), 0.00 (Novel) - IMG+QUES chart-specific answers: 0.09 (Familiar), 0.00 (Novel) - SANDY (Oracle) chart-specific answers: 52.55 (Familiar), 52.70 (Novel) - MOM helps, but appears limited by string generation/localization: - MOM chart-specific answers: 12.78 (Familiar), 2.93 (Novel) - MOM (±1) improves substantially (23.62 / 12.47), suggesting many errors are "near miss" OCR/string issues. ### Limitations and open issues (as discussed) - SANDY's dynamic encoding is **cascade-sensitive**: if OCR misses a word, the positional chaining used to index the local dictionary can corrupt the whole mapping. - MOM depends on **accurate bounding box prediction** for the answer region; localization errors can propagate to decoding errors. - Dataset scope: DVQA in this paper is **bar charts only**; authors state a follow-up with pie charts/plots/other diagrams is planned. - Real-world validation is limited (mostly structure questions, ~500 examples), and reported without end-to-end real-chart data retrieval/reasoning evaluation. * * * ## Model ### Shared components / preprocessing - Image backbone for image-processing models: **ResNet-152** (ImageNet pretrained), input resized to $448 \\times 448$, producing a $14 \\times 14 \\times 2048$ feature tensor (unless noted). - Question encoder: 1-layer **LSTM** with 1024 hidden units; word embeddings are 300-dimensional. ### Baseline models - **YES:** always answers "YES" (noted as slightly more common than "NO"). - **IMG:** question-blind; pooled CNN features $\\rightarrow$ MLP (hidden 1024) $\\rightarrow$ softmax. - **QUES:** image-blind; LSTM features $\\rightarrow$ MLP (hidden 1024) $\\rightarrow$ softmax. - **IMG+QUES:** concat CNN+LSTM embeddings $\\rightarrow$ MLP (hidden 1024) $\\rightarrow$ softmax. - **SAN-VQA:** Stacked Attention Network over last conv feature maps, conditioned on LSTM question embedding (implementation aligned to a "strong SAN baseline" variant the paper cites). ### MOM (Multi-Output Model) - Dual-network architecture: 1. **Classification sub-network:** SAN-VQA for generic answers 2. **OCR sub-network:** predicts a text box then decodes characters from that region. - OCR branch details: - **BBox predictor:** regression with MSE loss - Crop patch from predicted region, resize to $128 \\times 128$ - Apply a small **3-layer CNN** - Use **N-step spatial attention** over the patch to get per-character features, with $N = 8$ (max sequence length in experiments) - Encode with **bidirectional GRU**, then decode with a character classifier trained with **CTC loss**. - Branch selection: - A separate binary classifier uses LSTM question features to choose "generic" vs "chart-specific"; reported as perfect on DVQA test data. ### SANDY (SAN with DYnamic encoding) - Replaces fixed word/answer dictionaries with a **Dynamic Encoding Model (DEM)**: - Assumes access to OCR boxes with positions + strings (oracle version uses dataset annotations; OCR version uses Tesseract). - Local dictionary construction: - Assign indices to detected text boxes based on a positional chaining procedure: start from lower-left as index 0, then repeatedly pick the nearest unassigned box to assign the next index. - Cap local dictionary size at **$M = 30$** (no training chart exceeded 30 text labels). - Usage: - Augments the global question dictionary (size $N$) with $M$ local entries for encoding. - Augments the global answer classes (size $L$) with $M$ local answer classes; predicting one maps back to a string via the local dictionary. - OCR version preprocessing (Tesseract): - keep only detections with alphabetic characters - filter confidence < 50% - drop single-character detections. * * * ## Data ### Dataset scale and splits - Total: **300,000 images**, **3,487,194** QA pairs, **1,576 unique answers** overall. - Splits: - Train: 200,000 images; 2,325,316 questions; 1,076 unique answers - Test-Familiar: 50,000 images; 580,557 questions; 1,075 unique answers - Test-Novel: 50,000 images; 581,321 questions; 577 unique answers. ### Chart text vocabulary construction - Uses nouns drawn from the **Brown Corpus**: - training + Test-Familiar: 1000 most frequent nouns - Test-Novel: 500 new words disjoint from training to create OOV conditions. ### Underlying bar-value regimes - Three data types: - **linear:** values from 10 randomly chosen values in range 1--10 - **percentage:** 10 randomly chosen values in range 10--100 - **exponential:** 10 randomly chosen values in range 1--$10^{10}$ - Some bars may be **zero**, rendered as a missing bar. ### Question families (examples) - **Structure understanding:** counts, grouping, stacked vs non-stacked, horizontal vs vertical, patterns vs solid, negative values, etc. - **Data retrieval:** scale type (log/percent), label identification (e.g., "third bar from the left"), legend-color mapping, "units sold" queries, etc. - **Reasoning:** argmax/argmin, comparisons, sums/differences, threshold counts, cross-category comparisons (e.g., algorithm $A_1$ on dataset $D_1$ vs $A_2$ on $D_2$). ### Bias minimization - Randomize chart generation to avoid correlations between styles/colors/labels. - Balance yes/no for selected question templates by randomly removing examples until balanced. * * * ## Algorithms / Training - Optimization: **Adam** with initial learning rate **0.001**. - Regularization: dropout **0.5** applied to inputs of convolutional, fully connected, and LSTM units. - Classification answer space: - global answer dictionary size from training: **1076** answers for most classifiers. - MOM OCR character decoder: - **27** output classes (26 letters + blank). - SANDY output layer: - described as **107** units in the paper's configuration, with indices 0--30 reserved for local dictionary entries and 31--107 for common answers. * * * ## Evaluation ### Overall accuracy (Table 3) Percent correct (exact match), Test-Familiar / Test-Novel: - **SAN-VQA:** - Structure: 94.71 / 94.82 - Data: 18.78 / 18.92 - Reasoning: 37.29 / 37.25 - Overall: 36.04 / 36.14 - **MOM:** - Data: 29.52 / 21.40 - Overall: 40.89 / 37.26 - **MOM (±1):** - Data: 38.20 / 29.14 - Overall: 45.03 / 40.90 - **SANDY (Oracle):** - Data: 65.40 / 65.55 - Overall: 56.48 / 56.62 - **SANDY (OCR):** - Data: 37.82 / 37.78 - Overall: 45.77 / 45.81 ### Chart-specific subsets (Table 4) - Chart-specific answers are near-impossible for fixed-vocabulary baselines (near 0), while SANDY (Oracle) is ~52--53%. - The paper attributes MOM's gap vs SANDY partly to small string generation errors and dependence on precise localization; edit-distance scoring (±1) partially recovers these. * * * ## Hardware / Production - The paper specifies architecture choices and image resolution/features, but does **not** provide concrete training hardware (GPU model/count), wall-clock training time, or throughput/latency numbers in the provided excerpt.