# NovaChart: A Large-scale Dataset towards Chart Understanding and Generation of Multimodal Large Language Models

> Large-scale chart instruction-tuning dataset with 47K high-resolution charts, 856K instruction-response pairs spanning 18 chart types and 15 tasks, backed by rich metadata including data points, visual elements, source tables, and rendering code

- Authors: Hunter Heidenreich
- Published: 2024-10-01
- Updated: 2026-03-04
- Code: NovaChart Code <https://github.com/Elucidator-V/NovaChart>
- Dataset: NovaChart Dataset <https://huggingface.co/datasets/ympan/novachart>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

NovaChart is a chart-focused instruction-tuning dataset designed to improve MLLMs on both chart understanding and chart generation. The authors report 47K high-resolution chart images and 856K instruction-response pairs, spanning 18 chart types and 15 tasks, backed by per-chart metadata including data points, visual elements, source tables, and rendering code.

* * *

## What kind of paper is this?

**Dominant:** $\\Psi\_{\\text{Resource}}$

-   Headline contribution is a **large-scale dataset + tooling** for chart instruction tuning, including metadata designed for scalability.

**Secondary:** $\\Psi_{\\text{Method}}$ (data engine/pipeline) + $\\Psi_{\\text{Evaluation}}$ (task suite + metrics + model comparisons)

**Rough superposition:**

-   $\\approx 0.6\\,\\Psi_{\\text{Resource}} + 0.25\\,\\Psi_{\\text{Method}} + 0.15\\,\\Psi\_{\\text{Evaluation}}$

* * *

## What is the motivation?

The authors identify three claimed limitations in existing chart datasets used for training chart-capable MLLMs:

**Chart type coverage is narrow and imbalanced**

-   Many datasets focus on bar/line/pie; long-tail types (histograms, radar, word clouds, etc.) are underrepresented.

**Task diversity is restricted**

-   Prior work often centers on extraction/QA/summarization; practical tasks like conditional extraction, visual element recognition, and chart-type conversion are missing or rare.

**Scalability is limited by sparse annotations**

-   Many datasets provide only data points (sometimes colors), but not **source tables** and **visualization code**, which blocks easy task/instruction expansion via LLMs.

* * *

## What is the novelty?

### Dataset scale + coverage

**47K chart images**, **856K instruction-response pairs**

**18 chart types** (explicitly enumerated in the paper's overview figures/tables):

-   Table (as a "chart type"), single/multi-class line plot, single/multi-class scatter plot, single/multi-class bar plot, univariate/bivariate histogram, correlation heatmap, pie, ring, rose, radar, box, sankey, knowledge graph, word cloud.

**15 tasks**, grouped into 4 capability buckets:

-   **Chart Data Understanding:** data identification, data comparison, conditional data extraction, data referring
-   **Chart Visual Understanding:** color recognition, style detection, chart classification, visual elements identification/retrieval, text extraction
-   **Chart Summarization and Analysis:** chart pattern recognition, chart analysis, chart summarization
-   **Chart Generation:** chart blueprint, chart type conversion, table-to-chart generation

### "Scalable metadata" design

For each chart, NovaChart includes **four kinds of annotations**:

1.  **Data points** (numeric/statistical values shown)
2.  **Visual elements** (e.g., colors, style choices)
3.  **Source data** (the originating sub-table)
4.  **Visualization code** (Python rendering code; exception noted for knowledge graphs where code is unavailable)

### Data generation engine (pipeline novelty)

They emphasize an end-to-end engine that produces: raw tables → curated subtables + stats → styled chart images + code → instruction-response pairs.

* * *

## What experiments were performed?

### Dataset comparisons

Compares NovaChart against ChartQA, PlotQA, Chart-to-text, SimChart9K, UniChart, MMC, ChartLlama.

**Key table claims:**

-   NovaChart: **18 types, 47K images, 856K instruction pairs, 15 tasks**, covering understanding + generation.
-   Metadata comparison table shows NovaChart is the only listed dataset providing **data points + visual elements + source data + visualization code** (others lack source data/code).

### Fine-tuning + evaluation

Fine-tune 3 open MLLMs:

-   **LLaVA-v1.5**, **InternLM-XComposer**, **Qwen-VL-Chat**

Evaluate on an **independent evaluation set** spanning all 15 tasks.

**Metrics (by task type):**

-   Exact Match (EM) for classification/QA
-   RNSS for numerical results in data-referring tasks
-   Levenshtein distance + SCRM for multi-point extraction
-   GPT-Score for open-ended summarization/analysis and chart generation tasks

### Reported outcomes

Across tasks, fine-tuning yields **large relative improvements** reported as **35.47%--619.47%** (range across tasks/models).

**Qualitative examples show improvements in:**

-   Correctly interpreting axes/distributions for analysis
-   Producing **executable** chart conversion code (baseline sometimes outputs "not directly executable").

* * *

## What are the outcomes/limitations?

### Outcomes the paper emphasizes

-   The authors report gains across all 15 tasks after tuning, including generation tasks. Relative improvement ranges are reported as 35.47% to 619.47%, though baseline performance levels vary significantly by task.
-   Chart classification and text extraction reportedly reach "excellent" performance in their evaluation setup.
-   Improvements are claimed across chart types beyond common bar/line/pie charts, attributed to broader type coverage in the training data. Per-type breakdowns are not provided.

### Limitations / open questions

**Dataset design and coverage:**

-   Chart type distribution is not reported; balance across the 18 claimed types is unclear, which may affect model generalization.
-   Knowledge graph charts explicitly lack visualization code, breaking the "full metadata" claim for at least one chart type.
-   Source data for the 47K charts is not specified; provenance, diversity, and potential biases are unclear.

**Evaluation concerns:**

-   Open-ended task evaluation relies on GPT-Score (LLM judge), introducing model-judge dependence and sensitivity to prompt formulation. No inter-rater reliability or correlation with human judgment is reported.
-   The paper acknowledges improvements are "limited" for certain analysis and generation tasks, suggesting the dataset may not sufficiently address those capabilities.
-   Accuracy drop magnitude when moving from extracted tables to predicted tables is not quantified for the fine-tuned models.

**Reproducibility gaps:**

-   Critical hyperparameters (fine-tuning learning rates, batch sizes, epochs) are deferred to appendices not included in the provided excerpt.
-   Chart-type-specific attribute rules and full metric definitions are also appendix-only.
-   Training compute (GPU hours, hardware specs) is not reported.

* * *

## Reproducibility Details

### Model

NovaChart itself is a dataset, but the paper fine-tunes 3 MLLMs: **LLaVA-v1.5**, **InternLM-XComposer**, **Qwen-VL-Chat**. Hyperparameters and fine-tuning details are stated as being in Appendix 2 (not included in the visible content).

### Data

**Dataset size and structure:**

-   **47K** chart images (high-resolution)
-   **856K** instruction-response pairs
-   Built from:
    -   **1.3K** raw tables (post-filtering)
    -   **28K** curated sub-tables ("source data") used to derive chart statistics

**Chart types (18):**

From the overview and distribution figure, the chart types include:

-   Table
-   Single-class and multi-class: line plot, scatter plot, bar plot
-   Univariate and bivariate histogram
-   Correlation heatmap
-   Pie, ring, rose, radar
-   Box plot
-   Sankey
-   Knowledge graph
-   Word cloud

**Tasks (15):**

Grouped as:

-   **Chart Data Understanding:** data identification; data comparison; conditional extraction; data referring
-   **Chart Visual Understanding:** color recognition; style detection; chart classification; visual elements identification/retrieval; text extraction
-   **Chart Summarization and Analysis:** pattern recognition; analysis; summarization
-   **Chart Generation:** blueprint; chart type conversion; table-to-chart generation

**Metadata fields (per chart):**

The paper's "chart metadata" is explicitly:

-   Data points
-   Visual elements
-   Source data (sub-table)
-   Visualization code (rendering code)

### Algorithms / Training (Data Engine)

The paper describes a 4-stage pipeline:

**1. Raw Data Acquisition**

-   Source: Kaggle relational tables with high user votes.
-   Filtering/preprocess:
    -   Remove non-English tables
    -   Remove tables with too few rows (`< 50`)
    -   Remove unnamed columns
    -   Remove columns with too many missing values (`> 90%`)
    -   Remove columns with overly long contents (example: movie reviews)

**2. Data Curation**

-   Goal: sample attributes + rows to produce many sub-tables and then compute chart statistics.
-   Attribute type schema (used for choosing columns):
    -   **Numeric**: numeric dependent variables (e.g., y-axis)
    -   **Unique-Numeric**: numeric, mostly unique (e.g., year), usable as independent variable (x-axis for lines)
    -   **Categorical**: string with `<= 5` unique values (class labels for multi-class charts)
    -   **Enumerable**: `<= 25` unique values (qualitative variable for bar/pie, etc.)
-   Attribute classification model:
    -   Uses **GPT-turbo-3.5** with in-context learning; prompt includes the attribute-type definitions, number of unique values, example values, and an instruction to classify the attribute.
-   Sub-table sampling:
    -   For each chart type: randomly select the required attribute types and sample **30--50 rows** from the raw table
    -   If a raw table cannot yield a sub-table meeting requirements, skip it
    -   Compute chart statistics from the sampled sub-table to derive chart data points

**3. Image Styling and Visualization**

-   Rendering libraries mentioned: **Matplotlib**, **Seaborn**, **Pyecharts**.
-   Visual diversity: randomize visual elements (example: colors, shadow, style) and render multiple images per data-point instance.
-   The paper states this stage yields ~**40K** charts initially, then expanded to **47K** after later extensions.

**4. Instruction Formulation**

-   Task design: 15 tasks, including "new" ones the authors emphasize (conditional extraction; some visual element checks like fitting-curve existence in histograms; chart-to-chart conversions).
-   LLM usage:
    -   For tasks with answers obtainable from metadata: use **GPT-4** to generate diverse instruction templates and convert to instruction-following format.
    -   For more open-ended tasks (summarization/analysis): use **GPT-turbo-3.5**, providing prompt with task instruction, in-context demos, and the relevant chart metadata; the model output becomes the response.

**Data expansion steps:**

-   Add line-chart source data from **Statista** due to shortage of suitable line-chart subtables from Kaggle.
-   Add **HTML/CSS-rendered tables** as an additional "chart" type.
-   Manually collect some chart types (explicitly mentioned: knowledge graphs and word clouds); knowledge-graph visualization code unavailable.

### Evaluation

**Metrics and protocols:**

-   EM for classification/QA tasks, RNSS for numeric data-referring, Levenshtein + SCRM for multi-point extraction, GPT-Score for open-ended and generation tasks.
-   Comparisons shown in figures:
    -   Per-task spider/radar charts before vs after tuning for the three models
    -   Per-chart-type performance plots on selected tasks (example shown for InternLM-XComposer).

**Reported improvement magnitude:**

-   Across tasks: **35.47%--619.47%** relative improvements after fine-tuning (paper-reported range).

### Hardware / Production

The main text does not list GPU types, wall-clock training time, or serving throughput; it points to appendices for fine-tuning details.

* * *

## Data Availability

### Is all the data publicly shared?

They **claim the dataset is publicly available**, and they point to a public GitHub repo as the distribution entry point.

That said, "**all the data**" depends on what you mean:

**Released (public):**

-   **GitHub repo** (code + toolkit; also points to where to download the dataset): [Elucidator-V/NovaChart](https://github.com/Elucidator-V/NovaChart)
-   **Hugging Face dataset** linked from the GitHub repo as the download location for the "full NovaChart dataset": [ympan/novachart](https://huggingface.co/datasets/ympan/novachart)

**Potentially not fully released / ambiguous:**

-   The paper emphasizes rich **chart metadata** (data points, visual elements, source data, visualization code).
-   But a [public GitHub issue](https://github.com/Elucidator-V/NovaChart/issues/2) reports that only the **instruction-tuning JSONL files** were found and asks whether metadata will be released, suggesting at least some parts may be missing or not obvious in the current release.

**Underlying raw sources are likely not fully redistributable as-is:**

-   Their pipeline uses **Kaggle tables** and additionally **Statista tables** for line charts.
-   The paper doesn't specify whether the _original_ Kaggle/Statista tables are redistributed (and those sources typically have their own licenses/ToS).

### Where is it shared?

-   **GitHub:** [Elucidator-V/NovaChart](https://github.com/Elucidator-V/NovaChart) (paper and repo both point here)
-   **Hugging Face (dataset download):** [ympan/novachart](https://huggingface.co/datasets/ympan/novachart) --- contains a ~10 GB `novachartv1.rar` file

### What license?

-   **GitHub repository license:** marked as **MIT** on the repo page (this generally covers the _code and repo contents_, unless the repo says otherwise)
-   **Hugging Face dataset license:** listed as **Apache-2.0** on the dataset page/README metadata
-   **Paper itself:** does not appear to state a dataset license in the main text; it only states availability and links