# OneChart: Purify the Chart Structural Extraction via One Auxiliary Token > Chart-to-Python-dict extraction model using an auxiliary token and decoder for improved numeric reliability, with self-consistency scoring for output filtering; introduces ChartY benchmark - Authors: Hunter Heidenreich - Published: 2024-04-01 - Updated: 2026-03-04 - Project: OneChart Project Page - Code: OneChart Code - Model: OneChart Model - Source: Bevaya Labs, --- ## TL;DR OneChart is a chart-to-**Python-dict** structural extraction model that adds a single **auxiliary token** (``) plus a small **auxiliary number decoder** trained with an $L_1$ loss to improve numeric reliability. It also uses a **self-consistency distance** between "numbers implied by the text dict" and "numbers predicted by the auxiliary decoder" as an optional reliability score to filter ("purify") outputs at inference. Authors report strong structural-extraction AP across several chart benchmarks with a relatively small model ($\\approx$200M params) and gains when feeding the extracted dict into downstream ChartQA systems. * * * ## What kind of paper is this? **Dominant:** $\\Psi\_{\\text{Method}}$ (new architecture element: auxiliary token + decoder + training/inference recipe; includes ablations and SOTA-style tables). **Secondary:** - $\\Psi\_{\\text{Resource}}$ (introduces **ChartY** benchmark; also describes large synthetic data engine). - $\\Psi\_{\\text{Evaluation}}$ (self-consistency reliability scoring and "purified output" evaluation protocol; additional comparisons/ablations). ## What is the motivation? Chart parsing is hard because charts vary widely in **style, values, text, and layouts**, and even large VLMs can struggle, especially when charts **lack explicit numeric annotations**. Authors attribute failures mainly to: - **"CLIP bias"**: CLIP-ViT encoders are trained on natural-image captioning and may miss chart-local details; English-heavy pretraining can also hurt non-English charts. - **Cross-entropy numerics problem**: token-level CE does not strongly distinguish "close-looking" numbers (example: "7008" vs "70.8" can have deceptively similar loss), which can slow convergence and reduce numeric accuracy. Public benchmarks are described as limited in **type/style/language diversity** (e.g., ChartQA and PlotQA skew heavily to bar/line; many synthetic sets have limited styling). ## What is the novelty? **Auxiliary token + auxiliary number decoder:** - Prefix a special token `` at the **start** of the output sequence; its hidden state embedding is routed to an auxiliary MLP decoder trained to predict a fixed-length vector of normalized chart numbers using **masked $L_1$ loss**. - Because the main LM is causal, later text tokens can attend to the `` token embedding, aiming to improve numeric fidelity in the generated Python-dict. **Self-evaluation / "purification":** - At inference, parse the raw Python-dict output, extract numbers, normalize them, and compare to the auxiliary decoder's numeric vector via a mean absolute distance **$S \\in [0,1]$**; filter outputs by a threshold (example $\\delta=0.1$) to keep only "reliable" predictions and report improved AP on the retained subset. **Data engine + benchmark:** - Large synthetic chart generation (Matplotlib + Pyecharts) with style randomization and bilingual (English/Chinese) content; and the **ChartY** chart-to-dict benchmark (reported as $\\sim$6K charts in the intro, bilingual and stylistically broader). ## What experiments were performed? - **Structural Extraction (SE)** evaluation on multiple sources (ChartQA-SE, PlotQA-SE, ChartX-SE, ChartY-en/zh), using **SCRM mean AP** under strict/slight/high tolerances; plus textual OCR accuracy using **Reverse Edit distance (RE)** for fields like title/source/axes. - **Comparisons** to chart parsing baselines (e.g., UniChart, DePlot, ChartVLM, ChartAst), emphasizing performance when charts **do not** have explicit numeric labels. - **Ablations**: auxiliary token presence and **position** (front vs behind), and training strategy (Stage2 vs Stage3 vs both). - **Downstream QA**: combine OneChart's extracted dict with LLM/VLM QA systems on ChartQA; report accuracy changes when giving models chart image, dict, or both. ## What are the outcomes/limitations? **Outcomes (as reported):** - The authors report OneChart at 0.2B parameters achieves competitive or leading AP across multiple SE benchmarks in their evaluation (Table 2). - Filtering by self-consistency distance threshold $\delta=0.1$ is reported to increase AP on the retained subset (example: ChartQA-SE AP@strict 72.02 $\rightarrow$ 81.97), though this comes at the cost of reduced coverage. - Token position matters: placing `` at the front performs better than at the end, consistent with causal attention (Table 4). - The authors report QA accuracy improvements when feeding the extracted dict into LLaVA variants (Table 6). **Limitations / open questions (based on what's in the paper):** **Dataset and generalization scope:** - Synthetic data generator focuses primarily on bar/line ("barline") and pie charts, with explicit constraints (e.g., barline charts "up to three legends"). Coverage of other chart types (scatter, area, stacked variants, multi-panel figures) is limited. - ChartY benchmark is reported as $\sim$6K charts, but exact split sizes and per-type distributions are not provided. - Generalization to chart styles outside the synthetic generation distribution is not systematically evaluated. **Architectural constraints:** - The auxiliary number decoder outputs a fixed-length 256 vector with padding. Behavior when charts contain many series or data points (e.g., dense scatter plots, time series with hundreds of points) is unclear. - The paper does not analyze failure modes when the fixed-length representation is insufficient. **Evaluation methodology concerns:** - "Purified" results are reported on filtered subsets with reduced coverage. Full risk-coverage curves are not presented; only snapshots at specific thresholds ($\delta=0.1$). - This shifts evaluation from "overall accuracy" to "accuracy at coverage," but the trade-off is not fully characterized. - The self-consistency distance may be biased toward certain chart types or value ranges; this is not analyzed. **Reproducibility gaps:** - The paper reports three training stages but does not provide full hardware specifications, wall-clock training time, or GPU hours. - Stage-specific data mixing ratios and sampling strategies are not detailed. - License complexity: while code is Apache-2.0, the model inherits constraints from Vary/OPT upstream dependencies, making the effective license "research use only" despite the stated Apache-2.0 for code. * * * ## Model ### Task and output format - Input: a chart image (resized to **1024$\\times$1024**). - Output: a serialized **Python-dict** containing at least: `title`, `source`, `x_axis`, `y_axis`, and `value`. ### Backbone architecture - Built on a VLM-style encoder--decoder; authors choose **Vary-tiny**: a vision encoder "from SAM-base" paired with an autoregressive **OPT-125M** decoder, connected via a linear projection for channel alignment. - Vision features occupy **256 tokens** in the conversation template. ### Auxiliary token and decoder - Add special token `` at the **start** of the token sequence (Figure 3 pipeline depiction). - Let the `` hidden embedding be $t \\in \\mathbb{R}^{768}$. Feed $t$ into an auxiliary decoder $F$ described as a **3-layer MLP** with **2 ReLU** activations. - Auxiliary output: $F(t) \\in \\mathbb{R}^{256}$ representing **min--max normalized numeric values** for the chart. * * * ## Data ### Synthetic generation ("data engine") - Uses both **Matplotlib** and **Pyecharts** to render charts of varied styles/types (Figure 2 summarizes the pipeline). - Adds a **"chart source"** field (beyond typical title/axes/body) to better match real-world charts; includes a two-stage rendering variant that stitches title/source after rendering the main body to increase layout diversity. - Style randomization: random **16-bit color codes** for text/graphics (beyond standard palettes), "hundreds" of fonts, variability in size/direction/quantity of visual elements. - Scale: about **10M** synthetic chart images with labels for pretraining. - Chart types emphasized: - **Barline** charts: single/multi column, single/multi line, combo; balanced between with/without numeric labels; up to **3 legends**. - **Pie** charts: labeled pies and legend-based pies in equal proportion. - Text/content generation: random corpora for pretraining; GPT-3.5 prompts for more "logical/practical" themed data across domains (finance/education/technology/etc.). ### Fine-tuning (SFT) data - Total SFT data reported: **2.7M** samples (Table 1). - Mix includes ChartQA (real), PlotQA (real), and multiple GPT-3.5-driven synthetic sets in English and Chinese rendered via Matplotlib/Pyecharts. ### ChartY benchmark - Bilingual (English/Chinese) chart-to-dict benchmark with broader style/type diversity than prior benchmarks. - Reported as $\\sim$6K charts total (split between ChartY-en and ChartY-zh). - Available via project page and GitHub repository. * * * ## Algorithms / Training ### Conversation template and tokens Uses Vicuna v1-style prompt template: USER: [image] Convert the key information of the chart to a Python-dict. ASSISTANT: [texts output] Adds ``, ``, `` to the OPT tokenizer as special tokens. ### Objectives **Text generation loss:** standard causal LM cross-entropy: $$L\_{\\text{text}}(\\theta, w) = -\\mathbb{E}\_{(w,v)\\sim D}\\log P_\\theta(w_m \\mid w_{<m}, v)$$ **Auxiliary numeric loss:** masked $L_1$ on non-padded entries: $$L_{\\text{num}}(\\theta, u) = \\mathbb{E}_{(u,t)\\sim D}\\left|F(t) - u\\right|\_{\\text{masked}}$$ with ground-truth values min--max normalized and padded with "nan" to length 256. ### Three-stage training recipe (with reported compute) **Stage 1 (Pretraining):** - **10M** synthetic charts - Batch size 16; lr $10^{-4}$; 3 epochs - Trains vision encoder + language model - Loss: $L\_{\\text{text}}$ - Compute: **32 A100 (80G)** for $\\sim$12 hours **Stage 2 (Warmup aux decoder):** - **2.7M** SFT samples - Freeze vision encoder; train LM + auxiliary decoder - Batch size 16; lr $5\\times10^{-5}$; 1 epoch - Loss: $L_{\\text{text}} + L_{\\text{num}}$ - Compute: **16 A100 (80G)** for $\\sim$3 hours **Stage 3 (SFT all params):** - Same SFT pool - Train all parameters - Batch size 16; lr $5\\times10^{-5}$; 1 epoch - Loss: $L_{\\text{text}} + L_{\\text{num}}$ - Compute: **24 A100 (80G)** for $\\sim$4 hours * * * ## Evaluation ### Metrics **Textual OCR** for `title`, `source`, `x_axis`, `y_axis`: uses normalized edit distance, reported as **Reverse Edit distance (RE) = 1 - normalized edit distance** (higher is better). **Structural extraction** for `value` dict: compute tuples of (key, item) pairs and evaluate via **SCRM** mean Average Precision with tolerances: - strict: $J_{thr}=0$, $e_{thr}=0$ - slight: $J_{thr}=2$, $e_{thr}=0.05$ - high: $J_{thr}=5$, $e_{thr}=0.1$ ### Benchmarks described - **ChartQA-SE / PlotQA-SE**: derived from ChartQA and PlotQA test sets; PlotQA-SE emphasizes charts without explicit numeric annotations. - **ChartX-SE**: Matplotlib-rendered bar/line/pie variants, with and without numeric labels. - **ChartY-en / ChartY-zh**: authors' added benchmark (Pyecharts, bilingual, partial numeric annotations). ### Key reported results (selected) **Table 2 (SE AP, OCR RE):** OneChart ("Ours", 0.2B) reports higher AP than compared baselines across many settings, with notable gains on no-numeric-annotation scenarios (authors highlight this in the text). **Table 3 (raw vs purified):** with threshold $\\delta=0.1$, AP@strict increases after filtering (example: ChartQA-SE 72.02 $\\rightarrow$ 81.97) while the number of evaluated samples decreases. **Table 4 (token position):** `` at sequence **front** outperforms "behind" and "no token" for ChartQA-SE and PlotQA-SE AP under all tolerances. **Table 6 (ChartQA QA):** authors report that providing the extracted dict can substantially improve LLaVA1.5/1.6 accuracy in their setting (e.g., LLaVA1.5 avg 17.5 $\\rightarrow$ 50.1 when given figure + dict; improvements are also shown for other configurations). * * * ## Hardware / Production - Inference preprocessing: resize to **1024$\\times$1024**, scale pixels to **[0,1]**, embed to $v \\in \\mathbb{R}^{256\\times768}$, and decode until `` token. - Reliability scoring (optional): parse the text dict (authors mention using `json.loads()`), extract numbers, min--max normalize to $u_r$, compare to auxiliary decoder output $u_c$ via: $$S=\\frac{1}{N}\\sum_{i=1}^{N}|u_{r_i}-u_{c_i}|$$ and threshold $S$ to decide whether to "trust"/retain outputs. - Efficiency note (authors' claim): model size is $\\approx$200M parameters and they report token-time comparisons (example: 1.3 ms vs 5.7 ms for "one token" compared to a much larger baseline), but the exact measurement setup is not fully detailed in the excerpted text.