# ChartOCR: Notes > Deep + rules hybrid pipeline for extracting numeric tables from bar, line, and pie chart images - Authors: Hunter Heidenreich - Published: 2021-01-01 - Updated: 2026-03-04 - Paper: ChartOCR Paper - Code: DeepRule Code - Dataset: DeepRule Dataset (ExcelChart400K) - Source: Bevaya Labs, --- ## TL;DR ChartOCR is a **deep + rules hybrid** pipeline for extracting the underlying numeric table from **bar, line, and pie** chart images by (1) detecting **chart keypoints** with a shared deep model, then (2) applying **type-specific geometric rules** to reconstruct chart elements and map pixels to values. The authors report strong results on their large ExcelChart400K dataset (scores: **0.919 bar**, **0.918 pie**, **0.962 line**) and show that with ground-truth keypoints the remaining rule module becomes near-perfect, suggesting keypoint localization is the dominant error source. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Method}}$. The core contribution is a **unified extraction method** that uses keypoint detection plus chart-type rules to support multiple chart types in one framework. - **Secondary:** $\\Psi_{\\text{Resource}}$ (they introduce **ExcelChart400K**, a large annotated dataset) and $\\Psi_{\\text{Evaluation}}$ (they propose **new chart-type-specific metrics**). ## What is the motivation? - Prior **rule-based** chart extractors struggle to generalize across chart styles and layouts, since rules are brittle and style-specific. - Pure **end-to-end deep** approaches can be accurate but often **specialize to a single chart type** and provide less controllable intermediate structure (plot area, range, components). - The paper targets a middle ground: use deep learning for what generalizes (keypoint detection) and rules for what is structured (geometry and grouping). ## What is the novelty? - **Key idea:** reduce chart extraction across types to a **keypoint detection problem**, then reconstruct components with **type-specific post-processing rules**. - A **shared "common information extraction" network** predicts (a) keypoints and (b) chart type; then downstream modules handle range estimation and object grouping per type. - **New dataset:** ExcelChart400K (386,966 images) for training deep models on diverse chart styles. - **New metrics:** separate evaluation metrics for bar, line, and pie that better match the "read the data values" objective than borrowed detection metrics. ## What experiments were performed? - Datasets: - **FQA** (100 synthetic chart images across bar/pie/line) and **WebData** (100 web-crawled images) from prior work. - **ExcelChart400K** (large-scale dataset introduced here). - Baselines compared (as reported): - Rule-based: **Revision**. - Deep: **Vis**, **ResNet+Faster-RCNN** (bar), **RotationRNN** (pie), **ResNet+RNN** (end-to-end). - Commercial: **think-cell** (bar only, qualitative). - Ablations: - Replace predicted keypoints with **ground-truth keypoints** to estimate an upper bound for the rule module. - Replace OCR with **ground-truth OCR** for FQA (since WebData lacks GT OCR). ## What are the outcomes/limitations? - Main outcomes (authors' reported): - ExcelChart400K scores: **0.919 (bar), 0.918 (pie), 0.962 (line)**; with GT keypoints: **0.989/0.996/0.991**, implying post-processing is strong when keypoints are accurate. - Public datasets (mean error): ChartOCR improves substantially over Vis, especially on pie and line. - Runtime: ChartOCR averages **0.206s (bar), 0.193s (pie), 0.507s (line)**; rule-based Revision is much slower. - Limitations called out: - **Line charts:** hard cases with multiple entangled line segments remain challenging; the QUERY network struggles in these settings. - Practical assumptions (implicit in method description, potential fragility): - Data range extraction assumes y-axis labels are on the **left of the plot area** and uses only top/bottom OCR numbers to infer scale. This may break for unconventional layouts, multi-axis charts, or non-linear scales (not addressed in the paper text). * * * ## Model ### Common information extraction: keypoints + chart type - Backbone: **modified CornerNet** with **Hourglass Net** backbone (104 layers). - Output: a **pixel-level probability map** with **3 channels** (top-left, bottom-right, background). Map size matches input image. - Uses **corner pooling** (from CornerNet) on the penultimate keypoint branch layer to expand receptive field along horizontal/vertical directions. - Loss: "CornerNet-style" combination of probability-map loss plus **smooth L1** for keypoint coordinates (the paper references CornerNet's settings rather than fully re-deriving them). ### Chart type classification head - Adds a conv layer from Hourglass output to downsample features (example given: $(32 \\times 32)$), then max-pools to a 1D vector and feeds FC layers to a softmax classifier. - Loss: cross-entropy (explicit formula provided). * * * ## Data ### FQA and WebData - **FQA:** 100 synthetic images total across chart types; limited style diversity per authors. - **WebData:** 100 images crawled from the web; larger style variation than FQA. ### ExcelChart400K - Size: **386,966** chart images, collected by crawling public Excel sheets; chart images captured with Excel APIs and underlying data extracted directly from the source spreadsheets. - Privacy: authors overwrite chart text with random characters for anonymization. - Splits (Table 1): - Bar: train **173,249**, val **6,935**, test **6,970** - Line: train **116,745**, val **3,073**, test **3,072** - Pie: train **73,075**, val **1,924**, test **1,923** * * * ## Algorithms / Training ### Data range extraction (bar + line) - Uses **Microsoft OCR API** to extract text from chart images (legend/title/axis labels). - Locates **plot area** via a similar keypoint detection routine (top-left + bottom-right corners). - Assumption: y-axis numbers are on the **left-hand side** of the plot area; filter OCR results accordingly. - **Algorithm 1 (Data Range Estimation)** (high-level): - Find nearest OCR number near bottom-left of plot area as $r_{\\max}$ and near top-left as $r_{\\min}$ (using a left-of-plot constraint like `r.r < Left - 4`). - Convert text to numbers, compute a pixel-to-value scale: $$Y_{\\text{scale}} = \\frac{r_{\\max}.num - r_{\\min}.num}{r_{\\min}.t - r\_{\\max}.t}$$ - Estimate $Y_{\\min}$ and $Y_{\\max}$ using scale and vertical offsets relative to plot area bounds. - For pie charts, they skip this step since the total is assumed to be **100%** by default. ### Type-specific chart object detection **Bar charts** - Threshold keypoint heatmap at **$s = 0.4$**, then match each top-left point to the nearest bottom-right point using a weighted distance: $$dist = \\gamma , dist_x + \\nu , dist_y$$ - For vertical bars, set $\\gamma > \\nu$ and constrain search to the right side of plot area; for horizontal bars, $\\nu > \\gamma$. **Pie charts** - Modify keypoint net: replace corner pooling with **center pooling** (from CenterNet-style methods) to capture 360-degree context. - Threshold at **$s = 0.3$**. - **Algorithm 2 (Sector Combining)** handles: 1. **Tight** pies (single center): sort arc points clockwise; pair consecutive arc points with the center. 2. **Exploded** pies (multiple centers): run **Pie Radius Estimation** (details in supplement per authors) and only connect center-to-arc pairs whose distances fall within a threshold of the estimated radius. **Line charts** - Adds an **embedding layer** (after an early conv) to encourage points on the same line to have similar embeddings; uses pull/push losses (CornerNet-style associative embedding). - Total keypoint loss for line: $loss'_{\\text{point}} = loss_{\\text{point}} + \\lambda \\cdot loss\_{\\text{embedding}}$, with **$\\lambda = 0.1$**. - Groups points into lines via hierarchical clustering using **union-find**. - Handles points shared by multiple lines ("intersection points") with a **QUERY network**: - For an intersection point $s$ and closest assigned point $e$, sample $K$ points evenly along segment $s \\rightarrow e$; obtain features via **linear interpolation**; classify whether $s$ and $e$ belong to the same line. ### Training setup (as reported) - Optimizer: **Adam**, LR **$2.5 \\times 10^{-4}$**, reduced to **$2.5 \\times 10^{-5}$** for last 5,000 batches. - Batch size: **27**. - Mentions **$\\alpha = 2$, $\\beta = 4$** (consistent with focal-loss-style params, though the paper does not re-explain them in detail here). - Postproc: **Soft-NMS** to merge keypoints. - Early stopping used; validation set from ExcelChart400K used for hyperparameter tuning. * * * ## Evaluation ### Proposed metrics (Section 6) **Bar** - Defines a custom distance between predicted and GT bar boxes $p$ and $g$ emphasizing $(x,y,h)$ (width is treated as less relevant for reading): $$D(p,g) = \\min\\left(1, \\frac{|x_p-x_g|}{w_g} + \\frac{|y_p-y_g|}{h_g} + \\frac{|h_p-h_g|}{h_g}\\right)$$ - Computes assignment cost via a minimum-cost matching (job assignment) over predictions vs GT; final score is $1 - cost/K$. **Line** - Treats a line as continuous and evaluates via interpolation-based error with a precision/recall and an $F1$ formula; weights segments by point intervals (larger gaps count more). - For multi-line charts, they enumerate combinations to find best matching score. **Pie** - Treats extraction as a **sequence matching** problem in clockwise order; uses dynamic programming with a per-element match reward of $1 - \\left|\\frac{x_i-y_j}{y_j}\\right|$, then normalizes. ### Key results (as reported) **ExcelChart400K (Table 2, higher is better)** - ChartOCR: **0.919 (bar), 0.918 (pie), 0.962 (line)** - ChartOCR + GT keypoints: **0.989, 0.996, 0.991** - ResNet+Faster-RCNN: **0.802** (bar) - Revision: **0.582 (bar), 0.838 (pie)** - RotationRNN: **0.797** (pie) - ResNet+RNN: **0.000 (bar), 0.411 (pie), 0.644 (line)** **FQA + WebData (Table 3, mean error, lower is better)** - FQA: ChartOCR **0.185 (bar), 0.038 (pie), 0.484 (line)**; with GT OCR: **0.093, 0.038, 0.496**. - WebData: ChartOCR **0.285 (bar), 0.439 (pie), 0.740 (line)**. - The authors note OCR quality is a major contributor to bar-chart extraction error on these sets. * * * ## Hardware / Production - Training environment: **4 $\\times$ Tesla P100 GPUs**. - Average runtime (Table 4): - ChartOCR: **0.206s (bar), 0.193s (pie), 0.507s (line)** - Revision: **20.032s (bar), 5.423s (pie)** - ResNet+Faster-RCNN: **0.120s (bar)** - RotationRNN: **0.421s (pie)** - Authors suggest merging QUERY with the keypoint backbone if runtime is critical (currently QUERY does not share parameters with keypoint net). * * * ## Data availability + licensing (ExcelChart400K / "CHARTEX Data") - **Publicly available?** Yes --- the paper explicitly states "the code and the dataset are publicly available" and points to the DeepRule GitHub repo. - **Online?** Yes --- the DeepRule repo is public, and its README links to a hosted dataset download ("Downloading CHARTEX Data") on Hugging Face. ([GitHub][1]) - **What license is it under?** - **Repo (code) license:** The DeepRule GitHub repository is labeled **BSD-3-Clause**. ([GitHub][1]) - **Dataset (as hosted) license:** The Hugging Face dataset page for _DeepRuleDataset_ declares the license as **MIT**. ([Hugging Face][2]) - **Important caveat:** The **paper does not state a dataset license** (it only says it's publicly available), and the dataset itself was collected by "crawling public Excel sheets" (with text anonymization). So if you need high confidence for downstream use, rely on the license/terms shipped with the dataset distribution you download (and treat GitHub's BSD-3-Clause as applying to the repo contents unless the dataset package states otherwise). [1]: https://github.com/soap117/DeepRule "GitHub - soap117/DeepRule" [2]: https://huggingface.co/datasets/asbljy/DeepRuleDataset/tree/main "niups/DeepRuleDataset at main"