# CHART-Infographics: Notes > ICDAR 2019 competition on harvesting raw tables from infographics with synthetic training and real test sets - Authors: Hunter Heidenreich - Published: 2019-09-01 - Updated: 2026-03-04 - Paper: CHART-Infographics Paper - Dataset: CHART2019-S (Synthetic) - Dataset: ICDAR-CHART-2019-PMC - Source: Bevaya Labs, --- ## TL;DR This paper reports the setup and results of the first CHART-Infographics competition: a task decomposition for chart understanding plus a large synthetic training set and a smaller, manually annotated real-chart test set from PubMedCentral. Main takeaway: systems do very well on synthetic data but drop substantially on real charts; the benchmark, annotation tooling, and evaluation scripts are meant to standardize comparisons. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Resource}}$ - Primary "artifact" is a benchmark ecosystem: synthetic dataset generation, real-chart dataset sampling/annotation tooling, and released evaluation scripts. - **Secondary:** $\\Psi\_{\\text{Evaluation}}$ - Defines task-specific metrics (notably for text detection+OCR scoring, axis tick localization scoring, legend marker localization scoring) and analyzes cross-domain generalization (synthetic to real). * * * ## What is the motivation? - Chart understanding has many proposed methods, but the field lacked **common benchmarks and tools** to compare approaches fairly. - The authors aim to (1) provide a **large-scale synthetic training regime** to enable deep models and (2) test robustness on **real charts** from scientific literature, where layout and rendering variability is higher. * * * ## What is the novelty? - **Task decomposition** of "extract raw tables from charts" into a pipeline of independently evaluable sub-tasks (Tasks 1--7). - **Synthetic dataset generation** from real-world data tables, rendered into 10 chart types with randomized stylistic/layout variations, and annotated automatically via Matplotlib API. - **New evaluation formulations** for chart-specific subtasks: - Text detection+recognition: combines IoU-based matching with OCR string similarity via normalized character error rate (NCER). - Axis analysis: weighted F-measure with partial credit based on tick-location accuracy as a function of distance relative to image diagonal. - Legend analysis: weighted F-measure with partial credit based on legend-marker bounding box IoU. * * * ## What experiments were performed? - Competition evaluation on **two domains**: - **Synthetic charts** (large-scale; used for training and testing). - **PMC (real charts)** from PubMedCentral Open Access, manually annotated for evaluation (with different annotation depths per task). - Tasks actually receiving submissions: **Tasks 1--5** (no submissions for Tasks 6--7). - Reported results include: - Task 1 confusion matrices for top methods on PMC (Figure 4). - Task-level summary tables: - Task 1: average F-measure on synthetic vs PMC (Table II). - Task 2: text detection/recognition scores (Table III). - Tasks 3--5: average/weighted F-measures (Table IV). * * * ## What are the outcomes/limitations? **Outcomes** - Strong synthetic performance across several tasks, but **substantial degradation on real charts**. - Example: Task 1 (chart classification) top system is near-perfect on synthetic (99.81) but lower on PMC (88.29). - For PMC, success correlated with **extra data augmentation or external datasets** beyond the provided synthetic set. - PMC text is hard: resolution limits, human-difficult regions, and many special symbols (Greek letters, superscripts/subscripts) hurt OCR. **Limitations (as evidenced by setup + results)** - **Domain gap** is the core difficulty: synthetic rendering does not capture the full diversity of real charts. - Only **5 teams submitted results**, and only for Tasks 1--5; **no participation** in full data extraction (Tasks 6--7), suggesting the end-to-end problem is still very challenging under this formulation. - For PMC axis ticks, the paper notes annotation patterns (minor ticks, separator ticks) that did not exist in synthetic; evaluation required adjusting GT to match the only tick-label pattern represented in synthetic training. * * * ## Model This is a **competition + benchmark** paper, not a single-model paper. The "model" content is participant system summaries. Representative approaches (submitted teams): - **ABC (Fintech team)**: - Task 1: ResNet-101 multi-label classifier. - Task 2: connected components + Faster R-CNN detection; attention modules for multi-orientation OCR. - Task 3: gradient boosting decision tree over engineered features (geometry, alignment patterns, numeric-ness, direction, relative position to axis/legend). - Task 4: axis detection via color + line segments; tick detection via gradients along axes. - Task 5: segmentation (ResNet + FPN) for legend markers + rule filtering; text-marker pairing by proximity. - **A-team**: PixelLink for text detection; Tesseract OCR for recognition; SVM for text-role classification; heuristic "skeleton" for tick localization; connected components for legend markers. - Other teams: ResNet-50 classifier (ANU-Team); small CNN (Boomerang); SVM text-role classification with geometric features (IITB-Team). * * * ## Data ### Synthetic chart dataset - **Rendered with Matplotlib** using tabular data from multiple public sources (World Bank indicators, India open data, UN commodity trade stats, US census, stock/ETF price-volume). - **Chart types (10 total)** shown in Figure 1: - Pie, Donut, Line, Scatter, Vertical Box, Horizontal Box, Vertical Bar (Grouped), Vertical Bar (Stacked), Horizontal Bar (Grouped), Horizontal Bar (Stacked). - Synthetic dataset generation includes explicit randomization/variation: - Title/legend placement, fonts, style (colors/line widths/borders/grids/markers), bar widths, pie radii, optional error bars. - **Annotation** obtained programmatically from Matplotlib API, including tight bounding boxes for text, axes, legends, and plot elements (bars/lines/pies). - **Counts** (Table I; synthetic): Train = 198,010; Test = 4,540, with per-type distributions (e.g., Line train 41,874; Scatter train 41,703). ### PubMedCentral (PMC) real chart dataset - Source: PubMedCentral Open Access (paper mentions $>1.8$ million papers). - Sampling strategy: - Select journals likely to contain charts (epidemiology, public health, pathology, genetics), restrict to journals with $>500$ publications, papers after year 2000. - Cluster extracted figures by visual similarity to separate chart candidates from non-charts; manually annotate images within chart-heavy clusters. - Final sample: **4,242 single-panel figures** across chart types (Table I "PMC Test" column shows per-type counts; e.g., Line 2,257; Scatter 532). - Annotation depth: - Entire 4,242 used for **Task 1** evaluation. - Two disjoint sub-samples fully annotated for **Task 2** and **Tasks 3--5** evaluation. - Annotation tooling and workflow (Figure 2, plus text): - Annotate figure type, panel type, chart type/orientation. - Annotate text regions (location, role, transcription). - Used **Tesseract OCR** for initial transcription, then manually corrected errors; special symbols recorded as **LaTeX strings**. - Annotate legends, then axes (axis type categorical vs numerical; first-quadrant bbox; tick locations; axis titles/labels; links between ticks and tick labels). - Annotate plot elements by chart type (bars/lines/boxes/scatter marks). - Tools are described as open-source and available post-competition. * * * ## Algorithms / Training This paper does not define a single training recipe, but it does define a **task pipeline** and the evaluation condition: - Overall chart extraction is treated as a **pipeline** where upstream outputs feed downstream tasks (Figure 3). - For evaluation of each task, participants were provided the **ideal outputs of previous tasks**, isolating errors per module rather than compounding pipeline failures. * * * ## Evaluation ### Task definitions (as used in the competition) - **Task 1: Chart Classification** - Classes: horizontal bar, vertical bar, horizontal box, vertical box, line, scatter, pie, donut (pie/donut only used in Task 1). - Metric: average per-class F-measure. - **Task 2: Text Detection and Recognition** (logical element level, not word level) - Inputs: chart image + correct chart class. - Detection match: IoU threshold = 0.5; many-to-one and one-to-many resolved by highest overlaps; others count as FP/FN. - Recognition score per matched pair: $\\max(1-\\text{NCER}, 0)$ where NCER is normalized edit distance (normalized by GT string length). - Aggregation: - Detection scores averaged per image by $\\max(#GT,#Pred)$. - Recognition scores averaged per image by $#GT$. - Final metric: harmonic mean of detection and recognition, averaged across test set. - **Task 3: Text Role Classification** - Inputs: text boxes + transcripts for all logical elements. - Classes: chart title, axis title, tick label, legend title, legend label. - Metric: average per-class F-measure. - **Task 4: Axis Analysis** - Output: for both x and y axes, list of tick-label text elements paired with a pixel point $(x,y)$ for tick location. - Metric: weighted F-measure with partial credit based on distance to GT tick location. - Scoring uses thresholds $a=1.0\\%$ and $b=2.0\\%$ of image diagonal. For prediction distance $d$: $$ s(d,a,b)= \\begin{cases} 1 & d \\le a \\ \\frac{b-d}{b-a} & a \\le d \\le b \\ 0 & d \\ge b \\end{cases} $$ Recall = sum(scores) / (# GT ticks); Precision = sum(scores) / (# predicted ticks). - **Task 5: Legend Analysis** - Output: list of text elements paired with bounding boxes around legend markers. - Metric: weighted F-measure, where partial TP credit is determined by marker BB IoU (text element must match, then IoU controls partial score). - **Tasks 6--7** (data extraction and end-to-end) were specified but received no submissions; the paper only briefly outlines them. ### Reported results (key numbers) - **Task 1 (avg F-measure)** (Table II): - Synthetic: ABC 99.81; A-Team 94.82; ANU-Team 89.78; Boomerang 9.59. - PMC: ABC 88.29; A-Team 77.52; ANU-Team 35.96; Boomerang 12.06. - **Task 2 (Synthetic + PMC)** (Table III): - Synthetic: ABC IoU 69.92, OCR 94.97, F 80.54; A-Team IoU 70.96, OCR 78.97, F 74.75. - PMC: A-Team IoU 48.48, OCR 58.81, F 53.15. - **Tasks 3--5 (Synthetic)** (Table IV): - Task 3 avg F: ABC 100.00; A-Team 99.95; IITB-Team 60.25. - Task 4 weighted F: ABC 96.49; A-Team 99.76. - Task 5 weighted F: ABC 78.14; A-Team 87.13. - **Tasks 3--4 (PMC)** (Table IV): - Task 3 avg F: A-Team 84.38; IITB-Team 35.58. - Task 4 weighted F: A-Team 77.33. ### Error patterns called out in analysis - Chart classification confusions: - Grouped vs stacked bars (synthetic + PMC). - On PMC: box plots misclassified as line/scatter; line vs scatter difficult (scatter may include fitted lines not representing raw data). - PMC text challenges: - Low/variable resolution; hard-to-read text; special symbols. - Legend analysis: - Since Task 3 is near-perfect on synthetic, the remaining gap in Task 5 is attributed mainly to marker BB localization quality; ABC has higher variance in IoU (more very good and more zero-IoU cases), A-Team is more consistently "good." * * * ## Hardware / Production - No concrete training hardware, runtime, or throughput specs are reported (expected for a competition summary rather than a single-system paper).