# Beagle: Notes > Automated pipeline for crawling and classifying SVG-based visualizations using hand-engineered features - Authors: Hunter Heidenreich - Published: 2018-04-01 - Updated: 2026-03-04 - Paper: Beagle Paper - Code: Beagle Annotator - Dataset: Beagle Dataset - Source: Bevaya Labs, --- ## TL;DR Beagle is an automated pipeline for crawling the web to extract **SVG-based** visualizations and classifying them into visualization types using 114 hand-engineered SVG statistics with a Random Forest classifier. The system achieves **82-99% accuracy** when training and testing within single collections, and about **85% accuracy** when mixing collections. The resulting dataset contains **42,000+ visualizations** from five web sources (D3, Plotly, Chartblocks, Fusion Charts, Graphiq), enabling analysis of what chart types appear on the web. --- ## What kind of paper is this? - **Dominant**: $\Psi_{\text{Method}}$. Primary contribution is an end-to-end system and classifier for SVG visualization extraction and automated labeling (Web Crawler + Annotator). - **Secondary**: $\Psi_{\text{Resource}}$. Produces a sizable mined dataset across multiple visualization "islands" (5 sites; 42,000+ visualizations). - **Secondary**: $\Psi_{\text{Evaluation}}$ (light). Provides multi-class, cross-collection evaluation methodology with metrics (weighted/non-weighted accuracy, F1; stratified 5-fold CV repeated 10 times). --- ## What is the motivation? - The authors want to understand **how visualization tools are actually used on the web**, which requires collecting many real examples and labeling them at scale. - A naive, unguided crawl is low-yield and redundant: after crawling approximately 20M pages, they found approximately 10k with visualizations (0.05%), heavily dominated by repeated StackOverflow profile charts. - They pivot to crawling "islands" of visualization usage: centralized sites where SVG-based visualizations are commonly hosted. --- ## What is the novelty? ### System design: crawler + annotator - **Two standalone components**: - **Web Crawler**: extracts SVGs from pages (raw SVG spec + snapshot). - **Annotator**: labels SVGs via an SVG-focused classifier. ### SVG feature design for classification (114 features) - Feature extraction computes **basic statistics over SVG elements** and feeds them into an off-the-shelf classifier. - Total **114 features**, grouped as: **general (6)**, **style (19)**, **per-element (89)**. - The per-element group targets five SVG element types with per-type feature counts: circle (16), rect (20), line (15), path (35), text (3). - Examples of what is measured: - **General**: counts of element types plus counts of horizontal/vertical axis lines to separate "often has axes" charts (bars) from "often no axes" (maps). - **Style**: unique fill/border colors, stroke width min/max, font size min/max, unique font sizes, font variance; also accounts for style inheritance and CSS. - **Per-element geometry/layout**: normalized x/y position stats, shared-position counts, class-name counts; plus element-specific stats like circle radii variance, rect width/height variance, and path "d" string length statistics. --- ## What experiments were performed? ### Data collection and labeling - Targeted crawl mines 5 SVG-heavy "islands": bl.ocks.org (D3), Plotly, Chartblocks, Fusion Charts, Graphiq. - Crawl yields **42,000+ total** SVG-based visualizations; per-island counts reported as: D3 2000+, Plotly 15000, Chartblocks 22000, Fusion Charts 500, Graphiq 2500+. - For analysis/evaluation, they omit about **one quarter** of extracted visualizations due to complex webpage features (example: animations) interfering with extraction. - Labeling approach: - **By hand**: D3 (bl.ocks.org), Fusion Charts, Graphiq. - **By code**: Plotly, Chartblocks (using site structure cues like page titles or organized link lists). - They note inconsistent metadata across sites as a general challenge. ### Classification model choice and evaluation protocol - They tried Multinomial NB, Gaussian NB, Decision Tree, and SVM with default scikit-learn params; no big performance differences among top performers, and chose a tree-based approach for interpretability, then moved to Random Forest to address overfitting. - Final classifier: scikit-learn `RandomForestClassifier` with **`n_estimators=14`**, defaults otherwise. - Evaluation: stratified **5-fold cross-validation**, repeated **10 runs**, reporting weighted accuracy and non-weighted (equal-class-weight) accuracy, plus F1. ### Within-group and between-group tests - They run within-group evaluation per collection and between-group evaluation by mixing collections. - Reported results (Table 2) include: - D3 weighted accuracy 0.8193 (non-weighted 0.7155) - Plotly weighted accuracy 0.9721 (non-weighted 0.9159) - Chartblocks weighted accuracy 0.9955 (non-weighted 0.9955) - Fusion Charts weighted accuracy 0.9258 (non-weighted 0.8753) - Graphiq weighted accuracy 0.9873 (non-weighted 0.9726) - Mixed-collection results: - Mixture weighted accuracy 0.8527 - Mixture (Revision) weighted accuracy 0.7952 (noted as a revision baseline in the table) - They also include a "Mixture + Non-Vis (binary)" line with accuracy/F1 reported as binary classification. --- ## What are the outcomes/limitations? ### Empirical findings about visualization usage - From the broad unguided crawl: SVG visualizations are rare on the open web in their sampling (0.05% of approximately 20M crawled pages). - Across the mined collections, four chart families dominate: **bar**, **line**, **scatter**, **geographic maps**. - Table 3 summarizes usage patterns; examples: - D3: "Map" most popular (30.4%); pie is 0.6% - Plotly: "Scatter" most popular (46.6%); pie is 0.4% - Chartblocks: only 4 types, with pie at 23.2% but still below line (34.7%) and bar (31.8%). - They interpret the distribution as suggestive of a flexibility vs. ease-of-use tradeoff, and note that line/bar charts are consistently in the top 3 across collections. ### Limitations acknowledged by the paper - **Format limitation**: results are about **SVG-based** (often interactive) web visualizations and may not generalize to raster images, Excel outputs, etc. - **Time sensitivity**: crawls represent a point-in-time view; multiple crawls would be needed to study evolution (example: newer D3 versions). - **Coverage limitation**: only five sites were crawled; they explicitly plan to extend to more websites, more metadata (tool info, docs, possibly raw data), and richer design analysis (coordinated views, cross-filtering). --- ## Model - Pipeline: feature extraction over SVG $\rightarrow$ scikit-learn classifier (ultimately Random Forest). - Features: 114 total, in three groups (general/style/per-element), with per-element subfeatures for circle/rect/line/path/text. - Normalization for geometry: x positions divided by visualization width; y positions by height; line/path lengths by diagonal; widths by max(width,height). --- ## Data ### Crawl strategy and scale - Unguided crawl: approximately 20M pages visited, approximately 10k pages with visualizations (0.05%), dominated by redundant StackOverflow profile charts. - Targeted crawl: five SVG "islands" (D3, Plotly, Chartblocks, Fusion Charts, Graphiq) yielding 42k+ SVG visualizations. ### Evaluation datasets (Table 1 excerpts) - Collection sizes and type counts (examples shown in the table excerpt): - Chartblocks: 22,730 visualizations, 4 types (pie/line/bar/scatter with counts listed). - Fusion Charts: 530 visualizations, 10 types. - Graphiq: 2,727 visualizations, 11 types. - Plotly: 6,544 visualizations, 11 types. - Omitted data: about one quarter of extracted visualizations excluded due to extraction issues from complex page features (animations mentioned). ### Label construction - Label superset creation steps: review site docs/galleries, compare against observed visualizations to find missing labels, then consolidate into a final superset. - Some label consolidation choices: group geographic maps (choropleth, projections) and graphs/trees (dendrograms, trees); merge style variants (stacked vs grouped bars). ### Dataset access and license - **Dataset URL**: [https://homes.cs.washington.edu/~leibatt/beagle.html](https://homes.cs.washington.edu/~leibatt/beagle.html) - **Code license**: MIT ([beagle-annotator](https://github.com/leibatt/beagle-annotator)) - **Dataset license**: Not explicitly specified; assumed MIT based on code license --- ## Algorithms / Training - Classifier family comparisons: Multinomial NB, Gaussian NB, Decision Tree, SVM (default scikit-learn params) with similar performance among best; tree-based chosen initially for interpretability, then Random Forest to address overfitting. - Final model config: scikit-learn `RandomForestClassifier`, `n_estimators=14`, default other parameters. - Cross-validation: stratified 5-fold, repeated 10 times; weighted and non-weighted accuracy, plus F1. --- ## Evaluation - Within-collection weighted accuracies span roughly 0.8193 (D3) up to 0.9955 (Chartblocks), with corresponding non-weighted scores lower on more imbalanced/multi-type sets like D3. - Between-collection (Mixture) weighted accuracy reported as 0.8527, and the "Mixture (Revision)" line drops further (0.7952 weighted), suggesting sensitivity to domain shifts or representation differences captured by that revision baseline. - They explicitly frame these experiments as testing labeling robustness across different rendering environments/tools. --- ## Hardware / Production Hardware, training time, and throughput are not described in the paper. The implementation is described at the level of "off-the-shelf scikit-learn classifier" and feature extraction over SVG.