# VisText: A Benchmark for Semantically Rich Chart Captioning > A chart captioning benchmark with semantically rich captions (L1-L3) for 12,441 area, bar, and line charts; demonstrates text-only representations outperform image-based approaches. - Authors: Hunter Heidenreich - Published: 2023-08-01 - Updated: 2026-03-04 - Code: VisText Code + Dataset - Source: Bevaya Labs, --- ## TL;DR VisText is a chart-captioning benchmark designed to push beyond surface-level captions by including human-written, semantically rich statements (notably "value/trend" style content) and multiple machine-consumable chart representations (image, data table, scene graph). Their experiments suggest **text-only representations (scene graph or table) outperform image-based approaches**, and semantic prefix-tuning is mainly useful for **controlling** semantic level rather than improving standard metrics. * * * ## What kind of paper is this? **Dominant:** **0.55 $\\Psi\_{\\text{Resource}}$** (new benchmark + dataset + standardized inputs) **Secondary:** **0.25 $\\Psi\_{\\text{Method}}$** (semantic prefix-tuning to control caption semantic level) **Secondary:** **0.20 $\\Psi\_{\\text{Evaluation}}$** (systematic comparisons across representations, models, and metrics) ## What is the motivation? Prior chart captioning benchmarks and methods often emphasize **surface descriptions** (or omit deeper semantics), while assistive technologies and visualization authoring workflows benefit from captions that include **derived insights** (values, comparisons, trends). The paper frames semantically rich captions as more useful and closer to human-written chart descriptions. ## What is the novelty? 1. **Dataset design for richer semantics:** Each chart has multiple representations (image, data table, scene graph) and captions intended to be more semantically informative than older datasets. 2. **Multi-level captioning framing:** Uses a semantic "level" lens (L1--L4) adapted from accessibility guidance and prior work, focusing on the first three levels for modeling. 3. **Semantic prefix-tuning:** A fine-tuning approach that aims to generate different semantic levels without training separate models per level. ## What experiments were performed? - **Quantitative benchmarking** comparing: - text-only models from **scene graphs** vs **data tables** - multimodal/image-guided models (image-only and image+text hybrids) - prior baseline (Kantharaj et al. 2022) using BLEU, perplexity, ROUGE-L, WMD, TER, and a relation-generation check. - **Ablations** over different language-model backbones (ByT5-small, T5-small, BART-base) and prefixes. - **Qualitative error analysis** categorizing common failure modes (identity/direction/value/etc.). ## What are the outcomes/limitations? - **Outcome:** Image-based captioning performed worst; text-only encodings (scene graph or table) performed best, with scene graphs often competitive with data tables. - **Outcome:** L1 captions are much easier than L2/L3 (models score far higher on L1 than on richer statements). - **Limitation:** Dataset focuses on **three univariate chart types**; generalization to more complex charts and real-world, messy charts remains open. - **Limitation:** Models are evaluated via proxies; the paper calls out the need for work on interactive authoring and better grounding/linking of text to chart regions. * * * ## Model ### Inputs (three chart representations) Each example includes: - **Rasterized image** of the chart, - **Underlying data table**, and - **Scene graph** derived from the rendered visualization. ### Baseline families evaluated - **Text-only seq2seq**: ByT5-style models mapping a text linearization of scene graph or data table to captions. - **Image-guided**: VL-T5 variants using image features (alone or paired with text representations). ### Semantic prefix-tuning They propose prefix-tuning to generate captions at different semantic levels **without training separate models**; importantly, they note it **does not necessarily improve standard metrics**, but supports semantic-level control. ## Data ### Source data and cleaning - Data tables come from the **Statista** public dataset; they convert tables to structured form, handle missing values, and normalize some date-like formats (e.g., week to datetime). ### Chart generation (synthetic but structured) - For each table, they iterate through field pairs to create univariate charts (quantitative measure vs categorical/temporal dimension), using Vega-Lite/Altair-like constraints (e.g., max rows/cols for certain field types). - Final dataset size and composition: **12,441 charts** across **3,189 area**, **6,238 bar**, **3,014 line**, split approximately **80:10:10** train/val/test. ### Caption construction (semantic levels) They use a 4-level caption taxonomy (L1--L4) adapted from accessibility guidance and prior work, focusing on **L1--L3** for this benchmark. - **L1 generation (synthetic):** Templates produce a first sentence (3 templates) plus follow-on sentences (26 combinations), with randomized synonym phrasing and optional omissions (e.g., dropping scale words or outlier mentions). - **L2/L3 collection (crowdsourced):** Captions are written by crowd workers; they are instructed to produce multiple "non-repeating" captions per chart and are screened for vision/location/approval criteria. ### Dataset analysis of semantic content On a 2% sample (230 captions), they find most semantic statements are **L2 or L3**, with L3 appearing about **1.4×** as often as L2 in that coding. ## Algorithms / Training ### Representation preprocessing (critical implementation detail) Before training, they **minimize and linearize** both scene graphs and tables to reduce token length: - Scene graphs are reduced by preserving specific elements (title, axis labels/ticks, marks, mark coordinates/sizes) and removing other details, then linearized with a depth-first traversal. - The paper reports average representation lengths (for the "reduced" forms) on the order of **948 characters** for scene graphs vs **426 characters** for data tables. ### Training setup (reported) - Text models (ByT5): 50 epochs, batch size 1024, learning rate 5e-5, dropout 0.1. - Image-guided VL-T5: 50 epochs, batch size 512, learning rate 2e-5. ## Evaluation ### Metrics They use BLEU, ROUGE-L, WMD, TER, plus **GPT-2-medium perplexity** as a fluency proxy, and a "relation generation" check based on matching chart fields (title, axis names, etc.) mentioned in captions. ### Main quantitative findings (Table 1) On combined captions (L1+L2+L3), representative rows include: - Prior baseline (Kantharaj et al. 2022): BLEU **0.30**, perplexity **28.51**. - Text-only (scene graph, no PT): BLEU **0.34**, perplexity **17.04**. - Text-only (data table, no PT): BLEU **0.34**, perplexity **17.86**. - Image-only: BLEU **0.13**, perplexity **51.65** (worst). Overall narrative: image models underperform; scene graphs and data tables do well and are close to each other. ### L1 vs L2/L3 difficulty They explicitly report that models do much better on L1 than on L2/L3. For example, scene-graph model BLEU is **0.71** on L1 but about **0.06** on L2/L3 in their breakdown table. ### Ablation observations Backbone comparison suggests performance differences across ByT5-small, T5-small, and BART-base, with notes about prefix-tuning feasibility (they mention not being able to prefix-tune BART in their setup). ### Qualitative error analysis (common failure modes) In manual analysis of generated captions, they report broad error categories such as: - **Identity errors** (wrong entity/label): 86 errors (22.93%). - **Direction errors** (wrong trend direction): 32 errors (8.53%). - **Value errors** (wrong numeric values): 12 errors (3.20%). - **Stability errors**: 4 errors (1.07%). - **Repetition**: 117 errors (31.2%). - **Nonsensical**: 9 errors (2.4%). ## Hardware / Production ### Reported training hardware + runtime They trained on **8× Titan XP (12 GB)** GPUs, with 16 CPU cores and 128 GB RAM, reporting per-epoch training times on the order of minutes (varying by model and prefix-tuning). ### Production status No deployed system is claimed; the paper positions this as a benchmark and a step toward mixed-initiative authoring workflows and richer accessibility outputs rather than a production-ready captioning pipeline.