# TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning > 3B-parameter multimodal LLM for chart understanding using visual token merging and program-of-thoughts learning to improve efficiency and numerical reasoning - Authors: Hunter Heidenreich - Published: 2024-04-25 - Updated: 2026-03-04 - Code: TinyChart Code - Dataset: TinyChart Data - Source: Bevaya Labs, --- ## TL;DR TinyChart is a 3B-parameter multimodal LLM for chart understanding that targets efficiency via (1) Visual Token Merging inside the vision transformer to reduce high-res visual sequence length, and (2) Program-of-Thoughts (PoT) learning to offload numeric reasoning into executable Python programs. Authors report strong results on ChartQA and other chart benchmarks, plus higher inference throughput than larger (7B--13B) chart-focused MLLMs. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Method}}$ - Introduces an end-to-end model recipe combining visual token merging in ViT + PoT learning and validates via ablations and benchmark comparisons. - **Secondary:** $\\Psi\_{\\text{Resource}}$ - Builds ChartQA-PoT (PoT-augmented supervision) and describes its construction and statistics. - **Secondary:** $\\Psi\_{\\text{Evaluation}}$ (light) - Emphasizes efficiency metrics (throughput, OOM behavior at high resolution) alongside accuracy. * * * ## What is the motivation? Chart understanding requires: (a) robust OCR/text retrieval, (b) numerical reasoning, and (c) high-resolution vision encoding. Authors identify three constraints of existing chart MLLMs: 1. Large parameter counts hinder deployment/training (13B chart models need substantial VRAM). 2. Numerical errors are common for calculation-heavy questions. 3. High-resolution images produce long ViT token sequences that are expensive for the LLM to process. * * * ## What is the novelty? ### Visual Token Merging inside the ViT - **Observation:** Charts contain large regions of blank space / uniform color blocks, yielding many similar patches. - **Mechanism:** In each transformer layer, tokens are split into two disjoint sets, then bipartite matching selects the top-$r$ most similar token pairs across the sets (similarity = cosine distance between self-attention key vectors), and merges each paired feature by average pooling. - Not limited to spatial neighbors: non-adjacent tokens can merge if similar and in opposite sets. - **Proportional attention fix:** Since merging reduces multiplicity of a feature, they add a patch-count term $s$ into attention: $$ \\text{Attention}=\\text{softmax}\\left(\\frac{QK^\\top}{\\sqrt{d}}+\\log s\\right)V $$ where $s$ is the number of original patches represented by the merged token. ### Program-of-Thoughts learning for numeric chart QA - Instead of learning arithmetic implicitly, model is trained to output executable Python assignment statements (with comments), then a Python interpreter produces the final numeric answer. - Authors construct **ChartQA-PoT** from ChartQA training split using two pipelines: - **Template-based PoT:** 40 question templates (from PlotQA) + manually written numpy-style code templates; placeholders filled from chart tables, then rule-based filtering removes bad fills. - **GPT-based PoT:** gpt-3.5-turbo generates PoT code from (question + chart data table text); they execute generated code and keep only samples where execution succeeds and matches the annotated ChartQA answer. * * * ## What experiments were performed? - **Benchmarks:** ChartQA, Chart-to-Text (Pew), Chart-to-Table, OpenCQA; plus ChartX cognition generalization. - **ChartQA evaluation settings:** - **Direct:** short answer. - **PoT:** emit Python program; interpreter gives answer. - **Combine (default):** keyword rule decides if question is "calculative"; PoT for calculative, Direct otherwise; fallback to Direct if program has syntax errors. - **Oracle:** choose the correct one between Direct and PoT after the fact (upper bound). - **Ablations:** - With/without PoT data; template-only vs template+GPT PoT. - Resolution changes ($384 \\rightarrow 512 \\rightarrow 768$) with/without token merging; varying merge rate $r$. - Tracks visual sequence length and inference throughput as efficiency metrics. - **Qualitative case studies:** QA, chart-to-table, chart-to-text, and chart redrawing; includes error examples. * * * ## What are the outcomes/limitations? ### Outcomes (as reported) - TinyChart@512 and TinyChart@768 (3B) are reported to outperform or match larger chart MLLMs on several benchmarks, while achieving higher throughput. - **ChartQA setting breakdown:** Combine improves over Direct; Oracle is notably higher than Combine, implying combination policy is not optimal. - **Calculative vs non-calculative:** PoT is much better on "calculative" subset than Direct, while Direct remains strong on non-calculative; Combine captures gains from both. - **Efficiency:** Token merging enables high resolution (notably 768) without OOM on 32GB V100. - **Key numbers:** - Throughput (ChartQA test, batch size 1 on V100 32GB): TinyChart@512 3.65 it/s, TinyChart@768 3.14 it/s - ChartQA results: TinyChart@768 Direct 76.36, PoT 80.84, Combine 83.60, Oracle 89.12 - Ablation: 768×768 without merging produces visual length 2916 and OOM; with merging r=84 produces visual length 732 at 3.14 it/s ### Limitations / failure modes (from paper) **Extraction and OCR brittleness:** - Chart-to-table extraction struggles when values must be inferred from axes without explicit OCR labels near data points. This is a fundamental limitation when text merging relies heavily on visible annotations. - The paper does not quantify how often this failure mode occurs in their test sets. **Generation quality issues:** - Chart-to-text generation can hallucinate mismatched content even when some extracted values are correct, suggesting weak grounding between vision and language outputs. - Chart redrawing struggles with unseen chart types (example given: 3D bar charts), indicating limited generalization beyond training distribution. **PoT combination policy gap:** - The simple keyword-based routing between Direct and PoT modes leaves a substantial gap to Oracle performance (83.60 vs 89.12 on ChartQA). - The paper does not explore learned routing or confidence-based selection, which could close this gap. - Error analysis on when the policy fails is not provided. **Evaluation and reproducibility concerns:** - ChartQA-PoT construction relies heavily on GPT-3.5 generation (21K of 140K pairs), which may introduce style biases and limit diversity of reasoning patterns. - Template-based PoT generation uses 40 question templates from PlotQA; coverage of reasoning types beyond these templates is unclear. - The paper does not report inter-annotator agreement or validation rates for the generated PoT programs beyond "execution succeeds and matches answer." - Full training cost breakdown (GPU hours, energy, cost) is not provided beyond "3 days on 32 V100s." * * * ## Model ### Architecture - Standard MLLM structure: Vision Transformer encoder → vision-language connector → LLM decoder. - **Vision Transformer:** Token merging reduces sequence length by approximately $r$ per layer; parameter-free merging module inserted into each ViT layer. - For image resolution $N \\times N$ and patch size $P \\times P$, number of vision tokens is $(\\lfloor N/P \\rfloor)^2$; without reduction, this long sequence is passed downstream. - **Vision-language connector:** Implemented as an MLP with one hidden layer and GeLU activation, mapping vision features into the LLM embedding space. - **LLM:** Transformer decoder with causal mask; supervised fine-tuning objective is LM loss over response tokens only. ### Training recipe - **Initialization:** from TinyLLaVA, using SigLIP as vision encoder and Phi-2 as LLM. - **Vision resolution changes:** base encoder at 384×384; extend to 512×512 and 768×768 with token merging. - **Token merging rates:** - 512×512: $r=20$ (authors also ablate $r=12,15,20$). - 768×768: $r=84$ (enables training vs OOM without merging). - Train entire model for 3 epochs, batch size 512, learning rate $1\\times 10^{-4}$, warmup 3% of steps, then decay to 0. - **Training cost:** 3 days on 32 Tesla V100 GPUs (32 GB VRAM). * * * ## Data ### ChartQA-PoT (PoT supervision) - Built from ChartQA training split; total 140,584 (question, PoT answer) pairs. - **Construction counts:** - Template-based: 119,281 pairs over 17,498 images. - GPT-based: 21,303 pairs over 15,521 images. - **GPT-based generation constraints:** assignment statements only; no loops/branches; one-line comment before each statement; last variable must be `Answer`; restricted operator/function list (numpy ops + simple arithmetic/comparators). ### Multitask supervised fine-tuning mix - Total training samples: 1,364,921 across tasks (QA, chart-to-text, chart-to-table, instruction-following), using datasets including ChartQA, PlotQA, DVQA, OpenCQA, Pew/Statista/Vistext/ChartSumm/Chart2Text-8k, and ChartLlama instruction data. * * * ## Evaluation ### Metrics and protocols - **ChartQA:** "relaxed accuracy" allowing numerical error within 5%. - **Chart-to-Text (Pew):** BLEU4. - **Chart-to-Table:** RMSF1 metric. - **OpenCQA:** BLEU4. - **ChartX cognition:** GPT-Accuracy for QA, GPT-score for summary/description/redrawing. ### Routing rule for "calculative" questions Uses a keyword detector; list includes terms like "sum, mean, average, ratio, subtract, divide, times, absolute, minus, greater, lowest, number, how many" etc.