# MathWriting: Notes > Large-scale online handwritten mathematical expression dataset with 230k human-written inks and 396k synthetic inks, each paired with raw and normalized LaTeX labels - Authors: Hunter Heidenreich - Published: 2024-04-01 - Updated: 2026-03-04 - Code: MathWriting Code - Dataset: MathWriting Dataset - Source: Bevaya Labs, --- ## TL;DR MathWriting is a large-scale online handwritten mathematical expression dataset with 230k human-written inks and 396k synthetic inks, each paired with raw and normalized LaTeX labels. The paper proposes a benchmark using token-level character error rate and reports baseline results for CTC Transformer, PaLI, and PaLIGemma. Adding synthetic data reduces test CER from 6.20 to 5.49. * * * ## What kind of paper is this? **Dominant:** $\\Psi\_{\\text{Resource}}$ Primary contribution is a dataset and benchmark protocol (splits, normalization statistics, licensing, baseline models). Secondary: $\\Psi\_{\\text{Evaluation}}$ through proposed test protocol with tokenized LaTeX CER. ## What is the motivation? Handwritten mathematical expression recognition is challenging due to inherent 2D spatial structure. Online handwriting (stroke sequences) differs from static bitmaps. Data scarcity is a bottleneck: collecting real handwritten math requires specialized hardware and human effort. Existing benchmarks (CROHME) have smaller vocabularies and fewer samples. MathWriting expands symbol coverage, formula diversity, and supports offline recognition via rasterization. ## What is the novelty? - **Scale:** Largest published online handwritten math dataset (650k inks vs 164k in CROHME23) - **Dual labels:** Raw LaTeX plus normalized labels to remove training/evaluation ambiguities - **Synthetic pipeline:** Constructs diverse expressions by pasting handwritten symbol inks into LaTeX-derived bounding boxes; bounding-box data published for custom synthesis - **Split strategy:** Test designed for low label overlap with train, motivated by findings that "seen vs unseen label" matters more than writer identity ## What experiments were performed? Baselines trained/fine-tuned on MathWriting (train + synthetic), evaluated on valid/test with token-level CER: - OCR API (rasterized inks) - CTC Transformer (online) - PaLI (encoder-decoder VLM, point sequence + raster) - PaLIGemma (decoder-only LLM, image input with speed features) - Synthetic data ablation for CTC Transformer ## What are the outcomes/limitations? **Outcomes:** - Test CER (lower is better): OCR API 7.17, CTC Transformer 5.49, PaLI 5.95, PaLIGemma 5.97 - Synthetic data improves CTC Transformer: 6.20 $\\rightarrow$ 5.49 CER - Vocabulary expansion: 254 tokens vs 105 in CROHME23; includes matrices not in CROHME23 **Limitations:** - Single-formula scope; models may not transfer to full handwritten pages - LaTeX-only labels; not intended for general handwritten language - Some distinctions intrinsically ambiguous from ink alone (e.g., "z" vs "2") - Noise remains: stray strokes <1%, incorrect ground truth ~1--2% - Normalization is syntactic; cannot resolve semantic cases (e.g., "cos" vs `\cos` in "tacos") --- ## Model ### Input representations - **Online ink:** Sequence of strokes; each stroke is a sequence of $(x, y, t)$ points where $t$ is timestamp - **Rasterized:** Used for OCR-style models; mixed with point sequences for VLM fine-tuning ### Baseline architectures **CTC Transformer (35M params):** - Transformer-base with CTC loss - 11 layers, embedding size 512, swish activation, dropout 0.15 **PaLI (700M params):** - Encoder-decoder VLM fine-tuned on MathWriting - Uses both point sequences and rasterized ink **PaLIGemma (3B params):** - Decoder-only LLM (Gemma) with image input at 448px - Trained using ink rendering with speed information --- ## Data ### Dataset splits and sizes Five splits: **train, valid, test** (human-written), **symbols** (isolated symbols for synthesis), **synthetic** (generated expressions). | Split | Inks | Distinct Labels | |-----------|------|-----------------| | train | 230k | 53k | | synthetic | 396k | 396k | | valid | 16k | 8k | | test | 8k | 4k | Total: 650k inks with 254-token vocabulary (vs 164k inks, 105 tokens in CROHME23). ### Collection protocol (human-written) - Collected via in-house Android app: contributors copy rendered prompt (bitmap from LaTeX) using finger or stylus - 6 campaigns (2016–2019), each 2–3 weeks; contributors hired internally - Prompt sources: ~95% Wikipedia; remainder generated for underrepresented structures (nested fractions, rare symbols, matrices) - Device diversity: ~150 device types; different sampling rates and artifacts ### Synthetic generation - Uses LaTeX compilation outputs (DVI-derived bounding boxes) to place handwritten symbol inks into expression layouts - Individual symbol inks manually extracted from train (20–30 occurrences per symbol) to build symbols split - Synthetic expressions tend to be longer (90th percentile: 68 chars vs 51 in train) for length generalization ### Label normalization Each sample includes **raw annotation label** (as collected) and **normalizedLabel** (for training/eval robustness). Normalization: remove spaces, standardize braces, order sub/superscripts, rewrite `\over` $\rightarrow$ `\frac`, collapse synonyms, rewrite function commands (e.g., `\sin`) to letter sequences, normalize matrix environments, drop size modifiers (`\left`/`\right`). --- ## Algorithms / Training ### Split strategy - Human-written split by writer (early) and by label (later), motivated by findings that "seen vs unseen label" mattered more than writer style - Label overlap: valid has substantial overlap with train; test kept low (355 shared labels train-test) ### Training recipes **CTC Transformer:** - Adam, lr $1 \times 10^{-3}$, batch 256, 100k steps **PaLI:** - 200k steps, batch 128, lr 0.3, dropout 0.2; three runs with different shuffles **PaLIGemma:** - lr $1 \times 10^{-4}$, batch 512; image input 448px --- ## Evaluation ### Benchmark protocol Evaluate on **test split** with **token-level CER**, where "character" = **LaTeX token** (not ASCII character). Tokenization code provided in Appendix M. ### Results | Model | Params | Valid CER | Test CER | |-----------------|--------|-----------|----------| | OCR API | - | 6.50 | 7.17 | | CTC Transformer | 35M | 4.52 | 5.49 | | PaLI | 700M | 4.47 | 5.95 | | PaLIGemma | 3B | 3.95 | 5.97 | ### Synthetic data ablation (CTC Transformer) | Configuration | Valid CER | Test CER | |-------------------|-----------|----------| | With synthetic | 4.52 | 5.49 | | Without synthetic | 4.64 | 6.20 | --- ## Hardware / Production - **CTC Transformer:** 4 hours on 4 TPU v2 per run (100k steps) - **PaLI:** 14 hours on 16 TPU v5p per run; total experiment cost: 2 TPU v2 days + 28 TPU v5p days - **PaLIGemma:** 36 hours on 64 TPU v5p