# ICFHR 2016 CROHME: Competition on Recognition of Online Handwritten Mathematical Expressions > CROHME 2016 competition report evaluating online handwritten math recognition across four tasks with updated evaluation tooling and Wikipedia formula corpus - Authors: Hunter Heidenreich - Published: 2016-09-01 - Updated: 2026-03-04 - Paper: CROHME 2016 Paper - Dataset: IAPR TC10/11 Dataset - Source: Bevaya Labs, --- ## TL;DR CROHME 2016 evaluated online handwritten math recognition across four tasks: end-to-end formula recognition, isolated symbol classification, structure parsing from provided symbols, and matrix recognition. **MyScript** achieved the best overall results (67.65% fully correct on Task 1) but used additional private training data. Among systems trained only on provided data, **WIRIS** led on most tasks, while structure recognition remained a key bottleneck even with perfect symbol recognition. * * * ## What kind of paper is this? **Dominant:** $\\Psi\_{\\text{Evaluation}}$ Competition protocol, standardized metrics, ranking methodology, error analysis, and comparative evaluation across four tasks. **Secondary:** $\\Psi\_{\\text{Resource}}$ New test sets for Tasks 1 and 4, updated evaluation tooling (CROHMELib/LgEval), and a Wikipedia formula corpus (592,000+ samples) for language modeling. ## What is the motivation? Progress in handwritten math recognition lacked **standardized benchmarks and evaluation metrics**, making it difficult to track improvements across systems. CROHME 2016 aims to: - Provide common datasets and evaluation protocols - Separately evaluate subproblems: symbol recognition, structural parsing, and end-to-end performance - Introduce experimental matrix recognition as a focused challenge ## What is the novelty? **Task structure:** - **Task 1:** End-to-end formula recognition from strokes - **Task 2a/2b:** Isolated symbol classification; Task 2b adds junk samples to test rejection - **Task 3:** Structure recognition from provided symbols (isolates spatial parsing from symbol recognition) - **Task 4:** Matrix recognition from strokes (experimental) **Resources:** - 592,000+ Wikipedia formulae (LaTeX + Presentation MathML) for language model training - Web-based submission and evaluation system for real-time scoring ## What experiments were performed? Multi-task evaluation with standardized metrics: - **Expression-level:** Recognition rates (exact match, $\\leq 1$ error, $\\leq 2$ errors) - **Symbol-level:** Recall/precision for segmentation and classification - **Relationship-level:** Recall/precision for spatial relation parsing - **Isolated symbols:** Top-1 accuracy, TMP (average rank of correct class in Top-10) - **Rejection (Task 2b):** TAR/FAR for junk sample handling - **Error analysis:** Symbol confusion matrices, frequent relationship errors ## What are the outcomes/limitations? **Key findings:** - End-to-end recognition remains challenging: best system achieves 67.65% fully correct on Task 1 - Structure parsing is the primary bottleneck: even with perfect symbols (Task 3), best system reaches only 90.67% structure accuracy - Isolated symbol classification performs well (low-90s Top-1) but hits ceiling due to ambiguity (`x` vs `X` vs `$\times$`, `o` vs `O` vs `0`) - Junk sample rejection (Task 2b) introduces TAR/FAR tradeoffs and degrades performance **Limitations:** - Best overall system (MyScript) used private training data, limiting fair comparison - Many symbol errors stem from context-independent classification - Relationship confusions (especially Right vs. Subscript/Superscript) remain frequent * * * ## Model ### Task definitions (what each system must output) - **Tasks 1/3/4:** output an interpreted formula structure represented as a labeled graph (Symbol Layout Tree, SLT) over strokes/symbols/relations; scoring supports structure-only and structure+labels. - **Task 2:** output ranked Top-10 class predictions for isolated symbols; Task 2b includes a reject/junk mechanism. ### Formula representation: Symbol Layout Trees as label graphs - Formulae are encoded as **label graphs** (adjacency matrices): - Diagonal entries label each stroke's symbol class association. - Intra-symbol grouping is represented by **bidirectional edges** among strokes in the same symbol, labeled by the symbol class. - Spatial relations are represented by directed edges from **each stroke of a parent symbol to each stroke of a child symbol**, labeled by relation type. - Matrices generalize label graphs to allow **sets of labels** per node/edge so a stroke can belong simultaneously to symbol/cell/row/column/matrix objects; segmentation can be recovered via labeled cliques. ### Participant system summaries **MyScript (Tasks 1--4):** - Integrated segmentation/recognition/interpretation with grammar-guided spatial relations - Features: dynamic trajectory (direction/curvature) + static bitmap - Deep MLP + RNN classifier with statistical language model - Used $\\sim$30k additional private training samples **WIRIS (Tasks 1--4):** - Probabilistic grammar with statistical LM trained on CROHME + Wikipedia data - Neural nets with mixed online/offline features (point sequences, HOG) - Matrix-specific handling (dimension matching, spatial segmentation) **Tokyo Univ. of Agriculture and Technology (Tasks 1/2/3):** - Symbol classifier: CNN (offline) + LSTM (online) - Structure: CYK parsing with stroke-order heuristics - Updated from CROHME 2014 system **University of Nantes (Task 1):** - Converts 2D ink to 1D paths in stroke graph - BLSTM + local CTC for path labeling - Merges paths to label graph; no language model **University of São Paulo (Tasks 1/3):** - Two-stage: hypotheses graph generation + grammar-based parsing **RIT (Task 2):** - Direction/order-tolerant shape descriptors - Features: crossings, 2D histograms, visual words (k-means) * * * ## Data ### Dataset splits and sizes (Table I) | Task | Training | Validation | Test | | ------------------------------- | -----------------------: | ----------------------: | -------------------------------: | | Task 1 (Formulae) | Train 2014: 8,836 expr. | Test 2014: 986 expr. | **Test 2016: 1,147 expr. (new)** | | Task 2a (Valid symbols) | Train 2014: 85,802 symb. | Test 2013: 10,061 symb. | Test 2014: 10,019 symb. | | Task 2b (Valid + Junk) | Junk: 74,284 | Junk: 9,161 | Junk: 8,416 (new seed) | | Task 3 (Structure; new in 2016) | Train 2014: 8,836 expr. | Test 2013: 671 expr. | Test 2014: 986 expr. | | Task 4 (Matrices; experimental) | M.Train 2014: 362 expr. | M.Test 2014: 175 expr. | **M.Test 2016: 250 expr. (new)** | (Table content summarized from the paper.) ### New data collection (Tasks 1 and 4) **Source:** ArXiv papers (2000--2001) from KDD 2003 Cup; 1,147 expressions selected using CROHME 2014 grammar/frequency constraints. **Collection setup:** - 50 writers - Three device types: pen-based tablet PC (12"), touch-screen (27", finger input), pen-based interactive whiteboard **Ground truth pipeline:** 1. Show rendered LaTeX for copying 2. Store LaTeX + strokes in InkML 3. Parse LaTeX to enumerate symbols/classes 4. Guide automatic recognition/parsing 5. Manual verification and correction 6. Average: 1 minute per expression ### Wikipedia formula corpus for language models - Provided **592,000+** English Wikipedia formulae in **LaTeX** and **Presentation MathML**, sourced from NTCIR-12 MathIR data, intended for language model parameter fitting. ### Data usage - All participants used provided training data - **WIRIS:** Wikipedia corpus for language model training - **MyScript:** Additional $\\sim$30,000 private formulae - **RIT:** Synthetic data generation ($\\sim 5\\times$ expansion for symbol training) * * * ## Algorithms / Training Key algorithmic patterns across participant systems: - **Grammar-based joint segmentation + parsing** (MyScript, WIRIS): probabilistic scoring with language modeling - **Sequence models** (Nantes): RNN/LSTM/BLSTM with CTC-style labeling for stroke/path labeling - **Graph-based approaches** (São Paulo): hypothesis generation + grammar-based parsing - **Feature fusion** (Tokyo): CNN (offline) + LSTM (online) ensembles for symbol classification Note: As a competition report, detailed training recipes are system-specific and not fully disclosed. * * * ## Evaluation ### Tooling - Evaluation uses updated **CROHMELib** and **LgEval**, providing metrics at stroke/symbol/expression levels plus automated error analyses (confusion matrices/histograms for symbol/relationship subgraphs). ### Metrics (by task) - **Tasks 1/3/4 (formula/structure/matrix):** - Expression recognition rates for (a) structure-only and (b) structure+labels; also reported are rates allowing $\\leq 1$ and $\\leq 2$ label errors in the label-graph adjacency matrix. - Recall/precision for symbol segmentation and segmentation+classification; likewise for relationship segmentation and relationship segmentation+classification. - **Task 2 (isolated symbols):** - Top-1 recognition and **TMP** (average rank of correct class in Top-10; missing treated as rank 11). - Task 2b adds **TAR** (true acceptance rate for valid) and **FAR** (false acceptance rate for junk). ### Results: Task 2 (symbols) | System | Task 2a Top-1 | Task 2a TMP | Task 2b Top-1 | Task 2b TMP | Task 2b TAR | Task 2b FAR | | -------- | ------------: | ----------: | ------------: | ----------: | ----------: | ----------: | | MyScript | 92.81 | 1.13 | 86.77 | 1.19 | 89.82 | 11.16 | | Tokyo | 92.27 | 1.15 | -- | -- | -- | -- | | RIT | 88.85 | 1.25 | 83.34 | 1.31 | 95.86 | 19.71 | (Table II summarized.) ### Results: Task 1 (end-to-end formulae from strokes; Test 2016) | System | Structure rate | Structure+labels rate | $\\leq 1$ err | $\\leq 2$ err | | --------- | -------------: | --------------------: | ------------: | ------------: | | MyScript | 88.14 | 67.65 | 75.59 | 79.86 | | WIRIS | 74.28 | 49.61 | 60.42 | 64.69 | | Tokyo | 61.55 | 43.94 | 50.91 | 53.70 | | São Paulo | 57.02 | 33.39 | 43.50 | 49.17 | | Nantes | 21.45 | 13.34 | 21.02 | 28.33 | (Table III summarized.) ### Results: Task 3 (structure from provided symbols; Test 2014) | System | Structure rate | Structure+labels rate | $\\leq 1$ err | $\\leq 2$ err | | --------- | -------------: | --------------------: | ------------: | ------------: | | MyScript | 90.67 | 84.38 | 85.90 | 87.62 | | WIRIS | 86.61 | 78.80 | 80.42 | 82.75 | | São Paulo | 69.27 | 64.81 | 67.34 | 70.69 | | Tokyo | 70.99 | 61.46 | 63.89 | 66.84 | (Table III summarized.) ### Symbol-level vs. relationship-level performance (Task 1) - Systems achieve competitive **symbol recall** but lower expression-level performance due to compounding errors across segmentation, labeling, and relation parsing - Many systems show **precision > recall**, indicating a tendency to under-segment when errors occur - Most common relationship confusions: **Right-adjacency** vs. **Subscript/Superscript** See Table IV for complete recall/precision breakdown. ### Results: Task 4 (matrices; Test 2016) | System | Expression rate | Symbol recall | Matrix recall | Row recall | Column recall | Cell recall | | -------- | --------------: | ------------: | ------------: | ---------: | ------------: | ----------: | | MyScript | 68.40 | 94.86 | 97.52 | 95.61 | 90.71 | 87.49 | | WIRIS | 56.40 | 87.03 | 85.67 | 87.16 | 82.22 | 84.68 | (Table V summarized.) ### Common error patterns **Symbol ambiguity:** - `x` vs `X` vs `$\times$` - `o` vs `O` vs `0` - `p` vs `P` - Punctuation size (comma vs. dot) without context **Error distribution:** - Frequent symbols dominate error lists (`1`, `2`, ambiguous `x`/`$\times$`) - See Tables VI--VII for complete symbol and bigram confusion matrices * * * ## Hardware / Production Compute requirements and inference specifications not reported; paper focuses on datasets, evaluation protocols, and comparative results. * * * ## Data Availability & Licensing ### Dataset availability **CROHME 2016 datasets (Tasks 1, 2a, 2b, 3, 4):** - Available via IAPR TC10/11 dataset package - Includes training + test data from CROHME 2011--2019 - Ground truth provided in InkML (LaTeX string, MathML structure, SLG/OLG formats) - More than 10,000 labeled handwritten formulae across all releases - [IAPR TC10/11 Resource][1] **Wikipedia formula corpus:** - 592,000+ formulae (LaTeX + Presentation MathML) - Sourced from NTCIR-12 MathIR - Intended for language model training ### Licensing **CROHME datasets (IAPR TC10/11):** - **CC BY-NC-SA 3.0** (Creative Commons Attribution-NonCommercial-ShareAlike 3.0) - Academic/research use only; no commercial use - [IAPR TC10/11 Resource][1], [CROHME Portal][2] **Wikipedia formula corpus:** - Source: NTCIR-12 MathIR - NTCIR distribution: restricted use scope, prohibits redistribution ([NTCIR Agreement][3]) - Wikipedia content: CC BY-SA 4.0 and GFDL ([Wikipedia Copyrights][4]) - Licensing depends on distribution source **Summary:** CROHME datasets use CC BY-NC-SA 3.0 for non-commercial research. Wikipedia corpus licensing varies by source. [1]: https://tc101-demo.github.io/datasets/ICDAR2019-CROHME-TDF_1 "ICDAR 2019 Competition on Recognition of Handwritten Mathematical Expressions and Typeset Formula Detection" [2]: https://www2.isical.ac.in/~crohme/CROHME_data.html?utm_source=chatgpt.com "CROHME - isical.ac.in" [3]: https://research.nii.ac.jp/ntcir/ntcir-12/permission/ntc12math-agrmnt-participant-e.pdf "Microsoft Word - ntc12math-agrmnt-participant-e-150804.cgd151028.doc" [4]: https://en.wikipedia.org/wiki/Wikipedia%3ACopyrights?utm_source=chatgpt.com "Wikipedia:Copyrights - Wikipedia"