# CROHME 2014: Competition on Recognition of On-Line Handwritten Mathematical Expressions > Competition report for CROHME 2014 with new tasks for isolated symbol recognition with junk rejection and matrix expression recognition - Authors: Hunter Heidenreich - Published: 2014-09-01 - Updated: 2026-03-04 - Paper: CROHME 2014 Paper - Dataset: TC11 Dataset - Code: CROHME 2014 GitHub - Source: Bevaya Labs, --- ## TL;DR Competition report for CROHME 2014 with two new tasks: isolated symbol recognition with junk rejection and matrix expression recognition. Best system achieves 62.68% exact expression match on standard expressions; matrix recognition proves substantially harder at 53% expression-level accuracy. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Evaluation}}$ --- It is primarily a **competition report**: tasks, metrics, protocols, and benchmarked system results (multiple tables of rates and error metrics). - **Secondary:** $\\Psi\_{\\text{Resource}}$ --- It introduces/organizes **new test data** (all tasks) and a **new matrix representation + multi-level evaluation objects** (matrix/row/column/cell), and notes public availability of data/tools. ## What is the motivation? Handwritten math recognition requires solving joint segmentation, recognition, and 2D structure parsing over a large symbol set (101 classes in CROHME 2014). Unlike linear text, math expressions have spatial relations (superscripts, fractions, roots) that complicate interpretation. The task has practical value for pen-based math input on tablets and search applications. ## What is the novelty? Two new tasks expand the competition scope: 1. **Task 1: isolated symbol recognition with reject option.** Systems must accept valid symbols and reject "junk" (random 1-4 stroke sequences that don't form real symbols). This tests robustness to segmentation errors. 2. **Task 3: matrix recognition.** Expressions contain matrices where systems must detect matrix boundaries, infer row/column structure, and recognize cell contents (which may be arbitrary expressions). Evaluation reports recall at four levels: matrix, row, column, and cell. Task 2 (standard expression recognition) uses the same training set and grammar as 2013 with a new test set to track year-over-year progress. ## What experiments were performed? 8 systems submitted 16 runs across three tasks. Data collection used whiteboards, iPads, and Wacom tablets from three labs; test expressions came from Wikipedia. **Metrics:** - Task 1: Top-1 accuracy, TAR (true acceptance rate), and FAR (false acceptance rate) for junk rejection - Task 2: Expression-level exact match plus object-level recall/precision for segmentation, classification, and structural relations - Task 3: Expression rate, symbol rate, and recall at matrix/row/column/cell levels Systems were also evaluated on the 2013 test set for year-over-year comparison, though this risks validation leakage. ## What are the outcomes/limitations? **Task 2 (standard expressions):** Best system achieves 62.68% exact match (MyScript). Expression-level scores show large gaps between systems, but object-level metrics compress: segmentation and relation detection recalls vary less dramatically. The authors note potential validation leakage since 2014 systems could tune on the 2013 test set. **Task 1 (symbol + junk):** Test set contains 10,061 valid symbols and 9,161 junk examples. Adding junk typically drops accuracy 5-14 points. False acceptance rates vary widely (6-37%), while true acceptance rates cluster tighter (78-87%). Top systems use different classifiers (MLP, RNN, SVM), suggesting no dominant architecture. **Task 3 (matrices):** Only two systems participated. Best expression-level accuracy is 53%, substantially below standard Task 2 performance. Both systems recognize rows better than columns; column recall is the weakest sub-object. **Limitations:** Year-over-year comparison is confounded by potential test-set leakage. The matrix task's low participation limits analysis. Expression length increased (9.06 to 10.20 symbols/expression) but difficulty effects are unclear. * * * ## Model Competition report; no single architecture. Selected system details: ### System I (UPV, "seshat") **Task 1:** 7 online point features + rendered image (40px height) with 9 offline features per column. Two BLSTM-RNNs (online/offline) combined via rnnlib. **Tasks 2/3:** 2D stochastic context-free grammar parsing with GMM for spatial relationships. Matrix extensions enforce row/column count consistency and use center-difference features for relations. ### System II (University of São Paulo) 3-stage pipeline: (1) generate symbol hypotheses by combining each stroke with 3 nearest neighbors, (2) extract baseline tree from low-cost hypotheses, (3) parse LaTeX for grammar legality and select lowest-cost legal expression. ### System III (MyScript) **Task 1:** MLP combining trajectory features (position/direction/curvature) and bitmap features (projections/histograms). **Tasks 2/3:** Joint segmentation-recognition-interpretation with grammar encoding spatial relations. Uses "symbol expert" producing segmentation probabilities, statistical language model with spatial context, and global discriminant training at equation level. Matrix task adds row/column post-processing. ### System IV (RIT DRPL) **Task 1:** RBF-kernel SVM with online/offline features (stroke count, curvature, crossings, fuzzy histograms). **Task 2:** AdaBoost for stroke merging; MST-based parsing that groups vertical structures, identifies dominant operators, then processes baselines. Relations use bounding box geometry and shape context. ### System V (RIT CIS) **Segmentation:** Merge/split decisions over adjacent stroke pairs using 405 features (geometric + multi-scale shape context + classification probabilities), PCA to 100D, then RBF SVM. **Parsing:** Strong relations (above/below/inside) detected first; weak relations (right/sup/sub) via polar histogram features (15 distance $\\times$ 20 angle) with PCA and SVM. ### System VI (Tokyo Univ. of Agriculture and Technology) **Task 1:** 512D NCGF features reduced to 300D via FLDA; GLVQ classifier. Junk clustered into 64 LBG clusters. **Task 2:** Two SVMs learn structural relations; stroke order reduces parsing complexity to $O(n^3|P|)$. ### System VII (University of Nantes, IRCCyN) Joint optimization of segmentation, recognition, and structure under grammar. MLP with reject capability; "global learning" trains directly from expressions with explicit junk modeling. ### System VIII (ILSP/Athena) Template elastic matching using 8-direction Freeman chain code. Distance computed over normalized codes weighted by stroke-length proportions. * * * ## Data ### Task 2: full expressions (core CROHME task) - Training: **8,836 expressions**, **85,781 symbols**. - Test: **986 expressions**, **10,061 symbols**. ### Task 1: isolated symbols with junk - Training: **85,781 valid symbols**; junk is user-generated by a provided script from Task 2 training data. - Test: **10,061 valid symbols**, **9,161 junk**. ### Task 3: matrices - Training: **362 matrices**, **2,332 cells**, **4,281 symbols**. - Test: **175 matrices**, **1,075 cells**, **2,101 symbols**. - Reported averages: about **6 cells per matrix** (roughly a $2 \\times 3$) and about **2 symbols per cell**. (See Figure 1 examples of matrix expressions, p.2.) ### Collection protocol - Organizing labs: IRCCyN/IVC (France), RIT/DPRL (USA), ISI/CVPR (India). - Device diversity: whiteboard/tablet PC, iPads (finger), Wacom stylus tablet. * * * ## Algorithms / Training No unified training recipe. Common patterns: - **Joint inference:** Systems I, III, VII jointly solve segmentation, classification, and structure under grammar constraints - **Reject modeling:** Some systems implement explicit rejection for junk; others report "n.r.o." (no reject option) - **Feature fusion:** Systems I, III, IV combine online trajectory and offline rendered-image features - **Grammar-driven parsing:** Multiple systems use grammars or staged relation detection * * * ## Evaluation ### Task 1 (isolated symbols) Selected results (Top-1 on valid symbols without junk; with junk where available): | System | No junk | With junk | TAR | FAR | | ---------- | ------- | --------- | ------ | ------ | | System I | 91.24% | 84.14% | 80.29% | 6.44% | | System III | 91.04% | 85.54% | 87.12% | 10.39% | | System IV | 88.66% | 83.61% | 83.52% | 9.03% | | System V | 85.00% | 71.19% | 86.84% | 36.85% | Top systems use different classifiers (MLP/RNN/SVM). FAR varies more than TAR across methods. ### Task 3 (matrix recognition) | System | Expression | Symbol | Matrix | Row | Column | Cell | | ---------- | ---------- | ------ | ------ | ------ | ------ | ------ | | System III | 53.28% | 89.81% | 92.57% | 92.00% | 69.16% | 71.07% | | System I | 31.15% | 87.43% | 73.14% | 70.59% | 50.84% | 55.35% | Both systems struggle with columns more than rows. Expression-level accuracy is substantially lower than Task 2. ### Task 2 (full expressions) Best system (System III): 62.68% exact match. Next best (System I): 37.22% exact, 50.20% with $\\leq$3 errors. **System III object-level (recall/precision):** - Segmentation: 98.42/98.13 - Seg+class: 93.91/93.63 - Tree relations: 94.26/94.01 Authors also report Hamming distances on labeled graphs and derived error rates ($\\Delta Bn$, $\\Delta E$) to weight segmentation/classification/parsing contributions. * * * ## Data Availability Train and test data for all tasks are publicly available via the [IAPR TC11 site](https://tc11.cvc.uab.es/datasets/CROHME-2014_2) and [GitHub](https://github.com/DPRL/CROHME_2014). The paper does not specify an explicit license; the broader CROHME package (2011-2019) is distributed as CC BY-NC-SA 3.0 via TC10/11, though individual dataset bundles may vary. Evaluation scripts are included.