# ICDAR 2019 CROHME + TFD: Competition on Recognition of Handwritten Mathematical Expressions and Typeset Formula Detection > CROHME 2019 competition report introducing symLG representation for evaluating LaTeX-output systems alongside graph-based systems, with expanded training data and a new typeset formula detection task - Authors: Hunter Heidenreich - Published: 2019-09-01 - Updated: 2026-03-04 - Paper: CROHME+TFD 2019 Paper - Paper: CROHME+TFD 2019 Paper (PDF) - Code: IoU Implementation - Source: Bevaya Labs, --- ## TL;DR CROHME 2019 introduces a **symbol-level label graph (symLG)** representation allowing systems that output LaTeX (common in encoder-decoder approaches) to be evaluated against prior CROHME graph-based formats. Best results improved versus CROHME 2016 (online expression rate up to 80.73%), though handwritten math recognition remains challenging with 101 symbol classes and complex 2D structure. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Evaluation}}$ --- Competition protocol with new symLG representation for metrics, ranking methodology, and comparative evaluation across three tasks. - **Secondary:** $\\Psi\_{\\text{Resource}}$ --- Expanded training data (added 2012/2013 test sets), new 2019 test set, TFD dataset (36 train PDFs, 10 test PDFs), and updated evaluation tooling. ## What is the motivation? The shift toward encoder-decoder systems that emit **LaTeX** without stroke segmentation created an **evaluation mismatch** with CROHME's historical stroke-level label graph format. CROHME 2019 addresses this by evaluating **symbolic structure** using a new **symLG** representation, enabling fair comparison between LaTeX-output systems and graph-output systems. ## What is the novelty? **symLG representation:** Converts both stroke-level label graphs and LaTeX outputs into a **symbol-level** graph/tree format compatible with existing CROHME evaluation tools (LgEval, CROHMELib). Nodes are identified by the **sequence of relation labels from root** (e.g., "oRRSup" for Right, Right, Superscript). Similarity computed using adjacency matrix with symbol labels on diagonal and spatial parent-child relations off-diagonal. **Three tasks:** - **Task 1 (online):** strokes → SLT; ranked by expression recognition rate. Subtasks: 1a isolated symbols (+ junk), 1b parsing from provided symbols. - **Task 2 (offline):** rendered grayscale images → SLT; ranked by expression rate. Subtasks: 2a isolated symbols (+ junk), 2b parsing from provided symbols. - **Task 3 (TFD):** detect formula bounding boxes on document pages (given character boxes); ranked by F-measure after one-to-one matching with IoU ≥ 0.75. **Dataset refresh:** Expanded training by adding prior CROHME test sets (2012, 2013) and introduced new 2019 handwritten test set from 80 writers and multiple devices. ## What experiments were performed? Evaluation across three tasks with standardized metrics: **Datasets:** Training = Train 2014 + Test 2013 + Test 2012 (9993 expr); validation = Test 2014 (986 expr); test = Test 2019 (1199 expr). TFD: 36 train PDFs (569 pages, 26,395 regions), 10 test PDFs (236 pages, 11,885 regions). **Metrics:** - Handwritten tasks: expression rate, symbol recognition rate, relationship metrics via LgEval/CROHMELib - TFD: F-score after one-to-one matching with IoU ≥ 0.75 (also reports IoU ≥ 0.5) **Selected participant approaches:** - **USTC-iFLYTEK (Tasks 1, 2):** Attention-based encoder-decoder; RNN encoder (online), CNN encoder (offline); external RNN language model from NTCIR-12 MathIR. - **Samsung R&D Team 2 (TFD, winner):** Graph-theoretic methods for multi-character formulas + statistical/context recognition for single-character math. - **RIT Teams (TFD):** Modified YOLOv3 and SSD512 with sliding windows and voting-based pooling. - **PAL-v2 (offline):** Heavy augmentation (330k images), Paired Adversarial Learning, ensemble of 6 models. ## What are the outcomes/limitations? **Outcomes:** Best expression rate improved versus CROHME 2016 (80.73% vs 67.65% on online task). USTC-iFLYTEK achieved 80.73% (Task 1 online) and 77.15% (Task 2 offline). Samsung R&D-2 achieved 93.45% F1 on TFD (IoU ≥ 0.75). **Limitations:** - **symLG tradeoffs:** Stroke segmentation performance cannot be computed; systems can achieve correct SLT without correctly segmenting symbols. Symbols identified by relationship paths means structural shifts manifest as missing symbols ("ABSENT"), potentially underestimating symbol recall. - **TFD metric sensitivity:** Large performance gap between winner and others. Using IoU ≥ 0.5 substantially raises F-scores for non-winning systems. - **Low participation:** No teams participated in symbol recognition subtasks (1a, 2a). * * * ## Model Competition report; architectures described at high level. Example: USTC-iFLYTEK uses attention-based encoder-decoder with RNN encoder (online) and CNN encoder (offline) plus RNN language model trained from NTCIR-12 MathIR text. ## Data ### CROHME 2019 splits (Table I) - **Formulae (Tasks 1, 2):** training = Train 2014 + Test 2013 + Test 2012 (9993 expr); validation = Test 2014 (986 expr); test = Test 2019 (1199 expr). - **Symbols (Tasks 1a, 2a):** train 180,440 symbols+junks; val 18,435; test 15,483 (Test 2016). - **Structure (Tasks 1b, 2b):** train 9993 expr; val 986; test 1147 expr (Test 2016). ### TFD dataset (Task 3) - Train: 36 rendered PDFs at 600 dpi (569 pages), 26,395 formula regions - Test: 10 PDFs (236 pages), 11,885 regions - Character boxes always provided; labels only in train ### Input specifications - Task 2 images: rendered **1000 × 1000** with **5 px padding** - Isolated symbols: **28 × 28** with 5 px padding - 2019 handwritten test: 1200 expressions, 80 writers, 3 device types; sourced from arXiv 2002-2003 documents (KDD 2003 Cup) ## Evaluation ### Tooling Online submission system (Django) with real-time leaderboard. LgEval/CROHMELib updated to support LaTeX-to-symLG and stroke-LG-to-symLG conversions. TFD uses one-to-one IoU matching (Padilla implementation). ### Results **Handwritten formula recognition (2019 test):** | Task | System | Expression Rate | | ---------------- | ------------ | --------------- | | Task 1 (strokes) | USTC-iFLYTEK | 80.73% | | Task 2 (images) | USTC-iFLYTEK | 77.15% | **TFD (IoU ≥ 0.75):** | System | F1 | Recall | Precision | | ------------- | ----- | ------ | --------- | | Samsung R&D-2 | 93.45 | 92.73 | 94.17 | | RIT 2 | 68.29 | --- | --- | | RIT 1 | 60.58 | --- | --- | **Key findings:** - Most common error: missing symbols, attributed to symLG's "absolute path" identification - Structural shifts in symLG can underestimate symbol recall - TFD gap partly due to better use of character locations (ignored by RIT teams)