# Syntax-Aware Network for Handwritten Mathematical Expression Recognition > Grammar-constrained parse tree decoding for HMER with syntax-aware attention and the HME100K dataset - Authors: Hunter Heidenreich - Published: 2022-03-03 - Updated: 2026-03-04 - Paper: SAN Paper - Dataset: HME100K Dataset - Dataset: HME100K Download Portal - Source: Bevaya Labs, --- ## TL;DR Grammar-constrained decoder for handwritten mathematical expression recognition using syntax-aware attention and stack-based tree expansion. Achieves 53-56% exact match on CROHME benchmarks and 67% accuracy on HME100K, a new 100K-image dataset with complex camera-captured handwriting. * * * ## What kind of paper is this? - **Dominant:** $\\Psi\_{\\text{Method}}$ --- Proposes grammar-driven parse tree decoding with syntax-aware attention and stack-based traversal for handwritten mathematical expression recognition. - **Secondary:** $\\Psi\_{\\text{Resource}}$ --- Introduces HME100K, a 100k-image dataset collected from approximately 10,000 writers. - **Secondary:** $\\Psi\_{\\text{Evaluation}}$ --- Defines expression-level accuracy and structure-only protocol (ESPR) with ablations on grammar and attention. ## What is the motivation? Encoder-decoder systems for handwritten mathematical expression recognition (HMER) often predict character-by-character, which produces structural errors on 2D math layouts and messy handwriting. Even tree-based decoders can behave like sequential models without explicit grammar constraints. The paper addresses this by embedding syntax constraints directly into the decoding process, predicting components and subtrees according to syntactic relationships rather than next-token likelihood. ## What is the novelty? - **Grammar formulation**: Converts LaTeX into a parse tree with constraints following reading order (left-to-right, top-to-bottom) and spatial relations between symbols. - **Tree expansion decoding**: Predicts production rules to expand non-terminals while traversing the tree with a stack-based algorithm. - **Syntax-aware attention**: Accumulates attention only along the path from root to current node, reducing drift between unrelated components. - **Attention self-regularization**: Uses a reversed decoder to predict parent nodes and regularizes forward vs. reversed attention with a KL term during training. ## What experiments were performed? The model is evaluated on CROHME 2014, 2016, 2019 (offline images rendered from InkML strokes) and HME100K (camera-captured handwriting). Metrics include ExpRate (exact expression match), relaxed ExpRate $\\leq 1$ and $\\leq 2$ (tolerating one or two symbol-level errors), and ESPR (structure correct regardless of symbol labels). Comparisons are made against prior HMER systems including DWAP-TD and BTTR. Ablations isolate the effects of grammar syntax and syntax-aware attention. ## What are the outcomes/limitations? **Outcomes:** - On CROHME 2014/2016/2019, SAN achieves the highest ExpRate among non-augmented methods and reports further gains with data augmentation. - On HME100K, SAN outperforms DWAP, DWAP-TD, and BTTR on overall accuracy and the hard subset (51.5% vs. 45.4-46.0%), with higher inference speed (23.9 FPS vs. 6.9-23.3 FPS on V100). - Ablations show grammar rules contribute the majority of improvement, with additional gains from syntax-aware attention. **Limitations:** - Distorted or overlapping components can cause under-translation or over-translation errors. * * * ## Model ### Grammar Formulation SAN defines a grammar $G = (N, \\Sigma, R, S, \\Gamma, C, D)$ with non-terminals $N$, terminals $\\Sigma$, production rules $R$, start symbol $S$, spatial relations $\\Gamma$, encoder $C$, and decoder $D$. **Spatial relations:** The system uses 7 relations (right, above, below, low right, upper left, upper right, inside) derived from 9 base relations by removing redundant directions under reading-order constraints. **Non-terminals:** $S$ (expression) and $E$ (extendable structure with relation slots). **Production rules:** 1. $S \\rightarrow \\sigma S \\mid E \\mid \\epsilon$ where $\\sigma \\in \\Sigma$ 2. $E \\rightarrow [((\gamma_1)S \mid \epsilon), \ldots, ((\gamma_7)S \mid \epsilon)]$ where $\\gamma_i \\in \\Gamma$ The grammar generates parse trees where leaves are terminals or relations and internal nodes are non-terminals. LaTeX is recovered via preorder traversal. ### Encoder A DenseNet backbone processes grayscale input $X \\in \\mathbb{R}^{1 \\times H \\times W}$ to produce a feature map of size $C \\times \\frac{H}{\\zeta} \\times \\frac{W}{\\zeta}$, flattened to $E(X) = [e_1, \ldots, e_L]$ where $e_i \\in \\mathbb{R}^{C}$ and $L = \\frac{H}{\\zeta}\\frac{W}{\\zeta}$. Implementation uses $C = 684$ and $\\zeta = 16$. ### Decoder Two GRUs with syntax-aware attention process the parse tree. For each node $\\alpha$, the context state includes: - **Historical state** $c_h^\\alpha$: how $\\alpha$ was produced - **Partner state** $c_p^\\alpha$: embedding of the latest terminal symbol or relation GRU-$\\alpha$ computes $c_o^\\alpha = \\text{GRU}(c_p^\\alpha, c_h^\\alpha)$. Attention produces compact visual feature $\\Omega = \\text{Att}(E(X), c_o^\\alpha, \\text{att}_\\alpha(X))$. GRU-$\\beta$ computes $c\_\\beta^\\alpha = \\text{GRU}(\\Omega, c_o^\\alpha)$. Two output branches: - **Symbol branch:** Softmax over $|\\Sigma| + 2$ (terminals, $E$, empty) - **Relation branch:** Sigmoid over 7 relations **Decoding logic:** 1. If terminal $\\sigma$ selected: apply $S \\rightarrow \\sigma S$ 2. If $E$ selected: apply relation branch; keep relations with probability $> 0.5$; create corresponding $(\\gamma)S$ children 3. If empty selected: apply $S \\rightarrow \\epsilon$ ### Syntax-Aware Attention Standard attention weights: $$ \\xi^\\alpha = \\text{softmax}\\left(W_w \\tanh(W_o c_o^\\alpha + W_\\alpha \\text{att}\_\\alpha(X) + W_e E(X))\\right) $$ Key modification: instead of summing over all past steps, accumulate only along the root-to-node path: $$ \\text{att}_\\alpha(X) = \\sum_{i \\in \\text{path}\_\\alpha} \\xi^i $$ This reduces attention drift between structurally unrelated components. ### Inference Stack-based traversal: 1. Encode image and initialize stack with start symbol $S$ 2. While stack non-empty: - Pop node and context - Compute production rule probabilities - Select highest-probability rule - Push produced non-terminals/relations with updated contexts - Update parse tree * * * ## Data ### CROHME (2014, 2016, 2019) The paper uses CROHME as the primary benchmark, converting InkML strokes to offline images. Training set: 8,836 expressions with 101 symbol classes. Test sizes: 986 (2014), 1,147 (2016), 1,199 (2019). ### HME100K A new dataset of 74,502 train + 24,607 test images with 245 symbol classes, collected from approximately 10,000 writers via camera-captured uploads. Characteristics include color variation, blur, complex backgrounds, perspective distortion, and illumination issues. Statistics: - Max sequence length: 184 (vs. 96 for CROHME 2019) - Average sequence length: 17.62 (vs. 15.79 for CROHME 2019) - Writer count: ~10,000 (vs. ~100 for CROHME 2019) **Dataset availability:** Download links provided via GitHub repository and official portal. No explicit license specified; verify terms before use. * * * ## Training ### Objective Multi-task loss combining symbol prediction, relation prediction, reversed symbol prediction, and attention regularization: $$ L = L\_{\\text{symbol}} + L\_{\\text{relation}} + L\_{\\text{rev. symbol}} + L\_{\\text{reg}} $$ Attention regularization uses a reversed decoder to predict parent nodes and enforces consistency: $$ L_{\\text{reg}} = -\\sum_\\beta \\hat{\\xi}^\\eta \\log \\frac{\\hat{\\xi}^\\eta}{\\xi^\\alpha} $$ The reversed decoder is removed at inference. ### Supervision Ground-truth LaTeX is parsed into a parse tree via depth-first search, generating parent-child training samples processed in preorder with teacher forcing. ### Implementation - Framework: PyTorch - Hardware: Single NVIDIA Tesla V100 (32GB) - Batch size: 8 - GRU hidden size: 256 - Word/relation embedding dimension: 256 - Optimizer: Adadelta ($\\rho = 0.95$, $\\epsilon = 10^{-6}$) - Learning rate: Linear warmup from 0 to 1 over first epoch, then cosine decay to 0 * * * ## Results ### CROHME Benchmarks Without data augmentation: | Dataset | ExpRate | ExpRate $\\leq 1$ | ExpRate $\\leq 2$ | | ----------- | ------- | ----------------- | ----------------- | | CROHME 2014 | 56.2 | 72.6 | 79.2 | | CROHME 2016 | 53.6 | 69.6 | 76.8 | | CROHME 2019 | 53.5 | 69.3 | 70.1 | A variant trained with data augmentation achieves higher scores across all benchmarks. ### HME100K | Model | Total Acc. | Hard Subset | FPS (V100) | Params | | ------- | ---------- | ----------- | ---------- | ------ | | DWAP | 61.9 | 45.4 | 23.3 | --- | | DWAP-TD | 62.6 | 45.4 | 6.9 | --- | | BTTR | 64.1 | 46.0 | 3.9 | --- | | **SAN** | **67.1** | **51.5** | **23.9** | 8.9M | ### Ablations ExpRate improvements on CROHME 2019: - Baseline → SAN-GS (with grammar): significant gain - SAN-GS → SAN (with syntax-aware attention): additional improvement Grammar syntax contributes the majority of gains, with syntax-aware attention providing further refinement.