# CDM (Character Detection Matching): Notes > Image-level evaluation metric for formula recognition that renders predicted and ground-truth LaTeX into images and matches per-character bounding boxes - Authors: Hunter Heidenreich - Published: 2024-09-01 - Updated: 2026-03-04 - Paper: CDM Paper (CVPR) - Code: CDM Code - Source: Bevaya Labs, --- ## TL;DR CDM is an **image-level** evaluation metric for formula recognition that renders predicted and ground-truth LaTeX into images and matches **per-character bounding boxes** instead of comparing LaTeX strings. CDM reduces unfairness from non-unique LaTeX representations and aligns better with human judgment than BLEU, Edit Distance, and ExpRate. * * * ## What kind of paper is this? **Dominant vector:** $\\Psi\_{\\text{Evaluation}}$ Headline contribution is a new evaluation metric and protocol for formula recognition, motivated by reliability and fairness issues in existing text-based metrics. **Secondary vectors:** $\\Psi\_{\\text{Method}}$ (matching algorithm design) CDM includes a concrete multi-stage algorithm: localization, Hungarian matching, invalid-match elimination, and scoring. ## What is the motivation? **Non-unique LaTeX representations:** Text-based metrics (BLEU/Edit Distance/ExpRate) misrepresent correctness because visually identical formulas can score poorly if written differently. **Unfair model comparisons:** Metrics favor outputs closer to a dataset's annotation style even when the prediction is objectively worse, particularly under distribution or style mismatch. **Human-perception mismatch:** Predictions with obvious visual errors can receive high BLEU scores, while visually correct predictions may score poorly due to stylistic differences. ## What is the novelty? **Render-to-image evaluation:** CDM evaluates formula recognition in image space, not LaTeX space, by rendering both predicted and ground-truth LaTeX and performing character-level matching with spatial awareness. **Character detection framing:** Each token/character is treated as an "object" with a bounding box; a match-based score analogous to detection evaluation uses an F1-style metric. **Robust matching pipeline:** Bipartite matching (Hungarian algorithm) plus post-filters (token consistency and geometric consistency via RANSAC with constrained affine transform) avoid cascading mismatch from local errors or layout differences. ## What experiments were performed? **Formula-level evaluation** on **UniMER-Test** (23,757 formulas) comparing Mathpix (API), UniMERNet, Texify, and Pix2Tex under BLEU, ExpRate, CDM, and ExpRate@CDM. **Human preference study** (1,008 samples) comparing whether CDM vs BLEU better reflects quality, with randomized score ordering in the UI. **Style-stability test:** 50 formulas rewritten 5 times using GPT-4 (250 variants), manually verified to render identically; compared score sensitivity of BLEU vs CDM. **Document-level evaluation** on **Tiny-Doc-Math** (12 PDFs, 196 pages, 437 formulas; post-June 2024 papers) using Nougat, GPT-4o, Mathpix, plus formula-level cropped inputs for GPT-4o/UniMERNet/Mathpix/Pix2Tex. ## What are the outcomes/limitations? **Outcomes** **Human preference:** 64% preferred CDM; 32% said both are good; 3% preferred BLEU; 1% neither. Authors interpret this as 96% consistency with human evaluation. **Style robustness:** CDM scores remain 1.0 under equivalent formula rewrites, while BLEU varies widely across writing styles. **Model comparison differences:** Subset analyses highlight cases where BLEU and CDM yield opposite conclusions due to annotation-style bias, notably in the SCE (Screenshot Expressions) subset. **Tiny-Doc-Math:** GPT-4o achieves the highest BLEU among cropped-formula inputs but the lowest CDM, suggesting BLEU may overstate formula recognition quality for some models. **Limitations** **Rendering dependency:** CDM requires successful LaTeX rendering; rendering failures are assigned CDM = 0, coupling evaluation to renderer robustness and LaTeX validity. **Token-synonym handling is partial:** CDM uses a low token mismatch cost (0.05) for some differently written but identically rendered tokens (e.g., `"("`, `"\left("`, `"\big("`), but coverage of the full LaTeX synonym space is not guaranteed. **Scaling to very long formulas:** Element localization uses a finite palette of 5,832 distinct colors; behavior when formulas exceed that many tokens is not described. **Geometric assumptions:** Positional consistency uses a constrained affine transform with rotation fixed to 0 (translation + scaling), which matches typical rendering but is an explicit assumption. * * * ## Model ### What CDM is (conceptually) CDM reframes formula recognition evaluation as **matching sets of detected character regions** between two rendered images (prediction vs ground truth), instead of comparing LaTeX strings. ### Outputs - **CDM score:** F1-style match score in $[0,1]$ - **ExpRate@CDM:** fraction of samples with perfect match (CDM = 1) * * * ## Data ### UniMER-Test **Size:** 23,757 formula samples **Subsets:** - SPE: Simple Printed Expressions - CPE: Complex Printed Expressions - SCE: Screenshot Expressions - HWE: Handwritten Expressions ### Tiny-Doc-Math **Construction:** arXiv math/CS papers published after June 2024 (intended to reduce training contamination); LaTeX and PDFs collected; displayed equations matched via regex and manually verified. **Size:** 12 PDFs, 196 pages, 437 formulas * * * ## Algorithms / Training ### CDM pipeline CDM has four stages: **(1) element localization**, **(2) element region matching**, **(3) invalid match elimination**, **(4) metric calculation**. #### 1) Element localization **LaTeX source normalization:** Tokenize both ground truth and prediction into tokens such as `"2"`, `"a"`, `"A"`, `"\alpha"`, `"\sin"`. Composite constructs are decomposed (e.g., `\frac ab` rewritten as `\frac {a} {b}`). **Element region localization via unique colors:** - Render each token in a unique RGB color using `\mathcolor[RGB]{r,g,b}` - Construct color list with interval 15 from (0,0,15) to (255,255,255), yielding $(255/15 + 1)^3 = 5832$ distinct colors - After rendering, extract pixels of each color to locate the token's bounding box #### 2) Element region matching (Hungarian assignment) Let $y$ be GT elements, $\\hat{y}$ predicted elements; sizes $N_y$, $N_{\\hat{y}}$, with $N=\\min(N_y,N_{\\hat{y}})$. Find a permutation $\\hat{\\omega}$ minimizing total cost: $$ \\hat{\\omega}=\\arg\\min_{\\omega\\in S_N}\\sum_{i=1}^{N} L_{\\text{match}}(y_i,\\hat{y}_{\\omega(i)}) $$ Matching cost is a weighted sum of: - **Token matching cost $L_t$:** 0 if tokens identical; 1 if different; 0.05 if different tokens render identically - **Positional proximity cost $L_p$:** $L_1$ distance between bbox coordinates (normalized by bbox coordinate dimension) - **Order similarity cost $L_o$:** $L_1$ distance between normalized token-order indices (approximate reading order from LaTeX source) Weights $W_t,W_p,W_o$ are defined but specific numeric values are not enumerated in the main text. #### 3) Invalid match elimination **Token consistency check:** Discard matched pairs whose characters are inconsistent. **Position relationship consistency check:** - Assume predicted bboxes follow an affine transform of GT bboxes: $\\hat{b}\_{\\omega(i)} = A(b_i)$ - Use **RANSAC** to estimate $A$ and remove outliers; rotation is fixed to 0 (translation + scaling only) to speed convergence and match typical rendering structure - Run multiple rounds to handle line breaks (multi-line layouts) #### 4) Metric calculation Define: - **TP:** matched bbox pairs after elimination - **FP:** unmatched predicted bboxes - **FN:** unmatched GT bboxes **CDM score (F1):** $$ \\text{CDM}=\\frac{2 \\cdot TP}{2 \\cdot TP+FP+FN} $$ **ExpRate@CDM:** proportion of samples with CDM = 1 * * * ## Evaluation ### Baseline metrics discussed - **BLEU:** $n$-gram overlap with brevity penalty - **Edit Distance:** insertion/deletion/substitution distance - **ExpRate:** exact string match rate (noted as coarse/strict and unreliable under LaTeX non-uniqueness) LaTeX "regularization" helps some syntax variations but cannot cover full symbol synonymy (e.g., `\leq` vs `\le`). ### Rendering success rate (CDM applicability) If rendering fails, CDM is set to 0. Reported render success on UniMER-Test: Pix2Tex 96.63%, Texify 94.77%, UniMERNet 99.71%, Mathpix 97.82%. ### Main results (UniMER-Test) | Model | ExpRate | ExpRate@CDM | BLEU | CDM | | --------- | ------- | ----------- | ------ | ------ | | Pix2Tex | 0.1237 | 0.2910 | 0.4080 | 0.6360 | | Texify | 0.2288 | 0.4950 | 0.5890 | 0.7550 | | Mathpix | 0.2610 | 0.5000 | 0.8067 | 0.9510 | | UniMERNet | 0.4799 | 0.8110 | 0.8425 | 0.9680 | ### Subset anomalies **SCE subset:** BLEU and CDM can disagree for Mathpix vs UniMERNet. SCE annotations were based on Mathpix outputs then manually corrected, biasing LaTeX style toward Mathpix and inflating BLEU alignment. **Pix2Tex:** Shows large BLEU drops on HWE/SCE but strong performance on SPE/CPE, attributed to training data skew toward printed arXiv formulas and lack of handwritten/screenshot styles. ### Tiny-Doc-Math results **Formula-level (cropped formula inputs):** | Model | BLEU | CDM | ExpRate@CDM | | --------- | ------ | ------ | ----------- | | Pix2Tex | 0.4648 | 0.7444 | 0.3684 | | GPT-4o | 0.6431 | 0.7330 | 0.4324 | | UniMERNet | 0.6056 | 0.9396 | 0.6887 | | Mathpix | 0.6112 | 0.9480 | 0.2105 | **Document-level (page screenshots / full-document outputs):** | Model | BLEU | CDM | ExpRate@CDM | | ------- | ------ | ------ | ----------- | | GPT-4o | 0.3411 | 0.6502 | 0.1670 | | Nougat | 0.5897 | 0.8326 | 0.6086 | | Mathpix | 0.5939 | 0.9567 | 0.6292 | Manual check note: Mathpix often misses trailing commas/periods, which impacts exact-match rate (ExpRate@CDM) even when CDM remains high. ### Human preference protocol 1,008 samples from Pix2Tex with balanced score distribution; annotators saw GT + predicted render and chose which score (BLEU vs CDM, randomized order) better matched perceived quality. ### Style sensitivity test 50 formulas rewritten 5 times by GPT-4 (250 variants), manually verified identical render; CDM remained 1 for all style variants while BLEU varied. * * * ## Hardware / Production GPU/throughput requirements for running CDM are not reported. Operational cost is primarily driven by (1) LaTeX rendering and (2) per-formula matching (Hungarian assignment + RANSAC iterations).