# UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition

> 1M training samples and architectural improvements for real-world printed, screen-captured, and handwritten mathematical expression recognition

- Authors: Hunter Heidenreich
- Published: 2024-04-23
- Updated: 2026-03-04
- Code: UniMERNet Code <https://github.com/opendatalab/UniMERNet>
- Model: UniMERNet Base Model <https://huggingface.co/wanderkid/unimernet_base>
- Model: UniMERNet Small Model <https://huggingface.co/wanderkid/unimernet_small>
- Model: UniMERNet Tiny Model <https://huggingface.co/wanderkid/unimernet_tiny>
- Dataset: UniMER Dataset <https://huggingface.co/datasets/wanderkid/UniMER_Dataset>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

UniMERNet introduces UniMER-1M (1,061,791 image-LaTeX pairs) and UniMER-Test (23,757 samples across four subsets) to address real-world mathematical expression recognition beyond clean rendered formulas. A Swin-Transformer + mBART encoder-decoder architecture with fine-grained embedding, local convolution modules, and decoder attention compression achieves stronger BLEU scores and throughput than Pix2tex and Texify baselines on complex printed, screen-captured, and handwritten expressions.

* * *

## What kind of paper is this?

-   **Dominant:** $\\Psi\_{\\text{Resource}}$ --- Releases UniMER-1M (1M training pairs), UniMER-Test benchmark (four subsets covering diverse real-world conditions), and model weights.
-   **Secondary:** $\\Psi\_{\\text{Method}}$ --- Proposes UniMERNet architecture modifications (fine-grained embedding, convolutional enhancement, decoder attention compression) for formula-specific recognition.
-   **Secondary:** $\\Psi\_{\\text{Evaluation}}$ --- Provides comprehensive baseline comparisons (Pix2tex, Texify) and ablations across architecture, training data, pretraining, and augmentation.

## What is the motivation?

Prior mathematical expression recognition (MER) benchmarks emphasize simple printed or handwritten formulas with limited complexity and clean backgrounds. Real-world deployments encounter long expressions, noisy screen captures, font inconsistencies, geometric distortions, and mixed capture modalities. Existing models trained on small clean datasets fail to generalize to these conditions. The paper addresses this by constructing a large-scale dataset reflecting practical distributions and an architecture tuned for formula-specific challenges such as local spatial relations and similar-looking symbols.

## What is the novelty?

**Dataset:**

-   **UniMER-1M:** 1,061,791 training pairs; maximum formula length 7,037 tokens; average length 79.48 tokens. Sourced from arXiv (89%), Wikipedia (9%), StackExchange (2%); rendered with multiple math fonts at 80-350 DPI.
-   **UniMER-Test:** 23,757 samples across four subsets:
    -   **SPE** (short printed expressions): rendered LaTeX, clean backgrounds
    -   **CPE** (complex printed expressions): long/complex rendered LaTeX, clean backgrounds
    -   **SCE** (screen-captured expressions): extracted from 1,000 PDF pages, annotated via Mathpix + manual correction
    -   **HWE** (handwritten expressions): CROHME + HME100K combined into 6,332 test samples

**Model:** UniMERNet modifies a Swin + mBART baseline with:

-   **FGE (Fine-Grained Embedding):** Overlapping 3×3 convolution stack (stride 2, padding 1) for patch embedding to reduce character fragmentation from non-overlapping patches.
-   **CE (ConvEnhance):** Depthwise 3×3 convolution + GELU before attention/MLP blocks for local perception of superscripts, subscripts, and adjacent symbols.
-   **RSW (Remove Shift Window):** Eliminates Swin shift-window mechanism when FGE+CE already enlarge receptive field, improving speed and accuracy.
-   **SA (Squeeze Attention):** Low-dimensional projection of query/key tensors in decoder attention to improve throughput without significant accuracy loss.

## What experiments were performed?

**Data scaling (Table 2):** Compared Pix2tex-only, Pix2tex+HWE, and UniMER-1M training on UniMER-Test subsets, measuring BLEU and edit distance.

**Architecture ablations (Table 3):** Evaluated baseline vs. incremental addition of FGE, CE, SA, and RSW, measuring BLEU by subset and throughput (images/s at batch size 128, max sequence length 1536 on A100).

**SOTA comparisons (Table 5):** Compared Pix2tex, Texify, Texify\* (Texify trained on UniMER-1M with their augmentations), and UniMERNet-T/S/B variants, reporting BLEU, edit distance, FPS, and parameter counts.

**Pretraining ablation (Table 6):** Tested with/without 16M arXiv-derived image-text pretraining corpus for Texify and UniMERNet-B.

**Augmentation ablation (Table 7):** Evaluated with/without Albumentations + custom augmentations (erosion, dilation, degradation, geometric transforms) for Pix2tex and UniMER-1M training.

**Depth sweeps (Tables 8-9):** Varied encoder depth per Swin stage and decoder depth, tracking BLEU and FPS.

## What are the outcomes/limitations?

**Outcomes:**

-   **UniMER-1M training improves generalization** on complex and real-world subsets:
    -   CPE BLEU: 0.724 (Pix2tex+HWE) → **0.925** (UniMER-1M); edit distance: 0.225 → **0.056**
    -   SCE BLEU: 0.529 (Pix2tex+HWE) → **0.626** (UniMER-1M); edit distance: 0.309 → **0.224**
-   **UniMERNet-B** (325M parameters) achieves the strongest reported results:
    -   SPE BLEU **0.915**, CPE BLEU **0.925**, SCE BLEU **0.626**, HWE BLEU **0.895**
    -   Throughput: **5.06 img/s** (batch 128, max seq len 1536, A100)
-   **SA (Squeeze Attention)** is the primary speed improvement: 4.07 img/s → 5.04 img/s with minimal BLEU change.
-   **Pretraining** improves printed/screen-captured more than handwritten: SCE BLEU 0.601 → **0.626** with 16M arXiv pretraining.

**Limitations:**

-   **Inconsistent counts:** UniMER-Test size described as 23,789 in text but Table 1 lists 23,757; CROHME test described as 3,332 but Table 1 lists 3,233.
-   **Sequence length mismatch:** Dataset reports max formula length 7,037 tokens while training/evaluation uses max sequence length 1,536; handling of longer sequences (truncation vs. filtering) is not specified.
-   **SCE labels via Mathpix:** Requires proprietary API for initial annotation followed by manual correction; reproducibility implications and licensing considerations are not discussed.
-   **Pretraining data availability:** 16M arXiv-derived corpus is described as "in-house"; public release status is not clarified.

* * *

## Model

### Architecture

UniMERNet follows the encoder-decoder paradigm used in Donut, Nougat, and Texify:

-   **Encoder:** Swin Transformer with modifications
-   **Decoder:** mBART

### Encoder Modifications

**Fine-Grained Embedding (FGE):**

-   Two convolutional layers, kernel size 3, stride 2, padding 1
-   Each followed by LayerNorm + GELU
-   Produces overlapping patches to reduce character feature fragmentation

**ConvEnhance (CE):**

-   Depthwise 3×3 convolution + GELU inserted before attention and MLP modules
-   Provides local perception for spatial relations (superscripts, subscripts)
-   Alternates local (conv) and global (attention) feature aggregation

**Remove Shift Window (RSW):**

-   Eliminates Swin shift-window mechanism
-   FGE and CE already expand receptive field sufficiently
-   Improves both speed and accuracy in ablation studies

### Decoder Modifications

**Squeeze Attention (SA):**

-   Projects query $Q$ and key $K$ to lower-dimensional space before attention computation
-   Reduces computational cost in decoder attention, which becomes the throughput bottleneck
-   Value $V$ remains full-dimensional

### Model Variants

| Variant     | Encoder Depth $N$ | Decoder Depth $M$ | Hidden Dim $C$ | Parameters |
| ----------- | ----------------: | ----------------: | -------------: | ---------: |
| UniMERNet-T |         [6,6,6,6] |                 8 |            512 |       100M |
| UniMERNet-S |         [6,6,6,6] |                 8 |            768 |       202M |
| UniMERNet-B |         [6,6,6,6] |                 8 |           1024 |       325M |

* * *

## Data

### UniMER-1M Composition

-   **Total:** 1,061,791 LaTeX-image pairs
-   **Sources:** arXiv (89%), Wikipedia (9%), StackExchange (2%); approximately 4M LaTeX expressions collected
-   **Length distribution:**
    -   Maximum formula length: 7,037 tokens
    -   Average formula length: 79.48 tokens
    -   Initial long-formula proportion: 2.3%; rebalanced by extracting longest formulas as CPE subset

**Rendering:**

-   XeLaTeX with multiple math fonts: Asana Math, Cambria Math, XITS Math, TeX Gyre (Bonum, Pagella, Schola, Termes), Latin Modern Math
-   DPI range: 80-350
-   Latin Modern Math used in ~22% of samples (default)

**Normalization:**

-   Based on Deng et al. (2017) LaTeX normalization
-   Adjusted for multi-line environments (`align`, `cases`, etc.)
-   Filters unsupported syntax

### UniMER-Test Subsets

| Subset | Type            |   Size | Description                                                          |
| ------ | --------------- | -----: | -------------------------------------------------------------------- |
| SPE    | Short printed   |  8,313 | Rendered LaTeX, clean backgrounds, short expressions                 |
| CPE    | Complex printed |  5,368 | Rendered LaTeX, clean backgrounds, long/complex expressions          |
| SCE    | Screen-captured |  4,744 | Extracted from 1,000 PDFs, Mathpix + manual annotation, deduplicated |
| HWE    | Handwritten     |  6,332 | CROHME + HME100K combined test set                                   |
| Total  |                 | 23,757 |                                                                      |

**SCE construction:**

-   1,000 PDF pages (Chinese + English)
-   Formula detection and bounding box annotation
-   Initial labels from Mathpix API
-   Manual correction pass
-   Perceptual hashing deduplication

**HWE construction:**

-   Training: 83,338 samples from CROHME + HME100K
-   Test: 6,332 samples (note: text describes CROHME test as 3,332, but Table 1 lists 3,233)

* * *

## Training

**Framework:** PyTorch

**Maximum sequence length:** 1,536 tokens

**Optimization:**

-   Loss: Cross-entropy language modeling
-   Learning rate schedule: Linear warmup + cosine decay
    -   Initial LR: $1 \\times 10^{-4}$
    -   Minimum LR: $1 \\times 10^{-8}$
    -   Warmup LR: $1 \\times 10^{-5}$
-   Weight decay: 0.05
-   Total iterations: 300,000

**Augmentation:**

-   Albumentations library + custom transforms
-   Morphological: erosion, dilation
-   Degradation: fog, frost, rain, snow, shadow simulation
-   Geometric: rotation, distortion

**Pretraining (optional):**

-   16M image-text pairs from arXiv
-   Constructed via layout detection + OCR on text blocks, matched to source LaTeX
-   Pretraining improves printed/screen-captured performance more than handwritten

**Hardware:**

-   NVIDIA A100 80GB
-   Batch size: 64
-   Training setup description includes both "single GPU" and "eight such GPUs" references (apparent inconsistency in source material)

* * *

## Evaluation

### Metrics

-   **BLEU:** Measures token-level similarity between predicted and ground-truth LaTeX
-   **Edit Distance (Levenshtein):** Character-level edit operations normalized by ground-truth length
-   **ExpRate:** Exact match rate (expression-level accuracy)

### Key Results

**Data Scaling (Table 2)**

Training on UniMER-1M vs. Pix2tex+HWE, evaluated on UniMER-Test:

| Subset | Metric        | Pix2tex+HWE | UniMER-1M |
| ------ | ------------- | ----------: | --------: |
| SPE    | BLEU          |       0.893 | **0.902** |
| SPE    | Edit Distance |       0.069 | **0.065** |
| CPE    | BLEU          |       0.724 | **0.925** |
| CPE    | Edit Distance |       0.225 | **0.056** |
| SCE    | BLEU          |       0.529 | **0.626** |
| SCE    | Edit Distance |       0.309 | **0.224** |
| HWE    | BLEU          |       0.874 | **0.882** |
| HWE    | Edit Distance |       0.088 | **0.082** |

**Architecture Ablation (Table 3)**

Impact of UniMERNet modifications on SCE subset (batch 128, A100):

| Configuration  | Params |  FPS |  SCE BLEU |
| -------------- | -----: | ---: | --------: |
| Baseline       |   342M | 4.12 |     0.579 |
| +FGE+CE        |   342M | 4.07 |     0.599 |
| +FGE+CE+SA     |   325M | 5.04 |     0.599 |
| +FGE+CE+RSW+SA |   325M | 5.06 | **0.601** |

**Model Comparison (Table 5)**

UniMERNet-B vs. baselines on UniMER-Test:

| Model       | Params |  FPS |  SPE BLEU |  CPE BLEU |  SCE BLEU |  HWE BLEU |
| ----------- | -----: | ---: | --------: | --------: | --------: | --------: |
| Pix2tex     |   350M | 7.67 |     0.750 |     0.575 |     0.378 |     0.806 |
| Texify      |   312M | 4.16 |     0.820 |     0.764 |     0.420 |     0.844 |
| Texify\*    |   312M | 4.16 |     0.901 |     0.916 |     0.599 |     0.884 |
| UniMERNet-T |   100M | 7.67 |     0.857 |     0.846 |     0.535 |     0.858 |
| UniMERNet-S |   202M | 6.08 |     0.896 |     0.906 |     0.598 |     0.888 |
| UniMERNet-B |   325M | 5.06 | **0.915** | **0.925** | **0.626** | **0.895** |

_Texify\* = Texify trained on UniMER-1M with their augmentation pipeline_

**Pretraining Impact (Table 6)**

UniMERNet-B with/without 16M arXiv pretraining:

| Subset | Without Pretraining | With Pretraining | Δ BLEU |
| ------ | ------------------: | ---------------: | -----: |
| SPE    |               0.904 |        **0.915** | +0.011 |
| CPE    |               0.913 |        **0.925** | +0.012 |
| SCE    |               0.601 |        **0.626** | +0.025 |
| HWE    |               0.895 |        **0.895** |  0.000 |

* * *

## Hardware

**Training:**

-   NVIDIA A100 80GB
-   Batch size: 64 per GPU
-   Setup: references both "single GPU" and "eight such GPUs" (inconsistent in source)

**Inference throughput tests:**

-   Batch size: 128
-   Max sequence length: 1536
-   Hardware: A100 80GB
-   UniMERNet-B: 5.06 img/s

* * *

## License & Availability

**Code:** Apache-2.0 ([GitHub](https://github.com/opendatalab/UniMERNet))

**Models:** Apache-2.0 ([Hugging Face](https://huggingface.co/wanderkid/unimernet_base))

**Dataset:** Apache-2.0 ([Hugging Face](https://huggingface.co/datasets/wanderkid/UniMER_Dataset), [OpenDataLab](https://opendatalab.com/OpenDataLab/UniMER-1M))

**Important caveat:** The dataset card notes that HME100K portion must be downloaded manually "for copyright compliance," and the dataset incorporates multiple upstream sources (Pix2tex, CROHME, HME100K). The Apache-2.0 tag on the Hugging Face dataset repository may not override upstream licensing terms for these components. Verify upstream licenses before commercial use.