# OmniParser V1: A Unified Framework for Text Spotting, Key Information Extraction, and Table Recognition

> OmniParser V1 unifies three visually-situated text parsing tasks under a single encoder-decoder using point-conditioned text generation, where structured center-point sequences act as bridges between task-specific structural tokens and polygon-plus-content decoding.

- Authors: Hunter Heidenreich
- Published: 2024-03-28
- Updated: 2026-03-26
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2403.19128>
- Paper: Published (CVPR 2024) <https://openaccess.thecvf.com/content/CVPR2024/html/Wan_OmniParser_A_Unified_Framework_for_Text_Spotting_Key_Information_Extraction_CVPR_2024_paper.html>
- Code: OmniParser V1 Code <https://github.com/AlibabaResearch/AdvancedLiterateMachinery>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

OmniParser V1 proposes a single encoder-decoder model for visually-situated text parsing (VsTP) that handles text spotting, key information extraction (KIE), and table recognition under one objective: point-conditioned text generation. A Structured Points Decoder first autoregressively produces center-point coordinates interleaved with task-specific structural tokens; a Region Decoder and a Content Decoder then predict polygon contours and character-level transcripts in parallel from those points. With two pre-training strategies (spatial-window prompting and prefix-window prompting), the model achieves competitive or leading results on 7 benchmarks spanning all three tasks.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$: The core contribution is a new architecture and training paradigm for multi-task visually-situated text parsing. The unified representation (structured points interleaved with task tokens), the three-decoder design, and the two pre-training strategies together constitute the methodological headline. Ablation tables and SOTA-style comparison tables across all three tasks form the bulk of the experimental content.

No strong secondary vector. The multi-task comparison tables serve the methodological argument rather than constituting an independent contribution to evaluation methodology.

## What is the motivation?

Visually-situated text parsing covers several tightly coupled subtasks: detecting and recognizing text instances in natural images (text spotting), linking detected text to structured named fields (KIE), and recovering both the logical structure and cell content of tabular layouts (table recognition). Despite their shared visual and textual substrate, prior work has almost universally addressed them with task-specific architectures, objectives, and post-processing pipelines.

This fragmentation causes two problems. First, specialist models trained for a single subtask cannot share representations or transfer knowledge across tasks, leading to duplicate engineering effort and modal isolation. Second, generalist models (typically large multimodal LLMs) achieve breadth at the cost of spatial precision and interpretability, and their performance degrades when an external OCR engine is unavailable.

The paper argues that a single, lightweight model with a shared representation and a unified generation objective can match or exceed specialist performance on all three tasks simultaneously.

## What is the novelty?

**Unified Input/Output Representation.** OmniParser represents the output of all three tasks as a structured sequence composed of three sub-sequences:

1. **Structured Points Sequence:** Discrete coordinate tokens encoding the center point of each text instance, interleaved with task-specific structural tokens (e.g., `<address>` for KIE entities, `<tr>` for table rows). Text spotting is treated as the special case with no structural tokens. Coordinates are normalized to image dimensions, then quantized to integers in $[0, n\_{\text{bins}} - 1]$.

2. **Polygon Sequence:** A 16-point polygon per text instance, tokenized with the same quantization scheme, representing the polygonal contour.

3. **Content Sequence:** Character-level tokens encoding the text transcript.

Points act as the bridging primitive: the Structured Points Decoder first generates the entire points sequence, and the Region and Content Decoders then generate polygons and transcripts in parallel, conditioned on each point.

**Three-Decoder Architecture.** The model uses a shared Swin-B encoder with FPN for multi-scale visual feature extraction. Three independent transformer decoders (same architecture, separate parameters) handle structure-point generation, polygon prediction, and content transcription. Each decoder has 4 layers, 8 attention heads, and hidden dimension 512. The independence of decoder weights, rather than weight sharing, is shown by ablation to be important: shared decoder weights degrade performance, suggesting the sub-tasks require different inductive biases.

**Training Objective.** All tasks share the same negative log-likelihood loss:

$$
L = -\sum\_{j=k}^{N} w\_j \log P(\bar{s}\_j \mid \mathbf{v}, s\_{k:j-1})
$$

where $\bar{s}$ is the target sequence, $k$ is the number of prompt tokens excluded from the loss, and $w\_j$ is a per-token weight: 4.0 for structural or entity tags, 1.0 for all other tokens.

**Spatial-Window Prompting.** During pre-training, the Structured Points Decoder is given a bounding box prompt $(X\_{\text{left}}, Y\_{\text{top}}, X\_{\text{right}}, Y\_{\text{bottom}})$ and tasked with predicting only the center points whose locations fall inside that window. Windows are sampled from fixed grid layouts (e.g., $3 \times 3$, $2 \times 2$) or randomly from the image (covering at least 1/9 of the area). This mechanism allows the decoder to handle images with many text instances despite a finite sequence length, and trains robust spatial coordinate perception.

**Prefix-Window Prompting.** The Structured Points Decoder is also pre-trained to predict center points only for text instances whose single-character prefix falls within a designated character range. The range is defined by start and end characters drawn from an ordered dictionary (26 uppercase, 26 lowercase, 10 digits, 34 ASCII punctuation). This strategy trains character-level semantic grounding, which the authors find particularly useful for KIE where entity types are associated with characteristic vocabulary.

Ablation shows that each prompting strategy provides an independent performance gain, and combining both yields the best results on both Total-Text and ICDAR 2015.

## What experiments were performed?

Experiments cover three tasks across five evaluation benchmarks, plus two ablation studies.

**Text Spotting.** Evaluated on Total-Text (1,255 train / 300 test, word-level polygon annotations for arbitrary-shaped text), CTW1500 (1,000 train / 500 test, line-level curved text), and ICDAR 2015 (1,000 train / 500 test, quadrilateral annotations for incidental text). End-to-end recognition metrics are the primary target: F1 under None and Full lexicons for Total-Text and CTW1500, and F1 under Strong, Weak, and Generic lexicons for ICDAR 2015. Baselines include TextDragon, ABCNet v2, TESTR, DeepSolo, UNITS, and others.

**Key Information Extraction.** Evaluated on CORD (1,000 receipts, 30 entity labels across 4 categories; 800/100/100 train/val/test split) and SROIE (626 train / 347 test receipts, 4 entity types: company, date, address, total). Metrics: field-level F1 and tree-edit-distance-based accuracy. Baselines: Donut, Dessurt, DocParser, SeRum.

**Table Recognition.** Evaluated on PubTabNet (500,777 train / 9,115 val, scientific document tables; validation set used due to missing public test annotations) and FinTabNet (92,000 train / 10,656 test, financial document tables). Metric: Tree-Edit-Distance-based Similarity (TEDS) and structure-only TEDS (S-TEDS). Baselines: WYGIWYS, EDD, and Donut (fine-tuned by the authors using the official training configuration). The comparison is correctly restricted to end-to-end methods that predict both table structure and cell content jointly, excluding non-end-to-end TSR methods.

**Ablations.**

- **Pre-training strategies (Table 5):** Removing spatial-window prompting, prefix-window prompting, or both shows additive degradation on Total-Text and ICDAR 2015 end-to-end metrics.
- **Encoder and decoder design (Table 6):** Swin-B outperforms ResNet-50, and three independent decoders outperform a shared-weight decoder.
- **Decoder length for table recognition (Table 7):** Performance plateaus for the Structured Points Decoder at 1,500 tokens; extending the Content Decoder length from 200 to 300 tokens yields small TEDS improvement. Inference speed (1.3 FPS) is faster than Donut (0.8 FPS) at a shorter maximum sequence length.

**Pre-training details.** Two-stage pre-training on a hybrid corpus (Curved SynthText, ICDAR 2013, ICDAR 2015, MLT 2017, Total-Text, TextOCR, HierText, COCO Text, Open Image V5). Stage 1: batch 128, resolution $768 \times 768$, 500k steps, AdamW with lr $5 \times 10^{-4}$, 5k-step linear warm-up then linear decay. Stage 2: batch 16, resolution $1920 \times 1920$, 200k additional steps, lr $2.5 \times 10^{-4}$.

## What are the outcomes/conclusions?

**Text Spotting.** On Total-Text and CTW1500, OmniParser V1 achieves 84.0% E2E (None) on Total-Text and 66.8% E2E (None) on CTW1500, the highest end-to-end scores reported on those benchmarks at the time of publication (+1.5% and +3.2% above the prior best, respectively). On ICDAR 2015, results are competitive with the then-leading methods across all three lexicon settings.

**Key Information Extraction.** On CORD, the model achieves 84.8% field-level F1, outperforming all compared generation-based methods. On SROIE, it matches the prior best in F1 (85.6%) and achieves the best TED-based accuracy (93.6%). Importantly, OmniParser V1 also provides explicit localization of extracted entities, unlike OCR-free generation methods such as Donut, and is pre-trained on scene text data only, not large document corpora.

**Table Recognition.** OmniParser V1 achieves 90.45 S-TEDS / 88.83 TEDS on PubTabNet and 91.55 S-TEDS / 89.75 TEDS on FinTabNet, surpassing EDD and the authors' Donut reproduction on both. The modularized architecture separates structural HTML tags from cell text, avoiding the attention drift and error accumulation that limits monolithic sequence-to-sequence approaches on long sequences.

**Limitations acknowledged by the authors:**

- The model requires precise word-level point location annotations during training, which may not be available in all real-world scenarios.
- It does not handle non-text elements (figures, charts, diagrams), limiting applicability to complex document parsing tasks.
- Future work is intended to extend the model to layout analysis and chart parsing.

**Unacknowledged limitations:**

- No error bars, seeds, or significance tests are reported; all results are single-run.
- The table recognition comparison set is small (only WYGIWYS, EDD, and Donut), reflecting the limited availability of end-to-end baselines at the time but making it difficult to assess the model's position in a broader context.
- Inference speed (1.3 FPS at 1,024 resolution for table recognition) may be insufficient for latency-sensitive document processing pipelines.

## Reproducibility

| Resource | Type | License | Link |
|---|---|---|---|
| Preprint | Paper | CC-BY-4.0 | [arXiv 2403.19128](https://arxiv.org/abs/2403.19128) |
| Published paper | Paper | Unknown | [CVPR 2024 open access](https://openaccess.thecvf.com/content/CVPR2024/html/Wan_OmniParser_A_Unified_Framework_for_Text_Spotting_Key_Information_Extraction_CVPR_2024_paper.html) |
| Code | Code | Apache-2.0 | [AlibabaResearch/AdvancedLiterateMachinery](https://github.com/AlibabaResearch/AdvancedLiterateMachinery) |
| Pretrained weights | Model | Unknown | Not confirmed released; check repository |

### Models

- **Architecture:** Swin-B encoder (pretrained on ImageNet-22k) with FPN for multi-scale features at strides 4, 8, 16, 32. Three independent transformer decoders (Structured Points, Region, Content), each with 4 layers, 8 attention heads, hidden dimension 512, MLP amplification factor 4. Separate randomly initialized positional encodings per decoder to accommodate varying sequence lengths.
- **Code:** Available at [AlibabaResearch/AdvancedLiterateMachinery](https://github.com/AlibabaResearch/AdvancedLiterateMachinery) under Apache-2.0. The OmniParser V1 implementation is in the OCR/ subdirectory.
- **Weights:** Not explicitly mentioned as released in the paper. The code repository should be checked for the current status of pretrained weights.

### Algorithms

- **Pre-training Stage 1:** AdamW, batch 128, resolution $768 \times 768$, 500k steps, lr $5 \times 10^{-4}$, 5k-step linear warm-up then linear decay to 0.
- **Pre-training Stage 2:** AdamW, batch 16, resolution $1920 \times 1920$, 200k steps, lr $2.5 \times 10^{-4}$, same schedule.
- **Fine-tuning:** lr $1 \times 10^{-4}$, cosine decay. Text spotting: 20k steps. KIE: 200k steps. Table recognition: 400k steps for Structured Points Decoder, 200k for Content Decoder.
- **Data augmentation:** Instance-aware random cropping, random rotation in $[-90^\circ, 90^\circ]$, random resizing, color jittering.
- **Loss:** Negative log-likelihood with $w = 4.0$ for structural/entity tags, $w = 1.0$ for others. Prompt tokens excluded from loss computation.
- **Coordinate quantization:** Coordinates normalized to image dimensions, quantized to $[0, n\_{\text{bins}} - 1]$; value of $n\_{\text{bins}}$ is not stated in the main paper.
- **Text ordering during pre-training:** Center points arranged in raster scan order.

### Data

- **Pre-training corpus (9 datasets):** Curved SynthText, ICDAR 2013, ICDAR 2015, MLT 2017, Total-Text, TextOCR, HierText, COCO Text, Open Image V5. All are publicly available.
- **Task fine-tuning:** Standard train/val/test splits for each benchmark (Total-Text, CTW1500, ICDAR 2015, CORD, SROIE, PubTabNet, FinTabNet).
- **Annotation requirements:** The model requires word-level center-point annotations during training. Polygon annotations (16-point format) are also required. Standard benchmarks provide these.

### Evaluation

- **Text spotting:** End-to-end F1 under None/Full (Total-Text, CTW1500) and Strong/Weak/Generic (ICDAR 2015) lexicons. Standard protocol for each dataset.
- **KIE:** Field-level F1 and TED-based accuracy. Standard for CORD and SROIE; SROIE point locations are generated by the authors where not provided in the original annotation.
- **Table recognition:** TEDS and S-TEDS. Evaluated on PubTabNet validation (no public test set) and FinTabNet test set. Comparison restricted to end-to-end methods.
- **Baselines:** Generally fair within each task track. Donut for table recognition is reproduced by the authors using official configuration.
- **Statistical rigor:** No error bars, seeds, or significance tests reported.

### Hardware

- Training hardware not specified in the paper.
- The two-stage pre-training (700k total steps at high resolution, batch sizes 16-128) implies multi-GPU training, likely on A100-class hardware. Specific GPU counts and training time are not reported.
- Table recognition inference runs at 1.3 FPS (compared to Donut at 0.8 FPS) at input size $1024 \times 1024$ with a maximum decoder length of 1,500 + 200 tokens.

## BibTeX

```bibtex
@inproceedings{wan2024omniparser,
  title={OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition},
  author={Wan, Jianqiang and Song, Sibo and Yu, Wenwen and Liu, Yuliang and Cheng, Wenqing and Huang, Fei and Bai, Xiang and Yao, Cong and Yang, Zhibo},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2024}
}
```