# OmniParser V2: Unified Visual Text Parsing with Structured-Points-of-Thought

> OmniParser V2 unifies text spotting, key information extraction, table recognition, and layout analysis under a single encoder-decoder with a two-stage Structured-Points-of-Thought (SPOT) prompting scheme, replacing task-specific heads with a token-router-based shared decoder.

- Authors: Hunter Heidenreich
- Published: 2026-03-23
- Paper: Preprint (arXiv) <https://arxiv.org/abs/2502.16161>
- Code: OmniParser V2 Code <https://github.com/AlibabaResearch/AdvancedLiterateMachinery>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

OmniParser V2 is a unified visually-situated text parsing (VsTP) model that handles text spotting, key information extraction (KIE), table recognition (TR), and layout analysis within a single encoder-decoder. The core contribution is Structured-Points-of-Thought (SPOT) prompting: a two-stage decoding strategy that first generates a structured sequence of text center points, then predicts polygon contours and content in parallel from those points. A token-router-based shared decoder (a supervised MoE variant) replaces three separate decoders from the prior version, reducing model size by 23.6% while improving performance.

## What kind of paper is this?

**Dominant:** $\Psi\_{\text{Method}}$: The paper's headline contribution is a new architecture and training paradigm. The token-router-based shared decoder and SPOT prompting scheme are the central contributions, supported by ablations that isolate each design choice and SOTA-style comparison tables across four tasks.

## What is the motivation?

Visually-situated text parsing spans several related subtasks: text spotting (detection + recognition), KIE (linking named fields to their values), table recognition (structure + cell content), and layout analysis (word/line/paragraph hierarchy). Existing approaches address each subtask with dedicated architectures and objectives, creating modal isolation and complex multi-stage pipelines. Generalist models (typically large MLLMs) offer breadth but sacrifice localization precision and often fail without an external OCR engine. Specialist models are strong on their subtask but compose poorly.

The authors argue that a single model with a shared representation and a unified objective can match or exceed specialist performance while eliminating task-specific heads and reducing inference complexity.

## What is the novelty?

**Structured-Points-of-Thought (SPOT) Prompting.** SPOT decomposes generation into two stages:

1. **Stage 1 (Structured Points Sequence):** The decoder autoregressively generates center-point coordinates of text instances, interleaved with task-specific structural tokens (e.g., `<tr>` for table rows, `<line>` for layout lines). Coordinates are quantized into discrete bins: $x, y \in [0, n\_{\text{bins}} - 1]$ with $n\_{\text{bins}} = 1000$.

2. **Stage 2 (Polygon + Content):** Given each center point from Stage 1 as a prompt, the same decoder generates the full 16-point polygon and the character-level text transcript in parallel across all instances.

This decoupling reduces sequence length relative to directly predicting full HTML (as in Donut) and provides an explicit spatial anchor that reduces attention drift in long sequences. The authors draw an analogy to chain-of-thought reasoning: structured points are intermediate "reasoning" steps before the final output.

**Token-Router-Based Shared Decoder.** Rather than three independent decoders (one per decoding mode: structure, detection, recognition), OmniParser V2 uses a single decoder in which each transformer layer contains three task-specific feed-forward networks (Structured FFN, Detection FFN, Recognition FFN). Token routing is not learned; the category of each input token is predetermined by task-specific priors, making training more stable than a standard MoE:

$$
L = -\sum\_{j=k}^{N} w\_j \log P(\tilde{s}\_j \mid v, s\_{k:j-1})
$$

where $w\_j = 4.0$ for structural/entity tags and $w\_j = 1.0$ for other tokens. The shared self-attention and cross-attention layers use the same parameters across all token types; only the FFNs differ. This yields 110M total parameters versus 144M for OmniParser V1, a 23.6% reduction.

**Pre-training Strategies.** Two auxiliary strategies improve the decoder's spatial and semantic grounding:

- *Spatial-window prompting:* the decoder is conditioned on a bounding box and tasked with predicting center points only within that window, training robust spatial coordinate perception.
- *Prefix-window prompting:* the decoder is conditioned on a character range and tasked with predicting points only for text instances whose first character falls within that range, training character-level semantic grounding.

**SPOT for MLLMs.** The authors demonstrate that SPOT prompting can be applied to existing multimodal LLMs (e.g., QwenVL) via supervised fine-tuning on a curated dataset of 181k-389k text spotting examples. The two-stage instruction-following formulation (first generate center points; then generate polygons and transcripts) substantially improves text localization in models that otherwise struggle with it.

## What experiments were performed?

Experiments span four tasks across eight datasets.

**Text Spotting:** Total-Text, CTW1500 (curved text, line-level), and ICDAR 2015 (quadrilateral, word-level). Metrics: end-to-end F1 under None/Full/Strong/Weak/Generic lexicons. Baselines include DeepSolo, UNITS, TESTR, ABCNet v2, and others.

**Key Information Extraction:** CORD (1k receipt samples, 30 entity labels) and SROIE (626/347 receipts, 4 entities). Metrics: field-level F1 and TED-based accuracy. Baselines restricted to end-to-end (OCR-free) methods comparable to the OMNIPARSER V2 formulation.

**Table Recognition (end-to-end):** PubTabNet (500k train, 9k val) and FinTabNet (113k, financial tables). Metric: TEDS and S-TEDS. Baselines: EDD, Donut (fine-tuned), and OmniParser V1. Importantly, this is an *end-to-end* evaluation (structure + cell content recognized jointly), not a pure TSR evaluation, so results are not directly comparable to non-end-to-end TSR methods like TFLOP or UniTabNet that use offline OCR.

**Layout Analysis:** HierText validation and test sets. Metric: Panoptic Quality (PQ) at word, line, and paragraph level. Baselines: UniDec and Hi-SAM-B.

**Ablations:**
- Pre-training strategies (spatial-window vs. prefix-window): each contributes independently; combined they add +1.8 points E2E on Total-Text.
- Decoder architecture: token-router-based decoder outperforms native shared decoder (+1.72% on text spotting) and standard MoE decoder, at lower parameter count than OmniParser V1.
- Decoder length for table recognition: performance saturates at 1,500 structure tokens + 200 content tokens; longer sequences yield marginal gains at significant speed cost.
- SPOT length for MLLMs (N-SPOT / S-SPOT / L-SPOT): normal SPOT (generating the full center-point sequence as an intermediate step) performs best; omitting it (S-SPOT) or extending to per-instance detection/recognition prompting (L-SPOT) both hurt.

## What are the outcomes/conclusions?

On text spotting, OmniParser V2 achieves competitive results across all three benchmarks, with Total-Text E2E (None) at 84.3%, and ICDAR 2015 Generic at 80.6%, matching or exceeding prior unified models.

On KIE, it reaches 85.0% field F1 on CORD and outperforms prior generation-based methods on SROIE TED accuracy, while being pre-trained on scene text data only (no large document corpora).

On end-to-end table recognition, it achieves S-TEDS 90.5 / TEDS 88.9 on PubTabNet and S-TEDS 93.2 / TEDS 90.5 on FinTabNet, improving over OmniParser V1 and substantially over Donut. However, results are lower than pure TSR methods (e.g., TFLOP at 99.56 S-TEDS on FinTabNet) that assume clean OCR input; the comparison set in the paper is correctly restricted to end-to-end methods.

On layout analysis (HierText test), OmniParser V2 surpasses Hi-SAM-B by +1.9 / +1.2 / +0.8 PQ at word, line, and paragraph level respectively, in an end-to-end fashion without offline OCR.

SPOT applied to QwenVL and similar MLLMs shows meaningful improvements in text localization over both the base MLLMs and some specialist baselines. A performance gap versus OmniParser V2 itself remains, which the authors acknowledge as an open problem.

**Limitations acknowledged by the authors:**
- Understanding and reasoning capabilities remain limited; they cite reinforcement learning (e.g., DeepSeek-R1 style) as a future direction.
- A gap remains between SPOT-augmented MLLMs and the lightweight OmniParser V2 model on localization tasks.
- The paper does not compare end-to-end table recognition to non-end-to-end TSR methods, which would require a different experimental setup.

**Unacknowledged limitations:**
- The evaluation on table recognition excludes widely used benchmarks (e.g., WTW for wild tables) where non-end-to-end methods are typically reported. The restricted benchmark set makes cross-paper comparison difficult.
- No inference latency breakdown per task is reported; the 2.1 FPS figure is for the table recognition setting only (the most sequence-length-constrained task).
- No error bars, seeds, or significance tests are reported.

## Reproducibility

### Models

- **Architecture:** Swin-B (pretrained on ImageNet-22k) encoder with FPN, producing multi-scale visual embeddings. Token-router-based shared decoder with 4 transformer layers, 8 attention heads, hidden dimension 512, MLP amplification factor 4. Total parameters: approximately 110M.
- **Code:** The paper states code will be released at [AlibabaResearch/AdvancedLiterateMachinery](https://github.com/AlibabaResearch/AdvancedLiterateMachinery). The repository exists and is licensed Apache-2.0, but as of the preprint date no OmniParser V2 subdirectory has been added (only OmniParser V1 is present in the OCR/ folder).
- **Weights:** Not released at time of preprint submission.

### Algorithms

- **Pre-training Stage 1:** AdamW, batch 128, resolution $768 \times 768$, 500k steps, lr $5 \times 10^{-4}$, 5k-step linear warm-up then linear decay.
- **Pre-training Stage 2:** AdamW, batch 16, resolution $1920 \times 1920$, 200k steps, lr $2.5 \times 10^{-4}$, same schedule.
- **Fine-tuning:** Learning rate $1 \times 10^{-4}$, cosine decay. Text spotting and KIE: 20k and 200k steps respectively. Table recognition and layout: up to 400k steps.
- **Data augmentation:** Instance-aware random cropping, random rotation in $[-90^\circ, 90^\circ]$, random resizing, color jittering.
- **Loss:** Negative log-likelihood with token weighting ($w = 4.0$ for structural tags, $w = 1.0$ for others). Prompt tokens excluded from loss.
- **Coordinate quantization:** $n\_{\text{bins}} = 1000$; coordinates normalized to image dimensions then quantized to integers.

### Data

- **Pre-training data (8 datasets):** Curved SynthText, ICDAR 2013, ICDAR 2015, MLT 2017, Total-Text, TextOCR, HierText, COCO Text, Open Image V5.
- **MLLM fine-tuning data:** TS180k (181k) and TS380k (389k) text spotting examples; R440k (447k) and R980k (981k) read-all-text examples. Constructed from public datasets and data collected from Platypus.
- **Task-specific fine-tuning:** Uses standard splits for each benchmark (PubTabNet, FinTabNet, CORD, SROIE, Total-Text, CTW1500, ICDAR 2015, HierText).
- **Public availability:** PubTabNet, FinTabNet, CORD, SROIE, Total-Text, CTW1500, HierText, and ICDAR benchmarks are all publicly available. Platypus data is publicly available. Curved SynthText is publicly available.

### Evaluation

- **Text spotting:** End-to-end F1 with lexicon (None, Full, Strong, Weak, Generic). Standard for Total-Text and ICDAR 2015.
- **KIE:** Field-level F1 and TED-based accuracy. Standard for CORD and SROIE.
- **Table recognition:** TEDS and S-TEDS. Comparison is restricted to end-to-end methods only (EDD, Donut, OmniParser V1); non-end-to-end TSR methods are explicitly excluded.
- **Layout analysis:** Panoptic Quality (PQ) at three levels. Standard for HierText.
- **Baselines:** Generally fair within each task track, though compute budgets are not normalized across methods.
- **Statistical rigor:** No error bars, seeds, or significance testing reported. Single-run results throughout.

### Hardware

- Hardware is not explicitly stated in the paper.
- From context: the two-stage pre-training (700k total steps, batch 128 at $768^2$ then batch 16 at $1920^2$) suggests substantial GPU resources, likely 8+ A100-class GPUs.
- Table recognition inference at 1,124 + 200 decoder tokens runs at 2.1 FPS; the default 1,500 + 200 configuration runs at 1.7 FPS (configuration-dependent).
- The paper notes GPU constraints limit Donut's max sequence length to 4,000 in their reproductions, providing a rough lower bound on memory requirements.

## BibTeX

```bibtex
@article{yu2025omniparserv2,
  title={OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models},
  author={Yu, Wenwen and Yang, Zhibo and Wan, Jianqiang and Song, Sibo and Tang, Jun and Cheng, Wenqing and Liu, Yuliang and Bai, Xiang},
  journal={arXiv preprint arXiv:2502.16161},
  year={2025}
}
```