# PaddleOCR-VL: Two-Stage Document Parsing with a 0.9B VLM

> Analysis of PaddleOCR-VL, a two-stage document parsing system with layout analyzer and 0.9B VLM supporting 109 languages

- Authors: Hunter Heidenreich
- Published: 2025-10-20
- Updated: 2026-03-04
- Paper: PaddleOCR-VL Paper <https://arxiv.org/abs/2510.14528v1>
- Code: PaddleOCR Code <https://github.com/PaddlePaddle/PaddleOCR>
- Model: PaddleOCR-VL Models & Demo <https://huggingface.co/PaddlePaddle/PaddleOCR-VL>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

PaddleOCR-VL is a **two-stage document parsing system**: a lightweight layout analyzer (detect + reading order) followed by a 0.9B VLM (NaViT-style dynamic-resolution visual encoder + ERNIE-4.5-0.3B LM) that recognizes text, tables (OTSL), formulas (LaTeX), and charts (Markdown tables) across 109 languages. It posts SOTA/near-SOTA on OmniDocBench v1.0/v1.5 and olmOCR-Bench, while keeping inference lean via multithreaded, batched pipelines. Compared with end-to-end OCR VLMs, the decoupled layout stage improves reading-order stability, latency, and resource use.

* * *

## What kind of paper is this?

-   **Dominant:** $\\Psi\_{\\text{Method}}$ (new system decomposition + VLM architecture choices + training recipe).
-   **Secondary:** $\\Psi\_{\\text{Resource}}$ (large-scale data construction pipeline with 30M+ samples across multiple synthesized/auto-labeled corpora).
-   **Also present:** $\\Psi\_{\\text{Evaluation}}$ (explicit throughput/VRAM benchmarking; inference efficiency comparisons across hardware).

## What is the motivation?

-   Document parsing needs **layout understanding + reading order + element extraction** (text/tables/formulas/charts) to support downstream retrieval and RAG workflows.
-   The paper frames a tradeoff:
    -   **Pipeline systems:** strong but complex integration and error propagation.
    -   **End-to-end VLM conversion:** simpler interface but suffers from reading-order issues, hallucinations, and long-sequence latency/memory costs.
-   Goal: get the stability benefits of pipelines with the simplicity of a compact VLM for element recognition.

## What is the novelty?

-   **Two-stage decomposition:** PP-DocLayoutV2 handles layout detection + reading order; PaddleOCR-VL-0.9B recognizes each cropped element. Outputs merge into structured Markdown + JSON.
-   **Decoupled reading order:** RT-DETR + pointer network before VLM decoding mitigates long-context autoregressive burden and stabilizes ordering (a common failure mode in end-to-end VLM OCR).
-   **NaViT-style native dynamic resolution:** fewer hallucinations on dense pages vs. fixed-size or heavy tiling; preserves aspect ratio without extreme token counts.
-   **0.9B "ultra-compact" VLM:** 0.3B decoder + strong visual encoder yields favorable accuracy/latency balance for production.
-   **Data engine:** loops LLM-refined labels and typed hard-case synthesis to target weaknesses with measurable metrics.

> **Contrast with DeepSeek-OCR:** DeepSeek uses SAM $\\rightarrow$ conv-compressor $\\rightarrow$ CLIP DeepEncoder to shrink tokens 16$\\times$ and studies text/vision token ratios ($\\approx$97% OCR at $\\sim$10$\\times$ compression). PaddleOCR-VL instead optimizes robustness + throughput under a document-parsing workflow with explicit reading order.

## What experiments were performed?

-   **Page-level parsing:** OmniDocBench v1.5 (1,355 pages), OmniDocBench v1.0 (981 pages), olmOCR-Bench; comparisons against pipeline tools, general VLMs, and specialized OCR VLMs.
-   **Element-level recognition:**
    -   Text: OmniDocBench-OCR-block (17,148 crops) + in-house OCR set (107,452 line-level samples) + Ocean-OCR-Handwritten.
    -   Tables: OmniDocBench-Table-block (512 crops) + in-house benchmark (20 table types).
    -   Formulas: OmniDocBench-Formula-block (1,050 crops) + in-house benchmark (34,816 samples).
    -   Charts: in-house benchmark (1,801 samples) evaluated with RMS-F1.
-   **Inference/efficiency:** end-to-end speed on OmniDocBench v1.0 (512 PDFs, A100), reporting pages/s, tokens/s, VRAM; cross-hardware configs (A10, RTX 3060/5070/4090D).

## What are the outcomes/limitations?

**Outcomes:**

-   **OmniDocBench v1.5:** overall 92.56; text edit distance 0.035; formula CDM 91.43; table TEDS 89.76; reading-order ED 0.043.
-   **olmOCR-Bench:** overall 80.0 $\\pm$ 1.0 unit-test pass rate (best), leading in ArXiv and Headers/Footers.
-   **Element-level wins** across text/table/formula/chart benchmarks with detailed multilingual breakdowns.
-   **Throughput:** 1.224 pages/s on A100 (+15.8% vs. MinerU2.5); ~43.7 GB VRAM (vs. dots.ocr ~78.5 GB).

**Limitations:**

-   **Reproducibility constraints:** multiple critical components depend on in-house datasets and proprietary annotators (ERNIE-4.5-VL family); exact replication is difficult without equivalents.
-   **Coupling to detector quality:** overall fidelity and order depend on RT-DETR + pointer network; OOD layouts may degrade.
-   **Less emphasis on token-economy:** no explicit vision-tokens-per-page vs. accuracy frontier like DeepSeek-OCR.
-   **Output format assumptions:** tables trained to OTSL, charts to Markdown tables, formulas to LaTeX; downstream consumers may need adapters.
-   **Charts grounding:** strong RMS-F1 in-house, but public chart test sets are noisy/imbalanced.

* * *

## Model

### High-level architecture (Figures 2--4, pp.5--6)

![PaddleOCR-VL system overview: documents flow through PP-DocLayoutV2 for layout analysis and reading order, then batched element crops go to the VLM for recognition, producing structured Markdown and JSON output](https://labs.bevaya.ai/images/ocr/paddleocr-vl-system-overview.webp)

-   **Two stages**:
    1) PP-DocLayoutV2 (layout + reading order):

    -   Detector: RT-DETR for element boxes/classes (text blocks, tables, formulas, charts).
    -   Pointer network (6-layer Transformer) infers reading order: predicts an $N \\times N$ pairwise order matrix using absolute 2D position encodings, class embeddings, and a geometry-biased attention head (Relation-DETR-style); decoded via **deterministic win-accumulation** to recover a topologically consistent ordering.
    -   Rationale: avoids end-to-end VLM hallucinations and long-sequence overhead for layout; keeps this step small and fast.

    ![PP-DocLayoutV2 layout analysis architecture: RT-DETR detects bounding boxes, features pass through 6 transformer layers, then a head produces relative-order logits that are post-processed into reading order](https://labs.bevaya.ai/images/ocr/paddleocr-vl-layout-analysis.webp)

    2) PaddleOCR-VL-0.9B (element-level recognition):

    -   NaViT-style dynamic-resolution visual encoder (from Keye-VL), so it digests native-resolution crops without tiling distortion.
    -   2-layer MLP projector (GELU; merge size 2).
    -   ERNIE-4.5-0.3B LM with 3D-RoPE as the text decoder (small decoder means faster AR decoding).

![PaddleOCR-VL-0.9B architecture: batch images of varying sizes pass through dynamic resolution preprocessing, NaViT 400M vision encoder, 2-layer MLP with patch merge, then ERNIE-4.5-0.3B LLM decoder with 3D RoPE](https://labs.bevaya.ai/images/ocr/paddleocr-vl-architecture.webp)

> **Design point vs. DeepSeek-OCR:** PaddleOCR-VL decouples layout (specialized detector + ordering) from recognition, whereas DeepSeek-OCR is an end-to-end encoder--decoder optimized for optical compression of long text into few vision tokens. The former trades some end-to-end elegance for stability, lower latency, and cheaper training; the latter explores token-economy limits via compression.

### "What the VLM emits"

-   Text OCR: block/line/word-level transcription.
-   Tables: OTSL structural tokens + content.
-   Formulas: LaTeX (distinguishes inline `\(...\)` vs. display `\[...\]`).
-   Charts: normalized Markdown tables.

* * *

## Algorithms / Training (Section 2.2; Table 1, p.8)

-   **Layout (PP-DocLayoutV2)**:
    -   Train RT-DETR first (100 epochs on ~20k curated layout pages), then freeze it and train the pointer network for order (200 epochs; AdamW; GCE loss for noisy labels).
-   **VLM (PaddleOCR-VL-0.9B)**: two stages, all components trainable (ERNIEKit); batch size 128, sequence length 16,384 for both stages.
    -   Stage-1 alignment: 29M image--text pairs, max res $1280 \\times 28 \\times 28$ (NaViT), LR $5 \\times 10^{-5} \\rightarrow 5 \\times 10^{-6}$, 1 epoch.
    -   Stage-2 instruction FT: 2.7M carefully curated samples, max res $2048 \\times 28 \\times 28$, LR $5 \\times 10^{-6} \\rightarrow 5 \\times 10^{-7}$, 2 epochs; teaches 4 task families (OCR/table/formula/chart).

* * *

## Data (Section 3; Figure 5, p.9)

**Scale:** 30M+ training samples across open datasets, synthesis, web-harvested documents, and in-house sets.

**Three pillars**:

1) Curation from open sets (CASIA-HWDB, UniMER-1M, MathWriting; chart corpora like ChartQA/PlotQA/UniChart/Beagle/ChartINFO/visText/ExcelChart), synthesized data, web-scale crawl, and in-house corpora spanning many doc genres.
2) Automatic annotation: PP-StructureV3 pseudo-labels $\\rightarrow$ prompt LLMs (ERNIE-4.5-VL, Qwen2.5-VL) for refinement $\\rightarrow$ hallucination filtering.
3) Hard-case mining: build typed eval engines (23 text categories, 20 table types, 4 formula types, 11 chart families), score with EditDist, TEDS, CDM, RMS-F1 to find failures, then synthesize targeted data (XeLaTeX, web renderers). Reported synthesis rate: ~10,000 samples/hour for tables alone.

* * *

## Hardware / Inference (Section 4.3; Table 13, p.18)

-   Pipeline: three asynchronous threads (1) page rendering, (2) layout model, (3) batched VLM connected by queues; VLM batching triggers by queue size or dwell-time. vLLM/SGLang backends; knobs for max-batched-tokens and GPU memory utilization.
-   **Throughput (A100, vLLM, OmniDocBench v1.0)**:
    -   Pages/s: 1.224 (vs. MinerU2.5 1.057; +15.8%).
    -   Tokens/s: 1881 (vs. 1648; +14.2%).
    -   VRAM: ~43.7 GB; significantly less than dots.ocr (~78.5 GB) while being faster.
-   Cross-hardware benchmarks also provided for A10, RTX 3060, RTX 5070, and RTX 4090D.

* * *

## Evaluation

### Page-level (full pages; Figures 1, Tables 2--3)

-   **OmniDocBench v1.5** --- Overall 92.56; Text-Edit 0.035; Formula-CDM 91.43; Table-TEDS 89.76 / TEDS-S 93.52; Reading-Order ED 0.043. Top overall against pipelines and large VLMs.
-   **OmniDocBench v1.0** --- Avg Overall-Edit 0.115; Text-Edit: EN 0.041, ZH 0.062; Table-TEDS: EN 88.0, ZH 92.1; Reading-Order ED: EN 0.045, ZH 0.063.
-   **olmOCR-Bench** --- Overall 80.0 $\\pm$ 1.0 (best), leading in ArXiv and Headers/Footers; strong on Multi-column and Long tiny text.

### Element-level (cropped blocks)

-   Text (OmniDocBench-OCR-block) --- lowest Edit Distance across most doc types (e.g., PPT2PDF 0.049, Academic 0.021, Newspaper 0.034).
-   Handwriting (Ocean-OCR-Bench) --- EN ED 0.118, ZH ED 0.034; best F1/Precision/Recall/BLEU/METEOR among compared systems.
-   Tables (OmniDocBench-Table-block) --- TEDS 0.9195, TEDS-struct 0.9543, Overall-ED 0.0561 (best).
-   Formulas (v1.5 Formula-block) --- CDM 0.9453; In-house formulas CDM 0.9882.
-   Charts (in-house) --- RMS-F1 0.844 overall, surpassing several specialized OCR VLMs and some 72B-scale VLMs.

* * *

## Additional Notes

-   **Typed hard-case mining loop:** the paper's bucketed eval sets + targeted synthesis approach is a well-documented methodology for iterative data improvement.
-   **Prompting surface (Stage-2 tasks):** the VLM is trained on four task families with distinct output formats:
    -   OCR: block/line/word transcription
    -   Tables: OTSL structural markup
    -   Formulas: LaTeX (inline vs. display)
    -   Charts: Markdown tables
-   **Contrast with DeepSeek-OCR:** represents different design priorities; DeepSeek-OCR explores token-economy limits via optical compression ($\\sim$97% OCR at $\\sim$10$\\times$ compression), while PaddleOCR-VL prioritizes robustness and throughput via decoupled layout.