# MinerU: Notes

> Open-source multi-module PDF extraction pipeline producing Markdown/JSON from diverse PDFs using LayoutLMv3, YOLOv8, UniMERNet, TableMaster, and PaddleOCR

- Authors: Hunter Heidenreich
- Published: 2024-09-01
- Updated: 2026-03-04
- Code: MinerU Code <https://github.com/opendatalab/MinerU>
- Code: PDF-Extract-Kit <https://github.com/opendatalab/PDF-Extract-Kit>
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


## TL;DR

MinerU is an open-source, multi-module PDF extraction pipeline built on PDF-Extract-Kit models with explicit preprocessing and post-processing rules to produce Markdown/JSON from diverse PDFs (academic papers, textbooks, exams, financial reports, slides). The system uses LayoutLMv3 for layout detection (~21K training pages), YOLOv8 for formula detection (~2.9K pages), UniMERNet for formula recognition (UniMER-1M data), TableMaster + StructEqTable for tables, and PaddleOCR per text region. Post-processing includes bbox overlap resolution and reading-order segmentation. The paper emphasizes robustness via diverse training data and improved end-to-end readability via rule-based ordering heuristics.

* * *

## What kind of paper is this?

**Dominant:** $\\Psi\_{\\text{Resource}}$ (open-source "all-in-one" extraction system released as a project with models, pipeline, and tooling).

**Secondary:** $\\Psi_{\\text{Method}}$ (pipeline design + preprocessing/post-processing algorithms) and $\\Psi_{\\text{Evaluation}}$ (module-level comparisons on layout, formula detection/recognition, plus qualitative end-to-end visualizations).

The primary contribution is the release of a production-ready document extraction system. The method component describes the engineering of preprocessing (language detection, scanned PDF handling) and post-processing (overlap resolution, reading order) around existing model components. Evaluation focuses on validating individual modules rather than comprehensive end-to-end benchmarking.

* * *

## What is the motivation?

High-quality document extraction is critical for LLM training data pipelines and RAG systems, but existing approaches have significant gaps:

-   **OCR-only approaches** introduce noise on non-text elements (formulas, tables, figures) and struggle with layout understanding
-   **Library parsing** (direct PDF text extraction) fails on formulas, tables, and scanned documents; reading order is often scrambled in multi-column layouts
-   **Multi-module pipelines** show promise but prior open-source models often overfit to academic papers, degrading on textbooks, financial reports, exam papers, and slides
-   **End-to-end MLLMs** can handle diverse content but incur high inference costs for large-scale document processing

The authors position MinerU as addressing the robustness and diversity gap in open-source document extraction via data-engineering-driven model training and explicit post-processing for layout-aware reading order.

* * *

## What is the novelty?

### System-level engineering

An end-to-end workflow combining:

-   **PDF preprocessing:** Parseability detection, language identification (Chinese/English only), encryption handling, scanned vs. parseable classification, metadata extraction
-   **Model-based region detection/recognition:** Five-model pipeline (layout detection, formula detection, formula recognition, table recognition, OCR)
-   **Rule-based post-processing:** Bounding-box overlap resolution (containment, partial overlap handling), reading-order segmentation using "top-to-bottom, left-to-right" heuristics
-   **Format conversion:** Intermediate JSON representation (with `_parse_type` and `_version_name` fields) converting to Markdown/JSON outputs

### Data-engineering emphasis

Models in PDF-Extract-Kit are trained/fine-tuned on **diverse document sources** beyond academic papers:

-   11-category document taxonomy: Academic Papers, Research Reports (financial), Standard/Special Image-Text Textbooks, Slides, Exam Papers, Historical Documents, Handwritten Notes, Picture Albums, Standard Books
-   Validation-guided sampling: cluster PDFs by visual features, sample across cluster centers to maximize diversity
-   Iterative annotation based on model validation feedback

### Post-processing for reading order

Explicit algorithms for handling common failure modes:

-   **Containment removal:** Text/formula boxes inside image/table regions are filtered out to prevent duplication
-   **Partial overlap resolution:** Vertically/horizontally overlapping text boxes are shrunk to avoid mutual coverage
-   **Column-aware segmentation:** Page is segmented into regions consistent with "top-to-bottom, left-to-right" reading, with each region containing at most one column

> **Contrast to MinerU2.5:** The original MinerU is a pipeline system combining specialized models with explicit preprocessing/post-processing rules. MinerU2.5 (released ~1 year later) replaces the multi-module pipeline with a unified 1.2B-parameter VLM using a two-stage coarse-to-fine architecture (thumbnail layout detection followed by native-resolution crop recognition). The successor achieves higher accuracy with learned representations but requires GPU inference, while the original MinerU targets broader deployment scenarios including CPU-only environments.

* * *

## What experiments were performed?

The paper provides **module-level evaluation** on three core components:

### 1. Layout detection (Table 3)

Compared LayoutLMv3-Finetuned against DocXchain, Surya, and 360LayoutAnalysis variants on academic papers and textbooks using mAP/AP50/AR50 metrics.

### 2. Formula detection (Table 4)

Compared YOLOv8-Finetuned against Pix2Text-MFD on academic papers and multi-source documents using AP50/AR50 metrics.

### 3. Formula recognition (Table 5)

Compared UniMERNet against Pix2tex, Texify, and Mathpix on CDM-adapted evaluation using ExpRate, ExpRate@CDM, BLEU, and CDM metrics.

### 4. End-to-end qualitative results (Figure 7)

Three-column visualization (layout → spans → Markdown) across document types: academic literature, textbooks, exam papers, and research reports. The authors argue module quality plus post-processing yields readable Markdown, but **no quantitative end-to-end metrics** are reported in the provided sections.

* * *

## What are the outcomes/limitations?

### Outcomes (as reported)

**Layout detection** (LayoutLMv3-Finetuned):

| Split               | mAP $\\uparrow$ | AP50 $\\uparrow$ | AR50 $\\uparrow$ |
| ------------------- | --------------- | ---------------- | ---------------- |
| Academic Papers Val | **77.6**        | **93.3**         | **95.5**         |
| Textbook Val        | **67.9**        | **82.7**         | **87.9**         |

The fine-tuned model substantially outperforms reported baselines (e.g., DocXchain academic mAP 52.8, Surya academic mAP 24.2).

**Formula detection** (YOLOv8-Finetuned):

| Split               | AP50 $\\uparrow$ | AR50 $\\uparrow$ |
| ------------------- | ---------------- | ---------------- |
| Academic Papers Val | **87.7**         | **89.9**         |
| Multi-source Val    | **82.4**         | **87.3**         |

**Formula recognition** (UniMERNet):

| Model         | ExpRate@CDM $\\uparrow$ | CDM $\\uparrow$ |
| ------------- | ----------------------- | --------------- |
| **UniMERNet** | **0.811**               | **0.968**       |
| Mathpix       | 0.5                     | 0.951           |

The authors emphasize CDM as the more reliable metric over ExpRate for formula recognition evaluation.

### Limitations / caveats

**Language support constraints:**

-   MinerU **currently supports only Chinese and English** for high-quality extraction
-   Other languages are not quality-guaranteed; preprocessing explicitly filters for these two languages

**Reading order assumptions:**

-   Post-processing assumes **"top-to-bottom, left-to-right"** reading order, which may not align with vertical scripts (Japanese/Chinese vertical) or right-to-left languages (Arabic, Hebrew)
-   The dataset taxonomy includes "Historical Document" with right-to-left reading order mentioned, but the algorithm description remains LTR-oriented

**Evaluation coverage gaps:**

-   **No quantitative end-to-end benchmarking** reported in the paper (no edit distance, F1, BLEU, or other full-document metrics)
-   Table recognition and OCR accuracy are described qualitatively with visualizations but lack detailed quantitative evaluation tables
-   Module-level metrics don't capture error propagation through the pipeline

**Runtime / compute not reported:**

-   The paper positions MinerU as "low inference cost" compared to end-to-end MLLMs, but provides **no concrete latency, throughput, or memory measurements**
-   Hardware requirements (CPU vs GPU, VRAM needs) are not specified

**Open questions:**

-   How does end-to-end accuracy compare to newer unified VLMs (GOT-OCR2.0, InternVL, Qwen2-VL) on standard benchmarks?
-   What is the error propagation impact from layout detection mistakes?
-   How does the system handle edge cases like overlapping text/tables, complex nested structures, or documents with non-standard layouts?

* * *

## Model

### Pipeline overview

MinerU follows an explicitly staged workflow:

1.  **Document preprocessing:** Language detection (Chinese/English), page size/count extraction, encryption/password handling, scanned vs. parseable classification
2.  **Content parsing** via five models: Layout detection → formula detection → formula recognition + table recognition + OCR (applied per-region)
3.  **Post-processing:** Bounding box overlap resolution, image/table cropping, header/footer removal, reading-order reconstruction
4.  **Format conversion:** Intermediate JSON → Markdown and final JSON outputs

### Core model components (v0.8.1)

The pipeline integrates **five specialized models**:

| Component           | Model                       | Training Data                                         | Notes                                                                                         |
| ------------------- | --------------------------- | ----------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Layout detection    | LayoutLMv3-Finetuned        | ~21K pages                                            | 11 categories including title, paragraph, images, captions, tables, formulas, headers/footers |
| Formula detection   | YOLOv8-Finetuned            | 2,890 pages, 24,157 inline + 1,829 displayed formulas | Three categories: inline, displayed, ignore (ambiguous)                                       |
| Formula recognition | UniMERNet                   | UniMER-1M                                             | Handles printed/scanned/handwritten variety                                                   |
| Table recognition   | TableMaster + StructEqTable | PubTabNet v2.0.0 + DocGenome                          | Table-to-LaTeX / Table-to-HTML                                                                |
| OCR                 | PaddleOCR                   | **NR**                                                | Applied per detected text region (not whole-page)                                             |

### OCR strategy for multi-column and inline formulas

**Per-region OCR** (not whole-page):

-   OCR is run **on detected text regions** (titles, paragraph blocks) individually to avoid merging columns into a single reading stream
-   Preserves column boundaries established by layout detection

**Inline formula masking**:

1.  Formulas are **masked** in the text region using formula detector bounding boxes
2.  OCR runs on the masked region (text only)
3.  Formulas are **reinserted** into OCR output at their original positions

This prevents OCR from garbling mathematical notation while maintaining text flow.

* * *

## Data

### Layout detection training data

**Size:** ~21K annotated pages

**Data collection process:**

1.  Collect diverse PDFs across document types
2.  Cluster pages by visual features (embeddings)
3.  Sample across cluster centers to maximize diversity
4.  Annotate using validation-guided feedback loop

**Annotation schema** (11 categories):

-   Content elements: title, paragraph text, images, captions, tables, table captions, image/table notes, inline formulas, formula labels
-   Discard types: headers, footers, page numbers, page notes

### Formula detection training data

**Size:** 2,890 pages with 24,157 inline formulas and 1,829 displayed formulas

**Sources:** Chinese and English academic papers, textbooks, books, financial reports

**Categories:**

-   **Inline formulas:** Embedded in text flow
-   **Displayed formulas:** Standalone equations
-   **Ignore:** Ambiguous cases like "50%", "NaCl", "1-2 days" that resemble formulas but are plain text

### Table recognition training data

-   **TableMaster:** Trained on **PubTabNet v2.0.0**
-   **StructEqTable:** Trained on **DocGenome** benchmark data

### Document diversity taxonomy (Table 2)

The paper analyzes 11 document categories for training/validation:

1.  Academic Paper
2.  Research Report (financial reports, prospectuses)
3.  Standard Textbook
4.  Special Image-Text Textbook
5.  Slides (presentation materials)
6.  Exam Papers
7.  Historical Documents (includes right-to-left reading order examples)
8.  Handwritten Notes
9.  Picture Albums
10. Standard Books
11. Special Image-Text Exam Papers

* * *

## Algorithms / Training

### Preprocessing decisions

**Language identification:**

-   Detects document language to set OCR language parameter
-   **Only Chinese and English are explicitly supported** for quality guarantees

**Garbled-text detection:**

-   Attempts direct PDF text extraction (library parsing via PyMuPDF)
-   If extracted text is garbled (encoding issues, corrupted fonts), switches to OCR-based extraction

**Scanned PDF identification heuristics:**

-   Large image area relative to text area
-   Full-page or near-full-page images
-   Near-zero average text length per page

**Parse path selection:**

-   **Parseable PDFs:** Direct text extraction (PyMuPDF `fitz` library) for speed and accuracy
-   **Scanned PDFs:** Enable full OCR pipeline for all content

### Post-processing for bounding box overlaps

**Containment removal:**

-   Remove text/formula boxes **contained within** image or table regions to prevent duplicate content
-   Remove text boxes **contained within** formula boxes (formulas are atomic units)

**Partial overlap handling:**

-   **Text-text overlaps:** Shrink overlapping text boxes (vertically or horizontally) to eliminate mutual coverage
-   **Text-image/table overlaps:** Temporarily ignore image/table boxes to preserve text integrity; prioritize text reading flow over image placement

### Reading-order segmentation

**Goal:** Produce stable "top-to-bottom, left-to-right" reading order across multi-column layouts

**Algorithm:**

1.  After overlap resolution, segment page into **regions**
2.  Each region contains **at most one column** of content
3.  Sort regions by vertical position (top-to-bottom primary)
4.  Within each region, sort elements by horizontal then vertical position (left-to-right, top-to-bottom secondary)
5.  Concatenate region contents in sorted order to produce final reading sequence

**Assumption:** Left-to-right, top-to-bottom reading convention (standard for English and Chinese horizontal text). The algorithm does not explicitly handle right-to-left scripts or vertical text layouts mentioned in the "Historical Document" category.

* * *

## Evaluation

### Layout detection (Table 3)

Metrics: mAP, AP50, AR50 on academic papers and textbook validation sets

| Model                           | Split               | mAP      | AP50     | AR50     |
| ------------------------------- | ------------------- | -------- | -------- | -------- |
| **LayoutLMv3-Finetuned (Ours)** | Academic Papers Val | **77.6** | **93.3** | **95.5** |
| **LayoutLMv3-Finetuned (Ours)** | Textbook Val        | **67.9** | **82.7** | **87.9** |
| DocXchain                       | Academic Papers Val | 52.8     | 75.4     | 81.2     |
| Surya                           | Academic Papers Val | 24.2     | 47.6     | 58.9     |

The fine-tuned model shows substantial improvements over baselines, particularly on academic papers. Textbook performance is lower across all models, suggesting increased layout complexity.

### Formula detection (Table 4)

Metrics: AP50, AR50 on academic papers and multi-source validation sets

| Model                       | Split               | AP50     | AR50     |
| --------------------------- | ------------------- | -------- | -------- |
| **YOLOv8-Finetuned (Ours)** | Academic Papers Val | **87.7** | **89.9** |
| **YOLOv8-Finetuned (Ours)** | Multi-source Val    | **82.4** | **87.3** |
| Pix2Text-MFD                | Academic Papers Val | 78.3     | 81.5     |
| Pix2Text-MFD                | Multi-source Val    | 71.6     | 76.8     |

Performance drops on multi-source validation reflect increased diversity (textbooks, financial reports, etc.) beyond academic papers.

### Formula recognition (Table 5)

Metrics adapted from CDM paper: ExpRate, ExpRate@CDM, BLEU, CDM

| Model                | ExpRate   | ExpRate@CDM | BLEU      | CDM       |
| -------------------- | --------- | ----------- | --------- | --------- |
| Pix2tex              | 0.584     | 0.407       | 0.753     | 0.881     |
| Texify               | 0.607     | 0.632       | 0.836     | 0.945     |
| Mathpix              | **0.831** | 0.5         | **0.879** | 0.951     |
| **UniMERNet (Ours)** | 0.817     | **0.811**   | 0.871     | **0.968** |

UniMERNet achieves the highest CDM score (0.968), which the authors emphasize as the most reliable metric. Mathpix leads on raw ExpRate but scores lower on ExpRate@CDM, suggesting potential overfitting to exact match without considering semantic equivalence.

### End-to-end qualitative results (Figure 7)

The paper shows three-column visualizations (layout detection → spans → Markdown output) across document types:

-   Academic literature: Complex formulas, tables, multi-column layouts
-   Textbooks: Mixed text, images, captions, inline formulas
-   Exam papers: Varied formatting, handwritten elements (in scanned cases)
-   Research reports: Dense tables, financial notation

The visualizations demonstrate module integration and post-processing effectiveness, but **no quantitative end-to-end metrics** (edit distance, F1, BLEU, etc.) are provided for full-document extraction accuracy.

* * *

## Hardware / Production

**Runtime and compute:** The paper positions MinerU as "low inference cost" compared to end-to-end MLLMs, but **does not provide concrete measurements**:

-   No latency or throughput numbers (pages/second, seconds/page)
-   No memory requirements (VRAM for GPU inference, RAM for CPU inference)
-   No hardware specifications for evaluation runs

**Deployment artifacts:**

-   **Intermediate JSON format:** `pdf_info.para_blocks` structure with `_parse_type` field (`txt` vs `ocr`) and `_version_name` for versioning
-   **Output formats:** Markdown and JSON with optional image/table cropping
-   **CLI interface:** Command-line tool for batch processing

**Production considerations:**

-   Pipeline modularity enables CPU-only deployment (though OCR and detection models benefit from GPU acceleration)
-   Open-source license (Apache 2.0 for code, model licenses vary by component) permits commercial use
-   Maintenance requires updates to five separate model components plus preprocessing/post-processing rules

* * *

_Note: This analysis follows the Bevaya research note guidelines and classification taxonomy. For academic or production use, consult the original paper and verify claims through independent evaluation._