# POINTS-Reader: Distillation-Free Document Conversion via Iterative Self-Improvement
> Analysis of POINTS-Reader, a 4B VLM for document conversion trained via synthetic warm-up and iterative self-improvement without distillation
- Authors: Hunter Heidenreich
- Published: 2025-09-15
- Updated: 2026-03-04
- Paper: POINTS-Reader Paper (EMNLP 2025)
- Paper: POINTS-Reader arXiv Preprint
- Code: POINTS-Reader Code
- Model: POINTS-Reader Model
- Source: Bevaya Labs,
---
## TL;DR
POINTS-Reader is a 4B-parameter document-conversion VLM built on POINTS-1.5 and trained via a two-stage, distillation-free pipeline: synthetic "uniform format" warm-up (800K samples across plain text, formulas, tables, multi-column layouts) and iterative self-improvement on real PDFs (DocMatix) with rule-based filtering. By unifying outputs for text (Markdown), tables (HTML), and formulas (LaTeX) and repeatedly bootstrapping on its own filtered annotations, the model surpasses many larger VLMs and OCR experts on OmniDocBench and Fox, especially for tables, while still trailing pipeline systems like MinerU and Mathpix.
* * *
## What kind of paper is this?
**Dominant:** $\\Psi\_{\\text{Method}}$ (training procedure: synthetic warm-up plus iterative self-improvement with rule-based filtering, rather than new architecture).
**Secondary:** $\\Psi_{\\text{Resource}}$ (implicitly releases carefully filtered HTML-table, formula-aware dataset of $\\sim$1.1M high-quality samples plus trained model); $\\Psi_{\\text{Evaluation}}$ (ablations on data scale, aspect ratio filtering, F1 thresholds, sampling ratios, initialization choices, all quantified on OmniDocBench).
The core contribution is a training methodology that avoids distillation from proprietary or very large models. The paper demonstrates that careful synthetic data design plus iterative bootstrapping with strict automated quality filters can produce competitive document-conversion models without inheriting teacher model biases or errors.
* * *
## What is the motivation?
High-quality labeled data for end-to-end document conversion (text plus tables plus math plus layout) is scarce and expensive to annotate. Existing approaches usually distill from proprietary or very large open VLMs (GPT-4o, Qwen2.5-VL-72B, etc.), which locks progress to teacher capabilities and transfers teacher errors and biases (missing tables, hallucinated or omitted text, wrong table structure). Purely synthetic data helps but diverges from real-world layouts; models trained only on synthetic images underperform on complex PDFs. The goal is to learn a strong OCR/VLM reader without distillation by warm-starting on synthetic but layout-diverse data with unified output format, then adapting to real documents via self-improvement on DocMatix with strict automated filtering.
* * *
## What is the novelty?
**Unified output representation** for document elements:
- Plain text: standardized Markdown (headers, lists, etc.)
- Tables: HTML `` with simplified attributes, no CSS except merged-cell info, no extra whitespace or indentation to keep tokens down
- Formulas: KaTeX-compatible LaTeX, `$...$` inline and `$$...$$` display
**Two-stage distillation-free training pipeline:**
1. **Uniform Format Warm-up Stage (UWS):** LLM-generated synthetic documents across four categories (plain text, text plus formulas, text plus tables, multi-column layouts with tables). HTML templates (1/2/3-column) rendered with headless Chrome to produce image--text pairs.
2. **Iterative Self-improvement Stage (ISS):** Use the warm-up model to annotate millions of real DocMatix pages; filter predictions (plain-text F1 vs PaddleOCR, HTML structural checks for tables, syntax checks for LaTeX); retrain on filtered set; repeat for several iterations.
**Rule-based data filtering at scale:**
- Bag-of-words F1 against PaddleOCR for text
- Structural validation for tables
- Syntax validity for formulas
**Empirical insights about data and training:**
- Sweet spot in synthetic data size ($\\sim$800K samples) beyond which performance drops due to distribution mismatch
- Restricting aspect ratio to $\\sim[0.4, 2.5]$ removes pathological images and improves results
- Always re-initializing each iteration from the base pre-trained model (POINTS-1.5) is better than continuing from the previous iteration
- Including UWS data in later ISS iterations helps, despite its synthetic nature, because its annotations are very clean
* * *
## What experiments were performed?
**Warm-up ablations** (OmniDocBench overall metric, lower is better):
- Incrementally adding each synthetic category (plain text, then plus formulas, then plus tables, then plus multi-column tables) while keeping 200K samples per category (800K total)
- Scaling total synthetic data from 100K to 1.2M to find saturation/overfitting point
- Filtering images by aspect ratio ranges to prune extreme shapes
**Self-improvement ablations:**
- Adding filters in stages (text, then table, then formula) and measuring improvements
- Sweeping F1 thresholds for text filtering (0.70, 0.80, 0.90, 0.95)
- Varying sampling ratios for plain text vs tables vs formulas during training
- Comparing initialization from previous iteration vs re-starting from the original POINTS-1.5 weights
- Tracking performance vs ISS iteration count (up to five rounds), with curves for OmniDocBench, global F1 vs PaddleOCR, and retained data volume
**Dataset comparison:**
- Compare the final distilled-free dataset ($\\sim$1.1M samples) to KOSMOS-2.5 training data and olmOCR data (size, table format, distillation vs non-distillation, language coverage)
**Baseline comparisons** on OmniDocBench (en) and Fox-Page-en:
- Pipeline methods: MinerU, Marker, Mathpix
- General VLMs: Qwen2.5-VL (3B, 7B, 72B)
- Expert OCR/VLM: GOT-OCR, Nougat, Mistral OCR, OLMOCR
- Additionally: a version of POINTS-Reader trained directly on distillation data from Qwen2.5-VL-72B
* * *
## What are the outcomes/limitations?
**Outcomes (as reported):**
On **OmniDocBench (en)**:
- POINTS-Reader (4B) overall edit distance **0.259**, beating several larger specialized and general models:
- Qwen2.5-VL-3B: 0.390; Qwen2.5-VL-7B: 0.331; GOT-OCR: 0.287; OLMOCR-7B: 0.326
- Particularly strong **table** performance: 0.335 vs 0.341 for Qwen2.5-VL-72B and much better than GOT-OCR on the same metric (gap $\\approx 0.197$)
On **Fox-Page-en** (edit distance, lower is better):
- POINTS-Reader: 0.023, outperforming all listed general VLMs and OCR baselines in the table (e.g., Qwen2.5-VL-72B at 0.027)
Compared to a model **directly distilled from Qwen2.5-VL-72B**, the distillation-free POINTS-Reader is decisively better (overall 0.259 vs 0.302) on OmniDocBench.
However, **pipeline systems** like MinerU and Mathpix still achieve lower error overall than any end-to-end VLM, including POINTS-Reader, highlighting remaining performance gaps.
ISS curves show monotonic improvement across iterations on OmniDocBench, global F1 vs PaddleOCR rising from $\\sim$0.70 up to $\\sim$0.84, and number of retained filtered samples increasing for text, tables, and formulas over iterations.
**Limitations (acknowledged by authors):**
- **Language:** Training and evaluation are **English-only**; multi-lingual and CJK support are future work
- **Content type:** Datasets are largely **printed fonts**; performance on handwritten notes is suboptimal. The model currently extracts only plain text, formulas, and tables, not images or figure locations
- **Data supervision:** Table and formula filters check only structure and syntax, not semantic correctness, so some residual label noise remains
- **Benchmark scope:** Focus is on Fox and OmniDocBench; there is no evaluation on real production workflows or domain-specific documents (e.g., handwritten forms, multi-language PDFs)
* * *
## Model
### Backbone and overall architecture
- Base model: **POINTS-1.5** VLM as the visual encoder plus multimodal connector
- Language backbone: **Qwen2.5-3B-Instruct** as the LLM head, chosen for efficiency vs quality trade-off
- Parameter scale: $\\sim$4B parameters
- Context length: **8192 tokens** in this work (longer than default POINTS-1.5)
- Training paradigm inherited from POINTS-1.5:
1. Pretraining on generic vision--language data
2. Visual instruction tuning (VIT) with both the synthetic UWS data and filtered ISS data, plus general data from POINTS-1.5
- All hyperparameters and settings except VIT data and context length are identical to POINTS-1.5, so this paper's novelty sits almost entirely in the data and training pipeline rather than architecture tweaks
### Input and output format
- **Input:** Document page images, often multi-column, with tables and formulas. Real data comes from PDF-derived PNGs in DocMatix
- **Output:** Single text sequence in the unified markup format:
- **Plain text:** Markdown headings, lists, bold/italic, etc.
- **Tables:** Minimal HTML table markup with no whitespace between tags, no CSS apart from merged-cell attributes; same format used for both synthetic and real annotations
- **Formulas:** LaTeX obeying KaTeX rules, `$...$` inline, `$$...$$` display
* * *
## Data
### 1. Uniform Format Warm-up (Synthetic)
**Unified output format:**
- Text: Markdown (following Kosmos-2.5 style choices)
- Tables: HTML to handle complex structures (merged cells) more flexibly than Markdown; LaTeX tables rejected due to non-standard and variable syntax
- Formulas: KaTeX-compatible LaTeX syntax
**Generation pipeline:**
1. **LLM text generation:** Prompts instruct a large language model to generate document-like Markdown, including topic selection, 300--800 words depending on category, style choices from exam papers, slides, academic papers, books, textbooks, magazines, notes, newspapers, financial reports, and optional use of Markdown constructs (headings, lists, bold/italic, subscripts/superscripts). Formula-containing prompts encourage varied LaTeX constructs (matrices, `align`, `gather`, `frac`, `sum`, etc.), mixing inline and display math. Multi-column prompts split content into two logical chunks separated by a marker string `"x----------x"` for later layout templating.
2. **Category design:** Four synthetic categories (plain text; text plus formulas; text plus tables, single-column; multi-column layouts with tables)
3. **Table enrichment:** LLM-prompting alone tended to produce simple tables, so they augment with real tables from **PubTabNet** training set, generating descriptive paragraphs and then inserting those tables into the text at random positions
4. **Filtering (synthetic side):** Apply the same LaTeX formula and HTML table filters as in ISS (syntax/structure checks) before rendering
5. **Rendering:** Convert the unified text into HTML and render images with 1/2/3-column templates using Chrome headless mode
**Synthetic dataset scale:**
- For ablations: 200K filtered samples per category yields **800K total**; this configuration gives the best performance
- Scaling from 100K to 1.2M samples shows gains up to $\\sim$800K, then degradation (overfitting to synthetic layouts that diverge from real PDFs)
- Aspect ratio filtering: Inspect distribution of width/height and cut off extreme aspect ratios outside $[2/5, 5/2]$. This removes overly tall or wide renderings and measurably improves OmniDocBench scores
### 2. Iterative Self-improvement (Real Data)
**Base real-world corpus:**
- **DocMatix:** More than 2M images derived from PDF/A, covering academic papers, textbooks, exams, and various document types
- Total used: 2,234,134 images; in the final ISS iteration, $\\sim$1.1M (1,096,325) pass filters and are used for training
**Element distribution (final ISS iteration):**
- Only plain text: **90.2%** of samples
- Contains tables: **6.5%**
- Contains formulas: **3.3%** (about 0.1% have both tables and formulas)
**Token length distribution:**
- Most training samples are shorter than 1,000 tokens
**Final dataset comparison:**
| Dataset | Size | Tables Format | Distillation | Language |
| ----------------- | ----------- | ------------- | ------------ | -------- |
| KOSMOS-2.5 | 357.4M | Markdown | Yes | English |
| olmOCR | 260K | Markdown | Yes (GPT-4o) | English |
| **POINTS-Reader** | $\\sim$1.1M | **HTML** | **No** | English |
* * *
## Algorithms and Training
### 1. Plain-text F1 filtering
For text quality control, they compare POINTS-Reader predictions to **PaddleOCR** outputs as pseudo-ground-truth:
1. Normalize both prediction and reference strings (strip non-alphanumeric characters, split on spaces and count occurrences of each token)
2. Let $P = {(u^p_i, c^p_i)}$ be prediction tokens and counts, $T = {(u^t_i, c^t_i)}$ be reference tokens and counts
3. Define:
$$\\text{Precision} = \\frac{\\sum_i \\min(c^p_i, c^t_i)}{\\sum_i c^p_i},\\quad
\\text{Recall} = \\frac{\\sum_i \\min(c^p_i, c^t_i)}{\\sum_i c^t_i}$$
$$F_1 = \\frac{2 \\times \\text{Precision} \\times \\text{Recall}}{\\text{Precision} + \\text{Recall}}$$
4. Discard samples with $F_1$ below a threshold
Ablation on thresholds: **0.90** gives the best overall OmniDocBench score, balancing quality and coverage; 0.70 and 0.95 both hurt performance relative to 0.90.
### 2. Table filtering
They do not use external structure-recognition models as teachers, due to robustness issues and table-only assumptions. Instead, they parse each HTML table in the output and enforce **structural validity:**
- Consistent numbers of cells per row/column
- Valid merged-cell attributes
Samples with malformed table structures are dropped.
### 3. Formula filtering
- Extract all LaTeX formulas from the output and run syntax checks (KaTeX-compatible)
- Samples containing any syntactically invalid formula are discarded
- They explicitly **do not check semantic correctness** (e.g., whether the math equals the underlying document)
### 4. Iterative self-improvement loop
Each ISS iteration:
1. Run the current model on the full DocMatix dataset to produce annotations (image to unified markup)
2. Apply the three filters in order: text F1, table structure, formula syntax
3. Use all retained image--text pairs to perform visual instruction tuning from the **base POINTS-1.5 weights**, not from the previous ISS iteration's weights (this choice is empirically better)
4. Include UWS synthetic data in ISS training; this reliably improves performance
Findings from ISS:
- Each iteration improves both OmniDocBench performance and DocMatix F1 vs PaddleOCR
- The volume of retained samples with text/tables/formulas grows per iteration, indicating the model is generating more usable annotations
- Improvements come from a combination of better original training data and increased diversity from new filtered samples
### 5. Sampling ratios
Because plain-text-only samples dominate, they tested aggressive rebalancing: lowering the plain-text sampling probability and up-weighting tables and formulas. This rebalancing **hurts** performance; best results come from sampling directly from the natural distribution (1.0:1.0:1.0 weights). Hypothesis: down-sampling text reduces diversity without increasing table/formula diversity (just repeats the same small set more often), degrading generalization.
* * *
## Evaluation
### Benchmarks and metrics
- **OmniDocBench (English split):** 19 layout types; evaluates text edit distance, formula edit distance, table edit distance, reading order correctness. Overall metric is a blend of these (all normalized edit distances; lower is better)
- **Fox-Page-en** (English split of Fox): 112 pages, single and double column, each with more than 1K words. Metric: normalized edit distance between model outputs and references; lower is better
### Core comparisons
**Pipeline methods** (MinerU, Marker, Mathpix):
- Achieve the **lowest** OmniDocBench error overall; still the strongest approach for document conversion in this comparison
**General VLMs:**
- Qwen2.5-VL-72B is the strongest among generic models (overall $\\sim$0.214) but much larger than POINTS-Reader
- Smaller Qwen2.5-VL (3B, 7B) lag behind POINTS-Reader on both OmniDocBench and Fox
**Expert VLM/OCR:**
- GOT-OCR, Nougat, Mistral OCR, OLMOCR all underperform relative to POINTS-Reader on the combined OmniDocBench metrics, especially for tables
- POINTS-Reader vs Mistral OCR: substantial gain in OmniDocBench overall (0.259 vs 0.268) and much better tables
### Ablation highlights
| Experiment | Configuration | OmniDocBench Overall $\\downarrow$ |
| ----------------------- | ---------------------------------- | ---------------------------------- |
| Synthetic diversity | Plain text only | 0.626 |
| | + Formulas | 0.579 |
| | + Tables | 0.538 |
| | + Multi-column tables | **0.510** |
| Aspect ratio filtering | No filtering | 0.515 |
| | $[2/5, 5/2]$ | **0.498** |
| ISS filtering | No filters | 0.493 |
| | + Text F1 filter | 0.463 |
| | + Table filter | 0.447 |
| | + Formula filter | **0.439** |
| Distillation comparison | Distilled from Qwen2.5-VL-72B | 0.302 |
| | **POINTS-Reader ISS (no distill)** | **0.259** |
* * *
## Hardware and Production
- All experiments run on **64 $\\times$ NVIDIA H800 GPUs**
- Approximate costs reported:
- Training on **1M samples $\\approx$ 7 hours**
- Inference on **2M DocMatix images $\\approx$ 10 hours** using **SGLang** for deployment-time serving
- The paper does not detail batching strategies or peak throughput, but given the 4B parameter count and 8192-token context, the model is engineered to be relatively efficient compared to 70B-class VLMs, while still relying on a sizable GPU cluster for each self-improvement iteration
* * *
_Note: This analysis follows the Bevaya research note guidelines and classification taxonomy. For academic or production use, consult the original paper and verify claims through independent evaluation._