# olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models > Analysis of olmOCR, a 7B vision-language model for PDF-to-text linearization using document anchoring and GPT-4o distillation - Authors: Hunter Heidenreich - Published: 2025-02-01 - Updated: 2026-03-04 - Paper: olmOCR Paper - Code: olmOCR Code - Demo: olmOCR Demo - Model: olmOCR-7B-0825 Model - Source: Bevaya Labs, --- ## TL;DR olmOCR is a PDF-to-text linearization toolkit centered on a 7B vision-language model fine-tuned from Qwen2-VL-7B-Instruct. The system converts PDFs (both born-digital and scanned) into clean plain text suitable for language model development. The core innovation is "document anchoring," a prompt technique that injects PDF-extracted text blocks with coordinates to reduce reading-order errors and hallucinations. On olmOCR-Bench (1,402 PDFs, 7,010 unit tests), the anchored model achieves 75.5% overall pass rate with an estimated cost of \\$176 per million pages on L40S hardware. * * * ## What kind of paper is this? Primarily **$\\Psi\_{\\text{Resource}}$**, with meaningful **$\\Psi\_{\\text{Method}}$** and **$\\Psi\_{\\text{Evaluation}}$** components. - **Dominant: $\\Psi\_{\\text{Resource}}$**: The headline deliverables are (1) a training dataset (olmOCR-mix-0225, 260k pages), (2) a benchmark suite (olmOCR-Bench with 7,010 unit tests), and (3) a released fine-tuned model with inference code. The paper emphasizes enabling reproducible PDF processing for the research community. - **Secondary: $\\Psi\_{\\text{Method}}$**: Document anchoring (layout-aware prompt engineering with extracted text blocks and bounding boxes) is presented as a novel technique for reducing VLM hallucinations in document contexts. - **Secondary: $\\Psi\_{\\text{Evaluation}}$**: The benchmark design itself represents a substantial contribution: deterministic unit-test-style pass/fail rules that avoid LLM-as-judge biases. ## What is the motivation? PDFs encode rendering primitives (glyphs with positions) rather than semantic text structure or ground-truth reading order, making them challenging inputs for language model training and document-grounded inference. - **Low-fidelity extraction hurts LM training**: Poor OCR quality can degrade training stability and downstream task performance. Cascading errors in document workflows compound the problem. - **Proprietary solutions are expensive**: The authors cite GPT-4o API costs exceeding \\$6,200 per million pages as a barrier to large-scale PDF processing for open research. - **Existing open tools lack validation**: Pipeline systems (Marker, MinerU) and open VLMs have no standardized, reproducible benchmark for comparing PDF linearization quality. ## What is the novelty? **Document anchoring:** The system extracts text blocks with bounding boxes from the PDF using `pypdf`, then injects a subset of these blocks (with coordinates and image placeholders) into the VLM prompt alongside the page image. This provides layout hints that help the model maintain correct reading order and reduce hallucinated content. **Silver-data recipe:** The authors generate approximately 260k page OCR targets with GPT-4o using structured JSON output, then fine-tune Qwen2-VL-7B-Instruct into olmOCR-7B-0225-preview. The training data is filtered for English-only content and sampled up to 3 pages per PDF. **Unit-test benchmark:** olmOCR-Bench uses deterministic binary checks (presence/absence/order/table structure/math formula accuracy) designed to be evaluator-model-independent, explicitly addressing concerns about LLM-judge bias in document evaluation. ## What experiments were performed? **olmOCR-Bench evaluation (Table 4):** 1,402 PDFs spanning 7,010 unit tests across categories including text presence/absence, natural reading order, table accuracy, and math formula rendering. Baselines include open pipeline tools (Marker v1.7.5, MinerU v1.3.10), proprietary OCR systems (Mistral OCR, GPT-4o, Gemini Flash), and Qwen2-VL variants with and without document anchoring. **Cost analysis (Table 6):** Estimates tokens/sec, pages/USD, and cost per million pages for multiple systems under specified hardware assumptions (L40S at \\$0.79/hr, H100 at \\$2.69/hr), including GPT-4o API vs. batch pricing. **Downstream LM pretraining ablation (Table 5):** Continues pretraining OLMo-2-1124-7B for 50B tokens using PDFs linearized by olmOCR vs. Grobid+rules (peS2o baseline), then evaluates on standard LM benchmarks (MMLU, ARC-Challenge, DROP, etc.). **Ablations on anchoring:** Compares Qwen2-VL variants and GPT-4o with and without document anchoring prompts to isolate the contribution of layout-aware context. ## What are the outcomes/limitations? **Key results:** - **olmOCR-Bench overall pass rate: 75.5%** (95% CI via bootstrap) for the anchored model, outperforming Marker (70.1%), MinerU (61.5%), Mistral OCR (72.0%), GPT-4o (68.9%), and Qwen2.5-VL (65.5%). - **Cost estimate: \\$176 per million pages** on L40S hardware (assumes 12% retry rate for JSON parsing failures and degenerate repetition), compared to \\$6,240 for GPT-4o batch API and \\$596 for MinerU on L40S. - **Downstream pretraining impact: +1.3 percentage points** average improvement across benchmark suite when replacing Grobid+rules PDF processing with olmOCR (53.9% $\\rightarrow$ 55.2% average score). **Limitations and open questions:** - **English-only training data**: Explicit filtering via Lingua removes non-English documents, limiting multilingual generalization. The 3-page-per-PDF sampling may bias coverage toward short documents. - **Teacher model inheritance**: Fine-tune targets are GPT-4o outputs, so the student model inherits teacher quirks and evaluation aligns to teacher behavior rather than purely human preference. - **Reliability challenges**: The paper describes retries for JSON parsing failures and degenerate repetition (mitigated with higher temperature $\\tau = 0.8$). The 12% retry rate impacts throughput in production. - **Benchmark generalization**: olmOCR-Bench is carefully engineered but remains an in-house suite. The document distribution (55.9% academic papers) may not reflect other production use cases. External validation on held-out benchmarks is not the core claim. - **Long document handling**: Sampling only a few pages per PDF during training raises questions about failure modes on documents with complex multi-page structure. > **Contrast to olmOCR 2:** This initial olmOCR release (February 2025) focuses on GPT-4o distillation with document anchoring prompts, whereas [olmOCR 2](2025-10-olmocr2.md) (October 2025) shifts to RL-based policy improvements using unit-test rewards from the same benchmark suite. olmOCR 2 reports 81.2% pass rate (vs. 75.5% here) and eliminates structured JSON output in favor of direct plain-text generation, improving robustness and reducing retry overhead. * * * ## Model **Base model:** Qwen2-VL-7B-Instruct fine-tuned into **olmOCR-7B-0225-preview** (7B parameters). **Output format:** The model generates structured JSON during training (matching GPT-4o's synthetic target schema), though the inference prompt can be simplified to return "plain text representation" directly. The JSON schema includes fields for `primary_language`, rotation validity/correction, boolean flags for tables/formulas/diagrams, and `natural_text`. ### Document anchoring representation The prompt construction pipeline: 1. Extract PDF text blocks and image blocks with bounding boxes using `pypdf`. 2. Sample a subset of blocks (preferring start/end of document) to inject into the prompt. 3. If the prompt exceeds 8,192 tokens, regenerate with reduced character limits per block. 4. Concatenate the page image, selected text blocks (with `RAW_TEXT_START/END` delimiters), and generation instructions. The authors describe this as "layout-aware retrieval" that provides the VLM with content hints and rough ordering cues, reducing hallucinations and reading-order errors. ## Data ### Training set: olmOCR-mix-0225 **Size:** 102,825 unique PDFs / 258,641 pages total (Table 1). **Source breakdown:** - **Web PDFs:** 96,929 documents / 240,940 pages (drawn from internal crawl of over 240 million PDF documents). - **Internet Archive books:** 5,896 documents / 17,701 pages. **Filtering and sampling:** - Remove non-English documents (Lingua language detector). - Filter parsing failures, spam keywords (explicit list), fillable forms, and documents with insufficient extractable text. - Sample up to 3 pages per PDF. **Document type distribution (Table 2, manual annotation):** - Academic papers: 55.9% - Brochures: 11.2% - Legal documents: 10.2% - Books: 6.8% - Table-heavy: 5.6% - Diagram-heavy: 4.7% - Slideshows: 1.9% - Other: 3.7% ### Synthetic labeling with GPT-4o The teacher model (GPT-4o) receives prompts with rules including: - Preserve natural reading order. - Use LaTeX for equations, Markdown for tables. - Remove headers/footers when appropriate. - Handle handwritten annotations if present. - Do not hallucinate content. - Output `null` if no text is present. The structured JSON schema captures language, rotation metadata, content flags, and the final `natural_text` field. ## Algorithms / Training ### Fine-tuning configuration - **Effective batch size:** 4 - **Optimizer:** AdamW - **Learning rate:** $1 \\times 10^{-6}$ - **Schedule:** Cosine annealing - **Steps:** 10,000 (approximately 1.2 epochs over 258k pages) - **Hardware:** 8 $\\times$ H100 80GB - **Runtime:** 16 node-hours - **Context truncation:** Training examples truncated to 8,192 tokens; loss masked to final response tokens only. ### Inference-time prompt The production prompt for olmOCR-7B-0225-preview is minimal: - Page image - Prior extracted raw text blocks (with `RAW_TEXT_START/END` delimiters) - Instruction: "return the plain text representation of this document as if you were reading it naturally" No explicit JSON schema enforcement at inference (though the model was trained to produce JSON). ## Evaluation ### olmOCR-Bench construction **Design philosophy:** Deterministic unit tests with binary pass/fail outcomes, avoiding LLM-as-judge and fuzzy reference matching. The authors explicitly cite concerns about evaluator bias and non-reproducibility in document benchmarks. **Size:** 1,402 PDFs / 7,010 tests. **Test categories (Table 3, selected examples):** - **Text presence (TP):** Fuzzy string matching to verify key content appears. - **Text absence (TA):** Verify headers/footers are removed. - **Natural reading order (NR):** Check relative ordering of text segments. - **Table accuracy (TT):** Validate neighbor-cell constraints in Markdown/HTML tables (HTML required for rowspan/colspan). - **Math formula accuracy (MF):** KaTeX-rendered symbol layout matching. **Document type distribution in benchmark (Table 3):** - arXiv Math (AR): 2,927 formula tests - Multi-Column (MC): 884 reading-order tests - Tables (TT): 1,020 table structure tests - Legal-Tabular-Tiny (LTT): 213 combined tests - Open Street Map (OSM): 123 specialized tests - Table Accuracy (TA): (additional table tests, count not specified in draft) ### Benchmark results **Overall pass rate (95% CI, Table 4):** | System | Pass Rate | | ----------------------------- | --------- | | Marker v1.7.5 | 70.1% | | MinerU v1.3.10 | 61.5% | | Mistral OCR | 72.0% | | GPT-4o (no anchor) | 68.9% | | GPT-4o (anchored) | 69.9% | | Qwen 2.5 VL (no anchor) | 65.5% | | **olmOCR v0.1.75 (anchored)** | **75.5%** | **Category-wise highlights:** - Anchored olmOCR leads on AR (74.9%), MC (78.3%), and LTT (73.3%). - Math-heavy OSM and table-heavy TA categories show more competitive performance across baselines. ### Cost comparison **Cost per million pages (Table 6, selected systems):** | System | Cost/M pages | | -------------------- | ------------ | | GPT-4o API | \\$12,480 | | GPT-4o Batch | \\$6,240 | | Marker (H100) | \\$1,484 | | MinerU (L40S) | \\$596 | | Gemini Flash 2 Batch | \\$249 | | **olmOCR (L40S)** | **\\$176** | | olmOCR (H100) | \\$178 | **Assumptions:** L40S at \\$0.79/hr, H100 at \\$2.69/hr, 12% retry rate for JSON parsing failures and repetition handling. ### Downstream LM pretraining ablation **Setup:** Continue pretraining OLMo-2-1124-7B for 50B tokens on PDFs linearized with olmOCR vs. Grobid+rules (peS2o baseline). Evaluate on standard benchmarks. **Results (Table 5):** - **Baseline (Grobid+rules) average score:** 53.9% - **olmOCR average score:** 55.2% - **Improvement:** +1.3 percentage points This suggests OCR/linearization quality can measurably impact LM performance at the 50B-token pretraining scale. ## Hardware / Production ### Inference pipeline and robustness **Orchestration:** The system uses SGLang for serving and chunks work into batches of approximately 500 pages, coordinated through shared cloud storage (S3). **Reliability heuristics:** - **Prompt regeneration:** If token count exceeds 8,192, reduce character budget for sampled text blocks and rebuild prompt. - **Retry on JSON parse failure:** Re-run generation with adjusted parameters. - **Rotation correction:** Use PDF metadata to detect and correct page orientation. - **Degenerate repetition mitigation:** Retry with higher temperature ($\\tau = 0.8$) if the model produces repetitive output; fall back to alternative strategies if retries fail. The reported 12% retry rate indicates non-trivial overhead in production deployments, though the authors frame this as necessary for quality assurance. ### Throughput estimates The paper provides pages-per-hour and cost-per-page calculations based on measured token throughput on L40S and H100 hardware, factoring in the retry rate. Specific throughput numbers are embedded in the cost estimates (Table 6). ## Implementation sketch ```python # Conceptual pipeline based on paper description for page in pdf: blocks = pypdf.extract_text_and_images_with_bboxes(page) selected = sample_blocks_preferring_doc_start_end(blocks) prompt = build_prompt(page_image, selected) if prompt_tokens > 8192: selected = regenerate_selected_blocks_with_lower_char_budget() prompt = build_prompt(page_image, selected) output = vlm.generate(prompt) if parse_fail_or_repetition(output): output = retry_with_adjustments(prompt, tau=0.8) plaintext_pages.append(output) return concatenate(plaintext_pages) ``` This matches the described use of `pypdf` for block extraction, selective sampling, prompt regeneration under token limits, and retry logic for robustness. ## Notes and open questions **Observations:** - Document anchoring is essentially layout-aware retrieval: injecting PDF primitives (text + bboxes) into the VLM prompt. The technique is straightforward to implement but depends heavily on PDF parser quality and block selection policy. - The benchmark design deliberately avoids LLM-judge evaluators, addressing real reproducibility concerns in document evaluation. The unit-test approach is deterministic but may miss nuanced quality issues that human evaluators would catch. - The downstream pretraining ablation (+1.3 points at 50B tokens) provides evidence that OCR quality matters for LM training, at least in this experimental setup. **Open questions:** - **Robustness beyond English academic PDFs:** How do the reported gains generalize to document distributions that are not 56% academic papers, and to multilingual documents (given explicit English-only filtering)? - **Anchoring vs. model quality:** How much of the Table 4 advantage comes from model weights vs. prompt engineering (anchoring) vs. post-processing heuristics (rotation, retries, truncation)? Ablations on GPT-4o and Qwen show modest anchoring gains (~1 point), but the fine-tuned model may benefit more. - **Long document failure modes:** Sampling only 3 pages per PDF during training may leave gaps in handling complex multi-page structures (cross-references, continued tables, section numbering). Production use cases likely encounter these patterns frequently. - **Structured output overhead:** The JSON schema adds tokens and introduces parsing failures (12% retry rate). olmOCR 2's shift to plain-text generation suggests this was a recognized pain point.