# DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation > DocPTBench is a benchmark for evaluating document parsing and translation on phone-captured images with perspective distortion, lighting variation, and blur, revealing substantial performance gaps in current MLLMs and specialized parsers. - Authors: Hunter Heidenreich - Published: 2026-03-30 - Benchmark: DocPTBench (Dataset and Code) - Paper: Preprint (arXiv) - Source: Bevaya Labs, --- ## TL;DR DocPTBench is a benchmark of over 1,300 photographed document images designed to measure how well current models handle parsing and translation under real-world capture conditions. Experiments show that shifting from digital-born to photographed documents causes average parsing edit distance to increase by roughly 18% for general MLLMs and 25% for specialized OCR models, with translation BLEU scores dropping by about 12%. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Evaluation}}$. The central contribution is a new benchmark and evaluation protocol for photographed document parsing and translation. The paper does not introduce a new model or training procedure; it systematically measures the robustness gap in existing systems using a carefully constructed dataset. **Secondary:** $\Psi\_{\text{Resource}}$. DocPTBench itself is a released community resource: a dataset of over 1,300 labeled photographed document images with human-verified annotations across eight language-pair translation directions, along with evaluation code. ## What is the motivation? Document parsing and translation benchmarks have historically focused on digital-born or high-quality scanned documents with clean geometric structure. Benchmarks such as OmniDocBench, FoxPage, and olmOCR-Bench represent this well-structured distribution, as do translation benchmarks like DoTA and DITrans. The result is that reported model numbers do not reflect what happens when a user points a phone at a document in practice. Real-world document photographs introduce a range of degradations absent from these benchmarks: perspective distortion from off-axis capture, page curvature and physical deformation, motion blur, uneven or harsh lighting, shadows, and overexposure. While the WildDoc benchmark incorporated in-the-wild images, it focused on visual question answering rather than structured parsing, and provided no translation annotations. The community therefore lacked a unified benchmark to measure end-to-end parsing and translation robustness under photographed conditions. ## What is the novelty? DocPTBench introduces three features that distinguish it from prior benchmarks: 1. **Unified dual-task evaluation.** A single dataset supports both document parsing and document translation evaluation. Prior benchmarks either covered parsing or translation in isolation; no earlier resource combined both tasks with photographed images. 2. **Three-tier image collection.** DocPTBench organizes its images into an Original set (981 born-digital documents from OmniDocBench), a Photographed set (981 simulated photographs plus 400 physically photographed images of 100 documents under four challenging conditions), and an Unwarping set (the Photographed images after a commercial geometric rectification API). This structure allows controlled ablation of geometric versus photometric degradation. 3. **Broad multilingual translation coverage.** Eight translation directions are supported: En-Zh, En-De, En-Fr, En-Ru, and their Zh-originated counterparts. Translation ground truth was produced by Qwen-Max generation followed by rigorous manual verification. The evaluation metrics for parsing follow OmniDocBench conventions, using Levenshtein-based edit distances for text ($\text{Text Edit}$), formulas ($\text{Formula Edit}$), tables ($\text{Table Edit}$), and reading order ($\text{Read Order Edit}$), plus a Tree-Edit-Distance-based Similarity score (TEDS) for table structure. Translation is assessed with BLEU, chrF, METEOR, and STEDS. Lower edit distance indicates better parsing; higher BLEU indicates better translation. ## What experiments were performed? The paper benchmarks two model categories across all three image conditions: **Expert OCR models** (evaluated on parsing only): PaddleOCR-VL, MinerU2.5, dots.ocr, MonkeyOCR, DeepSeek-OCR, olmOCR2, Dolphin, olmOCR, OCRFlux, SmolDocling, Nanonets-OCR, and Nanonets-OCR2. These systems are purpose-built for document content extraction. **General MLLMs** (evaluated on both parsing and translation): Gemini 2.5-Pro, Qwen-VL-Max, GLM-4.5v, Kimi-VL, and Doubao-Seed-1.6-vision. Qwen2.5-VL-72B is also evaluated on parsing in the supplementary full results table. For translation, four smaller open-source models were also included: Qwen3-VL-4B, Qwen2.5-VL-3B, InternVL3-2B, and InternVL3.5-2B. Parsing is evaluated across the Original, Photographed, and Unwarping conditions. Translation is evaluated with: - A text-only baseline (the source Markdown text is provided directly, bypassing vision) to establish an upper bound for translation quality independent of OCR. - Original and Photographed image inputs. - Two prompting strategies: a **Simple prompt** (direct image-to-target-language output) and a **Chain-of-Thought (CoT) prompt** (explicit instruction to first parse the document into source-language Markdown, then translate). The CoT formulation decouples perception from translation and helps diagnose where errors originate. ## What are the outcomes/conclusions? **Parsing degradation is severe and widespread.** Across the tested expert models, transitioning from Original to Photographed documents produces average overall edit distance increases far exceeding typical benchmark variance. DeepSeek-OCR's Overall Edit Score (En) deteriorates from 13.4 to 54.4; SmolDocling's rises from 49.3 to 90.1. General MLLMs fare comparatively better, with an average increase of roughly 18%, while expert models average roughly 25%. The authors attribute the relative resilience of MLLMs to training on more diverse and noisy data. **Geometric rectification is necessary but not sufficient.** Processing photographed images through a commercial unwarping API before evaluation consistently recovers substantial performance. For example, DeepSeek-OCR's Overall Edit Score (En) improves from 54.4 back to 22.1 after unwarping, and PaddleOCR-VL's Table TEDS (En) rebounds from 54.3 to 82.5. However, a consistent gap remains between Unwarping and Original scores. The authors interpret this as evidence that photometric degradations (blur, uneven lighting) are not addressed by geometric correction and continue to impair parsing quality. **Translation exhibits a large modality gap.** Comparing text-only inputs to image inputs on the same documents reveals a large gap. Qwen-VL-Max's BLEU score on En-Zh drops from 69.41 (text) to 41.04 (image, original). For some models on low-resource scripts, the collapse is near-total: InternVL3-2B's BLEU on En-Ru falls from 37.09 (text) to 8.15 (image, original), suggesting that complex scripts compound the difficulty. **Photographed images compound translation errors.** When the image source is photographed rather than digital-born, BLEU scores drop a further average of 12% across models and directions. This degradation occurs consistently whether a simple or CoT prompt is used, indicating that visual image quality is the primary bottleneck rather than the prompting strategy. **CoT prompting mitigates some modality gap.** For most models and language pairs, the CoT strategy (parse first, then translate) outperforms the simple direct approach. For example, Qwen-VL-Max's En-Zh BLEU on original images rises from 41.04 with a simple prompt to 47.60 with CoT. The benefit, while consistent, does not close the gap to text-only baselines. **Dual bottleneck for translation.** The paper identifies two independent constraints on end-to-end translation quality: the model's visual parsing capability and its underlying translation competence. Doubao-Seed-1.6-vision achieves strong parsing scores but delivers only modest translation quality because its text-only translation performance is weaker than competing models. This finding suggests that improving either component in isolation will not fully solve end-to-end translation. **Gemini 2.5-Pro shows strongest overall robustness.** Across parsing, translation, and photographed conditions, Gemini 2.5-Pro exhibits the smallest performance degradation among the general MLLMs tested, dropping only 3.4 Overall Edit Score (En) points on photographed images compared to original. ## Reproducibility ### Models No new model weights are introduced. DocPTBench evaluates existing, externally released systems. All evaluated models are referenced by their original papers and repositories. Gemini 2.5-Pro was accessed via API; the open-source models (Qwen series, InternVL series) are available on HuggingFace under their respective licenses. ### Algorithms No training is performed. The evaluation pipeline applies each model's default inference settings. Two prompting strategies are systematically varied: the Simple prompt and the Chain-of-Thought prompt. The exact prompts are illustrated in Fig. 5 of the supplementary material (page 14 of the paper). The commercial unwarping API used for the Unwarping tier is provided by textin.com (https://www.textin.com/market/detail/crop_enhance_image). Inference scripts for each prompting mode (simple, CoT, and text-only) are provided in `code/translation/model_infer/` in the repository, organized by source language. The translation evaluation script is at `code/translation/evaluate_metric/evaluate.py`. Parsing evaluation uses the OmniDocBench evaluation harness (v1_0 branch of https://github.com/OmniDocBench/OmniDocBench); the repository documentation (`docs/parsing.md`) includes a step-by-step walkthrough for reproducing parsing results with any model. ### Data The DocPTBench dataset is built on 981 documents from OmniDocBench. The Photographed tier combines: - 981 photographed-simulation images (exact simulation pipeline details are in the paper and code repository). - 400 physically photographed images of 100 source documents, each photographed under four conditions (strong illumination, shadows, perspective distortion, physical wear such as folds or wrinkles). The 1,381 Unwarping images are derived from the Photographed set using a commercial API. Translation ground truth for all eight language directions was generated by Qwen-Max and then manually verified and corrected. The dataset and code are released under Apache-2.0 at https://github.com/Topdu/DocPTBench. The dataset is also hosted on HuggingFace (`topdu/DocPTBench`) and ModelScope (`topdktu/DocPTBench`) and can be downloaded with: ```bash huggingface-cli download topdu/DocPTBench --local-dir ./DocPTBench --repo-type dataset ``` The downloaded dataset contains a ground-truth file (`DocPTBench_combined.json`) and four image directories: `images_synreal` (981 simulated photographed images), `images_synreal_dewarp` (their unwarped counterparts), `images_pic_synreal` (400 physically photographed images), and `images_pic_dewarp` (their unwarped counterparts). Translation ground truth is stored under `translation_gt/src_en/` and `translation_gt/src_zh/` for the eight language pairs. ### Evaluation Parsing metrics follow OmniDocBench conventions: Levenshtein-based Text Edit, Formula Edit, Table Edit, and Read Order Edit (lower is better), plus TEDS for table structure (higher is better). An Overall Edit score aggregates these. Translation metrics are BLEU, chrF, METEOR, and STEDS. Baselines cover 18 evaluated systems in total across the supplementary full results table (Tab. 5). The three-tier design (Original/Photographed/Unwarping) controls for degradation type. No statistical significance tests or error bars over multiple runs are reported; results reflect single-run evaluations. The authors acknowledge that the benchmark builds on OmniDocBench's digital-born subset, so domain coverage reflects OmniDocBench's composition (academic papers, invoices, forms, magazines, and others). ### Hardware No training hardware is relevant. Inference hardware specifics are not reported for the evaluated models. The paper does not report GPU-hours, inference latency, or throughput. ## BibTeX ```bibtex @article{du2025docptbench, title={DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation}, author={Du, Yongkun and Chen, Pinxuan and Ying, Xuye and Chen, Zhineng}, journal={arXiv preprint arXiv:2511.18434}, year={2025} } ```