# MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding > Analysis of MosaicDoc (arXiv 2025), a 72K-image bilingual (Chinese/English) benchmark sourced from newspapers and magazines, featuring block- and line-level reading order annotations alongside DocVQA, OCR, and localization tasks. - Authors: Bevaya AI Team - Published: 2026-04-02 - Paper: Preprint (arXiv) - Code: MosaicDoc Code - Source: Bevaya Labs, --- ## TL;DR MosaicDoc is a 72K-image bilingual (Chinese and English) benchmark sourced from newspapers and magazines, annotated with block- and line-level reading order, OCR, DocVQA, and content-aware localization. It is constructed using DocWeaver, a multi-agent LLM pipeline that computes correct reading order explicitly for non-Manhattan and multi-column layouts. Evaluation of 13 models reveals that all current models, including Gemini-2.5, fail to correctly order text across columns in complex layouts. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: The headline contribution is the MosaicDoc benchmark dataset itself, along with the DocWeaver pipeline used to construct it. The paper devotes most of its space to data collection, annotation methodology, quality assurance, and dataset statistics. **Secondary:** $\Psi\_{\text{Evaluation}}$: The benchmark evaluation of 13 models establishes baseline performance and identifies systematic weaknesses, with meaningful diagnostic findings on multi-span reasoning, dense OCR, and multi-column reading order. ## What is the motivation? Existing VRDU benchmarks have three converging weaknesses: 1. **English-centric and visually simple.** Benchmarks like VisualMRC and Docmatix rely on simple single-column documents. Chinese datasets like XFUND and DuReadervis replicate this simplicity. 2. **No reading order annotations.** Models trained without reading order supervision default to left-to-right, top-to-bottom heuristics, which fail on multi-column and non-Manhattan layouts. 3. **Single-task.** Prior reading order datasets (ReadingBank, ROOR) lack VQA annotations; VQA datasets lack reading order ground truth. The authors argue this gap is particularly severe for newspapers and magazines, which feature irregular column layouts, cross-page articles, and dense bilingual text. ## What is the novelty? ### DocWeaver: Multi-Agent Annotation Pipeline DocWeaver automates benchmark construction in three phases: **Phase 1: Document Decomposition and Structuring** - PDF parsing via PyMuPDF and pdfminer.six; encoding recovery via chardet. - Layout detection using PP-DocLayout fine-tuned on the M$^3$Doc magazine dataset (13 element types). - Multi-engine OCR with weighted-voting for final text: $$ \hat{s} = \arg\max\_{t \in \mathcal{T}} \sum\_{i=1}^{N} w\_i p\_i \mathbf{1}[s\_i = t] $$ where $\mathcal{T}$ is the set of all candidate strings, $w\_i$ is engine reliability weight, and $p\_i$ is the confidence score. - Table structure via TableStructureRec (HTML output). - Cross-page linking via DeepSeek semantic similarity (threshold 0.8, up to 4 pages merged). **Reading Order Modeling** Two strategies depending on data availability: - *Structured data* (magazines, Chinese newspapers with HTML): fuzzy matching of PDF text spans to HTML template, constrained by bounding box and contextual proximity. Reading order derived from the validated HTML sequence. - *Unstructured data* (English newspapers): binarization-based layout line detection partitions pages into rectangular article blocks; within each block, line-level reading order is inferred from PDF metadata (font features, positional cues) and semantic coherence. Pages with more than five detected reading order errors (by distance-based heuristics) are discarded entirely. **Phase 2: QA Generation** GPT-4o and DeepSeek-R1 generate QA pairs using specialized prompts for text, tables, and charts. QWQ-32B serves as an auxiliary discriminator to promote multi-span questions requiring cross-sentence reasoning. **Phase 3: Quality Assurance** Five-criteria hallucination guardrail scored by QWQ (Completeness, Consistency, Conciseness, Clarity, Inference). Only pairs passing a high threshold on all criteria are retained. Retention rates: 87.7-89.8% across subsets (Table 2). A 200-image human-validated gold subset is used for all reported evaluations. ### MosaicDoc Benchmark - **Scale:** 72.3K images (42.7K magazines, 29.6K newspapers), 620K+ QA pairs. - **Languages:** Chinese and English from 196 publishers across 24 domains. - **Tasks:** DocVQA (single-span, multi-span, table, chart), word- and line-level OCR, block- and line-level reading order prediction, content-aware localization. - **Layout complexity:** Average tokens per document exceeds prior VQA datasets by a wide margin (magazines: ~1,075; newspapers: ~3,558 vs. ReadingBank ~315). Low sentence-level BLEU scores versus left-to-right baselines confirm layout complexity. ## What experiments were performed? ### Models Evaluated 13 models in three tiers: - **Expert Models:** Donut (253M), ViTLP (259M), LayoutReader (line-level bbox input only). - **Expert VLMs:** Vary-7B, TextMonkey-7B, mPLUG-DocOwl2-7B, GOT-OCR-0.5B, olmOCR-8B. - **General VLMs:** CogVLM2-19B, InternVL3-9B, Qwen2.5-VL-7B, GPT-4o (API), Gemini-2.5 (API). All evaluated zero-shot on A100 GPUs (80GB) via vLLM at maximum supported resolution. ### DocVQA (ANLSL Metric) Best overall performance: Gemini-2.5 (61.3/68.8 Mag., 62.5/59.8 News. en/zh). All models drop sharply on multi-span questions; even Gemini-2.5 sees a ~10-point decline from single-span to multi-span. Expert VLMs with token reduction (TextMonkey, mPLUG-DocOwl2) collapse on dense layouts. ### Page-Level OCR (CRR / OCRR) Gemini-2.5 dominates (89.4/87.3 CRR for magazines en/zh; 87.9/66.6 for newspapers). All models degrade significantly on Chinese newspapers. A common failure mode: models enter repetitive generation after processing a fraction of a dense page, collapsing CRR. ### Reading Order Prediction (Micro-F1) Evaluated on text line sequences. Results from Table 5: | Model | Mag. F1 (en/zh) | News. F1 (en/zh) | |:---|:---|:---| | LayoutReader | -- / -- | -- / -- | | olmOCR | 81.9 / 71.4 | 33.8 / 0.71 | | InternVL3 | 72.2 / 76.6 | 63.9 / 41.2 | | Qwen2.5-VL | 65.4 / 75.7 | 50.9 / 57.2 | | Gemini-2.5 | 87.9 / 89.4 | 87.9 / 56.3 | All models exhibit high precision but low recall: correct local sequences within a column, but failure to capture the full page or bridge across columns. Gemini-2.5 achieves the best results but still fails to establish correct cross-column ordering in multi-column newspaper layouts. ## What are the outcomes/conclusions? 1. **Complex layouts break current models.** All 13 models degrade substantially on newspaper layouts relative to magazine layouts, with Chinese newspaper OCR near zero for several models. 2. **Multi-span reasoning is a universal weakness.** No current model handles multi-span QA reliably on dense documents. 3. **Reading order recovery fails at the column boundary.** High-precision/low-recall patterns indicate models read within columns correctly but cannot establish inter-column reading order. This directly degrades OCR and VQA tasks that depend on correct full-page sequencing. 4. **General VLMs outperform specialized Expert VLMs** on MosaicDoc, suggesting that domain-specific pre-training on simpler documents does not transfer to complex layouts. ### Limitations - The human-validated gold set is only 200 images; the full 72K set uses automated annotations whose error rate is not independently characterized. - Reading order ground truth for unstructured sources (English newspapers) is derived via heuristics; no inter-annotator agreement study is reported. - The dataset is accepted at AAAI 2026 but published as a preprint; full release ("as soon as possible") was pending at submission time. - No reported training set; MosaicDoc is an evaluation-only benchmark. Training data for reading order on complex layouts remains an open gap. ## Reproducibility ### Models No new model weights introduced. All evaluated models are existing releases. ### Algorithms DocWeaver pipeline code at [github.com/DOCLAB-SCUT/MosaicDoc](https://github.com/DOCLAB-SCUT/MosaicDoc) (Apache-2.0). Full release pending as of arXiv submission. ### Data - Source: Official public websites, content aggregation platforms, open-source PDF repositories (compliance details in Appendix D of the paper). - 196 publishers, 24 domains, Chinese + English. - Gold evaluation subset: 200 human-validated images. Full dataset release pending. - Reading order annotations: HTML-derived for structured sources; heuristic for unstructured. No per-source license breakdown provided in the main text. ### Evaluation - DocVQA: ANLSL (Average Normalized Levenshtein Similarity for Lists). - OCR: CRR (ground-truth normalized), OCRR (output-normalized precision). - ROP: Micro-F1 on text line sequences. - Zero-shot evaluation throughout; no fine-tuning baselines. - No error bars or significance tests. ### Hardware - Evaluation: NVIDIA A100 80GB via vLLM framework. - Training hardware: not applicable (evaluation-only benchmark paper). ## BibTeX ```bibtex @article{chen2025mosaicdoc, title={MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding}, author={Chen, Ketong and Chen, Yuhao and Xue, Yang}, journal={arXiv preprint arXiv:2511.09919}, year={2025} } ```