# PubLayNet: The 'ImageNet' of Document Layout Analysis > Large-scale document layout dataset with 360K+ automatically annotated pages from PubMed Central, serving as the standard pre-training corpus for layout models. - Authors: Bevaya AI Team - Published: 2019-09-01 - Updated: 2026-03-12 - Paper: Preprint (arXiv) - Paper: Published (ICDAR 2019) - Dataset: PubLayNet Data - Code: Toolkit - Model: Faster R-CNN - Model: Mask R-CNN - Source: Bevaya Labs, --- ## TL;DR PubLayNet is a large-scale dataset for document layout analysis, providing over **360,000 annotated pages** from PubMed Central scientific articles. By automatically aligning XML source text with PDF visual content, the authors created a dataset orders of magnitude larger than any prior public alternative, enabling effective training of deep object detection models. Its **5-class taxonomy** (Text, Title, List, Table, Figure) became the de facto standard across the field until more diverse datasets like DocLayNet emerged. --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$ The primary contribution is the dataset and the automated annotation pipeline. By demonstrating that models trained on large-scale automatically generated annotations achieve strong performance and transfer well to other domains, PubLayNet helped shift the field's focus toward scalable data synthesis. **Secondary:** $\Psi\_{\text{Evaluation}}$ The paper benchmarks Faster R-CNN and Mask R-CNN on the new dataset and runs two transfer learning experiments (ICDAR 2013 table detection and SPD insurance documents) to validate the dataset's usefulness as a pretraining source. ## What is the motivation? Prior to 2019, document layout analysis suffered from a severe data scarcity problem. Datasets like ICDAR 2013 or UNLV contained only hundreds of pages, insufficient for training deep networks like Faster R-CNN or Mask R-CNN, which require thousands of examples to generalize. The authors identified an opportunity in the duality of scientific publishing: PubMed Central (PMC) stores articles as both **PDF** (visual layout) and **XML** (logical structure). By systematically aligning the XML text with bounding boxes in the PDF, they could generate annotations for millions of pages without manual effort. ## What is the novelty? 1. **Scale:** At 360k+ pages across 6,500+ journals, it was orders of magnitude larger than any prior public dataset. 2. **Automated alignment pipeline:** A robust heuristic algorithm maps logical XML nodes to visual PDF tokens using fuzzy string matching. The maximum Levenshtein distance allowed for a match ($d\_{\max}$) adapts to the length of the target string ($l\_{\text{target}}$): $$ d\_{\max} = \begin{cases} 0.2 \cdot l\_{\text{target}} & \text{if } l\_{\text{target}} \le 20 \\\\ 0.15 \cdot l\_{\text{target}} & \text{if } 20 < l\_{\text{target}} \le 40 \\\\ 0.1 \cdot l\_{\text{target}} & \text{if } l\_{\text{target}} > 40 \end{cases} $$ Shorter strings tolerate proportionally more mismatches (up to 20%), while longer strings use a tighter threshold (10%), reflecting the expectation that longer sequences provide more matching context. 3. **Proof of transfer:** The paper demonstrates empirically that models pre-trained on this synthetic "silver" data generalize better to real-world document tasks than models pre-trained on ImageNet or COCO alone. ## What experiments were performed? Three experiments were designed to evaluate the dataset: 1. **Layout recognition on PubLayNet itself:** Faster R-CNN and Mask R-CNN (ResNeXt-101-64x4d backbone) trained on PubLayNet and evaluated on the development and test sets using mAP@IoU[0.50:0.95]. 2. **Table detection transfer to ICDAR 2013:** Models pretrained on the table-only subset of PubLayNet were fine-tuned on the 170 training pages from the ICDAR 2013 Table Competition and evaluated on the 238-page competition test set. Evaluated using precision, recall, and F1. 3. **Transfer to a different domain (SPD documents):** 20 Summary Plan Description (SPD) health insurance documents (2,131 pages) were manually annotated by the authors for Text, Table, and List. A 5-fold cross-document-validation was used to compare three initialization strategies: ImageNet backbone only, full COCO-pretrained model, and full PubLayNet-pretrained model. Zero-shot PubLayNet was also tested. All training used the Detectron (v1) framework. PDF pages were rasterized using `pdf2image`. ## What are the outcomes/conclusions? ### Layout Recognition (Table III) | Class | F-RCNN (Test) | M-RCNN (Test) | | :--- | ---: | ---: | | Text | 0.913 | 0.917 | | Title | 0.812 | 0.828 | | List | 0.885 | 0.887 | | Table | 0.943 | 0.947 | | Figure | 0.945 | 0.955 | | **Macro average** | **0.900** | **0.907** | Both models achieve mAP > 0.90 overall. Tables and Figures score highest (~0.94-0.96), attributed to their regular shapes and visual distinctiveness. Titles score lowest (~0.81-0.83), because titles are small and their appearance varies across journal templates. ### ICDAR 2013 Table Detection (Table IV) Fine-tuning a PubLayNet-pretrained F-RCNN on only 170 training pages achieves F1 = 0.968, matching the prior state of the art (Schreiber et al. 2017, F1 = 0.968) which required 1,600 samples from a general vision model. This demonstrates that pretraining on domain-relevant data substantially reduces the labeled data requirement. ### SPD Domain Transfer (Table V) On the SPD insurance documents, fine-tuning from PubLayNet outperforms COCO and ImageNet initializations for Text and List detection. For Table, COCO initialization is slightly stronger for F-RCNN, which the authors attribute to greater visual dissimilarity between SPD and PMC tables compared to SPD and PMC text. The zero-shot PubLayNet result is substantially below fine-tuned models (macro average ~0.47 vs. ~0.66), confirming that the domain gap between scientific and business documents is significant. ### Taxonomy Mapping PubLayNet defines five broad functional classes. Understanding their exact scope matters for downstream use: | Class | XML Sources | Key Limitations | | :--- | :--- | :--- | | **Text** | Body paragraphs, abstract, affiliations, footnotes, **figure & table captions** | Captions are labeled as Text, not merged into Figure/Table boxes. Inline section titles (where the title runs into the first line of the section) are also merged here. | | **Title** | Article title, **standalone (sub)section titles**, standalone figure/table labels | Standalone section headings ARE captured as Title. Only inline section titles are subsumed into Text. | | **List** | List elements | Nested (child) lists are annotated as a single object with the parent. | | **Table** | Main body of table only | Caption is a separate Text instance, not included in the Table box. | | **Figure** | Main body of figure only | Caption is a separate Text instance. Sub-figures are grouped into one box. | ## Mapping to Matter vs. Meaning Framework How does PubLayNet's 5-class taxonomy map to our [Matter vs. Meaning framework](../matter-vs-meaning/)? | PubLayNet Class | Visual Primitive | Logical Role | Notes | | :--- | :--- | :--- | :--- | | **Text** | `Text` | `Body` / `Caption` / `Footnote` / `Abstract` | Catch-all for all text regions. Captions are labeled as Text (not grouped with Figure/Table). Inline section titles are also merged here. Footnotes and abstracts are not separated. | | **Title** | `Text` | `Title` / `SectionHeader` | Covers both the main paper title and standalone section headings. Only inline section titles (where the title runs into the first paragraph line) are subsumed into Text. | | **List** | `Text` | `ListItem` | Nested child lists are annotated as a single object with the parent. No separation of individual list items. | | **Table** | `Table` | *(primitive only)* | Main table body only. Caption is a separate Text instance, not included in the Table bounding box. | | **Figure** | `Image` | `Figure` | Main figure body only. Caption is a separate Text instance. Sub-figures are grouped into one box. No distinction between charts, diagrams, and photographs. | **Coverage gaps:** PubLayNet has no explicit classes for `PageHeader`, `PageFooter`, `PageNumber`, `Footnote`, `Formula`, `Caption`, or `BibEntry`. All of these are collapsed into the `Text` class. The absence of `Formula` is by design: the annotation pipeline explicitly removes `tex-math` and `disp-formula` XML nodes before matching, because their PDF rendering diverges too much from the XML source text for fuzzy matching to work reliably. The `Caption` role is particularly notable: it exists as a separate bounding box but shares the `Text` label, making it indistinguishable from body paragraphs without post-processing heuristics (e.g., proximity to Figure/Table boxes). ## Reproducibility > **Note:** The official IBM Data Asset eXchange page is offline. The code depends on the deprecated [Detectron (v1)](https://github.com/facebookresearch/Detectron) framework, which requires older PyTorch versions. ### Models - **Architectures:** Faster R-CNN and Mask R-CNN. - **Backbone:** ResNeXt-101-64x4d, initialized from ImageNet pretrained weights. - **Pretrained weights:** Both Faster R-CNN and Mask R-CNN checkpoints released (Apache-2.0). ### Algorithms - **Training:** 180k iterations. Base learning rate 0.01, reduced by factor of 10 at 120k and 160k iterations. - **Batch size:** 8 GPUs, 1 image per GPU (effective mini-batch size of 8). - **PDF rendering:** `pdf2image` package to rasterize PDF pages. - **ICDAR fine-tuning:** Base learning rate 0.001, reduced by 10 at iteration 100 of 200. Confidence threshold selected by 5-fold cross-validation on the 170 training pages. ### Data - **Source:** 1,162,856 PMCOA articles downloaded 3 October 2018. - **Splits (journal-level partitioning):** - Training: 340,391 pages / 3,311,660 instances - Development: 11,858 pages / 127,815 instances (human-curated to remove egregious errors) - Testing: 11,983 pages / 131,775 instances (human-curated) - **Quality control:** Pages excluded if annotation coverage is below 99% for non-title pages, 90% for title pages. - **License:** CDLA-Permissive-1.0 covers the annotations. However, the page images are rendered from PDFs obtained via the PMC FTP service, which [explicitly states](https://pmc.ncbi.nlm.nih.gov/tools/ftp/) that individual PDFs are only available for non-commercially licensed articles. Since training requires images and annotations together, **PubLayNet is not suitable for commercial model training** regardless of the annotation license. - **Format:** COCO JSON with rasterized page images. ### Evaluation - **Metric:** mAP@IoU[0.50:0.95] (COCO standard), macro-averaged over classes. - **ICDAR 2013:** Official evaluation toolkit; precision/recall/F1 on 238 test pages. - **SPD transfer:** 5-fold cross-document-validation on 20 manually annotated documents (2,131 pages). ### Hardware - Not specified in the paper beyond "8 GPUs." ## BibTeX ```bibtex @inproceedings{zhong2019publaynet, title={PubLayNet: largest dataset ever for document layout analysis}, author={Zhong, Xu and Tang, Jianbin and Yepes, Antonio Jimeno}, booktitle={2019 International Conference on Document Analysis and Recognition (ICDAR)}, pages={1015--1022}, year={2019}, organization={IEEE}, url={https://arxiv.org/abs/1908.07836} } ```