# DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis > DocLayNet provides 80k human-annotated pages across 6 diverse domains (including Finance and Law), establishing a rigorous benchmark for layout analysis models. - Authors: Bevaya AI Team - Published: 2022-08-01 - Updated: 2026-03-15 - Paper: Preprint (arXiv) - Paper: Published (ACM KDD 2022) - Dataset: DocLayNet - Code: GitHub Repository - Source: Bevaya Labs, --- ## TL;DR DocLayNet is a human-annotated document layout dataset containing **80,863 pages** from six diverse domains: **Financial Reports, Scientific Articles, Laws & Regulations, Government Tenders, Manuals, and Patents**. Unlike previous large-scale datasets (like PubLayNet or DocBank) that rely on automatic matching of XML/LaTeX to PDFs, DocLayNet provides human annotations for **11 distinct classes** (including `Page-header`, `Page-footer`, `Formula`, `Footnote`, and `Section-header`). --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$ The primary contribution is the dataset itself, which addresses the lack of high-quality, human-annotated layout data for non-scientific documents. It provides a rigorous benchmark for evaluating layout analysis models on complex, real-world documents. **Secondary:** $\Psi\_{\text{Evaluation}}$ The authors perform extensive baselining to measure the "human ceiling" (inter-annotator agreement) vs. model performance, demonstrating that even top baseline models lag behind human layout perception on average, though top models exceed human agreement on the most visually distinctive classes. ## What is the motivation? Prior datasets like **PubLayNet** (~300K pages in the authors' rounding; ~358K pages in the actual release) and **DocBank** (500K pages) relied on **weak supervision** (automatically aligning PDF bounding boxes with source XML/LaTeX files). While scalable, this approach has two critical limitations: 1. **Domain Restriction:** They are almost exclusively scientific papers, which have clean, predictable columns and few complex layouts. 2. **Label Noise:** Automatic matching often produces loose bounding boxes or misses elements entirely, creating a noisy ground truth. The authors built DocLayNet to address a generalization problem: models trained on PubLayNet fail on financial reports, invoices, or legal documents. By providing human-verified distinctions (e.g., distinguishing a "Footnote" from body "Text"), the dataset requires models to learn layout semantics rather than fitting to the uniform templates of scientific publishing. ## What is the novelty? 1. **Diversity (The "Big 6"):** It covers six distinct document styles, covering layouts that penalize models overfit to scientific articles: * **Financial Reports:** Annual reports with complex tables/branding, including both free-style formats and formal SEC filings. * **Scientific Articles:** Standard double-column layouts (comparable to PubLayNet). * **Laws & Regulations:** Dense text, numeric lists. * **Government Tenders:** Forms, lists, tables. * **Manuals:** Graphic-heavy instruction booklets. * **Patents:** Unique technical flow with diagrams. 2. **Human Annotation:** Every page was annotated by trained humans (after passing two exam levels in Phase 3), providing tight bounding boxes and resolving ambiguities that confuse automated parsers. 3. **Inter-Annotator Agreement as a Ceiling:** The authors double- and triple-annotated roughly 10% of the data to calculate a human agreement ceiling (IAA). The best-performing baseline models achieve $\approx 72\text{-}77\%$ mAP (`mAP@0.5:0.95`), while human agreement is $\approx 82\text{-}83\%$, leaving a meaningful gap for the community to close. 4. **Pre-defined document-wise splits:** The dataset ships with fixed train/test/validation splits partitioned on document boundaries, preventing pages from the same document from appearing in multiple sets. ## What experiments were performed? The authors establish baselines using four standard object detection architectures: **Mask R-CNN R50-FPN 3x**, **Mask R-CNN R101-FPN 3x**, **Faster R-CNN R101-FPN 3x** (all via detectron2 defaults), and **YOLOv5x6**. All models were pretrained on COCO 2017 and fine-tuned on $1025 \times 1025$ RGB page images. Performance is reported as `mAP@0.5:0.95` (COCO standard, averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05). Beyond the baseline comparison, the paper runs four ablation studies: 1. **Dataset size ablation (learning curve):** Mask R-CNN R50 was trained on increasing fractions of DocLayNet to assess whether the dataset is large enough. The learning curve flattens between the 80% and 100% data mark (within a 1% error bar from five runs), suggesting the dataset is close to saturated for this model family and that further size increases would yield diminishing returns. 2. **Label set reduction:** The 11-class label set was reduced to 6, 5, and 4 classes by collapsing or dropping labels, to understand the trade-off between label richness and model performance. 3. **Document-wise vs. page-wise split:** The same Mask R-CNN R50 was trained on both the official document-wise split and a randomized page-wise split to quantify how much information leaks when pages from the same document appear in both train and test. 4. **Cross-dataset comparison:** Mask R-CNN R50 was trained on each of PubLayNet, DocBank, and DocLayNet, then evaluated on the other datasets using their common labels, to test transfer robustness. ## Methodology: Class Definitions How does DocLayNet fit into our [Matter vs. Meaning framework](../matter-vs-meaning/)? The paper defines 11 distinct classes, chosen for their semantic distinctiveness. | DocLayNet Class | Visual Primitive | Logical Role | Notes | | :--- | :--- | :--- | :--- | | **Text** | `Text` | `Body` | Standard body text. | | **Title** | `Text` | `Title` | The main document title. | | **Section-header** | `Text` | `SectionHeader` | Crucial for RAG chunking hierarchies. | | **Page-header** | `Text` or `Image` | `PageHeader` | Running headers. DocLayNet does not distinguish text-based from image-based headers (logos, decorative bands); both are labeled identically. | | **Page-footer** | `Text` or `Image` | `PageFooter` | Running footers. Same caveat as Page-header: image-based footers are not separated from text-based ones. | | **Footnote** | `Text` | `Footnote` | Bottom-of-page notes. Link to `Body` via reference. | | **Caption** | `Text` | `Caption` | Text describing a Picture/Table. The guideline requires exactly one Caption per Picture or Table. | | **Table** | `Table` | *(primitive only)* | DocLayNet does not assign a further role. The `Table` visual primitive is the full classification; no `TOC`, `FormGroup`, or other role distinction is made. | | **Picture** | `Image` | `Figure` / `Chart` / `Diagram` | DocLayNet collapses all non-text visual content into one class. Photos, data charts, and technical diagrams are indistinguishable here. M vs. M separates these as distinct roles; expect precision loss when mapping. | | **Formula** | `Formula` | `DisplayEquation` | **Display only**. Inline math is merged into `Text`. Equation numbers are included *inside* the Formula bounding box (unlike M vs. M, which defines `FormulaNumber` as a separate `Text` role). | | **List-item** | `Text` | `ListItem` | Bullets/Enumerations. Each item is a separate atomic instance (not a grouped container). A List-item is defined as a paragraph with hanging indentation; bullet symbols are not required. | > **Critical for RAG & Extraction:** The explicit `Page-header` and `Page-footer` classes are invaluable. Most other datasets merge these into "Text". By isolating them, you gain control: > * **Filter:** Remove "Page 42" or "Annual Report" repetition to clean up prose chunks for RAG. > * **Extract:** Retain critical metadata (e.g., Policy Numbers, Claimant Names) often found only in the margins. > > *Constraint:* If your dataset lumps these into `Text`, you lose both capabilities (the prose is polluted, and the metadata is unrecoverable). --- ## What are the outcomes/conclusions? **Best model: YOLOv5x6 at 76.8% overall mAP.** The Mask R-CNN and Faster R-CNN variants cluster tightly between 72.4% and 73.5%, indicating that pixel-level segmentation masks do not help beyond bounding boxes for this task. The human IAA ceiling sits at **82-83% mAP**, leaving a gap of roughly 6 to 10 percentage points that current vision models have not closed. ### Per-Class Breakdown (Table 2, YOLOv5x6 vs. Human IAA) | Class | YOLOv5x6 AP | Human IAA | Notes | | :--- | ---: | :--- | :--- | | Caption | 77.7 | 84-89 | Below human; caption/text confusion | | Footnote | 77.2 | 83-91 | Below human; small and visually similar to text | | Formula | 66.2 | 83-85 | Hardest text class; visually variable and easily confused with surrounding text | | List-item | 86.2 | 87-88 | Near-human; very frequent | | Page-footer | 61.1 | 93-94 | Lowest model AP; humans are highly consistent here | | Page-header | 67.9 | 85-89 | Model struggles; high human consistency | | Picture | 77.1 | 69-71 | **Model exceeds human IAA** | | Section-header | 74.6 | 83-84 | Below human; confused with title/text | | Table | 86.3 | 77-81 | **Model exceeds human IAA** | | Text | 88.1 | 84-86 | **Model exceeds human IAA** | | Title | 82.7 | 60-72 | **Model far exceeds human IAA** | | **All** | **76.8** | **82-83** | | The hardest classes for models are **Page-footer** (61.1), **Formula** (66.2), and **Page-header** (67.9). On the most visually distinctive and abundant classes (Text, Table, Picture, Title), YOLOv5x6 surpasses human annotator agreement. The gap is concentrated in semantically ambiguous classes like Footnote, Section-header, and the header/footer pair. The paper attributes the model's advantage on Text, Table, and Picture to their abundance and visual distinctiveness, while the human advantage on header/footer classes reflects that humans bring page-level context that the model lacks. ### Learning Curve The dataset size ablation (Figure 5) shows the mAP learning curve flattens between the 80% and 100% data marks, within the 1% error bar estimated from five full-dataset runs. The authors interpret this as evidence that the dataset is large enough for this model family: adding more similar data would not yield substantial gains. Future improvement is more likely to come from better annotation consistency, data augmentation, or additional document categories. ### Label Reduction Tradeoff (Table 3) Reducing the label set from 11 classes to 5 (keeping List-item, Picture, Section-header, Table, and Text; mapping Caption, Footnote, and Formula into Text; excluding Page-footer and Page-header) raises the overall Mask R-CNN R50 mAP from **72% to 78%**. Going to 4 classes (the closest match to PubLayNet, further collapsing List-item into Text) yields 77%. The 5-point gain from collapsing to 5 classes comes almost entirely from removing Page-footer and Page-header, the two classes where model performance is lowest. If your downstream application does not need to distinguish headers and footers, reducing the label set is a practical option. ### Methodological Warning: Document-Wise vs. Page-Wise Split (Table 4) Many documents in DocLayNet have a distinctive visual style. If pages from the same document appear in both the training set and the test set (a "page-wise" random split), the model can memorize document-specific styling. Table 4 quantifies the effect: **page-wise splitting inflates mAP by roughly 10 to 12 points** compared to the document-wise split used in the official benchmark (72% vs. 84% for 11 classes with Mask R-CNN R50). DocLayNet ships with a pre-defined document-wise split. When benchmarking your own model, use that split. Page-wise splits produce results that look better on paper but do not reflect real generalization performance. ### Cross-Dataset Generalization (Table 5) To test whether layout diversity translates to robustness, the authors trained Mask R-CNN R50 on each available dataset and evaluated on the others using their common label classes. A PubLayNet-trained model achieves 93% mAP on its own test set but only 30% when transferred to DocLayNet. A DocLayNet-trained model achieves 78% on its own test set but only drops to 59% on PubLayNet and 47% on DocBank: a far smaller performance gap. This supports the core motivation that diverse training data leads to more generalizable layout models. ## Reproducibility ### Annotation Geometry Each annotation is an **axis-aligned rectangle** (no polygons, no rotated boxes). The CCS tooling enforced this as a hard constraint: annotators could only draw vertically oriented, non-overlapping rectangular bounding boxes. Annotations are at **block level**, not line, word, or character level. A whole paragraph is one box; a whole table is one box; a whole figure is one box. The one deliberate exception is `List-item`: each item in a list receives its own individual box rather than a single grouped container, which differs from PubLayNet's approach. **Text-cell snapping:** For all purely text-based classes (everything except `Table` and `Picture`), the CCS tool automatically shrinks the user-drawn box to the minimum bounding box enclosing the PDF text cells within it. This produces pixel-accurate tight boxes without requiring annotators to hand-draw precise boundaries. `Table` and `Picture` have no underlying text-cell layer, so these were drawn entirely by hand. **Non-overlapping:** Enforced at the tooling level. Boxes cannot overlap. **Coordinate space:** Pixel coordinates in the $1025 \times 1025$ PNG image, stored in augmented COCO JSON format. ### Annotation Campaign The campaign ran in four phases. Phases 1 (data selection) and 2 (label definition and guideline creation) required a small expert team. Phases 3 and 4 involved the full annotator group. - **Staff:** 40 annotators assembled; 32 admitted to production after passing two exam levels in Phase 3. - **Phase 3 (training):** 974 pages were reference-annotated by one proficient core team member. Annotators completed the same pages (blinded from the reference) at two complexity levels, each with a practice and an exam part. Only those meeting quality thresholds on both levels advanced to production. - **Phase 4 (production):** Roughly three months of annotation. - **Throughput:** 20 to 60 seconds per page, depending on layout complexity. - **Tooling:** IBM Corpus Conversion Service (CCS), a cloud-native annotation platform with a visual PDF overlay interface. Annotators could not see each other's annotations, by design, to avoid biasing the IAA measurement. - **Guideline:** Over 100 pages; planned for public release alongside the dataset. ### Models - **Architectures:** Mask R-CNN R50-FPN 3x, Mask R-CNN R101-FPN 3x, Faster R-CNN R101-FPN 3x (all via detectron2 default configs), YOLOv5x6. - **Pretraining:** COCO 2017 weights for all models. - **Input:** $1025 \times 1025$ pixels, RGB. - **No custom hyperparameters reported** beyond default detectron2/YOLOv5 configurations. - **No official pretrained checkpoints** for these specific experiments were released. ### Algorithms - **Training procedure:** Not detailed. The paper states all models used default detectron2 or YOLOv5 configurations with COCO 2017 pretrained weights. No optimizer, learning rate schedule, batch size, epoch count, or data augmentation settings are reported. - **Loss functions:** Standard object detection losses (as defined by detectron2 and YOLOv5 defaults). ### Data - **Size:** 80,863 unique pages; 7,059 double-annotated, 1,591 triple-annotated (91,104 total page-annotator instances). - **Total bounding boxes:** 1,107,470 across the full dataset (train/test/val: 941,123 / 99,816 / 66,531 bounding boxes respectively). - **Domains:** Financial Reports, Scientific Articles, Laws & Regulations, Government Tenders, Manuals, Patents. - **Language:** ~95% English; also German (2.5%), French (1.0%), Japanese (1.0%). - **Splits:** Pre-defined document-wise train/test/validation split. A page-wise split is also available but should not be used for fair evaluation (see Table 4 finding above). - **Format:** Augmented COCO JSON with PNG page images ($1025 \times 1025$), original PDFs, and sidecar JSON files with parsed text and text-cell coordinates. - **License:** CDLA-Permissive-1.0. #### Source Acquisition Documents were collected from publicly available repositories with open intellectual property: - **Scientific Articles:** arXiv - **Financial Reports:** Company websites and financial data directory services (including SEC filings and free-style annual reports) - **Laws & Regulations / Government Tenders:** Government office websites, mixed across independent providers to maximize layout variability - **Patents:** Patent databases - **Manuals:** Various public sources **Licensing caveat:** The CDLA-Permissive-1.0 license covers the *annotations* only. The underlying page images inherit the copyright of their source documents, which varies significantly by domain. **SEC filings** (mandatory US government disclosures) and **patents** (published by patent offices for public disclosure) are effectively public domain and are the safest categories for commercial training use. **arXiv papers** carry per-paper licenses that are not uniformly commercial-safe (many are CC-BY-NC or all-rights-reserved). **Financial reports** beyond SEC filings (free-style annual reports from company websites) and **manuals** ("various public sources") have unverified per-document licenses. **Government tenders** vary by jurisdiction. If you need a commercially safe training subset, filter by `doc_category` in the dataset (values: `financial_reports`, `scientific_articles`, `laws_and_regulations`, `government_tenders`, `manuals`, `patents`) and restrict to `patents` and the SEC filing portion of `financial_reports`. **Selection criteria:** The authors required medium-to-large documents (more than 10 pages) with technically dense content, prioritizing pages rich in tables, figures, and captions. **Scanned documents were excluded wherever possible.** Scans introduce rotation and skew that make rectangular bounding-box annotation unreliable, and the CCS annotation tool's text-cell snapping feature (which tightens boxes to enclosed PDF text cells) cannot function without a programmatic PDF layer. **Page subsampling:** Title pages were always included. For the remaining pages, a pre-trained PubLayNet object detection model was used to estimate figure and table density, biasing selection toward pages with more complex visual content. #### Rendering Pipeline The dataset images are **rasterized vector PDFs**, not photographs or scans. The pipeline was: 1. Source PDFs uploaded to IBM's Corpus Conversion Service (CCS) 2. CCS parsed each PDF and extracted text-cell coordinates (character bounding boxes from the PDF layer) 3. Each page was rendered to a PNG at $1025 \times 1025$ pixels 4. Annotators drew bounding boxes on the rendered PNG inside the CCS interface; the tool automatically snapped purely text-based boxes to the minimum enclosing text-cell boundary for pixel-accurate annotation 5. The released dataset includes the PNG page images, the original PDF pages, and sidecar JSON files linking parsed text and text-cell geometry to each page image This rendering-from-PDF approach is significant: it means bounding boxes can be tightly fit to PDF text cells, and it is why scanned pages (which lack a programmatic text layer) were excluded. ### Evaluation - **Metric:** `mAP@0.5:0.95` (COCO standard API), macro-averaged over 11 classes. - **Human ceiling:** Computed as pairwise mAP between independent annotators on triple-annotated pages; yields 82-83% overall. - **Baselines:** Detectron2 and YOLOv5 defaults; no architecture search or hyperparameter tuning reported. ### Hardware - Not specified in the paper. Standard GPU training with detectron2 and YOLOv5 default configurations. ## BibTeX ```bibtex @inproceedings{pfitzmann2022doclaynet, title={DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis}, author={Pfitzmann, Birgit and Auer, Christoph and Dolfi, Michele and Nassar, Ahmed S and Staar, Peter W J}, booktitle={Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining}, pages={3743--3751}, year={2022}, publisher={ACM}, doi={10.1145/3534678.3539043} } ```