# HRDoc: Dataset and Baseline Method toward Hierarchical Reconstruction of Document Structures > HRDoc introduces hierarchical document structure reconstruction as a multi-page task, pairing a 2,500-document dataset with fine-grained line-level annotations with an encoder-decoder system (DSPS) that combines multi-modal bidirectional encoding with a structure-aware GRU decoder. - Authors: Bevaya AI Team - Published: 2023-03-24 - Updated: 2026-04-02 - Dataset: HRDoc Dataset - Code: DSPS Code - Paper: Preprint (arXiv) - Paper: Published (AAAI 2023) - Source: Bevaya Labs, --- ## TL;DR HRDoc defines hierarchical document structure reconstruction as a formal multi-page task, releasing a 2,500-document dataset with nearly 2 million line-level annotated semantic units. The authors also propose DSPS, an encoder-decoder model that pairs a multi-modal bidirectional encoder with a structure-aware GRU decoder and a soft-mask operation, achieving substantially better performance than prior page-level systems on the new benchmark. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Resource}}$: The central contribution is the HRDoc dataset itself: 2,500 annotated multi-page documents with line-level semantic labels and explicit parent-child relational annotations. The dataset introduces a new task formulation, defines annotation protocols, and provides train/test splits. Without the dataset, the paper has little to offer. **Secondary:** $\Psi\_{\text{Method}}$: The Document Structure Parsing System (DSPS) is a meaningful model contribution: a multi-modal bidirectional encoder paired with a structure-aware GRU decoder and a novel soft-mask operation. It is benchmarked against baselines with ablations. **Secondary:** $\Psi\_{\text{Evaluation}}$: The paper introduces STEDS (Semantic Tree-Edit-Distance Score), a new metric for evaluating hierarchical reconstruction quality at the document level. Note on classification: the paper's title lists "Dataset and Baseline Method" as co-equal contributions, and the dataset arguably has independent value. We classify it as Resource-dominant because the benchmark it establishes has driven downstream work independently of DSPS, but a Method-dominant reading is defensible given the experimental section's focus on ablating DSPS components. ## What is the motivation? Most document layout analysis research treats each page in isolation. The dominant paradigm (object detection models like Cascade-RCNN applied to document images) produces flat lists of bounding boxes per page: text regions, figures, tables, and so on. These approaches do not address how elements relate to one another across pages, how paragraphs belong to sections, how sections nest under one another, or how captions attach to figures. The authors observe that converting a PDF to a structured, editable format (such as Markdown or a knowledge graph) requires more than bounding boxes: it requires a tree. Yet at the time of writing, no benchmark captured this hierarchical, cross-page structure at scale with fine-grained line-level annotations. Existing partial solutions either only extracted tables of contents (ignoring body content) or applied brittle rule-based heuristics that do not generalize beyond fixed layouts. Specifically, DocParser's rule-based page-level reconstruction produced poor generalization across layout types, and DocStruct's pair-wise relation approach scaled poorly with document length. HRDoc is built to fill this gap, providing both the annotation standard and the scale needed to train and evaluate learned approaches to hierarchical document structure recovery. ## What is the novelty? The paper makes three distinct contributions. **Task definition.** The authors define hierarchical document structure reconstruction as three coordinated subtasks given a multi-page document $D$ with semantic units $U$: - **SUBTASK 1 (Semantic unit classification):** Assign a semantic label to each unit $u\_i$ from 14 classes: Title, Author, Mail, Affiliation, Section, First-Line, Para-Line, Equation, Table, Figure, Caption, Page-Footer, Page-Header, and Footnote. - **SUBTASK 2 (Parent finding):** For each unit $u\_i$, identify its nearest ancestor $p\_i$ in the document tree (potentially across page boundaries). - **SUBTASK 3 (Relation classification):** Classify the relation $r\_i$ between $u\_i$ and $p\_i$ as one of *Connect* (sequential continuation), *Contain* (hierarchical nesting), or *Equality* (same-level siblings). **HRDoc dataset.** The dataset comprises two splits: HRDoc-Simple (HRDS, 1,000 ACL papers with uniform two-column layout) and HRDoc-Hard (HRDH, 1,500 arXiv papers spanning 17 research areas and more than 30 layout varieties). Ground truth for HRDS is produced by PDF parsing + heuristic rules + human review; for HRDH by re-compiling LaTeX source with injected color tags + rule-based parsing + human review. Together they contain approximately 2 million annotated semantic units. All source documents carry non-commercial open licenses (CC-BY-NC-SA 3.0 / CC-BY 4.0 for ACL; CC-BY-NC-SA 4.0 for arXiv). **DSPS model.** The Document Structure Parsing System consists of three components: 1. *Multi-modal bidirectional encoder.* For each semantic unit $u\_i$ with text $s\_i$, bounding box $b\_i$, page number $p\_i$, and page image $I\_{p\_i}$, the model constructs a fused embedding: $$x\_i = \text{LN}(x\_i^{\text{text}} + x\_i^{\text{layout}} + x\_i^{\text{pos}} + x\_i^{\text{vis}} + x\_i^{\text{page}})$$ Text embeddings come from `paraphrase-multilingual-mpnet-base-v2` Sentence-BERT projected to dimension $H$; layout embeddings follow LayoutLMv2; visual embeddings use ResNet-50 with FPN and RoIAlign over page images rendered with PyMuPDF. A Transformer-based bidirectional encoder produces contextual representations $x\_i^{\ast}$. 2. *Structure-aware GRU decoder with soft-mask.* A GRU processes the full unit sequence across pages to produce hidden states $h\_i$. Attention pooling computes a query vector: $$q\_i = \sum\_{j=0}^{i} \alpha(h\_i, h\_j)\, h\_j$$ The soft-mask operation introduces a child-parent distribution matrix $M\_{cp} \in \mathbb{R}^{(C+1) \times C}$ derived from corpus statistics, encoding which semantic class types can legally parent which child types. The parent probability is then: $$P\_{\text{par}(i,j)} = \text{softmax}\!\left(\alpha(q\_i, h\_j) \cdot P\_{\text{dom}(i,j)}\right)$$ where $P\_{\text{dom}(i,j)} = \tilde{P}\_{\text{cls}\_j}\, M\_{cp}\, (P\_{\text{cls}\_i})^T$ acts as a domain-knowledge gate. Additive smoothing with pseudo-count 5 handles unseen relation pairs. 3. *Relation classifier.* A linear projection over concatenated hidden states classifies the relation type for each child-parent pair. **Multi-task loss.** The three subtasks are trained jointly: $$L\_{\text{tol}} = L\_{\text{cls}} + \alpha\_1 L\_{\text{par}} + \alpha\_2 L\_{\text{rel}}$$ where $L\_{\text{cls}}$ and $L\_{\text{rel}}$ use Focal Loss for class-imbalance robustness and $L\_{\text{par}}$ uses cross-entropy. The specific values of $\alpha\_1$ and $\alpha\_2$ are not reported. **STEDS metric.** The Semantic Tree Edit Distance Score measures reconstruction quality as: $$\text{STEDS}(T\_D, \hat{T}\_D) = 1 - \frac{\text{EditDist}(T\_D, \hat{T}\_D)}{\max(|T\_D|, |\hat{T}\_D|)}$$ A node is considered a match only when both label and text content agree, making the metric sensitive to semantic correctness. Both Micro-STEDS (frequency-weighted) and Macro-STEDS (class-equal) are reported. ## What experiments were performed? **Baselines for semantic unit classification (SUBTASK 1):** Four methods are compared on 14-class F1: - T1: Cascade-RCNN (vision-only detection) - T2: ResNet-50 + RoIAlign (vision features with bounding boxes) - T3: Sentence-BERT alone (text-only) - T4: DSPS encoder (full multi-modal) **Baselines for hierarchical reconstruction (OVERALL TASK):** DocParser (Rausch et al., AAAI 2021) is used as the external baseline. DSPS is then ablated across: semantic modality, visual modality, soft-mask operation, and document-level vs. page-level decoding. Both HRDS and HRDH test sets are evaluated separately to capture layout generalization. ## What are the outcomes/conclusions? **Semantic unit classification.** The full DSPS encoder (T4) achieves the best Micro F1 across both splits: 99.52% on HRDS and 96.74% on HRDH, compared to 79.32% / 73.37% for Cascade-RCNN (T1). The multi-modal fusion consistently outperforms single-modality approaches, though for categories with visually similar appearances (Mail, Affiliation), visual features can occasionally reduce performance. **Hierarchical document reconstruction.** DSPS (all modalities, soft-mask, document-level) achieves Micro-STEDS of 0.8143 on HRDS and 0.6903 on HRDH, compared to 0.2361 and 0.1873 for DocParser. Key ablation findings: - Adding visual features over semantic-only raises HRDS Micro-STEDS by +0.049. - Adding the soft-mask raises it by +0.068, showing domain-knowledge gating is slightly more impactful than vision. - Forcing page-level decoding drops performance from 0.8143 to 0.6482, confirming multi-page context is essential. The gap between HRDS and HRDH performance reflects the greater layout diversity in the hard split. The STEDS metric requiring both label and text to match may be overly strict given imperfect OCR inputs. The dataset inherits non-commercial licensing from its source documents, limiting downstream use. **Lineage note.** HRDoc and DSPS serve as the direct foundation for Detect-Order-Construct (Wang et al., 2024), which extends the three-subtask formulation with an explicit reading-order prediction stage and introduces the Comp-HRDoc benchmark for joint evaluation of detection, ordering, TOC extraction, and full structure reconstruction. ## Mapping to Matter vs. Meaning Framework How does HRDoc's 14-class taxonomy and 3 relation types map to our [Matter vs. Meaning framework](../matter-vs-meaning/)? ### Element Classes | HRDoc Class | Visual Primitive | Logical Role | Notes | | :--- | :--- | :--- | :--- | | **Title** | `Text` | `Title` | The main document title. | | **Author** | `Text` | `Author` | Author name(s). | | **Mail** | `Text` | `Address` | Author email addresses. | | **Affiliation** | `Text` | `Affiliation` | Institutional affiliations. | | **Section** | `Text` | `SectionHeader` | Section and subsection headings. | | **First-Line** | `Text` | `Body` | First line of a paragraph; marks paragraph boundaries at line level. | | **Para-Line** | `Text` | `Body` | Continuation lines of a paragraph. | | **Equation** | `Formula` | `DisplayEquation` | Block-level mathematical expressions. | | **Table** | `Table` | *(primitive only)* | Tabular regions (detected at region level, not line level). | | **Figure** | `Image` | `Figure` | Visual content (detected at region level, not line level). | | **Caption** | `Text` | `Caption` | Text describing a Figure or Table. | | **Page-Footer** | `Text` | `PageFooter` | Running footers. | | **Page-Header** | `Text` | `PageHeader` | Running headers. | | **Footnote** | `Text` | `Footnote` | Bottom-of-page notes. | ### Relation Types | HRDoc Relation | M-vs-M Mapping | Notes | | :--- | :--- | :--- | | **Connect** | Sequential reading order | Continuation (e.g., paragraph line to next line). Maps to reading order edges. | | **Contain** | `header_of` / hierarchical nesting | Parent-child (e.g., section to subsection). Maps to hierarchical structure. | | **Equality** | Sibling grouping | Same-level siblings (e.g., two subsections under one section). No direct M-vs-M equivalent; closest is implicit via shared parent. | **Structural notes:** HRDoc is one of few datasets that provides **cross-page** parent-child relationships. The First-Line / Para-Line distinction is unique: it encodes paragraph boundaries at the line level, finer-grained than most datasets that annotate whole paragraphs as single boxes. **Coverage gaps:** No explicit classes for `ListItem`, `BibEntry`, `Abstract`, `PageNumber`, inline `Formula`, or any form primitives. The dataset is restricted to scientific papers (ACL and arXiv). ## Reproducibility ### Models DSPS uses a Transformer-based bidirectional encoder fed five embedding streams. The visual branch is ResNet-50 with FPN and RoIAlign over page images rendered with PyMuPDF. Text embeddings come from `paraphrase-multilingual-mpnet-base-v2` Sentence-BERT. The GRU decoder is single-layer with attention pooling. No parameter counts are reported. Source code is at the project repository; model weights are not separately released as pretrained checkpoints. The repository does not specify a license. ### Algorithms The training procedure details (optimizer, learning rate, batch size, number of training steps or epochs) are not reported in the paper. Loss weighting coefficients $\alpha\_1$ and $\alpha\_2$ are mentioned but not specified. Focal Loss is used for classification and relation tasks; cross-entropy for parent finding. Additive smoothing with pseudo-count 5 is applied to the soft-mask distribution matrix. ### Data HRDoc-Simple: 1,000 ACL papers (NAACL, EMNLP, ACL) with fixed layout; CC-BY-NC-SA 3.0 or CC-BY 4.0. HRDoc-Hard: 1,500 arXiv papers from 17 research areas; CC-BY-NC-SA 4.0. Train/test splits are used but split ratios are not explicitly stated. Bounding boxes for Equation, Table, and Figure regions are detected with Cascade-RCNN rather than extracted from PDF metadata. Dataset files are distributed via Google Drive links from the GitHub repository. ### Evaluation Primary metric for SUBTASK 1: per-class F1, with a predicted box accepted when it overlaps a ground-truth box of the same label with IoU above 0.65. Micro-STEDS and Macro-STEDS for the overall task use tree-edit-distance derived from Pawlik and Augsten (2016). Comparison against DocParser is conducted by running DocParser inference on HRDoc, excluding the table structure module, and matching detected objects to ground-truth units using an overlap ratio threshold of 0.7. No statistical significance tests or confidence intervals are reported. ### Hardware The paper does not report training hardware, GPU count, training time, or GPU-hours. ## BibTeX ```bibtex @inproceedings{ma2023hrdoc, title={HRDoc: Dataset and Baseline Method toward Hierarchical Reconstruction of Document Structures}, author={Ma, Jiefeng and Du, Jun and Hu, Pengfei and Zhang, Zhenrong and Zhang, Jianshu and Zhu, Huihui and Liu, Cong}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, volume={37}, number={2}, pages={1870--1877}, year={2023}, doi={10.1609/aaai.v37i2.25277} } ```