# Comp-HRDoc: Comprehensive Benchmark for Hierarchical Document Structure Analysis > Research note on the DOC framework and Comp-HRDoc benchmark for hierarchical document structure analysis, covering page object detection, reading order, TOC extraction, and tree reconstruction. - Authors: Hunter Heidenreich - Published: 2026-03-15 - Paper: Preprint (arXiv) - Paper: Published (Pattern Recognition 2024) - Benchmark: Comp-HRDoc Benchmark - Source: Bevaya Labs, --- ## TL;DR This paper introduces DOC (Detect-Order-Construct), a three-stage pipeline for hierarchical document structure analysis that casts each stage as a relation prediction problem solved by multi-modal transformers. Alongside the method, the authors release Comp-HRDoc, a comprehensive benchmark built on HRDoc-Hard (1,500 documents) that evaluates four tasks simultaneously: page object detection, reading order prediction, table of contents extraction, and hierarchical structure reconstruction. DOC improves over prior methods across all four tasks, with margins detailed below. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The headline contribution is the DOC framework itself: a three-stage architecture that decomposes hierarchical document structure analysis into Detect (page object detection + text region grouping), Order (reading order prediction), and Construct (TOC and tree building). Each stage is formulated as a relation prediction problem with dedicated transformer-based models, ablations, and comparisons against published results across multiple benchmarks. **Secondary:** $\Psi\_{\text{Resource}}$. Comp-HRDoc is a new benchmark extending HRDoc-Hard with annotations for four tasks, publicly released with evaluation scripts under MIT license. Document images must be obtained separately from the original HRDoc dataset. **Secondary:** $\Psi\_{\text{Evaluation}}$. The paper introduces REDS (Reading Edit Distance Score), a new metric for reading order evaluation that uses Levenshtein distance with Hungarian matching across multiple reading order groups. ## What is the motivation? Prior work on document structure analysis treats sub-tasks (layout detection, reading order, TOC extraction, hierarchical reconstruction) independently, using separate models and datasets. No unified benchmark existed to evaluate all four tasks together on the same documents. The original HRDoc dataset only provided line-level semantic labels and hierarchical annotations, lacking standard page-object-level detection and reading order evaluation. The authors also argue that existing reading order metrics (BLEU, nDCG, Spearman) are borrowed from other domains and do not adequately capture document-specific errors like paragraph segmentation mistakes. ## What is the novelty? ### DOC Framework The core insight is that all three stages of hierarchical document structure analysis can be unified as **relation prediction** problems: 1. **Detect:** A hybrid approach combining top-down graphical object detection (DINO or Mask2Former) with bottom-up text region detection. Text-lines from a PDF parser are grouped into text regions via intra-region reading order prediction, formulated as dependency parsing: $$P(j \mid i) = \frac{\exp(f\_{ij})}{\sum\_{k} \exp(f\_{ik})}$$ where $f\_{ij}$ is a pairwise compatibility score between text-lines $i$ and $j$, computed from multi-modal features (visual via RoIAlign, text via BERT-BASE, 2D positional embeddings). Union-Find groups text-lines into regions based on predicted links. 2. **Order:** Detected page objects are encoded with attention-fused text-line features and region type embeddings, processed through a 3-layer transformer encoder. Inter-region reading order is predicted using the same dependency parsing formulation, with a separate classification head distinguishing text region reading order from graphical region relationships (caption-to-figure/table links). 3. **Construct:** Section headings (in predicted reading order) are encoded with Rotary Positional Embeddings (RoPE) and processed by a transformer encoder. A tree-aware TOC relation prediction head predicts both parent-child and sibling relationships: $$P\_{\text{parent}}(j \mid i) = \frac{\exp(f^{p}\_{ij})}{\sum\_{k} \exp(f^{p}\_{ik})}$$ $$P\_{\text{sibling}}(j \mid i) = \frac{\exp(f^{s}\_{ij})}{\sum\_{k} \exp(f^{s}\_{ik})}$$ A serial tree insertion algorithm (Algorithm 1) scores candidate positions by multiplying parent and sibling probabilities along the rightmost sub-tree and inserts each heading at the best-scoring position. ### Comp-HRDoc Benchmark Extends HRDoc-Hard (1,500 documents, 1,000 train / 500 test) with: - **Page object detection** annotations (12 categories: Title, Author, Mail, Affiliate, Section heading, Paragraph, Table, Figure, Caption, Footnote, Header, Footer) evaluated via segmentation-based mAP (not box-based, since logical paragraphs may span multiple columns) - **Reading order prediction** evaluated via REDS - **TOC extraction** and **hierarchical structure reconstruction** evaluated via Semantic-TEDS (Micro and Macro) ### REDS Metric REDS (Reading Edit Distance Score) addresses shortcomings of borrowed metrics (BLEU, nDCG) by using Levenshtein edit distance with Hungarian matching across reading order groups: $$\text{REDS} = 1 - \frac{D}{N}$$ where $D$ is the minimum total Levenshtein distance after Hungarian matching across predicted and ground-truth reading order groups, and $N$ is the total number of basic evaluation units. It includes a special paragraph-ending tag to penalize paragraph segmentation errors. ## What experiments were performed? The authors evaluate DOC on four benchmarks: - **PubLayNet** (360K pages, 5 classes): Page object detection. COCO box-based mAP. - **DocLayNet** (80K pages, 11 classes): Page object detection. COCO box-based mAP. - **HRDoc** (Simple: 1,000 docs; Hard: 1,500 docs): Semantic unit classification (F1) and hierarchical structure reconstruction (Semantic-TEDS). - **Comp-HRDoc** (1,500 docs): All four tasks with dedicated metrics. Baselines include Mask R-CNN, Faster R-CNN, YOLOv5, DINO, Mask2Former, DiT-L, VSR, LayoutLMv3, and the DSPS Encoder from HRDoc. Ablation studies cover: (1) hybrid detection strategy effectiveness, (2) modality contributions in the Construct module, (3) component contributions to TOC extraction (sibling head, tree insertion algorithm, loss function choice). ## What are the outcomes/conclusions? ### Page Object Detection | Benchmark | DOC | Previous Best | Delta | | :--- | :---: | :---: | :---: | | DocLayNet (mAP) | 81.0 | 76.8 (YOLOv5) | +4.2 | | PubLayNet val (mAP) | 96.5 | 96.0 (TRDLU) | +0.5 | | PubLayNet test (mAP) | 96.5 | 95.7 (VSR) | +0.8 | | Comp-HRDoc (seg mAP) | 88.06 | 73.54 (Mask2Former) | +14.52 | The large Comp-HRDoc gap (+14.52) comes from the hybrid strategy: top-down detection handles graphical objects well, while bottom-up text-line grouping produces more precise text region boundaries than pure detection approaches. ### Reading Order, TOC, and Hierarchy (Comp-HRDoc) | Task | Metric | DOC | Previous Best | | :--- | :--- | :---: | :---: | | Reading Order (Text) | REDS | 0.932 | 0.774 (Lorenzo et al.) | | Reading Order (Graphical) | REDS | 0.864 | 0.858 (Lorenzo et al.) | | TOC Extraction | Micro-STEDS | 0.861 | 0.676 (MTD) | | TOC Extraction | Macro-STEDS | 0.879 | 0.710 (MTD) | | Hierarchical Reconstruction | Micro-STEDS | 0.837 | 0.690 (DSPS Encoder) | | Hierarchical Reconstruction | Macro-STEDS | 0.837 | 0.697 (DSPS Encoder) | ### Key Ablation Findings - Section number information is critical for the Construct module: without section numbers in the text features, Micro-STEDS drops from 0.861 to 0.641. - The tree insertion algorithm is essential: removing it drops Micro-STEDS from 0.861 to 0.711. - Softmax cross-entropy loss significantly outperforms binary cross-entropy for relation prediction (0.861 vs. 0.700). ### Limitations - The Construct module depends on accurate section heading recognition; errors in detection propagate to tree construction. - Section number information is critical for TOC extraction; the approach may not generalize to documents without numbered sections. - Only tree-structured documents are supported; graph-based logical structures are not addressed. - The system assumes text-line bounding boxes and content are provided by a PDF parser or OCR engine, limiting direct applicability to image-only documents without a prior OCR step. - Inconsistent paragraph annotations in HRDoc may affect paragraph detection performance. ## Reproducibility ### Models - **Detect:** ResNet-50 backbone (PubLayNet/DocLayNet) or ResNet-18 (HRDoc/Comp-HRDoc). Graphical object detector: DINO (PubLayNet/DocLayNet) or Mask2Former (HRDoc/Comp-HRDoc). Custom bottom-up text region detector with 1-layer transformer encoder (12 heads, 768-dim, 2048-dim FFN). - **Order:** 3-layer transformer encoder for inter-region relation prediction - **Construct:** Transformer encoder with RoPE; tree-aware relation prediction head with parent-child and sibling sub-heads - **Text encoder:** BERT-BASE (pretrained), averaged token embeddings per text-line - No model weights or training code released. The [GitHub repo](https://github.com/microsoft/CompHRDoc) contains only benchmark annotations and evaluation scripts. ### Algorithms - **Optimizer:** AdamW (betas 0.9/0.999, epsilon 1e-8) - **Learning rates:** 1e-5 (backbone) / 2e-5 (BERT) for PubLayNet/DocLayNet; 2e-4 (backbone) / 4e-5 (BERT) for HRDoc/Comp-HRDoc - **Weight decay:** 1e-4 (backbone, PubLayNet/DocLayNet); 1e-2 (backbone, HRDoc/Comp-HRDoc); 1e-2 (BERT, all) - **Schedule:** PubLayNet/DocLayNet: step decay (LR / 10 at epoch 11 / 20 respectively). HRDoc/Comp-HRDoc: linear warmup (2 epochs) then linear decay to 0. - **Epochs:** 12 (PubLayNet), 24 (DocLayNet), 20 (HRDoc/Comp-HRDoc) - **Batch size:** 16 (PubLayNet/DocLayNet), 1 (HRDoc/Comp-HRDoc, multi-page documents) - **Multi-scale training:** Shorter side randomly sampled from [512, 640, 768] (max 800) or [320, 416, 512, 608, 704, 800] (max 1024) - **Test resolution:** Shorter side 640 (PubLayNet/DocLayNet), 512 (HRDoc/Comp-HRDoc) - **Loss:** Softmax cross-entropy for relation prediction; standard detection losses for DINO/Mask2Former - **Framework:** PyTorch v1.10 ### Data - **Comp-HRDoc:** 1,000 train / 500 test documents, built on HRDoc-Hard. Annotations publicly available at [GitHub](https://github.com/microsoft/CompHRDoc) under MIT license. Document images must be obtained separately from the [HRDoc dataset](https://github.com/jfma-USTC/HRDoc) (license unknown). - **PubLayNet:** 340K train / 12K val / 12K test. CDLA-Perm-1.0 (annotations); underlying PDFs non-commercial. - **DocLayNet:** 69K train / 6.5K test / 5K val. CDLA-Perm-1.0 (annotations). - **HRDoc:** Simple (1,000 docs) and Hard (1,500 docs). Mixed licenses (CC-BY, CC-BY-NC-SA). ### Evaluation - **Page object detection:** COCO-style mAP (box-based for PubLayNet/DocLayNet; segmentation-based for Comp-HRDoc) - **Reading order:** REDS (new metric introduced in this paper) - **TOC extraction and hierarchy:** Semantic-TEDS Micro and Macro (from HRDoc) - **Semantic unit classification (HRDoc):** Per-category F1, Micro/Macro averages - Comp-HRDoc evaluation scripts released at GitHub - No error bars or multi-run statistics reported ### Hardware - **Training:** 8 NVIDIA Tesla V100 GPUs (32 GB each) - No inference latency, throughput, or cost estimates reported - Batch size of 1 for Comp-HRDoc due to multi-page document processing; memory constraints likely driven by processing dozens of pages per document ## BibTeX ```bibtex @article{wang2024detect, title={Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysis}, author={Wang, Jiawei and Hu, Kai and Zhong, Zhuoyao and Sun, Lei and Huo, Qiang}, journal={Pattern Recognition}, volume={156}, pages={110836}, year={2024}, publisher={Elsevier}, doi={10.1016/j.patcog.2024.110836} } ```