# DSG: An End-to-End Document Structure Generator > DSG replaces heuristic relation rules with LSTM-based relation prediction and grammar constraints to make hierarchical document structure parsing fully end-to-end trainable, and introduces E-Periodica, a dataset of 542 annotated magazine pages. - Authors: Bevaya AI Team - Published: 2023-10-13 - Updated: 2026-04-02 - Paper: Preprint (arXiv) - Paper: Published (ICDM 2023) - Code: DSG Code - Dataset: E-Periodica Annotations - Source: Bevaya Labs, --- ## TL;DR DSG (Document Structure Generator) is, to the authors' knowledge, the first fully end-to-end trainable system for hierarchical document structure parsing. It extends a Faster R-CNN entity detector with LSTM-based relation classification heads and grammar-based postprocessing to produce tree-structured document representations. The authors also introduce E-Periodica, a dataset of 542 annotated magazine pages with 11,446 entities spanning complex, heterogeneous layouts. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$: The paper's headline contribution is a new system architecture that replaces heuristic relation inference with a trainable LSTM relation head, enabling joint end-to-end optimization of entity detection and relation classification. The ablation studies, comparison tables, and architectural descriptions all center on the modeling contribution. **Secondary:** $\Psi\_{\text{Resource}}$: The paper also contributes E-Periodica, a manually annotated dataset of real-world magazine pages that fills a gap left by earlier datasets limited to scientific articles. ## What is the motivation? Vast amounts of institutional and research information is stored as rendered documents (PDFs, scans) rather than structured, machine-readable formats. Downstream tasks such as database storage, knowledge base construction, and document retrieval require access to the hierarchical structure of these documents, not just their raw text. The state-of-the-art system at the time, DocParser, followed a two-stage pipeline: a trained entity detector identified bounding boxes and semantic categories, while hand-coded heuristics inferred the relations (sequence and nesting) between entities. Those heuristics were manually engineered for a specific document domain (scientific articles) and could not adapt to other domains without manual re-engineering. No system existed that jointly learned both entity detection and relation classification from labeled data. ## What is the novelty? DSG introduces a trainable relation classification head that operates alongside a standard Faster R-CNN entity detector, enabling the full pipeline to be optimized end-to-end. The system models documents through five components: 1. **Image preprocessing (C1):** Bilinear resizing with size augmentation during training; no geometric augmentations (flipping, rotation) that would corrupt layout geometry. 2. **Entity detection (C2):** Faster R-CNN with a ResNet-50 backbone. Produces bounding boxes, semantic categories, and confidence scores. 3. **Relation classification and entity refinement (C3):** A Neural Motifs architecture with two bidirectional LSTM heads. The refinement head takes visual, categorical (GloVe), and positional embeddings per entity to produce refined category predictions. The relation head processes pairs $(E\_{\text{subj}}, E\_{\text{obj}})$ using the union bounding box visual feature, the concatenated entity refinement outputs, and a frequency bias term to predict $\Psi \in \\{\text{parent\_of}, \text{followed\_by}, \text{null}\\}$. 4. **Grammar-based postprocessing (C4):** Resolves invalid predictions (cycles, missing parents, illegal relation types) using confidence-guided rules to enforce a valid tree structure. 5. **hOCR conversion (C5):** Outputs enriched hOCR markup compatible with standard document processing tools, extended with DSG-specific XML nodes for hierarchical information. The pairwise output feature used to score relation types is: $$p\_{\text{out}}^{\text{pair}} = W\_{\text{rel}}^{\text{pair}} \left( (W\_1^{\text{pair}} \rho\_1^{\text{pair}}) \odot (W\_2^{\text{pair}} \rho\_2^{\text{pair}}) \right) + \rho\_3^{\text{pair}}$$ where $\rho\_1^{\text{pair}}$ is the visual feature for the union bounding box, $\rho\_2^{\text{pair}}$ concatenates entity refinement features, and $\rho\_3^{\text{pair}}$ is the category-pair frequency bias. The joint training objective combines entity detection losses from C2 and relation losses from C3: $$\mathcal{L} = L\_{C2} + L\_{C3} = (L\_{\text{RPN}}^{\text{cls}} + L\_{\text{RPN}}^{\text{loc}} + L\_{\text{cls}}^{\text{cls}} + L\_{\text{cls}}^{\text{loc}}) + (L\_{\text{ref}} + L\_{\text{rel}})$$ End-to-end training allows gradient signal from relation-level supervision to propagate into the entity detector, improving entity detection quality, particularly for entity categories that are critical for building correct hierarchical structures. ## What experiments were performed? **Datasets:** The system is evaluated on two datasets: arXivdocs-target (362 hand-annotated scientific article pages from DocParser) and E-Periodica (542 manually annotated magazine pages introduced by this paper). The two cover distinct document types: structured academic articles and heterogeneous real-world magazines. **Baselines:** DocParser (heuristic relation classification) is the primary comparison. Ablation variants of DSG include DSG 2-stage (C2 frozen) and DSG 2-stage (C2 unfrozen), which separate entity detection training from relation training rather than training jointly. **Metrics:** Entity detection uses mean average precision (mAP) at IoU thresholds of 0.5 and 0.75. Bounding box overlap is measured by: $$\text{IoU} = \frac{\text{area}(B\_j \cap \hat{B}\_j)}{\text{area}(B\_j \cup \hat{B}\_j)}$$ A predicted entity is a true positive if its IoU exceeds the threshold and it uniquely matches a ground-truth entity (highest-IoU wins; ties broken by confidence). Structure generation uses precision, recall, and F1 on exact-match relation triples $(E\_{\text{subj}}, E\_{\text{obj}}, \Psi)$: $$F\_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$$ Both entity bounding boxes in a predicted triple must uniquely match ground-truth entities at the specified IoU threshold. This is stricter than conventional scene graph evaluation, which allows one-to-many matching. **Training:** Two pre-training stages before fine-tuning: COCO-pretrained Faster R-CNN weights, then arXivdocs-weak (weakly supervised scientific articles). Fine-tuning runs for up to 200,000 iterations with batch size 4 and learning rate 0.001. Up to 50 entities and 128 sampled entity pairs per iteration are passed to C3. **Hardware:** Experiments run on a single NVIDIA Titan Xp GPU (12 GB VRAM). Inference averages approximately 0.1616 seconds per page without postprocessing and 0.1776 seconds with postprocessing. ## What are the outcomes/conclusions? **Entity detection:** DSG end-to-end achieves mAP@0.5 of 79.61 on arXivdocs-target vs. 77.70 for DocParser (a 1.91 pp gain). On E-Periodica, the gap widens to 62.35 vs. 55.32 (a 7.03 pp gain, or 12.71% relative). The larger gap on the more heterogeneous magazine data reflects that heuristics tuned to scientific articles generalize poorly. **Structure generation:** On arXivdocs-target, DSG end-to-end (without postprocessing) achieves F1 of 0.7592 vs. 0.7054 for DocParser (7.63% relative improvement). On E-Periodica, the gap is much larger: F1 of 0.5340 vs. 0.1884, a 183.44% relative improvement. DocParser's heuristics, tuned for scientific articles, generalize poorly to magazine layouts. **Ablations:** End-to-end training consistently outperforms both 2-stage variants, confirming that joint optimization with relation-level supervision improves entity detection quality, particularly for structurally important categories such as ARTICLE. **Postprocessing trade-off:** Grammar-based postprocessing slightly reduces F1 in strict evaluation (because resolving missing parents can introduce incorrect edges), but it guarantees a valid tree structure required for downstream use. Qualitative results suggest that even when F1 is around 0.5, the recovered structures are visually close to ground truth. **Limitations:** The system processes one page at a time and does not model cross-page structure. The strict evaluation metric (exact entity matching) means small localization errors carry a high penalty that may understate practical utility. Some images in the E-Periodica dataset remain under copyright restriction and were substituted with similar pages, introducing minor inconsistency in the test set. ## Reproducibility ### Models DSG is built on the Neural Motifs architecture, adapted from Detectron2 and segmentation-sg. The entity detection backbone is ResNet-50. The relation and refinement heads each use a one-layer bidirectional LSTM with hidden size 512 and dropout 0.2. GloVe embeddings (200-dimensional) represent entity category names. Visual features are 1024-dimensional vectors from a four-scale FPN alignment. Pretrained model checkpoints for both DSG trained on arXivdocs and DSG trained on E-Periodica are available through the project's Google Drive link. ### Algorithms - Optimizer: Not stated explicitly in the paper; the codebase inherits Detectron2 defaults (typically SGD). Learning rate 0.001 for end-to-end training, 0.01 for the initial entity-only pretraining stage. - Batch size: 4 (end-to-end stage), 8 (entity pretraining stage and 2-stage ablation second stage). - Training steps: up to 200,000 for end-to-end training; up to 100,000 for entity pretraining; up to 100,000 for the second stage in 2-stage ablations. - Early stopping: based on mAP@0.5 on the validation set. - Relation sampling: 128 entity pairs per iteration, sampled to balance positive and negative examples (most pairs have relation type null). - Image resizing: smallest side capped at $\phi^s\_{\max} = 600$; longest side $\phi^l\_{\max}$ randomly chosen from [250, 300, ..., 550] during training. At test time, $\phi^s\_{\max} = 400$ and $\phi^l\_{\max} = 600$. - Pre-training: COCO object detection weights with copy-paste augmentation; then arXivdocs-weak weak supervision (from the DocParser project). - No geometric augmentation (flipping, rotation) to preserve layout reading order. ### Data - **arXivdocs-target:** 362 hand-annotated scientific article pages from the DocParser work. Used for training and evaluation after conversion to COCO-format annotations. Availability and license controlled by the DocParser repository. - **arXivdocs-weak:** Weakly supervised dataset from the DocParser project, used only for pre-training (not fine-tuning or evaluation). No direct download link or license is provided in the DSG paper; availability is controlled by the DocParser repository. - **E-Periodica:** 542 pages from historical and contemporary Swiss magazines. Split into 270 training / 135 validation / 137 test. Manual two-step annotation: entity bounding boxes and categories, then relation types (`followed_by`, `parent_of`). UNORDERED GROUP and ORDERED GROUP categories handle magazine-specific reading order conventions. Two images were substituted due to copyright restrictions; those become publicly available in 2024 and 2026 respectively. The dataset license is not stated explicitly; the repository notes copyright restrictions on some source images. ### Evaluation - **Entity mAP:** Standard detection mAP at IoU thresholds 0.5 and 0.75. One-to-one ground-truth matching (highest-IoU prediction wins each GT box). - **Structure F1:** Exact-match precision/recall/F1 on relation triples $(E\_{\text{subj}}, E\_{\text{obj}}, \Psi)$; both entity bounding boxes must uniquely match GT at the specified IoU threshold. - The stricter evaluation (one-to-one entity matching, exact relation typing) makes scores harder to interpret for deeply nested or closely overlapping entities. The authors acknowledge that practical utility may be higher than the metric suggests. - No statistical significance tests or multi-seed variance reported. ### Hardware - Training and inference: single NVIDIA Titan Xp GPU (12 GB VRAM). - Inference: ~0.16 seconds per page without postprocessing, ~0.18 seconds with postprocessing. - No cloud cost or energy estimates reported. ## BibTeX ```bibtex @inproceedings{rausch2023dsg, title={DSG: An End-to-End Document Structure Generator}, author={Rausch, Johannes and Rashiti, Gentiana and Gusev, Maxim and Zhang, Ce and Feuerriegel, Stefan}, booktitle={2023 IEEE International Conference on Data Mining (ICDM)}, year={2023}, doi={10.1109/ICDM58522.2023.00061} } ```