# DLAFormer: An End-to-End Transformer for Document Layout Analysis > Research notes on DLAFormer, a DETR-based model that unifies graphical object detection, text region detection, logical role classification, and reading order prediction into a single end-to-end transformer via a unified label space. - Authors: Hunter Heidenreich - Published: 2026-03-13 - Paper: Preprint (arXiv) - Paper: Published (ICDAR 2024) - Source: Bevaya Labs, --- ## TL;DR DLAFormer reformulates multiple document layout analysis sub-tasks (graphical object detection, text region detection, logical role classification, reading order prediction) as relation prediction problems over a unified label space, enabling a single end-to-end DETR-based transformer to handle all tasks concurrently. It outperforms multi-branch and multi-stage baselines on both Comp-HRDoc and DocLayNet. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$ The headline contribution is a new architecture and training formulation. The paper introduces type-wise queries, a unified label space, and a coarse-to-fine detection strategy. The majority of the paper is devoted to describing the architecture, ablation studies, and SOTA comparisons. **Secondary:** None. No new datasets, benchmarks, or model weights are released. ## What is the motivation? Prior DLA systems treat sub-tasks (graphical object detection, text region grouping, logical role classification, reading order) as separate problems, each handled by a dedicated model or branch. Multi-branch and multi-stage pipelines introduce cascading errors: mistakes in early stages propagate downstream. They also scale poorly because adding a new task requires adding a new branch or stage. The authors argue for a single unified model that handles all DLA sub-tasks simultaneously, reducing cascading errors and improving scalability. ## What is the novelty? DLAFormer introduces three key ideas: ### 1. Unified Label Space Three relationship types are defined over text-lines and graphical objects: - **Intra-region relationships:** link adjacent text-lines belonging to the same text region (handles text region detection) - **Inter-region relationships:** link pairs of regions with logical connections, e.g., a table and its caption (handles reading order prediction) - **Logical role relationships:** link each text-line to a predefined logical role query, e.g., "paragraph" or "section heading" (handles logical role classification) These are unified into a single label matrix $M \in \mathbb{Z}^{H \times W}$, where $H$ indexes text-line and graphical object queries and $W$ indexes all queries including logical role queries. Each cell takes one of four values: no relation, intra-region, inter-region, or logical role. ### 2. Type-wise Queries Standard DETR initializes content queries as learnable but semantically opaque vectors. DLAFormer replaces the binary classifier in the Deformable DETR encoder's auxiliary detection head with a multi-class classifier. The predicted category selects a type-specific learnable embedding as the content query, giving queries a concrete "physical meaning" tied to their predicted class (table, figure, formula, etc.). ### 3. Unified Relation Prediction Head A relation prediction module computes pairwise scores: $$ f\_{ij} = FC\_k^q(q\_i) \circ FC\_k^r(q\_j) $$ where $\circ$ denotes the dot product. Separate softmax normalization is applied over text-line/graphical queries ($j \le H$) and logical role queries ($H < j \le W$). A bilinear relation classification module then determines the relation type: $$ p\_{ij} = \text{BiLinear}(FC\_c^q(q\_i), FC\_c^k(q\_j)) $$ Both modules share the same enhanced query representations from the decoder, so all relations are predicted in a single forward pass. ## What experiments were performed? ### Benchmarks - **Comp-HRDoc** (HRDoc-Hard split): 1,000 train / 500 test documents. Tasks: page object detection (Segm. mAP) and reading order prediction (Text Region REDS, Graphical Region REDS). - **DocLayNet**: 69,375 train / 6,489 test / 4,999 val pages across 6 document categories, 11 classes. Task: page object detection (box mAP). ### Baselines - Mask2Former (R18, R50) - Detect-Order-Construct (DOC) (R18, R50) - Mask R-CNN, Faster R-CNN, YOLOv5, DINO (on DocLayNet) ### Key Results **Comp-HRDoc (R50):** | Task | Metric | DOC | DLAFormer | | :--- | :--- | :---: | :---: | | Page Object Detection | Segm. mAP | 86.5 | **89.6** | | Reading Order (Text) | REDS | 94.4 | **96.6** | | Reading Order (Graphical) | REDS | 88.6 | **90.0** | **DocLayNet:** 83.8 mAP, up from 79.6 (DOC) and 76.8 (YOLOv5). This exceeds the reported human inter-annotator agreement range of 82-83. ### Ablations - Removing type-wise queries drops Segm. mAP by 1.4 points (89.6 to 88.2) on Comp-HRDoc. The effect on reading order is uneven: Text Region REDS is unchanged (96.6), but Graphical Region REDS drops by 0.6 points (90.0 to 89.4), suggesting type-wise queries mainly help with graphical region ordering. - Replacing the deformable encoder/decoder with vanilla DETR encoder + DAB-DETR decoder drops Segm. mAP from 89.6 to 86.9. An intermediate configuration (DETR encoder + Deformable decoder) reaches 89.0, indicating both components contribute but the deformable encoder provides the larger gain. ## What are the outcomes/conclusions? DLAFormer demonstrates that unifying DLA sub-tasks into a single relation prediction framework is viable and outperforms multi-branch alternatives. The unified label space is the core enabling idea: it allows a single prediction head to handle detection, classification, and ordering simultaneously. ### Limitations - **Per-class weaknesses on DocLayNet:** DLAFormer does not uniformly dominate all categories. It scores 63.1 mAP on Footnote versus 77.2 for YOLOv5, and also trails DOC or YOLOv5 on List-item, Page-footer, Text, and Table. The authors attribute the Footnote gap to the long-tail problem: text-lines inside graphical objects are visually similar to standard text-lines, confusing the classifier. Multi-stage methods like DOC can filter these using predicted graphical object boxes first. - **Requires OCR text-line bounding boxes as input.** The model is described as "purely vision-based" (no text embeddings), but it requires a PDF parser or OCR engine to provide text-line bounding boxes. This is not a zero-dependency vision model. - **Single-page only.** The framework handles page-level tasks; cross-page hierarchical analysis (e.g., table of contents extraction, multi-page reading order) is left to future work. - **No code or weights released.** Reproducibility is limited to the paper description. - **Logit-adjusted loss on DocLayNet.** The authors use logit-adjusted softmax cross-entropy to handle the long-tail distribution, but the improvement from this specific choice is not ablated. ## Reproducibility ### Models - Architecture: Deformable DETR with 3 encoder layers + 3 decoder layers. - Hidden dimension: 256; feedforward dimension: 1,024; 8 attention heads. - Backbone: ResNet-50 (ImageNet pre-trained) or ResNet-18. - Multi-scale feature maps: $C\_3, C\_4, C\_5$ (1/8, 1/16, 1/32). - Relation prediction/classification FC layers: 1,024 nodes each. - Top-K encoder features for query selection: 50 (Comp-HRDoc), 100 (DocLayNet). - No weights or checkpoints released. ### Algorithms - Optimizer: AdamW ($\beta\_1 = 0.9$, $\beta\_2 = 0.999$, $\epsilon = 10^{-8}$). - Backbone LR: $10^{-5}$; transformer LR: $10^{-4}$; weight decay: $10^{-4}$. - Batch size: 4 per GPU (effective global batch size: 64 across 16 GPUs). - Epochs: 40 (Comp-HRDoc), 24 (DocLayNet). - Multi-scale training: shorter side randomly in [320, 416, 512, 608, 704, 800], longer side capped at 1,024. - Test resolution: shorter side = 512. - Loss: softmax cross-entropy for all relation prediction/classification heads. Logit-adjusted variant used on DocLayNet for class imbalance. - Shared unified relation prediction head at each decoder layer during training. - Loss weights/balancing across detection, relation prediction, and relation classification heads are not specified. Hungarian matching cost coefficients are inherited from Deformable DETR defaults but not explicitly stated. - Implemented in PyTorch v1.11. ### Data - **Comp-HRDoc:** Based on HRDoc-Hard. 1,000 train + 500 test documents. Publicly available at [github.com/microsoft/CompHRDoc](https://github.com/microsoft/CompHRDoc) under MIT license. OCR files with text-line bounding boxes and reading order provided. Text-lines within graphical objects are pre-filtered. - **DocLayNet:** 80,863 pages total. Publicly available on [HuggingFace](https://huggingface.co/datasets/ds4sd/DocLayNet) under CDLA-Permissive-1.0. OCR files included but contain text-lines within graphical objects (a noted source of difficulty). - No new data released. ### Evaluation - Comp-HRDoc: COCO-style Segmentation mAP for page object detection; REDS for reading order. - DocLayNet: COCO-style box mAP. - Baselines: DOC results replicated by the authors (marked with dagger); other DocLayNet baselines from original papers. - No error bars, confidence intervals, or multi-run statistics reported. - No per-class breakdown of reading order performance. ### Hardware - Training: 16 NVIDIA Tesla V100 GPUs (32 GB each). - No training time, GPU-hours, or inference latency reported. - No information on inference hardware requirements or throughput. ## BibTeX ```bibtex @inproceedings{wang2024dlaformer, title={DLAFormer: An End-to-End Transformer For Document Layout Analysis}, author={Wang, Jiawei and Hu, Kai and Huo, Qiang}, booktitle={International Conference on Document Analysis and Recognition (ICDAR)}, pages={40--57}, year={2024}, series={Lecture Notes in Computer Science}, volume={14807}, publisher={Springer}, doi={10.1007/978-3-031-70546-5_3} } ```