# RoDLA: Benchmarking the Robustness of Document Layout Analysis Models > Research notes on RoDLA, which introduces the first robustness benchmark for document layout analysis with 450K perturbed document images, two new evaluation metrics (mPE and mRD), and a robust DLA model. - Authors: Hunter Heidenreich - Published: 2026-03-14 - Paper: Preprint (arXiv) - Paper: Published (CVPR 2024) - Code: RoDLA - Benchmark: PubLayNet-P / DocLayNet-P / M6Doc-P - Source: Bevaya Labs, --- ## TL;DR RoDLA introduces the first systematic robustness benchmark for Document Layout Analysis (DLA), comprising ~450K perturbed document images across three datasets (PubLayNet-P, DocLayNet-P, M6Doc-P) with 12 perturbation types at 3 severity levels. The authors propose two metrics (mPE for perturbation assessment and mRD for robustness evaluation) and a robust model that integrates channel attention and average pooling into a DINO-based detector to resist perturbation-induced attention shifting. ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Evaluation}}$. The headline contribution is a robustness benchmarking framework: a perturbation taxonomy, two new evaluation metrics (mPE and mRD), and a systematic assessment of 10+ DLA methods under controlled perturbations. The paper's center of gravity is measuring how existing models fail under realistic document corruptions. **Secondary:** $\Psi\_{\text{Resource}}$. The perturbed benchmark datasets (PubLayNet-P, DocLayNet-P, M6Doc-P) and perturbation generation code are released as reusable community assets. **Secondary:** $\Psi\_{\text{Method}}$. The RoDLA model itself is a methodological contribution (channel attention + average pooling in a DINO encoder), though it serves primarily to demonstrate the benchmark's utility. ## What is the motivation? DLA models are typically evaluated on clean digital documents, but real-world documents suffer from scanning artifacts, camera distortions, noise, and other degradations. Prior work had not systematically studied how these perturbations affect DLA performance. The authors observed that existing high-performing models (e.g., SwinDocSegmenter achieving 93.7% mAP on clean PubLayNet) can drop drastically under perturbation (down to 29.2% mAP under speckle noise). This gap between clean-data performance and real-world robustness motivates both the benchmark and the robust model. Existing robustness metrics (borrowed from ImageNet-C) conflate perturbation severity with model robustness because they rely on a single baseline model's performance as reference. The authors argue this introduces metric uncertainty from model randomness. ## What is the novelty? ### Perturbation Taxonomy A hierarchical taxonomy of 12 document-specific perturbations organized into 5 groups, each with 3 severity levels: 1. **Spatial Transformation:** Rotation (P1), Warping (P2), Keystoning (P3) 2. **Content Interference:** Watermark (P4), Complex Background (P5) 3. **Inconsistency Distortion:** Non-uniform Illumination (P6), Ink-bleeding (P7), Ink-holdout (P8) 4. **Blur:** Defocus (P9), Vibration (P10) 5. **Noise:** Speckle (P11), Texture (P12) ### Mean Perturbation Effect (mPE) A model-independent metric for quantifying perturbation severity that combines IQA metrics with degradation scores: $$\text{mPE}\_p = \frac{1}{NMK} \sum\_{s=1}^{N} \left(\sum\_{i=1}^{M} f\_{s,p}^i + \sum\_{g=1}^{K} D\_{s,p}^g\right)$$ where $f\_{s,p}^i$ are IQA scores (MS-SSIM, CW-SSIM) and $D\_{s,p}^g$ is the degradation of a reference model. By combining multiple assessment methods, mPE reduces sensitivity to any single metric's blind spots (e.g., MS-SSIM is insensitive to warping; CW-SSIM is insensitive to defocus at fine severity levels). ### Mean Robustness Degradation (mRD) A robustness metric that normalizes a model's performance drop by the perturbation's inherent difficulty: $$\text{RD}\_p = \frac{1}{N} \sum\_{s=1}^{N} \frac{D\_{s,p}^g}{\text{mPE}\_{s,p}}$$ The overall mRD averages $\text{RD}\_p$ across all 12 perturbation types. Values above 100 indicate the model degrades more than expected; below 100 indicates better-than-expected robustness. Lower is better. ### RoDLA Model Built on DINO with an InternImage backbone, the model adds: - **Channel-wise Attention (CA)** in the encoder: performs self-attention along the channel dimension to aggregate correlated feature channels, reducing attention shifting on perturbed tokens - **Average Pooling Layers (APL)** with a dilation predictor: leverages neighboring token attention to mitigate overemphasis on individual (potentially corrupted) tokens The channel-wise attention operates as: $$Y = \sigma\left(\frac{\text{Softmax}(Q) \cdot \text{Softmax}(K^T)}{\sqrt{d}} \cdot \text{MLP}(V)\right)$$ where $Q, K, V \in \mathbb{R}^{d \times n}$. ## What experiments were performed? ### Benchmark Setup - **Datasets:** PubLayNet (360K pages, 5 classes), DocLayNet (80.8K pages, 11 classes), M6Doc (9K pages, 74 classes) - **Perturbed versions:** Each dataset $\times$ 12 perturbations $\times$ 3 severity levels = ~450K total perturbed images - **Baselines:** 10+ methods including LayoutParser, Faster R-CNN, Mask R-CNN, Cascade R-CNN, DocSegTr, SwinDocSegmenter, DINO, Co-DINO, and multi-modal models (DiT, LayoutLMv3) - **Backbones:** ResNet, ResNeXt, Swin Transformer, DiT, LayoutLMv3, InternImage - **Framework:** All models reproduced in MMDetection with consistent training settings - **Training:** Models trained on clean data only; perturbations applied only at test time - **Metrics:** Clean mAP, P-Avg (average mAP across all perturbations), mRD ### Key Results **PubLayNet-P:** | Method | Clean mAP | P-Avg | mRD | | :--- | :---: | :---: | :---: | | LayoutParser | 89.0 | 58.0 | 212.7 | | Faster R-CNN | 90.2 | 66.2 | 175.5 | | SwinDocSegmenter | 93.7 | 57.1 | 214.4 | | DiT + Cascade R-CNN | 94.5 | 76.7 | 95.8 | | LayoutLMv3 + Cascade R-CNN | 95.1 | 73.6 | 116.2 | | DINO (InternImage) | 95.4 | 69.0 | 120.7 | | **RoDLA** | **96.0** | **70.0** | **116.0** | RoDLA achieves the best clean mAP (96.0%) and the best mRD (116.0) among single-modal methods, with +3.8% P-Avg over Faster R-CNN. Among all methods including multi-modal ones, DiT + Cascade R-CNN achieves the best mRD (95.8), benefiting from document-specific pre-training on IIT-CDIP. **DocLayNet-P:** RoDLA achieves 82.8% clean mAP (+1.9% over DINO baseline), 62.5% P-Avg (+7.1%), and mRD of 135.4. **M6Doc-P:** RoDLA achieves 67.5% clean mAP, 52.9% P-Avg (+12.1%), and mRD of 150.4. ### Ablation Studies - **Backbone:** InternImage outperforms Swin Transformer, but RoDLA narrows the gap (only 0.1% P-Avg difference and 8.2 mRD difference between backbones) - **CA placement:** Encoder-only CA placement is optimal; adding CA to the decoder hurts robustness - **APL placement:** Encoder-only APL is optimal; the combined CA (encoder) + APL (encoder) configuration yields the best mRD of 115.7 - **Multiple perturbations:** Stacking perturbations degrades performance progressively (P-Avg drops from 64.9 with 2 perturbations to 41.9 with 5 on M6Doc-P) ## What are the outcomes/conclusions? **Key findings:** - Existing DLA models are fragile under realistic perturbations, with some experiencing 60%+ mAP drops under severe blur or noise - Document-specific pre-training (DiT, LayoutLMv3 on IIT-CDIP) substantially improves robustness, particularly against texture and noise perturbations - Multi-modal models are not uniformly more robust; LayoutLMv3 shows vulnerability to content interference (watermark, background) despite strong overall numbers - The RoDLA model demonstrates that attention mechanism modifications can improve both clean performance and robustness simultaneously - mPE provides more balanced perturbation assessment than individual IQA metrics, and mRD decouples perturbation difficulty from model robustness measurement **Limitations the authors acknowledge:** - The benchmark focuses on vision-based single-modal DLA models; multi-modal robustness testing is limited - No human-in-the-loop evaluation for deployment readiness - Perturbations are applied only at test time (no robustness-aware training or augmentation strategies explored) **Limitations not discussed:** - The 12 perturbations, while comprehensive, do not cover all real-world degradations (e.g., partial occlusion, page folding, mixed-resolution scanning) - mPE still depends on a baseline model (Faster R-CNN) for the degradation component, partially undermining the claim of model independence - The benchmark uses existing datasets that are predominantly English and scientific/business documents; robustness for non-Latin scripts and diverse document types is not evaluated - Severity levels are somewhat arbitrary; real-world degradation distributions may not align with the L1/L2/L3 bins ## Reproducibility ### Models - RoDLA: DINO-based detector with InternImage backbone (~323M parameters), pre-trained on ImageNet - Channel-wise Attention module and Average Pooling Layers added to the encoder - Pre-trained checkpoints for PubLayNet, DocLayNet, and M6Doc are released via the GitHub repo ### Algorithms - Optimizer: AdamW (lr=2e-4, weight decay=1e-4) - Schedule: Step-based with warmup at epochs {16, 22}, warmup ratio 1e-3 - Training: 24 epochs, batch size 2 per GPU - Augmentation: Horizontal flip (0.5 ratio), random crop (384, 600) - All models trained on clean data only; perturbations applied at test time only - 38 total models trained for the benchmark (24 on PubLayNet, 7 each on DocLayNet and M6Doc) ### Data - PubLayNet-P: PubLayNet test set + 12 perturbations $\times$ 3 levels - DocLayNet-P: DocLayNet test set + 12 perturbations $\times$ 3 levels - M6Doc-P: M6Doc test set + 12 perturbations $\times$ 3 levels - Perturbation generation code and pre-generated benchmarks available via the GitHub repo - Background perturbation (P5) uses images from ILSVRC (ImageNet) ### Evaluation - Primary metrics: mAP (COCO-style), P-Avg (average mAP across all perturbations/levels), mRD - mPE uses MS-SSIM + CW-SSIM + Faster R-CNN degradation as components - All baselines reproduced in MMDetection for fair comparison - No error bars or multiple-run variance reported - Rotation perturbation (P1) causes the largest performance drops across all models due to annotation misalignment at high severity ### Hardware - Training: 4$\times$ NVIDIA A100 (40 GB each), 300 GB CPU memory per node - No GPU-hours or cost estimates reported - No inference latency or throughput measurements ## BibTeX ```bibtex @inproceedings{chen2024rodla, title={RoDLA: Benchmarking the Robustness of Document Layout Analysis Models}, author={Chen, Yufan and Zhang, Jiaming and Peng, Kunyu and Zheng, Junwei and Liu, Ruiping and Torr, Philip and Stiefelhagen, Rainer}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pages={15556--15566}, year={2024}, doi={10.1109/CVPR52733.2024.01473} } ```