# M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis > Research notes on M2Doc, a pluggable multi-modal fusion approach that adds early-fusion (pixel-level) and late-fusion (block-level) modules to existing object detectors for document layout analysis. - Authors: Hunter Heidenreich - Published: 2026-03-14 - Paper: Published (AAAI 2024) - Code: M2Doc - Source: Bevaya Labs, --- ## TL;DR M2Doc is a pluggable multi-modal fusion approach for document layout analysis that adds two lightweight modules to existing object detectors: an early-fusion module that gates pixel-level textual features into backbone outputs, and a late-fusion module that injects block-level BERT embeddings into candidate bounding boxes via IoU matching. Applied to DINO and Cascade Mask R-CNN, it achieves 89.0 mAP on DocLayNet (+11.3 over unimodal DINO) and 69.9 mAP on M6Doc (+1.9). ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$ : The core contribution is a multi-modal fusion architecture designed specifically for document layout analysis. The paper introduces two pluggable fusion modules (early and late), demonstrates them across multiple detector families, and validates with extensive ablations and SOTA comparisons. The majority of the paper describes architectural design choices and ablation experiments. **Secondary:** None. The paper does not introduce new datasets, benchmarks, or evaluation protocols. Code is released, but as a companion to the method, not as an independent infrastructure contribution. ## What is the motivation? Document layout analysis has traditionally been treated as a vision-only object detection problem, even when using multimodal pre-trained models. The authors identify three specific gaps: 1. **Unimodal pipelines from multimodal pre-training**: Models like DiT and LayoutLM undergo multimodal pre-training but are reduced to unimodal (image-only) backbones when fine-tuned for layout detection. The textual modality learned during pre-training is discarded at inference time. 2. **Weak multi-modal baselines**: The primary multi-modal method (VSR) uses a complex two-backbone architecture with Chargrid/Wordgrid/Sentencegrid inputs and Transformer-based relation modeling, yet it performs worse than unimodal detectors on complex logical layout datasets like DocLayNet. 3. **Detector-specific solutions**: Existing DLA enhancements (TransDLANet, SwinDocSegmenter, SelfDocSeg) modify specific detector internals, making them non-transferable across detector families. The authors argue that documents are inherently multimodal (rich text + visual structure) and that layout analysis should exploit textual semantics, particularly for semantically distinct categories like page headers, footers, and section headers. ## What is the novelty? ### Textual Grid Representation M2Doc uses a BERTgrid-style approach to create pixel-aligned textual features. Given a document image $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$ with $N$ OCR-detected words, each word $w\_i$ with bounding box $b\_i$ is encoded by BERT: $$(T\_1, \dots, T\_N) = \text{BERT}(w\_1, \dots, w\_N)$$ The sequential embeddings $T\_i \in \mathbb{R}^{d \times 1}$ (where $d = 768$) are placed into a 2D grid $\mathbf{G} \in \mathbb{R}^{H \times W \times d}$: $$G\_{x,y} = \begin{cases} T\_i, & \text{if } (x, y) \in b\_i \\\\ 0, & \text{otherwise} \end{cases}$$ Unlike VSR's two-backbone approach, M2Doc feeds both the image and the textual grid through a single shared backbone (ResNet), using convolutions to align channel dimensions before the first block. ### Early Fusion (Pixel-Level) After feature extraction, each scale $\theta \in \\{1, 2, 3, 4\\}$ produces visual features $P\_\theta$ and textual features $S\_\theta$. A gated mechanism adaptively weights the textual contribution: $$\alpha\_\theta = \eta(S\_\theta)$$ $$F\_\theta = \text{LayerNorm}(\alpha\_\theta \odot S\_\theta + P\_\theta)$$ where $\eta(\cdot)$ is two $1 \times 1$ convolutions with ReLU and Tanh activations (restricting scores to $(0, 1)$), and $\odot$ is element-wise multiplication. LayerNorm handles the zero-valued regions of the textual grid. ### Late Fusion (Block-Level) After the RPN or Transformer encoder generates candidate bounding boxes $r\_j$, late fusion matches each candidate to OCR boxes via IoU: $$\text{IoU}\_{i,j} = \frac{|r\_j \cap b\_i|}{|r\_j \cup b\_i|}$$ Words with IoU above a threshold are aggregated into a block-level textual feature: $$E\_j = \Gamma(T \cdot J\_j)$$ where $J\_j$ is a binary inclusion vector and $\Gamma$ is an MLP mapping BERT dimensions to the decoder's channel dimensions. For end-to-end detectors (DINO), these features are added to content queries: $$\text{Query}\_j = \text{Query}\_j + \lambda\_1 E\_j$$ For two-stage detectors (Cascade Mask R-CNN), block-level textual features are added to RoI features scaled by $\lambda\_2$ before the R-CNN head. ### Design Insight The authors find that gated fusion works best for early fusion (where textual features are backbone-extracted and need adaptive weighting), while simple summation works best for late fusion (where textual features come directly from the pre-trained language model and are already high-quality). ## What experiments were performed? ### Datasets | Dataset | Pages | Categories | OCR Source | Layout Type | | :--- | :--- | :--- | :--- | :--- | | PubLayNet | 360K | 5 | PDFMiner | Physical | | DocLayNet | 80K | 11 | Human sentence-level | Logical | | M6Doc | 9K | 74 | OCR engine | Logical | ### Detectors Evaluated - **End-to-end**: DINO (DINO-4Scale, 900 queries) - **Two-stage**: Cascade Mask R-CNN - **Pluggability test**: Mask R-CNN, Faster R-CNN, Deformable DETR All use ResNet-101 + FPN as the visual backbone and BERT-Base-Multilingual-Cased as the language model. Detectors are initialized from COCO 2017 pre-trained weights. ### Training Configuration - DocLayNet and M6Doc: 36 epochs - Cascade Mask R-CNN: SGD, lr = 2e-2, decayed to 2e-3 at epoch 27 and 2e-4 at epoch 33 - DINO: AdamW, lr = 1e-4, decayed to 3.3e-5 at epoch 27 and 1e-5 at epoch 33 - PubLayNet: 6 epochs, same initial lr, decayed by 10$\times$ at epoch 5 - Custom anchor scales for Cascade Mask R-CNN: [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2, 5, 10] - Built on MMDetection ### Key Results **DocLayNet** (Table 1): DINO + M2Doc achieves 89.0 mAP, up from 77.7 for unimodal DINO. This is reported as the first method to significantly exceed the human baseline of 83 mAP. Cascade Mask R-CNN + M2Doc reaches 86.7 mAP (+11.1). The largest per-category gains come from semantically distinct classes: Page-footer (+27.3), Page-header (+14.4), Section-header (+18.6), Formula (+17.1). **M6Doc** (Table 2): DINO + M2Doc achieves 69.9 mAP (+1.9). Gains are more modest due to M6Doc's 74 fine-grained categories and diverse input scales limiting recall. **PubLayNet** (Table 3): DINO + M2Doc achieves 95.5 mAP (+0.1 over unimodal DINO). Minimal improvement because PubLayNet's 5-category physical layout is already near-saturated and semantically simple. ### Ablations (M6Doc) - **Module ablation** (Table 4): Both early and late fusion contribute. Together they yield +1.9 mAP for DINO and +2.1 mAP for Cascade Mask R-CNN. Individual modules contribute roughly equally (+0.4 to +0.5 for DINO). - **Fusion strategy ablation** (Table 5): Gate mechanism is best for early fusion; summation is best for late fusion. Concatenation and other combinations underperform. - **Pluggability** (Table 6): All five detectors improve with M2Doc. Gains range from +1.6 mAP (Deformable DETR) to +2.9 mAP (Mask R-CNN). The authors note that their reproduced baselines are higher than M6Doc's originally reported numbers due to different experimental settings, so absolute mAP values should be compared with that caveat in mind. ### Notable Observation The only category where multimodal fusion hurts performance is "Figure" (which mostly lacks text content). Text-overlaid pictures can also degrade boundary detection for multimodal models. ## What are the outcomes/conclusions? ### Strengths - The pluggable design is detector-agnostic, demonstrated across five architectures spanning two-stage and end-to-end families. - The DocLayNet results (+11.3 mAP) are substantial and indicate that multimodal fusion addresses a real gap in logical layout analysis where semantic categories require textual understanding. - The single-backbone design is simpler and more parameter-efficient than VSR's two-backbone approach. - Ablations are thorough, testing both module contributions and fusion strategy alternatives. ### Limitations - **OCR dependency**: The method requires pre-computed OCR results (word boxes + text). DocLayNet provides human-annotated OCR, but real-world OCR introduces noise. The paper does not evaluate robustness to OCR errors. - **Marginal gains on simple datasets**: On PubLayNet (+0.1 mAP), the multimodal overhead (BERT inference + fusion modules) provides almost no benefit, suggesting the approach is only worthwhile for complex logical layout tasks. - **No latency or throughput analysis**: Adding BERT inference and fusion modules has a computational cost that is never quantified. Inference time comparisons with unimodal baselines are absent. - **Missing comparison with pre-trained multimodal models**: The paper compares against unimodal detectors and VSR, but does not compare with LayoutLMv3 or DiT fine-tuned for DLA, which also leverage multimodal features (through pre-training). LayoutLMv3 appears in the PubLayNet table but not DocLayNet or M6Doc. - **Figure category degradation**: The authors acknowledge that multimodal fusion hurts text-overlaid image detection but offer no mitigation strategy. ## Reproducibility ### Models - Visual backbone: ResNet-101 + FPN, initialized from COCO 2017 pre-trained weights. - Language model: BERT-Base-Multilingual-Cased (110M params), from HuggingFace. - Early fusion: two $1 \times 1$ conv layers with ReLU + Tanh per feature scale ($\theta \in \\{1,2,3,4\\}$). - Late fusion: IoU matching + MLP mapping BERT dim (768) to detector channel dim. - Pre-trained weights for R50 and R101 variants (4 checkpoints: Cascade Mask R-CNN and DINO, each with R50 and R101) available via BaiduNetDisk and Google Drive (per the GitHub repo). Weights are only released for DocLayNet; no PubLayNet or M6Doc checkpoints are provided. Full training and inference code is provided, including multi-GPU scripts. ### Algorithms - Cascade Mask R-CNN: SGD, lr = 2e-2, step decay at epochs 27 and 33, 36 epochs. - DINO: AdamW, lr = 1e-4, step decay at epochs 27 and 33, 36 epochs. - PubLayNet: 6 epochs, lr decayed by 10$\times$ at epoch 5. - IoU threshold for late fusion: not stated in the paper text. Config files in the GitHub repo (`mmdetection/m2doc_config/`) likely contain the actual values used. - $\lambda\_1$ and $\lambda\_2$: described as adjustable hyperparameters; exact values not stated in the paper text but may be recoverable from the repo config files. - Framework: MMDetection. - Environment: Python 3.8.0, CUDA 10.2, PyTorch 1.8.1, mmcv, mmengine, transformers. ### Data - PubLayNet: 360K pages, word-level OCR from PDFMiner. Publicly available (CDLA-Permissive-1.0). - DocLayNet: 80K pages, human sentence-level OCR annotations included. Publicly available (CDLA-Permissive-1.0). - M6Doc: 9K pages, sentence-level OCR obtained via OCR engine (unspecified which). Restricted access (CC BY-NC-ND 4.0); requires application and approval from HCIILAB. - Standard train/val/test splits used for all three datasets. ### Evaluation - Metric: COCO-style mAP @ IoU [0.50:0.95:0.05]. - Per-category AP reported for DocLayNet (11 classes) and PubLayNet (5 classes). - AP50, AP75, and Recall additionally reported for M6Doc. - No error bars, significance tests, or multi-run statistics. - Baselines reproduced in the same MMDetection framework with consistent settings. ### Hardware - Not explicitly reported in the paper. The GitHub repo's distributed training script (`dist_train.sh`) defaults to 8 GPUs, suggesting an 8-GPU setup was used, but GPU type, training time, and memory requirements are unspecified. - No inference speed benchmarks. ## BibTeX ```bibtex @inproceedings{zhang2024m2doc, title={M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis}, author={Zhang, Ning and Cheng, Hiuyi and Chen, Jiayu and Jiang, Zongyuan and Huang, Jun and Xue, Yang and Jin, Lianwen}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, volume={38}, number={7}, pages={7233--7241}, year={2024}, doi={10.1609/aaai.v38i7.28552} } ```