# LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding > LayoutLMv2 integrates visual features into the pre-training stage via a ResNeXt-FPN backbone, spatial-aware self-attention, and two new cross-modal alignment objectives, achieving SOTA on six document understanding benchmarks. - Authors: Bevaya AI Team - Published: 2026-03-15 - Paper: Preprint (arXiv) - Paper: Published (ACL 2021) - Model: LayoutLMv2 Base - Model: LayoutLMv2 Large - Code: Code (GitHub) - Source: Bevaya Labs, --- ## TL;DR LayoutLMv2 is the second generation of the LayoutLM family, introducing visual features directly into the pre-training stage (rather than only at fine-tuning time, as in LayoutLMv1). It combines a multi-modal Transformer encoder with a ResNeXt-101 FPN visual backbone, a spatial-aware self-attention mechanism for 2D relative position modeling, and two new pre-training objectives (Text-Image Alignment and Text-Image Matching). It achieved SOTA on six document understanding benchmarks at the time of publication. --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$ The headline contribution is the model architecture and pre-training strategy. The paper introduces three key innovations: (1) early visual integration via a trainable CNN backbone, (2) spatial-aware self-attention with 2D relative position biases, and (3) two new cross-modal pre-training objectives. The majority of the paper describes these components and their ablations. **Secondary:** $\Psi\_{\text{Evaluation}}$ The paper evaluates on six diverse benchmarks (FUNSD, CORD, SROIE, Kleister-NDA, RVL-CDIP, DocVQA) and provides a systematic ablation study isolating the contribution of each component. ## What is the motivation? LayoutLMv1 demonstrated that jointly pre-training text and layout (2D position) embeddings from large-scale unlabeled documents improves downstream document understanding tasks. However, v1 had a significant limitation: visual features from a CNN were only added during fine-tuning, not during pre-training. This meant the model could not learn cross-modal interactions between text, layout, and visual appearance from the 11M unlabeled IIT-CDIP pages. The authors also identified that absolute 2D position embeddings (as used in v1) encode where a token is on the page but not how tokens relate to each other spatially. Two tokens in the same relative arrangement (e.g., a label above a value) at different absolute positions would produce different representations. ## What is the novelty? ### 1. Early Visual Integration Instead of adding visual features only at fine-tuning time, LayoutLMv2 passes the document page image through a **ResNeXt-101 FPN** visual encoder during pre-training. The output feature map is average-pooled to $W \times H = 7 \times 7 = 49$ visual tokens, each projected to the same dimensionality as text tokens. Visual and text tokens are concatenated into a single sequence: $$x\_i^{(0)} = X\_i + l\_i, \quad X = \\{v\_0, \ldots, v\_{WH-1}, t\_0, \ldots, t\_{L-1}\\}$$ where $l\_i$ is the layout (2D position) embedding and $X\_i$ is either a visual or text embedding. The CNN backbone is updated through backpropagation during pre-training, initialized from a Mask R-CNN trained on PubLayNet. ### 2. Spatial-Aware Self-Attention The standard self-attention score $\alpha\_{ij}$ is augmented with learnable relative position biases: $$\alpha'\_{ij} = \alpha\_{ij} + b^{(1D)}\_{j-i} + b^{(2D\_x)}\_{x\_j - x\_i} + b^{(2D\_y)}\_{y\_j - y\_i}$$ where $(x\_i, y\_i)$ are the top-left bounding box coordinates of the $i$-th token. The biases are shared across encoder layers but differ across attention heads, adding minimal parameters while encoding spatial relationships. ### 3. New Pre-training Objectives In addition to Masked Visual-Language Modeling (MVLM, inherited from v1): - **Text-Image Alignment (TIA):** Randomly cover line-level image regions and train the model to predict which text tokens have been covered. This enforces fine-grained spatial correspondence between text and image. Image regions corresponding to MVLM-masked tokens are also covered to prevent visual leakage. - **Text-Image Matching (TIM):** Binary classification on whether the image and text come from the same page. 15% of images are replaced with a different document page; 5% are dropped entirely. ## What experiments were performed? ### Pre-training Pre-trained on 11M pages from IIT-CDIP Test Collection. Text is extracted via OCR (Microsoft Read API). Two model sizes: BASE (200M params, 12 layers, $d=768$) and LARGE (426M params, 24 layers, $d=1024$). Both use the same ResNeXt-101 FPN visual backbone. Text encoder initialized from UniLMv2; visual backbone initialized from a Mask R-CNN trained on PubLayNet. ### Downstream Tasks Six benchmarks spanning four task types: | Task Type | Dataset | Metric | v1 LARGE | v2 LARGE | | :--- | :--- | :--- | ---: | ---: | | Entity Extraction | FUNSD | F1 | 0.7895 | **0.8420** | | Entity Extraction | CORD | F1 | 0.9493 | **0.9601** | | Entity Extraction | SROIE | F1 | 0.9524 | **0.9781**\* | | Entity Extraction | Kleister-NDA | F1 | 0.8340 | **0.8520** | | Classification | RVL-CDIP | Acc | 94.43% | **95.64%** | | Visual QA | DocVQA | ANLS | 0.7295 | **0.8672** | \*SROIE F1 of 0.9781 excludes OCR mismatch errors; the raw F1 is 0.9661. The DocVQA score of 0.8672 uses train+dev fine-tuning with question-generation data augmentation. ### Ablation Study (DocVQA val, BASE size) The ablation isolates each component's contribution: | Setting | ANLS | | :--- | ---: | | LayoutLM (text+layout only, BERT init) | 0.6841 | | + visual module (MVLM only) | 0.6915 | | + TIA | 0.7061 | | + TIM | 0.6955 | | + TIA + TIM | 0.7124 | | + TIA + TIM + spatial-aware attention | 0.7217 | | + UniLMv2 init (final v2 BASE) | 0.7421 | TIA contributes more than TIM. Spatial-aware self-attention adds roughly 1 ANLS point. Better text-side initialization (UniLMv2 over BERT) provides a further 2-point gain. ## What are the outcomes/conclusions? LayoutLMv2 LARGE set new SOTA on all six benchmarks at the time of publication, with particularly large gains on DocVQA (+13.8 ANLS over v1) and FUNSD (+5.3 F1 over v1). The results demonstrate that integrating visual features during pre-training, rather than only at fine-tuning, yields substantial improvements across diverse document understanding tasks. The ablation confirms that each component contributes: visual integration, TIA, TIM, and spatial-aware attention all provide additive gains. The largest single contributor is the visual module with TIA, suggesting that fine-grained text-image spatial alignment is the most impactful pre-training signal. ### Limitations - The model requires OCR as a preprocessing step, inheriting OCR errors. - The visual backbone (ResNeXt-101 FPN) adds computational cost; the 49 visual tokens increase sequence length. - Pre-training on 11M IIT-CDIP pages is computationally expensive (exact GPU-hours not reported). - The CC-BY-NC-SA-4.0 license on released model weights restricts commercial use. - No layout detection experiments are reported; the paper focuses on document understanding (entity extraction, classification, VQA) rather than bounding-box detection. ## Reproducibility ### Models - **LayoutLMv2 BASE:** 200M parameters. 12-layer, 12-head Transformer encoder, $d = 768$. ResNeXt-101 FPN visual backbone. - **LayoutLMv2 LARGE:** 426M parameters. 24-layer, 16-head Transformer encoder, $d = 1024$. Same visual backbone. - Both model checkpoints released on Hugging Face under CC-BY-NC-SA-4.0. - Text encoder initialized from UniLMv2; visual backbone initialized from Mask R-CNN trained on PubLayNet ("MaskRCNN ResNeXt101\_32x8d FPN 3X" from detectron2). ### Algorithms - **Pre-training:** Maximum sequence length $L = 512$. Visual tokens: $7 \times 7 = 49$. MVLM masks 15% of tokens (80% [MASK], 10% random, 10% unchanged). TIA covers 15% of lines. TIM replaces 15% of images and drops 5%. - **Optimizer:** Adam with learning rate $2 \times 10^{-5}$, weight decay $1 \times 10^{-2}$, $(\beta\_1, \beta\_2) = (0.9, 0.999)$. Linear warmup over the first 10% of steps, then linear decay. - **Pre-training schedule:** BASE is trained with batch size 64 for 5 epochs; LARGE is trained with batch size 2048 for 20 epochs on IIT-CDIP. (The ablation study uses 1-epoch runs for efficiency.) - **Fine-tuning:** Task-specific heads on top of encoder outputs. DocVQA additionally benefits from a question-generation data augmentation step. Fine-tuning hyperparameters (learning rates, batch sizes, epochs per task) are not reported in the paper. ### Data - **Pre-training:** IIT-CDIP Test Collection 1.0 (11M document images). Text extracted via Microsoft Read API (a commercial OCR service). The IIT-CDIP dataset requires institutional access and is not freely downloadable. - **Downstream:** FUNSD (149/50 train/test), CORD (800/100/100), SROIE (626/347), Kleister-NDA (254/83/203), RVL-CDIP (320K/4K/4K), DocVQA (39K/5K/5K). ### Evaluation - Entity extraction: entity-level F1 score. - Classification: accuracy. - DocVQA: Average Normalized Levenshtein Similarity (ANLS). - Ablation on DocVQA validation set with BASE models pre-trained for one epoch. - No error bars or multi-seed runs reported. ### Hardware - Not specified in the main paper. Pre-training on 11M pages with a 200M-426M parameter model with a ResNeXt-101 FPN backbone requires substantial GPU resources. The original LayoutLM was pre-trained on 8x V100 GPUs; LayoutLMv2 likely requires similar or greater resources. ## BibTeX ```bibtex @inproceedings{xu2021layoutlmv2, title={LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding}, author={Xu, Yang and Xu, Yiheng and Lv, Tengchao and Cui, Lei and Wei, Furu and Wang, Guoxin and Lu, Yijuan and Florencio, Dinei and Zhang, Cha and Che, Wanxiang and Zhang, Min and Zhou, Lidong}, booktitle={Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)}, pages={2579--2591}, year={2021}, publisher={Association for Computational Linguistics}, doi={10.18653/v1/2021.acl-long.201} } ```