# LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding > LayoutXLM extends LayoutLMv2 to 53 languages via multilingual pre-training on 30M documents, and introduces the XFUND benchmark for multilingual form understanding. - Authors: Bevaya AI Team - Published: 2026-03-15 - Paper: Preprint (arXiv) - Paper: Published (ACL 2022) - Model: LayoutXLM Base - Model: LayoutXLM Large - Dataset: XFUND Dataset - Code: Code (GitHub) - Source: Bevaya Labs, --- ## TL;DR LayoutXLM is the multilingual extension of LayoutLMv2. It uses the same architecture (multimodal Transformer with ResNeXt-FPN visual backbone and spatial-aware self-attention) but pre-trains on 30M documents across 53 languages. The paper also introduces XFUND, a manually labeled multilingual form understanding benchmark in 7 languages. LayoutXLM outperforms multilingual text-only baselines (XLM-R, InfoXLM) on cross-lingual document understanding tasks. --- ## What kind of paper is this? **Dominant:** $\Psi\_{\text{Method}}$. The headline contribution is extending LayoutLMv2 to the multilingual setting with a large-scale cross-lingual pre-training corpus. **Secondary:** $\Psi\_{\text{Resource}}$. The XFUND dataset (1,393 forms across 7 languages with key-value pair annotations) is a meaningful data contribution. **Secondary:** $\Psi\_{\text{Evaluation}}$. Benchmarks on XFUND with zero-shot transfer and multitask fine-tuning settings. ## What is the motivation? Nearly 40% of digital documents on the web are in non-English languages. LayoutLMv2 was pre-trained only on English documents (IIT-CDIP), limiting its cross-lingual capabilities. Multilingual text-only models (mBERT, XLM-R) lack the layout and visual understanding needed for document tasks. Translating documents via machine translation is unsatisfactory because it loses layout structure. ## What is the novelty? 1. **Multilingual pre-training corpus.** 22M multilingual PDFs scraped from Common Crawl across 53 languages, combined with 8M English documents from IIT-CDIP (30M total). The multilingual PDFs are filtered for quality and classified by language using fastText. 2. **Architecture.** Same as LayoutLMv2 (multimodal Transformer + ResNeXt-FPN + spatial-aware self-attention), but initialized from InfoXLM (a multilingual pre-trained model) instead of UniLMv2. Three pre-training objectives: MVLM, TIA, TIM. 3. **XFUND benchmark.** 1,393 manually annotated forms across 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese). 199 forms per language (149 train / 50 test). Two tasks: Semantic Entity Recognition (SER) and Relation Extraction (RE) for key-value pairs. ## What experiments were performed? Three evaluation settings on XFUND: - **Language-specific fine-tuning:** Train and test on the same language. - **Zero-shot transfer:** Fine-tune on English FUNSD, evaluate on each target language. - **Multitask fine-tuning:** Fine-tune on all 8 languages jointly. LayoutXLM BASE and LARGE are compared against XLM-R, InfoXLM, and LayoutLMv2. ## What are the outcomes/conclusions? LayoutXLM outperforms text-only multilingual baselines on both SER and RE tasks. On XFUND SER (language-specific), LayoutXLM BASE achieves an average F1 of 0.8056 across 7 languages, compared to 0.7047 for XLM-R BASE. Zero-shot transfer from English to other languages is feasible but shows substantial performance drops, confirming that layout understanding transfers better than textual semantics across languages. ### Limitations - CC-BY-NC-SA-4.0 license on model weights restricts commercial use. - Requires OCR as preprocessing. - The 30M pre-training corpus is not publicly released. - XFUND covers only form understanding; no layout detection or document classification benchmarks for multilingual evaluation. ## Reproducibility ### Models - **LayoutXLM BASE:** 369M parameters. Initialized from InfoXLM BASE + ResNeXt-101 FPN. - **LayoutXLM LARGE:** 625M parameters. Initialized from InfoXLM LARGE + ResNeXt-101 FPN. ### Data - **Pre-training:** 30M documents (22M multilingual PDFs + 8M IIT-CDIP English). OCR via Microsoft Read API. - **XFUND:** 1,393 forms, 7 languages, manually annotated key-value pairs. ### Algorithms - Optimizer, learning rate, batch size, warmup schedule, and total training steps are not reported in the paper. - Three pre-training objectives are identical to LayoutLMv2: Masked Visual-Language Modeling (MVLM), Text-Image Alignment (TIA), and Text-Image Matching (TIM). - OCR preprocessing uses the commercial Microsoft Read API. ### Evaluation - Semantic Entity Recognition (SER) is evaluated with entity-level F1. - Relation Extraction (RE) is evaluated with pair-level F1. - No error bars or multi-run statistics are reported. ### Hardware - Not specified in the paper. ## BibTeX ```bibtex @inproceedings{xu2022layoutxlm, title={LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding}, author={Xu, Yiheng and Lv, Tengchao and Cui, Lei and Wang, Guoxin and Lu, Yijuan and Florencio, Dinei and Zhang, Cha and Wei, Furu}, booktitle={Findings of the Association for Computational Linguistics: ACL 2022}, pages={3184--3196}, year={2022}, doi={10.18653/v1/2022.findings-acl.253} } ```